We basically need to find the length of the common part between a line AB and a polygon P.

First for each line XY in polygon P, compute the intersection point between AB and XY.

These are the points where the line either enters the polygon or leaves the polygon.

Sort these points according to their order on the line (based on their distance from A or distance from B).

Your answer is the sum of distance between each Entering point and its immediate next Leaving Point in this order.

This is the basic idea. But the implementation can be quite tricky, since vertex points may occur twice as intersection points and 180-degree angles are allowed in the input. If you can't figure it out on your own, just look into someone else's code.