At first you've to notice that for F(n) be divisible by P, at least one part of the multiplication (n — i²) must be a multiple of P. So we have that n — i² (mod P) = 0 (mod P), so n (mod P) = i² (mod P). It's clear that i must be less than sqrt(n), so we can iterate over all possible i's and for each of them count how many n's exists that are less than some integer A. After some math it's clear that counting this is equal to (A — i²)/P. That's not the complete solution, since we're overcalculating some n's, and this occurs when we have some positive integer j different from i and less than sqrt(n) such that j² (mod P) = i² (mod P). Using an array or a set to mark for each i if i² module P has been already visited is enough.

Just calculate the answer for A being hi and subtract it from A being lo — 1 and you're done.

P.S: To better explain why (A — i²)/P, we know that n (mod P) = i² (mod P) and we can express it as n = i² + P*k. We know that n must be less than A, so i² + P*k < A, so k is actually the number of different solutions. Isolating k, we have:

P*k < A — i²

k < (A — i²)/P

k = floor((A — i²)/P)

My code: http://pastie.org/private/gltsff0nslzgn6cyqdexa