I was thinking of a way to solve this problem in O(1) and I am not getting the right answer..

My logic was that..

the answer is (a+b)(a+1)(b+1)/2 and the equation is x+my=mc(I changed b to c for convinience..)

so upon applying the am>=gm inequality in the following way..

((a+b)+(a+1)+(2*m-1)(b+1)/3>=((2*m-1)^1/3)*answer

so answer would be floor(LHS/(2*m-1)^1/3) and LHS is (2*m*c+2*m)/3 as (a,b) lies on the line..

but I was getting a wrong answer..

Sorry if I missed something obvious and any help would be really appreciated..