Let \(C\) be a curve \(a x^2+2 h x y+b y^2+2 g x+2 f y+c=0\) in \(a\) cartesian plane. By rotating the coordinate axes through an angle \(\frac{\pi}{4}\) in the positive direction, if the transformed equation of \(C\) is \(Y^2+X Y-X=0\), then \(\left(h^2-a b\right)-2 g f=\)

Apply the relevant identity from the chapter and substitute the values established in the previous step, keeping the original constraint in view.

Use the simplified expression to isolate the unknown, then plug the result back into the question setup to confirm the working is internally consistent.

Cross-check against a boundary or limiting case. Both the dimensional balance and the qualitative behaviour at the extreme should agree with the candidate answer.

Combine the steps above to identify the correct option. The remaining choices each fail at least one of the consistency, dimensional, or boundary checks.