After the coordinate axes are rotated through an angle \(\frac{\pi}{4}\) in the anti clockwise direction without shifting the origin, if the equation \(x^2+y^2-2 x-4 y-20=0\) transforms to \(a x^2+2 h x y+b y^2+2 g x+2 f y+c=0\) in the new coordinate system, then \(\left|\begin{array}{lll}a & h & g \\ h & b & f \\ g & f & c\end{array}\right|=\)

Apply the relevant identity from the chapter and substitute the values established in the previous step, keeping the original constraint in view.

Use the simplified expression to isolate the unknown, then plug the result back into the question setup to confirm the working is internally consistent.

Cross-check against a boundary or limiting case. Both the dimensional balance and the qualitative behaviour at the extreme should agree with the candidate answer.

Combine the steps above to identify the correct option. The remaining choices each fail at least one of the consistency, dimensional, or boundary checks.