If \(\left(\tan ^{-1} x\right)^2+\left(\cot ^{-1} x\right)^2=\frac{5 \pi^2}{8}\), then the value of \(x\) is

\(\left(\tan ^{-1} x\right)^2+\left(\cot ^{-1} x\right)^2=\frac{5 \pi^2}{8}\)
\(\Rightarrow\left(\tan ^{-1} x+\cot ^{-1} x\right)^2-2 \tan ^{-1} x \cot ^{-1} x=\frac{5 \pi^2}{8}\)
\(\Rightarrow\left(\tan ^{-1} x+\cot ^{-1} x\right)^2\)
\(-2 \tan ^{-1} x\left(\frac{\pi}{2}-\tan ^{-1} x\right)=\frac{5 \pi^2}{8}\)
\(\begin{aligned} & \Rightarrow \frac{\pi^2}{4}-2 \times \frac{\pi}{2} \tan ^{-1} x+2\left(\tan ^{-1} x\right)^2=\frac{5 \pi^2}{8} \\ & \Rightarrow 2\left(\tan ^{-1} x\right)^2-\pi \tan ^{-1} x-\frac{3 \pi^2}{8}=0 \\ & \Rightarrow \tan ^{-1} x=-\frac{\pi}{4}, \frac{3 \pi}{4} \\ & \Rightarrow \tan ^{-1} x=-\frac{\pi}{4} \Rightarrow x=-1\end{aligned}\)