The unit vector which is orthogonal to the vector \(5 \hat{i}+2 \hat{j}+6 \hat{k}\) and is coplanar with the vectors \(2 \hat{i}+\hat{j}+\hat{k}\) and \(\hat{i}-\hat{j}+\hat{k}\) is

Let \(\overline{\mathrm{a}}=5 \hat{\mathrm{i}}+2 \hat{\mathrm{j}}+6 \hat{\mathrm{k}}, \overline{\mathrm{b}}=2 \hat{\mathrm{i}}+\hat{\mathrm{j}}+\hat{\mathrm{k}}\) and \(\bar{c}=\hat{i}-\hat{j}+\hat{k}\)
Then, required unit vectors are given by
\(\bar{\alpha}= \pm \frac{\bar{a} \times(\bar{b} \times \bar{c})}{|\overline{\mathrm{a}} \times(\bar{b} \times \bar{c})|}\)
Now, \(\overline{\mathrm{a}} \times(\overline{\mathrm{b}} \times \overline{\mathrm{c}})=(\overline{\mathrm{a}} \cdot \overline{\mathrm{c}}) \overline{\mathrm{b}}-(\overline{\mathrm{a}} \cdot \overline{\mathrm{b}}) \overline{\mathrm{c}}\)
\(\begin{aligned}
& =9(2 \hat{i}+\hat{j}+\hat{k})-18(\hat{i}-\hat{j}+\hat{k}) \\
& =27 \hat{j}-9 \hat{k}
\end{aligned}\)
\(\therefore \quad|\overline{\mathrm{a}} \times(\overline{\mathrm{b}} \times \overline{\mathrm{c}})|=\sqrt{729+81}=\sqrt{810}=9 \sqrt{10}\)
Hence, required unit vectors are
\(\bar{\alpha}= \pm \frac{27 \hat{\mathrm{j}}-9 \hat{\mathrm{k}}}{9 \sqrt{10}}= \pm \frac{3 \hat{\mathrm{j}}-\hat{\mathrm{k}}}{\sqrt{10}}\)