Two adjacent sides of a parallelogram $ABCD$ are given by $\overrightarrow{AB} = 2\hat{i} + 10\hat{j} + 11\hat{k}$ and $\overrightarrow{AD} = -\hat{i} + 2\hat{j} + 2\hat{k}$ The side $AD$ is rotated by an acute angle $\alpha$ in the plane of the parallelogram so that $AD$ becomes $AD'$. If $AD'$ makes a right angle with the side $AB$, then the cosine of the angle $\alpha$ is given by

$\text{Given: } \overrightarrow{AB} = 2\hat{i} + 10\hat{j} + 11\hat{k}$ $\overrightarrow{AD} = -\hat{i} + 2\hat{j} + 2\hat{k}$ $\overrightarrow{AD} \ \xrightarrow{\ \text{rotate by acute } \alpha\ } \ \overrightarrow{AD'}$ $\overrightarrow{AD'} \perp \overrightarrow{AB} \qquad \text{To find: } \cos\alpha$ $\text{Parallelogram } ABCD \text{ and the plane it lies in}$ $\vec{a} = \overrightarrow{AB}, \quad \vec{b} = \overrightarrow{AD}, \quad \angle(\vec{a}, \vec{b}) = \theta$ $\overrightarrow{AD} \;\to\; \overrightarrow{AD'} \perp \overrightarrow{AB} \qquad (\text{rotation in the same plane})$ $\text{Plan: } 1.\ |\vec{a}|, \ |\vec{b}|$ $2.\ \cos\theta = \dfrac{\vec{a}\cdot\vec{b}}{|\vec{a}||\vec{b}|} \qquad 3.\ \cos\alpha = \sin\theta$ $|\vec{a}| = \sqrt{2^2 + 10^2 + 11^2} = \sqrt{4 + 100 + 121} = \sqrt{225} = 15$ $|\vec{b}| = \sqrt{(-1)^2 + 2^2 + 2^2} = \sqrt{1 + 4 + 4} = \sqrt{9} = 3$ $\vec{a}\cdot\vec{b} = (2)(-1) + (10)(2) + (11)(2) = -2 + 20 + 22 = 40$ $\cos\theta = \dfrac{\vec{a}\cdot\vec{b}}{|\vec{a}||\vec{b}|} = \dfrac{40}{15 \times 3} = \dfrac{40}{45} = \dfrac{8}{9}$ $\overrightarrow{AD'} \perp \overrightarrow{AB} \ \text{ in the plane} \quad \Rightarrow \quad \alpha = 90^\circ - \theta$ $\cos\alpha = \cos(90^\circ - \theta) = \sin\theta$ $\sin\theta = \sqrt{1 - \cos^2\theta} = \sqrt{1 - \left(\dfrac{8}{9}\right)^2}$ $= \sqrt{\dfrac{81 - 64}{81}} = \sqrt{\dfrac{17}{81}}$ $\cos\alpha = \sin\theta = \dfrac{\sqrt{17}}{9}$ $\cos\alpha = \dfrac{\sqrt{17}}{9}$ $\therefore \text{Option } (2)$