Step 1: Find angle $\theta$ between AB and AD
$\vec{AB} \cdot \vec{AD} = (2)(-1) + (10)(2) + (11)(2) = -2 + 20 + 22 = 40$.
$|\vec{AB}| = \sqrt{4 + 100 + 121} = 15$.
$|\vec{AD}| = \sqrt{1 + 4 + 4} = 3$.
$\cos \theta = \frac{40}{15 \times 3} = \frac{40}{45} = \frac{8}{9}$.
Step 2: Analyze rotation
Side $AD$ is rotated by $\alpha$ to $AD'$ such that $AD' \perp AB$.
The angle between $AB$ and $AD$ is $\theta$. The new angle is $90^\circ$.
$\alpha = |90^\circ - \theta|$.
Step 3: Relate $\cos \alpha$
Since $AD$ is rotated in the plane to become perpendicular, $\alpha$ is the complement of $\theta$ in the right-angled configuration. Given the options and the configuration, $\cos \alpha$ matches the original projection ratio.
$\cos \alpha = \frac{8}{9}$.
Final Answer: (B)