Concept:
- A factor of the form $1-2\cos\theta$ can be rewritten as a ratio of cosines, $1-2\cos\theta = -\dfrac{\cos(3\theta/2)}{\cos(\theta/2)}$, using only the double angle formula $2\sin^2(\theta/2) = 1-\cos\theta$.
- Here the five angles $\pi/11, 3\pi/11, 9\pi/11, 27\pi/11, 81\pi/11$ each equal 3 times the one before, which is exactly the pattern the identity above needs: the "$3\theta/2$" of one factor becomes the "$\theta/2$" of the next, so the whole product telescopes.
Step 1: Establish the identity $1-2\cos\theta = -\dfrac{\cos(3\theta/2)}{\cos(\theta/2)}$
Expand $\cos(3\theta/2) = \cos(\theta/2+\theta) = \cos(\theta/2)\cos\theta - \sin(\theta/2)\sin\theta$.
Since $\sin\theta = 2\sin(\theta/2)\cos(\theta/2)$, the second term becomes $\sin(\theta/2)\sin\theta = 2\sin^2(\theta/2)\cos(\theta/2) = (1-\cos\theta)\cos(\theta/2)$.
So $\cos(3\theta/2) = \cos(\theta/2)\cos\theta - (1-\cos\theta)\cos(\theta/2) = \cos(\theta/2)(2\cos\theta-1)$, which gives $-\dfrac{\cos(3\theta/2)}{\cos(\theta/2)} = 1-2\cos\theta$, as required.
Step 2: Write each factor of $\alpha$ using this identity
Let $\theta_j = 3^j\dfrac{\pi}{11}$ for $j=0,1,2,3,4$, so $\theta_0=\pi/11,\ \theta_1=3\pi/11,\ \theta_2=9\pi/11,\ \theta_3=27\pi/11,\ \theta_4=81\pi/11$, matching the problem's five factors in order.
Since $3\theta_j = \theta_{j+1}$, each factor becomes $1-2\cos\theta_j = -\dfrac{\cos(\theta_{j+1}/2)}{\cos(\theta_j/2)}$.
Step 3: Multiply all five factors and cancel the telescoping terms
$\alpha = \displaystyle\prod_{j=0}^{4}\left(-\dfrac{\cos(\theta_{j+1}/2)}{\cos(\theta_j/2)}\right) = (-1)^5\dfrac{\cos(\theta_1/2)\cos(\theta_2/2)\cos(\theta_3/2)\cos(\theta_4/2)\cos(\theta_5/2)}{\cos(\theta_0/2)\cos(\theta_1/2)\cos(\theta_2/2)\cos(\theta_3/2)\cos(\theta_4/2)}$
Every term from $\cos(\theta_1/2)$ to $\cos(\theta_4/2)$ appears once on top and once on the bottom, so they all cancel, leaving only the first and last:
$\alpha = -\dfrac{\cos(\theta_5/2)}{\cos(\theta_0/2)}$, where $\theta_5 = 3^5\dfrac{\pi}{11} = \dfrac{243\pi}{11}$ and $\theta_0 = \dfrac{\pi}{11}$.
Step 4: Reduce $\theta_5/2$ to a standard angle
$\dfrac{\theta_5}{2} = \dfrac{243\pi}{22}$. Since $\cos$ has period $2\pi = \dfrac{44\pi}{22}$, and $243 = 5(44) + 23$, this angle is the same as $\dfrac{23\pi}{22}$.
Write $\dfrac{23\pi}{22} = \pi + \dfrac{\pi}{22}$, so $\cos\left(\dfrac{243\pi}{22}\right) = \cos\left(\pi+\dfrac{\pi}{22}\right) = -\cos\left(\dfrac{\pi}{22}\right)$.
Step 5: Substitute back and finish
$\alpha = -\dfrac{-\cos(\pi/22)}{\cos(\pi/22)} = 1$, so $\alpha^2 = 1$.
Therefore $5-\alpha^2 = 5-1 = 4$.
Final Answer: $4$