Step 1: Write the given vectors and find their angle.
We are given:
\[
\overrightarrow{PS}=\hat{i}+\hat{j}
\]
and
\[
\overrightarrow{PQ}=-\hat{j}+\hat{k}
\]
Now, let the angle between \( \overrightarrow{PS} \) and \( \overrightarrow{PQ} \) be \( \theta \). Then:
\[
\cos \theta=\frac{\overrightarrow{PS}\cdot \overrightarrow{PQ}}{|\overrightarrow{PS}|\,|\overrightarrow{PQ}|}
\]
First, calculate the dot product:
\[
(\hat{i}+\hat{j})\cdot(-\hat{j}+\hat{k})=0-1+0=-1
\]
Now, calculate the magnitudes:
\[
|\overrightarrow{PS}|=\sqrt{1^2+1^2}=\sqrt{2}
\]
\[
|\overrightarrow{PQ}|=\sqrt{(-1)^2+1^2}=\sqrt{2}
\]
So:
\[
\cos \theta=\frac{-1}{\sqrt{2}\cdot \sqrt{2}}=\frac{-1}{2}
\]
Hence:
\[
\theta=120^\circ
\]
Step 2: Find the angle of rotation \( \alpha \).
Initially, the angle between \( \overrightarrow{PS} \) and \( \overrightarrow{PQ} \) is \(120^\circ\). We need to rotate \( \overrightarrow{PS} \) so that it becomes perpendicular to \( \overrightarrow{PQ} \), that is, the new angle should be \(90^\circ\).
Therefore, the required minimum rotation is:
\[
\alpha=120^\circ-90^\circ=30^\circ
\]
Step 3: Evaluate the required expression.
We need to find:
\[
\sin^2\frac{5\alpha}{2}-\sin^2\frac{\alpha}{2}
\]
Substitute \( \alpha=30^\circ \):
\[
\sin^2\frac{5\cdot 30^\circ}{2}-\sin^2\frac{30^\circ}{2}
\]
\[
=\sin^2 75^\circ-\sin^2 15^\circ
\]
Now use the identity:
\[
\sin^2 A-\sin^2 B=\sin(A+B)\sin(A-B)
\]
So:
\[
\sin^2 75^\circ-\sin^2 15^\circ=\sin(90^\circ)\sin(60^\circ)
\]
\[
=1\cdot \frac{\sqrt{3}}{2}
\]
\[
=\frac{\sqrt{3}}{2}
\]
Step 4: Conclusion.
Thus, the value of \( \left( \sin^2 \frac{5\alpha}{2}-\sin^2 \frac{\alpha}{2} \right) \) is \( \frac{\sqrt{3}}{2} \).
Final Answer: \( \frac{\sqrt{3}}{2} \)