Concept:
When axes are rotated through an angle \(\theta\), the new coordinates are related to the old coordinates by:
\[
x=x'\cos\theta-y'\sin\theta,
\]
\[
y=x'\sin\theta+y'\cos\theta.
\]
The purpose of rotation is usually to eliminate the \(xy\)-term from the equation.
Step 1: Identify coefficients of the quadratic equation.
Given equation:
\[
x^2+2\sqrt3xy-y^2=2a^2.
\]
Comparing with the standard form
\[
Ax^2+2Hxy+By^2=0,
\]
we obtain
\[
A=1,\qquad H=\sqrt3,\qquad B=-1.
\]
Step 2: Find angle of rotation.
The angle \(\theta\) required to eliminate the \(xy\)-term is determined by
\[
\tan2\theta=\frac{2H}{A-B}.
\]
Substituting values,
\[
\tan2\theta=\frac{2\sqrt3}{1-(-1)}
=\frac{2\sqrt3}{2}
=\sqrt3.
\]
Therefore,
\[
2\theta=60^\circ
\]
and hence
\[
\theta=30^\circ.
\]
This matches the given rotation.
Step 3: Find transformed coefficients.
The transformed coefficients are:
\[
A'=A\cos^2\theta+2H\sin\theta\cos\theta+B\sin^2\theta,
\]
\[
B'=A\sin^2\theta-2H\sin\theta\cos\theta+B\cos^2\theta.
\]
Using
\[
\cos30^\circ=\frac{\sqrt3}{2},
\qquad
\sin30^\circ=\frac12,
\]
we get
\[
A'
=1\cdot\frac34
+2\sqrt3\cdot\frac{\sqrt3}{4}
-1\cdot\frac14.
\]
Thus,
\[
A'=\frac34+\frac{6}{4}-\frac14
=\frac{8}{4}
=2.
\]
Similarly,
\[
B'
=1\cdot\frac14
-2\sqrt3\cdot\frac{\sqrt3}{4}
-1\cdot\frac34.
\]
Hence,
\[
B'=\frac14-\frac64-\frac34
=-2.
\]
Therefore transformed equation becomes
\[
2x'^2-2y'^2=2a^2.
\]
Dividing throughout by \(2\),
\[
x'^2-y'^2=a^2.
\]
Hence the required transformed equation is
\[
\boxed{x'^2-y'^2=a^2}.
\]