Step 1: Concept
For a rotation $\theta$, the transformations are $x = X\cos\theta - Y\sin\theta$ and $y = X\sin\theta + Y\cos\theta$.
Step 2: Analysis
Given $\tan\theta = 2$, we have $\cos\theta = \frac{1}{\sqrt{5}}$ and $\sin\theta = \frac{2}{\sqrt{5}}$. Thus, $x = \frac{X-2Y}{\sqrt{5}}$ and $y = \frac{2X+Y}{\sqrt{5}}$.
Step 3: Evaluation
Substituting these into $4xy - 3x^{2} = a^{2}$:
$4(\frac{X-2Y}{\sqrt{5}})(\frac{2X+Y}{\sqrt{5}}) - 3(\frac{X-2Y}{\sqrt{5}})^{2} = a^{2}$.
$\frac{4(2X^{2} - 3XY - 2Y^{2})}{5} - \frac{3(X^{2} - 4XY + 4Y^{2})}{5} = a^{2}$.
Step 4: Conclusion
Simplifying gives $5X^{2} - 20Y^{2} = 5a^{2}$, which reduces to $X^{2} - 4Y^{2} = a^{2}$.
Final Answer: (a)