Step 1: Write the given quadratic equation in standard form.
The equation is
\[
3x^2+\sqrt3xy-5=0.
\]
Comparing with
\[
Ax^2+Bxy+Cy^2+D=0,
\]
we obtain
\[
A=3,\qquad B=\sqrt3,\qquad C=0.
\]
Step 2: Use the invariant property under rotation of axes.
Rotation of axes changes only the orientation of the coordinate system. The eigenvalues of the quadratic form remain unchanged.
The transformed equation is
\[
ax'^2+by'^2=10,
\]
or
\[
ax'^2+by'^2-10=0.
\]
Hence,
\[
ab
=
\lambda_1\lambda_2,
\]
where \(\lambda_1,\lambda_2\) are the eigenvalues of the quadratic form.
Now,
\[
\lambda_1\lambda_2
=
AC-\left(\frac{B}{2}\right)^2.
\]
Substituting,
\[
ab
=
3(0)-\left(\frac{\sqrt3}{2}\right)^2
=
-\frac34.
\]
Since the constant changes from \(5\) to \(10\), the coefficients are multiplied by \(4\).
Therefore,
\[
ab
=
4\left(-\frac34\right)
=
-3.
\]
\[
\boxed{ab=-3.}
\]
Hence, the correct option is \(\boxed{(B)}\).