Concept:
This question is based on two concepts:
\[
\text{Sum of roots and product of roots of a quadratic equation}
\]
and
\[
\text{Geometric Progression}
\]
For a quadratic equation:
\[
x^2-Sx+P=0
\]
if its roots are \(r_1\) and \(r_2\), then:
\[
r_1+r_2=S
\]
and
\[
r_1r_2=P
\]
Also, if four numbers are in G.P., then they can be written as:
\[
k,\ kr,\ kr^2,\ kr^3
\]
where \(r\) is the common ratio.
Step 1: Use the first quadratic equation.
Given:
\[
x^2-3x+a=0
\]
Its roots are:
\[
\alpha,\beta
\]
Therefore, by sum of roots:
\[
\alpha+\beta=3
\]
and by product of roots:
\[
\alpha\beta=a
\]
Step 2: Use the second quadratic equation.
Given:
\[
x^2-12x+b=0
\]
Its roots are:
\[
\gamma,\delta
\]
Therefore:
\[
\gamma+\delta=12
\]
and:
\[
\gamma\delta=b
\]
Step 3: Write \(\alpha,\beta,\gamma,\delta\) as an increasing G.P.
Since \(\alpha,\beta,\gamma,\delta\) in order form an increasing G.P., let:
\[
\alpha=k
\]
\[
\beta=kr
\]
\[
\gamma=kr^2
\]
\[
\delta=kr^3
\]
where \(r>1\), because the G.P. is increasing.
Step 4: Use the sum of first two roots.
Since:
\[
\alpha+\beta=3
\]
we get:
\[
k+kr=3
\]
\[
k(1+r)=3
\]
Step 5: Use the sum of next two roots.
Since:
\[
\gamma+\delta=12
\]
we get:
\[
kr^2+kr^3=12
\]
Taking \(kr^2\) common:
\[
kr^2(1+r)=12
\]
Step 6: Divide the two equations.
We have:
\[
k(1+r)=3
\]
and:
\[
kr^2(1+r)=12
\]
Now divide the second equation by the first equation:
\[
\frac{kr^2(1+r)}{k(1+r)}=\frac{12}{3}
\]
After cancellation:
\[
r^2=4
\]
\[
r=2
\]
Since the G.P. is increasing:
\[
r=2
\]
Step 7: Find the value of \(k\).
Using:
\[
k(1+r)=3
\]
Substitute:
\[
r=2
\]
\[
k(1+2)=3
\]
\[
3k=3
\]
\[
k=1
\]
Step 8: Find the four roots.
The G.P. terms are:
\[
k,\ kr,\ kr^2,\ kr^3
\]
Substitute \(k=1\) and \(r=2\):
\[
\alpha=1
\]
\[
\beta=2
\]
\[
\gamma=4
\]
\[
\delta=8
\]
So:
\[
\alpha,\beta,\gamma,\delta=1,2,4,8
\]
Step 9: Find \(a\).
From the first quadratic equation:
\[
a=\alpha\beta
\]
\[
a=1\times 2
\]
\[
a=2
\]
Step 10: Find \(b\).
From the second quadratic equation:
\[
b=\gamma\delta
\]
\[
b=4\times 8
\]
\[
b=32
\]
Hence, the correct answer is:
\[
\boxed{(C)\ a=2,b=32}
\]