Step 1: Convert the pair of lines into equations in slope.
Let a line through the origin be
\[
y=mx
\]
For
\[
2x^2+axy+3y^2=0,
\]
put \(y=mx\).
Then,
\[
2x^2+ax(mx)+3m^2x^2=0
\]
\[
x^2(2+am+3m^2)=0
\]
Hence, the slopes of the two lines satisfy
\[
3m^2+am+2=0
\]
For
\[
2x^2+bxy-3y^2=0,
\]
put \(y=mx\).
Then,
\[
2x^2+bx(mx)-3m^2x^2=0
\]
\[
x^2(2+bm-3m^2)=0
\]
Hence, the slopes of the two lines satisfy
\[
3m^2-bm-2=0
\]
Step 2: Use the common line condition.
The two pairs have one common line, so the two quadratic equations in \(m\) have one common root.
Check option (3):
\[
a=5,\quad b=1
\]
Then the first slope equation becomes
\[
3m^2+5m+2=0
\]
Factorizing,
\[
3m^2+5m+2=(3m+2)(m+1)=0
\]
So,
\[
m=-\frac{2}{3},\quad m=-1
\]
The second slope equation becomes
\[
3m^2-m-2=0
\]
Factorizing,
\[
3m^2-m-2=(3m+2)(m-1)=0
\]
So,
\[
m=-\frac{2}{3},\quad m=1
\]
Thus, the common slope is
\[
m=-\frac{2}{3}
\]
Step 3: Check perpendicularity of the remaining lines.
The remaining slopes are
\[
-1
\]
and
\[
1
\]
Their product is
\[
(-1)(1)=-1
\]
Therefore, the remaining two lines are perpendicular.
Step 4: Final conclusion.
Hence, the required values are
\[
\boxed{a=5,\ b=1}
\]