Step 1:
For orthogonal matrix, rows are mutually perpendicular.
Step 2:
Dot product of first and third row:
1· a+2·2+2· b=0
⟹ a+4+2b=0 (1)
Step 3:
Dot product of second and third row:
2a+1·2+(-2)b=0
⟹ 2a+2-2b=0 (2)
Step 4:
Solving (1) and (2):
a=2, b=-1