Step 1: Understand the concept
The bisector of angle \(O\) meets \(AB\) at \(D\) so that \(\dfrac{AD}{DB} = \dfrac{OA}{OB}\) (angle bisector theorem).
Step 2: Find the lengths
\(OA = \sqrt{1 + 4} = \sqrt{5}\) and \(OB = \sqrt{1 + 9} = \sqrt{10}\). So \(AD : DB = \sqrt{5} : \sqrt{10} = 1 : \sqrt{2}\).
Step 3: Find D
By the section formula, \(\overrightarrow{OD} = \dfrac{\sqrt{2}\,\overrightarrow{OA} + 1\cdot\overrightarrow{OB}}{\sqrt{2} + 1} = \dfrac{(-\sqrt{2} + 1,\ 2\sqrt{2} + 3)}{\sqrt{2} + 1}\).
Step 4: Dot product
\(\overrightarrow{AB} = (2, 1)\). Then \(\overrightarrow{OD}\cdot\overrightarrow{AB} = \dfrac{2(1 - \sqrt{2}) + 2\sqrt{2} + 3}{\sqrt{2} + 1} = \dfrac{5}{\sqrt{2} + 1} = 5(\sqrt{2} - 1)\), option (D).
Final Answer:
The dot product is 5(sqrt 2 - 1). This is option (D).
\[ \boxed{\text{(D) }5(\sqrt{2}-1)} \]