Step 1: Compute A squared
\(A^2 = \begin{pmatrix}2 & -1\\ 0 & 2\end{pmatrix}\begin{pmatrix}2 & -1\\ 0 & 2\end{pmatrix} = \begin{pmatrix}4 & -4\\ 0 & 4\end{pmatrix}\).
Step 2: Form the equation
\(A^2+xA+yI = \begin{pmatrix}4+2x+y & -4-x\\ 0 & 4+2x+y\end{pmatrix} = O\).
Step 3: Solve
From the (1,2) entry, \(x=-4\). Then \(4+2(-4)+y = 0\) gives \(y = 4\).
Step 4: Check
Option (B), \(x=-4, y=4\). The characteristic equation of \(A\) is \(\lambda^2-4\lambda+4 = 0\), which agrees by the Cayley-Hamilton theorem.
Final Answer:
x = -4 and y = 4.
\[ \boxed{\text{(B)}\ x=-4,\ y=4} \]