Step 1: Expand the given matrix expression.
\[
(2A+B)^2
=
4A^2+2AB+2BA+B^2
\]
and
\[
(A-3B)^2
=
A^2-3AB-3BA+9B^2.
\]
Adding,
\[
(2A+B)^2+(A-3B)^2
=
5A^2-AB-BA+10B^2.
\]
Step 2: Compare with the given equation.
Given,
\[
5A^2-AB-BA+10B^2
=
5A^2-2AB+10B^2.
\]
Cancelling common terms,
\[
-AB-BA=-2AB.
\]
Therefore,
\[
AB+BA=2AB.
\]
Hence,
\[
BA=AB.
\]
Thus \(A\) and \(B\) commute.
Step 3: Evaluate \(ABAB\).
Since
\[
AB=BA,
\]
we have
\[
ABAB=A(BA)B.
\]
Replacing \(BA\) by \(AB\),
\[
ABAB=A(AB)B.
\]
Therefore,
\[
ABAB=A^2B^2.
\]
Step 4: Final conclusion.
Hence,
\[
\boxed{ABAB=A^2B^2}
\]