Step 1: Identify the coefficients of the transformed equation and use invariants.
The transformed equation is $25x'^2+9y'^2=225$. Comparing this to the general form $a'x'^2+2h'x'y'+b'y'^2=c'$, we have:
\[
a' = 25, b' = 9, h' = 0, c' = 225.
\]
Under rotation of axes, the sum of the coefficients of the second-degree terms and the constant term are invariants.
\[
a+b = a'+b' \implies a+b = 25+9 = 34.
\]
\[
c = c' \implies c = 225.
\]
Step 2: Use the transformation formula for the coefficient of the xy-term.
The coefficient of the $x'y'$ term in the transformed equation is given by $2h' = (b-a)\sin(2\theta) + 2h\cos(2\theta)$.
We are given $h'=0$ and the angle of rotation is $\theta = \pi/4$, so $2\theta = \pi/2$.
\[
0 = (b-a)\sin(\pi/2) + 2h\cos(\pi/2) \implies 0 = (b-a)(1) + 2h(0) \implies b-a=0 \implies a=b.
\]
Step 3: Use the transformation formula for the coefficient of the $x^2$-term to find h.
The coefficient of the $x'^2$ term is given by $a' = a\cos^2\theta + b\sin^2\theta + 2h\sin\theta\cos\theta = a\cos^2\theta + b\sin^2\theta + h\sin(2\theta)$.
Substituting $\theta=\pi/4, a'=25, a=b$:
\[
25 = a(\cos^2(\pi/4)) + a(\sin^2(\pi/4)) + h\sin(\pi/2).
\]
\[
25 = a(\frac{1}{2}) + a(\frac{1}{2}) + h(1) \implies 25 = a+h.
\]
From Step 1, $a+b=34$ and $a=b$, so $2a=34 \implies a=17$.
Substituting $a=17$ into $25=a+h$, we get $25 = 17+h \implies h=8$.
Step 4: Calculate the required expression.
We need to calculate $(a+2h+b-\sqrt{c})^2$.
We can group this as $((a+b)+2h-\sqrt{c})^2$.
We have $a+b=34$, $h=8$, and $\sqrt{c}=\sqrt{225}=15$.
\[
(34 + 2(8) - 15)^2 = (34 + 16 - 15)^2 = (50 - 15)^2 = 35^2.
\]
\[
35^2 = \boxed{1225}.
\]