Step 1: Write the given function clearly.
We are given
\[
f(x)=ax+bx^2
\]
and we need to find the coefficient of \(x^3\) in \(f(f(x))\).
This means we have to substitute \(f(x)\) in place of \(x\) in the same function.
Step 2: Use the definition of composition of functions.
Since
\[
f(t)=at+bt^2
\]
for any input \(t\), replacing \(t\) by \(f(x)\) gives
\[
f(f(x))=a\bigl(f(x)\bigr)+b\bigl(f(x)\bigr)^2
\]
Now substitute \(f(x)=ax+bx^2\):
\[
f(f(x))=a(ax+bx^2)+b(ax+bx^2)^2
\]
Step 3: Expand the first part.
Let us first simplify
\[
a(ax+bx^2)=a^2x+abx^2
\]
This part contains only \(x\) and \(x^2\), so it does not directly contribute any \(x^3\) term.
Step 4: Expand the square in the second part.
Now consider
\[
(ax+bx^2)^2
\]
Using
\[
(p+q)^2=p^2+2pq+q^2
\]
with \(p=ax\) and \(q=bx^2\), we get
\[
(ax+bx^2)^2=a^2x^2+2abx^3+b^2x^4
\]
Step 5: Multiply by \(b\).
Now multiply the above expression by \(b\):
\[
b(ax+bx^2)^2=b\left(a^2x^2+2abx^3+b^2x^4\right)
\]
\[
= a^2bx^2+2ab^2x^3+b^3x^4
\]
Here, the coefficient of \(x^3\) is clearly
\[
2ab^2
\]
Step 6: Combine all parts of \(f(f(x))\).
Putting both expanded parts together:
\[
f(f(x))=\left(a^2x+abx^2\right)+\left(a^2bx^2+2ab^2x^3+b^3x^4\right)
\]
\[
f(f(x))=a^2x+\left(ab+a^2b\right)x^2+2ab^2x^3+b^3x^4
\]
So the coefficient of \(x^3\) is
\[
2ab^2
\]
Step 7: Final conclusion.
Hence, the required coefficient of \(x^3\) in \(f(f(x))\) is
\[
\boxed{2ab^2}
\]
Therefore, the correct option is
\[
\boxed{(5)\ 2ab^2}
\]