Step 1: Understanding the Concept:
Divide the numerator and denominator by \(x^{10}\):
\[ \frac{x^4(x^{10}-1)}{x^{20}+3x^{10}+1} = \frac{x^4 - x^{-6}}{x^{10} + 3 + x^{-10}} \]
Step 2: Substitute:
Let \(t = x^5 + x^{-5}\). Then \(dt = 5(x^4 - x^{-6})\,dx\), so \((x^4 - x^{-6})dx = \frac{dt}{5}\).
Also \(t^2 = x^{10} + 2 + x^{-10}\), so \(x^{10} + x^{-10} + 3 = t^2 + 1\).
Step 3: Integrate:
\[ \int \frac{dt/5}{t^2+1} = \frac15\tan^{-1}t + c = \frac15\tan^{-1}\left(x^5 + \frac{1}{x^5}\right) + c \]
Step 4: Why the other options are wrong.
Options (A) and (C) miss the factor \(\frac15\) from \(dt = 5(\dots)dx\). Option (D) uses \(x^{10}\) in the inverse tangent and the factor \(\frac{1}{10}\), which does not match the substitution.
Final Answer:
The integral is \(\frac15\tan^{-1}(x^5 + x^{-5}) + c\), option (B).
\[ \boxed{\frac{1}{5}\tan^{-1}\left(x^5+\frac{1}{x^5}\right)+c} \]