Step 1: Simplify the integrand $\frac{1}{\cot^2 x - 1}$.
Recall that $\cot^2 x - 1 = \frac{\cos^2 x}{\sin^2 x} - 1 = \frac{\cos^2 x - \sin^2 x}{\sin^2 x}$.
Also, $\cos^2 x - \sin^2 x = \cos 2x$.
So, $\frac{1}{\cot^2 x - 1} = \frac{1}{\frac{\cos 2x}{\sin^2 x = \frac{\sin^2 x}{\cos 2x}$.
Step 2: Use the identity $\sin^2 x = \frac{1 - \cos 2x}{2}$.
\[ \int \frac{\sin^2 x}{\cos 2x} \,dx = \int \frac{\frac{1 - \cos 2x}{2{\cos 2x} \,dx = \frac{1}{2} \int \left(\frac{1}{\cos 2x} - \frac{\cos 2x}{\cos 2x}\right) \,dx \]
\[ = \frac{1}{2} \int \left(\sec 2x - 1\right) \,dx \]
Step 3: Integrate term by term.
Recall $\int \sec(ax) \,dx = \frac{1}{a} \log |\sec(ax) + \tan(ax)|$.
\[ = \frac{1}{2} \left[\frac{1}{2} \log |\sec 2x + \tan 2x| - x\right] + c \]
\[ = \frac{1}{4} \log |\sec 2x + \tan 2x| - \frac{x}{2} + c \]
Step 4: Compare the result with the given form $\frac{1}{A} \log |\sec 2x + \tan 2x| - \frac{x}{B} + c$.
By comparison:
\[ \frac{1}{A} = \frac{1}{4} \implies A = 4 \]
\[ \frac{1}{B} = \frac{1}{2} \implies B = 2 \]
Step 5: Calculate $A + B$.
\[ A + B = 4 + 2 = 6 \]