Step 1: Expand the nonlinearity
The given signal passes through a nonlinear device, producing
\[
y(t) = 2A\cos\phi(t) + \frac{5A^{2}}{2}\big[1 + \cos(2\phi(t))\big],
\]
where \(\phi(t) = \omega_c t + k_f \int m(\lambda)\, d\lambda.\)
Step 2: Identify the spectral components
The output has three distinct parts:
(i) A DC component: \(\tfrac{5A^{2}}{2}\), centered at 0 Hz.
(ii) An FM term: \(2A \cos\phi(t)\), centered at \(\omega_c\), with the same bandwidth as the original FM signal, namely \(B_T\).
(iii) A doubled FM term: \(\tfrac{5A^{2}}{2}\cos(2\phi(t))\), centered at \(2\omega_c\).
Step 3: Bandwidth of the doubled FM
For the doubled term, the instantaneous frequency is
\[
\frac{d}{dt}(2\phi(t)) = 2\omega_c + 2k_f m(t).
\]
Thus, both the carrier frequency and the frequency deviation are doubled. This means the FM bandwidth doubles as well: the bandwidth of this component is \(2B_T\).
Step 4: Separation condition
To recover the original FM term, we must isolate the band around \(\omega_c\) and reject the band around \(2\omega_c\). These two spectral regions should not overlap. Hence the highest edge of the band around \(\omega_c\) must not exceed the lowest edge of the band around \(2\omega_c\):
\[
\omega_c + \frac{B_T}{2} \;\le\; 2\omega_c - \frac{2B_T}{2}.
\]
Simplify:
\[
\omega_c + \frac{B_T}{2} \le 2\omega_c - B_T
\;\;\Rightarrow\;\; \omega_c \ge \frac{3}{2} B_T.
\]
Step 5: Check the weaker condition
There is also the trivial requirement \(\omega_c > \tfrac{B_T}{2}\) to avoid overlap with DC. But since \(\tfrac{3}{2}B_T > \tfrac{1}{2}B_T\), the stricter requirement dominates.
Final Answer:
\[
\boxed{\omega_c^{\min} = \tfrac{3}{2}B_T}
\]