The component of \( \mathbf{a} \) orthogonal to \( \mathbf{b} \) is given by the formula:
\[
\mathbf{a}_{\perp} = \mathbf{a} - \left( \frac{\mathbf{a} \cdot \mathbf{b}}{\mathbf{b} \cdot \mathbf{b}} \right) \mathbf{b}
\]
First, calculate the dot products:
\[
\mathbf{a} \cdot \mathbf{b} = (5)(3) + (7)(-1) + (2)(6) = 15 - 7 + 12 = 20
\]
\[
\mathbf{b} \cdot \mathbf{b} = (3)^2 + (-1)^2 + (6)^2 = 9 + 1 + 36 = 46
\]
Now, find the orthogonal component of \( \mathbf{a} \):
\[
\mathbf{a}_{\perp} = \mathbf{a} - \left( \frac{20}{46} \right) \mathbf{b} = \left( 5 \hat{i} + 7 \hat{j} + 2 \hat{k} \right) - \left( \frac{20}{46} \right) \left( 3 \hat{i} - \hat{j} + 6 \hat{k} \right)
\]
Simplify:
\[
\mathbf{a}_{\perp} = \left( 5 \hat{i} + 7 \hat{j} + 2 \hat{k} \right) - \left( 1.304 \hat{i} - 0.434 \hat{j} + 2.609 \hat{k} \right)
\]
\[
\mathbf{a}_{\perp} = \left( 3.696 \hat{i} + 7.434 \hat{j} - 0.609 \hat{k} \right)
\]
Now, find the magnitude of the orthogonal component:
\[
|\mathbf{a}_{\perp}| = \sqrt{(3.696)^2 + (7.434)^2 + (-0.609)^2} = \sqrt{13.66 + 55.27 + 0.37} = \sqrt{69.3} \approx 8.33
\]
Thus, the magnitude of the component of \( \mathbf{a} \) orthogonal to \( \mathbf{b} \) is \( \boxed{8.33} \).