Step 1: Find the set S
\(x^2-11x+30\le0\Rightarrow(x-5)(x-6)\le0\Rightarrow5\le x\le6\).
Step 2: Check monotonicity
\(f'(x)=9x^2-36x+27=9(x-1)(x-3)\), which is positive for \(x>3\). So \(f\) is increasing on \([5,6]\).
Step 3: Maximum value
The maximum is at \(x=6\): \(f(6)=3(216)-18(36)+27(6)-40=648-648+162-40=122\).
Step 4: Compare the options
The maximum value is 122, option (A).
Final Answer:
The maximum value of the function on the set is 122.
\[ \boxed{\text{(A) }122} \]