Shrugging off the shoulder during neck surgery is often related to accessory nerve (cranial nerve XI) damage. This nerve innervates the trapezius muscle, which is responsible for shoulder shrugging. Damage to this nerve during surgery can result in weakness or paralysis of the trapezius muscle, leading to the inability to shrug the shoulder.