Your shoulder is the most mobile joint in the body, but also the most unstable. It's supported only by thin ligaments and small muscles.
Because the shoulder is so mobile, a brace that shifts or slips even a few millimeters fails to provide the stability required to protect those thin ligaments.
So when your brace doesn't fit right, your joint can still start to "drift", which leads to clicking, grinding and aching... And eventually maybe even a worse injury, like a tear.