Conjugacy invariants and rigidity in Garside groups: a uniformity phenomenon