Surface roughness as a critical parameter for frictional sliding instability: Experiments and modelling