The unconstrained growth rate of COVID-19 is crucial for measuring the impact of interventions, assessing worst-case scenarios, and calibrating mathematical models for policy planning. However, robust estimates are limited, with scientific focus on the time-insensitive basic reproduction number R0. Using multiple countries, data streams and methods, we consistently estimate that European COVID-19 cases doubled every three days when unconstrained, with the impact of physical distancing interventions typically seen about nine days after implementation, during which time cases grew eight-fold. The combination of fast growth and long detection delays explains the struggle in countries' response better than large values of R0 alone, and warns against relaxing physical distancing measures too quickly. Testing and tracing are fundamental in shortening such delays, thus preventing cases from escalating unnoticed.