Editorial: AAOS Orthobiologics Registry—Sometimes, More is Less

Recently, one of us was speaking with a friend about a treatment that several good studies found to be no better than placebo. He replied wryly, “That’s from the patient’s viewpoint. From the surgeon’s viewpoint, it’s much better.” Obviously, he meant it in jest, but the underlying sentiment is tinged with cynicism. That word—cynicism—is at the front of our minds now in the wake of the American Academy of Orthopaedic Surgeons (AAOS) announcing its Orthobiologics Registry [1]. We wish the AAOS had not created this registry, but given that it has, we’ve got a few thoughts to share about its shortcomings, as well as a few suggestions for those interested in “biologic treatments” for knee arthritis, whether as part of a surgical practice or in research. The Academy’s announcement indicated that the new registry will “measure the long-term safety, efficacy, and real-world outcomes of orthobiologic therapies for patients diagnosed with knee osteoarthritis,” and that this registry “directly addresses AAOS’ mission to provide the highest-quality musculoskeletal care through evidence-based practice” [1]. Regarding the Academy's first goal: We don't need a registry to answer the efficacy question, since years of randomized trials already have found these therapies generally to be ineffective. The Cochrane group—the world’s leading clearinghouse for high-quality meta-analyses of randomized control trials (RCTs)—pooled seven RCTs on stem cells for knee arthritis and found a mean improvement in short-term pain of only 1.2 points out of 10 [10]. This is below any reasonable minimum clinically important difference, a finding similar to that from high-quality systematic reviews and meta-analyses [3, 8]. And remember that favorable claims about these treatments deriving from lower-quality study designs (that is, observational, nonrandomized studies, or registries with incomplete follow-up) generally will be inflated because of the impacts of selection bias, assessment bias, the placebo effect, and, again, follow-up that is insufficiently long or complete. The absence of robust evidence that these treatments regenerate hyaline cartilage in patients with osteoarthritis of the knee is worth remembering here, as well.A lack of biologic plausibility should further depress our enthusiasm for this family of interventions. These issues—as well as the tendency of so many studies to report findings of benefit based on p values rather than as a function of clinically important effect sizes—mean that any slim “advantages” we think we’re perceiving in support of these new injectables are overstated, likely imperceptible to patients, or altogether absent. Selling these ineffective treatments to patients seems wrong to us. All of us know how difficult it is to be in the office with a patient when surgery seems inappropriate or premature, yet it feels like we’ve already tried a number of reasonable nonsurgical alternatives. But the solution in those challenging moments isn’t to offer something unreasonable (that is, something shown not to be clinically superior to a placebo), and the answer isn’t to say there’s nothing more we can do. Instead, expressing empathy and compassion, talking to the patient about how to live well with arthritis, affirming how the patient already has adapted to his or her body, and making specific suggestions about how he or she might further accommodate it when needed is the superior and more honest approach [7]. This approach takes a little more time, but from firsthand experience, we can share that it’s well worth it. Physician-patient relationships based on that kind of care and integrity endure; they’re healthier for patients and more gratifying for surgeons. As for the AAOS’s laudable goal of advancing musculoskeletal care through evidence-based practice, we’re concerned that this registry will have the opposite effect: More overtreatment of patients with products that don’t work. Here’s why. Unlike the more socialized healthcare systems of Scandinavia, the United Kingdom, and Australia, where registries evaluate relatively captive populations, any registry that draws from the US healthcare system at large will be leaky by design. The patients who return for follow-up, and thus the ones who will be “counted” in registry-derived studies, generally will be those who are doing better. That’s the consistent impact of loss to follow-up both in research and in practice. Just think of which patients come to your office after having surgery from your colleagues across town: The ones who transfer their care in this context generally aren’t doing so well. The same is true in a leaky registry. The missing—those who don’t experience a benefit from stem cells, PRP, or some yet-to-be-discovered biologic—can’t be counted in registry studies about those products. As a result, it’s certain—not likely, but certain—that the findings from the AAOS Orthobiologics Registry [2] will overstate the benefits of biologics. To be clear, we are great fans of registry research from all over the world [4, 5], even though we’ve occasionally made suggestions about how those important institutions can do their work better [9]. We also realize that life is complicated, and so sometimes a leaky registry is better than no registry. For example, the AAOS’s American Joint Replacement Registry (AJRR) can answer clinically useful questions, even though many patients enrolled in the AJRR will get revisions that go uncaptured by that registry. The difference between the AJRR and the new biologics database is that the underlying treatment studied in the AJRR—lower extremity arthroplasty—is generally effective. In the setting of an effective treatment, even a leaky registry can help us compare alternatives, identify things that are failing prematurely, gauge trends in usage patterns of an established treatment against less well-established alternatives (at least among participating centers), and, when possible, marry results within that registry to other data sources that might help capture those missing revisions. But where biologics are concerned, there are no other data sources through which we can capture the missing, and leaving them out will result in inflated, erroneous conclusions about clinical efficacy favoring the new but ineffective interventions. Of course, if any treatments in a leaky registry are shown not to be effective, you can take that to the bank. But because of missingness as well as selection bias—for the first 3 to 5 years, only 10 enthusiastic, participating sites can enroll patients in this registry [2]—that outcome seems unlikely. The people who are behind this know as much about registries and clinical research as we do, so we suspect that they’ve considered and discounted all these concerns. And for the commercial entities supporting this effort, these concerns are features, not bugs. The registry may become a tool to support the promotion of ineffective but remunerative treatments whose substantial costs—often more than USD 5000 out of pocket—will be borne by patients. This seems cynical. Those involved could have used the considerable resource they’re deploying here to enable a number of more definitive RCTs to get robust answers to the remaining questions people have about biologics for arthritis. They chose not to, and this choice imposes a real opportunity cost. Every dollar spent studying ineffective care is a dollar not spent learning about worthwhile treatments [6]. Given all that, here are a few things you can do: If you’re a surgeon: Don’t use injectable biologics for knee arthritis until or unless robust, high-quality randomized trials consistently support the products you use for the indications you treat, with effect sizes that are large enough for patients to care about. Despite contentions about registries having more “real-world” applicability, for the reasons covered here, RCTs are more trustworthy in this context. If you’re a clinician-scientist: Focus your attention on performing those RCTs. If you want to report results from the AAOS Orthobiologics Registry, be absolutely explicit in reporting the number of individuals treated and the percentage of patients in each group who were lost to follow-up. Where possible, consider using analytic methods that mitigate the optimism that missingness injects into study results, such as sensitivity analyses, as well. If you work at the AAOS Orthobiologics Registry: We suggest requiring investigators to report on missingness as a condition of using data from the registry. Obvious as this seems, it’s too often not done in the registry work we read in orthopaedic surgery journals.