Home › Blog › Registry Data Analysis: What We Do at Regenexx that Doesn’t Exist Elsewhere

Credit: Wikipedia-By Martin Grandjean (vector), McGeddon (picture), Cameron Moll (concept) – Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=102017718

There’s a super fascinating story about how mathematicians helped armor WWII bombers that can teach us about how to interpret Registry data in Orthobiologics. Let’s learn about how that impacts how we analyze our Registry data to help chose who should get our procedures. Let’s dig in.

A WWII Bomber Problem

For the pilots of WWII bombers, the biggest problem was getting shot down by smaller fighters sent to intercept you. The math behind planes shot down versus those that returned was often a big problem. Meaning it took lots of resources to build a bomber and months to years to train a pilot and crew, so if you lost bombers more quickly than you could replace them, that was unsustainable.

In addition, there was another practical problem facing WWII bombers. You could armor them with steel, but steel was heavy, which meant that the bombers began to lose payload capacity when you added too much armor. Hence, this problem was turned over to the SRG (Statistical Research Group) at Columbia University. The goal of the project was to figure out select places in a bomber where you could concentrate armor and where you didn’t need it. Basically to thread the needle between weight and protection.

Missing Bullet Holes

The SRG began collecting data on the bullet holes found in returning bombers and produced the statistical map above. At first blush, this answers the question, right? You just concentrate your armor in all of these places with the bullet holes. That was the plan, until a smart mathematician who was part of the SRG (Abraham Wald), looked at the data. He reasoned that everyone else’s conclusions suffered from a problem called “survivorship bias”. Basically, the place to put the armor was where the bullet holes weren’t. Meaning that the model was created with planes that returned, but the real story was in the planes that never made it back. When discussing all of this with real pilots, they agreed that taking fire in the engines and the rear mid-section, where this model had no bullet holes, usually resulted in a plane that crashed. Hence, the real story behind the data was telling us that the armor was needed where the bullet holes weren’t found.

What Missing Bullet Holes Can Teach Us About Registry Data

One part of my job as the Chief Medical Officer (CMO) for Regenexx is using our Registry data to help our providers make the best decisions for patients. That’s a unique job as nobody else on earth has been collecting large-scale Registry data on as many patients for as long as Regenexx. What’s a Registry?

A Registry is a system where you collect outcome data on all consented patients. We began that process in 2005, so we have 17 years of Registry data on tens of thousands of patients. There isn’t a close second on the amount and longevity of data collected on orthobiologics. Meaning we have more data on these issues than anyone else on planet earth.

One of the biggest questions we ask ourselves in analyzing that data is who is a great candidate for Orthobiologic injections and shouldn’t be offered these treatments. In answering those questions we often encounter the same kind of problem as the bullet hole analysis. What real story does the data tell and what should we do about it?

Survivorship Bias and Registries

The first question our research team asked itself was simple. A Registry relies on sending a questionnaire and then someone responding. However, what about the people that never respond? Depending on the type of Registry, those non-response rates can be as high as 30-50%. Are they like the bombers that never returned? Meaning, even though the patients that respond seem to do well with your procedure, are the patients who never respond doing more poorly and that’s why they fail to answer their questionnaire?

We tested that hypothesis years ago. We created a random sample of 1oo patients where we worked hard to get a response rate of 100%. The good news was that the mean positive outcome in that 100% response group was about the same as in the registry with the non-responders. Meaning in our data set, lost patient data wasn’t adversely impacting the data analysis.

The Right Story?

As shown by the bomber problem above, data paints a picture. Which picture you see is sometimes like which animal you see when you look at clouds. Meaning different people can get different pictures out of the same data set.

Our team calls that picture “the story”. We know that there’s likely one story that fits with the clinical observations of the doctors and makes sense and others that are tangents that won’t help anyone. Take for example looking at MRI findings to determine which ones make a knee arthritis patient a poor candidate for a Bone Marrow Concentrate procedure (SD).

When we first began looking at our Registry data on this issue, I was pretty convinced that we would find some relationship between the severity of Arthritis and a poor outcome. Meaning that those patients with more severe Arthritis would be the same ones that did poorly with the procedure When I first looked at a small dataset in about 2013, I thought I saw an association, but that was based on the candidacy grades being given by the doctors as a proxy for more severe Arthritis. However, when I looked at a larger dataset that association went away.

Then about 7 years ago, I sat down during a trip to our Cayman site and personally read about 50+ MRIs and compared the findings to the outcome of the patients. I couldn’t find a relationship. Since then, we have tried this experiment many times, most recently last year when we had our data scientist dig into even more MRI data. Again, no real relationship emerged. Not to be deterred, we’re still adding more MRI data and will take yet another look, but the story after years of looking seems clear. For knee Arthritis patients, there is no relationship between MRI findings and outcome. Meaning patients with more Arthritis do as well with a Bone Marrow Concentrate procedure as those with less.

However, that’s not the case for other joints. For example, early on we observed a clear story for patients with more severe hip Arthritis. Meaning there is a relationship between worse hip Arthritis and a poor outcome. For shoulder Arthritis, there’s a single finding that we identified on MRI that is associated with a poor outcome and this was relayed to all of our network doctors so that they can caution patients with this finding. In both of these datasets, especially hip, we are still adding MRI data to refine these assessments.

Candidacy Grading Is a Bare Minimum

As I reviewed above, one of the points of having all of this data and spending the thousands of hours it requires to create a credible story is so we can give each patient an accurate Candidacy assessment. The problem is that since no other group on earth possesses enough data on a single procedure/technique to make these calls, everyone else is merely guessing about whether they know if you’re a good or bad Candidate. Or even worse, they never make that Candidacy assessment in the first place. Meaning that no Candidacy assessment is ever provided.

Hence, as a consumer, you need to ask for a Candidacy assessment from any provider who might perform a PRP or Bone Marrow Concentrate injection or any other Orthobiologic procedure. These are a few of the questions that need to be asked:

• Do you collect Registry data and if so, how much have you collected and for how long?
• Am I a good or poor Candidate?
• What data is that assessment based on?
• What percentage of patients are poor Candidates for this procedure?

Meaning, figuring out Candidacy requires large datasets collected for many years on a specific procedure and method. Then the clinic needs to spend a HUGE number of hours analyzing that data to create a Candidacy assessment. Meaning the doctor should be able to tell you the data story behind why you’re a good or poor Candidate.

The upshot? I love the WWII bomber story because it helps inspire our data team to think outside the box as we analyze mountains of registry data to help pick the right patients for our procedures. In the end, if the place where you get your Orthobiologic injection can’t tell you how they create their data stories, then find another spot to get treated!

Chris Centeno, MD is a specialist in regenerative medicine and the new field of Interventional Orthopedics. Centeno pioneered orthopedic stem cell procedures in 2005 and is responsible for a large amount of the published research on stem cell use for orthopedic applications. View Profile