Facebook and organ donation: does getting people to sign up as organ donors actually make a difference?

A year ago today, Facebook launched a PR blitz announcing that they were making it possible to publicize your organ donation status on Facebook. Donate Life America, the national coalition of state organ donation organizations, made it pretty clear that Facebook was launching a concerted PR effort:

In an exclusive interview with ABC News’ Robin Roberts on “Good Morning America” this morning, Facebook founder Mark Zuckerberg announced the new initiative and became one of the first Facebook users to sign up to donate on the social network. Tonight Facebook’s COO and one of America’s most powerful businesswomen Sheryl Sandberg will sit down exclusively with Diane Sawyer. ABC News affiliates, ABC News Radio, Yahoo! and ABCNews.com will also feature coverage of the exciting new initiative. ESPN will be running two powerful organ donation stories throughout the day on SportsCenter that encourage organ and tissue donation.

Aaron Swartz, Kieran Healy and I had a short discussion about the prospects for an effect on Twitter. I was optimistic, but Healy said, "public support for donation hasn't been the bottleneck for a very long time."

The best data I've found on the impact of Facebook's efforts comes from a CBC news article from June 25th:

The Facebook drive had an immediate effect. In the first few days, more than 100,000 people changed their status to indicate they were willing to be donors.

To be official, a willing organ donor needs to sign up with their own government's donor program, so Facebook also provided direct links to local donor registries. Donate Life America (DLA), a non-profit dedicated to raising donor awareness, subsequently reported an average 1,000-per-cent increase in online donor registrations across the U.S. over the six days following the addition of the donor feature.

To put that in concrete terms, during that time period 33,406 people legally registered to be donors; other years the number of signups was much smaller – in 2011 for the same time period, it was 3,288. Since then, the signup numbers have stabilized at around 1,150 per day, which was still more than double the historical daily average of 548.

So the Facebook effort led some people who probably weren't on the list before to sign up as organ donors - if the pace kept up, which seems unlikely, Facebook would have led to around 200,000 additional people joining the list this year. But does getting people who aren't yet donors to sign up make much of a difference?

This may seem like a crazy question, so some context is useful: if you die in a way such that your organs could be donated, even if you're not a registered donor, the "organ procurement organization" (OPO) will still ask your family if they'd like your organs to be donated. The OPOs are good (and getting better) at asking, and most people do support organ donation, so they actually get permission from most families that are asked when their loved one wasn't a registered donor. Conversely, in most cases, even if you're registered as a donor, if your family says they don't want your organs donated, the OPO will follow their wishes (this happens pretty rarely, luckily). So, structurally, it's actually not all that obvious that getting more people to sign up as organ donors will make much of a difference, especially if the people you get to sign up are the ones whose families already support donation (which seems pretty likely).

The ideal way to address this problem would be to run a large randomized control trial: spend a bunch of money on randomly giving specific people messages about registering as donors, hopefully getting a big difference in registration rates between treatment and control groups, and then follow them for years to see if the treatment group actually ends up donating at higher rates. This would be expensive, time consuming, and difficult, and to my knowledge it's never been tried.

There are two easier strategies we can pursue to try to identify the impact of marginal donor registry sign-ups, though they won't have the same rigor as the RCT:

1. compare the actual donation rates of people who are eligible to donate upon death who are and aren't registered donors
2. comparing states based on the proportion of registered people and the proportion of eligible donors who actually donate. The data and code for my analysis are on github here and figshare here. Feel free to email me if you need any help interpreting; I didn't do much to document well.

Neither of these strategies is as good as an RCT, and in particular both will be unable to distinguish whether registration efforts are just leading more people who would donate anyway to sign up on the list, but they still provide useful information.



1. How much more often do eligible deaths turn into donations for registered than un-registered donors?

We need a few data points, some assumptions, and easy algebra to answer this question:

• Overall, the consent rate for eligible deaths nationally in 2011 was 75.4% (big XLS; linked to from here). This figure includes both registered and unregistered donors. (I use 2011 data because it's the most recent available for the figures below and consistency seems more important than recency.)
• Amongst all deceased donors in the U.S. in 2011, 36.2% had registered as donors prior to their deaths. (PDF)
• There is conflicting data on how often families of registered donors consent for their loved ones' organs to be taken, but 96% seems to be a reasonable, if optimistic, forward-looking estimate:
• The most recent data comes from a 2013 study by Heather Traino and Laura Simonoff, covering 1,090 families who were asked to donate on behalf of their loved ones, found that 97.6% of families of registered donors actually consented to donation. However, they also found that 85.6% of families of non-registered eligible donors consented to donation, much higher than the national average of 75.4%. This is particularly worrisome because they had some nonresponse bias, with families that consented to donation being more likely to participate, and the same problem may have arisen with families of registered donors, leading to an underestimate of the proportion of families of registered donors who refuse donation. If we assume the ratios of non-donation scale the same way from the sample toa  national group across registered and non-registered donors, we'd estimate that 24.6%/14.4% * 2.4% = 4.1% of registered donor families would refuse donation. Despite the nonresponse bias and the need to extrapolate, this study is the one I regard are most credible, as it is the largest and most recent. The estimated impact of the registry could be made more conservative by assuming that scaling to a nationally representative group occurs in the same proportion as the rate of consent -- 75.4%/85.6% * 97.6% = 86% -- but this strikes me as too conservative.
• An older study from Ohio and Pennsylvania from 1994-1999 (Siminoff et al. 2001) finds that 89.3% of families that knew the deceased has a donor card consented to donation, but only 44% of families who knew the deceased didn't have a donor card consented, implying 10.7% of families of registered donors refuse donation. In the period since this study, all the states have passed laws allowing OPOs to obtain organs of registered donors even if families object, even if they don't do so in practice, and it seems likely that the rate of consent amongst registered donors would have risen as a result; 90% strikes me as a reasonable lower bound assumption.
• A smaller study from North Carolina found that 20% of families of registered donors refused donation, but I think it's an outlier.
• The key other assumptions that we need to get the calculation off the ground is that "eligible" donors are representative of all donors.

And then the basic equation is:

36.2%*95.9% + (1-36.2%)*x = 75.4%

Solving for x, we get:

x = (75.4%-36.2%*95.9%)/(1-36.2%) = 64% of non-registered donors consent to donating

If instead of assuming that 95.9% of registered donors consent when given the option, we assumed that 90% do, we'd estimate that 67% of non-registered donors consent to donating.

The national conversion rate for eligible donors (i.e. the rate at which they actually end up donating) in 2011 was 73.1% (big XLS; linked to from here), or 97% of the 75.4% consent rate. Applying that discount, we would estimate that:

• Under the 95.9% assumption, registration is associated with a 31 percentage point increase in conversion: (95.9%-64%)*97%
• Under the 90% assumption, registration is associated with a 22 percentage point increase in conversion: (90%-67%)*97%.

Since there were 8,993 eligible deaths reported in 2011, and an additional 1,554 "non-eligible" donations actually occurred (big XLSX; linked to from here). If we assume that the additional deceased donors consented at the same rate as eligible donors (couldn't find data on this), the 1,554 additional donors would come out of a pool of 2,061 potential additional donors (there were probably many more and conversion rates are lower, but I think it makes sense to focus on this), for a but actually occurring deceased donors, for a pool of 11,054 potential deceased donors. 2.51 million people died in the U.S. in 2011 (CDC PDF), so the rough chance of ever dying in a way that makes you donation-eligible is 0.44%. And since 311.8 million people lived in the U.S. in 2011, the chance of dying in a way that makes you donation eligible in 2011 was 0.0035%.

Multiplying these out, we'd expect adding 1 random person to the organ donation registry to increase the number of deceased donors in that year by 0.0011% (0.0035%*31%) and in their lifetime by 0.14% (0.44%*31%). So adding a million donors to the registry is expected to increase donation by 11 donors that year and 1400 donors per lifetime. Using the 90% instead of 95.9% assumption above would cut both of these estimates by about one third.

I've only been able to find one other attempt at this kind of analysis in the literature, and it wasn't very rigorous, but the value it comes out with is within an order of magnitude of mine: Tyler Harrison estimates that 1% of people die in a way such that their organs can be harvested, and that 95% of those who sign donor cards actually donate, while only 50% of those who don't sign their donor cards end up donating, for a net lifetime increase in donation due to registration of 0.45%, about three times my estimate. He claims, "Regardless, if we go by earlier assumptions on rates of consent and eligible donors, adding 200 names to a registry should result in one new donor over a lifetime."

Looking back on the Facebook data, my methodology for this part would suggest that, aggressively assuming that Facebook led to 200,000 new registered donors, that would have led to ~2 more actual donors in the past year, and would lead to ~280 over the course of a lifetime. Assuming that the benefits dropped off more quickly (which would be verifiable if Facebook or DonateLife America released the data) might cut the estimated effect in half or more.

All of this still assumes that the people Facebook got to sign up wouldn't have signed up, or, when the time came, donated, otherwise. In both cases, I find that assumption pretty dubious, and my arbitrary guess would be that as a result the estimates above are probably several times more optimistic than they should be, though it could easily be as much as a couple of magnitudes too optimistic; it's hard to see how they could be much too pessimistic.

2. Comparing states by the proportion of registered donors and eligible donor conversion rates

If registering donors makes a difference, we would expect that states that have a higher proportion of people registered as organ donors would also have a higher proportion of eligible deceased donors who actually ended up donating. I've looked, and I see surprisingly little evidence to support that hypothesis.

First, it's worth noting that, nationally, the number of registered organ donors has almost doubled since 2006. In 2006, there were an estimated 57 million registered donors in the U.S. (PDF), while at the end of 2012 data pointed to more than 109 million registered donors in the U.S. Over the same period, we haven't seen any massive increase in the total number of deceased donors, which continues to hover around 2,000 per quarter (8,000 per year):

(There is less variability on the left hand side of the graph because the 2006, 2007, and 2008 figures are imputed from annual data.)

This chart is inconclusive because it could still be possible that the increase in registered organ donors did a lot to prevent a secular decline in total donations, but this chart doesn't look very good for the case that registering donors makes a big difference to donation rates.

Applying the roughly 50 million increase in the number of registered donors above to the optimistic figures calculated in the previous section, we would predict another ~550 deceased donors per year, or an extra ~140 deceased donors per quarter. There is enough natural variability in the annual donation figures that we might not be able to detect such a change graphically, but there doesn't appear to be much evidence of that kind of systematic increase in the chart.

To control for the possibility that there might be a secular decline in the number of people who die in ways that would make them eligible to donate, we should look at the proportion of eligible potential donors who actually end up donating, rather than the absolute number of donors:

There's a subtle but detectable increase in the "conversion rate" (i.e. the proportion of eligible donors who end up donating) over the period we look at, leaving open the possibility that increases in the number of registered donors may be making a difference. Re-scaling the chart would make the growth in the conversion rate more obvious.

The data underlying these charts is actually available at the state level, so it's possible to make some more informed inferences (and some prettier charts).

For instance, we can look at the state-specific trend in the proportion of the population that are registered as organ donors (never above 60% because the denominator includes, e.g., children, who don't register):

There's a general upward trend over time across almost all of the states, but there's considerably more noise than is apparent in the national data, and a fair amount of missing data (e.g. in Puerto Rico).

Looking at a similar chart for the conversion rate (i.e. the proportion of eligible donors who end up donating), we see considerably more variation, which makes sense because of the smaller sample sizes:

Comparing the conversion rate to the proportion of eligible donors by state, there are many states that look like they fit the hypothesis decently well (e.g. Alaska, Colorado, Hawaii) and others that look like counterexamples (e.g. Alabama, Maine, Montana):

The obvious thing to do in this case is a regression (or a series of regressions). Cutting to the chase, we can reject effect sizes of the magnitude predicted by the algebra above (e.g. 11 additional deceased donors per year per million additional registered donors). To my knowledge, this analysis has never been done before in the literature.

Brief regression results:

Model 1 is a simple OLS regression of the quarterly number of deceased donors on the number of eligible deaths, population, and number of registered donors in each state. However, the standard errors aren't appropriate because they don't reflect the clustering of observations in by state and time: this regression basically assumes that I was able to observe the data in 966 random places. Model 1 finds that for every million additional donors, we get 5 (=4*1.3) additional deceased donors per year, and the result is highly statistically significant.

Model 2 more appropriately controls for the fact that we actually only have observations from 52 "states" (DC and Puerto Rico are included), and that there could be correlated shocks across states by time. In this model, we're unable to reject the possibility than the number of registered donors by state is uncorrelated with the number of deceased donors. To be statistically significant, adding an additional million donors to the registry would have to be correlated with about ~4 more deceased donors per year, and we don't observe that correlation. I consider this the most credible model of the three.

I'm not sure what to make of the switch in the sign of the coefficient on state populations between Models 1 and 2, but I find it a little worrisome. In addition, dropping population from Model 2 results in a positive and statistically significant coefficient on the number of designated donors for reasons I don't really understand. (I mean, I know it's because population and the number of designated donors are correlated, but I'm not really sure why the population continues to be correlated with the number of deceased donors after eligible deaths and quarter and state fixed effects are included; it seems likely that it's because states that have faster-growing populations are also experiencing higher conversion rates, which I wouldn't necessarily expect.)

Model 3 also includes state and quarter fixed effects, and just regressed conversion rate on the portion of the state population registered as donors, not finding anything statistically significant.

I wasn't all that thorough in exploring the solution space of possible regressions on this data, and I'm not highly confident in the regression results, so both for the sake of replicability and to allow others to continue the analysis if they see fit, my poorly-documented R script is up on github here and the data it uses are on figshare here.

I didn't spend much time documenting the code or data, so drop me an email if you need help interpreting.

Conclusion

I've worked through two approaches to trying to estimate the impact of adding people to the list of registered donors. Though they differ quite a bit in assumptions and conclusions, both support the conclusion that the impact of adding a million people to the organ donor registry is likely less than 10 additional deceased organ donors per year, and probably quite a bit smaller than that. This would imply that Facebook's efforts to lead people to sign up as organ donors over the past year have led to <2 additional organ donors this year.

My vague impression is that the organ donation community spends a huge amount of its time, money, and effort trying to get more people to sign up as organ donors. My analysis here has been tentative and exploratory, but I think the lack of stronger empirical evidence for an actual impact of those registries is startling, and that it would be worth a fuller exploration by people with more subject-matter expertise than I have. Given that there are a lot of other strategies for improving organ donation, continuing to focus on donor registration outreach seems like it may be leaving a lot on the table.