Monday I wrote about a new antibody study carried out in LA County which concluded that around 4 percent of the county had already developed antibodies to the virus. That would work out to around 224,000 people in the county who have had the virus and either had very mild symptoms or none at all. That study and an earlier one from Santa Clara County which reached similar conclusions seemed to indicate that the death rate of the virus could be much lower than previously believed, maybe between 0.1 and 0.2 percent.

But there are some potential problems with those results which made me want to give them a second look. The first point comes from Josh Marshall at Talking Points Memo. In a post he wrote yesterday, he points out that both the Santa Clara and LA County studies were organized by a pair of doctors from Stanford. Here’s Dr. Jay Bhattacharya touting the results of the Santa Clara study:

Dr. Jay Bhattacharya, Stanford School of Medicine, from Uncommon Knowledge pic.twitter.com/B6HByva5ZW

— TPM Livewire (@TPMLiveWire) April 21, 2020

The problem, as Marshall points out is that this low death rate Dr. Bhattacharya arrives at doesn’t seem possible when you look at the actual death toll in New York:

The numbers are straightforward: as of two days ago, there were 9,101 lab confirmed cases and 4,582 presumptive diagnosed cases for a total of 13,683 fatalities In New York City. The population of New York City is 8,398,748. That comes to either .11% or .16% depending on which death toll number is used.

To restate this, if you assume a death rate like what the Santa Clara and LA County studies suggest, then to get to the actual death toll reported in New York City, you have to assume very close to 100 percent of the city has been infected with the virus already. Not even the Swedes would believe those numbers without some strong evidence. And if you assume that everyone in NYC has not had the virus yet, then that means the actual death rate is higher than the Santa Clara study claims:

Even if we assume what I think most experts would consider the highly unlikely possibility that 50% of New Yorkers have been infected with COVID19 that would mean a .33% IFR. To be generous, let’s say a third of the population of New York City had already been infected with COVID19 – very high but not inconceivable. That would mean a IFR of .49%.

The point here is that the actual death toll in NYC seems to be sending us a signal that the death rate calculated from the Santa Clara and LA County studies isn’t accurate. Those studies calculated the death rate based upon the results to antibody tests in those counties. In Santa Clara they tested 3,330 people and got 50 positive results. But there are some potential problems with the study. The most significant problem was pointed out in detail by Columbia statistician Andrew Gellman on his blog. It involves the accuracy, technically the “specificity,” of the antibody tests used.

If X% of the population have the antibodies and the test has an error rate that’s not a lot lower than X%, you’re in big trouble. This doesn’t mean you shouldn’t do testing, but it does mean you need to interpret the results carefully…

Specificity . . . measures the proportion of actual negatives that are correctly identified as such.” OK, here [our] concern is actual negatives who are misclassified, so what’s relevant is the specificity…

If the specificity is 90%, we’re sunk. With a 90% specificity, you’d expect to see 333 positive tests out of 3330, even if nobody had the antibodies at all. Indeed, they only saw 50 positives, that is, 1.5%, so we can be pretty sure that the specificity is at least 98.5%. If the specificity were 98.5%, the observed data would be consistent with zero… On the other hand, if the specificity were 100%, then we could take the result at face value.

The manufacturer of the test put the specificity at 99.46 percent, which sounds pretty good. But Gellman concludes that when you add a standard 95% confidence interval to that figure there’s a small possibility that all of the positive results in the Santa Clara study (again 50 positives out of 3,330 tests) could have been false positives.

If the specificity data in the paper are consistent with all the tests being false positives—not that we believe all the tests are false positives, but this suggests we can’t then estimate the true positive rate with any precision—then how do they get a confidence nonzero estimate of the true positive rate in the population?…

It’s pretty unlikely that all of the positives were false positives but even if 1/3 of the positives were false the estimated prevalence of the virus drops dramatically and the death rate jumps up. The point is that the accuracy of the test used (that photo above shows the actual test used in LA County) has a lot to do with the results when the number of actual positives is this small.

One possible solution would be to re-contact those 50 people who tested positive in Santa Clara (or the similar number who tested positive in LA County) and perform a more accurate lab test to verify the results of the field test. That might help pin down the specificity of the test.

Unfortunately, all of this is pretty important. If the coronavirus is only slightly more deadly than the seasonal flu that could result in one set of policies to manage it. If on the other hand it is five times deadlier than the seasonal flu (or higher), that would probably have a significant impact on the kind of mediation efforts the public is willing to tolerate, i.e. masks, social distancing, the duration of stay-at-home orders. How soon the country re-opens for business will depend on getting accurate answers to these questions.