But if you’re not accounting for testing patterns, it can throw your conclusions entirely out of whack. You don’t just run the risk of being a little bit wrong: Your analysis could be off by an order of magnitude. Or even worse, you might be led in the opposite direction of what is actually happening. A country where the case count is increasing because it’s doing more testing, for instance, might actually be getting its epidemic under control. Alternatively, in a country where the reported number of new cases is declining, the situation could actually be getting worse, either because its system is too overwhelmed to do adequate testing or because it’s ramping down on testing for PR reasons.
Failure to account for testing strategies can also render comparisons between states and countries meaningless. According to two recent epidemiological studies, which tried to infer the true number of infected people from the reported number of deaths, there is roughly a 20-fold difference in case detection rates between the countries that are doing the best job of it, such as Norway and the worst job, such as the United Kingdom. (The United States is probably somewhere in the middle of the pack by this standard.) That means, for example, that in one country that reports 1,000 COVID-19 cases, there could actually be 5,000 infected people, and in another country that reports 1,000 cases, there might be 100,000!
There is also a lot of uncertainty about the true numbers of infections within a given country. According to an expert survey published by FiveThirtyEight, the number of detected cases in the United States could underestimate the true number of infected people by anywhere from a multiple of two times to 100 times. The same holds in other countries. A recent paper published by Imperial College London estimated that the true number of people who had been infected with the coronavirus in the U.K. as of March 30 was somewhere between 800,000 and 3.7 million1 — as compared to a reported case count through that date of just 22,141.