It’s no secret that the Ninth Circuit has a reputation for giving the Supreme Court fits, but is it true that they have an 80% reversal rate? The answer is yes … and really no. Confused? Many are, and the claim of an 80% reversal rate makes its way into many corners of the media. It’s mainly used to support arguments to break up the Ninth Circuit or find ways to circumvent it in order to prevent almost-certain erroneous outcomes.

The Daily Caller is hardly unique, although they did do a better job than most in qualifying the way in which the calculation is made (emphasis mine):

The 9th U.S. Circuit Court of Appeals — which ruled Thursday against reinstating President Trump’s travel ban affecting seven terror-prone countries — has an unusually high reversal rate before the U.S. Supreme Court.

Eight of out of 10 cases from the 9th Circuit reviewed by the Supreme Court are overruled, according to a 2010 analysis published by the American Bar Association. The 9th Circuit, which is known for its liberal tendencies, has the second-highest reversal rate of the 13 appellate courts below the Supreme Court.

“Reviewed by the Supreme Court” is the operative qualifier — and it’s a very, very important one. Very few cases actually get reviewed by the Supreme Court from any of the circuit courts, and most of them don’t even generate appeals to the Supreme Court in the first place. Parties file appeals to the Supreme Court, which then has to decide whether the justices want or need to review the case. If fewer than four of the justices think that the appeal has merit, the application for certiorari is denied, keeping the appellate decision in place. This happens in most cases.

What does that mean in practical terms? It means that the Supreme Court’s grant of certiorari tend to favor those cases that are likely to be overturned. It’s a major selection bias, and as we’ll see, it gives a very distorted picture of what happens in the appellate court system.

Let’s take a look at the ABA report that generated this talking point. The study covered ten years (1999-2008) across all appellate circuits. During that period of time, the total number of cases decided by all appellate courts was 604,665. How many did the Supreme Court accept for their review? A mere 660 cases, or 0.109% of all decisions reached by the appellate level. The Ninth Circuit accounted for 175 of the cases reviewed, or about 26.5%, but the same circuit handled 114,199 of all appellate cases — 18.9% of the total.

The surpassingly small number of reviews made it difficult to set up a metric that could distinguish between the courts. Dealing with variant outcomes within 0.109% of an overall data pool makes outcomes statistically insignificant, as the ABA hinted in its report:

Reversal rates for each court of appeals would be very small, in the range of a tenth of a percent, if calculated as the total number of cases reversed over the total number of appeals terminated by that court. Conversely, if the reversal rate is calculated as the total number of cases reversed over the total number of cases reviewed by the Supreme Court, the ratio increases dramatically. So, in the big picture, i.e., considering all of the appeals terminated by each circuit, reversal rates for all courts of appeals could be very low, if calculated by the former method, or very high, if calculated by the latter method.

The ABA decided to use both to some extent, but even the latter method (reviewed cases only) does not show the Ninth Circuit as wildly off the scale. In fact, they’re only second highest in reversal rates, and the distribution is remarkably close to the median:

This is where the selection bias of using only certiorari-granted cases shows up. Every circuit has more than half of its decisions selected by the Supreme Court for review overturned, and the median is 69%. The deviation of performance between the circuits is hardly dramatic.

Perhaps a better metric would be the percentage of cases reviewed by the Supreme Court. In this comparison, the Ninth Circuit ends up a little farther off from the median, but it’s only third “worst”:

The DC Circuit deals with more constitutional issues with federal government, the kind of cases in which the Supreme Court will naturally have a higher interest, which likely accounts for its highest review rate. But the percentages are so low here as to make comparisons insignificant statistically, the very problem that the ABA noted in its own report.

It’s true in a very narrow sense that the Ninth Circuit has an 80% reversal rate for that ten-year period, but it’s also true that they have a 0.123% reversal rate — and that’s actually the more accurate way to view their performance.

However, the ABA report does give evidence of another kind about the performance of the Ninth Circuit. As noted above, that circuit handled 18.9% of all appellate cases, far higher than any other circuit; the next closest was the Fifth Circuit, with 14.1%, and then the Eleventh Circuit with 12.3% of all appellate cases decided. Those three circuits heard slightly over 45% of all appeals, while the rest heard fewer than 10% each. That certainly seems like a good argument for splitting the Ninth Circuit:

They say it’s overloaded and overly liberal. But they’ll still face the difficulty of getting a plan past California Democrats, who have traditionally wanted nothing of it.

Sullivan and Republican Sen. Steve Daines of Montana have introduced two bills. One establishes a commission to study the appeals court system and find a quick and effective way to divide up the 9th Circuit’s caseload. The other would split the court into two: the 9th Circuit and a newly created 12th Circuit. The 9th Circuit would include California, Guam, Hawaii and the Northern Mariana Islands. The new circuit would include Alaska, Arizona, Idaho, Montana, Nevada, Oregon and Washington.

The ABA data makes it clear that the burden is much higher on the Ninth Circuit, or at least was between 1999-2008. That’s a better reason to create a new circuit and to shift the burden more equitably, and this Congress should take that up soon.

Update: A couple of followers on Twitter argue that the subset of reviewed cases is a “sample” that shows an overall problem. That’s wrong for a number of reasons. First, to draw conclusions from an audit or sample, the sample has to be random. It also has to be larger than 0.109% of the whole, for the same reasons I stated above — the sample would be so small that a few random adverse examples would skew the overall accuracy of the conclusions, which is exactly what this does here. The subset of reviewed cases (660 out of 604,665) are chosen by the Supreme Court specifically for their likelihood of resulting in a reversal, while many many more are rejected because of their low likelihood of reversal. That’s not sampling at all.

Update, 2/11: Edited slightly to emphasize which method is used for each result.