Why isn't court-packing unconstitutional?

After I read this report, I found it disturbingly easy to imagine a judicial opinion invalidating court-packing (many of these points are in Todd’s piece):

First it would talk about how in general Congress has power to structure the Court, but that under the long-established principles of the separation of powers, no one branch can be allowed to effectively destroy another.

Then it would argue that Congress has never before engaged in overtly outcome-motivated court-packing. (It’s not clear whether this is true, but there’s a respectable historical debate about the Civil War/Reconstruction era and other relevant moments, so the Court could say it with a straight face.)

Then it would note that the closest precedent was the New Deal battle over court packing, in which the legislative branch—dominated by a supermajority of the President’s own party—responded with powerful constitutional arguments, which may have helped carry the day against the President.