Video: East Anglia CRU’s below-standard computer modeling

posted at 10:55 am on December 16, 2009 by Ed Morrissey

While the rest of the world focused on the e-mails from East Anglia’s CRU describing attempts to “hide the decline” and silence critics, a programmer in the UK focused on the programming code for CRU’s computer modeling. John Graham-Cumming says he’s not an AGW skeptic, but he’s becoming a skeptic of East Anglia, thanks to its below-standard computer programming for its climate modeling. Breitbart TV has his assessment and damning conclusion in this clip, which gets down to the brass tacks: would you spend money on any conclusions offered by these models? Watch the clip below for the answer:

Breaking on Hot Air

Blowback

Note from Hot Air management: This section is for comments from Hot Air's community of registered readers. Please don't assume that Hot Air management agrees with or otherwise endorses any particular comment just because we let it stand. A reminder: Anyone who fails to comply with our terms of use may lose their posting privilege.

Trackbacks/Pings

Trackback URL

Comments

Comment pages: 1 2

But in all fairness, this probably isn’t the actual model.

I would tend to believe that most of the models are developed as “genetic algorithms” using a large “training” data set. The real downside to most “genetic algorithms” is that the users usually have no idea what is going on “inside the box,” the whole thing is a “black box” usually consisting of thousands of “fudge factors.”

J_Crater on December 16, 2009 at 12:08 PM

What’s the language with comefrom statements? I remember one of my professors always talking about a programming language designed to look like signal noise.

So it could be worse.

jhffmn on December 16, 2009 at 12:08 PM

BH 0bama’s economic algorithm:

1) Steal Underpants
2) ???
3) PROFITS!

noblejones on December 16, 2009 at 12:09 PM

MobileVideoEngineer on December 16, 2009 at 11:59 AM

Hopefully you only have to see languages that can be traced to C. Time to wax nostalgic for code with & (address of), ** (double pointers), and some serious computer crashes while debugging code.

WashJeff on December 16, 2009 at 12:10 PM

‘Fudge Factor’ is a technical term, Hershey’s scientists use it all the time.

mudskipper on December 16, 2009 at 11:33 AM

The scientists in San Francisco use the “Fudge Packer Factor” in their calculations.

csdeven on December 16, 2009 at 12:10 PM

Hopefully you only have to see languages that can be traced to C. Time to wax nostalgic for code with & (address of), ** (double pointers), and some serious computer crashes while debugging code.

WashJeff on December 16, 2009 at 12:10 PM

I still write in-line assembly on occasion, when run-time performance is pri 1

DarkCurrent on December 16, 2009 at 12:14 PM

http://www.youtube.com/watch?v=qlwPf-4eMig

My submission for the dumbest climate video in history.

Assembler, pah, not not a real programmer until you have coded in hex code using nothing but displacements from the c register!!!

sonofdy on December 16, 2009 at 12:18 PM

I learned in APL where my comments were likely to be far longer than the actual code, itself… but cryptic programming does that to a coder.

From what I’ve remembered after some reading around is that EA CRU started out with a FORTRAN base of programs from the 70′s and 80′s, then started to meld them together. After that is anybody’s guess as they didn’t keep back-up copies, didn’t document code changes, didn’t have an interation system for documenting code changes so each set of work was documented and generally, with all the AGW funds going through the place, didn’t hire a competent coder. I took some years of CS in imperative and non-imperative coding, and got a geology degree. Seemed to me to work good code as a scientist I had to know quite a bit of programming… there were very few that could do both the science and the coding competently.

From what I can see that is still the case.

Unless you use stock programs that are well documented and known and change NOTHING in the underlying program, itself, then any scientific team needs a competent coder to do anything for them. Of course you can work veritable magic with modern modeling and scientific analysis programs… but they weren’t using modern tools at EA CRU. Not even from the 1990′s. And if you can’t familiarize yourself with the tools, expect to get your thumb in the way of the hammer a few times…

ajacksonian on December 16, 2009 at 12:18 PM

“I know I’ve made some very poor decisions recently, but I can give you my complete assurance that my work will be back to normal. I’ve still got the greatest enthusiasm and confidence in the mission. And I want to help you.”

I honestly think you ought to sit down calmly, take a stress pill, and think things over. …

R Square on December 16, 2009 at 12:20 PM

Hopefully you only have to see languages that can be traced to C. Time to wax nostalgic for code with & (address of), ** (double pointers), and some serious computer crashes while debugging code.

WashJeff on December 16, 2009 at 12:10 PM

I still write in-line assembly on occasion, when run-time performance is pri 1

DarkCurrent on December 16, 2009 at 12:14 PM

I never liked C, but then again, my professor for C didn’t exactly teach because she asked the class how many people had taken C or C++ in high school and all but 3 or 4 people out of 50 or so raised their hands. They only offered Visual Basic 6.0 at my high school.

However, now my favorite language is C#. I liked my Assembly class though, you had to do a lot of stuff to get very little done, but it was pretty cool because you could actually control the hardware pretty easily.

MobileVideoEngineer on December 16, 2009 at 12:22 PM

I still write in-line assembly on occasion, when run-time performance is pri 1

DarkCurrent on December 16, 2009 at 12:14 PM

Must be working on some embedded systems (e.g., cell phones). Had to do the same while working at Motorola in the 90′s.

WashJeff on December 16, 2009 at 12:23 PM

They “Wee-weed up” that code. Missing basic loop logic that should have been cover in the first course/instruction book offered. If like was that easy, in case of Error; ignore.

jlemieu1 on December 16, 2009 at 12:31 PM

How DARE you so-called programmers deny the truth of this.

The programming is settled.

Lily on December 16, 2009 at 12:31 PM

More code snippets from the model:

double GetAverageHomigenizedTemp(int y)
{
double AvgTempThisYear = GetRawData(y);
double AvgTempLastYear = GetRawData(y – 1);
if (AvgTempThisYear < AvgTempLastYear)
{
AvgTempThisYear = AvgTempLastYear + 0.5;
}
return (AvgTempThisYear);
}

WashJeff on December 16, 2009 at 11:04 AM

Hey Jeff, how’d you cause the indents on your code snippet? Tabs?

VoyskaPVO on December 16, 2009 at 12:33 PM

However, now my favorite language is C#. I liked my Assembly class though, you had to do a lot of stuff to get very little done, but it was pretty cool because you could actually control the hardware pretty easily.

MobileVideoEngineer on December 16, 2009 at 12:22 PM

I use assembly when the compiler won’t be smart enough to take full advantage of the hardware instruction set and it’s important to get maximum speed. I don’t write most code directly in assembly of course, but there are hotspots that can benefit trememdously from a programmer who knows the processor instruction set and can code those sections for maximum perf.

DarkCurrent on December 16, 2009 at 12:33 PM

Must be working on some embedded systems (e.g., cell phones). Had to do the same while working at Motorola in the 90’s.

WashJeff on December 16, 2009 at 12:23 PM

No, Intel-platform apps and system components.

DarkCurrent on December 16, 2009 at 12:37 PM

I never liked C, but then again, my professor for C didn’t exactly teach because she asked the class how many people had taken C or C++ in high school and all but 3 or 4 people out of 50 or so raised their hands..

MobileVideoEngineer on December 16, 2009 at 12:22 PM

..C or C++ is far better than Fortran..

VoyskaPVO on December 16, 2009 at 12:39 PM

For example, current project using artificial neural networks. There are huge numbers of floating point multiplications that can be done a lot faster using the Intel SSE instructions directly.

DarkCurrent on December 16, 2009 at 12:41 PM

VoyskaPVO on December 16, 2009 at 12:33 PM

Hotair accepts the forced space html code, which is (I need to insert spaces between characters otherwise you will not see it): & n b s p ;

Tighten that up and you have a fixed space character. I also used the html <code> tag.

WashJeff on December 16, 2009 at 12:50 PM

Henrik Svensmark has an alternate theory of climate change that thoroughly debuks the myth that climate change is driven solely by CO2 and the greenhouse effect. He can show his work, too, unlike CRU and NASA. In fact, he has developed a whole new branch of science: cosmoclimatology. Fascinating stuff — yet he is labeled a ‘denier’ and a ‘skeptic’, rather than celebrated as the Einstein of our time.

This is part 1 of a 5 part youtube video on his work: http://www.youtube.com/watch?v=n1qGOUIRac0

The thing is, as long as ‘skeptics’ concentrate only on tearing down the ‘alarmists’ theories and substandard science they are always going to be playing catch-up. It’s time to get out in front. Henrik Svensmark’s work captures the imagination and takes the whole discussion on climate change onto a brand new track.

starboardhelm on December 16, 2009 at 12:51 PM

..C or C++ is far better than Fortran..

VoyskaPVO on December 16, 2009 at 12:39 PM

Oh I don’t doubt that. I learned C++ later after I learned Java. I went to Syracuse and they were big into Java and then I transferred to Wright State and they were more of a C++ school. Java used to be my favorite language, but then I taught myself C# and I like it because it’s so much faster than Java. C# is really a mix of the best of Java and C++.

MobileVideoEngineer on December 16, 2009 at 12:52 PM

With a little research it looks like the model is written in the IDL language. One benefit being, “Operators and functions work on entire arrays (without using loops), simplifying interactive analysis and reducing programming time.”

If you’re interested search the web for “if !d.name eq ‘X’ then begin” and you’ll find a ton of examples out there, including a bunch from .gov domains.

There’s also a lively discussion of this particular piece of code here. I don’t think anything has been resolved with this yet. We can examine the code all we want, but WHERE IS THE ORIGINAL & MODIFIED DATA.

Heck, I’ve been a programmer for 24 years, and I wouldn’t try to validate someone else’s code without also having the original inputs, and the steps taken to produce the output.

Keith_Indy on December 16, 2009 at 12:52 PM

Yeah, assembly can help significantly, generally, as it is a lower-level language than C. However, using C (which is a low-level language) with embedded assembly and/or compiler options can often do at least as good a job in considerably less time than writing strictly assembler.

In fact, the number of ways that the GCC compiler (which seems to be the up and coming Defacto standard) can be tweaked is absolutely frightening.

Generally, the programming I do is more I/O bound than anything else, so out and out assembler won’t help me that much (plus, programming datagrams or packets in C is bad enough. The thought of debugging an assembler program doing so, is enough to give me the chilly willies).

Natrium on December 16, 2009 at 1:02 PM

How the f**k do you model the universe or even our solar system which composes the many variables that one must take into account for climate change. Any climate change model has to be full of guesstimations and abitrary limiting of many variables to fixed constants, and we are supposed to swallow all of this as gospel? Utterly ridiculous!!!

docdave on December 16, 2009 at 1:04 PM

OK, the UK Met Office has a webpage that allows you to download some of the raw data being used.

http://www.metoffice.gov.uk/climatechange/science/monitoring/subsets.html

So, anyone with time on their hands, and large bandwidth, could download this data, and start analyzing it independently.

Here is one such guy doing exactly that.

Keith_Indy on December 16, 2009 at 1:11 PM

Bet you anything the source code used in your browser or the server this is being read from has similar problems. Not that this isn’t a problem given the stakes, but most programs out there are rather frightening and inscrutable once you look at them up close.

Anyway, I do like the fact that this thread has gotten people talking about assembly language….

calbear on December 16, 2009 at 1:12 PM

Where can I find a copy of this code?

I can read C++, Java and Visual Basic. I would just LOVE to sink my eyes into it and give a copy to one of my professors and see if his findings agree with mine…

newton on December 16, 2009 at 1:23 PM

Oh what a tangled web we weave when first we practice code to deceive!
- Sir Walter Scott

MB4 on December 16, 2009 at 1:29 PM

hey, they put a man on the Moon with 48k of RAM. How’s the manned space program doing in the XP era?

Chris_Balsz on December 16, 2009 at 11:03 AM

The thought of NASA using Windows is chilling.

unclesmrgol on December 16, 2009 at 2:36 PM

Mr. Jones, ahead Fudge Factor 9!

Aye, Captain!

GT on December 16, 2009 at 2:51 PM

hey, they put a man on the Moon with 48k of RAM. How’s the manned space program doing in the XP era?

Chris_Balsz on December 16, 2009 at 11:03 AM

I think the AGC (Apollo Guidance Computer) had 4K RAM (data area), and 72K ROM (program area). For both code and data, 16 bits — of which one was parity and, for data, another was sign. Maybe you were adding up the RAM in all the other computers used for the Apollo (each Apollo spacecraft had one AGC, and the lunar module had another). Then there was the AGS (Abort Guidance System), — that had 4K 18-bit words of memory, equally divided into RAM and ROM. Finally, I think the Saturn launch vehicle had another computer, whose characteristics I can’t remember, except that it was more complex than either the AGC or the AGS. And then there were the ground computers, which I think were IBM 7040s which would have had about a megabyte of main memory.

unclesmrgol on December 16, 2009 at 3:11 PM

basic is okay for some things but i would have thought fortran, pascal, or assembler.

sonofdy on December 16, 2009 at 11:19 AM

I would have thought they’d use a special purpose language specifically designed for data reduction, regression analysis, etc. which interfaces well with both databases (via SQL) and Excel spreadsheets, such as SAS.

unclesmrgol on December 16, 2009 at 3:24 PM

Garbage In = Garbage Out

AGW = Anthropogenic GARBAGE Warming

dont taze me bro on December 16, 2009 at 3:24 PM

I still write in-line assembly on occasion…

DarkCurrent on December 16, 2009 at 12:14 PM

Haven’t done that in over thirty years since I worked with a Digital PDP8… I have a couple of microfiche in the back of my thesis to remind me of having done it… sigh

Which brings me to my next point: these CRU geniuses’ models probably wouldn’t pass the “slide rule accuracy” criterion that once was used to grade exams.

ya2daup on December 16, 2009 at 3:36 PM

What is lacking in the argument is that simply being a competent programmer is not enough if you do not understand the science. No matter how good you are at programming if you have no grasp of the science behind the hydrogen atom you’ll never write a decent program to model the hydrogen atom.
What the leaks indicate is a failure on all levels.

jdkchem on December 16, 2009 at 3:38 PM

If look at the stair-stepping of the results plotted, you can see something weird was done there. No data set from nature should plateau and and then rise only to flatten out again.

The emails themselves confirm the results they were getting showed a decline in temps and they had to “fix” the data/model to hide it otherwise their whole point was moot (that climate was irrevocably getting hotter and hotter).

Anyone with a brain should understand that there was an ice age 10,000 years ago that ended – so what made the planet warm up? There were no cities and cars and jet planes then.

Neo on December 16, 2009 at 3:39 PM

Another thing on climate modeling, a guy named Lorenze proved long ago that climate changes are non-linear but I’ll bet that the climate models are all based on linear functions. If one models non-linear functions with linear equations, they are going to be way off much of the time.

docdave on December 16, 2009 at 3:43 PM

A statistics PhD guy on Rush’s show a few day’s ago said he was looking at the code and said that there where points in the code that were data dependent. I interpreted what he said that their where conditional statements based on the data the SW was processing (i.e., the program was rigged for a predetermined result).

WashJeff on December 16, 2009 at 11:16 AM

I interpreted his statement to mean that they had hard-coded some of the data into the programs — which lends some credence to rigging the results — rather than reading it in and letting the chips fall where they may.

ya2daup on December 16, 2009 at 3:48 PM

Why are they using BASIC? That should discredit anything they’re doing, right there.

Daggett on December 16, 2009 at 12:08 PM

It’s not BASIC its FORTRAN. It’s not uncommon for governments and universities to still be using FORTRAN and COBOL.

CableDude on December 16, 2009 at 3:53 PM

The way to get by with sloppy programs is to do it in house and not have outside people see or review it.

seven on December 16, 2009 at 11:29 AM

…which is more or less what the folks at CRU did, no?

ya2daup on December 16, 2009 at 4:02 PM

I interpreted his statement to mean that they had hard-coded some of the data into the programs — which lends some credence to rigging the results — rather than reading it in and letting the chips fall where they may.

ya2daup on December 16, 2009 at 3:48 PM

If I were being paid to write program to do what this one does, I would want to make sure it was as hard to read as possible.
Or go exactly the opposite. Comments like ” Added this line so that it returns the value that Dr. Quackenstein said he wanted” , might have went a long way in covering this guys butt.

CableDude on December 16, 2009 at 4:13 PM

Another thing on climate modeling, a guy named Lorenze proved long ago that climate changes are non-linear but I’ll bet that the climate models are all based on linear functions. If one models non-linear functions with linear equations, they are going to be way off much of the time.

docdave on December 16, 2009 at 3:43 PM

Chaos theory basically says all this climate modeling is a waste of time anyway, regardless of how valid your code or data may be. It’s amazing we can even predict the weather next week with any accuracy, much less what the weather will be 50 years from now.

And yes, weather IS climate, just aggregated over time.

ZenDraken on December 16, 2009 at 4:54 PM

It’s so simple people…They’re using frog DNA to fill in the missing areas of data.

MechEng5by5 on December 16, 2009 at 4:59 PM

It’s so simple people…They’re using frog DNA to fill in the missing areas of data.

MechEng5by5 on December 16, 2009 at 4:59 PM

Aha! That explains the worldwide decline in amphibian populations that zoologists have been whining about in recent years! You evil, evil warm-mongers: how could you have done this and yet still claim to love Gaia?

ya2daup on December 16, 2009 at 5:27 PM

PROGRAMMING LESSON:

If your data plots as a flat line and you want a Hockey Stick, simply add a Hockey Stick to the data.

….but they could have added in a reclining Dolly Parton and made it much more entertaining…and equally valid!

Models which have not been VALIDATED against the real world are completely useless.

landlines on December 16, 2009 at 6:06 PM

For a very good review of what should be done to get good scientific results from climate models I recommend a piece by Jerry Pournelle on his blog. It details how real scientists would act if they were adhering to established scientific principles.

http://www.jerrypournelle.com/view/2009/Q4/view600.html#Wednesday

donbury on December 16, 2009 at 7:54 PM

Ah, I do long for the days of my Vic 20 and Commodore 64!!!
The only programming I ever did was on those two machines.

THIS IS WHAT HAPPENS WHEN YOU TRY TO FIX DATA TO THEORY AND NOT THEORY TO DATA!

Reminds me of the Gilligan’s Island episode where Mr. Howell is trying to complete a puzzle and the piece he has will not fit in the given space. His answer was to, “Break off a piece and make it fit!”.

Sounds like we have a bunch of Gilligans here . . . .
But then we have been saying this for how long now . . . .

Good grant money (and that is the key here folks – it is all about the grant money) down the drain.

Bubba Redneck on December 16, 2009 at 11:49 PM

Comment pages: 1 2