CNN: Blowing up HealthCare.Gov and starting over still looking like their best option, say more tech experts

posted at 2:41 pm on October 23, 2013 by Erika Johnsen

There’s a growing consensus on the mid- to late-November time frame by which the Obama administration really, really needs to have ObamaCare’s online portal running smoothly to avoid inducing more death-spiral risks and subsequent industry panic, and all of their plans for public outreach and directing people to call centers and whatnot is only going to get them so far.

“Tech surge” or no tech surge, CNN reports that still more experts and computer engineers are piling on to the suggestion that fixing HealthCare.Gov’s major problems before the end of 2013 just isn’t a feasible task, and that rebuilding the system from scratch would be the administration’s easiest and safest bet:

After assessing the website, Dave Kennedy, the CEO of information-security company Trusted Sec, estimates that about 20% of Healthcare.gov needs to be rewritten. With a whopping 500 million lines of code, according to a recent New York Times report, Kennedy believes fixing the site would probably take six months to a year.

… Nish Bhalla, CEO of information-security firm Security Compass, said it “does not sound realistic at all” that Healthcare.gov will be fully operational before that point.

“We don’t even know where all of the problems lie, so how can we solve them?” Bhalla said. “It’s like a drive-by shooting: You’re going fast and you might hit it, you might miss it. But you can’t fix what you can’t identify.” …

“Projects that are done rapidly usually have a lot of [repetitive] code,” said Arron Kallenberg, a software engineer and tech entrepreneur. “So when you have a problem, instead of debugging something in a single location, you’re tracking it down all through the code base.”

A whopping 500 million lines of code is “so excessive,” says Kennedy, and that a more normal number for a project like the ObamaCare site would lie somewhere in the range of 25 million to 50 million. Dayum.

Admitting that they screwed up and throwing out more than $300 million taxpayer dollars’ worth of work would be one heck of a politically bitter pill to swallow, but this looks like it’s quickly turning into a damned-if-they-do, damned-if-they-don’t sort of scenario. Who could’ve seen that coming?


Related Posts:

Breaking on Hot Air

Blowback

Note from Hot Air management: This section is for comments from Hot Air's community of registered readers. Please don't assume that Hot Air management agrees with or otherwise endorses any particular comment just because we let it stand. A reminder: Anyone who fails to comply with our terms of use may lose their posting privilege.

Trackbacks/Pings

Trackback URL

Comments

Comment pages: 1 2

Implementing a “command and control economy” is much harder than many people thought.

mrjavascript on October 23, 2013 at 2:43 PM

Or …

… We repeal the whole thing! The PROBLEM is NOT the website!

The Death Spiral is already in place for the young:

1) Thanks to the Democrats incompetence on the economy there are not enough new jobs being created for new workers entering the job market.

2) Thanks to the marriage of economic incompetence with Obamacare – Many of the jobs BEING created are part time, poorly paid positions without benefits.

3) College graduates are running up unprecedented levels of education debt …. but having the degree means only 50% of the graduates are finding jobs … the other 50% have government sponsored debt, which is driving up the cost of college, they CAN’T get out of even if they declare bankruptcy.

and last but not least …

4) If they’ve found a decent paying job, young workers now get to FUND their parent’s & granparent’s healthcare costs OR pay 1% of their income as a punitive TAX for not taking the WORST DEAL EVER OFFERED!

Hey, Hot Air editors! Tell us again what a HUGE MISTAKE it was to shut down the government to try to prevent our children being RAPED by Obamacare.

PolAgnostic on October 23, 2013 at 2:43 PM

We must blow it up to see what’s in it…..

Electrongod on October 23, 2013 at 2:44 PM

“Who could’ve seen that coming?”

uhmmm..any sane person

going2mars on October 23, 2013 at 2:44 PM

And what happens when the call centers are overwhelmed? Are people expected to sit on hold for hours? I can’t imagine that signing up for insurance is a quick 3 minute call? Even Geico needs 15 minutes to sign you up for car insurance.

kooly on October 23, 2013 at 2:45 PM

Admitting that they screwed up and throwing out more than $300 million taxpayer dollars’ worth of work would be one heck of a politically bitter pill to swallow, but this looks like it’s quickly turning into a damned-if-they-do, damned-if-they-don’t sort of scenario. Who could’ve seen that coming?

And if we follow Juan Williams line of thinking – the administration isn’t about to let the Rethuglicans bully or shame them into redoing it – so we’ll proceed full steam ahead with the existing program.

Hill60 on October 23, 2013 at 2:45 PM

Admitting that they screwed up and throwing out more than $300 million taxpayer dollars’ worth of work would be one heck of a politically bitter pill to swallow, but this looks like it’s quickly turning into a damned-if-they-do, damned-if-they-don’t sort of scenario. Who could’ve seen that coming?

Oh please. They threw away nearly twice that amount on Solyndra and didn’t even hint at any remorse. I don’t think it’s the wasted money they’re concerned about. It’s admitting that they need to go back to square one.

Doughboy on October 23, 2013 at 2:46 PM

Remember they spent all that money…hundreds of billions for over a decade trying to update the intel community computers, database relationships, and software…even basic Boolean searching? Had to eventually abandon it and the project was never completed as envisioned.

Wonder if that was no-bid, too.

winoceros on October 23, 2013 at 2:47 PM

$300 million? even the CNN article pegs the costs at $640 million.

dbilly76 on October 23, 2013 at 2:49 PM

How much is Obamacare already over budget? Heeeeellllllooooo GOP why aren’t you talking about this?

Oil Can on October 23, 2013 at 2:50 PM

Relax. Everything is going according to plan. And it is all the fault of the dirty Tea Baggers anyway. Or the Koch brothers. Everyone knows they DOS attacked the site and planted saboteurs to damage the code.

SunSword on October 23, 2013 at 2:50 PM

I love how a failed big government program is an even BIGGER big government program.

melle1228 on October 23, 2013 at 2:51 PM

How much is Obamacare already over budget? Heeeeellllllooooo GOP

why aren’t you talking about this?

Oil Can on October 23, 2013 at 2:50 PM

Because the GOP has become the party of “we can manage the entitlements better” instead of the party of “we need to get rid of the entitlements.”

melle1228 on October 23, 2013 at 2:52 PM

We need a government insurance program that pays out if something happens to your website. If they had that when Obamacare site failed the insurer would have to pay for this and not the government. Oh wait…nevermind.

Rocks on October 23, 2013 at 2:53 PM

I see an Ambassadorship in Sebelius’ future…

d1carter on October 23, 2013 at 2:53 PM

This whole “500 million lines of code” thing doesn’t make sense. The number of lines of code is not really a meaningful measurement of anything. See this explanation, for example: http://www.slate.com/articles/technology/bitwise/2013/10/healthcare_gov_problems_what_5_million_lines_of_code_really_means.html?wpisrc=burger_bar. My point is not that the website isn’t a horrible mess, just that talking about lines of code is mostly meaningless.

tneloms on October 23, 2013 at 2:53 PM

They could have just copied wwww.ehealthinsurance.com, integrated it with the government systems and been done.

Kill it.

Charlemagne on October 23, 2013 at 2:54 PM

But Prezzy Stompyfoot will NOT do this. It would be tantamount to saying the Repubs were right about the shutdown. Prezzy Stompyfoot has shown that he will do ANYTHING, including harming the American people to get his way.

neyney on October 23, 2013 at 2:54 PM

Too many “w’s”

Ehealthinsurance

Charlemagne on October 23, 2013 at 2:55 PM

Bring in Al Gore. He invented the Internet. He should be able to invent an Internet web site.

The Rogue Tomato on October 23, 2013 at 2:56 PM

Did the people that wrote HealthCare.gov previously work on the global warming models\simulations?

WashJeff on October 23, 2013 at 2:56 PM

CNN: Blowing up HealthCare.Gov and starting over still looking like their best option, say more tech experts

Why not? That’s exactly what Barky did with his Afghan Strategic Review (that he said took the Pentagon 6 months to finish and was, therefore, obsolete). In that, Barky sent the Pentagon back to do a second review – which took over 4 months – and then arbitrarily pulled his own numbers out of his azz, for which we then got the worst fatalities and casualties in Afghanistan since the beginning of the war and a deterioration fo our position there … not that the MSM reported any of that …

And just think of all the advantages for Barky and the America-hating left, another 3 years to build another web site that doesn’t work, meanwhile collecting the penalties taxes fines tribute tot he feral government while providing nothing. It’s a win-win for the feral government monstrosity!!

ThePrimordialOrderedPair on October 23, 2013 at 2:57 PM

Implementing a “command and control economy” is much harder than many people thought.

mrjavascript on October 23, 2013 at 2:43 PM

They should have consulted Gorbachev and Deng about that.

kcewa on October 23, 2013 at 2:58 PM

So more than half a billion down the drain. This is worse than Solyndra. Defend this LiveFreeOffOther and LostMotherFluker!!

jawkneemusic on October 23, 2013 at 2:58 PM

The GOP should be out with a five- to ten-point plan to improve the healthcare and health insurance industries. NOW. Now that it’s clear Obama lied about the reduction of premiums, the reduction of deficits, and keeping plans and doctors, the GOP has people’s attention. Show how the solutions will solve problems. The one thing Obama never did was explain how all these magical things were going to happen. I’ll bet voters would be saying, “Why the phuck didn’t they do those things in the first place?”

And for crying out loud, why isn’t the GOP reminding Americans that Obama built this?

BuckeyeSam on October 23, 2013 at 2:58 PM

They better not ask me to pitch in any more tax dollars…

djl130 on October 23, 2013 at 2:58 PM

Why yes, that should do it. I suggest calling it The Great Leap Forward.

Drained Brain on October 23, 2013 at 2:58 PM

And what happens when the call centers are overwhelmed? Are people expected to sit on hold for hours? I can’t imagine that signing up for insurance is a quick 3 minute call? Even Geico needs 15 minutes to sign you up for car insurance.

kooly on October 23, 2013 at 2:45 PM

Have you been to a DMV office lately?
They don’t give a dam about how long YOU sit on hold….

dentarthurdent on October 23, 2013 at 2:59 PM

They should have consulted Gorbachev and Deng about that.

kcewa on October 23, 2013 at 2:58 PM

They have turned the economy into deng, which is far worse than being all “wee wee’d up”, I am guessing.

ThePrimordialOrderedPair on October 23, 2013 at 2:59 PM

They better not ask me to pitch in any more tax dollars…

djl130 on October 23, 2013 at 2:58 PM

They won’t. The IRS will come in the middle of the night to TAKE it.

dentarthurdent on October 23, 2013 at 3:00 PM

I see an Ambassadorship in Sebelius’ future…

d1carter on October 23, 2013 at 2:53 PM

No, she’ll go to head up Planned Parenthood.

BuckeyeSam on October 23, 2013 at 3:00 PM

Hey did I miss the thread about Egypt severing diplomatic ties with the U.S. over Syria or is it B.S.?

Cindy Munford on October 23, 2013 at 3:01 PM

Desperation: White House Plans Obamacare Blitz, Dispatching Top Officials Across Country…

Like that will fix the website or something.

Resist We Much on October 23, 2013 at 2:56 PM

They’re focusing on the real problem:

“Looking to combat headlines dominated by the technical problems plaguing the ObamaCare website”

kcewa on October 23, 2013 at 3:01 PM

BuckeyeSam on October 23, 2013 at 2:58 PM

been saying that for months…..well put!!

going2mars on October 23, 2013 at 3:01 PM

I see an Ambassadorship in Sebelius’ future…

d1carter on October 23, 2013 at 2:53 PM

Preferably to Libya or Syria….

dentarthurdent on October 23, 2013 at 3:01 PM

Is it time to dig up Steve Jobs yet?

Cindy Munford on October 23, 2013 at 3:01 PM

Hey did I miss the thread about Egypt severing diplomatic ties with the U.S. over Syria or is it B.S.?

Cindy Munford on October 23, 2013 at 3:01 PM

Saudi Arabia…we already list Egypt

kcewa on October 23, 2013 at 3:03 PM

Desperation: White House Plans Obamacare Blitz, Dispatching Top Officials Across Country…
Like that will fix the website or something.
Resist We Much on October 23, 2013 at 2:56 PM

They’re focusing on the real problem:
“Looking to combat headlines dominated by the technical problems plaguing the ObamaCare website”
kcewa on October 23, 2013 at 3:01 PM

It’s all about the spin / PR – whether the web site actually works is irrelevant.

dentarthurdent on October 23, 2013 at 3:04 PM

And the C(r)icket Caucus remains dutifully silent.

AmeriCuda on October 23, 2013 at 3:05 PM

And what happens when the call centers are overwhelmed? Are people expected to sit on hold for hours? I can’t imagine that signing up for insurance is a quick 3 minute call? Even Geico needs 15 minutes to sign you up for car insurance.

kooly on October 23, 2013 at 2:45 PM
Have you been to a DMV office lately?
They don’t give a dam about how long YOU sit on hold….

dentarthurdent on October 23, 2013 at 2:59 PM

This is just the website. Now imagine the future when you are having a heart attack, have a broken bone, pneumonia, etc…You will be sitting in a dirty DMV type waiting room for HOURS. When you finally are seen it will be by some hack Dr. who couldn’t get a real gig due to incompetence (because most good Doc’s will be long gone rather than work for the govn’t for peanuts). If you survive and get sent to the hospital if you can get care your deductible will be in the thousands and you will be in an open area with dozens of patients. How do I know this? I have a friend from Ireland who was telling all of us about the joys of national healthcare. The only thing I’ve added is the deductible, over there they don’t pay one…but then again they let people with cancer sit on waiting lists for treatment so long they are too far gone by the time they get to the front of the line to save their lives and they are sent home with the UK version of the red pill.

neyney on October 23, 2013 at 3:07 PM

I think everyone is missing one huge point here. There are over 1 million people who have had their insurance cancelled effective 1 January 2014 because it does not comply with the O care health insurance mandates. There is no way on this green earth that those Insurance companies are going to reinstate those policies. Those people will be without any form of health insurance at midnight on New Years Eve.

Johnnyreb on October 23, 2013 at 3:08 PM

If 50 million lines of code should suffice…you have to ask yourself… why are there 500 million lines of code… and what was the intended purpose of the addition 450 million lines?

thatsafactjack on October 23, 2013 at 3:09 PM

“Blowing up Healthcare.Gov?” Erika, if you weren’t already, you just got yourself put on a terrorist watch list with that violent rhetoric.

besser tot als rot on October 23, 2013 at 3:14 PM

We must blow it up to see what’s in it…..

Electrongod on October 23, 2013 at 2:44 PM

…blow it up…with Botox?

KOOLAID2 on October 23, 2013 at 3:14 PM

$300 million? I think it’s closer to $600 million.

Pretty soon, we’ll be talking about real money!

LASue on October 23, 2013 at 3:15 PM

I think everyone is missing one huge point here. There are over 1 million people who have had their insurance cancelled effective 1 January 2014 because it does not comply with the O care health insurance mandates. There is no way on this green earth that those Insurance companies are going to reinstate those policies. Those people will be without any form of health insurance at midnight on New Years Eve.

Johnnyreb on October 23, 2013 at 3:08 PM

Let me guess Obama’s answer for this ‘glitch’ of his own making….
Government Takeover!!!

LASue on October 23, 2013 at 3:17 PM

“500 million lines of code”

I’ve seen this metric all over the place. It cannot be correct unless they are including the other ~50 disparate secondary systems (many preexisting).

If coding was started in “the spring” as claimed, the only way to produce 500M SLOC is to write an app that generates SLOC.

I also noted that there is a tremendous amount of convoluted ‘business logic’ based on policy. I guess if you don’t use a formal rules-engine for that and you implement it with if/then/else statements, you may indeed get 500M SLOC. But you’d need 50K developers that are very proficient at cut/paste/modify.

Whatever. We can talk about maintenance and extensibility for the ever-evolving business logic once it’s up and running…

Tsar of Earth on October 23, 2013 at 3:20 PM

Before it’s all over, the Dems will be begging to kick this ClusterfvckNado can down the road, past the 2014 mid-terms. There’s no way they’re going to merely lengthen the enrollment period or delay the mandate, either of which will precipitate the Death Spiral before November. The whole thing is going to have to be postponed, there’s really no other choice.

If they don’t, pass the popcorn, baby!

TXUS on October 23, 2013 at 3:20 PM

neyney on October 23, 2013 at 3:07 PM

Hell ya – you nailed it.

A few years ago – long before 0care, I went to a city owned hospital ER on my doctor’s orders – my blood pressure was up around 175 / 120. I checked in, they gave me a quick BP test and EKG in the triage room off the lobby, and then told me to go sit and wait until they called me. When I sat down, I noticed lots of other people waiting – some with broken bones, some running to the bathroom to barf every 3 minutes, and such. After 2 hours, and they still hadn’t gotten to the broken bones and barfers yet, who had been there longer than me, I gave up and told them I was going home. The receptionist/nurse warned me I might have a stroke or heart attack and die – and I told her “well, if that was really likely you would have done something in the last 2 hours, and besides at this point I’d rather die comfortable at home than sitting here in an uncomfortable chair in a crowded waiting room”.

Oh BTW – I didn’t die (as far as I know) – just to be clear for the libtard trolls….

That’s our healthcare future under 0care.

dentarthurdent on October 23, 2013 at 3:21 PM

Hey did I miss the thread about Egypt severing diplomatic ties with the U.S. over Syria or is it B.S.?

Cindy Munford on October 23, 2013 at 3:01 PM

You have to go to the foreign press to get the news…

d1carter on October 23, 2013 at 3:22 PM

If 50 million lines of code should suffice…you have to ask yourself… why are there 500 million lines of code… and what was the intended purpose of the addition 450 million lines?

thatsafactjack on October 23, 2013 at 3:09 PM

Speaking as a Software Engineer, and knowing how government software projects work, I’m pretty sure the contractor’s productivity was measured by LOC. Which is the absolutely worst way to run a project.

kcewa on October 23, 2013 at 3:24 PM

Haven’t watched fox news or tv for that matter in a long, long time. Seeing how cold and damp it’s been today.I think I’ll go home and make a tub of popcorn and watch the lying heads spin this epic failure tonight. Lol.

AH_C on October 23, 2013 at 3:24 PM

Just a reminder, scrapping a failed government system that cost taxpayers hundreds of millions has been done before. In 2005, the FBI was forced to scrap it’s “Virtual Case File” system. $581 million in taxpayer money flushed down the crapper. ObamaCare has $200 million to go before it becomes the biggest IT clusterfark in U.S. history. The champ is still “Trilogy”:

Some FBI agents ruefully refer to the trilogy project, a massive initiative to modernize the FBI’s aging technology infrastructure, as the “Tragedy” project. It certainly has all the earmarks of tragedy: the best intentions, catastrophic miscommunication, staggering waste.

After more than four years of hard work and half a billion dollars spent, however, Trilogy has had little impact on the FBI’s antiquated case-management system, which today remains a morass of mainframe green screens and vast stores of paper records. As Senator Judd Gregg observed, “the software, which runs the hardware, is a huge problem.”

The problem with that software, known as VCF (Virtual Case File), is that it isn’t in production and may never be. VCF may be one of the most extreme examples of requirements bloat in IT history. What began as a fairly modest software project swelled into an all-encompassing replacement for a panoply of woefully outmoded applications and procedures. Along the way, the FBI went through five different CIOs, 10 project managers, and 36 contract changes. The result, said Senator Patrick Leahy at February’s Senate Appropriations Committee hearings, “has been a kind of train wreck in slow motion.”

rcpjr on October 23, 2013 at 3:25 PM

Now they are saying that 20% of the 500M lines of code needs to be changed. That’s 100,000,000 lines of code that needs to be changed. At least 5 years delay, unless they steal someone else’s code like they did before

ConservativePartyNow on October 23, 2013 at 3:26 PM

If 50 million lines of code should suffice…you have to ask yourself… why are there 500 million lines of code… and what was the intended purpose of the addition 450 million lines?

thatsafactjack on October 23, 2013 at 3:09 PM

Tsar of Earth on October 23, 2013 at 3:20 PM

I’d bet there’s a lot of useless spaghetti code that really does nothing at all – produced by someone who was being paid by the number of LOC they produced.
This IS Dem gubmint contracting we’re talking about, and they started with a company with a poor performance record.

dentarthurdent on October 23, 2013 at 3:26 PM

Have you been to a DMV office lately?
They don’t give a dam about how long YOU sit on hold….

dentarthurdent on October 23, 2013 at 2:59 PM

My most recent trip to the local DMV was so pleasent, first stand in line to get to the “Information Desk” for 25-30 minutes then the nice lady checks you into the system and makes sure you have all the needed forms and then she gives you a number and send you into the “Waiting Area” where you sit or stand cause there never is enough seats for another 60-75 minutes waiting for your number to be called. And if your DMV is anything like mine, they have some Orwellian numbering system that has letters and numbers, so when they call for example A256 and your holding A257 you would assume (yeah I know) that you would be called next, ha ha ha, the next number called is C842. Yeah, they don’t care how long you have to sit and wait as evidenced by the clusterfark going on with the current web based sign ups. The call centers will be just peachy.

D-fusit on October 23, 2013 at 3:26 PM

as a software developer who works on software requiring government compliance, I can tell you it’s not just about fixing “buggy” code. i’m betting there is a lot of bad interpretations of the law that went into that code, and that is not fixed easily. we may be talking about months just to determine what the code should be doing – before they even begin to rewrite it!

jetch on October 23, 2013 at 3:26 PM

More B.S. A replacement system cannot be built and tested over a 6-12 period. This thing is years out unless they decide to take a real flyer and risk privacy infringements and leaving actual coverage- as opposed to presumed coverage- to the whims of chance to be discovered only when it is needed and doesn’t exist.

Mason on October 23, 2013 at 3:27 PM

You see it all the time as a programmer – trying to fix someone else’s busted code, only to throw out the whole thing in disgust and start over.

It’s a pretty sad day when you see an inefficient multi-hundred line code reduced to a couple of dozen lines… or you throw out the old bubble sort that was being used on some gigantic database for a sorting algorithm that reduces clock-cycles and data hits significantly.

Yes, trash the code and start over – or better yet, flush the whole d@mned thing and call it a dumb idea.

Turtle317 on October 23, 2013 at 3:27 PM

I love how a failed big government program is an even BIGGER big government program.

melle1228 on October 23, 2013 at 2:51 PM

Unfortunately, this to far too often the case….as are major IT initiative disasters in local, state, and federal government projects. The politicians / managers never pay a price. The contractors who were supposed to implement the project do, at times, pay a bit of a price (usually having to return some of the monies they collected for building the clusterfark). But the one’s who ALWAYS pay a price are the taxpayers – who are always basically held in contempt by both the politicians and contractors.

The real challenge with the approach of nuking the site from orbit is that the clusterfark that is Healthcare.gov isn’t the root of the problem. Nor is it CGI or the other contractors who built this clusterfark.

The root cause is the fundamental design of Obamacare and combinations of 380,000+ words in the legislation, the 11,800,000+ words in the regulations, the clusterfark foundation of the government IT, and the incompetent bureaucrats and political appointees that fill the federal government.

Athos on October 23, 2013 at 3:30 PM

$300 million? I think it’s closer to $600 million.

Pretty soon, we’ll be talking about real money!

LASue on October 23, 2013 at 3:15 PM

It’s never real money. Except ,of course, for the scammers who oversaw and sold us this piece of crap. They get real money and they buy real cars, drugs and houses with it. Luckily, the cost to us serfs is never real money. It comes from the gov money tree.

BoxHead1 on October 23, 2013 at 3:30 PM

tneloms on October 23, 2013 at 2:53 PM

Lines of code are only meaningless if you are in a well run, well documented environment in which there are few personnel changes over the course of the project. Modular software code that has good objects in it make code lines meaningless: modular pieces interact in given ways and the individual modules cohere to a common framework of well documented function calls and data structures.

About… oh… 10 months ago at this point, the people trying to put Obamacare together were just then starting to think about data structures. In other words two years in to the process. There was plenty of code generated by then, but without the data structures coming first, the idea of getting anything useful from the code is problematical. Without the structures first and having to be added in second, all the code that was generated needs to be modified because of all the new function calls that need to be put into the structure.

THEN the number of lines of code becomes a massive problem as you now have to shoehorn in all the stuff that should have been there in the first place. Which means that prior modules (if you are working with modular code or object oriented code) now have to be edited or re-coded, debugged at the module level and then at each and every succeeding level to ensure that it now conforms to the new set of data structures and function calls.

Modular or object oriented code bases are relatively easy to deal with, but the idea that you can drop in entire sections of data structures or remove existing functions due to last minute changes via an RFC process is just nuts. If you’ve had to develop a simple software project on your own, just to do a few simple things, then when you are half-way through the coding and realize that you forgot some vital commands and components, the process of going through all the prior work becomes a nightmare. If you have never even programmed a simple bit in BASIC then you cannot easily touch base with the problems of a large and poorly managed code base versus one that is tightly managed and well documented. Once you step into the realm of C++, LISP, FP or APL, then documented code is a MUST. And that internal documentation doesn’t even count against the lines of code as it is simple internal documentation not read by the compiler.

Just looking in from the outside and hearing what was asked for and when it was asked for will tell anyone who knows a thing about program management of software projects that Obamacare was not well run, not well thought out, poorly managed and would then have all the problems of every project of that class of projects just in the program manager’s realm. Telling people to ‘fix the code’ doesn’t work so well on a project that isn’t well run… as the ‘fixes’ will tend to break other parts of the code. Then that 500 million level becomes daunting and untouchable.

ajacksonian on October 23, 2013 at 3:30 PM

500 million lines of code? That doesn’t seem plausible to me, unless they are counting libraries and such.

neuquenguy on October 23, 2013 at 3:30 PM

Who blowed up the greatest healthcare system in the World?

Obama did. Uh huh! Yes he can, and yes he did.

Key West Reader on October 23, 2013 at 3:31 PM

Luckily, Lucky for the LIV.

BoxHead1 on October 23, 2013 at 3:32 PM

500 million lines of code? That doesn’t seem plausible to me, unless they are counting libraries and such.

neuquenguy on October 23, 2013 at 3:30 PM

I’m no IT person but it sounds like they tried to map data with several competing legacy systems and then mixed in some Obamacodes and then blowed it all up with some choom and pizza pies.

Key West Reader on October 23, 2013 at 3:32 PM

Speaking as a Software Engineer, and knowing how government software projects work, I’m pretty sure the contractor’s productivity was measured by LOC. Which is the absolutely worst way to run a project.

kcewa on October 23, 2013 at 3:24 PM

Bingo!
And I’ll also bet a lot of what they’re counting is just comment code.

I once worked on a small DoD system built by SAIC in C. Their lead programmer was a PhD who wanted to show how much he could do with the C language. When we took it over for maintenance, we found pointers to pointers to pointers, and all kinds of other crap that made it darn near unmaintainable. My company spent the next 2 years cleaning it up and streamlining the code to make it work better and be far easier for a new programmer to jump and figure it out.

dentarthurdent on October 23, 2013 at 3:32 PM

I talked to my friendly IT guy this morning. When I mentioned that reports are that the base code for the system is COBOL 2002, he went off sort of like a Trident from an Ohio-class boomer.

According to him, COBOL is an obsolete programming language that is next to impossible to interface with any other; most systems today use C-Plus, which is among other things the base code language of Windows. BASIC and BASIC II are other common languages that are older and a bit clunkier, but at least they work with C-Plus; and no, they won’t work with COBOL in any of its iterations.

COBOL 2002 was a last-ditch effort to make it competitive with later, more developed languages. And it failed.

He also stated that COBOL is very prone to “spaghetti coding” which is one of the major problems cited with the system. And that the only way it even might work properly with commercial systems (like those used by insurance companies) is by the use of “translation programs” added on; he said it’s much like Google Translate, but for computer languages rather than telling you what a Russian imprecation like “Chort Vozmi!” means in English. (“Devil take it!” if you’re interested.)

Trouble is, COBOL doesn’t “like” such programs and tends to lock up when you try to use them with it.

I asked him if the Obamatrons’ preference for Macs over IBM-type PCs might have something to do with this decision. He said yes, quite possibly, as Macs tend to at least work with COBOL, unlike later IBM systems. In short, they may have “trial-run” a COBOL-based system on networked Macs (maybe at the White House?) without realizing that they were using base code that 99% of modern networked computers can’t work with.

If their intent was to use COBOL because it’s public domain to avoid the “taint” of paying a private supplier for the use of a commercial system, they were just being ignorant and/or stupid a-holes, in his opinion. C-plus is also PD.

His solution? They can keep trying to “patch” the system, which requires going through every single line of code (all 500 million) to debug it. Or, they can scrap it, and hire an actual IT firm that handles commercial systems (like amazon.com, for instance) to do a complete new system “build”. The procedure being, give the pros a list of what they want the system to do, and then leave them alone until they call back to tell the “managers” that it’s ready to roll.

If they choose the former course, things will still be totally FUBARed until some time in 2016, by his estimate. If they choose the latter course, the pros can probably have the new system up and running by Christmas.

He also said that for a December 31 “drop-dead” date, they better make that phone call to the IT firm sometime before noon tomorrow.

My guess is, they’ll go with the in-house FUBAR. After all, it’s the “philosophically pristine” thing to do.

He agreed. Because he’s convinced that they are dogmatic idiots.

I can’t argue with that.

clear ether

eon

eon on October 23, 2013 at 3:35 PM

The root cause is the fundamental design of Obamacare and combinations of 380,000+ words in the legislation, the 11,800,000+ words in the regulations, the clusterfark foundation of the government IT, and the incompetent bureaucrats and political appointees that fill the federal government.

Athos on October 23, 2013 at 3:30 PM

And you have to realize that the first people to read those millions of words were the poor bastards trying to write Functional Requirements. And when they found it wasn’t possible to implement the wording couldn’t be changed without legislation .

kcewa on October 23, 2013 at 3:35 PM

Remember that early code that is developed and accepted must stay in the project. Yet when you later change the requirements for the project you are stuck working on the prior lines of code… or convincing the program manager or the customer to accept new code to replace the old code.

Hope all the function calls got changed in the rest of the structure!

If you want the game equivalent you can look up your favorite game and the things that are hidden in the game… City of Heroes has entire areas that didn’t get developed but were left in due to code structures so that enterprising gamers can find their way to these partially made and forgotten realms that didn’t make it… but were left in since, hey!, it was paid for!

Now imagine that sort of thing with Obamacare.

ajacksonian on October 23, 2013 at 3:37 PM

Anyone want to guess about the state of documentation for this monstrosity?

Mason on October 23, 2013 at 3:38 PM

jetch on October 23, 2013 at 3:26 PM

ajacksonian on October 23, 2013 at 3:30 PM

Something I’ve been saying for a couple days on these threads – it all goes back to requirements definition. And in this case, they had to figure out system requirements based on 10K pages of horribly written regulations. So I’ll bet the requirements phase was either a total CF – or they skipped it entirely to just start cranking out code – figuring they’d eventually mash it all together “magically”.

I remember seeing a good cartoon about that years ago – the manager telling the programmers “you guys start writing code, and I’ll go find out what we’re supposed to be doing”.
Sadly – that could be what actually happened here.

dentarthurdent on October 23, 2013 at 3:41 PM

rcpjr on October 23, 2013 at 3:25 PM

The 2005-2007 US Air Force debacle around the development of the ECSS – a ERP/Logistical system, cost the federal government over $1B when it was finally scrapped.

Back in 2005, the Enterprise Resource Planning (ERP) project began to take shape, with a $628 million contract for Computer Sciences Corp. (CSC) to update the Air Force’s logistics systems with Oracle software. The project was dubbed the Expeditionary Combat Support System (ECSS), and was to be a significant contributor to a Department of Defense goal to be in line with Congressional auditability requirements by 2017. According to the Air Force Times, bickering over the suitability of the Oracle commercial off-the-shelf (COTS) software prevented much work from proceeding until 2007. When work did begin, even with that delay, it did so without a master plan or schedule of deadlines.

Speaking about the decision to cancel the project, Air Force Director of System Integration Brig. Gen. Kathryn Johnson submitted that five major problems contributed to the project’s failure:

Lack of a master schedule
Acquisition strategy changes
Infrastructure changes mid-project that caused delays
CSC
Oracle

When CSC was awarded its contract, it was given responsibility for developing operational requirements. Seven years later, CSC is being criticized for not having the capacity to carry out that portion of its tasks sufficiently, and new procedures within the Department of Defense have been put in place to prevent that responsibility from being placed outside DoD agencies in future projects.

The project was known early on to be in trouble, and in the three years that it was active, it was restructured three times with ‘firm’ deadlines added – and nothing improved.

Another point of note – when the AF listed out the reasons for the project’s doom, it, surprise / surprise, fail to list out it’s own management team and their responsibilities for playing a role.

In 2011, the U.S. Senate called the situation “one of the most egregious examples of mismanagement in recent memory.”

I wonder if Dingy Harry will ever call Healthcare.gov the same?

Athos on October 23, 2013 at 3:41 PM

kcewa on October 23, 2013 at 3:03 PM

Opps, you are right, I meant Saudi Arabia. Sorry about that.

Cindy Munford on October 23, 2013 at 3:43 PM

Flawlessly.

Bmore on October 23, 2013 at 3:43 PM

This whole “500 million lines of code” thing doesn’t make sense. The number of lines of code is not really a meaningful measurement of anything… My point is not that the website isn’t a horrible mess, just that talking about lines of code is mostly meaningless.
tneloms on October 23, 2013 at 2:53 PM

No it means a lot. I used to develop in house for a federal agency. My friends mainly worked in private. One wrote code in the OS of one of the computers I programmed on. You just cant get there – 500 million lines. There are literally not enough routines required, not enough error modules required, not enough of anything to reach 500 million

500 million transactions, sure. Not 500 million ‘verbs’ or ‘calls’
assuming one line per call and assuming you count one line for each byte reserved to define data, each buffer definition, and anything else spewed out at the compiler level

Rumor is Windows 7 has 50 million lines of code. It has to do a lot more functions than Obamacare

Not saying they couldn’t hit 500 million. Either they wanted to suck up maximum dollars, by faking it, or they needed it to not work before the next election. I suspect they needed it to fail

entagor on October 23, 2013 at 3:44 PM

Anyone want to guess about the state of documentation for this monstrosity?

Mason on October 23, 2013 at 3:38 PM

I Like Turtles! Seems apropos.

D-fusit on October 23, 2013 at 3:45 PM

eon on October 23, 2013 at 3:35 PM

Yes – COBOL is pretty much an “ancient” language. My wife was taught COBOL for her Business Info Systems BS almost 30 years ago. COBOL is not object-oriented which most modern languages are – which makes the code more modular and easier for code re-use.
Using COBOL for a large scale web-based system these days is darn near equivalent to trying to put a wood-fired steam engine into a Ferrari to build a race car.

dentarthurdent on October 23, 2013 at 3:47 PM

And you have to realize that the first people to read those millions of words were the poor bastards trying to write Functional Requirements. And when they found it wasn’t possible to implement the wording couldn’t be changed without legislation .

kcewa on October 23, 2013 at 3:35 PM

And most of those millions of words of regulations weren’t available to be read until after November 2012 – as the Administration made the politically expedient decision to delay completing the regulations until after the election.

The flowchart of Obamacare….

Athos on October 23, 2013 at 3:47 PM

I remember seeing a good cartoon about that years ago – the manager telling the programmers “you guys start writing code, and I’ll go find out what we’re supposed to be doing”.
Sadly – that could be what actually happened here.

dentarthurdent on October 23, 2013 at 3:41 PM

There was no triage of expectations at the start. I’ve done the dance getting a decent sized project together for contract work and made sure that all the people who had to be in it were gathered together for a couple of days to figure out what was actually needed.

Day 1 was everyone contributing to the master list of everything they would ever like to have the software do.

Day 2 was the triage. What MUST IT DO? that was first. What SHOULD IT HAVE? was second. What would be NICE FOR IT TO DO? was third. All lists must be approximately the same length and if they aren’t you then spend more time figuring it out since you didn’t do enough on Day 1.

When this doesn’t happen your project is doomed as you will remember things after the contract is let that is vital and then begins the Escher painting with people on the never ending upward stair march that goes nowhere. If you can’t figure out those three sections then you don’t know what you want.

ajacksonian on October 23, 2013 at 3:47 PM


Actual NPR Headline: ‘Despite Glitches, HealthCare.gov Could’ve Been Worse’

Jonah Goldberg:

‘For instance, logging on could have permanently blinded you!’

Resist We Much on October 23, 2013 at 3:47 PM

Not buying the whole ’500 millions lines of code’ claim. Sounds like a journalist made an error, someone utterly doesn’t know what they’re talking about, or else someone is deliberately inflating the number.

Robert_Paulson on October 23, 2013 at 3:49 PM

Things that make you wanna say “Hmmmmmm” (thanks Arsineo).

Windows 7 is guesstimated to be 50-80 million lines of code.

An.entire.operating.system.

VietVet_Dave on October 23, 2013 at 3:50 PM

eon on October 23, 2013 at 3:35 PM

Interesting. I can’t imagine the cost for an IT company to create this thing from scratch by the end of December, but then the cost is already too high.

txhsmom on October 23, 2013 at 3:51 PM

They need a Department of Glitches to fix this

faraway on October 23, 2013 at 3:52 PM

ajacksonian on October 23, 2013 at 3:47 PM

With you 100% on that one.

dentarthurdent on October 23, 2013 at 3:52 PM

It wouldn’t be that hard to get an insane amount of code if you hard coded business logic, failed to reuse existing services and code, made unique calls to connecting systems (instead of adopting a standard), failed to remove “bandaid” code once additional code was developed, etc etc.

Chances are you had multiple groups off in their own world with no one connecting them together to talk and get everyone on the same page. It’d be like building a house without a foreman and every group (plumbers, electricians, framers, concrete, tiling, landscapers, etc) just did what they thought needed to be done without worrying about what anyone else did. You’d have sockets covered by drywall, sewer lines in the closet, fan outlets in the shower, etc.

nextgen_repub on October 23, 2013 at 3:53 PM

*snip*

About… oh… 10 months ago at this point, the people trying to put Obamacare together were just then starting to think about data structures. In other words two years in to the process. There was plenty of code generated by then, but without the data structures coming first, the idea of getting anything useful from the code is problematical. Without the structures first and having to be added in second, all the code that was generated needs to be modified because of all the new function calls that need to be put into the structure.

*snip*

Modular or object oriented code bases are relatively easy to deal with, but the idea that you can drop in entire sections of data structures or remove existing functions due to last minute changes via an RFC process is just nuts.

*snip*

ajacksonian on October 23, 2013 at 3:30 PM

.
These are the best comments I have seen to date regarding the Obamacare system implementation.

The only thing I would add: You sit all your stakeholders down and map out the work processes BEFORE you begin to THINK about data structures.

If you don’t know what you are going to be doing with the data through out the entire system, you are guaranteeing the system will not meet your needs.

PolAgnostic on October 23, 2013 at 3:55 PM

500 millions line of code my @$$. The space shuttle had 500,00 lines of code for on-board and 3.5 million for ground ops.

db on October 23, 2013 at 3:56 PM

After assessing the website, Dave Kennedy, the CEO of information-security company Trusted Sec,

… Nish Bhalla, CEO of information-security firm Security Compass,

I’m sorry but these people don’t have any idea how to write enterprise software.

If these are the types of gurus Obama is using, ObamaCare is doomed.

Enterprise software is exponentially more complex than websites, and other simple database stuff.

faraway on October 23, 2013 at 3:57 PM

ajacksonian on October 23, 2013 at 3:47 PM

.
Didn’t read far enough BEFORE commenting … you’ve got it ALL covered.

PolAgnostic on October 23, 2013 at 3:57 PM

Not buying the whole ’500 millions lines of code’ claim. Sounds like a journalist made an error, someone utterly doesn’t know what they’re talking about, or else someone is deliberately inflating the number.

Robert_Paulson on October 23, 2013 at 3:49 PM

Depends on the software language, the number of functions, the amount of code that was accepted early, the number of changes, the number of pieces of code that had to replace major parts already accepted and yet would tend to leave some functions going to earlier accepted code…

You are thinking of a tightly run, well documented system with firm functions and capabilities known up-front that are well defined.

Obamacare has NONE of those salient features to it. A year should have been spent just trying to figure out what all the major functions were and how they were going to interoperate with State exchanges and with insurance companies. But Congress knew better, you see…

And just how many changes in personnel did the contractors go through once people realized that their previously written code was going to be butchered due to the customer not knowing what they wanted? The entire morale and personnel side of things plays a huge part in this and the code becomes indicative of the problems created by the customer, the program managers and the individual projects within each program area.

Obamacare will go down as one of the greatest examples of poorly run software development to-date. Windows ME has nothing on it. Nor MS Bob. Name the largest prior flop in software development and this will beat it, hands down.

ajacksonian on October 23, 2013 at 3:57 PM

Whoops, left out a zero after the comma. 500,00 should be 500,000 (five hundred thousand).

db on October 23, 2013 at 3:58 PM

#5 was left off the screen grab shot on the post’s photo.

Bmore on October 23, 2013 at 3:59 PM

Didn’t read far enough BEFORE commenting … you’ve got it ALL covered.

PolAgnostic on October 23, 2013 at 3:57 PM

Thats ok!

Been on the inside doing this stuff before my health went south… and was on more process improvement groups (whatever they got called in each iteration) and kept on having to make the same points over and over again for a decade.

It is hard work to run a program well and if you don’t know up front what you want, then the problems can snowball out of all proportion to the program itself. I’ve seen large scale programs that had to have their own error tracking system just to keep track of the bugs… and that system turned out to be larger than the entire deliverable.

ajacksonian on October 23, 2013 at 4:00 PM

The root cause is the fundamental design of Obamacare and combinations of 380,000+ words in the legislation, the 11,800,000+ words in the regulations, the clusterfark foundation of the government IT, and the incompetent bureaucrats and political appointees that fill the federal government.

Athos on October 23, 2013 at 3:30 PM

Yup. Right on.

500 MILLION lines of code (if that’s even true) isn’t going to get reviewed, much less understood, any more than the behemoth of Obamacare itself. (Wasn’t it Conyers who asked “Why bother to read it?”).

This is the impedance mismatch between the nonsensical output of partisan political hacks meeting up against the logical reality of attempting to implement it in code.

There’s a saying: “Plan to write the first one to throw away”. And trying to understand, and fix, someone else’s crappy code is much harder and slower than cleanly writing a new one. Plus, forget about maintenance – except that’s moot, since the plan clearly was to junk this for single payer ASAP.

bofh on October 23, 2013 at 4:00 PM

The GOP should be out with a five- to ten-point plan to improve the healthcare and health insurance industries. NOW. Now that it’s clear Obama lied about the reduction of premiums, the reduction of deficits, and keeping plans and doctors, the GOP has people’s attention. Show how the solutions will solve problems. The one thing Obama never did was explain how all these magical things were going to happen. I’ll bet voters would be saying, “Why the phuck didn’t they do those things in the first place?”

And for crying out loud, why isn’t the GOP reminding Americans that Obama built this?

BuckeyeSam on October 23, 2013 at 2:58 PM

Two issues with that…

1. The GOP doesn’t want to repeal Obamacare. They would like to be the ones running it, of course, but they don’t want it to go away. The GOP is not a party of small government. They are the party of “minutely smaller government than that other party.”

2. The federal government can do nothing to “fix” health care other than getting the hell out of it. Unfortunately, with our current American populace, actually trying to sell “we just want to completely extricate the federal government from health care and let the market do its job” will never sell.

Shump on October 23, 2013 at 4:01 PM

“Projects that are done rapidly usually have a lot of [repetitive] code,” said Arron Kallenberg, a software engineer and tech entrepreneur. “So when you have a problem, instead of debugging something in a single location, you’re tracking it down all through the code base.”

Repetitive code is a sign that the coders didn’t know what they were doing. In any very complex software system, a series of calculations, or a series of logical operations (if true, do this, if false, do something else) that are done repetitively are usually grouped into a sub-program or subroutine, which can be “called” from many locations in the main program, or higher-level subroutine. A “call” to a subroutine passes it input variables, and the subroutine passes output variables back to the calling program.

In such a structure, a “test program” is built around the subroutine to test it under many conditions, and modify the subroutine until it is “robust”, able to function correctly under all conceivable conditions. Then it is incorporated into the main program structure, and programmers then assume it works correctly. If a situation is later discovered where the subroutine doesn’t work correctly, it can be fixed once, not in many locations.

Most programmers, if they see that a section of code is repeated more than three or four times in program, will put it into a subroutine and replace the repeated code by one-line subroutine calls in the main program.

Setting up a structure of calling programs and subroutines takes PLANNING, and extra time and teamwork up front, but usually results in code that is much shorter and much easier to debug in the long term.

But since nobody really started planning the actual implementation of Obamacare until after the 2012 election, they ran out of time, and had to copy code many times, resulting in many copies of code all having the same bugs. Happy bug-hunting season!!!

Steve Z on October 23, 2013 at 4:02 PM

ajacksonian on October 23, 2013 at 3:30 PM
PolAgnostic on October 23, 2013 at 3:55 PM
Steve Z on October 23, 2013 at 4:02 PM

All of this, to me, is clear evidence that they did not do a proper requirements definition, which likely can never really be done just because of how 0care is written.
If the requirements aren’t properly defined up front, the system cannot be properly designed or coded, and there is no way the system will EVER work properly.

It’s like sending 100 people out to a junk yard, each to find one part (unspecified as to what part they are to get) – and then put all of those parts together to make a functional vehicle.
Good luck with that.

dentarthurdent on October 23, 2013 at 4:13 PM

Comment pages: 1 2