Romney’s ORCA system was never beta-tested before going live on election day
State officials were kept in the dark about exactly how it would work in the lead-up to election day, and there was never a dry run that included early voting, said one of the sources.
Among other issues, the system was never beta-tested or checked for functionality without going live before Election Day, two sources said. It went live that morning but was never checked for bugs or efficiencies internally. The volunteer at Ace of Spades also cited this issue but as one by which field workers couldn’t get to know the system ahead of Election Day. But inside Romneyland, officials were experiencing similar problems as votes were being cast…
It’s been reported the system crashed at 4 p.m., but multiple sources familiar with the war room operation said it had actually been crashing throughout the day. Officials mostly got information about votes either from public news sources tracking data, like CNN.com, or by calling the counties for information, the source said. Officials insisted the day after the election that they had still believed they were close, and that they had hit their numbers where they needed to, even as Fox News and other outlets called the race.
The numbers in the interface never moved, leaving officials in Boston and out in the states “flying blind” — a phrase used by several people.
Blowback
Note from Hot Air management: This section is for comments from Hot Air's community of registered readers. Please don't assume that Hot Air management agrees with or otherwise endorses any particular comment just because we let it stand. A reminder: Anyone who fails to comply with our terms of use may lose their posting privilege.
Trackbacks/Pings
Trackback URL
Comments
oops!
ted c on November 9, 2012 at 12:14 PM
I didn’t have problems logging into it. The demo vidoes they showed put forth enough that there was some testing to it. More accurate to say it was never stress tested. My problem was losing connection and the roll seemed to be out of date (or was not properly updating so I couldn’t find voters).
Zaggs on November 9, 2012 at 12:15 PM
… And this was the master of business that was supposed to get the US back on its feet? Wow what a joke . In our corporate it environment where I work for example, this is the most basic of project management and implementing new systems.
Realclearblue on November 9, 2012 at 12:15 PM
Orca is something the RNC should take ownership of now and fix it (also make it an actual app). Would be very useful once the bugs are worked out.
Zaggs on November 9, 2012 at 12:16 PM
I said this yesterday, but I was an ORCA volunteer and had no issues. The web “app” never crashed, and the list of voters was consistent with the paper file they sent.
changer1701 on November 9, 2012 at 12:21 PM
Come on, people.
Test-driven development.
Please.
The Schaef on November 9, 2012 at 12:21 PM
Frankly this is ridiculous. How many primaries did they have to test this thing? This sort of thing is something the RNC should have been in control of, not the campaign, and put to use for all republicans. Instead the developers were probably keeping it under lock and key to protect their copyright while dreaming of the cash they would make off it.
Sometimes having a business mindset is not the best thing.
Rocks on November 9, 2012 at 12:21 PM
That’s great but it seems like you were the exception rather than the rule which was not what they were going for.
Rocks on November 9, 2012 at 12:23 PM
It most likely wasn’t thought up until AFTER the primaries. The whole thing seemed rushed.
Zaggs on November 9, 2012 at 12:23 PM
Are these NASA people?
portlandon on November 9, 2012 at 12:25 PM
Having it NOT be a smartphone app may have killed the whole project before it got off the ground.
I can do a “mobile-friendly web app”, no problem. But I have no experience designing native apps for iOS/Android. But here’s the thing: my salary is no better than the national average. And I’m not designing the site for the President of the Fracking United States.
Please tell me, with a billion dollars to spend on the election, you didn’t leave your fate in the hands of a single, moderately-experienced, lower-middle-class web developer.
The Schaef on November 9, 2012 at 12:27 PM
Please, let’s not pretend that ORCA would have made a difference. America lost because we have more greedy takers than producers. That is what we have to overcome, the rest is all a waste of time.
VegasRick on November 9, 2012 at 12:31 PM
At the end of the day, the mathematical models based on state polls nailed this election. To hell with Nate Silver, Drew Linzer’s model predicted a 332-206 result back in June!
Had Romney won, we’d be reading process stories about the genius ORCA get out the vote system, and how overrated Obama’s ground game was.
Instead, we’re reading about Obama’s amazing ground game, despite the fact that many millions of his voters didn’t show up this time.
Based on the lower turnout in this election, you’d have to conclude that neither ground game was really that good…
In the end, these process stories are overrated. Ground games and super Pacs just don’t have that much to do with the final result in Presidential races, IMO.
Chameleon on November 9, 2012 at 12:33 PM
I understand that. I’m just providing another perspective, especially since people somehow believe that this is why we lost. Any Republican out there that needed someone in Boston calling them to get out and vote, after four years of Obama, takes the blame, imo.
changer1701 on November 9, 2012 at 12:41 PM
The entire Republican effort was cut too close. I finally got a card in the mail telling me where I could call to help with GOTV efforts … two days AFTER the election. That mailer should have gone out to be received a week before the election.
Here’s an idea: Next time, lets make sure only Republicans are working on this stuff. I have a suspicion that this clusterfark was by design. Same with delaying the GOTV information to people’s mailboxes.
crosspatch on November 9, 2012 at 12:45 PM
They must be… Muslims feel great about Tuesdays election.
Alberta_Patriot on November 9, 2012 at 12:49 PM
We now have a Republican Congressman in PA-12 thanks to a great GOTV effort. When I voted, there was a local GOP poll striker there, operating the old fashioned way with pencil & paper.
Just sayin’
allanbourdius on November 9, 2012 at 12:50 PM
Washoe County, NV doesn’t allow electronic devices in the polling places (and this may be true throughout NV,) and poll watchers aren’t allowed to sit close enough to the sign-in table to hear the names of the voters, so the app was useless in my polling place in real-time.
We do, however, receive printed lists of the voters from the poll workers every two hours, and these are easy to check-off against the printed list — though one has to cooperate with the Democrats and union thugs that arrive like clockwork when working with the lists. The other complication is the fact that there is a unique list for each sign-in computer, so one has to go over three lists.
After receiving the list, it was a simple process — if a little time consuming — to check those who voted in the past couple of hours against the volunteer’s printed list of all voters in the precinct.
Then it was time to go outside to transmit the data to ORCA, which was where the app, because it was designed to work in a real-time environment where voters might move through the sign-in at one or two per minute, failed, because the volunteer was now trying to input the data for several hundred voters in as short a time as possible.
The app required the volunteer to look up the voter by name (after inputting the name from the printed list,) and once found, move a slider to indicate that he/she had voted. Of course, this meant retrieval from a server and the attendant time lags. One of the volunteers at my precinct took nearly an hour to enter the data on fifty to sixty voters.
I, OTOH, didn’t have a smart phone, so had to transmit a unique numeric code, printed on the voter list, for each voter, followed by the pound sign, eg., 657#4598#34#4623#.
I found that the quickest way to do this was to, when checking the printed voter roll against the list of those who voted, simply copy the codes onto a second piece of paper, in columns. This allowed me to go outside and phone-in the codes at the rate of about ten per minute. IOW, I was able to enter 50 – 60 voters’ data in five or six minutes, a ten to twelve-fold improvement over the app.
Reno_Dave on November 9, 2012 at 12:50 PM
up until a group like anonymous blows the thing up the one day you need it.
bannor on November 9, 2012 at 12:57 PM
Opps.
It could have worked perfectly for you and still be the reason the GOP lost.
alwaysfiredup on November 9, 2012 at 12:58 PM
everyone is picking over the carcass trying to be the smartest persion in the room and point to ‘the one thing’ that doomed the campaign.
deploying a system such as this should be a methodical process. maybe some corners were cut here and there.. at the same time, we can’t take as gospel anecdotal field reports..
gatorboy on November 9, 2012 at 12:59 PM
exactly.. and the campaign may have erred on the side of safety/security by not releasing this to the field in advance to diminish the ability of others to tamper with it..
gatorboy on November 9, 2012 at 1:01 PM
I was signed up for Orca and went through all the training (kind of a joke). This was thrown together with the typical webapp mentality… throw it together and see what happens when you go live. I heard about the app problems, but their implementation was just as sucky… I live in Raleigh and when I got the precinct assignment I got was Elizabeth City, NC — three hours away. o_0
Called the local GOP office and was told “we’re working on it” and never heard another thing, other than a call from the main campaign on election day that the app had crashed and to go to the backup plan. Needless to say, I hadn’t driven to Elizabeth City and took another sip of my beer since I had taken the day off.
What a clusterfark and it left me very discouraged. GOTV should be boots on the ground or a well oiled (software) machine.
SteveInRTP on November 9, 2012 at 1:02 PM
The REAL problem is that RNC forced a “moderate” as the party nomineee.
The nation saw no real difference between Romney and Obama…
,,,and it turns out that many Republicans couldn’t tell the difference, either!!!
So maybe ORCA had trouble making the distinction, too!
landlines on November 9, 2012 at 1:07 PM
I, also, had no problems on my end with ORCA. In fact, I sent out a quick anecdotal blurb on the chat icon to them around 730 am and someone from HQ called me back immediately to check to see if I was having any problems. Too bad it didn’t work like they wanted. I do think it has potential.
1nolibgal on November 9, 2012 at 1:10 PM
My friends, it’s official: Mitt Romney ran a worse campaign than John McCain.
Mr. Wednesday Night on November 9, 2012 at 1:12 PM
That’s a way to look at it, then there is reality. Even in the highest participation election ever something like 40% of eligible voters don’t vote. This election it will be close to 50%. 20% of REGISTERED voters don’t vote at the best of times. There are a lot of people who are busy or need to be motivated to get out. That is reality.
There is a lot of talk about minorities but the fact that never gets mentioned is that if every eligible white voter were to turn out and vote 60% Republican as white voters that do vote then the minority vote could go 100% democrat and the democrat would still lose outside of large minority House districts.
Rocks on November 9, 2012 at 1:13 PM
So…….no cabinet position, ambassadorship, or other plum assignment within the Romney administration for the ORCA developer?
pain train on November 9, 2012 at 1:14 PM
If nothing else, don’t count on him being hired to help them move government websites onto Drupal.
The Schaef on November 9, 2012 at 1:58 PM
I would guess that with his connections he might just get that Drupal port. If it’s not an election, results are not important for those people.
BoxHead1 on November 9, 2012 at 2:32 PM
Many of us here in the east Central Florida red county couldn’t even get to our assigned watch spot because our applications never made it to the county Supervisor of Elections. I even got a call from Boston asking why I wasn’t at my assigned spot. I told them, and they were pretty helpful after that, but it was clear they hadn’t read the 4 emails I’d sent over the weekend saying I had no credentials. When I couldn’t log in to the site the night before, the helpline was saturated with calls despite that it wasn’t live. Those who could get through learned it would be up at 5am. Then at 5am those who could get through learned it would be up at 6am. I got the Boston call at 7:30. They called back 5 minutes later with instructions to go to the local HQ, so I drove 25 minutes and found several other people there with the same problem.
I don’t think this can be directly blamed on WMR, but the person 1-2 below him clearly was over his head.
321mdl on November 9, 2012 at 3:13 PM
I dunno… say what you will about the administration as a whole… their websites actually look nice and clean… and if I were to pick a CMS as a foundation for one (or all) of these sites, Drupal would have been my choice as well. I mean, it’s not like they tried to build them in Mambo or anything…
I don’t claim to be savvy enough to build their whole system – not by a long shot – but in my limited experience, I tend to think of system designs as a). modular and b). in trees.
What this would mean in terms of an app like this is that there would be a model for e.g. Team Romney to look at results from the top down. They don’t need to know every voter in every district and every home address. They just need to know how they’re performing on a by-state and sometimes by-district basis. A proper GOTV effort would have delegates down the line for taking care of the finer details anyway.
Then a state-level model could have specific district and probably precinct information (again, I’m taking wild stabs based on practically zero knowledge of the internal workings of GOTV; these are just examples to demonstrate the point: what information is/is not necessary at each level), and then a district model with specific precinct information including home addresses and what not.
The major benefits to this layered system would be:
1). At the state and especially the district level, you can collect additional data, not only for the presidential race, but for state and local offices, and issues of importance. You would have a broad base of information but it would be unique and isolated to your district, which is preferable anyway since the election models are so different throughout the country. It also lessens the possibility of people being asked to drive halfway across their own state.
2). the load on the app would be distributed across some 500 different locations instead of bombing one central location. Now, given apps like Facebook or Twitter, there’s no reason top-shelf programmers couldn’t write a single program that could handle a bajillion updates at once, but those companies have hundreds of geniuses busting code for like a decade to get to this point, and I’m trying to conceive an idea that could be pulled off in less than two years without trying to convince Larry Page to a). aid Republicans and b). not turn around and sell the same idea to the Dems (assuming they would even want an ORCA system)
3). distributed information not only means finer details at the lowest level, it also means that the information that trickles back up the tree contains less data and requires fewer transactions. Team Romney doesn’t need to know the addresses and phone numbers of people living on Bleaker St., and if he wants to make a personal phone call, he can call down the line to the district office. What he does need is to know how the numbers in Prince William County are looking, without his system crashing in midstream.
Again, I’m talking in broad strokes and platitudes, I know, but the point is that I’m not sure how much of this was taken into consideration at the design level: what kind of pieces you could break the data into, how much would you need at what level, what specifically you would want to do with it. Clearly I’m a big fan of breaking things into pieces, and I think it’s not unthinkable to give another run at a national-level real-time tracker. Just be sure to account for some of these other niggles (incidentally, a district office with its own implementation would also have helped sort out the paper-only vote tracking issue, as an example).
The Schaef on November 9, 2012 at 3:29 PM