The data is pretty bad. How can that be? The answer is clear if you consider the role of the site in the first place. At its core the most fundamental function of the site is to access and manage data from a variety of massive databases, from sources like the IRS, Veterans Administration, insurance carriers, the DHS, and Medicare. We are talking about a lot of incongruent and varied information that has to be extracted, transformed and analyzed, ideally on the fly. Does that sound like anything? To our dedicated readers, that’s a classic Big Data use case. So where has this gone wrong?

Critics are going to point out that no one knows what they’ve implemented as far as data technology goes, but these issues are self evident. It turns out that even if they had the technology, Big Data, analysis, massive concurrency from various sources, at scale, whatever they used – well that’s just one piece of a properly functioning solution.

There’s a process behind it. Creating and imposing the required data structuring onto this variety of sizable databases means a tremendous, effective effort should have taken place. Data from these systems needs to be identified, correlated, and processed in accordance with the target system. It would be nice if that was automated, but the truth is that many, many data points would have had to been manually evaluated, then categorized. …