Don’t get too excited or nervous on the basis of the new Washington Post poll out today. It’s an ambitious effort to expand the depth of polling by conducting 50 state polls rather than a national survey, which theoretically should give a much clearer look at the actual state of the race, but it suffers from two critical structural flaws. The Post’s Chris Cillizza introduces the results in a short but very enthusiastic video:

With nine weeks until Election Day, Donald Trump is within striking distance in the Upper Midwest, but Hillary Clinton’s strength in many battlegrounds and some traditional Republican strongholds gives her a big electoral college advantage, according to a 50-state Washington Post-SurveyMonkey poll.

The survey of all 50 states is the largest sample ever undertaken by The Post, which joined with SurveyMonkey and its online polling resources to produce the results. The state-by-state numbers are based on responses from more than 74,000 registered voters during the period of Aug. 9 to Sept. 1. The individual state samples vary in size from about 550 to more than 5,000, allowing greater opportunities than typical surveys to look at different groups within the population and compare them from state to state.

First of all, the effort and the investment should be applauded. National polls usually involve 800-2000 respondents spread out over 50 states and the District of Columbia rather than representative samples from each state. The normal model works pretty well, but generally speaking, reliability increases as samples get larger. Pollsters stick with that model because of the expense. It costs a lot of money to conduct polls with scientific samples, and doing 50 at a time costs a fortune.

However, two glaring issues become immediately apparent in the explanation of the model above. The first is that Survey Monkey’s universe of potential respondents is self-selected and therefore arguably not entirely representative. The WaPo’s Scott Clement explains:

The Post-SurveyMonkey poll used an online-based sampling methodology that differs from previous polls by The Washington Post. Those are telephone surveys based on random samples of cellular and landline phones.

The new poll was conducted online as part of SurveyMonkey’s 2016 Election Tracking project, which recruits respondents from the large number people who take polls on the company’s do-it-yourself survey platform, roughly three million each day.

We’ve used Survey Monkey in the past, and it’s a fun way to conduct relatively inexpensive polling, as long as one knows exactly what it produces. This is a universe of people who like to take polls, and that does not necessarily mirror the universe of people who like to vote. The decision to use Survey Monkey rather than traditional polling in the broader universe of voters seems antithetical to the concept of getting a broader and deeper level of polling. (Survey Monkey is particularly helpful when conducting site surveys, it should be noted.)

The second major problem is the length of time it took to conduct the poll. The period between August 9 and September 1 is over three weeks, and the presidential race changed significantly during that period of time. Polling reflects a slice of time, but make that slice too thick and it becomes difficult to determine the point in time the results really reflect. Polls that take three and a half weeks to conduct during a constantly changing race end up telling us almost nothing about where the race stands at the time the results get released.

Take the Texas results, for instance, which show Hillary Clinton edging Donald Trump 46/45. For what time period was that true — the second week of August? Third? Fourth? Now? Do the people who like taking online surveys in Texas reflect the general electorate? For what it’s worth, two other polls taken within the WaPo/SM survey period showed Trump up in Texas by six points and eleven points. The most recent of the two showed the six-point lead, and it came from Democratic pollster PPP. The WaPo/SM result looks very much like an outlier, and since they’re highlighting it as a big part of their story, it raises suspicions about the credibility of the entire result.

As for the rest of the results, well … Guy Benson puts it best:

WaPo/SM numbers just don't make sense: Trump ~tied in OH, FL, CO, WI, MI…but in danger of losing AZ, TX, GA, MS? C'mon.

— Guy Benson (@guypbenson) September 6, 2016

The Post had a good idea, but the execution makes the outcome difficult to credit. Perhaps they’ll try again with better methodology.