Our Polling in IL-10

Obviously our polling on the 10th Congressional District Democratic primary in Illinois was way off, one of our worst misses ever. We apologize to our clients, the Progressive Change Campaign Committee and MoveOn, for letting them down on this race.

PCCC and MoveOn have been transparent with their polling on the race and have posted all the polls we did for them over the last 3 months here. This error is on PPP, not them.

We did see the race tightening significantly over the final week. The tracking poll we did on Sunday, March 11th found Sheyman leading by 21 points, 46-25. The tracking we did on Wednesday, March 14th found Sheyman's lead down to 14 points at 44-30. The numbers that were released publicly last Thursday combined the interviews from Sunday and Wednesday, for the purposes of a larger sample size.

The final night of tracking we did on Sunday, March 18th continued to show Schneider gaining ground while Sheyman stayed in place, and that the lead was down to 11 points at 44-33. Still we never would have expected this.

We have a few theories on what went wrong:

-We called people who had a past history of voting in Democratic primaries, as we do for all primary polls. But I wonder if with the open primary and Schneider's status as the more conservative candidate if some people who usually vote in GOP primaries crossed over to support him. We would have missed those folks.

I think the bigger possibility though and a lesson learned for our work with all of our clients is this: we may have overpolled the district:

-The pool of likely Democratic primary voters in this district is small enough that we had to call everyone for both a message testing poll we did at the end of February and for one of the 3 nights of tracking we did over the last week. Calling people twice in such a short period may cause respondent fatigue.

There's 2 reasons I think this may have benefited Sheyman. The first is that on the message testing poll a much higher percentage of Schneider supporters hung up in the middle of the call than Sheyman ones, presumably because they thought the poll was being too negative to him, even though the poll tested negatives against both candidates and was nothing out of the ordinary for the polls we conduct. I think it's possible that when we called again for the tracking over the last week Schneider voters may have been more reluctant to participate. We actually had a different person record the tracking poll than the message poll so people previously called would hear it with a fresh ear but there may have still been some aspect of that.

The 2nd reason I think respondent fatigue would have helped Sheyman's numbers is that we did continuously find that his supporters were more excited than Schneider's were, and that may have extended to their being more excited to answer polls than Schneider's were too.

One thing I know is that it didn't have anything to do with the way the data was weighted.  Weighting had very little impact on the final results.

For PPP I think the lesson  is that a Congressional primary electorate is too small to call for multiple polls in less than a month, especially if there's some aspect of negative message testing involved. It needs to be a big enough pool of voters that you can call completely different universes every time.

We apologize that our polling skewed perceptions of where this race stood. When you poll as many races as we do you're going to badly screw one up every now and then. But we strive to be as close to perfect as possible and in this instance we fell badly short.

Comments

[Art Kelly]

PPP was very accurate in the Republican presidential primary! Your mea culpa is appreciated, but you should also take bows for your successes.

[Michael Weissman]

Thanks for this very forthright analysis. The bottom line seems to be that primaries are much harder than generals.

[Sophie]

Thank you for the candid thoughts.

A minor note: I think the link (to the detailed poll info) is broken; could you fix it?

[John]

Although you missed the result, I have to give props for owning up to your mistakes. Not many people/companies would do that.

[Jeff L]

Thanks says a lot about your integrity to discuss this so openly.

[dz]

I think it goes beyond this.....people in this district understood that for 32 years a republican has held this seat and that in order to turn it blue, we needed a moderate dem and not a progressive dem...Dold must go and Sheyman was not the candidate to oust him...Dold's campaign feared Schneider and not Sheyman - very telling....

[Fred Andes]

Well, speaking as someone who was initially torn between the candidates, but who was then utterly turned off by the negative messages emanating from Sheyman, I think that the most likely cause of the error is the one that you initially highlighted: Schneider supporters hanging up in the middle of the call, because they were very tired of hearing a real Democrat being called a Republican because he voted in one R primary (for a friend) and for contributing to a few R candidates because they were pro-Israel. The Sheyman people just went way out of bounds, and turned off a lot of solid Democrats, like me, who were seriously considering voting for him. Maybe try a positive message next time.

[Josh Kilroy]

It seems unlikely that GOP voters would opt out of a competitive presidential primary to intervene in a congressional race.

[fladem]

Any reason to suspect cell phone vs. home phones was an issue. People may be less likely to answer a cell phone poll on a second or third attempt.

[Davidsfr]

This is your 1 out of 20 outlier!

[Aaron]

I'm a Republican through and through, but I appreciate PPP's honest and open policy. The comments that you get about being a ridiculous hack for the Democrats are definitely off base. Keep up the good polling work.

[Luke]

No...Sheyman was never in the lead. You simply just polled people to skew polls to make people believe that he was the front-runner. You got a fool made out of yourselves and it serves you right. A very unethically run campaign was beaten by one of integrity-- that's justice being done.

[gentlewind]

I was polled, expressed support for Shayman, and then changed my mind at the last minute. It was a head over heart decision. I felt torn about the race for several weeks and know many others struggled as I did. Ultimately, we all felt that Schneider has a better chance of winning in the 10th and many switched our allegiance within a few hours of voting. I last minute support for Schneider by voters like me could explain some of the polling issue.

[Virginia]

In MA, I have three major observations over the past five years in working for Progressive Candidates for primaries. First, voters get excited when canvassed in person. Very few people answer their phones these days because they are message-deaf during elections. Second, voters still need to be taken to vote (huge GOTV activism). New citizens, college folks, older folks, and people who may be excited but timid about the voting process need to be car-pooled when the race is tight. Third, incumbent Democrats who are "in name only" really shape up for their constituency if they are seriously challenged by a progressive Dem. Thank you Ilya. Great job, everybody. We will follow you in the months ahead.

[Seans1965]

GOP out performed Democrats 3 to 1 in an already dismal 23.3% turnout in Lake County. If you look at the county map, the Congressional Candidates (all of them)spent a great amount of time in the east part of the county (Old 10th District). Turnout there was higher than the county average. The candidates did not truly work the new parts of the district (they had surrogates)and in many parts of the new district, turnout was far less than the county average. It takes a candiate to inspire voters. I hope there is a lesson learned here.

[David Hathaway]

Voting should be changed from weekdays to a three day event Saturday through Monday. The voting times do not match most people's schedules. Anyone working under a "point system" is being unjustly held from voting because it is barring their right to vote by effectively creating a "poll tax" A person is charged in one way or another, for exercising their right to vote. One could argue there is ample time, but for many families, that "ample time" falls at times when they have family duties. Ignoring those can also be seen as a "poll tax"

[jamel levin]

A fine progressive candidate - BUT very unsuited to the district and one who was hurt by unnecessarily negative campaign.

[mj]

I was calling to encourage voters to get out. Many are staying home this year. People did not respect Schneider's Republican votes, nor his donations. Even Conservative Dems who stated they were going to vote Repub this year (like Schneider had in the past) preferred Ilya's clearer positions to Schneider's Republican Lite. I think Ilya suffered from a lack of support at the party level - The Democratic party still wants to be Repub lite, rather than be a party of real change. And so, voters will stay at home, in droves, like 2010, and the Repubs in Illinois will clean up.
Many are discouraged - angry at the Tea party after 2010, angry at the Dems lack of leadership from 2008-2010, bailing out banks rather than people, etc.

[Rhysdux]

Speaking for myself, I hate being called on the phone by people pushing candidates. I never know if the person calling is really a campaigner or a nutjob stalker who's trying to garner information by pretending a campaigner. Add in the fear of stalkers and con artists with the fact that most calls about politics happen at night and campaign calls become REALLY creepy. I tend to be supportive of candidates just so that I can get off the phone.

And when campaigners call two or three nights in a row...that's a guarantee that I WON'T vote for the candidate. Hard sells don't work on me. I don't like being pestered.

[John Salch]

Possibly part of the problem might have been the vastly more interesting Republican Primary. If an independent or a Democrat voted Republican just to vote against Romney, etc, he could not switch to the Democrat ballot to vote for
Ilya. Also since many Republicans were turned off by War on Women, they may have picked the more acceptable (to Republicans) Democrat candidate.

[Maria-Viktoria Abricka]

I was a Sheyman supporter and actually did some volunteer work. And yes, I voted for Sheyman. My perceptions:way too much polling and calling. Sheyman's campaign started earlier and that was to the good, but the unintended consequence was that there were way too many calls ... sometimes two to three a night, day after day. I heard many people complain about the phone; I took to hanging up on every robocall regardless of the candidate. Eventually, I think I was hanging up on real people, too.

[jane]

Did you check the voting machines?

[Divine Primate]

Lots of great observations here! Polling is just not effective in a primary like this. My own view agrees that Schneider benefited from support of local Democrat establishment that seeks democrats just a bit to the left of Kirk. Schneider's attacks on Dold as Tea Party and his aggressive raising of the Progressive banner from the start, were excellent, excellent strategic moves, with his gentle hammering on Sheyman's "outside supporters," and too young, too progressive, can't beat Dold themes. These kept Sheyman on the defensive and were probably even more effective after Schneider started running negative Internet ads (Sheyman: "Bogus"), especially those quoting Rep. Mays. These sent a chilling effect and were more personal and powerful an image than the faceless grotesque mailers making a Federal case out of something everyone already knew about.

The "outside" attack ads on Schneider worked at first, but were pounded too hard and probably begun too early. Way, Way too negative for this district, especially in a primary race!!! They backfired and gave Schneider the opportunity (and plenty of time) to claim victimhood and sneak in his own attack ads with impunity.

In all, this is a very sophisticated, open-conflict-adverse district. Lots of older democrats who think/hope/project that Kirk/Dold is a moderate, but few progressives with clout. The intra-party bickering turned people off who really are more repulsed by disrespecting a successful middle-aged businessmen than they are drawn to demanding a progressive vision for America.

[Kman]

Luke is clearly a nut job. PPP I respect you more for admitting your mistake.

[eb]

There has been so much press on the republican candidates perhaps it got lost in the circus. ALso, many are disillusioned and wonder what it all means??? This was disappointing. Also, in metropolitan areas like Chicago , transportation can play a factor. This was the case in the 2008 election. You all did a great job. Thanks for keeping on!

[TrueBlue]

Elections in the USA are "rigged". It happens in both parties. Nobody voted for Joey Libermann in Connecticut but he WON, and the media didnt say boo. Same thing with Rahm Emanuel in Chicago, the guy had NO constituency but he....WON?
George W Bush????

Or how about Nobody Sanotorum? Who's voting for him?? Romney??? The guy is spending MILLIONS to tie....Sanotorum????

Ron Paul is the only guy with real support and he hasnt won a single race??? Yeah right!!

I am sorry for Sheymans loss. We certainly need more LEFT leaning progressives if we are to save America.

[Midwesterners]

I think Divine Primate's analysis above is excellent--and hits just about every nail on the head that could be nailed! First of all, I too campaigned/canvassed for Ilya Sheyman in several precincts in the 10th, including in the newer portions of the district that were formerly in Jan Schakowsky's 9th District. Just about everyone I talked to was very tired of the continuous phone calls and door knocks from the Sheyman campaign--that is, unless they were a "Raging Progressive" or "Unabashed Liberal" like myself of which sadly few such folks exist. It wasn't just that they began to see the campaign as being too negative, but they seemed to become irritated from feeling overly "badgered." I desperately hope we don't make this same type of "mistake" with the Wisconsin Recall because (at least for me) almost all the Tea in China is resting on this upcoming election to remove the Koch Addict puppets from FitzWalkerStan.

Also, and even more sadly, most people I talked to in the 10th District did not seem to mind that a Democrat would contribute to what THEY believed was a "moderate" Republican. It really did not do any good to try to explain to these people that there is no such thing as a "moderate" Republican any more. Tragically, most people still believe that such an animal exists. I honestly do not know how to alter their perceptions because they unknowingly have bought into the Republican narrative --and do not realize that the Right Wing and Corporations are in complete control of the Republican Party. And ironically, if you add to this what was said just above here, Schneider was able to cleverly distinguish himself from the Republican Party by suggesting that since he was not a Tea Party candidate, then he was a "Progressive."

The new 10th District may be somewhat more Democratic than it previously was, but I think this analysis belies the fact that it still contains many wealthy constituents, as well as voters in general, who are "adverse" to conflict --as "Devine Primate" so succinctly described above. If we Progressives do not realize this both here and elsewhere, I am afraid that we will continue to delude ourselves in other races as well. Indeed, most people I met probably rarely thought about politics on a day to day basis--as hard as this may seem for a left-wing political junkie like myself to comprehend. Thus, I am not surprised that Schneider won--partially because many folks saw him as a father with roots in the community as opposed to a young man in his twenties with little experience even as a member of a school board or park district. And, if any of the voters were politically-minded at all, I think many of these folks believed that Schneider did have the better chance to beat Dold in November and finally turn the district over to the Democrats--plain and simple. (So yes, they would have definitely bought the argument that Schneider was the more "electable" candidate.)

Again, I hope we don't turn off voters in the Wisconsin Recall--despite everything that Scotty Walker himself has done! If Walker-the-Koch-Addict-Puppet wins this, I don't want to be any where near Planet Earth for years to come. I also think that there are very similar parallels between the Illinois 10th District and to what is going on in the Massachusetts' Senate Race. Somehow, even in Massachusetts, it seems as if too many voters STILL believe that Scott Brown in a "moderate"--just like the voters in the 10th saw Mark Kirk (and now Bob Dold) as being moderate. Obviously for Brown, it helps that he rides around in a pick-up truck and once posed as a Cosmo Centerfold Pin-up to get voters to believe anything he says. But, it is horrible to think that Elizabeth Warren must fight this myth at the same time that she must address the "real" issues affecting voters while running for the Senate!

[Dan Van Riper]

Oh well gee, maybe the secret software in the voting machines controlled by one of three corporations may have something to do with the results. You think? Whenever results are wildly off from the polls we have to suspect the "proprietary software" that no voting inspector or anyone else is allowed to examine.

Yes yes I know, mentioning this horrible fact at the heart of our corrupted voting process is politically incorrect. It's taboo in the corporate media and even groups like Move On and Progressive Change are too scared to even mention this deadly issue.

Too bad the person who wrote this article is also not allowed to mention the obvious.

[Judy Meek]

So how much did all of this polling cost? Polling is very expensive. It seems to me much of the money could have been better spent communicating with voters rather than reassuring the campaign and the activists they were going to win. Elections, not polls, determine the victor.

[Brainburst]

" try to explain to these people that there is no such thing as a "moderate" Republican any more. "

Therein lies the problem. You can't try to talk people out of something that they want to believe. And you can't get votes from independents republicans by bashing the republican party. If Progressives want to win, they have to redefine what progress is. They need to take a long view and make slow steady progress. For too long Progressives in this country look at elections as a way to make change happen. This is fundamentally wrong. An election must be a capstone to change that has occurred in the electorate. The Progressive movement needs to win the hearts and minds of more of the American people before they try to elect change.

[Brainburst]

Whenever results are wildly off from the polls we have to suspect the "proprietary software"
Not necessarily true unless you are referring to exit polling.

[Larry Barton]

Welcome to quantum politics! The perceptions (of a reality) based on polls are highly problematic and subject to unaccounted for influences, eg, crossover voting, that actually determine the final recognized reality. This is coming from a poll worker and some one active in the local community (Cape Coral, FL) where only 11% of the voters came out for the primary and 16% for the regular election. Is this voter apathy or what? In such an environment, all bets are off unless there are know and solid commitments of target voters.

This also points out that it is the effective "organization" of the voting interests as opposed to that of the polling team that is the more determining and dependable factor. This also assumes that the voting was clean.

[Gerald]

I am so impressed with your candor and integrity. You should be proud of how you run your organization.

This year you have polled a lot of primaries and special elections, all very difficult tasks. You have had incredible successes, such as the specials in NY-9 and in the Iowa Senate and the primary in South Carolina, to name just a few. You have had a couple of broad misses, IL-10 being one. It might be prudent to lay the successes next to the misses and closely compare what PPP did in each. It might reveal a pattern which you might not detect if you simply dissect IL-10 alone.

[Taylor Mills]

Luke - How would PPP benefit by skewing poll results? The goal of any reliable pollster is to predict outcomes accurately. Being wrong can only hurt their reputation, hence the thorough attempt to explain the results. I am a republican but still review PPP polls at least once a week because I feel that they are very accurate 99% of the time. Nobody can always be right, and clearly, you don't understand how polling works. If you think you can do a better job, then perhaps you should try!

[Donna]

"Luke - How would PPP benefit by skewing poll results? The goal of any reliable pollster is to predict outcomes accurately. Being wrong can only hurt their reputation, hence the thorough attempt to explain the results."

Don't be naive. Answer: we do not know how, specifically, but there are certainly many possibilities. Organizations and corporations, including "non-profits" can and often do have many different goals, motivations and agendas that benefit their funders/sponsors/patrons in various ways. These will not be publicly disclosed, for obvious reasons. Just use Washington and our State Legislatures as a model for envisioning and understanding how this might happen.

There are often key people within these organizations who can exert a controlling influence over what they do and how they do it, and would be the only ones who would necessarily know why they do it. It might be helpful to keep in mind that, at last count, over 70% of intelligence agency employees were outsourced, corporate contractors. Who are they working for, do any of us really know??

As for hurting their reputation, we will just have to see about that. Based on the above comments, many are willing to give them a pass on this, or even shower them with praise for their 'candor', or, more accurately, their excuses and mea culpas offered ex post facto. If I were "Bold Progressives", I would not use them again -- at least until one of their "1%" poll errors occurred in a poll(s)of Republican candidate(s). Only time will tell if PPP's errors are truly random errors. Something to watch for in the future!