The Problem With Polls

[0] From the New York Times, I'm Michael Barbaro.

[1] This is the Daily.

[2] Today, two years ago, news organizations, including the Times, were blamed for misleading the country about the projected outcome of a major election.

[3] What we're doing differently this time, it's Friday, November 2nd.

[4] Right now, we have the presidency at about 89 % for Hillary Clinton, 11 % of.

[5] for Donald Trump, obviously.

[6] A lot of people have no idea that Trump is headed for a historic defeat.

[7] If I can ask you to travel back to that night two years ago, why were people so angry about the polls leading up to and on election night, 2016?

[8] We were both in the newsroom together that night.

[9] We were.

[10] I think there were two reasons.

[11] The first is that I think a lot of people entered the election very confident that Hillary Clinton was going to win.

[12] In my predictions, but I think that she's going going to have a very good night.

[13] So the technical term for that would be blowout.

[14] Lanslap.

[15] I give a lands.

[16] I don't know.

[17] You call that a landslide.

[18] I would say a landslide.

[19] The polls showed Hillary Clinton comfortably ahead in enough states to win 270 electoral votes.

[20] And they showed her ahead and even more states that would get her well over 300 electoral votes.

[21] Nate Cohn covers polling and elections for the times.

[22] In some states, there were no polls that showed Hillary Clinton trailing.

[23] Donald Trump.

[24] The electoral vote for Anderson this night is turning out to be a real nail bite.

[25] Florida's been going back and forth, back and forth.

[26] I mean, she is in profound trouble right now in the electoral college.

[27] I think the second reason was that there was a communication problem of sorts in using pre -election polls to forecast results.

[28] Between who?

[29] Between the people who are most familiar with the data and basically everybody else.

[30] Donald Trump will carry the state of Florida.

[31] Right.

[32] I think we all remember the needle.

[33] Yeah, which started at about an 85 % chance for Clinton to win.

[34] And then when the results came in, it became clear pretty quickly that the election was going to be very close.

[35] ABC and NBC project here that North Carolina has gone to Donald Trump.

[36] Again, the map updating behind you, ABC and NBC, those two networks projecting here that Donald Trump has won the state of North Carolina.

[37] And by 11 o 'clock, it had Trump at a greater than 95 % chance to win.

[38] The waiting will aggravate you.

[39] The calling it early and making a mistake will bury you.

[40] And here we go.

[41] This is a Fox News election alert.

[42] Pennsylvania goes to Donald Trump, and there's the big board, the 20 electoral votes from the state of Pennsylvania.

[43] We just got new metrics in.

[44] Donald J. Trump is the president of the United States elect.

[45] Donald J. Trump, the son of a queen's millionaire, defied all conventional wisdom and all expectations.

[46] I think after the election, it struck us that many people were surprised the polls could be wrong.

[47] If you're someone like me, you're not surprised when the polls are wrong.

[48] You expect polling can be wrong all the time.

[49] From my point of view, it's a miracle that polling works.

[50] There are probabilities.

[51] They are meant to communicate uncertainty in the range of possibilities.

[52] I mean, you can focus on Clinton having 85 % chance to win, which we certainly did.

[53] But you can also take, I hope, serious note of Trump having a 15 % chance to win, which is very substantial.

[54] I mean, that's like if you're down by one touchdown at the beginning of the fourth quarter, you have a 15 % chance to win.

[55] You certainly wouldn't turn off the game and conclude it was over or something.

[56] You know, if there's a 15 % chance that there's a hurricane tomorrow, you're under a hurricane warning.

[57] Ah, fair point.

[58] We didn't put up Trump warning.

[59] And for people like me, it's pretty obvious that almost all the elections that we cover that the candidate who's trailing can go on to win.

[60] And I don't think people had that message.

[61] And look, it's, I actually feel pretty good.

[62] about the numbers that we reported.

[63] But I think that we're all willing to concede that from a communication standpoint that it wasn't very effective.

[64] So with the lessons of that in mind, how are you as somebody really steeped in polling here at the times using polling data in this election, the midterms?

[65] Traditionally, the results of a poll are reported at its conclusion after all of the interviews have been conducted.

[66] And then it's released in a single news article that says Clinton's a head by two.

[67] we felt like we needed to demystify polling a little bit for people.

[68] So this time, we are doing live polling.

[69] And what does a live poll look like?

[70] So at the top of the page, we have a stream of dots that represent each of the people who we attempt to call.

[71] And most of the time, in some cases more than 99 % of the time, those dots are empty because people didn't respond to our call.

[72] And then on occasion, we're lucky enough to get some blue, red or gray dots, which indicate whether they voted for a Democrat, Republican, or they're undecided.

[73] And as each colored dot comes in, our result changes.

[74] The polls go up to 500 respondents in general.

[75] So by the end, you have a perfectly reasonable final poll result that's completely indistinguishable from a standard poll of a similar size that wouldn't have been reported in real time.

[76] But you got to see everything before that, rather than at the conclusion of the survey, once all of the interviews have been conducted.

[77] And what is the value of live polling versus the traditional polling in which you don't see that kind of process unfold, call by call?

[78] You just get a result at the end.

[79] I think that the most important lesson that I would have people take out of it that can't be accomplished any other way is to experience the change in the result over time.

[80] That's basically the margin of error playing out.

[81] And you think that by showing how dynamic polling is, live polling helps solve that communication problem that we had in 2016?

[82] I think it's one of many different ways of potentially solving it.

[83] From the standpoint of the public, a poll is a single, final, and fairly precise figure.

[84] Right.

[85] And I think the way it gets reported often adds to that.

[86] We're doing the opposite of that in a lot of ways.

[87] We are trying to be extremely transparent in a way that highlights the fuzziness of polling and show the extent that it's actually quite inexact.

[88] So that the public doesn't misunderstand that final figure.

[89] That's the hope.

[90] Julia Child presents the chicken sisters.

[91] What I see is that people are sort of treating it like a cooking show or something or being in the kitchen when a chef is cooking.

[92] We're roasting Miss Chicken.

[93] today on the French chef and you know you sort of know not to eat the food that is raw that is raw because people can just see that it's underway they know it's not done there's something very intuitive about that idea and you know I think the hope is sort of in part that sort of in the same way that there are a bunch of people who maybe appreciate cooking a little more after watching a cooking show or being in the kitchen that maybe some of that same feeling rubs off on polling which has a pretty bad rap in the last couple years So people are looking at it as what it is, which is a process with steps rather than a verdict.

[94] And then that's a good thing.

[95] You get to watch it come in and you can absolutely eat it at the end.

[96] But I think that if you're really smart, the biggest lesson is actually that your food is never completely done.

[97] It's never as precise as just looking at the final top line number would make it seem.

[98] Everything you just said is a reminder that polling is limited.

[99] It is in exact.

[100] it is kind of flawed, and that we shouldn't put too much stock into it.

[101] So can you quickly remind us why we should care it all about the result of a poll, even a really good, transparent live poll?

[102] I think that we live in a really diverse country, and it is also one that is increasingly divided, demographically, and polarized along a variety of geographic lines.

[103] And what that means is that very few people, I think, have a good understanding of public opinion simply by talking the people that are around them.

[104] In fact, often it's a very homogenous view.

[105] I mean, if you were a reporter and you wanted to report on Ohio, I mean, imagine all the little enclaves of the state that you would have to go into before you even begun to piece together, farm country in the northwest and old coal towns in Appalachia and majority black parts of Cleveland, white -collar and blue -collar suburbs around Cleveland.

[106] Piecing that picture together from the standpoint of an individual person is extraordinarily difficult.

[107] And polling is one of, the few.

[108] Actually, I think polling is basically the only way that we can broaden our perspective on what the public thinks beyond the very small group of people that we get to talk to individually.

[109] How did polling get this so wrong, Eric?

[110] So how did the polls lead us astray, including our CBS News poll, which is considered to be one of the best in the industry?

[111] What went wrong with the numbers, the predictions that polls that suggested a late surge for Clinton?

[112] This is a very I mean, we're all sitting here sort of silence in shock.

[113] We'll be right back.

[114] All right.

[115] So with all that in mind, and with four days to go until the midterms need, what's the story that this real -time polling that you're doing is telling?

[116] I think it shows that there are a lot of really, really close races.

[117] We have pulled dozens of house races over the last couple of weeks, including nearly all of the races that are considered toss -ups.

[118] And we have a majority of.

[119] those races right now within one point.

[120] Wow.

[121] And how many of these races are you talking about that are basically toss -ups?

[122] There are about 30 districts in this category.

[123] In most elections, we go into election day and we look at the districts or states that we believe we'll decide control, and we have a good picture of who we believe is ahead in those districts or states.

[124] This time, we do not have a clear picture about which candidate leads in the districts that will decide to control.

[125] So a couple of weeks ago, we talked to our colleague Alex Burns about the state of things, especially in the House.

[126] And he said that there were about 60 Republican held seats in the House that were competitive, that were in play.

[127] Either side could win them.

[128] And that the Democrats needed to win about 23 of those to take back control of the House.

[129] How does this latest polling of having around 30 toss -up seats fit into that?

[130] So of the 60 most vulnerable Republican -held seats, there are about 16 or 17, where the Democrats appear to have a comfortable and clear lead, that gets them a lot of the way to the 23.

[131] Right.

[132] But after those 17 or 16, there are these 30 toss -up races that look very, very close.

[133] The Democrats don't need to win all of them.

[134] They actually only need to win about a third of them in order to take the chamber.

[135] But they haven't put any of them away yet.

[136] It's not even evident that they have a clear lead in many of them.

[137] That's just way too close to call in all these races.

[138] Yeah, I think that's right.

[139] I think that if you wanted to be an optimist for the Democrats, you could say, oh, just as a matter of probability.

[140] I mean, they have so many options.

[141] How can't they win six or seven of them?

[142] But you can't go down the list and say that they have claimed a commanding advantage in six or seven of them.

[143] So tell us about these 30 toss -up districts.

[144] They're an interesting mix.

[145] I mean, there are really two types.

[146] One, are the well -educated and diverse suburbs where Clinton did well and where the Republicans have a incumbent who's not run before in this kind of political environment.

[147] And then there's another group that overall is larger, and that's predominantly white, middle -to -working -class areas of the president won narrowly, or in some cases by a comfortable margin, where the Democrats have a strong nominee, where the Republican incumbent has never been particularly.

[148] strong themselves, and often where there's like a little bit of residual democratic heritage.

[149] Like maybe Obama won the district, or maybe Bill Clinton would have won the district.

[150] But they've trended towards the Republicans over the last decade, and particularly in 2016.

[151] So given that the Democrats are well positioned in many of these 30 districts, just given the enthusiasm for the party, but they are districts that Trump won in 2016.

[152] So Republicans must, in the back of their minds, think, these are ours to lose, we can find a way to get our voters out.

[153] What are the Republicans doing to try to make sure that they win most, if not all, of these races?

[154] Well, if you look at the president's conduct in the last few weeks.

[155] What's happening right now as a large group of people, they call it a caravan.

[156] Seems like he's trying to wrap up the base.

[157] That is an assault on our country.

[158] That's an assault.

[159] He's been talking about this caravan of migrants from Latin America.

[160] And in that caravan, you have some very bad people.

[161] You have some very bad people.

[162] And we can't let that happen to our country.

[163] He's sending troops to the border, try and defend against their not particularly imminent arrival.

[164] Our military is out.

[165] We have about 5 ,0008.

[166] We'll go up to anywhere between 10 and 50.

[167] 15 ,000 military personnel on top of border patrol, ICE, and everybody else at the border.

[168] Nobody's coming in.

[169] We're not allowing people to come in.

[170] He has argued against birthright citizenship.

[171] You can definitely do it with an act of Congress, but now they're saying I can do it just with an executive order.

[172] He has redefined gender so as not to recognize transgendered people anymore.

[173] They have a lot of different things happening with respect to transgender right now.

[174] You know that as well as I do.

[175] looking at it very seriously.

[176] And then on Wednesday, the president tweeted a provocative new advertisement that alleges that Democrats let an undocumented immigrant into the country who killed two police officers and then cuts to images of the caravan.

[177] It implies that Democrats are going to let in many more murderers.

[178] Attempt of murder.

[179] So it seems that the Republicans are trying to rev up the base in the final stretch.

[180] So the theory, basically, is that if you return to the issues that got you over the top in these places in 2016 you know maybe you can do it again he's basically bringing back the band I think that's a nice way to put it from 2016 who got him elected he's trying to reassemble those people in each one of these toss -up districts yeah hmm so is this strategy working in these toss -up districts from what you can tell from your polling we can't say that it's working district by district yet these toss -ups are all still extremely close what we can say is that the president's approval rating is up from a month ago, in these districts.

[181] In these districts, as well as nationwide, which seems to indicate that some number of Republicans who have had some reservations about his presidency to this point are returning to his side in the final stretch.

[182] And maybe that's because he's returning to the sort of issues that won them over in the first place.

[183] So the fact that the president's approval rating is going up in these toss -up districts suggests that the messaging that's so polarizing on issues like immigration is making Republican voters like him more and therefore making people more likely to vote for a Republican congressional candidate in that district, which may give Republicans some much -needed momentum in the final days of this race.

[184] The last part's the hope.

[185] I should note, though, that these are all still toss -ups because that hasn't happened yet.

[186] So we should understand what has seemed like a kind of random assemblage of very polarizing campaign tactics in the past couple of days.

[187] every day or every hour, it feels like a new provocation by the president.

[188] We should understand that as a measure of the presidents and the Republicans' anxiety that so many of these toss -up races remain toss -ups this close to the midterms.

[189] And they're basically just trying to prime the pump and hope that the voters who came out in 2016 and made him president will come out as a result of this for the congressional candidates in these districts.

[190] I think that's a fair point.

[191] These are districts that shouldn't be their most vulnerable districts.

[192] In the Republican dream world, these districts would already be fairly comfortably ahead.

[193] And then they'd be focusing on these toss -up districts that voted for Hillary Clinton, where this strategy wouldn't be effective.

[194] But what the strategy is focused on is sort of stemming the losses and keeping themselves even in play.

[195] It's not really a winning strategy necessarily.

[196] It's a bit of a desperation strategy.

[197] It's a strategy to prevent the big blue wave, basically.

[198] Not a strategy to hold the house.

[199] That's not to say that they couldn't win, to be clear.

[200] It is to say, though, that on the Republican path to victory, there's two steps, right?

[201] You've got to hold the seats that are really conservative, and then you've got to win these Clinton districts, and they haven't done step one yet.

[202] So now they're trying to do step one to prevent a blowout.

[203] What the president's doing is not designed to get them in one swoop to a House majority.

[204] It doesn't get them far enough.

[205] Nate, what you're describing is essentially a targeted, intense recreation of the kind of campaign President Trump ran in 2016 deployed in the final days of, a midterm election, in order to try to stave off a Democratic wave in the House.

[206] But I wonder, based on your polling, how much has changed or not changed about voter attitudes since 2016 when the president first deployed this divisive strategy?

[207] I see very little evidence of significant change.

[208] In many of our polls right now, the president's approval rating looks a lot like the final result of the 2016 election.

[209] So what's going to have?

[210] happen on Tuesday?

[211] I think we're going to have a lot of really close races.

[212] And it wouldn't surprise me if they all went one way or the other.

[213] What do you mean?

[214] The things can break one way at the end.

[215] Undecided voters can all go one way.

[216] Our turnout model could be sort of off across the board in one way.

[217] I just don't think people should be surprised if when you start out with 30 close races and suddenly someone wins 20 or 25 of them.

[218] So despite the fact that there are 30 races that are excruciatingly close, so close that there's no way to predict them, it's also very possible that the vast majority then we'll go Republican or the vast majority than we'll go Democrat.

[219] Yep.

[220] And I think it's worth thinking about just how different that election night would feel like even though it just turns on a couple of points.

[221] You know, if the Democrats win 20 plus of these seats, suddenly we're talking about the Democrats winning 40 seats in the House.

[222] If the Republicans narrowly squeak out these wins, even though in terms of what it would really say about the country's attitudes, it's basically the same thing.

[223] You're talking about a couple of points in a handful of districts.

[224] Nate, why even do this work?

[225] Does it actually matter if polling is, as you say it is, a miracle if it actually works.

[226] Polling is inevitably going to be out there.

[227] Someone's going to do it.

[228] And so then the question is, are those polls going to be transparent, nonpartisan, high quality?

[229] And I think that it all matters because, you know, this is democracy.

[230] And in the end, our system of government is based in part on how our elected officials and our democratic institutions react to the will of the public.

[231] and the decisions that are being made by our elected officials are all based on their assumptions about where the race is at.

[232] I mean, this whole conversation about the conduct of the president of the United States has been based on us trying to read into what they think is going on in the race.

[233] And they think they know what's going on the race based on their polling.

[234] And we can go down a long list of things that have happened over the last decade, potentially including the election of the president himself, that are really based on decisions.

[235] that were made based on the interpretation of polls.

[236] And bad polling is really dangerous.

[237] Like, I think that it's not a coincidence that a big chunk of Americans, white voters without a college degree, have been systematically underrepresented in polls probably for a decade.

[238] No one really noticed.

[239] And then, you know, in 2016, that same group declared that they were a silent majority and backed an insurgent candidate who listened to them, even though most of the smart analysts, based on their bad interpretation of mediocre polling data, had concluded that Democrats didn't need.

[240] need white working class voters to win or stuff like that.

[241] Those things are deeply related.

[242] You might not wish that horse race polling was the way that our democracy worked.

[243] You might wish that politicians didn't make decisions based on who's up or who's down or how they might win, but they do.

[244] And so it is important for the democracy to have the most accurate information at how the public is behaving.

[245] Right.

[246] Because the people who were elected look at those polls too, and they use it to figure out all kinds of things.

[247] I mean, you remember after 2012 when President Obama, you know, decided the top of his agenda needed to be immigration reform?

[248] And you might remember that, like, Sean Hannity briefly supported immigration reform, and Jeb Bush supported immigration reform, and Marco Rubio supported immigration reform.

[249] I do remember that.

[250] All of that stuff was based on, you know, their conclusion that Hispanic voters won the election for President Obama.

[251] You remember that?

[252] That conclusion was basically wrong, by the way.

[253] Obama, but based on polls.

[254] But based on polls that were, did not do an appropriate a job of representing the country.

[255] And I think it's pretty easy to draw a straight line from bad measurement of the way that the 2012 election was won and lost to a lot of the events that have unfolded in our politics since then.

[256] Whoa, you're kind of blowing my mind.

[257] In other words, misunderstanding polls, bad polling, literally can change the entire direction of the country.

[258] Yeah.

[259] There's no doubt in my mind that polls profoundly affect the decision -making of basically all parts of our Democratic.

[260] process, whether that's the behavior of officials in Washington, the conduct of the campaigns on the campaign trail, and even the decision -making of journalists and media organizations.

[261] So if the polls are not working, then the whole democratic process isn't working in some way.

[262] So the stakes of polling are higher than whether we get it right on Tuesday, whether we get it right on any Tuesday in November.

[263] Yeah.

[264] So we better do them well.

[265] Thank you, Nate.

[266] Thanks for having me. Here's what else you need to know today.

[267] The Times reports that after weeks of deliberations, the Trump administration has decided to stand by Saudi Arabia's crown prince, Muhammad bin Salman, despite the role that those close to him played in the murder of the Saudi journalist Jamal Khashoggi.

[268] The president and his aides have concluded that bin Salman is almost certain to stay in power and that the most practical path is to find ways to work with him.

[269] and perhaps exploit his weakened standing for the benefit of the U .S. And on Thursday, thousands of Google employees staged a dramatic protest of how the company handles sexual harassment claims by walking out of their offices around the world.

[270] I stand here today as an ally to anybody who has suffered, anyone who is suffering, or anyone who may suffer as a result of any such unacceptable The protest was a response to revelations made by the Times last week that Google paid millions of dollars to departing male executives accused of harassment and then remained silent about both the accusations and payments.

[271] We demand to representation in conversation with HR and we demand that a better reporting process be available to all Google and Alphabet workers, FTEs, Interns.

[272] The Daily is produced by Theo Balcom, Lindsay Garrison, Rachel Quester, Annie Brown, Andy Mills, Ike's Comeraja, Claire Tennisgetter, Michael Simon Johnson, Jessica Chung, Alexandra Lee Young, and Nina Pawtuck, and edited by Paige Cowett, Larissa Anderson, and Wendy Doer.

[273] Lisa Tobin is our executive producer.

[274] Samantha Henig is our editorial director.

[275] Our technical manager is Brad Fisher.

[276] Our engineer is Chris.

[277] Wood, and our theme music is by Jim Brunberg and Ben Lansford of Wonderly.

[278] Special thanks to Sam Dolman, Michaela Bouchard, and Stella Tan.

[279] That's it for the Daily.

[280] I'm Michael Barbaro.

[281] See you on Monday.