This thread is about working out why more people are paying more attention to him than he deserves.
Actually, I think he deserves more attention, but I also think all applications of math, statistics, game theory, need more attention. If everyone looked at electoral college vote prediction trackers every day, the USA might be a better place.
Anyway, I think we've answered my initial question. It's good to have a real debate on a subject, where both sides can learn something and come to a greater understanding of a topic.
It's down to what you want. I like a more transparent, "this is our data, our method, and our results" approach, and Nate has a "I have secret sauce, and run thousands of simulations, and have all these graphs of uncertainty" approach, which comes off as more magical, or at least more mystical. And when you want to create a modern myth, transparency and sticking with pure data is the opposite of what you need.
That's not entirely fair. I see little reason to think Silver wants to create any kind of modern myth, or that Silver's main goal is not to make his predictions as good as possible. Clearly, making your predictions as good as possible will necessarily involve a lot of complexity. Yes, it would be better if his methodology was open-source, but it's hard to blame him for the fact that it isn't.
I don't follow FiveThirtyEight because it has "secret sauce"; I've chosen to follow it because it clearly and concisely shows a reasonable prediction based on a large amount of data that I have no desire to trawl through. I agree that based on what I've seen so far, HuffPost Pollster has shown itself to be more accurate, but ultimately a single election is a tiny sample for judging such a thing, and Pollster doesn't do as good a job of presenting their data and predictions as Nate does.
You've brought up some things that I agree ought to be fixed to improve the clarity of the site (the colour scheme, the fact that it's not made clear that the number of 313.0 is (I think) the mean value rather than the mode or median, and the rounding). I think it would be nice if those things were fixed, but they are relatively minor issues.
This thread is about working out why more people are paying more attention to him than he deserves.
Actually, I think he deserves more attention, but I also think all applications of math, statistics, game theory, need more attention. If everyone looked at electoral college vote prediction trackers every day, the USA might be a better place.
Anyway, I think we've answered my initial question. It's good to have a real debate on a subject, where both sides can learn something and come to a greater understanding of a topic.
Agreed.
One thing that I think hasn't been discussed enough is what the predictions were like well in advance of election day - that information is in many ways much more telling, given that the outcome is more uncertain the farther back you go. I want to see a comparison of what some of these models were predicting before and after the Denver debate, for one thing. This is another thing that is shown conspicuously on FiveThirtyEight - you can see a graph of the chance of Obama's victory over time.
It's down to what you want. I like a more transparent, "this is our data, our method, and our results" approach, and Nate has a "I have secret sauce, and run thousands of simulations, and have all these graphs of uncertainty" approach, which comes off as more magical, or at least more mystical. And when you want to create a modern myth, transparency and sticking with pure data is the opposite of what you need.
That's not entirely fair. I see little reason to think Silver wants to create any kind of modern myth, or that Silver's main goal is not to make his predictions as good as possible. Clearly, making your predictions as good as possible will necessarily involve a lot of complexity. Yes, it would be better if his methodology was open-source, but it's hard to blame him for the fact that it isn't.
Again, I'm going with the perception of Nate's site, not the actual goal of Nate himself. That Electoral Vote has all its data online and a clear outline of the methodology used, it gives a certain impression to visitors. Nate, by inserting himself and his (unclear or intentionally closed-source) methods between the original data and the presented outcome, APPEARS to be more magical. Again, my own wording here isn't very precise.
One thing that I think hasn't been discussed enough is what the predictions were like well in advance of election day - that information is much more telling, given that the outcome is more uncertain the farther back you go. I want to see a comparison of what some of these models were predicting before and after the Denver debate, for one thing.
Electoral Vote has a very easy back-in-time machine to look at that day's map and blog. The first date I can see with a prediction is May 29. 294 for Obama, 235 for Romney, 9 ties.
The best graph is the predictions over time. There are even the big news events marked on the graph. Simply put, there was never a moment when Romney was ahead for the entire year. The cool thing about that page is that the 208 and 2004 graphs are right there on the same page, at the same scale, and you can see how much closer it was back in 2004 using the same data and models.
Again, I'm going with the perception of Nate's site, not the actual goal of Nate himself. That Electoral Vote has all its data online and a clear outline of the methodology used, it gives a certain impression to visitors. Nate, by inserting himself and his (unclear or intentionally closed-source) methods between the original data and the presented outcome, APPEARS to be more magical. Again, my own wording here isn't very precise.
Yeah, a clear outline is given for Electoral Vote's methodology, and I think there are some clear flaws. Their use of a one-week window results in too few polls being used, and they're also missing out on some polling firms like YouGov and Gravis, which doesn't help. This issue is highlighted by their North Carolina prediction, which quite simply was not as much of a tossup as they made it seem to be. Their prediction in this case was significantly harmed by the fact that they only used two polls to make it.
It's unfortunate that FiveThirtyEight isn't open-source software, and there's a lot of data that goes into it that isn't directly available, but at least the poll numbers are shown, along with approximate weightings.
Also, it's not like Nate is keeping everything about his model secret. This article (linked earlier by Trogdor), although it's specifically about his approach to Senate predictions, describes quite a lot of what goes into the model. For further detail on how things are different for the presidential election, you can look at this article and then this one, describing how the model combines information from both state and national polls. Here's a quick quote from the first of the two:
In each of these states, I’m listing what I call the FiveThirtyEight modified polling average. The modified polling average starts out with our regular polling average, which weights the polls based on how recently they were conducted, the reliability of the pollster and the sample size of the poll. Then it makes three other adjustments: a likely voter adjustment (which reflects how polls of registered rather than likely voters typically underestimate the standing of the Republican candidate), a trendline adjustment (which reflects changes in the overall standing between the two candidates since the poll was conducted) and a “house effects” adjustment (which corrects for systematic biases toward either the Democratic or Republican candidates among certain pollsters). We’ve explained how each of these adjustments function in the past so I won’t dwell on them now.
Another interesting factor is Nate's use of an economic index, the details of which are given here. The relative weight of the polls and the economic index is described by this graph:
Simon Jackman of Pollster also gives a reasonable description of his own methodology, which is unsurprisingly very similar, albeit lacking an economic index. The first step in Jackman's model is to construct a model-based polling average, as is described here. Following this, he uses a Monte Carlo approach over the joint distributions to account for correlations between states, as he describes in the quite below (from this article):
The model uses poll data (and house effect corrections) to generate estimates of Obama and Romney levels of support in the states (and at the national level). The modeling is done simultaneously: if you will, there are up to 52 latent quantities (e.g., Obama support in 50 states, the District of Columbia, plus the national level) moving over time, with polls giving us (noisy) snapshots as to where the latent targets might be on any given day. Patterns of correlation in historical election results supplies prior information as to which latent targets are more likely to move together than others. The national trajectory is estimated with considerable precision (e.g., there is so much national polling in the last few months of a presidential election); moreover, by construction, fluctuations in the national level target will be correlated with state level fluctuations. Quite aside from the prior, historical information, the polling data strongly suggests correlated trajectories across states. All this is to say is there is a lot of "correlation across states" baked into the model.
Now, to generate a distribution over Electoral College outcomes, I sample repeatedly from the joint distribution of the latent quantities on a given day. Obama's level of support in Ohio on day t is correlated with Obama's level of support in, say, Iowa on that day (and in many other states too). For a given draw from the joint distribution, I simply note if Obama > Romney in state j = 1, ..., 52; if so, then I assign the Electoral College votes for state j to Obama, otherwise to Romney. Because I sample from the joint distribution for Obama support levels across states, the simulated Electoral College outcomes are likewise correlated. I do not compute the probabilities that Obama > Romney in each state (averaged across Monte Carlo samples) and then flip state-specific coins independently.
It's unfortunate that FiveThirtyEight isn't open-source software, and there's a lot of data that goes into it that isn't directly available, but at least the poll numbers are shown, along with approximate weightings.
Also, it's not like Nate is keeping everything about his model secret. This article (linked earlier by Trogdor), although it's specifically about his approach to Senate predictions, describes quite a lot of what goes into the model. ...
I understand he goes into a lot of detail, but I don't have time to read it now. On the other hand, Electoral Vote's method is simply "The most recent poll is always used, and, if any other polls were taken within a week of it, they are all averaged together. This method tends to give a more stable result, so the colors don't jump all over the place due to one unusual poll."
It's this kind of simplicity, which, if I had the same data, I could literally work out in my head and plot the results in a map or graph. I like that clarity, and I know that any error in the prediction isn't down to the model used, but in the polling data itself. Of course other models which are more complex may or may not have better predictive power, but I feel a "distance" from their final number due to the added complexity. Hence, I didn't clearly understand Silver's prediction in the way that other readers of his site did, even AFTER looking for a long time and trying to get to the bottom of it.
Sadly, complexity is a thing that happens in the real world, and this inevitably results in complex models.
I can agree that Electoral-Vote.com does a reasonably good job, but the shortcomings are obvious if you look at its final predictions - its final estimate for Nevada was based on a single poll, and its estimate for North Carolina was based on only two. Moreover, all three of the aforementioned polls were from just one polling organization, PPP.
I can understand wanting to avoid being mislead by possible flaws in the modeling process, but there are better ways to do so.
Yup! But I understand the shortcomings. If Nate Silver's predictions have a shortcoming it's that they are too complex (or not even publicly readable) for me to understand if they might have shortcomings in the first place.
As far as simplicity is concerned, I'm fairly sure RealClearPolitics uses a simple, straight-up average, but they tend to use more polls so I expect they would be more reliable than Electoral-Vote.com. They also list which polls they have used in calculating the average (see this for an example). It's unfortunate (and a poor decision on their part) that RCP doesn't come out and explicitly say on their site exactly what their approach is, though.
If you wanted to be sure of avoiding model-related issues, I think the most important thing to do would be to watch more than one site. I expect you'd do well watching, say, FiveThirtyEight, HuffPost Pollster, and RealClearPolitics; that way you would easily notice a discrepancy and could look into it further.
Well, if we imbue the drones with enough AI to be able to react to threats as they appear, we could legitimately say it wasn't the USA that was shooting down their fighters, but Rogue Drones.
Also, did we ever use the drones to actually shoot missiles into Iran? I'd say that's a pretty hostile act as well, and theirs could be interpreted as self defense.
Iranian fighters shot down a US recon drone. Time to start putting air to air missiles on them.
They didn't shoot it down. They shot at it, but the shot missed.
I do stand corrected. Still a hostile act.
True, though it's arguable as to what the proper response would be, especially since this was "no harm, no foul." It was certainly stupid on the Iranian's part, however.
It may have also been a screw-up on the pilots' part - they may have lost track of their position (not realizing they were over international waters) and been jumpy at the radar contact of the drone. They may have been trying to "scare off" the drone from Iranian airspace.
It may have also been a screw-up on the pilots' part - they may have lost track of their position (not realizing they were over international waters) and been jumpy at the radar contact of the drone.
an unarmed drone. And to the best of my knowledge (someone please correct me if I'm wrong) we don't arm our drones with AA missiles to begin with so even if it WAS armed it was no threat to the fighters.
It may have also been a screw-up on the pilots' part - they may have lost track of their position (not realizing they were over international waters) and been jumpy at the radar contact of the drone. They may have been trying to "scare off" the drone from Iranian airspace.
Also a possibility. The drone was 16 miles away from Iranian airspace. International waters is defined as 12 miles away. I can believe a 4 mile mistake to be an honest mistake and not trying to pick a fight (although with the wackjobs running Iran, who knows). I think the justified response to this is a stern warning. Now, if the shot hit and especially if the shot hit a manned aircraft, something more severe would be warranted.
It may have also been a screw-up on the pilots' part - they may have lost track of their position (not realizing they were over international waters) and been jumpy at the radar contact of the drone.
an unarmed drone. And to the best of my knowledge (someone please correct me if I'm wrong) we don't arm our drones with AA missiles to begin with so even if it WAS armed it was no threat to the fighters.
1) Drones can be armed, however. 2) You're assuming the pilot could tell that it was a drone (although admittedly they're slightly hard to mistake)
At a reasonably long engagement distance, it's perfectly reasonable to expect that a fighter might not be able to tell directly from a radar signature, and if his instruments were on the fritz it's possible (though unlikely) that he could have mistaken it for another type of aircraft.
Moreover, note that the Iranians fired machine guns at the drone and missed. Given how slow drones are, it shouldn't have been hard to hit even with guns, and the lack of missile fire means that the Iranian pilots were definitely firing warning shots, not trying to hit the drone.
True enough, but you're still not allowed to fire warning shots while the perceived threat is in international airspace. Its a gaff on Iran's part, though I doubt much will come of it. Still its like on the playground when someone threw a water balloon at you and missed. The intent was clear, but no harm came of it.
True enough, but you're still not allowed to fire warning shots while the perceived threat is in international airspace. Its a gaff on Iran's part, though I doubt much will come of it. Still its like on the playground when someone threw a water balloon at you and missed. The intent was clear, but no harm came of it.
Very true, but given how close this was to Iranian airspace (4 miles outside the 12 mile limit), there is a reasonable chance to assume it was a mistake and not a provocation. Now, if this was 50 miles into international airspace then it would be a more clear-cut case of provocation. Because it was just barely international airspace, a risk/benefit analysis of what the proper response should be is something along the lines of giving them a stern warning and perhaps sending out a few more drones into the same area, while remaining in international airspace, to show we aren't backing down.
If you wanted to be sure of avoiding model-related issues, I think the most important thing to do would be to watch more than one site. I expect you'd do well watching, say, FiveThirtyEight, HuffPost Pollster, and RealClearPolitics; that way you would easily notice a discrepancy and could look into it further.
The point is that there are no model-related issues. All of these state poll-based prediction all called the final outcome perfectly well. Some of them hit closer to the mark than others, but all of them admit that they were within the margin of error of the polls they used.
I'm not interested in looking at all those sites because none of them will tell me anything new beforehand. If one of them was waaaay out of line cough unskewed cough, then I'd look elsewhere. But none of them were. I'll *check* more than one site, but I won't *watch* more than one. Since about June (when I started checking regularly) Obama was never in danger of losing this election. All the sites like this said the same thing, all summer and autumn.
It's only when the election results come in do we see who did a good job or not, and by then it's too late to go back and start following a different site.
I follow Electoral Vote because of the interesting blog posts, and also because the numbers are simple. I don't want any more complexity, nor to spend my own time and effort keeping up with more than one site.
Comments
Actually, I think he deserves more attention, but I also think all applications of math, statistics, game theory, need more attention. If everyone looked at electoral college vote prediction trackers every day, the USA might be a better place.
Anyway, I think we've answered my initial question. It's good to have a real debate on a subject, where both sides can learn something and come to a greater understanding of a topic.
I don't follow FiveThirtyEight because it has "secret sauce"; I've chosen to follow it because it clearly and concisely shows a reasonable prediction based on a large amount of data that I have no desire to trawl through. I agree that based on what I've seen so far, HuffPost Pollster has shown itself to be more accurate, but ultimately a single election is a tiny sample for judging such a thing, and Pollster doesn't do as good a job of presenting their data and predictions as Nate does.
You've brought up some things that I agree ought to be fixed to improve the clarity of the site (the colour scheme, the fact that it's not made clear that the number of 313.0 is (I think) the mean value rather than the mode or median, and the rounding). I think it would be nice if those things were fixed, but they are relatively minor issues. Agreed.
One thing that I think hasn't been discussed enough is what the predictions were like well in advance of election day - that information is in many ways much more telling, given that the outcome is more uncertain the farther back you go. I want to see a comparison of what some of these models were predicting before and after the Denver debate, for one thing. This is another thing that is shown conspicuously on FiveThirtyEight - you can see a graph of the chance of Obama's victory over time.
The best graph is the predictions over time. There are even the big news events marked on the graph. Simply put, there was never a moment when Romney was ahead for the entire year. The cool thing about that page is that the 208 and 2004 graphs are right there on the same page, at the same scale, and you can see how much closer it was back in 2004 using the same data and models.
It's unfortunate that FiveThirtyEight isn't open-source software, and there's a lot of data that goes into it that isn't directly available, but at least the poll numbers are shown, along with approximate weightings.
Also, it's not like Nate is keeping everything about his model secret. This article (linked earlier by Trogdor), although it's specifically about his approach to Senate predictions, describes quite a lot of what goes into the model. For further detail on how things are different for the presidential election, you can look at this article and then this one, describing how the model combines information from both state and national polls. Here's a quick quote from the first of the two: Another interesting factor is Nate's use of an economic index, the details of which are given here. The relative weight of the polls and the economic index is described by this graph:
Simon Jackman of Pollster also gives a reasonable description of his own methodology, which is unsurprisingly very similar, albeit lacking an economic index. The first step in Jackman's model is to construct a model-based polling average, as is described here. Following this, he uses a Monte Carlo approach over the joint distributions to account for correlations between states, as he describes in the quite below (from this article):
It's this kind of simplicity, which, if I had the same data, I could literally work out in my head and plot the results in a map or graph. I like that clarity, and I know that any error in the prediction isn't down to the model used, but in the polling data itself. Of course other models which are more complex may or may not have better predictive power, but I feel a "distance" from their final number due to the added complexity. Hence, I didn't clearly understand Silver's prediction in the way that other readers of his site did, even AFTER looking for a long time and trying to get to the bottom of it.
I can agree that Electoral-Vote.com does a reasonably good job, but the shortcomings are obvious if you look at its final predictions - its final estimate for Nevada was based on a single poll, and its estimate for North Carolina was based on only two. Moreover, all three of the aforementioned polls were from just one polling organization, PPP.
I can understand wanting to avoid being mislead by possible flaws in the modeling process, but there are better ways to do so.
If you wanted to be sure of avoiding model-related issues, I think the most important thing to do would be to watch more than one site. I expect you'd do well watching, say, FiveThirtyEight, HuffPost Pollster, and RealClearPolitics; that way you would easily notice a discrepancy and could look into it further.
Edit: Yes, international airspace http://gizmodo.com/5958932/iran-tried-to-shoot-down-american-drone
Also, did we ever use the drones to actually shoot missiles into Iran? I'd say that's a pretty hostile act as well, and theirs could be interpreted as self defense.
At a reasonably long engagement distance, it's perfectly reasonable to expect that a fighter might not be able to tell directly from a radar signature, and if his instruments were on the fritz it's possible (though unlikely) that he could have mistaken it for another type of aircraft.
Moreover, note that the Iranians fired machine guns at the drone and missed. Given how slow drones are, it shouldn't have been hard to hit even with guns, and the lack of missile fire means that the Iranian pilots were definitely firing warning shots, not trying to hit the drone.
I'm not interested in looking at all those sites because none of them will tell me anything new beforehand. If one of them was waaaay out of line cough unskewed cough, then I'd look elsewhere. But none of them were. I'll *check* more than one site, but I won't *watch* more than one. Since about June (when I started checking regularly) Obama was never in danger of losing this election. All the sites like this said the same thing, all summer and autumn.
It's only when the election results come in do we see who did a good job or not, and by then it's too late to go back and start following a different site.
I follow Electoral Vote because of the interesting blog posts, and also because the numbers are simple. I don't want any more complexity, nor to spend my own time and effort keeping up with more than one site.