The outcome of this election was relatively obvious in this case, however. It was pretty easy to predict that Obama was going to win, and pretty much every state except Florida was rather predictable. Had the election been a genuinely close one, things might have been more interesting, and the models would have had something more interesting to show us.
With respect to multiple sites, by "watching" in this case I mostly just mean what I guess you meant by "checking" - basically just verifying that the one site you happen to be following (likely including blog posts, etc.) is on the mark. That in itself is enough to avoid any systematic or one-off mistakes in the one site.
In any case, unless you were doing something like betting money on the outcome, it's not going to matter that much if the site you happen to follow doesn't do a particularly good job. Consequently, the choice of any site you pay attention to is going to be a matter of personal preference; FiveThirtyEight's blog posts are also quite good, so I've been following those.
Personally, I'm perfectly happy with a complex model because I can understand how that added complexity is justified, and there is sufficient evidence to suggest that Nate's model is a reliable one. In fact, I think I could probably (with plenty of work and study) replicate a lot of the core methodology myself, so it really doesn't feel like magic to me the way it does to you.
Oh, and on the topic of accuracy, I want to see multi-site comparisons that analyse the models' accuracy using genuine statistics instead of this "50 out of 50" bullshit.
This article is a start, but there's quite a lot missing from it.
In fact, this is something I could put together myself, with some effort. If it's the kind of thing other people are interested in, I just might do it.
EDIT: Notably, Nate himself has said some interesting things on the topic of accuracy; this FiveThirtyEight post is somewhat enlightening; here's a quick quote:
Not only will the underdog — Mr. Romney — win some of the time, but he should win some of the time if we have estimated the odds correctly. If the set of candidates you have listed as 67 percent favorites in fact win 95 percent of the time, or 100 percent of the time, you’ve done something wrong. Over the long run, such candidates should win two out of three times — no less and no more.
Of course, it takes a very long time to realize the long run in presidential elections, since there is only one of them every four years. To the extent that one is evaluating the accuracy of political forecast models — whether they calibrate the odds correctly — it is probably better to look at something like races for the Senate. In that case, there are roughly 35 races held every other year, as opposed to just one every four years. Although these races are not completely independent from one another (there have been years in which Democrats or Republicans overperformed their Senate polls across the board), they are substantially more informative on balance for measuring the effectiveness of a series of forecasts.
I'd note here that in terms of Senate predictions, Nate's model did in fact do worse than a couple of the other models this year.
Oh, and on the topic of accuracy, I want to see multi-site comparisons that analyse the models' accuracy using genuine statistics instead of this "50 out of 50" bullshit.
So, was Nate Silver the most accurate 2012 election pundit? It depends which set of predictions you’re talking about. But the general lesson is this: the statistical methods used by the best-scoring pundits are not very advanced. Mostly, the top pundits outperformed everyone else by bothering to use statistical models in the first place, and by not making elementary mistakes of probability theory like the one committed by Brandon Gaylord.
which is basically what I said before:
The biggest reason is also the simplest one - people don't understand statistics. To the vast majority of people, (sufficiently advanced) statistics is indistinguishable from magic, and hence anyone using statistics is a witch.
I have a perfect answer to any question posed to a caucasian male involving women's health in any capacity: "That's a clown question, I decline to answer."
I have a perfect answer to any question posed to a caucasian male involving women's health in any capacity: "That's a clown question, I decline to answer."
Not meant as flamebait, because any answer anyone who is posed any kind of question like that is liable to offend one segment of the population or another, as well as usually specifically brought up for the purpose of riling up said segment, so its better just to not answer (which is a legitimate option).
Oh, and on the topic of accuracy, I want to see multi-site comparisons that analyse the models' accuracy using genuine statistics instead of this "50 out of 50" bullshit.
And again, the framing of the question isn't "Who was the most accurate 2012 Election Pundit?" but "Was Nate Silver the Most Accurate 2012 Election Pundit?" It's amazing to me that one guy can dominate the mind share about a single topic so much that no conversation about it can avoid mentioning him in the opening and closing statements, and that website after website is sticking his name in the title to increase readership.
I think that's a good position to aim for. In the juggling world, there is no conversation to be had about who is the best technical juggler which doesn't start and end with Anthony Gatto, and yet no conversations about who is the most interesting or creative juggler ever touch on him.
ARGH NOT EVERY CRITICISM OF AN ANGRY, RACIST AND BIGOTED RANT AGAINST THE ELECTION IS A VIOLATION OF YOUR FIRST AMENDMENT RIGHTS! STOP RUNNING TO DADDY GOVERNMENT TO PROTECT YOU FROM THE MEAN OLD LIBERALS, YOU FUCKING ASSHATS!
Huh, weird. Gamespot reviewed NS2 then they took down their review. WHAT DOES IT ALL MEAN?!
It was a really bad/negative review that a bunch of people got mad over. I swear nobody uses video game reviews to make purchasing decisions. Sites just post them because they get views from fans and haters who want to have flamewars.
It was a really bad/negative review that a bunch of people got mad over. I swear nobody uses video game reviews to make purchasing decisions. Sites just post them because they get views from fans and haters who want to have flamewars.
There are some that are OK. Giantbomb does theirs really well, for example, I occasionally use them to make purchasing decisions if I'm on the fence about something.
It was a really bad/negative review that a bunch of people got mad over. I swear nobody uses video game reviews to make purchasing decisions. Sites just post them because they get views from fans and haters who want to have flamewars.
There are some that are OK. Giantbomb does theirs really well, for example, I occasionally use them to make purchasing decisions if I'm on the fence about something.
Also, consider this.
Games you are a huge fan of are very unlike to have reviews before you buy it because you are going to pre-order. NS2 I pre-ordered before it was even playable. Even if I was a reader of reviews, they wouldn't effect that purchasing decision.
For other games, I'm just going to wait until they are old and cheap. In that case, a review isn't going to make a difference either. The game will have been out for so long that you don't need a review or score to know what the game is about. It's also so cheap, that buying it when it sucks is not going to be such a big mistake you needed to read reviews.
I don't really see any video games that fall into any other category except free games. I pretty much always either pre-order or wait until the cheap days.
Huh, weird. Gamespot reviewed NS2 then they took down their review. WHAT DOES IT ALL MEAN?!
It was a really bad/negative review that a bunch of people got mad over. I swear nobody uses video game reviews to make purchasing decisions. Sites just post them because they get views from fans and haters who want to have flamewars.
Actually, I just read the review, and it's rather sub-par journalism. Even if I agreed with his stated subjective enjoyment of the game, he is objectively wrong on many, many factual points.
Huh, weird. Gamespot reviewed NS2 then they took down their review. WHAT DOES IT ALL MEAN?!
It was a really bad/negative review that a bunch of people got mad over. I swear nobody uses video game reviews to make purchasing decisions. Sites just post them because they get views from fans and haters who want to have flamewars.
Actually, I just read the review, and it's actually sub-par journalism. Even if I agreed with his stated subjective enjoyment of the game, he is objectively wrong on many, many factual points.
I also read the review, and I agree with you. But I do agree with one thing from the review. NS2 desperately needs an interactive tutorial for playing and also commanding. A bunch of long ass YouTube videos and explore mode do not cut it.
I also read the review, and I agree with you. But I do agree with one thing from the review. NS2 desperately needs an interactive tutorial for playing and also commanding. A bunch of long ass YouTube videos and explore mode do not cut it.
Yup! I've not played it yet, but even as a pretty experienced NS1 player I was hoping for some good tutorials before I start.
You know? I've noticed that there are a lot of more modern games out there that lack a tutorial mode. Planetside 2, NS2, Counter Strike: Global Obfuscation... I wonder if that's a trend.
I often make purchasing decisions based on gamespot and giantbomb, but this is moreso because I understand what the people reviewing these games like and dislike. You can get a pretty good idea of whether or not you will like a game based on how much your interests line up with theirs.
You know? I've noticed that there are a lot of more modern games out there that lack a tutorial mode. Planetside 2, NS2, Counter Strike: Global Obfuscation... I wonder if that's a trend.
You know? I've noticed that there are a lot of more modern games out there that lack a tutorial mode. Planetside 2, NS2, Counter Strike: Global Obfuscation... I wonder if that's a trend.
Games you are a huge fan of are very unlike to have reviews before you buy it because you are going to pre-order. NS2 I pre-ordered before it was even playable. Even if I was a reader of reviews, they wouldn't effect that purchasing decision.
For other games, I'm just going to wait until they are old and cheap. In that case, a review isn't going to make a difference either. The game will have been out for so long that you don't need a review or score to know what the game is about. It's also so cheap, that buying it when it sucks is not going to be such a big mistake you needed to read reviews.
I don't really see any video games that fall into any other category except free games. I pretty much always either pre-order or wait until the cheap days.
You say that like I haven't already considered it. We have - as has been proven time and time again - different tastes, buying habits, and situation. I have broader tastes, and tend to end up on the fence about more games than you do - as you said, you only have the two modes, Buy right now, and don't buy right now(or ever), whereas I tend to be a little more "Do I buy this now, or this now, or this now, deferring the other two choices for next time?"
It also doesn't help that I don't want to spend a hundo on a game that I'm not really sure that I'll enjoy. Even you would balk at buying games you rather enjoy, if they were around a hundred bucks - and they don't go on sale or drop in price that fast here, either. Or at least, not with any great predictability.
I do try to pre-order things, though. For example, Sleeping dogs I bought for 40 bucks, about 4-5 months out from release, because I knew that I'd like it. I pre-ordered NS2 - not as early as you, but still, quite a while ago. Pre-ordered Guns of Icarus sight unseen.
So if I wait two years and no one is playing NS2 any more am I going to be happy that I waited? Probably not. I also am wary of purely multiplayer games that are described as having a high learning curve because people tend to be assholes if you're not good and learning aka. LoL.
Your non-review method only really works for single player games and not games that are 100% multiplayer that are a complete mistake to buy if no one is playing them or they're not good.
Comments
With respect to multiple sites, by "watching" in this case I mostly just mean what I guess you meant by "checking" - basically just verifying that the one site you happen to be following (likely including blog posts, etc.) is on the mark. That in itself is enough to avoid any systematic or one-off mistakes in the one site.
In any case, unless you were doing something like betting money on the outcome, it's not going to matter that much if the site you happen to follow doesn't do a particularly good job. Consequently, the choice of any site you pay attention to is going to be a matter of personal preference; FiveThirtyEight's blog posts are also quite good, so I've been following those.
Personally, I'm perfectly happy with a complex model because I can understand how that added complexity is justified, and there is sufficient evidence to suggest that Nate's model is a reliable one. In fact, I think I could probably (with plenty of work and study) replicate a lot of the core methodology myself, so it really doesn't feel like magic to me the way it does to you.
This article is a start, but there's quite a lot missing from it.
Basically, I want to see more of this kind of thing:
http://www.huffingtonpost.com/simon-jackman/pollster-predictive-perfo_b_2087862.html
but with graphs involving multiple sites.
In fact, this is something I could put together myself, with some effort. If it's the kind of thing other people are interested in, I just might do it.
EDIT:
Notably, Nate himself has said some interesting things on the topic of accuracy; this FiveThirtyEight post is somewhat enlightening; here's a quick quote:I'd note here that in terms of Senate predictions, Nate's model did in fact do worse than a couple of the other models this year.
http://appliedrationality.org/2012/11/09/was-nate-silver-the-most-accurate-2012-election-pundit/
In short,which is basically what I said before:
Sorry, I haven't made mine yet.
I think that's a good position to aim for. In the juggling world, there is no conversation to be had about who is the best technical juggler which doesn't start and end with Anthony Gatto, and yet no conversations about who is the most interesting or creative juggler ever touch on him.
That is all.
Games you are a huge fan of are very unlike to have reviews before you buy it because you are going to pre-order. NS2 I pre-ordered before it was even playable. Even if I was a reader of reviews, they wouldn't effect that purchasing decision.
For other games, I'm just going to wait until they are old and cheap. In that case, a review isn't going to make a difference either. The game will have been out for so long that you don't need a review or score to know what the game is about. It's also so cheap, that buying it when it sucks is not going to be such a big mistake you needed to read reviews.
I don't really see any video games that fall into any other category except free games. I pretty much always either pre-order or wait until the cheap days.
..or should soon.
@Muppet: They make wikis now.
It also doesn't help that I don't want to spend a hundo on a game that I'm not really sure that I'll enjoy. Even you would balk at buying games you rather enjoy, if they were around a hundred bucks - and they don't go on sale or drop in price that fast here, either. Or at least, not with any great predictability.
I do try to pre-order things, though. For example, Sleeping dogs I bought for 40 bucks, about 4-5 months out from release, because I knew that I'd like it. I pre-ordered NS2 - not as early as you, but still, quite a while ago. Pre-ordered Guns of Icarus sight unseen.
Your non-review method only really works for single player games and not games that are 100% multiplayer that are a complete mistake to buy if no one is playing them or they're not good.