> Keep in mind that the statement “candidate X will win with the probability of 60%” is also completely unfalsifiable in this case: there is no course of events that can prove it wrong.
Forecasters use the Brier score to translate these to a quantifiable track record of how well a model/a person predicts election outcomes. In this case 60% would be the confidence of the model.
If there is a systematic predictor (ie algorithm), the hypothesis is that the model can predict outcomes with a certain precision and therefore with multiple elections it could be falsifiable.
The Signal and the Noise by Nate Silver is an excellent book on this.
Of course apart from the fact thats its meaningless to apply the Brier score to the Octopus I do agree that measuring a model's performance on elections are hard to measure because they happen so rarely -- there is zero statistical signifance.
My main argument is even simpler than that. None of those predictors are ever wrong, because a probabilistic interpretation of a singular event is dishonest outside of a multi-universe interpretation. Nate Silver himself used this exact argument: after predicting Hillary's victory in 2016 with a probability of 70% he never said "I was wrong", he said "I gave Trump full 30% to win, and he won, so I was right". Well, that helped a lot.
In fact, he could have given any number instead of 70%, and he would be correct with any outcome, not because he is such a talented predictor but because probabilistic interpretation of a singular event is intrinsically misleading.
This is what I mean when I say this approach is not scientific and not falsifiable. The fact that he is sometimes correct in other elections and other circumstances (with other data sets, etc.) does not change that one bit, and Hume would be sad if he heard this argument.
Sure, but this is exactly what the brier score counteracts-- doesn't matter what Nate Silver claims, because he expressed an outcome with confidence we can say: person A is a better predictor than Nate Silver because his brier score is lower. Next time who should we give the money to? Not Nate Silver then!
Brier scores can be "pumped" just like any metric. Goodhart's law states that "When a measure becomes a target, it ceases to be a good measure", and Brier score is not immune. In a comment below, I gave a very simplistic example of how to do it.
I would trust the Octopus, but I'm unsure how much I would trust Yours trully 😜 joking aside the Brier score is hard to apply in this case but I can steelman the argument and chose any other metric and instead of the octopus choose a random model (or a random model that expresses 100% confidence each tine)-- yes there would be models that will have a good brier score and so elections are hard to quantify.
Btw the number of models the model was chosen from is also space of selection is super important: the Octopus was selected from a billion phenomena (that could be intepreted as a predictor).
If I only had one random model (not knowing it is random) and running it on all election of the last 200 years and getting perfect results -- even though its still somewhat small sample size I would believe it to be highly likely to be a good predictor.
I will give you an example of a semi-random model getting -- not perfect, but good -- results in election predicitons:
Let's say, we have 10 elections, 8 of which are have very clear results (clearly blue/red states, for example, very little margin of error), and 2 are a toss-up in a swing state. We correctly predict the 8 easy ones and guess 1 of the 2 toss-ups. Now, our prediction score is 9/10: exceptionally good, much better than that of 538.
Now, comes the general election and it's again a toss-up. We predict candidate A winning with 60% confidence.
Is this prediction more valuable than a coin toss?
Is there a situation where we can be ever wrong, or lose credibility, or be called a charlatan?
That's why I'm so often right, if not to say "almost always" lol
but no, really, -Thank you, that was a delightful read
(not sure about "de-facto leader, but okay. Let's assume))
Thanks!
> Keep in mind that the statement “candidate X will win with the probability of 60%” is also completely unfalsifiable in this case: there is no course of events that can prove it wrong.
Forecasters use the Brier score to translate these to a quantifiable track record of how well a model/a person predicts election outcomes. In this case 60% would be the confidence of the model.
If there is a systematic predictor (ie algorithm), the hypothesis is that the model can predict outcomes with a certain precision and therefore with multiple elections it could be falsifiable.
The Signal and the Noise by Nate Silver is an excellent book on this.
So, the Brier score for the predicting octopus was 0 after 8 games (it correctly predicted all of them).
Would you seriously consider his prediction for game number 9? Would you risk money on the outcome of game number 9, if this is your only information?
I've read The Signal and the Noise, and I think Nate Silver is very good at popularizing half-science and making sure it looks like proper science.
Did the octopus express confidence in the prediction?
Of course apart from the fact thats its meaningless to apply the Brier score to the Octopus I do agree that measuring a model's performance on elections are hard to measure because they happen so rarely -- there is zero statistical signifance.
My main argument is even simpler than that. None of those predictors are ever wrong, because a probabilistic interpretation of a singular event is dishonest outside of a multi-universe interpretation. Nate Silver himself used this exact argument: after predicting Hillary's victory in 2016 with a probability of 70% he never said "I was wrong", he said "I gave Trump full 30% to win, and he won, so I was right". Well, that helped a lot.
In fact, he could have given any number instead of 70%, and he would be correct with any outcome, not because he is such a talented predictor but because probabilistic interpretation of a singular event is intrinsically misleading.
This is what I mean when I say this approach is not scientific and not falsifiable. The fact that he is sometimes correct in other elections and other circumstances (with other data sets, etc.) does not change that one bit, and Hume would be sad if he heard this argument.
(edit: typos)
Sure, but this is exactly what the brier score counteracts-- doesn't matter what Nate Silver claims, because he expressed an outcome with confidence we can say: person A is a better predictor than Nate Silver because his brier score is lower. Next time who should we give the money to? Not Nate Silver then!
Brier scores can be "pumped" just like any metric. Goodhart's law states that "When a measure becomes a target, it ceases to be a good measure", and Brier score is not immune. In a comment below, I gave a very simplistic example of how to do it.
Yes! He was 100% confident!
Which is why his Brier score is actually 0.
Haha, I would then argue that predictions were then made by the octopus and You, the interpreter of its predicitons
Not me, his owners at the Sea Life Centre in Oberhausen.
I would trust the Octopus, but I'm unsure how much I would trust Yours trully 😜 joking aside the Brier score is hard to apply in this case but I can steelman the argument and chose any other metric and instead of the octopus choose a random model (or a random model that expresses 100% confidence each tine)-- yes there would be models that will have a good brier score and so elections are hard to quantify.
Btw the number of models the model was chosen from is also space of selection is super important: the Octopus was selected from a billion phenomena (that could be intepreted as a predictor).
If I only had one random model (not knowing it is random) and running it on all election of the last 200 years and getting perfect results -- even though its still somewhat small sample size I would believe it to be highly likely to be a good predictor.
I will give you an example of a semi-random model getting -- not perfect, but good -- results in election predicitons:
Let's say, we have 10 elections, 8 of which are have very clear results (clearly blue/red states, for example, very little margin of error), and 2 are a toss-up in a swing state. We correctly predict the 8 easy ones and guess 1 of the 2 toss-ups. Now, our prediction score is 9/10: exceptionally good, much better than that of 538.
Now, comes the general election and it's again a toss-up. We predict candidate A winning with 60% confidence.
Is this prediction more valuable than a coin toss?
Is there a situation where we can be ever wrong, or lose credibility, or be called a charlatan?