3 September 2014
A Perspective on The Accuracy of Meteorologists
Posted by Dan Satterfield
This is a guest post by Brad Panovich the Chief Meteorologist at WCNC-TV in Charlotte, NC
I know what you are thinking. “Here we go another Meteorologist telling me how great they are”. That is not the purpose of this blog post. I am simply attempting to put what we do in perspective. Plus I’m fascinated by how we perceive certain things about weather forecasting versus other professions we are all familiar with.
The joke starts like this……
Fool me once, shame on you. Fool me twice, shame on me. Fool me 350,000 times, you are a weather man.
— Bill Murray (@BiIIMurray) October 8, 2013
Or the always popular,
“I wish I could be wrong ____% of the time and keep my job”
I get it we aren’t always right, even though we are accurate more times than we get credit for. What I always found fascinating is that when people give forecasters a tough time over a busted forecast. I wonder why they never apply this same logic to other professions?
For instance did you know that according to the Journal of Clinical Oncology that 44% of some types of cancer are misdiagnosed? The Mayo Clinic in 2012 found that there was about a 26% misdiagnosis rate amount all illnesses. Doctors have very tough jobs and medicine has advanced way more than meteorology. Not to mention this one gigantic caveat is that doctors are not trying to predict the future.
Now I’m not picking on doctors but we would never get angry at a doctor for being overly cautious if he or she thought we might have cancer. Common sense says if you have a lump, dis-colored mole or other symptoms that might be cancer you get it check out ASAP. We have done a great job educating people on early diagnosis and being safe not sorry when it comes to cancer.
Yet people get mad that we issued so many tornado warnings even though on average about 75% don’t produce a tornado.
Tornado Accuracy Rates & Lead Time:
So what about tornadoes how good or bad are we?
(1996 – 2007 Tornado Warnings used County-Based Methodology. 2008 – Present Tornado Warnings use new Storm-Based Methodology.Weather Enterprise: Protecting Lives, Livelihoods, and Your Way of Life.)
The above chart shows the percentage of times a tornado touched down with a warning issued. That’s an average of 69% of the time a tornado touches down a warning is in place over this time period.
The average lead time over this time period 11.6 minutes.
What about the forecast in general?
As I have blogged before we take getting the forecast correct very seriously more so than anyone can imagine. http://wxbrad.com/charlottes-most-accurate-forecast-its-not-just-marketing-for-us/
Over the past 4 years our 24 hour forecast for the next day has been right 94% of the time. I bet you are thinking, NO WAY!!! Well it’s true but that still means that 6% of the time we blow it just 24 hours out. Over one calendar year that means 26 days we didn’t get it right.
Problem for me is that people remember those 26 days really well and really forget about the other 339 days I did get it right.
I get it…. because if we are getting it right then no one notices and that’s what we are supposed to be doing. I will remind you again though we are trying to predict the future.
Sports; what’s good for them sucks for us!
You saw the tornado warning accuracy stats around 70% and our 24 hour forecast accuracy of 94%. What if we compared that to All-Star and Hall a Fame athletes, who are doing incredible things yet still not trying to predict the future?
Amir Johnson let the league in field goal percentage this last season at 65%. Which means he missed 35% of the time. Kevin Durant was the MVP and he only shot 50.3% from the field. Meaning he missed half his shots. He did make 87% of his free throws though.
Again not predicting the future.
Philip Rivers led the league in competition percentage last season at 69.5%. That was awesome but he still missed 30.5% of his passes. Again a very hard thing to do is to play NFL quarterback but still not predicting the future.
Always a stats field day here. Miguel Cabrera led the league in hitting last season at .348 which basically means he got a hit 35% of the time. That means he didn’t get a hit the other 65% of the time. Ty Cobb is considered the best hitter of all time at .366 but that’s only a 37% success rate. Again hitting a baseball maybe the hardest thing to do in sports, yet still not predicting the future. Though hitters always try to guess what type of pitch is coming next.
Breaking News….we aren’t always right. The beauty is I know this and I work within the inherent uncertainty of weather forecasting. I know I can’t get it right all the time my goal is to reduce the error to as small as possible. I also learn more from a blown forecast just as much or more than a correct one. Like any walk of life you have to fall down to learn to get up. Mistakes are only bad if you don’t learn from them.
Just remember to keep things in perspective. Whatever job or career you have. Think about the times you messed up. Some small, some large, but usually always correctable. Think about the mistakes that happen in everyone’s line of work that have nothing to do with predicting the future. Then realizing how far we have come and how awesome it is that in 2014 we can get the 24 hour forecast correct 94% of the time and give you on average 12 minutes warning before a tornado hits. It amazes me and I work as a Meteorologist.
Brad Panovich | Chief Meteorologist
WCNC-TV NBC Charlotte, NC
I think you are right on the nose when you point out that people ignore the 94% of the time you are right. It is the same as sports fans who believe that certain media outlets or announcers are against their favorite team. They don’t pay attention to all of the positives said; they only hear the negatives which confirm what they think. This is known as “confirmation bias”.
The same is true of weather forecasts. Since people are convinced you toss a coin to come up with a forecast, they only see it when you are wrong.
I want to pick on your tornado graphic just a bit – As I read your words I believe you’re saying that when a tornado touches down, there’s a 69% chance (or slightly better, if the apparent upward trend is to be believed) that a warning is in place – but of course in the logical extreme one could get this number to 100% by simply keeping tornado warnings in place at all times.
Tornado forcasters must balance the risk of failing to predict a tornado against the risk of making people not heed the warnings. This isn’t an indictment of the field – predicting chaotic events is hard and will likely remain hard – but the statistic you quote is meaningless without balancing it against the number of false positives, which is quite high.
I do appreciate good info on forecast validation, though – as a layperson I’m fascinated by it, and I’m always struck at how good a job y’all are doing, and confused by the scorn send your way.