ForecastAdvisor Weather Forecast Accuracy Blog
Sunday, April 4, 2010
Last Year Stats Now 2009
Submit to Digg
Submit to RedditMonday, March 29, 2010
Moving to a New Server
Tuesday, March 16, 2010
ForecastWatch Founder On Bloomberg TV
The founder of ForecastWatch, Eric Floehr, appeared on Bloomberg TV during the 1pm news hour.
The segment discussed how weather forecasts can be used in investing and trading, and how weather strongly affects consumer decisions and the economy. Eric made the following points:
- The commercial weather forecast providers and the NWS and EC all do a good job
- Different weather forecasters do better in different regimes: A forecasting company may do better in the plains, another better on the coasts
- ForecastWatch customers include The Weather Channel, Telvent DTN, and CustomWeather
- ForecastWatch also works with weather forecast customers to understand how weather forecasts can be used in modelling and decision-making
- A one-degree improvement in the accuracy of weather forecasts could result in a savings of one billion dollars in energy costs
- The more a weather forecast changes, the less accurate it will be
You can watch the segment on Sling here or on Bloomberg here.
Sunday, September 6, 2009
Ten Things Your Weather Forecaster Won't Tell You
The August 2009 issue of SmartMoney Magazine featured an article called "10 Things Your Weather Forecaster Won't Tell You". It was a well-researched article with a number of good points. The ten things are:
- "Long-term forecast? Your guess is as good as ours."
- “We’re pretty accurate—as long as the sun is shining.”
- “We’re often more show biz than science.”
- “Our high-tech gizmos do everything but predict weather.”
- “Want the temperature? Don’t ask the National Weather Service.”
- “Weather is big business.”
- “Bad weather means big ratings...”
- “...and it’s always bad during sweeps week.”
- “Accuracy? Who cares.”
- “Weather is recessionproof.”
You can read the full article here. ForecastAdvisor is mentioned prominently. It's a very good article and definitely worth the time to read.
Thursday, March 5, 2009
How Good Are Week Out Weather Forecasts?
I am often asked if forecast for a week or more out is worth anything. Worth is a hard thing to measure, but skill is easier. And if there is no skill, then likely the forecast would not be worth much.
At ForecastWatch we use two measures of skill. How much better is a weather forecast than climatology, and how much better is a weather forecast than persistence. Those are both unskilled forecasts, in that it doesn't take any skill to create them. The climate weather forecast says that the weather will be exactly like the 1971-2000 average. A persistence forecast says the weather however many days in the future will be exactly like today.
The easiest way to measure skill, at least for temperature forecasts, is to compare average absolute error. The difference between the temperature forecast, and the actual temperature that occurred, is the error. Take the positive difference, average it over a lot of forecasts, and that is a measure of how good a forecast is at predicting the temperature. An average absolute error of 3 degrees is generally better than an average absolute error of 6 degrees.
The chart below shows the average absolute high temperature forecast error for all of 2008, for all stations, and all providers. Lower is better. Red is the unskilled persistence forecast. Yellow is the climate average forecast. The green line is the average of all weather forecast providers. Lower is better. The climate average forecast is a straight line because it doesn't matter how many days you forecast out, the climate average forecast for a given date is always the same.
What sticks out on this graph is the intersection of the yellow climate average line with the green forecasters' line at nine days out. What that means is that the weather forecasters predicted high temperatures nine days out just as well (or just as badly) as using the 1971-2000 climate average high temperatures as the forecast prediction for each location.
What is even more interesting is that for high temperature forecasts greater than nine days out, weather forecasters (at least the ones ForecastWatch tracks) do WORSE than the climate average forecast. This means that you would do better just looking at historical average temperatures when determining the temperature more than nine days out.
The American Meteorological Society recently said as much. Every ten years, they release an information statement describing the current state of weather science. The statement released August 2007, made this observation:
"The current skill in forecasting daily weather conditions beyond eight days is relatively low."
The AMS is very wise. And in fact, for this sample, composed of nearly 18 million forecasts over the year from multiple national providers, there is no skill in daily weather forecasts beyond nine days.
On the positive side, meteorologists always did better than the persistence (tomorrow is like today) forecast. There are many other interesting things about this graph that I'll talk about later.
Tuesday, February 6, 2007
New Monthly and Yearly Accuracy Data
New data for both the monthly and yearly accuracy data tables is now in. Last year now is for 2006, and last month is December 2006. There was a small delay in getting the hourly observations, and we did some year-end audits before we ran the full-year numbers.
January 2007 data is loading now, and should be reflected over the weekend.
I'll be posting more 2006 statistics soon.
Tuesday, January 2, 2007
Educational Vacation
We just returned from a Christmas vacation to Fort Myers, Florida to visit the folks. The weather was beautiful, and we were able to go to the beach one day, go on an airboat ride in the Everglades another day, and swim pretty much every day (though I must confess that the pool was heated).
We've visited Fort Myers every holiday season for the past several years. One of our kids' favorite things to do besides swimming and the beach is visiting the Imaginarium, Fort Myers' children's science museum. The museum is owned and operated by the city of Fort Myers on the site of a historical water plant.
The kids really enjoy the weather exhibit, and the hurricane simulation (which blows air in a chamber at 45 miles per hour). The weather exhibit includes several stations. One is about clouds and includes a "cloud maker" where kids can move their hands around in the "cloud" and see how solid clouds really are. Another shows current conditions and radar maps, along with a NOAA weather radio. Yet another simulates a thunderstorm.
What my two girls enjoyed the most was the interactive TV weather studio. There is a desk and microphone, with a US map with stick on symbols for high pressure, low pressure, sunny, etc. There is a camera pointed to the desk and map, which is "broadcast" to a television. See the picture below:
The weather map is from American Educational Products (note: I have not been paid nor asked to mention this company...I just think they offer some nice educational weather products) and could be purchased for your home, school, or science center for $36 here. Thankfully, you can also buy extra weather symbols. The Imaginarium weather map was missing quite a few compared to our last visit. I'm sure they "disappear" quite frequently. I'm going to contact them to see if they need a donation.
Have you been to a good educational weather display? Please let me know!
Wednesday, December 20, 2006
Anchorage Daily News Article
George Bryson of the Anchorage Daily News was interested in helping his readers understand the accuracy of Alaska weather prediction. He wanted to give his readers a better understanding of how difficult it is to forecast weather in Alaska. In addition to consulting local meteorologists in media and the National Weather Service, he contacted ForecastWatch for data and insights about forecasting weather.
I think the article is very well written and presents the data accurately and realistically. The article was on the front page of the Sunday, November 12th issue of the newspaper, and also appeared online. You can view a PDF of the online version.
Thursday, November 23, 2006
What I'm Thankful For
On this Thanksgiving day, I hope all of you are enjoying sunny skies, good food, and are surrounded by the love of family and friends.
I am certainly thankful for my wife, two children, my family, friends, the great people I get to work with every day, and the beautiful Earth we have been given.
I am also very thankful for meteorologists this Thanksgiving. They provide a service that is often under-appreciated. Their work can and does save lives. I know that people sometimes like to kid that they would like to be a meteorologist because they'd like to be in a job where they could be right only half the time and not get fired. Understanding the complex dynamics of the atmosphere and its interaction with land and sea, and predicting out many days in advance takes a lot of skill, a lot of brains, and a lot of dedication.
I will never forget the National Weather Service bulletin that went out last year on August 28. It was right before Hurricane Katrina made landfall. It began "...DEVASTATING DAMAGE EXPECTED...". It was followed by:
.HURRICANE KATRINA...A MOST POWERFUL HURRICANE WITH UNPRECEDENTED STRENGTH...RIVALING THE INTENSITY OF HURRICANE CAMILLE OF 1969. MOST OF THE AREA WILL BE UNINHABITABLE FOR WEEKS...PERHAPS LONGER. AT LEAST ONE HALF OF WELL CONSTRUCTED HOMES WILL HAVE ROOF AND WALL FAILURE. ALL GABLED ROOFS WILL FAIL...LEAVING THOSE HOMES SEVERELY DAMAGED OR DESTROYED.
Reading the bulletin sent shivers down my spine. I've been very involved with the weather as an amateur and in my business (ForecastWatch), but I'd never read anything like that. It was unprecedented. It scared me. I cannot imagine how it made people in the path of the storm feel. But however they felt, that strongly worded statement saved lives.
Don't take my word for it, though. The government report on the government's response to Hurricane Katrina titled "A Failure of Initiative: Final Report of the Select Bipartisan Committee to Investigate the Preparation for and Response to Hurricane Katrina" was very critical of many areas of our federal government. But it had this to say about the weather forecasters:
We reaffirmed what we already suspected — at least two federal agencies passed Katrina's test with flying colors: the National Weather Service (NWS) and the National Hurricane Center. Many who escaped the storm's wrath owe their lives to these agencies' accuracy. This hearing provided a backdrop for the remainder of our inquiry. We repeatedly tried to determine how government could respond so ineffectively to a disaster that was so accurately forecast.
In addition to the National Weather Service, Accuweather, The Weather Channel, and other private sector meteorologists helped warn citizens and helped in the response to the devastation. The Weather Channel, for example, created a message board for people looking for information about loved ones to connect, and gave more than $1 million dollars to Hurricane Katrina relief efforts.
So this Thanksgiving I am thankful for all meteorologists whether employed by the government or the private sector, and for all they do to help us plan our weekend, and keep us safe from weather disasters.
Happy Thanksgiving!
Wednesday, October 11, 2006
A Very Cool September
The September accuracy data has been aggregated and is now available. You should see the "Last month" accuracy values on your forecast page have updated. For Columbus, Ohio, for example, there wasn't much change...Weather Channel moved up from #3 to #2 while the National Weather Service did the opposite.
Temperature accuracy is beginning its seasonal dip. Overall high temperature accuracy was at its lowest in July, at 4.05 degrees error. In August it started moving back up to a peak in mid-winter. September continued the trend. Overall high temperature error in September was 4.57 degrees. You can read more about the seasonal nature of weather forecast accuracy in this blog entry.
One interesting thing to note is that it was a very cool September. ForecastWatch tracks how an unskilled climate forecast compares to weather forecasts by the weather forecast providers. But this also tells us how the climate is doing, because all we are doing is comparing climate normals with what actually happened. In September, for the about 800 observations locations we track, high temperatures were 2.33 degrees below 1971-2000 climate normals. Low temperatures, on the other hand, were only 0.14 degrees below normal. The National Climatic Data Center has said that September was the 31st coolest on record. There is more data from the NCDC here.
The map below is one available in ForecastWatch. Because it is from the perspective of the forecast, red means the forecast was too high, blue means the forecast was too low. If you want to look at it from the perspective of the actual temperatures, red areas are areas where temperatures were below climate normals, and blue where they were above.
The map shows how a climate normal forecast did in September. Red areas indicate areas where a forecast of the climate normal high was too high (in other words, the actual high temperature was below normal, on average). The blue areas are areas where the actual high temperature was above climate averages.
You can compare this map to the one produced by the NCDC. I think they are pretty comparable.
Sunday, September 24, 2006
Accuracy of Temperature Forecasts
ForecastAdvisor provides an accuracy measurement for one- to three-day-out forecasts combined. But ForecastWatch keeps data on forecasts out to nine days, and has data on forecasts for each day out. The percentage of temperatures within three degrees is a basic measure of accuracy. It isn't the only measure, but it is the accuracy measure most commonly known to non-meteorologists. Every city, it seems, has a television meteorologist who proclaims a "three degree guarantee".
Another interesting measure is a forecast "miss". If a temperature forecast is off by ten degrees or more, it is called a miss. That means that if the actual temperature was 80 degrees, a forecast is considered a miss (or a blown forecast) if the forecast temperature was 70 degrees or below, or 90 degrees or above.
The chart below shows the "within three degrees" and "missed forecasts" for both high and low temperature forecasts from all providers for one to nine days out in 2005. Not all of the providers tracked provide forecasts out nine days (and some offer even more). Each bar at one day out represents about one and a half million forecasts. Each bar at nine days out represents about 800,000 forecasts.
At one day out, for the entire country, high temperature forecasts are within three degrees of the observed afternoon high about 68%% of the time. High temperature forecasts are blown one day out about 3%% of the time. Many of these blown forecasts one day out are because of climate extremes that the models don't handle well, or timing errors with cold or warm fronts.
You might notice that the low temperature accuracy is lower than the high temperature accuracy. There are a couple of reasons for this. One, forecasts are taken at 6 pm, and the high is usually around 3-6 pm, whereas the low is around 3-6 am the next morning. Most forecasters, when they forecast a low temperature, forecast the overnight low. For "tomorrow's" (one-day out) forecast, the high will occur in around 24 hours from the forecast, the low, 12 hours after that. That 12 hour difference is important one day out, but becomes less important the farther out the forecast is. This is apparent in the graph. At nine days out, the difference between high and low temperature accuracy is only 1.5%%, whereas at one day out its 7%%.
Notice also that the "within three degrees" accuracy seems to taper off, and if you draw an imaginary line, it looks almost like if you continued the accuracy forward to 10, 11, 15, etc. days out, that it would converge on an accuracy around 30%% or 35%%. You might need to click on the graph to view the larger version to notice this. This is significant because the average accuracy of a climate forecast is about 33%%. A climate forecast is taking the normal, average high and low for the day and making that your forecast. So at nine days out, forecasters still show some skill. They are better than just using the normal temperature for the day. But not by much.
Sunday, September 17, 2006
The Wall Street Journal Online Article about ForecastAdvisor
On Thursday, the Wall Street Journal Online published a column by Carl Bialik, The Numbers Guy, called "Grading Weather Forecasts". It was about what I do here at ForecastAdvisor. Thank you everyone for all the positive comments and suggestions that I have recieved from people who have read the article and have tried ForecastAdvisor.
In the article, Dr. Bruce Rose, Vice President and Principal Scientist for weather systems at The Weather Channel, stated in the article that July and August are the easiest months to forecast for temperature, with February the toughest. He's right, but I wanted to expand on that comment.
The graph below shows high and low temperature error by month for all forecasters ForecastWatch tracks. The error measurement used is what's called "RMS error", or "root-mean-squared error." This error measurement takes the error value (forecasted temperature minus actual temperature) and squares it. This makes all error values positive, and also penalizes forecasts that are way off much more than forecasts that are close. A forecast 10 degrees off is given an error four times one that is 5 degrees off, rather than just two times if the error value was not squared. All the squared errors are then averaged and the square root is taken, so that the unit value of error is still degrees.
In the graph below, each month's data point is the aggregation of about 600,000 forecasts one- to five-days out from all the providers. I think it is a fairly representative sample. Note the dips and peaks in the error graph. The error lines peak in the winter, and bottom out in the summer. The graph's y-axis starts at 3 degrees error to emphasize the difference, but even so, a winter temperature forecast has about 75%% more error on average than a summer temperature forecast.
Just like Dr. Rose said, this past February was the worst month for error in 2006 so far, and the previous July had seen the least error before that. But why is it easier to forecast temperatures in the summer than in winter? For one, even in places like Key West, Florida, with some of the most unchanging weather in the continental U.S., there is more temperature variation in winter than in summer. The more temperatures fluctuate, the harder it is to predict.
Just for fun, I've added a linear trend line to the high and low temperature error graphs. If it's to be believed, the linear trend is down, which means forecasts are slowly getting better. This past winter, temperature forecasts did better overall than the winter of 2004-2005. It could also just be El Nino starting...
Saturday, August 26, 2006
Weather Forecast Accuracy Gets Boost with New Computer Model
The National Center for Atmospheric Research (or NCAR), sent out a press release announcing that the high-resolution Weather Research and Forecasting model (WRF), developed by a partnership of NCAR, the National Weather Service, and over 150 research institutions, has been adopted for day-to-day operational use by civilian and military weather forecasters.
According to the press release, tests over the last year at NOAA and AFWA have shown that the new model offers multiple benefits over its predecessor models. For example:
- Errors in nighttime temperature and humidity across the eastern United States are cut by more than 50%%.
- The model depicts flight-level winds in the subtropics that are stronger and more realistic, thus leading to improved turbulence guidance for aircraft.
- The model outperformed its predecessor in more than 70%% of the situations studied by AFWA.
- WRF incorporates data from satellites, radars, and a wide range of other tools with greater ease than earlier models.
It will be very interesting to see how use of the new model trickles down into the public forecasts that ForecastWatch tracks. We'll be certainly keeping an eye on the trends and will let you know about any we see.
If you are interested in learning more about the new model, you can visit the WRF website here.
The WRF model is the replacement for the widely used MM5 model, which can run on anything from a Linux desktop to a supercomputer. The model is primarily written in Fortran, and comprises about 360,000 lines of code. You can run the model yourself by getting the source code here. It features a module-based approach, which will allow researchers to plug in their own specific models (say for hail formation, etc.) and physics schemes/solvers.
It is certainly exciting times in weather forecasting!
Monday, July 17, 2006
See Previous Forecasts
We recently introduced a new feature to ForecastAdvisor. If you click on any forecast, it will bring up a set of previous forecasts for that day. For example, go to a forecast page, say Fort Collins, Colorado (which was recently Number One on Money Magazine's 2006 Best Places To Live). The day I'm writing this, you see this forecast:
The current forecast for "today" is for a chance of rain showers with a low of 59° and high of 88° Fahrenheit. If you click on that forecast, the current forecast will dim and you will be shown the current and past weather forecasts for "today". As I write this, it looks like:
The forecast on the left of this weather forecast trend page is the current forecast for today. Today's forecast for today, you might say. It matches the forecast on the 5-day forecast page. The following forecasts are also forecasts for today. These forecasts for today were made on previous days.
For example, look at the forecast from two days ago. It is the third forecast (the forecast created today is first, the forecast made one day ago is second, so the forecast made two days ago is third). On that day (July 15, 2006) the forecast for today was for Mostly Sunny skies. And the temperature forecast was for 95°, now it's for 88°. That's good. It looks like the high today is going to be 89° in Fort Collins, and there have been showers in the area today.
So why was this feature added? Well, for one, curiosity. Being a weather geek I knew that weather forecasts changed frequently, but I didn't know by how much. I also didn't know if knowing how stable or unstable a forecast is would help someone understand how much the forecast should be trusted.
It's quite interesting to see how a forecast changes over time, and I do believe you can learn from it. At any rate, it gives a serious weather person more information than is presently available to help them understand the weather.
Are there any numbers to back up this feature? I took 2005 forecast data from Accuweather, The Weather Channel, Intellicast, CustomWeather, and the National Weather Service, a total of almost 1.2 million forecasts, and ran some numbers. What I was looking for is the average accuracy of the one-day-out forecast relative to how much the forecast changed. In the Fort Collins example above, the forecast changed from 89° to 88° from the current forecast to the one-day-out forecast, or 1 degree. What I did then is take all one-day-out forecasts that were one degree different and averaged how accurate they were. I did the same for zero degree different, two degree different, and so on.
The chart looks like this (click here for a larger version):
What this graph shows is that the smaller the difference in high temperature forecasts between the one-day-out forecast and any other day out forecast (two-day-out, three-day-out, etc.) the smaller the overall forecast error. This means that a forecast that changes a lot is likely to be more incorrect than a forecast that is stable. The dashed line is the average high temperature forecast error for all one-day-out forecasts. A forecast that changes two degrees or less between any previous forecast and the one-day-out forecast has an error average below the overall average, and that the larger the difference in forecasts, the greater the average error.
There is a lot of further analysis required before any definitive conclusions can be reached, but it's promising. And it certainly is more reason why we added the ability to view previous forecasts. I hope that you find the ability to view previous forecasts useful and enlightening as I have.
Please use the comments link below to let us know your thoughts!
Sunday, June 11, 2006
New, Cleaner Design and More
If you are reading this, you've noticed that we have unveiled our new design. This new design is cleaner and will allow us to add additional features easily. I think you will find that the forecast is easier to understand at a glance.
Thanks to Ben Hunt and the folks at Scratchmedia for the design and graphics. They did an awesome job!
If you have any comments about the design, or anything else, don't hesitate to contact us!
Archives
December 2005 January 2006 March 2006 June 2006 July 2006 August 2006 September 2006 October 2006 November 2006 December 2006 January 2007 February 2007 March 2009 September 2009 March 2010 April 2010 Current Posts