Forecast accuracy from World Weather Online have been added, and you can see them on the "Last Month" table on both the summary and detail pages. They were added because they provide forecasts worldwide, and they use their own forecasting models and algorithms. They don't just repackage the NWS forecast or purchase forecasts from another provider. From World Weather Online's about page, World Weather Online...

"developed our own weather forecasting model which could deliver reliable and accurate weather information for any geo-point in the world. Our weather model is run along with other meteorological models to compare and deliver accurate weather forecasts."

I hope this makes ForecastAdvisor more useful and interesting!

permalink    1 comment

When ForecastAdvisor started, the National Weather Service website was much different that it is today. Today, forecast pages come from the national site, with point forecasts being pulled directly from the National Digital Forecast Database.

Back then, the forecast pages were the responsibility of the regional offices: Eastern, Western, Central, Southern, Alaska and Pacific regions. When you typed a city, state on the main weather.gov site, you were redirected to one of regional sites for your forecast. The problem at the time was two-fold. There was no way to programmatically generate an URL to link to a specific zip code. And, the western region at that time didn't provide a zip code to city mapping. The western region's reluctance is understandable, in the west, zip codes cover huge areas, and it's difficult to pinpoint a specific city with a zip code.

So at that time, when you clicked on "National Weather Service" or "NWS Digital Forecast" in the weather forecast accuracy tables below the ForecastAdvisor forecast, it took you to a "Why Not Us?" page, letting you know that the ForecastAdvisor forecast is the NWS forecast, and that we couldn't link to the NWS forecast for your city directly.

Now that the NWS is a fully national site, and there is a way to generate an URL to link to a specific zip code's forecast, I've changed it so clicking on "National Weather Service" will now take you directly to the NWS forecast page for your city.

The "NWS Digital Forecast" link still takes you to "Why Not Us?" since the NWS Digital Forecast is the NDFD forecast and there isn't a web page for that. However, on the "Why Not Us?" page, there is a link to go to the NWS forecast page for your city, rather than just to the home page of weather.gov.

I hope this makes ForecastAdvisor even more useful to you, and I'd like to thank Anthony DeRobertis for emailing me and letting me know the NWS now has a way to generate URLs to get to specific zip code forecasts. Thanks Anthony!

permalink    0 comments

Last year we had a bit of trouble producing the full year aggregations. Our server was under-powered and there were some database architecture and code changes that we needed to make.

All of that is complete, and we were able to complete the 2010 aggregations in a matter of days, instead of the month it took last year!

The result is, the "Last Year" statistics you see on the site now are for the full-year 2010!

permalink    0 comments

The 2009 full year aggregations have run, and so the "Last Year" stats you see on ForecastAdvisor are now 2009 data. Also, "Last Month" is now February 2010 data.

permalink    0 comments

We are moving ForecastAdvisor and ForecastWatch websites to a new, bigger, more powerful server today (Monday, March 29, 2010). We have seen large increases in the number of visitor, customers, and forecasts we collect over the past few years. Moving to this new server will allow us to continue to grow and add new features in the coming years. You may see intermittant down-time or issues with the pages (data not being displayed, etc.) over the next 24 to 48-hours as everything gets moving and caches get updated. Thanks for your patience during this time.

permalink    0 comments

The founder of ForecastWatch, Eric Floehr, appeared on Bloomberg TV during the 1pm news hour.

The segment discussed how weather forecasts can be used in investing and trading, and how weather strongly affects consumer decisions and the economy. Eric made the following points:

• The commercial weather forecast providers and the NWS and EC all do a good job
• Different weather forecasters do better in different regimes: A forecasting company may do better in the plains, another better on the coasts
• ForecastWatch customers include The Weather Channel, Telvent DTN, and CustomWeather
• ForecastWatch also works with weather forecast customers to understand how weather forecasts can be used in modelling and decision-making
• A one-degree improvement in the accuracy of weather forecasts could result in a savings of one billion dollars in energy costs
• The more a weather forecast changes, the less accurate it will be

You can watch the segment on Sling here or on Bloomberg here.

permalink    0 comments

The August 2009 issue of SmartMoney Magazine featured an article called "10 Things Your Weather Forecaster Won't Tell You". It was a well-researched article with a number of good points. The ten things are:

• "Long-term forecast? Your guess is as good as ours."
• “We’re pretty accurate—as long as the sun is shining.”
• “We’re often more show biz than science.”
• “Our high-tech gizmos do everything but predict weather.”
• “Want the temperature? Don’t ask the National Weather Service.”
• “Weather is big business.”
• “Bad weather means big ratings...”
• “...and it’s always bad during sweeps week.”
• “Accuracy? Who cares.”
• “Weather is recessionproof.”

You can read the full article here. ForecastAdvisor is mentioned prominently. It's a very good article and definitely worth the time to read.

permalink    0 comments

I am often asked if forecast for a week or more out is worth anything. Worth is a hard thing to measure, but skill is easier. And if there is no skill, then likely the forecast would not be worth much.

At ForecastWatch we use two measures of skill. How much better is a weather forecast than climatology, and how much better is a weather forecast than persistence. Those are both unskilled forecasts, in that it doesn't take any skill to create them. The climate weather forecast says that the weather will be exactly like the 1971-2000 average. A persistence forecast says the weather however many days in the future will be exactly like today.

The easiest way to measure skill, at least for temperature forecasts, is to compare average absolute error. The difference between the temperature forecast, and the actual temperature that occurred, is the error. Take the positive difference, average it over a lot of forecasts, and that is a measure of how good a forecast is at predicting the temperature. An average absolute error of 3 degrees is generally better than an average absolute error of 6 degrees.

The chart below shows the average absolute high temperature forecast error for all of 2008, for all stations, and all providers. Lower is better. Red is the unskilled persistence forecast. Yellow is the climate average forecast. The green line is the average of all weather forecast providers. Lower is better. The climate average forecast is a straight line because it doesn't matter how many days you forecast out, the climate average forecast for a given date is always the same.

What sticks out on this graph is the intersection of the yellow climate average line with the green forecasters' line at nine days out. What that means is that the weather forecasters predicted high temperatures nine days out just as well (or just as badly) as using the 1971-2000 climate average high temperatures as the forecast prediction for each location.

What is even more interesting is that for high temperature forecasts greater than nine days out, weather forecasters (at least the ones ForecastWatch tracks) do WORSE than the climate average forecast. This means that you would do better just looking at historical average temperatures when determining the temperature more than nine days out.

The American Meteorological Society recently said as much. Every ten years, they release an information statement describing the current state of weather science. The statement released August 2007, made this observation:

"The current skill in forecasting daily weather conditions beyond eight days is relatively low."

The AMS is very wise. And in fact, for this sample, composed of nearly 18 million forecasts over the year from multiple national providers, there is no skill in daily weather forecasts beyond nine days.

On the positive side, meteorologists always did better than the persistence (tomorrow is like today) forecast. There are many other interesting things about this graph that I'll talk about later.

permalink    3 comments

New data for both the monthly and yearly accuracy data tables is now in. Last year now is for 2006, and last month is December 2006. There was a small delay in getting the hourly observations, and we did some year-end audits before we ran the full-year numbers.

January 2007 data is loading now, and should be reflected over the weekend.

I'll be posting more 2006 statistics soon.

permalink    2 comments

We just returned from a Christmas vacation to Fort Myers, Florida to visit the folks. The weather was beautiful, and we were able to go to the beach one day, go on an airboat ride in the Everglades another day, and swim pretty much every day (though I must confess that the pool was heated).

We've visited Fort Myers every holiday season for the past several years. One of our kids' favorite things to do besides swimming and the beach is visiting the Imaginarium, Fort Myers' children's science museum. The museum is owned and operated by the city of Fort Myers on the site of a historical water plant.

The kids really enjoy the weather exhibit, and the hurricane simulation (which blows air in a chamber at 45 miles per hour). The weather exhibit includes several stations. One is about clouds and includes a "cloud maker" where kids can move their hands around in the "cloud" and see how solid clouds really are. Another shows current conditions and radar maps, along with a NOAA weather radio. Yet another simulates a thunderstorm.

What my two girls enjoyed the most was the interactive TV weather studio. There is a desk and microphone, with a US map with stick on symbols for high pressure, low pressure, sunny, etc. There is a camera pointed to the desk and map, which is "broadcast" to a television. See the picture below:

The weather map is from American Educational Products (note: I have not been paid nor asked to mention this company...I just think they offer some nice educational weather products) and could be purchased for your home, school, or science center for $36 here. Thankfully, you can also buy extra weather symbols. The Imaginarium weather map was missing quite a few compared to our last visit. I'm sure they "disappear" quite frequently. I'm going to contact them to see if they need a donation.

Have you been to a good educational weather display? Please let me know!

permalink    0 comments

George Bryson of the Anchorage Daily News was interested in helping his readers understand the accuracy of Alaska weather prediction. He wanted to give his readers a better understanding of how difficult it is to forecast weather in Alaska. In addition to consulting local meteorologists in media and the National Weather Service, he contacted ForecastWatch for data and insights about forecasting weather.

I think the article is very well written and presents the data accurately and realistically. The article was on the front page of the Sunday, November 12th issue of the newspaper, and also appeared online. You can view a PDF of the online version.

permalink    0 comments

On this Thanksgiving day, I hope all of you are enjoying sunny skies, good food, and are surrounded by the love of family and friends.

I am certainly thankful for my wife, two children, my family, friends, the great people I get to work with every day, and the beautiful Earth we have been given.

I am also very thankful for meteorologists this Thanksgiving. They provide a service that is often under-appreciated. Their work can and does save lives. I know that people sometimes like to kid that they would like to be a meteorologist because they'd like to be in a job where they could be right only half the time and not get fired. Understanding the complex dynamics of the atmosphere and its interaction with land and sea, and predicting out many days in advance takes a lot of skill, a lot of brains, and a lot of dedication.

I will never forget the National Weather Service bulletin that went out last year on August 28. It was right before Hurricane Katrina made landfall. It began "...DEVASTATING DAMAGE EXPECTED...". It was followed by:

.HURRICANE KATRINA...A MOST POWERFUL HURRICANE WITH UNPRECEDENTED STRENGTH...RIVALING THE INTENSITY OF HURRICANE CAMILLE OF 1969. MOST OF THE AREA WILL BE UNINHABITABLE FOR WEEKS...PERHAPS LONGER. AT LEAST ONE HALF OF WELL CONSTRUCTED HOMES WILL HAVE ROOF AND WALL FAILURE. ALL GABLED ROOFS WILL FAIL...LEAVING THOSE HOMES SEVERELY DAMAGED OR DESTROYED.

Reading the bulletin sent shivers down my spine. I've been very involved with the weather as an amateur and in my business (ForecastWatch), but I'd never read anything like that. It was unprecedented. It scared me. I cannot imagine how it made people in the path of the storm feel. But however they felt, that strongly worded statement saved lives.

Don't take my word for it, though. The government report on the government's response to Hurricane Katrina titled "A Failure of Initiative: Final Report of the Select Bipartisan Committee to Investigate the Preparation for and Response to Hurricane Katrina" was very critical of many areas of our federal government. But it had this to say about the weather forecasters:

We reaffirmed what we already suspected — at least two federal agencies passed Katrina's test with flying colors: the National Weather Service (NWS) and the National Hurricane Center. Many who escaped the storm's wrath owe their lives to these agencies' accuracy. This hearing provided a backdrop for the remainder of our inquiry. We repeatedly tried to determine how government could respond so ineffectively to a disaster that was so accurately forecast.

In addition to the National Weather Service, Accuweather, The Weather Channel, and other private sector meteorologists helped warn citizens and helped in the response to the devastation. The Weather Channel, for example, created a message board for people looking for information about loved ones to connect, and gave more than $1 million dollars to Hurricane Katrina relief efforts.

So this Thanksgiving I am thankful for all meteorologists whether employed by the government or the private sector, and for all they do to help us plan our weekend, and keep us safe from weather disasters.

Happy Thanksgiving!

permalink    0 comments

The September accuracy data has been aggregated and is now available. You should see the "Last month" accuracy values on your forecast page have updated. For Columbus, Ohio, for example, there wasn't much change...Weather Channel moved up from #3 to #2 while the National Weather Service did the opposite.

Temperature accuracy is beginning its seasonal dip. Overall high temperature accuracy was at its lowest in July, at 4.05 degrees error. In August it started moving back up to a peak in mid-winter. September continued the trend. Overall high temperature error in September was 4.57 degrees. You can read more about the seasonal nature of weather forecast accuracy in this blog entry.

One interesting thing to note is that it was a very cool September. ForecastWatch tracks how an unskilled climate forecast compares to weather forecasts by the weather forecast providers. But this also tells us how the climate is doing, because all we are doing is comparing climate normals with what actually happened. In September, for the about 800 observations locations we track, high temperatures were 2.33 degrees below 1971-2000 climate normals. Low temperatures, on the other hand, were only 0.14 degrees below normal. The National Climatic Data Center has said that September was the 31st coolest on record. There is more data from the NCDC here.

The map below is one available in ForecastWatch. Because it is from the perspective of the forecast, red means the forecast was too high, blue means the forecast was too low. If you want to look at it from the perspective of the actual temperatures, red areas are areas where temperatures were below climate normals, and blue where they were above.

The map shows how a climate normal forecast did in September. Red areas indicate areas where a forecast of the climate normal high was too high (in other words, the actual high temperature was below normal, on average). The blue areas are areas where the actual high temperature was above climate averages.

You can compare this map to the one produced by the NCDC. I think they are pretty comparable.

permalink    0 comments

ForecastAdvisor provides an accuracy measurement for one- to three-day-out forecasts combined. But ForecastWatch keeps data on forecasts out to nine days, and has data on forecasts for each day out. The percentage of temperatures within three degrees is a basic measure of accuracy. It isn't the only measure, but it is the accuracy measure most commonly known to non-meteorologists. Every city, it seems, has a television meteorologist who proclaims a "three degree guarantee".

Another interesting measure is a forecast "miss". If a temperature forecast is off by ten degrees or more, it is called a miss. That means that if the actual temperature was 80 degrees, a forecast is considered a miss (or a blown forecast) if the forecast temperature was 70 degrees or below, or 90 degrees or above.

The chart below shows the "within three degrees" and "missed forecasts" for both high and low temperature forecasts from all providers for one to nine days out in 2005. Not all of the providers tracked provide forecasts out nine days (and some offer even more). Each bar at one day out represents about one and a half million forecasts. Each bar at nine days out represents about 800,000 forecasts.

At one day out, for the entire country, high temperature forecasts are within three degrees of the observed afternoon high about 68%% of the time. High temperature forecasts are blown one day out about 3%% of the time. Many of these blown forecasts one day out are because of climate extremes that the models don't handle well, or timing errors with cold or warm fronts.

You might notice that the low temperature accuracy is lower than the high temperature accuracy. There are a couple of reasons for this. One, forecasts are taken at 6 pm, and the high is usually around 3-6 pm, whereas the low is around 3-6 am the next morning. Most forecasters, when they forecast a low temperature, forecast the overnight low. For "tomorrow's" (one-day out) forecast, the high will occur in around 24 hours from the forecast, the low, 12 hours after that. That 12 hour difference is important one day out, but becomes less important the farther out the forecast is. This is apparent in the graph. At nine days out, the difference between high and low temperature accuracy is only 1.5%%, whereas at one day out its 7%%.

Notice also that the "within three degrees" accuracy seems to taper off, and if you draw an imaginary line, it looks almost like if you continued the accuracy forward to 10, 11, 15, etc. days out, that it would converge on an accuracy around 30%% or 35%%. You might need to click on the graph to view the larger version to notice this. This is significant because the average accuracy of a climate forecast is about 33%%. A climate forecast is taking the normal, average high and low for the day and making that your forecast. So at nine days out, forecasters still show some skill. They are better than just using the normal temperature for the day. But not by much.

permalink    0 comments

On Thursday, the Wall Street Journal Online published a column by Carl Bialik, The Numbers Guy, called "Grading Weather Forecasts". It was about what I do here at ForecastAdvisor. Thank you everyone for all the positive comments and suggestions that I have recieved from people who have read the article and have tried ForecastAdvisor.

In the article, Dr. Bruce Rose, Vice President and Principal Scientist for weather systems at The Weather Channel, stated in the article that July and August are the easiest months to forecast for temperature, with February the toughest. He's right, but I wanted to expand on that comment.

The graph below shows high and low temperature error by month for all forecasters ForecastWatch tracks. The error measurement used is what's called "RMS error", or "root-mean-squared error." This error measurement takes the error value (forecasted temperature minus actual temperature) and squares it. This makes all error values positive, and also penalizes forecasts that are way off much more than forecasts that are close. A forecast 10 degrees off is given an error four times one that is 5 degrees off, rather than just two times if the error value was not squared. All the squared errors are then averaged and the square root is taken, so that the unit value of error is still degrees.

In the graph below, each month's data point is the aggregation of about 600,000 forecasts one- to five-days out from all the providers. I think it is a fairly representative sample. Note the dips and peaks in the error graph. The error lines peak in the winter, and bottom out in the summer. The graph's y-axis starts at 3 degrees error to emphasize the difference, but even so, a winter temperature forecast has about 75%% more error on average than a summer temperature forecast.

Just like Dr. Rose said, this past February was the worst month for error in 2006 so far, and the previous July had seen the least error before that. But why is it easier to forecast temperatures in the summer than in winter? For one, even in places like Key West, Florida, with some of the most unchanging weather in the continental U.S., there is more temperature variation in winter than in summer. The more temperatures fluctuate, the harder it is to predict.

Just for fun, I've added a linear trend line to the high and low temperature error graphs. If it's to be believed, the linear trend is down, which means forecasts are slowly getting better. This past winter, temperature forecasts did better overall than the winter of 2004-2005. It could also just be El Nino starting...

permalink    5 comments