OpenWeatherMap - how enthusiasts make the weather

We live in an amazing time, something that used to seem very difficult suddenly becomes accessible and simple. The tasks for the solution of which it was necessary to create scientific institutes and large organizations are now being solved by self-organized groups of enthusiasts. Enthusiasts create maps and encyclopedias make films and develop software. I want to talk about the area in which this fresh wind of change has not yet blown in full force - this is the weather. And our project - OpenWeatherMap - we want to change this situation!

Why are we used to the fact that the weather forecast is the lot of specialized organizations?

It happened in our world that the institutes, laboratories and large state organizations, which are isolated from the outside world, have always been engaged in the weather. Who and how creates weather forecasts has always remained a mystery, covered with a touch of mysticism. Most of them are large state organizations united in the World Meteorological Organization.
')
And everyone knows how fantastically accurate forecasts are, and we rejoice like children when this is so. And how unpleasant and even dangerous are misses. In the end, the correct weather forecast does not only affect our daily mood. Our lives depend on it! In short, the weather is everything. Therefore, we thought and decided that such a vital topic as the weather should be accessible to all. It belongs to people and should be created by the same people who need it!
It should be available, convenient, and, most importantly, free!

We are inspired by the example of strikingly successful Wikipedia and OpenStreetMap. We believe that enthusiasts united by one idea can always achieve better results than large bureaucratic organizations.

So the OpenWeatherMap project appeared.

The goal of the project is a free API.

The idea of ​​free and accessible weather information led us to create and provide all application developers with a free API to get a variety of weather data, such as:
- Interactive map with current weather data
- The forecast for the week in the city
- Historical data in 120,000 cities of the world.
- Data from 40,000 weather stations all over the world obtained almost online. (Delay from seconds to an hour)
- Many different web maps , including maps of clouds, precipitation, wind, temperature, etc.

How it works?

At the entrance we have (1) data from weather stations, as well as (2) forecasts of meteorological services and scientific laboratories. This data is stored in the OWM database, and after processing using unique mathematical algorithms
street magic they turn into interpolated data about current weather anywhere in the world, as well as a set of maps with weather phenomena (3). And finally, (4) API is provided for all weather data, including maps with weather events. And now a little more about each of the 4 points.

What is at the entrance?

Weather station data

The OpenWeatherMap service receives data from professional and private weather stations. Today, there are more than 40 thousand such stations. Most of them are professional stations that are installed in airports and major cities of the world. But also equally important for the service are data from non-professional stations, which are collected and installed by amateurs wherever possible. And it is very important for us, because The level of amateur stations is now very high, the accuracy and, most importantly, the efficiency of the transmitted information are also at a very high level. And the more such stations will be connected to OWM, the higher will be the accuracy of both current weather information and forecasts. Amateur weather stations are generally a separate topic. But in this article I would like to emphasize that the range of such stations is very wide. And it will be interesting to establish such a station at home or at the dacha not only for a serious radio amateur, but also, for example, for father and son. You can buy a ready-made station costing from $ 100 to $ 1000, or you can collect it yourself, for example, on Arduino.

Forecasts

In addition to weather stations, the OWM service collects already processed weather forecast data. Calculating the global forecast for the whole world requires incredible computer capacity, and, alas, as long as we do not have our own IBM Deep Thunder. Fortunately, many weather services share our love for free data and the results of their systems are available. We use data from two weather services - the American NOAA, which drive the GFS model, and the Canadian one from Environment Canada. Both models are global and are aimed not at predicting a specific weather situation at a particular point on the earth, but at determining the general atmospheric dynamics on the planet as a whole. They have a big grid step - about 50 km and a large time range of the forecast - 5-7 days.
In addition, more detailed, with a smaller step, models are calculated for individual regions.
We combine data of different-scale forecasts - from the average and global, to local and more accurate, respectively. As a result, the OpenWeatherMap web maps work very conveniently - global forecasts are used for large scale, and in the process of increasing more and more detailed data is loaded.



But all this would not be so interesting if meteorologists would not share our love for free software! Some models used by major weather services are available in OpenSource - for example, the WRF model.

For example, a group of French enthusiasts has deployed a similar data processing system on their home servers and provides detailed and accurate forecasts throughout France. By the way, the cost of their entire system was 5 thousand dollars (Blade server for 16 cores), which does not look inaccessible.



The OpenWeatherMap service combines such accurate local models together with global ones. After that, we assimilate super-operative data from weather stations. And already on the basis of all this data, interactive maps are constructed, in which the degree of detail of the data depends on the scale of the map.

What is the output?

Maps - a variety of species

After processing the raw data that OWM receives from weather stations and meteorological institutes, various interactive maps are created with data on weather and weather phenomena. These are maps of clouds, pressure, temperature, precipitation, wind. Also, this data radar, weather stations and just the current weather anywhere in the world.
More about maps

API

The OpenWeatherMap service provides a free API to all weather data, their history, forecasts and all the variety of weather maps.
There are two types of APIs - JSON for receiving data and Tile / WMS for cartography.

Using JSON you can get:
- Weather data in more than 120 thousand cities. At the same time cities do not need to choose from a strictly limited list, they can be found on the map and see the estimated weather forecasts both in the city itself and in the nearest regions.
- Current weather data at the selected point in lat / lon coordinates
- The forecast for 7 days in a compact or full form
- “Raw” data received from meteorological stations
- Weather data for past periods
More about Meteo API

How to use our API

The range of API applications is infinitely wide. These are mobile applications for all platforms. These are various websites that can use the API to display current weather, various weather charts, widgets, etc. These are smart home systems.
For example, one of the users of the OpenWeatherMap from the UK organized an automatic watering system for their English garden. For planning the amount of water and irrigation mode, he uses data on precipitation forecast.

Why is everything in Russia as always?

I will say right away that we do not provide what is called a weather forecast on the territory of Russia. And in general, we do not conduct any activities in the country. This type of activity in Russia requires licensing.

However, let's see what happens with weather stations and weather enthusiasm in Russia. In the illustration below you can see the current picture of the distribution of weather stations. Compared to the dense coverage of the entire European part, Russia looks more than modest. And this is one of the reasons for the inaccuracy of weather forecasts in our vast territory.

And what can we change?

We are confident that with the help of enthusiasts you can make a difference. If you are interested in this topic and you want to help the project, then there are so many possibilities.
You can, for example, the following:
- connect your weather station
- if you are a developer - use our data in your projects
- or finish our mobile version - m.openweathermap.org
- yes, just write about us in your blog

Or if you are interested in mathematics - there are a lot of tasks around this. For example - one of the most important tasks in the system is the definition of current weather. As I mentioned above, we receive operational data from weather stations, which need to be interpolated into a grid of important for geographical points - cities or individual regions. The data from the stations is heterogeneous and not received regularly. Moreover, a lot of garbage comes from erroneous and incorrect measurements, they must be sifted out. And errors can appear in the data from quite reliable weather stations.
Now we are using a fairly rigid and non-adaptive algorithm. I would love to try some learning algorithm in this problem.
It is very interesting to try a neural network in a similar task.

Or, for example, the appearance of maps. I'd like to make my own layer based on OpenStreetMap data with fewer details, but more natural looking. If you have ideas, we will be very happy, especially if you know how to implement them at mapnik.

We need everything capable of holding a weapon!

Architecture

I will not dwell on the system architecture in detail; I will conduct a general picture for illustration:

Architecture - one piece.

We use only Open Source software, including NgInx, Apache, PHP, Tilecache, OpenLayers, Leaflet, Mapnik, PostGIS, Memcache, MongoDB, Gearman, MySQL, Python and more.

PS

A small lyrical digression.
Now the fashionable term bigdata is becoming a selling brand that is actively used by all sharks of the IT market. But apart from pure marketing, this has a huge impact on the entire industry - the point is that the cost of a transaction and the cost of storing large amounts of information are falling at a tremendous pace, and practically tends to zero. This opens up extraordinary opportunities not only for large market players, but also for small ones. Roughly speaking, several years ago the cost of developing and maintaining a system with a data volume of several terabytes and a load of thousands of transactions per second was too heavy for a small company, and even more so for enthusiasts. Now everything is changing!
Bigdata becomes available. This is a revolution in which everyone can participate!

Actually from here and the answer to the question that I often ask - and why is it free? We believe that building a business with a paid API is a way to hell, both technologically and commercially. Our API will always remain free - this is the main idea and goal of the project.

What will happen next?

The idea of ​​free weather itself provides opportunities for the development of the project. At the moment we are already working on new directions and functions, such as:
• Sociality. This is an opportunity to say - no, in our city now the snow is not falling, but the sun is shining and flowers are blooming. This means that the weather station in the city gives incorrect data, and we will look for other possibilities to support accurate weather in this city. Automatic algorithms are good, but people are better.
• We will continue to connect external settlement modules to the project for detailed information by region.
• Promotion of the idea of ​​amateur weather stations and connecting them to the OpenWeatherMap
• Using OpenWeatherMap for the agro-industrial industry, the need for local forecasts is very high here.
• Development of specialized meteorological applications
• And of course we really hope for YOUR help!