How accurately does Yandex forecast rainfall in the winter? Analyzing the accuracy of forecast services.
In November, I published an article “Yandex.Meteum - technology without technology. Marketing is accurate to the district ” , where he correlated the quality of Yandex forecasts with other services. The emphasis was on temperature, without parsing other parameters. The conclusion was the following: the temperature forecast of Yandex does not show any exceptional results in comparison with the already proven forecasting services. I recommend to read the full text. This time it's time to check another key parameter - precipitation.
The main goal of the study is to understand the quality of precipitation forecast for Yandex in comparison with other predictive services. I draw your attention to the fact that the emphasis was on predicting the phenomenon, and not on the precipitation phase or their quantity.
The data of intellicast.com, which in January combined with the wundeground service , were used as control forecasts, but this had no effect on the forecasts, only the interface changed. The second control forecast is a comprehensive forecast from the site of the methodical cabinet of the Hydrometeorological Center of Russia. I told about this forecast in the previous publication.
')
Forecast intercepted in manual mode. The evaluation period is a meteorological day, that is, from about 8–9 am to 8–21 pm. I did not take night precipitation into account. The forecasts were estimated from November 3, 2018 to March 31, 2019, that is, for the period of time that the climatic winter falls on in the European part of Russia and part of the Urals, and the effect of convection is absent.
Yandex forecasts were also captured only for the daytime. At the same time, I tried to correlate forecasts from the site with those forecasts that Alice sent in push notifications. Later it turned out that the forecasts do not agree with each other, but we'll talk about this later.
The cities were chosen so that I could capture part of the south of the Urals and the main part of the European territory of Russia. I drew a line from Kurgan to Moscow, and chose cities on this site with approximately equal intervals. These are Kurgan, Ufa, Kazan and Moscow. The average distance between the points is 591 km.
As an assessment methodology, the Manual on short-term general-purpose weather forecasts was used, which is used in Roshydromet to estimate forecasts up to 72 hours. As I said, it was important for me to assess the fact of precipitation itself, and not their quantity or phase. Therefore, I made two parameters in the tables: dry (without precipitation) or precipitation.
If the site predicted any precipitation at any hour of the day, then the forecast included a gradation of “Precipitation”, if there was no precipitation in the forecast, then “Sukho”.
Official records of Roshydromet stations were used as actual information. If the precipitation gauge at the meteorological station recorded at least 0.1 mm in 12 hours, the status of “Precipitation” fit into the table. Further, prognostic and actual grades correlated with each other. I cite the numbers of weather stations that were used in the study: 28661 (Kurgan), 28722 (Ufa), 27595 (Kazan), 27612 (Moscow, VDNH).
For example, Yandex predicts light snow during the day, and the meteorological station does not record precipitation, then a prediction of 0% is set for the forecast. If precipitation is in any quantity, then 100%. Thus, all forecasts were placed on equal evaluation terms.
For transparency research, publish a link to the original table with all the data . I also tried not only to evaluate the forecasts, but also to correlate them with zero-time synoptic maps, which made it possible to identify those synoptic situations in which Yandex gives the lowest results of justification.
Initially, I hypothesized that Yandex will show similar results with other sites, or they will be slightly below the level of competitors. I had to say goodbye to this hypothesis at the end of November, when it turned out that the accuracy of Yandex was comparable to a coin toss. The results were at the level of random forecasts and ranged from 50 to 60%.
For 5 months, the accuracy of Yandex in precipitation for 24 hours was 58%. At intellicast / wunderground - 81%, and at the integrated forecast of the weather center - 80%!
Yandex was able to find an interesting feature - the farther from Moscow, the lower the accuracy. In Kurgan, the average accuracy was 45.6%, and in Moscow - 67.8%. The competitors' forecasts ranged from 75% to 85%, which looks very decent for the winter season.
The main problem of Yandex in the winter time is false precipitation. Yandex predicts rainfall, and other services expect dry weather, a typical situation. The saddest thing is that Yandex predicts rainfall even in those areas over which there is an extensive anticyclone. For example, on November 12, Yandex predicted precipitations for Kazan all day long, but they should not have been, because the anticyclone center was passing over the region, there was clear weather and there could be no rainfall physically.
A similar situation repeated on December 20, but already in Kurgan. Again an anticyclone was established over the city, but Yandex persistently gave precipitations. There were a lot of such strange situations, I just gave the most vivid examples.
Another problem turned out to be with Alice and her push-notifications, which she sends through the main application “Yandex”. For example, on January 3, it was written on the site that there would be no precipitation on January 4, and Alice sent me: “Snow is beautiful. A little snow tomorrow. ” For all 5 months there were only a few such cases, but the fact of disagreement between the services leads to different thoughts.
Of course, most often Alice reported the same forecast, which was listed on the site, but there was no precipitation. The error of “Meteum” is transmitted to the work of all other services and something must be done about it.
The precipitation forecast for 24 hours was sorted out, but I would like to indicate another important problem that is present in the work of the Yandex service. Weather".
In the previous article, I already wrote that Yandex publishes average monthly rainfall in all cities. The problem is the data source. The site indicates the source of NOAA, I can assume that Yandex is simply using data from the American computer reanalysis CFSR. Reanalysis is suitable for studying temperature, but it is extremely poor for studying precipitation. The computer model reproduces poorly actual rainfall, especially of low intensity. The precipitation meter at the meteorological station records the actual amount of precipitation, and the computer (reanalysis) calculates the virtual. Because of this, an error may occur.
The difference is especially strong in summer, when convective processes are launched, and models are very bad friends with them. Frontal rainfalls are much easier to calculate than rainfalls after passing through a thunderstorm cell (thundercloud). The error increases in the conditions of the highlands or in the conditions of the sea climate. Therefore, when studying climate, it is desirable to rely only on data from the nearest weather station . We should resort to computer reanalysis only if we simply don’t have analog data.
For example, Yandex Ufa for the year counted 803 mm of precipitation, although in fact the annual rate is 586 mm. Yandex overestimated rainfall by 37%. In Vladivostok, Yandex, on the contrary, underestimated precipitation by 40%. According to Yandex, there is more rainfall in Ufa than in Vladivostok, which is absolute nonsense. I wrote to Yandex at the expense of this problem, but my appeal was ignored. Although, download ready-made arrays by stations and process them, for Yandex, I think, not such a big problem.
The forecast of Yandex on precipitation for 24 hours is much inferior in quality to other forecasting services. For 5 months, the accuracy of Yandex in precipitation for 24 hours was 58%. At intellicast / wunderground - 81%, and at the integrated forecast of the weather center - 80%! Yandex was able to find an interesting feature - the farther from Moscow, the lower the accuracy. In Kurgan, the average accuracy was 45.6%, and in Moscow - 67.8%. The forecast parameters do not agree with each other: precipitation is predicted in case of clear anticyclonic weather.
I strongly advise against using Yandex as a baseline or sole source of forecast.
Main image: insidecorp.ru
GFS synoptic maps - meteoweb.ru/alter/prec.php
Test conditions and control forecasts
The main goal of the study is to understand the quality of precipitation forecast for Yandex in comparison with other predictive services. I draw your attention to the fact that the emphasis was on predicting the phenomenon, and not on the precipitation phase or their quantity.
The data of intellicast.com, which in January combined with the wundeground service , were used as control forecasts, but this had no effect on the forecasts, only the interface changed. The second control forecast is a comprehensive forecast from the site of the methodical cabinet of the Hydrometeorological Center of Russia. I told about this forecast in the previous publication.
')
Forecast intercepted in manual mode. The evaluation period is a meteorological day, that is, from about 8–9 am to 8–21 pm. I did not take night precipitation into account. The forecasts were estimated from November 3, 2018 to March 31, 2019, that is, for the period of time that the climatic winter falls on in the European part of Russia and part of the Urals, and the effect of convection is absent.
Yandex forecasts were also captured only for the daytime. At the same time, I tried to correlate forecasts from the site with those forecasts that Alice sent in push notifications. Later it turned out that the forecasts do not agree with each other, but we'll talk about this later.
The cities were chosen so that I could capture part of the south of the Urals and the main part of the European territory of Russia. I drew a line from Kurgan to Moscow, and chose cities on this site with approximately equal intervals. These are Kurgan, Ufa, Kazan and Moscow. The average distance between the points is 591 km.
As an assessment methodology, the Manual on short-term general-purpose weather forecasts was used, which is used in Roshydromet to estimate forecasts up to 72 hours. As I said, it was important for me to assess the fact of precipitation itself, and not their quantity or phase. Therefore, I made two parameters in the tables: dry (without precipitation) or precipitation.
If the site predicted any precipitation at any hour of the day, then the forecast included a gradation of “Precipitation”, if there was no precipitation in the forecast, then “Sukho”.
Official records of Roshydromet stations were used as actual information. If the precipitation gauge at the meteorological station recorded at least 0.1 mm in 12 hours, the status of “Precipitation” fit into the table. Further, prognostic and actual grades correlated with each other. I cite the numbers of weather stations that were used in the study: 28661 (Kurgan), 28722 (Ufa), 27595 (Kazan), 27612 (Moscow, VDNH).
For example, Yandex predicts light snow during the day, and the meteorological station does not record precipitation, then a prediction of 0% is set for the forecast. If precipitation is in any quantity, then 100%. Thus, all forecasts were placed on equal evaluation terms.
For transparency research, publish a link to the original table with all the data . I also tried not only to evaluate the forecasts, but also to correlate them with zero-time synoptic maps, which made it possible to identify those synoptic situations in which Yandex gives the lowest results of justification.
Research results
Initially, I hypothesized that Yandex will show similar results with other sites, or they will be slightly below the level of competitors. I had to say goodbye to this hypothesis at the end of November, when it turned out that the accuracy of Yandex was comparable to a coin toss. The results were at the level of random forecasts and ranged from 50 to 60%.
For 5 months, the accuracy of Yandex in precipitation for 24 hours was 58%. At intellicast / wunderground - 81%, and at the integrated forecast of the weather center - 80%!
Yandex was able to find an interesting feature - the farther from Moscow, the lower the accuracy. In Kurgan, the average accuracy was 45.6%, and in Moscow - 67.8%. The competitors' forecasts ranged from 75% to 85%, which looks very decent for the winter season.
The main problem of Yandex in the winter time is false precipitation. Yandex predicts rainfall, and other services expect dry weather, a typical situation. The saddest thing is that Yandex predicts rainfall even in those areas over which there is an extensive anticyclone. For example, on November 12, Yandex predicted precipitations for Kazan all day long, but they should not have been, because the anticyclone center was passing over the region, there was clear weather and there could be no rainfall physically.
A similar situation repeated on December 20, but already in Kurgan. Again an anticyclone was established over the city, but Yandex persistently gave precipitations. There were a lot of such strange situations, I just gave the most vivid examples.
Another problem turned out to be with Alice and her push-notifications, which she sends through the main application “Yandex”. For example, on January 3, it was written on the site that there would be no precipitation on January 4, and Alice sent me: “Snow is beautiful. A little snow tomorrow. ” For all 5 months there were only a few such cases, but the fact of disagreement between the services leads to different thoughts.
Of course, most often Alice reported the same forecast, which was listed on the site, but there was no precipitation. The error of “Meteum” is transmitted to the work of all other services and something must be done about it.
Climate data
The precipitation forecast for 24 hours was sorted out, but I would like to indicate another important problem that is present in the work of the Yandex service. Weather".
In the previous article, I already wrote that Yandex publishes average monthly rainfall in all cities. The problem is the data source. The site indicates the source of NOAA, I can assume that Yandex is simply using data from the American computer reanalysis CFSR. Reanalysis is suitable for studying temperature, but it is extremely poor for studying precipitation. The computer model reproduces poorly actual rainfall, especially of low intensity. The precipitation meter at the meteorological station records the actual amount of precipitation, and the computer (reanalysis) calculates the virtual. Because of this, an error may occur.
The difference is especially strong in summer, when convective processes are launched, and models are very bad friends with them. Frontal rainfalls are much easier to calculate than rainfalls after passing through a thunderstorm cell (thundercloud). The error increases in the conditions of the highlands or in the conditions of the sea climate. Therefore, when studying climate, it is desirable to rely only on data from the nearest weather station . We should resort to computer reanalysis only if we simply don’t have analog data.
For example, Yandex Ufa for the year counted 803 mm of precipitation, although in fact the annual rate is 586 mm. Yandex overestimated rainfall by 37%. In Vladivostok, Yandex, on the contrary, underestimated precipitation by 40%. According to Yandex, there is more rainfall in Ufa than in Vladivostok, which is absolute nonsense. I wrote to Yandex at the expense of this problem, but my appeal was ignored. Although, download ready-made arrays by stations and process them, for Yandex, I think, not such a big problem.
Conclusion
The forecast of Yandex on precipitation for 24 hours is much inferior in quality to other forecasting services. For 5 months, the accuracy of Yandex in precipitation for 24 hours was 58%. At intellicast / wunderground - 81%, and at the integrated forecast of the weather center - 80%! Yandex was able to find an interesting feature - the farther from Moscow, the lower the accuracy. In Kurgan, the average accuracy was 45.6%, and in Moscow - 67.8%. The forecast parameters do not agree with each other: precipitation is predicted in case of clear anticyclonic weather.
I strongly advise against using Yandex as a baseline or sole source of forecast.
Main image: insidecorp.ru
GFS synoptic maps - meteoweb.ru/alter/prec.php
Source: https://habr.com/ru/post/447456/
All Articles