How to find a place on earth and not get to the Yandex counter
... this story began a long time ago in a far-away country of Cracovia, whose inhabitants blithely lived their lives and did not know ...
But I myself am a local, and today I will tell you a terrible story about what prevented sleep (for me personally) for many years. And these are not taxes (everything is fine with them), this is Yandex.Maps geocoder !
It is the geocoder that will tell you where the wonderful country of Cracovia is on the map. And it is he who will be the main character of this story, the outset of which was described in a completely different book - in the ancient folio Yandex.Maps API User Agreement . Legend has it that there is a limit on the number of requests to the geocoding function. The maximum allowed to do per day is no more than 25,000 requests to HTTP and JS geocoder per day. Or oatmeal, sir.
And just the other day this rule has turned from ordinary words into a threat to the performance of the site. The API Maps Club published a message about mass bans of various services that exceed limits.
')
What to do? For clarity, we will get a piano out of the bushes - 8 years ago, the project “eCity” was lit up on Habré - “Map of interesting places”. All these years I continue to work on it.
Some of the places that users drew on the map had only coordinates on the map - for them it was necessary to find out the address. Other places found in various sources, on the contrary, only the address was known. So, it was necessary to translate it into coordinates for display on the map.
Moreover, the translation of coordinates to the address and back had to be repeated regularly!
In addition, tens of thousands of people visited the e-Neighborhood website every day who searched for places, which again led to the use of a geocoder.
In fact, everything could not be so bad if it were not for laziness and faith in the MIRACLE! See:
As a result of a simple operation, one or more points will appear on the map and exactly in the right place. Isn't it a miracle?
Blinded by the ease of use of the JS geocoder, many developers programmed their “cartographic solutions” this way. Displaying ATMs, post offices, your points of sale, or any other “organizations” associated with an address is really easier if you only work with addresses.
The texts will be swallowed up by the API, somewhere in the depths they will magically digest - and the labels will appear on the map in the right places. I will even tell you more - many people do not see the difference between assigning a location to a label with numerical coordinates or a legal address.
The trouble is that you have to pay for all the magic! And still it is necessary to go to the handle and to rise on the counter. And if you have one page, there are 10 tags on it, and 2,500 people per day - it means you have rested against the limit.
That is what prevented me from sleeping the last few days. One of the parts of the "e-Neighbors" is the materialization of KLADR (cities, streets, addresses) - postindex.esosedi.ru . This database is “about addresses” and, of course, it is described by addresses (text) and often does not contain coordinates. Even now, many of the villages of our large Motherland are not geocoded (remember about Cracovia). So the coordinates are secondary.
Each time you visit a page of a city or street, a map is shown. At the same time, a request is sent to translate a known address (Timur Frunze, 11k2) into coordinates. By the way, geocoding is not instantaneous, and if you have a lot of points, it can slow down the loading of the map.
Everything would be fine, but 40,000 post offices, 211,000 settlements, a million streets, 55,000 visitors yesterday ... That's enough for two daily limits with a tail. But for many years, this is exactly what worked. Why all?
We will not wait for the evil robot - we will solve the problem. Fortunately, there is a solution, and not one, but two.
Modify the backend so that it is where to save the coordinates. And that means making a normal admin panel, planting a girl who, based on a visual inspection of the “answer” of the geocoder, will display the TTL data, move the icons, reaching the centimeter mark placement accuracy ...
Pros: high-tech, reliable, beautiful girl in the office.
Minuses: a backend is required, a database is required, it is not entirely clear what to do with a user search; it is not quite understood how the wife will react.
Make your search! Such that something in itself retains (ie, a caching proxy), is out of the box, to cope with the same simultaneous requests, and so on.
In general, the idea is very simple: go to some kind of your own pen, check the status of the cache there and, if you don’t have the necessary data, send a request to the HTTP geocoder Yandex.Maps.
For many large projects, the right decision will be to write something of your own. Maybe even deploy your geocoder completely, for example, based on OSM.
But I know from experience that for the majority the most profitable option is to take something ready with github. By tradition, I give a link to the geocode-tool solution to the problem from dimik . Dima once worked Yandex. Maps and can not stop.
In the general case, the geocode-tool is the server part (“pen”), a web-interface with statistics, and a module for Yandex.Maps .
Minuses: it is required to launch one more service on the server, besides on the node. There will be no girl in the office.
Pros: All work has been reduced to a turnip jump and adding a couple of lines on the client (well, almost).
1000 people who will visit 10 pages of an abstract store, where on each page there is a location map, 2 points (addresses) on each page - these are already 20,000 requests, which is dangerously close to the limit. Suddenly, one day you decide to buy ads, and the site will be visited by a few more people?
How did this end with us, on the “e-Neighbor”:
That is - do not forget that the geocoder is not an artificial mind - it can produce garbage.
A geocoder is a simple mechanism for searching the database, which means it can give a little bit of the wrong object that you want. For example, because there is actually no searchable (remember Cracovia).
It's more fun than it seems. But! I! Finally! I can sleep well. What I advise you.
Good geo-sharing.
PS: In fact, I work on “eCity” in my spare time. And the main work is in Yandex.Maps. And I warned you.
PPS: All pictures from fevrony .
But I myself am a local, and today I will tell you a terrible story about what prevented sleep (for me personally) for many years. And these are not taxes (everything is fine with them), this is Yandex.Maps geocoder !
A geocoder is one of the HTTP services of Yandex.Map that receives a textual representation of the address in the request and returns the objects found on its basis in the response. Or vice versa: receiving coordinates and responding with the address.
It is the geocoder that will tell you where the wonderful country of Cracovia is on the map. And it is he who will be the main character of this story, the outset of which was described in a completely different book - in the ancient folio Yandex.Maps API User Agreement . Legend has it that there is a limit on the number of requests to the geocoding function. The maximum allowed to do per day is no more than 25,000 requests to HTTP and JS geocoder per day. Or oatmeal, sir.
And just the other day this rule has turned from ordinary words into a threat to the performance of the site. The API Maps Club published a message about mass bans of various services that exceed limits.
')
What to do? For clarity, we will get a piano out of the bushes - 8 years ago, the project “eCity” was lit up on Habré - “Map of interesting places”. All these years I continue to work on it.
Some of the places that users drew on the map had only coordinates on the map - for them it was necessary to find out the address. Other places found in various sources, on the contrary, only the address was known. So, it was necessary to translate it into coordinates for display on the map.
Moreover, the translation of coordinates to the address and back had to be repeated regularly!
According to the rules of using the service, the response of the geocoder cannot be saved to the database forever, but it can be cached for up to 30 days. The point is not that Yandex does not want to share data, just the quality of data and coverage is constantly changing, but for me, as the creator of the service, it was important that the latest information was available to the user. By the way, Krakow still does not find it.
In addition, tens of thousands of people visited the e-Neighborhood website every day who searched for places, which again led to the use of a geocoder.
In fact, everything could not be so bad if it were not for laziness and faith in the MIRACLE! See:
ymaps.geocode(' .12').then(function(res){myMap.geoObjects.add(res.geoObjects);}); // ymaps.geoQuery(ymaps.geocode('')).addToMap(myMap); As a result of a simple operation, one or more points will appear on the map and exactly in the right place. Isn't it a miracle?
Blinded by the ease of use of the JS geocoder, many developers programmed their “cartographic solutions” this way. Displaying ATMs, post offices, your points of sale, or any other “organizations” associated with an address is really easier if you only work with addresses.
The texts will be swallowed up by the API, somewhere in the depths they will magically digest - and the labels will appear on the map in the right places. I will even tell you more - many people do not see the difference between assigning a location to a label with numerical coordinates or a legal address.
The trouble is that you have to pay for all the magic! And still it is necessary to go to the handle and to rise on the counter. And if you have one page, there are 10 tags on it, and 2,500 people per day - it means you have rested against the limit.
Previously, when crossing the border of 25,000 requests per day, a letter could come from a manager from Yandex explaining what you are doing wrong. Now come the evil robot with plusmetom - and all.
That is what prevented me from sleeping the last few days. One of the parts of the "e-Neighbors" is the materialization of KLADR (cities, streets, addresses) - postindex.esosedi.ru . This database is “about addresses” and, of course, it is described by addresses (text) and often does not contain coordinates. Even now, many of the villages of our large Motherland are not geocoded (remember about Cracovia). So the coordinates are secondary.
Each time you visit a page of a city or street, a map is shown. At the same time, a request is sent to translate a known address (Timur Frunze, 11k2) into coordinates. By the way, geocoding is not instantaneous, and if you have a lot of points, it can slow down the loading of the map.
Everything would be fine, but 40,000 post offices, 211,000 settlements, a million streets, 55,000 visitors yesterday ... That's enough for two daily limits with a tail. But for many years, this is exactly what worked. Why all?
We will not wait for the evil robot - we will solve the problem. Fortunately, there is a solution, and not one, but two.
The first solution is the right one.
Modify the backend so that it is where to save the coordinates. And that means making a normal admin panel, planting a girl who, based on a visual inspection of the “answer” of the geocoder, will display the TTL data, move the icons, reaching the centimeter mark placement accuracy ...
Pros: high-tech, reliable, beautiful girl in the office.
Minuses: a backend is required, a database is required, it is not entirely clear what to do with a user search; it is not quite understood how the wife will react.
The second solution is cheap and angry.
Make your search! Such that something in itself retains (ie, a caching proxy), is out of the box, to cope with the same simultaneous requests, and so on.
In general, the idea is very simple: go to some kind of your own pen, check the status of the cache there and, if you don’t have the necessary data, send a request to the HTTP geocoder Yandex.Maps.
For many large projects, the right decision will be to write something of your own. Maybe even deploy your geocoder completely, for example, based on OSM.
But I know from experience that for the majority the most profitable option is to take something ready with github. By tradition, I give a link to the geocode-tool solution to the problem from dimik . Dima once worked Yandex. Maps and can not stop.
In the general case, the geocode-tool is the server part (“pen”), a web-interface with statistics, and a module for Yandex.Maps .
Minuses: it is required to launch one more service on the server, besides on the node. There will be no girl in the office.
Pros: All work has been reduced to a turnip jump and adding a couple of lines on the client (well, almost).
If you have a map with at least a location map, and geocoding is used (check), and it also seems to you that it runs frequently, connect a proxy server to yourself. If you have an address transformation on every page, use the server geocoder or set the coordinates directly.
1000 people who will visit 10 pages of an abstract store, where on each page there is a location map, 2 points (addresses) on each page - these are already 20,000 requests, which is dangerously close to the limit. Suddenly, one day you decide to buy ads, and the site will be visited by a few more people?
How did this end with us, on the “e-Neighbor”:
- Connecting the local cache made it possible to speed up the page by an average of 150ms - access to the geocoder under the conditions of mass upload of pictures and other advertising is not a quick matter.
When generating the page code, the geocoder's cache twitches at the beginning, bypassing the original service, and only if there is no answer is the function call added to the client. PS: there is again a server version of this case on github. - Finally, statistics appeared on the requests!
- After the appearance of the administrative division ( article on Habré ) and the normal cache of the geocoder, it became possible after the fact to check the correctness of the work of both.
That is - do not forget that the geocoder is not an artificial mind - it can produce garbage.
As is well known in the south of Moscow, there are four ( 1 , 2 , 3 , 4 ) Svitino and all are approximately in one place, but two of them are in Moscow, one in the Moscow region, and one more in Kaluga.
A geocoder is a simple mechanism for searching the database, which means it can give a little bit of the wrong object that you want. For example, because there is actually no searchable (remember Cracovia).
It's more fun than it seems. But! I! Finally! I can sleep well. What I advise you.
Good geo-sharing.
PS: In fact, I work on “eCity” in my spare time. And the main work is in Yandex.Maps. And I warned you.
PPS: All pictures from fevrony .
Source: https://habr.com/ru/post/263863/
All Articles