HeadHunter Job Analysis

Once I wondered what if I tried to analyze the vacancies and compose some tops on them. Find out who pays the most, who is most in demand and much more.

As a data source, I used the well-known HeadHunter. Were collected and processed jobs for May of this year. Only for a month, because the API does not allow getting more.

Data collection

The HeadHunter API has excellent documentation that is located in the repository . Requests should be made to the https://api.hh.ru/ domain with the installed User-Agent , preferably of type _/_ (__) (other User-Agent options sometimes work, but if the server doesn’t like something, it will return an error ).

The logic of the collection is very simple, so I implemented it in bash using cURL and jq . However, I want to share a few nuances.

Pagination

For search of vacancies in various parameters there is endpoint GET /vacancies .

 curl -A 'irenica (https://irenica.com/)' 'https://api.hh.ru/vacancies' 

Search results will be divided into pages, the size of which is the parameter per_page (20 by default and 100 maximum). You can select a specific page by specifying the page parameter (the numbering starts from 0).

In the pages service information returned with job openings, the total number of pages of the result will be indicated.

With this, you can easily search through all pages:

 declare -ii=0 while true; do declare url="https://api.hh.ru/vacancies?per_page=100&page=$i" declare page="$(curl -A 'irenica (https://irenica.com/)' "$url")" #  $page ((i++)) declare -i totalCount=$(echo "$page" | jq '.pages') if ((i >= totalCount)); then break fi done 

Full job details

However, the search results contain only part of the job data. To get everything, you need to make a separate request for the endpoint of the form GET /vacancies/id_ .

Partial data on vacancies are in the items field of search results. At first we will collect from them ID of vacancies:

 declare vacanciesIds="$(echo "$page" | jq -r '.items[].id')" 

Then we will request complete information about the relevant vacancies separately:

 for vacancyId in $vacanciesIds; do declare url="https://api.hh.ru/vacancies/$vacancyId" declare vacancy="$(curl -A 'irenica (https://irenica.com/)' "$url")" #  $vacancy done 

Search limit bypass

The HeadHunter API has one feature - no matter how many are found, a maximum of 2000 will be returned. At the same time, the actual amount found will also be returned to the found field of the search results. Thanks to this, it is possible to know for sure whether you received all the requested data, or if there are losses.

To get around this limitation, I came up with the following. When searching, you can specify the length of time when vacancies of interest were published (via the date_from and date_to , which accept the date in ISO 8601 format). You can take a small interval and sort through all the results with such pieces: after all, the shorter the time interval, the less vacancies were published for it.

It is worth paying attention that the vacancies published only for the last month are returned. Therefore, it makes no sense to set the range anymore.

To iterate over the length of time, the latter is best represented as Unix time:

 declare -i startTime=$(date -d '-1 month' +%s) declare -i endTime=$(date -d now +%s) while ((startTime <= endTime)); do declare -i intervalEnd=$((startTime + 60*60)) declare startTimeIso="$(date -d @$startTime +%FT%T)" declare intervalEndIso="$(date -d @$intervalEnd +%FT%T)" # ... declare url="https://api.hh.ru/vacancies?per_page=100&page=$i&date_from=$startTimeIso&date_to=$intervalEndIso" # ... startTime=$intervalEnd done 

Payroll processing

To collect statistics, it was necessary to group vacancies on certain grounds. At bash, doing this was already problematic, so I used Python.

The logic of the collection is nothing special - the accumulation of data in the associative array, sorting and output to CSV. However, again a few nuances.

Salary fork

It should be noted that the salary is presented in the form of two numbers - the minimum and maximum, and any of them may be absent.

Since for analysis it was necessary to have one number, I decided to use the lower limit, and only if it is absent, the upper one.

 salary = None if vacancy['salary']: if vacancy['salary']['to']: salary = vacancy['salary']['to'] if vacancy['salary']['from']: salary = vacancy['salary']['from'] 

Exchange rates

Salary in a job can be specified in different currencies, and they - have a different rate. The HeadHunter API has endpoint GET /dictionaries containing all the necessary predefined values. Currency rates are presented in the currency field. For convenience, it would be better to put their list in an associative array, where the key is the alphabetic currency code:

 currencies = {} dictionaries = requests.get('https://api.hh.ru/dictionaries').json() for currency in dictionaries['currency']: currencies[currency['code']] = currency['rate'] 

Now, during processing, it will be easy to convert all salaries into one currency:

 salary /= currencies[vacancy['salary']['currency']] 

NDFL accounting

In some vacancies, the salary is indicated before the payment of personal income tax, in some - after. The specific field is indicated by the gross field: it is true in the first case and false in the second.

I decided to transfer all salaries to the option after tax:

 if vacancy['salary']['gross']: salary -= salary * 0.13 

Results analysis

Now is the time to show the numbers.

Remote work

Probably many of those who read this post, would like to work on the remote. But as we see, work from home in our country is not very much quoted yet. Salary is much lower, the number of vacancies is significantly less. And therefore there is less opportunity to choose for the applicant.

And it is rather strange, because in many professions and many firms (by the specific nature of the tasks), the presence of a person in the office is completely unnecessary. But this is an eternal argument.

Applicants with disabilities

However, there is an even smaller category of vacancies - for people with disabilities. And this is completely illogical - even if employers do not want remote workers, but of those who are ready for this, why are there so few who think about people with disabilities? If you do not care that a person is in three time zones, what difference does it make to you, is he able to walk, for example?

Perhaps many of you are familiar with people with disabilities. I, too, and I wondered how difficult it is for them to find a job, and what they can count on.

Students

We all start with something, namely, with a job search, without any experience. I decided to assess the situation with positions open to such candidates.

The number of vacancies is encouraging for quick employment. And I do not know how realistic it is to get the maximum salary, but you can even somehow live by the average figures.

Common top

And now the most interesting thing: who pays the most? Sorted all vacancies found without any filters.

Of course, this is top management. Who would doubt that.

A curious fact: if you pay attention to the average salary in all tables, you can see that it is not that different.

Cleaning woman

And here is the easiest way: why study for 5 years, if you can just wash the office? Below is the result of filtering the top vacancies for the query "cleaning *".

What if you get a job in several offices and come in the evening for a couple of hours for cleaning? So you can live quite luxurious. We will consider it life hacking.

Top by city

Finally, I decided to check the number of open positions in cities. The first places are not surprising, but then there are curious and even unexpected positions.

Repository

All code from the article, with improvements and instructions, is available in the repository .