Analyzing market requirements for data scientist

There is a lot of information on the Internet that the data sciencist should know and be able to. But I decided that becoming a data sciencist is necessary immediately, so we will find out the requirements for specialists by analyzing the text of vacancies.





First, we will formulate the task and develop a plan:



Task:

')

View all job vacancies on the market and find out the general requirements indicated in them.



Plan:



1. Collect all the vacancies at the request of Data Scientist in a convenient for processing format

2. Find out frequently encountered in the description of the words and phrases.



For implementation, you need a little knowledge of SQL and Python.

Data collection

Source: hh.ru

At first I thought that you can sparse the site. Fortunately, I found that hh.ru has an API .



For a start, let's write a function that gets a list of job id for analysis. In the parameters, the function receives the search text (here we will send 'Data Scientist') and the search area (according to the api documentation), and returns a list of id. For data we use the function api job search :





For debugging, I sent directly to the API. I recommend using the chrome Postman application for this.



After that you need to get detailed information about each vacancy:





We now have a job list and a function that receives detailed information about each job. We must decide where to write the data. I had two options: save everything to a csv file or create a database. Since it is easier for me to write SQL queries than to analyze in Excel, I selected a database. First you need to create a database and tables in which we will take notes. To do this, we analyze what the API responds to and decide which fields we need.



We paste the api link into Postman, for example, api.hh.ru/vacancies/22285538 , make a GET request and get the answer:





Everything that we do not plan to analyze is deleted from JSON.





On the basis of this JSON we make a DB. It's easy, so I'll leave it out :)



We implement the module of interaction with the database. I used MySQL:





Now we put everything together by adding a main () method to the file.





Changing the text_search and area variables we get different vacancies from different regions.

On this data mining is completed and go to the interesting.

Text analysis

The main inspiration was the article about the search for popular phrases in the series How I met your mother



First, we will receive a description of all vacancies from the database





To work with the text, we will use the nltk package. By analogy with the above article, we write the function for obtaining popular phrases from the text:





We combine all the above methods in the main method and run it:





Perform and see:



li - 2459

/ li - 2459

and - 1297

p - 1225

/ p - 1224

in - 874

strong - 639

/ strong - 620

and - 486

ul - 457

/ ul - 457

from - 415

on - 341

data - 329

data - 313

the - 308

experience - 275

of - 269

for - 254

work - 233



We see that the result got a lot of words that are typical for all vacancies and tags that are used in the description. Remove these words from the analysis. For this we need a list of stop words. Let's form it automatically, having analyzed vacancies from another sphere. I chose "cook", "cleaner" and "fitter".



Let's go back to the beginning and get vacancies for these requests. After that, add a function to get stop words.





We also see the English of and of. Let's do it easier and remove jobs in English.

Make changes to main ():





Now the result looks like this:



data - 329

data - 180

analysis - 157

training - 134

machine - 129

models - 128

areas - 101

algorithms - 87

python - 86

tasks - 82

tasks - 82

development - 77

analysis - 73

construction - 68

methods - 66

will be - 65

statistics - 56

higher - 55

knowledge - 53

learning - 52



Well, this is one word, it does not always reflect the truth. Let's see what the phrases of 2 words show:



machine learning - 119

data analysis - 56

machine learning - 44

data science - 38

data scientist - 38

big data - 34

mathematical models - 34

data mining - 28

machine algorithms - 27

mathematical statistics - 23

will be a plus - 21

statistical analysis - 20

data processing - 18

English - 17

data analysis - 17

including 17

as well - 17

methods of machine - 16

areas of analysis - 15

probability theory - 14

The results of the analysis.

More clearly gives the result of the query in two words, we need to know:

• Machine learning
• Mathematical models
• Data mining
• Python
• Mathematical statistics
• English
• Probability theory

Nothing new, but it was fun :)

Findings.

This is far from the perfect solution.



Errors:



1. It is not necessary to exclude vacancies in English, it is necessary to translate them.

2. Not all stop words are excluded.

3. It is necessary to bring all the words to the basic form (machine -> machine, analysis -> analysis, etc.).

4. Invent a method by which to calculate a more optimal list of stop words. Answer the question "why 200?", "Why the cleaner?".

5. We need to figure out how to analyze the result automatically, to understand that one or two words carry meaning or more.