As I added 30,000 people to the first circle of contacts, they will block this social network in the Russian Federation
Recalling the interviews from his past, he decided to “not go up the hill,” but rather that the mountain itself came to me, or rather, staff members with vacancies. Again, looking for a job and now in a market where real income has become less, and there are more applicants ...
But I didn’t want to waste time on routine search operations and adding social network professionals. After 2000 manual clicks, my search criteria was formalized. Therefore, I wrote a script that saves my time and vision. I do not allow the soulless "robot" to communicate with people. I myself write and answer myself.
')
I was lucky to meet and talk with interesting people from the industry, get hundreds of spam messages and links to hundreds of vacancies. The finished script will not be here, but the publication has a couple of code snippets for webdriver magic and links. You will also find out the latest news about job search in Russia and a description of my experience in social networking.
The way from 160 contacts in the social network to 30,000 contacts took me half a year. Added a small amount of requests per day. First, I was in no hurry. I spent a couple of weeks in the hospital - underwent an ENT operation and a couple of weeks after the hospital, I was recovering. It was summer in the courtyard and you could sunbathe at lunch and ride a bike. At the end of the summer we got to the sea for a couple of weeks. I assembled another self-developed CNC machine on rail guides, with the ability to "steer" from the program in the JVM. I continued to do what I loved, but now it's all 8 hours a day. Programmed in java and groovy, but for his hobby projects. Studied new technologies that are popular in the market.
At the same time, the script worked for me a couple of times a day - I searched and added new contacts. I responded to messages, talked to people and spent no more than half an hour on this.
Hope, as on a platform for publishing news on professional topics, did not justify itself. Hundreds of views of publications, minimal user activity.
More than 30,000 will not add due to artificial network restrictions. Now I have 320 requests from people to add them to contacts, but because of this restriction I cannot add them.
For a long time I was in 1st place in terms of the number of views among my first lap and first place in views of employees of the last place of work. It is verified that this superiority does not give anything useful !!! I have always used a free account and did not see the point of joining the network’s marketing actions “enter your bank card number - the first month is free.”
This is the link from xcore78 in the comments on this article. Here, for some reason, my legible script is identified with spam, although in fact my profile is more relevant to the quality of contacts than IT is than that of so revered LION. And I do not send any messages with goods or services. I sent a request to establish a contact and if they did not answer, go further!
What I liked - you can get interesting messages and learn about the new, when you add an unfamiliar user to your contacts. Well, the most pleasant and unexpected influx of messages was congratulations on your birthday - about 200 incoming!
In general, job offers were moving abroad, but not in the most interesting locations. In our market, many vacancies were offered either in uninteresting subject areas where my expertise and technical knowledge are not particularly needed, or these are vacancies in companies about the internal conditions in which I knew from different people, or the payment did not suit me.
To go through a series of interviews and through the millstones of formal tests and millions of algorithmic tasks in google and companies with a similar competition, I feel sorry for the strength and most likely wasted time. Although I was invited to participate in this exciting process. In addition, the negative experience says that even complex test tasks with writing TDD code during an interview or after it do not guarantee that after this you will be engaged in an interesting job in a profession with technically complex tasks .
In addition to buying the social network Microsoft, which was a surprise to me, the following news shocked even more. Moscow City Court has recognized the legitimate decision to block LinkedIn in Russia. The essence of the claims to the social network is non-compliance with the law “On Personal Data”, which requires that the servers with the data of citizens are located on the territory of Russia. And this news was about the rejection of the appeal of the social network by the previous decision of the Tagansky District Court of Moscow.
The funniest thing about all this is Olga Golodets' comment. She said, judging by the publication in the news, that one of the main employment opportunities for Russians is the portal of Rostrud - the all-Russian job base “Work in Russia”. I first learned of its existence from the news. Do you, as IT professionals, often use it and hear about it?
For obvious reasons, including to avoid habraeffect, I will not distribute the script. But the information that I published on Habré along with my examples from the links to the githab is enough for those who want to automate the routine interaction with almost any web sites.
There is a saying: give a hungry fish - and you feed him for one day. Give him a fishing rod, teach him how to fish - and you feed him for life.
Everything I am talking about is based on open source.
Back to technology again. Previously, I already told about the web driver in Habré “What should we parse the site. Basics webdriver API . I recommend reading this article for a start, but here I will only describe the new information from personal experience about how to automatically load and connect a web driver for chrome to the program, how to scroll the page vertically from the script. And if the “headless” browser of the PhantomJS website is easy to detect and block your account, then with chromium + webdriver you can't do it so easily.
In the webdriver for a real browser, besides the client-side API, there is also a part that makes the browser a puppet of our application in a separate process.
Often it is downloaded manually and indicate the path in the application. But why, if you can do it automatically.
To do this, in the JVM application you need to add the dependency com.github.igor-suhorukov: mvn-classloader: 1.6 .
Depending on what kind of operating system you have, pass as an os value win32, linux64 or mac64. win32 works on 64-bit systems.
Compare with how it was done for PhantomJS:
The code below works for me in chromium and page scrolling is called from the submitRequest method. To scroll the window using javascript and to execute it from the context of the page, you need to cast the web driver type to org.openqa.selenium.JavascriptExecutor. These pages are dynamically loaded when scrolling.
Writing a script with webdriver solves only a small part of what you need to do to search. And the golden fish is still not caught. In addition, life is full of surprises! And what will happen next after blocking the social network, we will find out soon. It is a pity only to lose the opportunity to communicate with professionals from areas far from IT, who still do not know how to use VPN or Tor ...
PS 11/18/2016 Milo, even my Internet provider did not return the money in the absence of the Internet ...
But I didn’t want to waste time on routine search operations and adding social network professionals. After 2000 manual clicks, my search criteria was formalized. Therefore, I wrote a script that saves my time and vision. I do not allow the soulless "robot" to communicate with people. I myself write and answer myself.
')
I was lucky to meet and talk with interesting people from the industry, get hundreds of spam messages and links to hundreds of vacancies. The finished script will not be here, but the publication has a couple of code snippets for webdriver magic and links. You will also find out the latest news about job search in Russia and a description of my experience in social networking.
This is just a saying ...
My experience
The way from 160 contacts in the social network to 30,000 contacts took me half a year. Added a small amount of requests per day. First, I was in no hurry. I spent a couple of weeks in the hospital - underwent an ENT operation and a couple of weeks after the hospital, I was recovering. It was summer in the courtyard and you could sunbathe at lunch and ride a bike. At the end of the summer we got to the sea for a couple of weeks. I assembled another self-developed CNC machine on rail guides, with the ability to "steer" from the program in the JVM. I continued to do what I loved, but now it's all 8 hours a day. Programmed in java and groovy, but for his hobby projects. Studied new technologies that are popular in the market.
At the same time, the script worked for me a couple of times a day - I searched and added new contacts. I responded to messages, talked to people and spent no more than half an hour on this.
Hope, as on a platform for publishing news on professional topics, did not justify itself. Hundreds of views of publications, minimal user activity.
More than 30,000 will not add due to artificial network restrictions. Now I have 320 requests from people to add them to contacts, but because of this restriction I cannot add them.
For a long time I was in 1st place in terms of the number of views among my first lap and first place in views of employees of the last place of work. It is verified that this superiority does not give anything useful !!! I have always used a free account and did not see the point of joining the network’s marketing actions “enter your bank card number - the first month is free.”
For example, there are people in LinkedIn, the so-called LIONs (LinkedIn Open Networkers). These are people who add to themselves everyone, indiscriminately. They have 10-20 thousand contacts each. If you draw all the links in the world in the form of a grid, LIONs will look like thick, thick dots. Thanks to LION before the president of Oracle, you may need not 6 connections, but only 3.
So, when you connect with people, add as many LIONs as possible. Maybe tomorrow you will go to the person (or he to you) to contact with whom you can just thanks to the freshly-added LION!
This is the link from xcore78 in the comments on this article. Here, for some reason, my legible script is identified with spam, although in fact my profile is more relevant to the quality of contacts than IT is than that of so revered LION. And I do not send any messages with goods or services. I sent a request to establish a contact and if they did not answer, go further!
What I liked - you can get interesting messages and learn about the new, when you add an unfamiliar user to your contacts. Well, the most pleasant and unexpected influx of messages was congratulations on your birthday - about 200 incoming!
In general, job offers were moving abroad, but not in the most interesting locations. In our market, many vacancies were offered either in uninteresting subject areas where my expertise and technical knowledge are not particularly needed, or these are vacancies in companies about the internal conditions in which I knew from different people, or the payment did not suit me.
To go through a series of interviews and through the millstones of formal tests and millions of algorithmic tasks in google and companies with a similar competition, I feel sorry for the strength and most likely wasted time. Although I was invited to participate in this exciting process. In addition, the negative experience says that even complex test tasks with writing TDD code during an interview or after it do not guarantee that after this you will be engaged in an interesting job in a profession with technically complex tasks .
Last week's news
In addition to buying the social network Microsoft, which was a surprise to me, the following news shocked even more. Moscow City Court has recognized the legitimate decision to block LinkedIn in Russia. The essence of the claims to the social network is non-compliance with the law “On Personal Data”, which requires that the servers with the data of citizens are located on the territory of Russia. And this news was about the rejection of the appeal of the social network by the previous decision of the Tagansky District Court of Moscow.
The funniest thing about all this is Olga Golodets' comment. She said, judging by the publication in the news, that one of the main employment opportunities for Russians is the portal of Rostrud - the all-Russian job base “Work in Russia”. I first learned of its existence from the news. Do you, as IT professionals, often use it and hear about it?
Software part
For obvious reasons, including to avoid habraeffect, I will not distribute the script. But the information that I published on Habré along with my examples from the links to the githab is enough for those who want to automate the routine interaction with almost any web sites.
The essence of automation ...
There is a saying: give a hungry fish - and you feed him for one day. Give him a fishing rod, teach him how to fish - and you feed him for life.
Everything I am talking about is based on open source.
And this anedot rather refers to proprietary technologies in software.
Socialism: feed the hungry fish.
Capitalism as we draw it: do not feed the hungry fish, and give him a fishing rod.
Capitalism as it really is: not to give a bait, but to sell it on credit, not letting the hungry understand that he still has no access to the fish pond or the right to fish, since both the pond and fish have long belonged to those to whom he now also owes a bait ...
Capitalism as we draw it: do not feed the hungry fish, and give him a fishing rod.
Capitalism as it really is: not to give a bait, but to sell it on credit, not letting the hungry understand that he still has no access to the fish pond or the right to fish, since both the pond and fish have long belonged to those to whom he now also owes a bait ...
Back to technology again. Previously, I already told about the web driver in Habré “What should we parse the site. Basics webdriver API . I recommend reading this article for a start, but here I will only describe the new information from personal experience about how to automatically load and connect a web driver for chrome to the program, how to scroll the page vertically from the script. And if the “headless” browser of the PhantomJS website is easy to detect and block your account, then with chromium + webdriver you can't do it so easily.
Automatic driver installation for chrome
In the webdriver for a real browser, besides the client-side API, there is also a part that makes the browser a puppet of our application in a separate process.
Often it is downloaded manually and indicate the path in the application. But why, if you can do it automatically.
To do this, in the JVM application you need to add the dependency com.github.igor-suhorukov: mvn-classloader: 1.6 .
Depending on what kind of operating system you have, pass as an os value win32, linux64 or mac64. win32 works on 64-bit systems.
String chromedriver = MavenClassLoader.usingCentralRepo().resolveArtifact("com.github.igor-suhorukov:chromedriver:bin:" + os + ":2.24").getFile(); // linux chmod(chromedriver); System.setProperty("webdriver.chrome.driver", chromedriver); Compare with how it was done for PhantomJS:
// com.github.igor-suhorukov:phantomjs-runner:1.1 import com.github.igorsuhorukov.phantomjs.PhantomJsDowloader ... String phantomJsPath = PhantomJsDowloader.getPhantomJsPath() Scrolling windows with dynamic data loading
The code below works for me in chromium and page scrolling is called from the submitRequest method. To scroll the window using javascript and to execute it from the context of the page, you need to cast the web driver type to org.openqa.selenium.JavascriptExecutor. These pages are dynamically loaded when scrolling.
private static void submitRequest(WebDriver driver) throws InterruptedException { ... scroll((JavascriptExecutor) driver, (long) (MIN_SCROLL + Math.random() * SCROLL_INTERVAL)); ... } private static void scroll(JavascriptExecutor javascriptExecutor, long maxScroll) throws InterruptedException { long valPrev = 0; long val = 0; while (valPrev< maxScroll && (val = scrollViewport(javascriptExecutor)) >valPrev){ valPrev = val; Thread.sleep((long) (MIN_TIME+Math.random()*TIME_INTERVAL)); } } private static Long scrollViewport(JavascriptExecutor driver) { return (Long) driver.executeScript("var h=document.documentElement.scrollHeight; window.scrollTo(0, h); return h;"); } What's next?
Writing a script with webdriver solves only a small part of what you need to do to search. And the golden fish is still not caught. In addition, life is full of surprises! And what will happen next after blocking the social network, we will find out soon. It is a pity only to lose the opportunity to communicate with professionals from areas far from IT, who still do not know how to use VPN or Tor ...
PS 11/18/2016 Milo, even my Internet provider did not return the money in the absence of the Internet ...
Source: https://habr.com/ru/post/315136/
All Articles