How do we take screenshots of pages
Hello!
This is our first article on Habré, and in it we would like to tell you about how we make beautiful screenshots of pages for our service and how we came to this, which rakes we attacked.
This service is extremely important to us in order to display updates. And the faster and closer to reality (read - flash support) it works, the more pleasant it is for users.
')
After studying the Internet, the following options were found:
In general, having looked at it and not seeing the ideal, our first attempt to implement the most authentic screenshots was the launch of Firefox under Xvfb , and the screenshot of the entire screen for 30 seconds.
Xvfb is an indispensable utility for such purposes. This is a virtual framebuffer, which allows you not to put full X on the server, and run the "displays" with any resolution. Well, the applications in them :)
The method was, of course, imperfect. Firstly, firefox turned out to be stubborn and all the time he strove to slip a window for updating himself or plug-ins. Secondly, sometimes I just crashed screenshots (a monochrome picture was obtained) or hung. Thirdly, taking a screenshot strictly after 30 seconds is clearly suboptimal - if the site was loaded earlier, there is nothing to wait :) And, of course, he ate the memory.
The next step we tried was to write our own utility on Qt - in order to be able to edit it for ourselves, because after all, the service of making screenshots is really very important for us. Qt is not for nothing that the authors of most of these programs use it - there is a modern engine and a convenient API. There are fewer problems, but they remain - sometimes the utility hangs, sometimes it makes a broken screenshot ... And for some reason it did not work on the combat servers with a flash.
The developer of this utility at some point suggested that we switch to PhantomJS - which we used. The situation again became somewhat better, but not perfect. It also sometimes hangs, and there are links that always cause an error, as well as cases of sporadic failures. The flash did not work either, showed black rectangles, which in principle already suited us more or less.
But PhantomJS at the expense of being able to perform js on the page allowed us to retrieve the page description from the DOM (from meta tags or the secret algorithm directly from the text), the title, and potentially do any interesting things, for example, to define pages that prohibit showing yourself in the frame, which is also quite a critical feature for us. HTML can be such a mess that it can be a pain to work with it on the server; Browser engines have been accustomed to endure any bullying for decades.
So we would have lived with him for a long time, but when we began to add a fair number of links to us, Phantom pleased us with something new - working in parallel several copies of the program, processing links with a flash, SUDDENLY began to take it off, but, let's say, in one screenshot I got a video from another link, and in the second screenshot there was a combination of two videos at once. In short, the blood-intestine-dismemberment.
Hell, I thought, and remembered that Google Chrome generally has a built-in flash player, and it was already clear that all self-written programs suffered from a lack of reliability. Chrome is hard to blame for this :) In addition, it is extremely easy to write extensions on pure js, which could do everything the same as PhantomJS, and even more.
He also had all the magic command line keys that allow him to run without an interface and without any annoying suggestions. He also knew how to write in STDERR, and work in incognito mode, which ensures the most repetitive results of screenshots. We have become available all the power of CSS-selectors and we have earned a flash .
In general, Google Chrome fulfilled all our erotic dreams ;)
Actually the screenshots are taken with a snapshot of the entire screen in which the browser works, well, plus some special cases for special, magical sites.
In a sense, history has made a spiral revolution - we started with a browser, and finished with a browser. Just working on this task with Chrome turned out to be much easier. Yes, it is possible to write an extension under Firefox that will probably be able to do the same thing that we do with Chrome - but it will be more difficult. After writing a toolbar for Firefox, I guarantee it :)
And he works with memory more carefully. By the way, Chrome even has the ability to take screenshots without external utilities - they can be received in the form of a data-uri, but, ironically, in this mode the flash is not removed :)
So, here is a small checklist, how to get your own screenshots service, avoiding long trial and error:
Do not forget the steps "..." and "Profit!";)
In the end, I think we got a solution that surpasses all commercial analogues that I have seen.
This is our first article on Habré, and in it we would like to tell you about how we make beautiful screenshots of pages for our service and how we came to this, which rakes we attacked.
This service is extremely important to us in order to display updates. And the faster and closer to reality (read - flash support) it works, the more pleasant it is for users.
')
After studying the Internet, the following options were found:
- 1. Sites
- BrowserShots - apparently honored veteran. Free, supports a bunch of browsers, for money can take screenshots with a higher priority than without it. The joy lies in the fact that it opensource, the source - in Python. And the sadness is that the screenshots in most of the browsers simply do not work, the flash is not supported, some of the screenshots that seem to have been broken. Focused on web designers.
- WebShots Pro - also from the first page of Google. They have api, which is already interesting, the cost of which depends on the volume, size of pictures and the availability of service tags. But - you will not believe, boys - they make screenshots using IE! In general, everything is clear with them. Well, in fact, they look dubious beyond that.
- thumbalizr beta - promising guys, affordable api, paid service is to turn off the watermark, but do not support flash.
- url2png - all previous sites, except to some extent thumbalizr, look, frankly, razdolbayski-pofigistichno. And it seems to me that they work the same way. url2png leaves a very pleasant impression - though, again, it does not remove the flash and costs more than others.
- websnapr - well, half to half, nothing special.
- 2. Programs
- kthml2png - deserved old man. Almost not supported. In general, we did not try it.
- CutyCapt is a more current program. But - "CutyCapt has a number of known quirks." Sometimes he simply does not manage to remove the screen. Making friends with a flash is difficult.
- PhantomJS - as far as I know, the most active of these projects. A lot of opportunities, not just screenshots. You can execute Javascript in the context of the page, for example. But the position of the author about the plug-ins (flash including) is as follows - it should work, but I don’t want to guarantee anything and I won’t. Interesting problems with this will be described below :)
In general, having looked at it and not seeing the ideal, our first attempt to implement the most authentic screenshots was the launch of Firefox under Xvfb , and the screenshot of the entire screen for 30 seconds.
Xvfb is an indispensable utility for such purposes. This is a virtual framebuffer, which allows you not to put full X on the server, and run the "displays" with any resolution. Well, the applications in them :)
The method was, of course, imperfect. Firstly, firefox turned out to be stubborn and all the time he strove to slip a window for updating himself or plug-ins. Secondly, sometimes I just crashed screenshots (a monochrome picture was obtained) or hung. Thirdly, taking a screenshot strictly after 30 seconds is clearly suboptimal - if the site was loaded earlier, there is nothing to wait :) And, of course, he ate the memory.
The next step we tried was to write our own utility on Qt - in order to be able to edit it for ourselves, because after all, the service of making screenshots is really very important for us. Qt is not for nothing that the authors of most of these programs use it - there is a modern engine and a convenient API. There are fewer problems, but they remain - sometimes the utility hangs, sometimes it makes a broken screenshot ... And for some reason it did not work on the combat servers with a flash.
The developer of this utility at some point suggested that we switch to PhantomJS - which we used. The situation again became somewhat better, but not perfect. It also sometimes hangs, and there are links that always cause an error, as well as cases of sporadic failures. The flash did not work either, showed black rectangles, which in principle already suited us more or less.
But PhantomJS at the expense of being able to perform js on the page allowed us to retrieve the page description from the DOM (from meta tags or the secret algorithm directly from the text), the title, and potentially do any interesting things, for example, to define pages that prohibit showing yourself in the frame, which is also quite a critical feature for us. HTML can be such a mess that it can be a pain to work with it on the server; Browser engines have been accustomed to endure any bullying for decades.
So we would have lived with him for a long time, but when we began to add a fair number of links to us, Phantom pleased us with something new - working in parallel several copies of the program, processing links with a flash, SUDDENLY began to take it off, but, let's say, in one screenshot I got a video from another link, and in the second screenshot there was a combination of two videos at once. In short, the blood-intestine-dismemberment.
Hell, I thought, and remembered that Google Chrome generally has a built-in flash player, and it was already clear that all self-written programs suffered from a lack of reliability. Chrome is hard to blame for this :) In addition, it is extremely easy to write extensions on pure js, which could do everything the same as PhantomJS, and even more.
He also had all the magic command line keys that allow him to run without an interface and without any annoying suggestions. He also knew how to write in STDERR, and work in incognito mode, which ensures the most repetitive results of screenshots. We have become available all the power of CSS-selectors and we have earned a flash .
In general, Google Chrome fulfilled all our erotic dreams ;)
Actually the screenshots are taken with a snapshot of the entire screen in which the browser works, well, plus some special cases for special, magical sites.
In a sense, history has made a spiral revolution - we started with a browser, and finished with a browser. Just working on this task with Chrome turned out to be much easier. Yes, it is possible to write an extension under Firefox that will probably be able to do the same thing that we do with Chrome - but it will be more difficult. After writing a toolbar for Firefox, I guarantee it :)
And he works with memory more carefully. By the way, Chrome even has the ability to take screenshots without external utilities - they can be received in the form of a data-uri, but, ironically, in this mode the flash is not removed :)
So, here is a small checklist, how to get your own screenshots service, avoiding long trial and error:
- Use the job queue. I advise Resque, but it can be * MQ, and Kestrel, even a relational database. The main thing is to work normally with several parallel handlers.
- Run the required number of Xvfb sessions, on different “displays”
- Write a queue handler that will run Google Chrome with the necessary parameters, like DISPLAY =: 1 / opt / google-chrome / google / chrome .... The list of parameters can be found here: http://src.chromium.org/svn/trunk/src/chrome/common/chrome_switches.cc
- Write an extension for Google Chrome, which will monitor the completion of the page load and send a signal to the outside world. Documentation: http://code.google.com/chrome/extensions/getstarted.html
- Set on the task queue the necessary number of handlers, each of which will use its own DISPLAY. Remember to check that DISPLAY is available. This is easy to do with console utilities from the X11 family.
- Take a screenshot by taking a screenshot of the signal from the extension, or use the captureVisibleTab method, but then the flash (and other plugins) will not be removed
Do not forget the steps "..." and "Profit!";)
In the end, I think we got a solution that surpasses all commercial analogues that I have seen.
Source: https://habr.com/ru/post/137847/
All Articles