The pipeline will save you from doubles in the tape

Hey.

Recently andorro wrote about nasty duplicate posts, which with some probability appear in the tape if you subscribe to Habr and GT. There is a great way to solve this problem, using rss and yahoo, and for one - to subscribe to intersecting hubs.

Yahoo Pipes

Pipes are a composition tool to aggregate, manipulate, and mashup content from around the web.

Pipes is a service that gets something at the entrance, does something with it within itself and gives back what happened. You can use csv, rss xml or even dsdjl of another pipe as input. On the way out you can get rss, json, email or widget.
')
For example, you can get the rss fly channel in Tumblr, regularly replace in all img a link to a small picture with a link to a larger one and return the resulting rss.

We are building a pipeline

Register on Yahoo (if needed), go to http://pipes.yahoo.com/ and open the constructor.



On the left is a list of blocks, in the center is the constructor itself, at the bottom is a debugger. For each block there is a description and an example of use. There are many options, but we need Fetch Feed.

Let's add it and, for an example, we will add in it hubs of Windows and Laptops with GT and still Development from Habr.



The log shows that the data is received. Now you can sort them by date (Sort block).



After sorting, it became clear that the posts from the Windows and Laptops hubs quite repeat themselves. This is easily solved by filtering by headers (the Unique block).



We connect the output of the Unique block, save and run it .



I hope this recipe will be useful to you.

Yahoo Pipes
Sample demo