Crossposting from Twitter to VKontakte using a router
Hello, my name is Eugene, and I am an alcoholic and I love social networks. Due to the absence of any hints of writing talent, I prefer Twitter - its 140 character limit eliminates the need to invent something long. In addition, only for twitter there is a normal java-client, which I used to use on a mobile phone.
On the other hand, most of my acquaintances use VKontakte, and, frankly, I like to listen to music there and read news. It is obvious that it is inconvenient to write yourself in two different social networks, it is necessary to automate it - use crossposting (post transfer).
Further, a brief description of my not very usual way of cross-hosting.
Previously, I used IFTTT to transfer records. It was not very convenient - posts were transferred once every 15 minutes, special characters were encoded ("turned into & quot;), links were transferred abbreviated (t.co) —Vkontakte cursed when trying to follow these links. However, due to recent changes in the Twitter API IFTTT was forced to turn off “recipes" that read tweets. I had to search for a new path. Suddenly ™ I learned that Vkontakte has an official way to import tweets, but for this you have to use the #vk hashtag. it is not always (apparently again with worked out the new Twitter API restrictions on the number of requests), plus there was a problem with links.
Of course, the easiest way would be to use a script that will transfer records. On Habré , they repeatedly wrote about different ways of cross-hosting, but all of these methods rested on my lack of a server. It is not enough to buy paid hosting (albeit inexpensive) for one simple script.
But just recently, crawling on the Internet in search of a new firmware version for my router, it dawned on me that on the basis of my Zyxel Keenetic you can make a small web server! Even before buying, I read about the capabilities of the router, but then it somehow got out of my head. And so I began to implement the conceived project - crossposting from Twitter on Vkontakte using a router.
To begin, talk a little about the possibilities of Zyxel Keenetic. In short, various ready-made modules can be installed quickly, easily and securely on the router (php, lighttpd, dlna, transmission, perl, python, etc.). More detail you can read on the forum .
Before the start of the project, I formed a list of women. The script should be able to:
At first it was necessary to choose the language of the script. There are Perl and Python modules for the router, and I was equally unfamiliar with these languages. I also never came across the Twitter API and Vkontakte API, so first of all I was interested in ready-made examples of working with them. Fortunately, there were enough such scripts in the network. Somehow I accidentally chose Perl :)
Then I proceeded directly to the documentation and examples on the Twitter API. As you know, there are two types - Streaming and REST. The latest changes in the API clearly show Twitter’s desire to transfer developers to the Streaming API. Therefore, it was decided to use Streaming in order to have no problems in the future.
Since I was making the script only for myself, I did not get access token, but took it immediately from the settings page of my Twitter application. For a couple of hours, a simple console application that output tweets was written using the perl module . Everything worked fine on my computer. To check on the router, I installed all the necessary packages (as I thought), launched the script, and ... nothing. Does not work. I installed all the available packages for Perl, did not help. By googling the text of the error, I found out that the problem is most likely due to the absence of some kind of Perl module, as a result, the script cannot work correctly with SSL. As a result, it was decided to leave Perl, and try Python.
')
Download PyCharm , began to learn the syntax. To be honest, after .NET was shocked by the use of indentation as a block notation :) I understood the syntax, found the tweepy module, and pretty quickly my tweet output script was rewritten in Python. I install the python packages on the router, run the script - oh, a miracle! Works! It remains to add side functionality and record posts in Vkontakte.
First, let's deal with the functionality.
Unfolding links. If the tweet contains a link, then Twitter returns us a JSON in which there is a
The tweet text contains
Image acquisition. If a picture is attached to a tweet, then a
A direct link to the image is contained in the
Let's look at the received code to work with Twitter.
First log in and start reading stream
We read the answers, if it is tweet - we process it
Tweet processing method:
VK has a slightly different authorization mechanism, and it will not work just to get access token.
First you need to create a desktop application and take it id. Then create a link of the form
Such a wrapper is used to work with VK API methods. The code is not mine, but I completely forgot where I got it from - I apologize for not indicating the author.
Here vkToken is the received token for VK, method is the name of the method from the list of VK API methods, data is some data for the method.
To attach a picture to a post on a wall, you need to upload it to the user's album. For photos on the wall in VK there is a special album and special methods for working with it. First we get a link to upload photos, then save the picture. The result was the following method:
This method returns the ID of the loaded picture, which can be specified directly when creating a message on the wall:
In this case, if the
I use EyeEm (this is an analog of Instagram, but with normal Privacy Terms), and recently I thought that it would be nice to upload photos from there to VKontakte. Fortunately, EyeEm has a very simple and straightforward API, and in just two lines you can add support for uploading photos from the script to the script.
The method of obtaining links to photos by her ID
Now we find the link to the photo in the tweet, we get its url and also upload it to VK
The finished script can be downloaded from github . Tokens in it, of course, changed :) You will need to get yours.
Now we are setting up a router to support third-party packages according to the instructions on the wiki . I will give here a brief instruction for Windows:
The router is ready to install opkg packages. Their list can be viewed on the website or by
Then, install python with the command:
This script works for me for about six months (the article was written sooo long), most of the errors I corrected. My wall in VC has become much easier to read because of the expanded links and attached photos. I am pleased. I plan to create a couple of Twitter scripts based on a router, such as a twitter bot.
I hope this article is useful to someone. At least to start using the existing router not only for its intended purpose :)
I honestly tried to correct all grammatical errors and typos, but if you notice something - write in a personal.
I also remind you once again that my knowledge of Linux and Python is not far from zero, and I will be happy to hear your tips on how to improve the script.
On the other hand, most of my acquaintances use VKontakte, and, frankly, I like to listen to music there and read news. It is obvious that it is inconvenient to write yourself in two different social networks, it is necessary to automate it - use crossposting (post transfer).
Further, a brief description of my not very usual way of cross-hosting.
A little more background
Previously, I used IFTTT to transfer records. It was not very convenient - posts were transferred once every 15 minutes, special characters were encoded ("turned into & quot;), links were transferred abbreviated (t.co) —Vkontakte cursed when trying to follow these links. However, due to recent changes in the Twitter API IFTTT was forced to turn off “recipes" that read tweets. I had to search for a new path. Suddenly ™ I learned that Vkontakte has an official way to import tweets, but for this you have to use the #vk hashtag. it is not always (apparently again with worked out the new Twitter API restrictions on the number of requests), plus there was a problem with links.
Of course, the easiest way would be to use a script that will transfer records. On Habré , they repeatedly wrote about different ways of cross-hosting, but all of these methods rested on my lack of a server. It is not enough to buy paid hosting (albeit inexpensive) for one simple script.
But just recently, crawling on the Internet in search of a new firmware version for my router, it dawned on me that on the basis of my Zyxel Keenetic you can make a small web server! Even before buying, I read about the capabilities of the router, but then it somehow got out of my head. And so I began to implement the conceived project - crossposting from Twitter on Vkontakte using a router.
Moscow was not built in a day
To begin, talk a little about the possibilities of Zyxel Keenetic. In short, various ready-made modules can be installed quickly, easily and securely on the router (php, lighttpd, dlna, transmission, perl, python, etc.). More detail you can read on the forum .
Before the start of the project, I formed a list of women. The script should be able to:
- transfer tweets immediately after they appear or when the script is run via cron
- do not carry answers, retweets and mentions
- expand t.co short links in tweet
- If possible, copy attached images (for example, Instagram)
At first it was necessary to choose the language of the script. There are Perl and Python modules for the router, and I was equally unfamiliar with these languages. I also never came across the Twitter API and Vkontakte API, so first of all I was interested in ready-made examples of working with them. Fortunately, there were enough such scripts in the network. Somehow I accidentally chose Perl :)
Then I proceeded directly to the documentation and examples on the Twitter API. As you know, there are two types - Streaming and REST. The latest changes in the API clearly show Twitter’s desire to transfer developers to the Streaming API. Therefore, it was decided to use Streaming in order to have no problems in the future.
Since I was making the script only for myself, I did not get access token, but took it immediately from the settings page of my Twitter application. For a couple of hours, a simple console application that output tweets was written using the perl module . Everything worked fine on my computer. To check on the router, I installed all the necessary packages (as I thought), launched the script, and ... nothing. Does not work. I installed all the available packages for Perl, did not help. By googling the text of the error, I found out that the problem is most likely due to the absence of some kind of Perl module, as a result, the script cannot work correctly with SSL. As a result, it was decided to leave Perl, and try Python.
')
Perl is dead, long live Python
Download PyCharm , began to learn the syntax. To be honest, after .NET was shocked by the use of indentation as a block notation :) I understood the syntax, found the tweepy module, and pretty quickly my tweet output script was rewritten in Python. I install the python packages on the router, run the script - oh, a miracle! Works! It remains to add side functionality and record posts in Vkontakte.
First, let's deal with the functionality.
Unfolding links. If the tweet contains a link, then Twitter returns us a JSON in which there is a
"entities" element containing "urls" . It looks like this: "entities": { "hashtags":[], "user_mentions":[], "urls":[{ "indices":[0,21], "display_url":"dev.twitter.com/terms/display-\u2026", "url":"https://t.co/Ed4omjYs", "expanded_url":"https://dev.twitter.com/terms/display-guidelines" }] } The tweet text contains
"url" , so to expand the link, you just need to replace the "url" value with the value of "expanded_url" in the text.Image acquisition. If a picture is attached to a tweet, then a
"media" element will be added to the "entities" "media" , like this: "entities": { "hashtags":[], "user_mentions":[], "urls":[], "media":[{ "type":"photo", "media_url":"http://p.twimg.com/A7kqLpACEAAUlwz.png", "indices":[0,20], "sizes": { "large":{"resize":"fit","h":454,"w":584}, "small":{"resize":"fit","h":264,"w":340}, "thumb":{"resize":"crop","h":150,"w":150}, "medium":{"resize":"fit","h":454,"w":584} }, "display_url":"pic.twitter.com/XzDoEpH9", "media_url_https":"https://p.twimg.com/A7kqLpACEAAUlwz.png", "url":"http://t.co/XzDoEpH9", "expanded_url":"http://twitter.com/TestTwVK/status/268292032273977344/photo/1", "id":268292032278171648, "id_str":"268292032278171648" }] }, A direct link to the image is contained in the
"media_url" element. I had to save the image to a temporary file on a local disk, did not find a way to upload it to VC directly from the twitter server.Let's look at the received code to work with Twitter.
First log in and start reading stream
url = "https://userstream.twitter.com/1.1/user.json" param = {"delimited":"length", "with":"user"} header = {} auth = tweepy.OAuthHandler(consumer_key, consumer_secret) auth.set_access_token(access_key, access_secret) auth.apply_auth(url, "POST", header,param) logging.info('Twitter authorization successful') # req = urllib2.Request(url) req.add_header("Authorization", header["Authorization"]) r = urllib2.urlopen(req, urllib.urlencode(param), 90) We read the answers, if it is tweet - we process it
while True: # length = "" while True: c = r.read(1) if c == "\n": break if c == "": raise Exception length += c length = length.strip() if not length.isdigit(): continue # tweet = json.loads(r.read(int(length))) # - if "user" in tweet and "text" in tweet and "created_at" in tweet and "id" in tweet: handleTweet(tweet) Tweet processing method:
# text = tweet["text"] #, , if not text.startswith('@') and not "@zzeneg" in text: # urls = tweet["entities"]["urls"] # for url in urls: text = text.replace(url["url"], url["expanded_url"]) # EyeEm - id # - if "media" in tweet["entities"]: photo = tweet["entities"]["media"][0] text = text.replace(photo["url"], "") Work with VK
VK has a slightly different authorization mechanism, and it will not work just to get access token.
First you need to create a desktop application and take it id. Then create a link of the form
oauth.vk.com/authorize?client_id={ID}&scope=wall,photos,offline&redirect_uri=http://oauth.vk.com/blank.html&display=page&response_type=token oauth.vk.com/authorize?client_id={ID}&scope=wall,photos,offline&redirect_uri=http://oauth.vk.com/blank.html&display=page&response_type=token , where {ID} is the application ID. This link also shows the permissions for the application - wall for posting on the wall, photos for uploading the image and offline for the token to be eternal. We copy this link in the browser. VC will ask us about granting rights to this application, and then redirect to the URL, which will contain the token.Such a wrapper is used to work with VK API methods. The code is not mine, but I completely forgot where I got it from - I apologize for not indicating the author.
def vkMethod(method, data={}): url = 'https://api.vk.com/method/%s.json' % (method) data.update({'access_token': vkToken}) response = requests.post(url, data).json if 'error' in response: print 'VK API error: %s' % (response['error']['error_msg']) return response Here vkToken is the received token for VK, method is the name of the method from the list of VK API methods, data is some data for the method.
To attach a picture to a post on a wall, you need to upload it to the user's album. For photos on the wall in VK there is a special album and special methods for working with it. First we get a link to upload photos, then save the picture. The result was the following method:
def uploadPhoto(fileUrl): # url response = vkMethod('photos.getWallUploadServer') uploadUrl = response['response']['upload_url'] # urllib.urlretrieve(fileUrl,'temp.jpg') # files = {'photo': open('temp.jpg', 'rb')} response = requests.post(uploadUrl, files = files).json # response = vkMethod('photos.saveWallPhoto', response) # id return response['response'][0]['id'] This method returns the ID of the loaded picture, which can be specified directly when creating a message on the wall:
attachments = uploadPhoto(photo["media_url"]) vkMethod('wall.post', {'message': text,'attachments':attachments}) In this case, if the
attachments is an empty line - then the message will still appear on the wall, without attachments.Work with EyeEm
I use EyeEm (this is an analog of Instagram, but with normal Privacy Terms), and recently I thought that it would be nice to upload photos from there to VKontakte. Fortunately, EyeEm has a very simple and straightforward API, and in just two lines you can add support for uploading photos from the script to the script.
The method of obtaining links to photos by her ID
def getEyeEmPhotoUrlById(photoId): url = "https://www.eyeem.com/api/v2/photos/{0}?access_token={1}".format(photoId, eyeEmToken) response = requests.get(url).json return response['photo']['photoUrl'] Now we find the link to the photo in the tweet, we get its url and also upload it to VK
if url["expanded_url"].startswith("http://www.eyeem.com/p/"): eyeEmId = url["expanded_url"].replace("http://www.eyeem.com/p/", "") photoUrl = getEyeEmPhotoUrlById(eyeEmId) attachments = uploadPhoto(photoUrl) Installing the script on the router
The finished script can be downloaded from github . Tokens in it, of course, changed :) You will need to get yours.
Now we are setting up a router to support third-party packages according to the instructions on the wiki . I will give here a brief instruction for Windows:
- Format the flash drive in NTFS, create system / bin folders on it
- Insert the flash drive into the router, open it in the explorer and copy the ext_init.sh file there
- Remount the disk - disconnect the disk through the web interface, rewind the flash drive. We are waiting until the post "
dropbear[4017] Running in background" appears in the logs of the router - Connect via SSH (root / zyxel) and execute finish_install.sh. We are waiting for the end of the swap.
The router is ready to install opkg packages. Their list can be viewed on the website or by
opkg list . Install python and the necessary modules:Then, install python with the command:
- Basic packages using
opkg install pythonandopkg install python-openssl - The tweepy and requests modules simply copy the folder to
system\usr\lib\python2.7\. The new version of requests did not work for me, try either the cache branch, or from the archive . - To activate cron, rename the K02cron file in the system \ etc \ init.d folder in S02cron
- Configuring the script to run in cron. Add the line to the system \ etc \ crontabs \ root file
*/15 * * * * killall -9 "python" ; /media/DISK_A1/system/usr/bin/python /media/DISK_A1/system/root/TwVk.py@rebootdid not work. Well, plus the script is also not quite perfect, restarting it does not hurt. If someone can come up with a better idea, you are welcome. - Finally, we copy the script to system \ root \ and remount the disk again.
Conclusion
This script works for me for about six months (the article was written sooo long), most of the errors I corrected. My wall in VC has become much easier to read because of the expanded links and attached photos. I am pleased. I plan to create a couple of Twitter scripts based on a router, such as a twitter bot.
I hope this article is useful to someone. At least to start using the existing router not only for its intended purpose :)
I honestly tried to correct all grammatical errors and typos, but if you notice something - write in a personal.
I also remind you once again that my knowledge of Linux and Python is not far from zero, and I will be happy to hear your tips on how to improve the script.
Acknowledgments
- zyxmon , man and
steamerforum. His contribution to the Zyxel community cannot be overemphasized. - kethlin_mil , my girlfriend, without her I wouldn’t have figured out linux, cron and Perl.
Source: https://habr.com/ru/post/158285/
All Articles