Android, Ubuntu and Python: automate Internet radio recording and on-air synchronization
Hello to the whole community Habra!
Surely, many along the way “home” <-> “work” listen to music from their Android background on the way. I also often sleep on the subway under the peppy breaks extra 10-20 minutes. In the next trip to the operational base, enjoying the tracks, which are already “a hundred years old at lunch,” I made a note in my head that I would need to update the music library in the evening. Of course, the note was safely forgotten in the whirlwinds of the working day, and the next morning I was driving again with a hackneyed record. He estimated a little and decided that it would be necessary to automate this process in order to exclude my godless influence on the matter of automation.
Interested citizens - welcome under cat.
The choice of strategy was determined during the trip, the benefit of a small “black coffin” aka server on the Atom, with Ubuntu on board was at my home disposal. The solution was this: some demon girl stream the Internet radio stream on the server, and the snake script does everything else: sorts the tracks into folders, creates playlists, and somehow merges all the good into a smartphone, preferably at night.
After the general concept, there was a choice of a specific implementation of the plan. So let's get started in order:
Record stream broadcast. As a "invader" I chose a small console utility streamripper . Able to do everything you need and even more. It is enough to give her the address of the stream and several more settings: where to store the tracks and how to call them.
I myself am not directly connected with the IT infrastructure by the nature of my work. Just sometimes, in the case of any standard actions, it is easier to write a script or handler, and with calm conscience to do the main work, and not to waste it on a routine. I can not stand, when suddenly it turns out you need to check 100 items and find out the date of delivery from the supplier. At the same time, of course, the supplier, as a sincerely caring vendor who is interested in sales, resolutely does not enter the 21st century, leaving one field for one position in the form on the website. Mass request processing and database synchronization? No, have not heard) Through this, it is easier to raise the Apache and incite the curl to this disgrace, than to sit for an hour and do copy-and-paste. Having estimated what would be better, I chose python from my knowledge bag. On it will be based the entire skeleton, which will bring together the data synchronization system.
')
Next - how to transfer information to a smartphone? In general, since Android itself is a Nikos * mega-meeting, there should be no problems with choosing a transfer interface. Quickly rummaging through GoogleMarket, I found everything that is needed as a server part for Androida. It turned out that Samba, SSHfs and standard FTP are available. After several test transfers from the server to the android, 2-3 GB of tracks stopped on FTP, although it was certainly more pleasant to work on SSHfs. But that's bad luck - whether my smart (peredito on Android HTC HD2) could not cope with the load, or the transfer of keys was crookedly implemented or something else, but as a fact I still have sludge along Samba and ssh. Record breaks occurred periodically, especially on large volumes (over 0.7-1Gb), keys were lost, or access rights were incorrectly issued for the sshfs user. In general, taking into account stable FTP, the bug of the developers or my hands decisively got sick and the last moment remained. The FTP server had to be raised on the phone, either by a command from the server or by its own scheduler. In any case, it was necessary to deal with the console in Android and transfer the application intent.action to start the server. And here I was disappointed - not one of the clients (maybe looking bad?) Had anything similar in the description of AndroidManifest.xml. Those. The launch of the application itself was possible, but the start of the server is no longer. I just happened to be lucky that I noticed in the functions to the wonderful FTPServer application such a feature as “start the FTP server when connected to the selected wi-fi network”. Now this problem has been solved.
And the last question. It is more connected with the device itself of the Android operating system, namely with its high power consumption. The FTP server itself keeps the wi-fi and display on at all times, thus preventing the device from going to sleep, which is correct, otherwise Android will turn off the network and the FTP server will be unavailable. Thus, the task has been reduced to the following: to start FTP, if there is a charge, at night and while in the coverage area of home wi-fi. I remembered the once-read note on Habré about the Tasker program, which supported writing simple scripts and switching phone functions depending on different conditions. The program was immediately downloaded, studied and tested. The result is more than satisfied.
As a result, in general, the bundle acted as follows: from the side of the smartphone at night (a few minutes before the script started on the server), if there was charging and the availability of the home network, the FTPServer started, which was on the network for 2 hours, after which the phone went to reboot, which update your files library and make playlists available. From the Ubuntu server side, the stream was recorded using streamrippera, and with the help of the crown a script was launched at night that did the following:
Such a small home automation has been living with me for a couple of weeks and serious bugs have not been noticed yet. It is especially pleasant to get up in the morning, put on headphones and be glad that there is plenty of new music and the whole process of transferring the recordings takes place in the background for me as a person.
Actually, such a small story is not so much about music, but about the opportunities that modern development gives us today in order to make life more comfortable, leaving time for creativity, and not for routine.
A couple of recommendations and comments:
Below I will provide the text of the script, in general there are quite a lot of explanations, it should be clear on the fly.
Surely, many along the way “home” <-> “work” listen to music from their Android background on the way. I also often sleep on the subway under the peppy breaks extra 10-20 minutes. In the next trip to the operational base, enjoying the tracks, which are already “a hundred years old at lunch,” I made a note in my head that I would need to update the music library in the evening. Of course, the note was safely forgotten in the whirlwinds of the working day, and the next morning I was driving again with a hackneyed record. He estimated a little and decided that it would be necessary to automate this process in order to exclude my godless influence on the matter of automation.
Interested citizens - welcome under cat.
The choice of strategy was determined during the trip, the benefit of a small “black coffin” aka server on the Atom, with Ubuntu on board was at my home disposal. The solution was this: some demon girl stream the Internet radio stream on the server, and the snake script does everything else: sorts the tracks into folders, creates playlists, and somehow merges all the good into a smartphone, preferably at night.
After the general concept, there was a choice of a specific implementation of the plan. So let's get started in order:
Attention! All conversations are recorded.
Record stream broadcast. As a "invader" I chose a small console utility streamripper . Able to do everything you need and even more. It is enough to give her the address of the stream and several more settings: where to store the tracks and how to call them.
Use Python
I myself am not directly connected with the IT infrastructure by the nature of my work. Just sometimes, in the case of any standard actions, it is easier to write a script or handler, and with calm conscience to do the main work, and not to waste it on a routine. I can not stand, when suddenly it turns out you need to check 100 items and find out the date of delivery from the supplier. At the same time, of course, the supplier, as a sincerely caring vendor who is interested in sales, resolutely does not enter the 21st century, leaving one field for one position in the form on the website. Mass request processing and database synchronization? No, have not heard) Through this, it is easier to raise the Apache and incite the curl to this disgrace, than to sit for an hour and do copy-and-paste. Having estimated what would be better, I chose python from my knowledge bag. On it will be based the entire skeleton, which will bring together the data synchronization system.
')
Samba, FTP or SSHfs
Next - how to transfer information to a smartphone? In general, since Android itself is a Nikos * mega-meeting, there should be no problems with choosing a transfer interface. Quickly rummaging through GoogleMarket, I found everything that is needed as a server part for Androida. It turned out that Samba, SSHfs and standard FTP are available. After several test transfers from the server to the android, 2-3 GB of tracks stopped on FTP, although it was certainly more pleasant to work on SSHfs. But that's bad luck - whether my smart (peredito on Android HTC HD2) could not cope with the load, or the transfer of keys was crookedly implemented or something else, but as a fact I still have sludge along Samba and ssh. Record breaks occurred periodically, especially on large volumes (over 0.7-1Gb), keys were lost, or access rights were incorrectly issued for the sshfs user. In general, taking into account stable FTP, the bug of the developers or my hands decisively got sick and the last moment remained. The FTP server had to be raised on the phone, either by a command from the server or by its own scheduler. In any case, it was necessary to deal with the console in Android and transfer the application intent.action to start the server. And here I was disappointed - not one of the clients (maybe looking bad?) Had anything similar in the description of AndroidManifest.xml. Those. The launch of the application itself was possible, but the start of the server is no longer. I just happened to be lucky that I noticed in the functions to the wonderful FTPServer application such a feature as “start the FTP server when connected to the selected wi-fi network”. Now this problem has been solved.
Android
And the last question. It is more connected with the device itself of the Android operating system, namely with its high power consumption. The FTP server itself keeps the wi-fi and display on at all times, thus preventing the device from going to sleep, which is correct, otherwise Android will turn off the network and the FTP server will be unavailable. Thus, the task has been reduced to the following: to start FTP, if there is a charge, at night and while in the coverage area of home wi-fi. I remembered the once-read note on Habré about the Tasker program, which supported writing simple scripts and switching phone functions depending on different conditions. The program was immediately downloaded, studied and tested. The result is more than satisfied.
Results
As a result, in general, the bundle acted as follows: from the side of the smartphone at night (a few minutes before the script started on the server), if there was charging and the availability of the home network, the FTPServer started, which was on the network for 2 hours, after which the phone went to reboot, which update your files library and make playlists available. From the Ubuntu server side, the stream was recorded using streamrippera, and with the help of the crown a script was launched at night that did the following:
- Checked: whether streamripper is running and whether there are no copies of it in the processes. If this was not the case, then the script organized a separate child process of streamrippera.
- sorted downloaded tracks into folders in the format 'Radio Name - DD.MM.YYYY'
- in the same folders created a playlist * .m3u according to the time of the creation of the mp3 file. Those. in fact in the order they sounded on the radio.
- to save space, delete old records on the server (the limit is registered in the script)
- deleted old entries on Android (the limit is registered in the script)
- added new ones to Android (the limit is registered in the script)
Such a small home automation has been living with me for a couple of weeks and serious bugs have not been noticed yet. It is especially pleasant to get up in the morning, put on headphones and be glad that there is plenty of new music and the whole process of transferring the recordings takes place in the background for me as a person.
Actually, such a small story is not so much about music, but about the opportunities that modern development gives us today in order to make life more comfortable, leaving time for creativity, and not for routine.
Used materials":
- FTP server for Android FTPServer
- an advanced task scheduler for Android Tasker
- Python
- operating system Ubuntu Server 12.04
- mobile phone HTC HD2 with Android operating system on board
PS
A couple of recommendations and comments:
- on the router, it is better to bind to the mac-address for your smartphone, in order to make a permanent issue of only one ip address of the ftp server
- periodically check the free space on the flash-card of the phone
- on average, one day of the stream (at 128kbit / s) takes 0.8-1.2GB
- the transfer rate from the server to the smartphone over the air is approximately 2.5-3Mb / C
Below I will provide the text of the script, in general there are quite a lot of explanations, it should be clear on the fly.
#!/usr/bin/python # -*- coding: utf-8 -*- #****************** ******************* #*********************************************************** import os import shutil from datetime import * import time from operator import itemgetter from ftplib import FTP import ftplib import subprocess import commands #*********************************************************** #****************** ******************* #****************** ************************ #*********************************************************** dir_exec = '' # , dir_music = '' #, -( streamripper) dir_music = os.path.join(dir_exec,dir_music) # alias_dir = '' # . : ' - ..' time_shift = 3 # time_sync = 4 # , time_storage = 30 # , server = '192.168.1.126' #FTP server, login = '' # FTP pass_ftp = ' ' # FTP port = '2121' # FTP andr_music = '/mnt/sdcard/Music' # FTP rec_proc = ['streamripper', ' ', '-d', '-D', '%S/%A-%T'] # streamrippera` #*********************************************************** #****************** ************************ #****************** ************************* #*********************************************************** #*** streamripper ( ) shell` def start_rec(proc_stream): num = commands.getoutput('pidof %s |wc -w' % proc_stream) if int(num) >= 2: os.system('killall ' + proc_stream) num = commands.getoutput('pidof %s |wc -w' % proc_stream) if int(num) == 0: proc = subprocess.Popen([rec_proc[0], rec_proc[1], rec_proc[2], dir_exec, rec_proc[3], rec_proc[4],], stdout=subprocess.PIPE) #*** ftp- def ftp_online(): ftpConnect = FTP() try: ftpConnect.connect(server,port) ftpConnect.login(login, pass_ftp) ftpConnect.quit() ftpConnect.close() enable = 1 except: enable = 0 return enable #*** , . def mdir(create_dir): if not os.path.isdir(create_dir): os.makedirs(create_dir) #*** def uniq_date(lst): return reduce(lambda y,z: not (z in y) and y.append(z) or y, lst, []) #*** mp3 def date_mp3(path_dir_music): out_date=[] names = os.listdir(path_dir_music) for name in names: if name.endswith('mp3'): path = os.path.join(path_dir_music,name) mtime = os.path.getmtime(path) - time_shift*60*60 out_date.append(date.fromtimestamp(mtime).strftime('%d.%m.%Y')) return out_date #*** mp3 def name_mp3(path_dir_music): out_name=[] names = os.listdir(path_dir_music) for name in names: if name.endswith('mp3'): out_name.append(name) return out_name #*** . . def m3u(dir,path): m3u_n_d = [] m3u_n_s = [] m3u_names = name_mp3(path) i=0 for m3u_name in m3u_names: m3u_mtime = os.path.getmtime(os.path.join(path,m3u_name)) - time_shift*60*60 m3u_n_d.append([m3u_name,m3u_mtime]) i=i+1 m3u_n_d = sorted(m3u_n_d, key=itemgetter(1)) w = open(path+'/'+dir+'.m3u','w') w.write('#EXTM3U\n') #EXTM3U m3U for i in range(i): #w.write('./'+m3u_n_d[i][0]+'\n') # w.write(m3u_n_d[i][0]+'\n') # PowerAmp w.close #*** NLST. def ftp_nlst(pwd): log = [] ftpConnect.cwd(pwd) ftpConnect.retrlines('NLST', callback=log.append) return log #*** LIST def ftp_list(pwd): log = [] file_ftp = [] ftpConnect.cwd(pwd) ftpConnect.retrlines('LIST', callback=log.append) files = (line.split(':')[1] for line in log) p = list(files) for name in p: file_ftp.append((name[3:])) return file_ftp #*** def num_day(dir_time): date_ = time.mktime(time.strptime(dir_time.split(' - ')[1],'%d.%m.%Y')) now_ = time.time() day_beet = (now_-date_)/(60*60*24) return int(day_beet) #*********************************************************** #****************** ************************* #****************** ***************************** #*********************************************************** # start_rec(rec_proc[0]) # creates_dirs = [] u_dates = uniq_date(date_mp3(dir_music)) j=0 for u_date in u_dates: mdir(os.path.join(dir_exec,alias_dir+u_date)) creates_dirs.append([alias_dir+u_date,os.path.join(dir_exec,alias_dir+u_date)]) j=j+1 # mp3 out_names = name_mp3(dir_music) for out_name in out_names: path = os.path.join(dir_music,out_name) mtime = os.path.getmtime(path) - time_shift*60*60 out_date_temp = date.fromtimestamp(mtime).strftime('%d.%m.%Y') shutil.move(os.path.join(dir_music,out_name),os.path.join(dir_exec,alias_dir+out_date_temp,out_name)) # for j in range(j): m3u(creates_dirs[j][0],creates_dirs[j][1]) # dir_rem_serv = os.listdir(dir_exec) for dir_rem_s in dir_rem_serv: if dir_rem_s[:15] == alias_dir: if num_day(dir_rem_s) > time_storage: shutil.rmtree(os.path.join(dir_exec,dir_rem_s)) # if ftp_online() == 1: ftpConnect = FTP() ftpConnect.connect(server,port) ftpConnect.login(login, pass_ftp) ftpConnect.cwd(andr_music) # st = ftpConnect.pwd() # dir_on_root = ftp_list(st) for dir_rem in dir_on_root: if dir_rem[:15] == alias_dir: if num_day(dir_rem) > time_sync: ftpConnect.cwd(os.path.join(st,dir_rem)) files_rem = ftp_list(os.path.join(st,dir_rem)) for file_rem in files_rem: ftpConnect.delete(file_rem) # ftpConnect.cwd(st) ftpConnect.rmd(os.path.join(st,dir_rem)) # ftpConnect.quit() ftpConnect.close() # dir_names = os.listdir(dir_exec) for dir_name in dir_names: if dir_name[:15] == alias_dir: if num_day(dir_name) <= time_sync and ftp_online() == 1: ftpConnect = FTP() ftpConnect.connect(server,port) ftpConnect.login(login, pass_ftp) ftpConnect.cwd(andr_music) # st = ftpConnect.pwd() # dir_on_root = ftp_list(st) if dir_name not in dir_on_root: ftpConnect.mkd(os.path.join(st,dir_name)) ftpConnect.cwd(os.path.join(st,dir_name)) list_files = os.listdir(os.path.join(dir_exec,dir_name)) files_on_dir = ftp_list(os.path.join(st,dir_name)) for file in list_files: st_file = str(os.path.join(dir_exec,dir_name,file)) if file not in files_on_dir: ftpConnect.storbinary('STOR ' + file, open(st_file, 'rb'),1024) if file.endswith('m3u'): ftpConnect.storbinary('STOR ' + file, open(st_file, 'rb'),1024) ftpConnect.quit() ftpConnect.close() #*********************************************************** #****************** ****************************** Source: https://habr.com/ru/post/146272/
All Articles