Organizing regular digest logs with python and ansible using asterisk as an example

When I created this tool, I was not familiar with logwatch. I wanted to see the situation with the logs on my servers in general, and so I did this bike. I think that this mechanism can help newcomers to understand the alternative possibilities of ansible.

Used software products:

• python 2.7.14
• ansible 2.3
• asterisk server based on FreePBX 13

The mechanism consists of two parts - a python script that processes the log file and sends a report to the mail, and a playbook for collecting logs from the servers and transferring them to the script for processing.

Playbook itself:
')

--- - name: parseastlogs hosts: production_asterisks vars: date: "{{ lookup('pipe', 'date +%Y%m%d') }}" ipaddr: "{{ ansible_default_ipv4.address }}" tasks: - debug: var=date - debug: var=ipaddr - fetch: src: /var/log/asterisk/full-{{ date }} dest: /tmp/full-{{ date }}-{{ ipaddr }} flat: yes - local_action: "shell /etc/ansible/localscripts/astReporter.py {{ ipaddr }} full-{{ date }}-{{ ipaddr }}"    ,            fetch.        -   flat,       . 

Script:

 #!/usr/bin/python2.7 import re from collections import Counter import yagmail import sys import datetime import os servername=sys.argv[1] yag=yagmail.SMTP(user='ansible@2.1',password=None,host='192.168.2.1',port='25',smtp_starttls=False,smtp_set_debuglevel=0,smtp_skip_login=True) recipients=['user@mydomain.local'] filename=sys.argv[2] workdir='/tmp/' log_file=open(workdir+filename,'r')#sys.argv[1] loglist=list() verbosity=['NOTICE','ERROR','WARNING'] regexes = ["Call from '.*' .* to extension '.*' rejected because extension not found in context '.*'.", "Identifier \d+, identifier_type \d+ not found in identifier list", "Invalid result identifier \d+ passed in aMYSQL_clear", "This function can only be used on SIP channels.", "fwrite() returned error: Broken pipe", "CDR requires a value \(CDR\(variable\)=value\)", "Received SIP subscribe for peer without mailbox: .*", "Removed interface '.*' from queue '.*'", "Peer '.*' is trying to register, but not configured as host=dynamic", "Registration from '.*' failed for '.*' - Peer is not supposed to register", "Unable to join queue '.*'", "Attempt to pause interface Local/@from-queue/n, not found", "PRESENCE_STATE unknown", "EXTENSION_STATE requires an extension", "Prodding channel '.*' failed", "Channel '.*' sent to invalid extension but no invalid handler: context,exten,priority=.*", "Can't send 10 type frames with PJSIP", "Attempt to pause interface .+, not found", "Attempt to unpause interface .+, not found", "no samples for ulawtolin", "Could not find matching INVITE transaction for CANCEL request", "Peer '.*' is now Reachable. \(.*\)", "Peer '.*' is now UNREACHABLE! Last qualify: .*", "Registration from .* failed for '.*' - Wrong password", "Retransmission timeout reached on transmission .*", "no samples for alawtolin", "Peer '.*' is now Lagged. \(\d+ms / \d+ms\)", "Call completed to .*", "Invalid retrytime at line \d+ of .*", "Not accepting call completion offers from call-forward recipient .*", "No such context '.*' for macro '.*'\. Was called by .*", "[pP]ickup .* attempt by .*", "Call failed to go through, reason \(5\) Remote end is Busy", "Deprecated syntax found\..* ", "No digits dialed for atxfer.", "Unable to create channel of type 'SIP' \(cause \d+ - Subscriber absent\)", "'tls' is not a valid transport type when tlsenable=no\. If no other is specified, the defaults from general will be used\.", "'tcp' is not a valid transport type when tcpenable=no\. If no other is specified, the defaults from general will be used\.", "Queued call to .* expired without completion after \d+ attempts", "Re-invite to non-existing call leg on other UA. SIP dialog .*\. Giving up.", "Channel .* not found! Variable 'BLKVM' not set to .*\.", "Remote host can't match request CANCEL to call .*\. Giving up\.", "Unable to execute query \[.*\]", "SQL Exec Direct failed \(-1\)!\[", "SQL Execute returned an error .*", " -- Re-registration for .*", "Outbound Registration\: Expiry for .* is \d+ sec \(Scheduling reregistration in \d+ s\)", "Correct auth, but based on stale nonce received from '.*'", "Unable to write frametype: 2", "Received response: \"Forbidden\" from '\".*\" .*'", "Huh\? Not an RDNIS SIP header .*", "Hanging up call .* - no reply to our critical packet .*", "Cancelling retransmit of OPTIONs \(call id .*\) ", "Still have a QUALIFY dialog active, deleting", "The use of '_\.' for an extension is strongly discouraged and can have unexpected behavior. Please use '_X\.' instead at line .*", "Context '.*' tries to include nonexistent context '(.*)'", "aMYSQL_query: mysql_query failed\. Error: Duplicate entry .*", "RTCP SR transmission error to .*, rtcp halted Operation not permitted", "Failed to write frame to '.*': Resource temporarily unavailable", "Unable to forward frametype: 2", "Timeout on .* on non-critical invite transaction.", "Unexpected control subclass '\d+'", "Context '.*' for macro '.*' lacks .*", "No response received from '.*' on registration attempt to '.*', retrying in '\d+'", "Unknown RTP codec 90 received from '.*'", "Invalid extension '.*', but no rule 'i' or 'e' in context '.*'", "Added interface '.*' to queue '.*'", "Exceptionally long voice queue length queuing to .*", "Request from '.*' failed for '.*' \(callid: .*\) - No matching endpoint found", "Junk at the beginning of frame \d+", "Unable to register extension at line .*", "Unable to register extension '.*' priority .* in '.*', already in use", "Unable to load config file .*", 'Purely numeric hostname \(\d+\), and not a peer--rejecting!', "Unknown directive .* at line \d+ .*", "Context '.*' already included in '.*' context on include at line \d+ of .*", "No closing parenthesis found\? '.*' at line \d+ of .*", "Can't use '.*' priority on the first entry at line \d+ of .*", "The .* options are deprecated. Please see UPGRADE.txt for information", "Can't use '.*' priority on the first entry at line \d+ of .*", "Call failed to go through, reason .*", "Unable to open .*", "Playback of message .* failed", "File .* does not exist in any format", "Playback failed on .* for .*", "Adding .* to .*", "Can't move channel. One or both is dead .*", "Unable to complete call pickup of .*", "Pickup .* failed by .*", "No entry in voicemail config file for .*", " \-\- Registration for '.*' timed out, trying again \(Attempt #[0-9]+\)", "Disconnecting call .* for lack of RTP activity in [0-9]+ seconds", "Failed to initialize .* declining image stream", "Can't send 10 type frames with SIP write", "Set requires an '=' to be a valid assignment.", "Timeout but no rule 't' or 'e' in context .*", "RTCP RR transmission error to .*, rtcp halted Operation not permitted", "Unable to acquire target extension for attended transfer.", "Unterminated comment detected beginning on line"] notmatched=list() matchedregex=list() combinedre="("+")|(".join(regexes) + ")" #           .        . class logitem: ldatetime=None ltype=None lchan=None lsource=None lcontent=None def __init__(self,ldate,ltype,lch,lsrc,lcont): self.ldatetime=ldate self.ltype=ltype self.lchan=lch self.lsource=lsrc self.lcontent=lcont #        def main(): i=0 dumpchanstart=False for line in log_file: #   Dumpchan if 'app_dumpchan.c' in line: dumpchanstart = 0 for line in log_file: if dumpchanstart==2: break if "================================================================================" in line: dumpchanstart+=1 #    ,    [ if line[0]=="[": buf=re.split('] ', line) #     verbosity if any(x in buf[1] for x in verbosity): buf1spl=buf[1].split('-') channum='' buf2spl=re.split('c: ',buf[2]) if len(buf1spl)>1: channum=buf1spl[1] itemcheck=logitem(buf[0][1:],buf[1].split('[')[0],channum,buf2spl[0],buf2spl[1]) filtration(itemcheck.lcontent) #      matchedregex,        notmatched,  . def filtration(logitem): found = False for regexitem in regexes: if re.match(regexitem,logitem): found= True matchedregex.append(regexitem) continue if found==False: notmatched.append(logitem) #    .   if..elif      def genContents(): content=u"<html>" content+=u"<head></head>" content+=u"<body>" content+=u"<h1>Log digest for " +servername+ u"</h1>" content+=u"<table style='width: 60%;' border = \"1\" cellpadding = \"1\"'>" for key, val in Counter(matchedregex).most_common(): if u"rejected because extension not found in context" in key: key=u"<b style='background-color:#e8edff'>"+key+u"</b>" elif u"No matching endpoint found" in key: key=u"<b style='background-color:#fd6161'>"+key+u"</b>" elif u"No closing parenthesis " in key: key = u"<b style='background-color:#98862a'>" + key + u"</b>" elif u"Wrong password" in key: key=u"<b style='background-color:#fd6161'>"+key+u"</b>" elif u"Unterminated comment detected beginning on line" in key: key=u"<b style='background-color:#fd6161'>"+key+u"</b>" content += u"<tr>" content+= u"<td>"+key+u"</td><td>" +str(val) +u"</td>" content += u"</tr>" content+=u"</table><br>" content+=u"<h1>not matched logitems:</h1><br>" for item in notmatched: content+=item content+=u"</body>\n" content+=u"</html>\n" return content main() cont=genContents() yag.send(recipients,subject='Log digest for '+servername,contents=cont) os.remove(workdiir+filename) 

the script is launched by the command: / usr / local / bin / ansible-playbook /etc/ansible/playbooks/parseastlogs.yml

The result of the command will be a number of letters (one letter to each server in the production_asterisk group) with something like the following:



If you have questions or suggestions, I am ready to answer them.