SQUID Proxy Automation

This post does not claim to be new, I give an example from real life. I myself am not an expert in Bash & PHP languages. Most likely, you can further simplify and improve this manual.
At work, it was necessary to automate the database update process and simplify the change of the prohibition files for SquidGuard. In a short time and by virtue of my knowledge, I implemented this task in this way ...

1) We write a script to automatically update the database of restrictions (update_squidGuard.sh). The script creates a backup copy of the current ban list, downloads a new database of sheets, unpacks it, updates and reconfigures Squid. He will work once a week.

1. #!/bin/sh echo '__________ __________' tar zcf old_blacklists.tgz /etc/squid/blacklists/ echo '============================' echo '!' echo '============================' echo '__________ __________' /usr/bin/wget -q --cache=off 'http://www.shallalist.de/Downloads/shallalist.tar.gz' –O /etc/squid/updatedb/shallalist.tar.gz tar zxf /etc/squid/updatedb/shallalist.tar.gz -C /etc/squid/updatedb/ cp -R -f /etc/squid/updatedb/BL/* /etc/squid/blacklists/ rm -R /etc/squid/updatedb/BL/ echo '============================' echo '!' echo '============================' echo '__________ SQUID__________' /etc/squid/updatedb/rebuid_base.sh squid -k reconfigure echo '============================' echo ' !' echo '============================'
2. #!/bin/sh echo '__________ __________' tar zcf old_blacklists.tgz /etc/squid/blacklists/ echo '============================' echo '!' echo '============================' echo '__________ __________' /usr/bin/wget -q --cache=off 'http://www.shallalist.de/Downloads/shallalist.tar.gz' –O /etc/squid/updatedb/shallalist.tar.gz tar zxf /etc/squid/updatedb/shallalist.tar.gz -C /etc/squid/updatedb/ cp -R -f /etc/squid/updatedb/BL/* /etc/squid/blacklists/ rm -R /etc/squid/updatedb/BL/ echo '============================' echo '!' echo '============================' echo '__________ SQUID__________' /etc/squid/updatedb/rebuid_base.sh squid -k reconfigure echo '============================' echo ' !' echo '============================'
3. #!/bin/sh echo '__________ __________' tar zcf old_blacklists.tgz /etc/squid/blacklists/ echo '============================' echo '!' echo '============================' echo '__________ __________' /usr/bin/wget -q --cache=off 'http://www.shallalist.de/Downloads/shallalist.tar.gz' –O /etc/squid/updatedb/shallalist.tar.gz tar zxf /etc/squid/updatedb/shallalist.tar.gz -C /etc/squid/updatedb/ cp -R -f /etc/squid/updatedb/BL/* /etc/squid/blacklists/ rm -R /etc/squid/updatedb/BL/ echo '============================' echo '!' echo '============================' echo '__________ SQUID__________' /etc/squid/updatedb/rebuid_base.sh squid -k reconfigure echo '============================' echo ' !' echo '============================'
4. #!/bin/sh echo '__________ __________' tar zcf old_blacklists.tgz /etc/squid/blacklists/ echo '============================' echo '!' echo '============================' echo '__________ __________' /usr/bin/wget -q --cache=off 'http://www.shallalist.de/Downloads/shallalist.tar.gz' –O /etc/squid/updatedb/shallalist.tar.gz tar zxf /etc/squid/updatedb/shallalist.tar.gz -C /etc/squid/updatedb/ cp -R -f /etc/squid/updatedb/BL/* /etc/squid/blacklists/ rm -R /etc/squid/updatedb/BL/ echo '============================' echo '!' echo '============================' echo '__________ SQUID__________' /etc/squid/updatedb/rebuid_base.sh squid -k reconfigure echo '============================' echo ' !' echo '============================'
5. #!/bin/sh echo '__________ __________' tar zcf old_blacklists.tgz /etc/squid/blacklists/ echo '============================' echo '!' echo '============================' echo '__________ __________' /usr/bin/wget -q --cache=off 'http://www.shallalist.de/Downloads/shallalist.tar.gz' –O /etc/squid/updatedb/shallalist.tar.gz tar zxf /etc/squid/updatedb/shallalist.tar.gz -C /etc/squid/updatedb/ cp -R -f /etc/squid/updatedb/BL/* /etc/squid/blacklists/ rm -R /etc/squid/updatedb/BL/ echo '============================' echo '!' echo '============================' echo '__________ SQUID__________' /etc/squid/updatedb/rebuid_base.sh squid -k reconfigure echo '============================' echo ' !' echo '============================'
6. #!/bin/sh echo '__________ __________' tar zcf old_blacklists.tgz /etc/squid/blacklists/ echo '============================' echo '!' echo '============================' echo '__________ __________' /usr/bin/wget -q --cache=off 'http://www.shallalist.de/Downloads/shallalist.tar.gz' –O /etc/squid/updatedb/shallalist.tar.gz tar zxf /etc/squid/updatedb/shallalist.tar.gz -C /etc/squid/updatedb/ cp -R -f /etc/squid/updatedb/BL/* /etc/squid/blacklists/ rm -R /etc/squid/updatedb/BL/ echo '============================' echo '!' echo '============================' echo '__________ SQUID__________' /etc/squid/updatedb/rebuid_base.sh squid -k reconfigure echo '============================' echo ' !' echo '============================'
7. #!/bin/sh echo '__________ __________' tar zcf old_blacklists.tgz /etc/squid/blacklists/ echo '============================' echo '!' echo '============================' echo '__________ __________' /usr/bin/wget -q --cache=off 'http://www.shallalist.de/Downloads/shallalist.tar.gz' –O /etc/squid/updatedb/shallalist.tar.gz tar zxf /etc/squid/updatedb/shallalist.tar.gz -C /etc/squid/updatedb/ cp -R -f /etc/squid/updatedb/BL/* /etc/squid/blacklists/ rm -R /etc/squid/updatedb/BL/ echo '============================' echo '!' echo '============================' echo '__________ SQUID__________' /etc/squid/updatedb/rebuid_base.sh squid -k reconfigure echo '============================' echo ' !' echo '============================'
8. #!/bin/sh echo '__________ __________' tar zcf old_blacklists.tgz /etc/squid/blacklists/ echo '============================' echo '!' echo '============================' echo '__________ __________' /usr/bin/wget -q --cache=off 'http://www.shallalist.de/Downloads/shallalist.tar.gz' –O /etc/squid/updatedb/shallalist.tar.gz tar zxf /etc/squid/updatedb/shallalist.tar.gz -C /etc/squid/updatedb/ cp -R -f /etc/squid/updatedb/BL/* /etc/squid/blacklists/ rm -R /etc/squid/updatedb/BL/ echo '============================' echo '!' echo '============================' echo '__________ SQUID__________' /etc/squid/updatedb/rebuid_base.sh squid -k reconfigure echo '============================' echo ' !' echo '============================'
9. #!/bin/sh echo '__________ __________' tar zcf old_blacklists.tgz /etc/squid/blacklists/ echo '============================' echo '!' echo '============================' echo '__________ __________' /usr/bin/wget -q --cache=off 'http://www.shallalist.de/Downloads/shallalist.tar.gz' –O /etc/squid/updatedb/shallalist.tar.gz tar zxf /etc/squid/updatedb/shallalist.tar.gz -C /etc/squid/updatedb/ cp -R -f /etc/squid/updatedb/BL/* /etc/squid/blacklists/ rm -R /etc/squid/updatedb/BL/ echo '============================' echo '!' echo '============================' echo '__________ SQUID__________' /etc/squid/updatedb/rebuid_base.sh squid -k reconfigure echo '============================' echo ' !' echo '============================'
10. #!/bin/sh echo '__________ __________' tar zcf old_blacklists.tgz /etc/squid/blacklists/ echo '============================' echo '!' echo '============================' echo '__________ __________' /usr/bin/wget -q --cache=off 'http://www.shallalist.de/Downloads/shallalist.tar.gz' –O /etc/squid/updatedb/shallalist.tar.gz tar zxf /etc/squid/updatedb/shallalist.tar.gz -C /etc/squid/updatedb/ cp -R -f /etc/squid/updatedb/BL/* /etc/squid/blacklists/ rm -R /etc/squid/updatedb/BL/ echo '============================' echo '!' echo '============================' echo '__________ SQUID__________' /etc/squid/updatedb/rebuid_base.sh squid -k reconfigure echo '============================' echo ' !' echo '============================'
11. #!/bin/sh echo '__________ __________' tar zcf old_blacklists.tgz /etc/squid/blacklists/ echo '============================' echo '!' echo '============================' echo '__________ __________' /usr/bin/wget -q --cache=off 'http://www.shallalist.de/Downloads/shallalist.tar.gz' –O /etc/squid/updatedb/shallalist.tar.gz tar zxf /etc/squid/updatedb/shallalist.tar.gz -C /etc/squid/updatedb/ cp -R -f /etc/squid/updatedb/BL/* /etc/squid/blacklists/ rm -R /etc/squid/updatedb/BL/ echo '============================' echo '!' echo '============================' echo '__________ SQUID__________' /etc/squid/updatedb/rebuid_base.sh squid -k reconfigure echo '============================' echo ' !' echo '============================'
12. #!/bin/sh echo '__________ __________' tar zcf old_blacklists.tgz /etc/squid/blacklists/ echo '============================' echo '!' echo '============================' echo '__________ __________' /usr/bin/wget -q --cache=off 'http://www.shallalist.de/Downloads/shallalist.tar.gz' –O /etc/squid/updatedb/shallalist.tar.gz tar zxf /etc/squid/updatedb/shallalist.tar.gz -C /etc/squid/updatedb/ cp -R -f /etc/squid/updatedb/BL/* /etc/squid/blacklists/ rm -R /etc/squid/updatedb/BL/ echo '============================' echo '!' echo '============================' echo '__________ SQUID__________' /etc/squid/updatedb/rebuid_base.sh squid -k reconfigure echo '============================' echo ' !' echo '============================'
13. #!/bin/sh echo '__________ __________' tar zcf old_blacklists.tgz /etc/squid/blacklists/ echo '============================' echo '!' echo '============================' echo '__________ __________' /usr/bin/wget -q --cache=off 'http://www.shallalist.de/Downloads/shallalist.tar.gz' –O /etc/squid/updatedb/shallalist.tar.gz tar zxf /etc/squid/updatedb/shallalist.tar.gz -C /etc/squid/updatedb/ cp -R -f /etc/squid/updatedb/BL/* /etc/squid/blacklists/ rm -R /etc/squid/updatedb/BL/ echo '============================' echo '!' echo '============================' echo '__________ SQUID__________' /etc/squid/updatedb/rebuid_base.sh squid -k reconfigure echo '============================' echo ' !' echo '============================'
14. #!/bin/sh echo '__________ __________' tar zcf old_blacklists.tgz /etc/squid/blacklists/ echo '============================' echo '!' echo '============================' echo '__________ __________' /usr/bin/wget -q --cache=off 'http://www.shallalist.de/Downloads/shallalist.tar.gz' –O /etc/squid/updatedb/shallalist.tar.gz tar zxf /etc/squid/updatedb/shallalist.tar.gz -C /etc/squid/updatedb/ cp -R -f /etc/squid/updatedb/BL/* /etc/squid/blacklists/ rm -R /etc/squid/updatedb/BL/ echo '============================' echo '!' echo '============================' echo '__________ SQUID__________' /etc/squid/updatedb/rebuid_base.sh squid -k reconfigure echo '============================' echo ' !' echo '============================'
15. #!/bin/sh echo '__________ __________' tar zcf old_blacklists.tgz /etc/squid/blacklists/ echo '============================' echo '!' echo '============================' echo '__________ __________' /usr/bin/wget -q --cache=off 'http://www.shallalist.de/Downloads/shallalist.tar.gz' –O /etc/squid/updatedb/shallalist.tar.gz tar zxf /etc/squid/updatedb/shallalist.tar.gz -C /etc/squid/updatedb/ cp -R -f /etc/squid/updatedb/BL/* /etc/squid/blacklists/ rm -R /etc/squid/updatedb/BL/ echo '============================' echo '!' echo '============================' echo '__________ SQUID__________' /etc/squid/updatedb/rebuid_base.sh squid -k reconfigure echo '============================' echo ' !' echo '============================'
16. #!/bin/sh echo '__________ __________' tar zcf old_blacklists.tgz /etc/squid/blacklists/ echo '============================' echo '!' echo '============================' echo '__________ __________' /usr/bin/wget -q --cache=off 'http://www.shallalist.de/Downloads/shallalist.tar.gz' –O /etc/squid/updatedb/shallalist.tar.gz tar zxf /etc/squid/updatedb/shallalist.tar.gz -C /etc/squid/updatedb/ cp -R -f /etc/squid/updatedb/BL/* /etc/squid/blacklists/ rm -R /etc/squid/updatedb/BL/ echo '============================' echo '!' echo '============================' echo '__________ SQUID__________' /etc/squid/updatedb/rebuid_base.sh squid -k reconfigure echo '============================' echo ' !' echo '============================'
17. #!/bin/sh echo '__________ __________' tar zcf old_blacklists.tgz /etc/squid/blacklists/ echo '============================' echo '!' echo '============================' echo '__________ __________' /usr/bin/wget -q --cache=off 'http://www.shallalist.de/Downloads/shallalist.tar.gz' –O /etc/squid/updatedb/shallalist.tar.gz tar zxf /etc/squid/updatedb/shallalist.tar.gz -C /etc/squid/updatedb/ cp -R -f /etc/squid/updatedb/BL/* /etc/squid/blacklists/ rm -R /etc/squid/updatedb/BL/ echo '============================' echo '!' echo '============================' echo '__________ SQUID__________' /etc/squid/updatedb/rebuid_base.sh squid -k reconfigure echo '============================' echo ' !' echo '============================'
18. #!/bin/sh echo '__________ __________' tar zcf old_blacklists.tgz /etc/squid/blacklists/ echo '============================' echo '!' echo '============================' echo '__________ __________' /usr/bin/wget -q --cache=off 'http://www.shallalist.de/Downloads/shallalist.tar.gz' –O /etc/squid/updatedb/shallalist.tar.gz tar zxf /etc/squid/updatedb/shallalist.tar.gz -C /etc/squid/updatedb/ cp -R -f /etc/squid/updatedb/BL/* /etc/squid/blacklists/ rm -R /etc/squid/updatedb/BL/ echo '============================' echo '!' echo '============================' echo '__________ SQUID__________' /etc/squid/updatedb/rebuid_base.sh squid -k reconfigure echo '============================' echo ' !' echo '============================'
19. #!/bin/sh echo '__________ __________' tar zcf old_blacklists.tgz /etc/squid/blacklists/ echo '============================' echo '!' echo '============================' echo '__________ __________' /usr/bin/wget -q --cache=off 'http://www.shallalist.de/Downloads/shallalist.tar.gz' –O /etc/squid/updatedb/shallalist.tar.gz tar zxf /etc/squid/updatedb/shallalist.tar.gz -C /etc/squid/updatedb/ cp -R -f /etc/squid/updatedb/BL/* /etc/squid/blacklists/ rm -R /etc/squid/updatedb/BL/ echo '============================' echo '!' echo '============================' echo '__________ SQUID__________' /etc/squid/updatedb/rebuid_base.sh squid -k reconfigure echo '============================' echo ' !' echo '============================'
20. #!/bin/sh echo '__________ __________' tar zcf old_blacklists.tgz /etc/squid/blacklists/ echo '============================' echo '!' echo '============================' echo '__________ __________' /usr/bin/wget -q --cache=off 'http://www.shallalist.de/Downloads/shallalist.tar.gz' –O /etc/squid/updatedb/shallalist.tar.gz tar zxf /etc/squid/updatedb/shallalist.tar.gz -C /etc/squid/updatedb/ cp -R -f /etc/squid/updatedb/BL/* /etc/squid/blacklists/ rm -R /etc/squid/updatedb/BL/ echo '============================' echo '!' echo '============================' echo '__________ SQUID__________' /etc/squid/updatedb/rebuid_base.sh squid -k reconfigure echo '============================' echo ' !' echo '============================'
21. #!/bin/sh echo '__________ __________' tar zcf old_blacklists.tgz /etc/squid/blacklists/ echo '============================' echo '!' echo '============================' echo '__________ __________' /usr/bin/wget -q --cache=off 'http://www.shallalist.de/Downloads/shallalist.tar.gz' –O /etc/squid/updatedb/shallalist.tar.gz tar zxf /etc/squid/updatedb/shallalist.tar.gz -C /etc/squid/updatedb/ cp -R -f /etc/squid/updatedb/BL/* /etc/squid/blacklists/ rm -R /etc/squid/updatedb/BL/ echo '============================' echo '!' echo '============================' echo '__________ SQUID__________' /etc/squid/updatedb/rebuid_base.sh squid -k reconfigure echo '============================' echo ' !' echo '============================'

2) Create a file that will update the database for SquidGuard every 20 minutes (rebuild_base.sh)

#!/bin/sh
chown -R squid:squid /etc/squid/blacklists
/usr/local/bin/squidGuard -u /etc/squid/blacklists/*/*.diff
chown -R apache:apache /etc/squid/blacklists/*/*.diff
/usr/sbin/squid -k reconfigure



3) Run scripts on crown:

tux# crontab –u squid –e
0,20,40 * * * * /etc/squid/updatedb/rebuid_base.sh
00 21 * * 7 /etc/squid/updatedb/update_squidGuard.sh



4) Naturally, there are many complex systems for remote control and monitoring of Squid. Historically, SAMS collects statistics from us, but no one wants to use its block lists for unknown reasons. I wrote my little file editor in 30 minutes. Since bans files are processed by crown every 20 minutes, I can easily change bans lists.
First, create the symlinks of the files (* .diff) of each of the lock groups in / var / www / html /:
Example for group sheets: bad, good, pron.

domains-bad.diff
domains-good.diff
domains-pron.diff
…
urls-bad.diff
urls-good.diff
urls-pron.diff
…



This is certainly a tedious and not rewarding exercise, but having done it once, you can no longer think about it.
4.1) The easiest protection of access to this section of the site through the htaccess file:

Order deny,allow
Deny from all
Allow from 192.168.0.1
Allow from 192.168.0.2
Allow from 192.168.0.3



4.2) An example of the index.html file:

1. < html >
2. < head >
3. < title > Editor </ title >
4. </ head >
5. < body >
6. < table >
7. < tbody >
8. < tr >
9. < td > DOMAINS-BA Block List </ td >
10. < td > <a href = "lists.php? action = domains-bad"> edit </ a > </ td >
11. </ tr >
12. < tr >
13. < td > URLS-BAD Block List </ td >
14. < td > <a href = "lists.php? action = urls-bad"> edit </ a > </ td >
15. </ tr >
16. </ tbody >
17. </ table >
18. </ body >
19. </ html >

* This source code was highlighted with Source Code Highlighter .

4.3) The file dealing with the output of the block lists lists.php:

1. <? php
2. header ( 'Content-Type: text / html; charset = UTF-8' );
3. $ var = "domains" ;
4. if ( isset ($ _GET [ 'action' ]))
5. {
6. $ var = $ _GET [ 'action' ];
7. }
8. ? >
9. < font style = "font-size: 20px" color = "# 000000" face = "Arial" > BL EDITOR: </ font >
10. <Br />
11. < form action = "update.php? actions = $ var" method = "get" >
12. < table >
13. < td >
14. < th >
15. List of Prohibitions: <br />
16. < textarea name = "$ var" style = "width: 300; height: 300" >
17. <? php
18. $ filename = "$ var.diff" ;
19. $ fp = fopen ($ filename , 'r' );
20. $ buffer = fread ($ fp , filesize ($ filename ));
21. $ buffer = str_replace ( "" , '' , $ buffer );
22. echo $ buffer ;
23. fclose ($ fp );
24. ? >
25. </ textarea >
26. </ th >
27. </ td >
28. </ table >
29. <Br />
30. < input type = "hidden" name = "action" value = "& # 60 ;? = $ var ? > " >
31. < input type = "submit" name = "update" value = "Save changes" >
32. </ form >
33. <br />
34. <a href = "#"> Home <a />

* This source code was highlighted with Source Code Highlighter .

4.4) After changing the lock files, run update.php, which writes the changes to the file.

1. <? php
2. header ( 'Content-Type: text / html; charset = UTF-8' );
3. $ var = "domains" ;
4. if ( isset ($ _GET [ 'action' ]))
5. {
6. $ var1 = $ _GET [ 'action' ];
7. }
8. // Add new data to the file
9. // domains list domains . diff
10. $ upd1 = $ _GET [ '$ var' ];
11. $ upd1 = str_replace ( "\ r" , '' , $ upd1 );
12. $ fd = "$ var1.diff" ;
13. $ fdomain = fopen ($ fd , "w +" );
14. fwrite ($ fdomain , $ upd1 );
15. fclose ($ fdomain );
16. echo "& # 60 ; b > all ok! </ b > <br /> ";
17. echo " <a href = index. php> Home </ a > ";
18. ? >

* This source code was highlighted with Source Code Highlighter .

That's all, you can try to test the result. I hope this experience or parts of it will be useful to someone. Successes.
PS: Criticism is welcome.
')