Proxy with Tor. Basis for multithreaded parsing
Hello! The interesting task arrived the other day: “Find free proxy servers”. Having taken it, I decided to go around all the sites of free proxies and understood - dead ... or with high ping.
After several hours of unsuccessful searches, it was decided to use their resources!
So, what should be the result:
')
1. VM Ubuntu / Debian 2 CPU, 2GB RAM, 8GB HDD (well, that is not demanding at all)
2. PRIVOXY for proxying requests (You can take nginx, varnish - I took Privoxy)
3. TOR server
In my experience I will say, 70 proxies spend about 30 megabits of the channel somewhere. We are quite satisfied. Perhaps your resources will be more demanding of traffic.
First of all we raise the necessary libraries:
In the future, we will clone the privoxy and tor scripts and run separate services, so the initial configuration is not needed.
Installation is over and let's start building a script for automatic service deployment.
Define the initial input parameters:
What will rise first, decide for yourself. I first pick up 99 copies of Tor, and later 99 copies of Privoxy.
In order for TOR to work on any particular port, you need to run this command.
But with privoxy, you will have to do terrible things: make duplicate configs and executable files.
I will cite once the execution scripts. In them I wrote everything in detail so as not to be mistaken.
You can start Tor and Privoxy like this:
Update streams like this:
And the Tor stream update script via the control port.
And do not forget to fasten monitoring!
PS You can use!
After several hours of unsuccessful searches, it was decided to use their resources!
So, what should be the result:
')
1. VM Ubuntu / Debian 2 CPU, 2GB RAM, 8GB HDD (well, that is not demanding at all)
2. PRIVOXY for proxying requests (You can take nginx, varnish - I took Privoxy)
3. TOR server
In my experience I will say, 70 proxies spend about 30 megabits of the channel somewhere. We are quite satisfied. Perhaps your resources will be more demanding of traffic.
First of all we raise the necessary libraries:
apt-get install tor privoxy expect In the future, we will clone the privoxy and tor scripts and run separate services, so the initial configuration is not needed.
Installation is over and let's start building a script for automatic service deployment.
Define the initial input parameters:
PID_END=99 # PRIVOXY_PORT=8100 # privoxy, 8100 8199 TOR_PORT=9100 # Tor, 9101 9199 TOR_CONTROL=20000 # Tor. , . BASE_IP=10.0.0.1 #, IP . . BASE_DIR=./data_tor # PID TOR BASE_DIR_PRIVOXY=/etc # Privoxy, 52 53 INSTANCE=$1 # . 99 What will rise first, decide for yourself. I first pick up 99 copies of Tor, and later 99 copies of Privoxy.
In order for TOR to work on any particular port, you need to run this command.
tor --RunAsDaemon 1 --CookieAuthentication 0 --HashedControlPassword "" --ControlPort $c_port --PidFile tor$i.pid --SocksPort $s_port --DataDirectory ${BASE_DIR}/tor$i But with privoxy, you will have to do terrible things: make duplicate configs and executable files.
I will cite once the execution scripts. In them I wrote everything in detail so as not to be mistaken.
You can start Tor and Privoxy like this:
./tor_privoxy_start.sh 50 Update streams like this:
./tor_privoxy_start.sh 50 update tor_privoxy_start.sh
#!/bin/bash #set -e # PID_END=99 # PRIVOXY_PORT=8100 # privoxy, 8100 8199 TOR_PORT=9100 # Tor, 9101 9199 TOR_CONTROL=20000 # Tor. , . BASE_IP=10.1.9.98 #, IP . . BASE_DIR=./data_tor # PID TOR BASE_DIR_PRIVOXY=/etc # Privoxy, 64 65 INSTANCE=$1 # . 99 ACTION=$2 UPDATE="update" if [ ! -d $BASE_DIR ]; then mkdir -p $BASE_DIR fi echo " :" echo "1 TOR Privoxy" echo "2 TOR" echo "3 TOR Privoxy" read doing case $doing in 1) # for i in $(seq 1 ${INSTANCE}); do p_port=$((PRIVOXY_PORT+i)) s_port=$((TOR_PORT+i)) c_port=$((TOR_CONTROL+i)) if [ $i -gt $PID_END ]; then echo " Treads , \n" break fi if [ ! -d "$BASE_DIR/tor${i}" ]; then echo " ${BASE_DIR}/tor${i}" mkdir -p "${BASE_DIR}/tor${i}" fi if [ ! -d "${BASE_DIR_PRIVOXY}/privoxy${i}" ]; then echo " ${BASE_DIR_PRIVOXY}/privoxy${i}" mkdir -p "${BASE_DIR_PRIVOXY}/privoxy${i}" fi cp -r ${BASE_DIR_PRIVOXY}/privoxy/* ${BASE_DIR_PRIVOXY}/privoxy$i # # echo "" > ${BASE_DIR_PRIVOXY}/privoxy$i/config; echo "forward-socks4a / 127.0.0.1:${s_port} ." >> ${BASE_DIR_PRIVOXY}/privoxy$i/config echo "confdir ${BASE_DIR_PRIVOXY}/privoxy${i}" >> ${BASE_DIR_PRIVOXY}/privoxy$i/config echo "logdir /var/log/privoxy/" >> ${BASE_DIR_PRIVOXY}/privoxy$i/config echo "logfile privoxy.log" >> ${BASE_DIR_PRIVOXY}/privoxy$i/config echo "#actionsfile standard" >> ${BASE_DIR_PRIVOXY}/privoxy$i/config echo "actionsfile default.action" >> ${BASE_DIR_PRIVOXY}/privoxy$i/config echo "actionsfile user.action" >> ${BASE_DIR_PRIVOXY}/privoxy$i/config echo "filterfile default.filter" >> ${BASE_DIR_PRIVOXY}/privoxy$i/config echo "debug 4096" >> ${BASE_DIR_PRIVOXY}/privoxy$i/config echo "debug 8192" >> ${BASE_DIR_PRIVOXY}/privoxy$i/config echo "listen-address ${BASE_IP}:${p_port}" >> ${BASE_DIR_PRIVOXY}/privoxy$i/config echo "user-manual /usr/share/doc/privoxy/user-manual" >> ${BASE_DIR_PRIVOXY}/privoxy$i/config echo "toggle 1" >> ${BASE_DIR_PRIVOXY}/privoxy$i/config echo "enable-remote-toggle 0" >> ${BASE_DIR_PRIVOXY}/privoxy$i/config echo "enable-edit-actions 0" >> ${BASE_DIR_PRIVOXY}/privoxy$i/config echo "enable-remote-http-toggle 0" >> ${BASE_DIR_PRIVOXY}/privoxy$i/config echo "buffer-limit 4096" >> ${BASE_DIR_PRIVOXY}/privoxy$i/config cp -r /etc/init.d/privoxy /etc/init.d/privoxy$i cp -r /usr/sbin/privoxy /usr/sbin/privoxy$i sed -i "s/NAME=privoxy/NAME=privoxy${i}/g" /etc/init.d/privoxy$i FROM="CONFIGFILE=\/etc\/privoxy\/config" TO="CONFIGFILE=\/etc\/privoxy${i}\/config" sed -i "s/$FROM/$TO/g" /etc/init.d/privoxy$i echo ": tor --RunAsDaemon 1 --CookieAuthentication 0 --HashedControlPassword \"\" --ControlPort $c_port --PidFile tor$i.pid --SocksPort $s_port --DataDirectory ${BASE_DIR}/tor$i" tor --RunAsDaemon 1 --CookieAuthentication 0 --HashedControlPassword "" --ControlPort $c_port --PidFile tor$i.pid --SocksPort $s_port --DataDirectory ${BASE_DIR}/tor$i echo "Update RC" update-rc.d privoxy$i defaults systemctl daemon-reload # /etc/init.d/privoxy$i restart done # netstat -4ln | grep $BASE_IP:80** | grep -Eo '10.{12}' > ./proxy_list.txt ;; 2) # echo " TOR\n" for i in $(seq 1 ${INSTANCE}); do c_port=$((TOR_CONTROL+i)) ./tor_renew.exp $c_port done ;; 3) for i in $(seq 1 ${PID_END}); do rm -r /etc/privoxy${i} rm /etc/init.d/privoxy${i} rm /usr/sbin/privoxy${i} done kill $(ps aux | grep -e 'tor' -e 'privoxy' | awk '{print $2}') ;; esac And the Tor stream update script via the control port.
tor_renew.exp
#!/usr/bin/expect -f set port [lindex $argv 0] spawn telnet 127.0.0.1 $port expect "Escape character is '^]'." send "AUTHENTICATE\r" expect "250 OK" send "signal NEWNYM\r" expect "250 OK" send "quit\r" And do not forget to fasten monitoring!
PS You can use!
Source: https://habr.com/ru/post/315002/
All Articles