Multihome Policy-Based Routing on pf
We continue to publish useful articles about difficult things.
Today we will talk about the publication of various services through several communication providers at the same time.
Due to the fact that over time almost all organizations have additional channels of communication for reservations or other needs, the question arises: “Is it possible to use these channels of communication for the simultaneous publication of corporate services?”
Some time ago this issue arose in our company, so it was decided to rebuild the outer perimeter.
In our case, there were 4 providers and the following list of services:
The initial wiring diagram looked like this.
I think the majority of those who publish services, it looks about the same.
The scheme is inconvenient because with the disappearance of communication with one provider, access to the service tied to this provider is lost.
I really wanted to get away from using a heap of external servers and preserve the convenience of configuring the system.
And build the following.
By trial and error, this is an interesting config for pf.
')
Now a little explanation on config.
frontend - nginx server in http, https, smtp, imap, pop3 proxy mode. At the same time, multi-domain maintenance of IMAP, SMTP, POP3 services is provided.
jabber - XMPP messaging server
vpn - openvpn server
The principle of operation is quite simple and is based on tagging packets when passing pf rules.
A packet arriving at a published port receives a label when it passes through a specific incoming interface of the router. Then this tagged packet is forwarded to the recipient server, while the label of the departed packet is stored in the state table. When a packet is returned from the recipient's server, a label is calculated and the packet is sent back to the interface and the gateway from which it came. Thus, the integrity of the session is maintained.
The default gateway for published servers is the router on which the above rules are processed.
ATTENTION! This scheme does not imply the use of any services on the server balancer!
The server balancer should deal only with handling incoming and outgoing connections! Attempting to publish services located on the balancer server will cause problems with the unavailability of the published service. This is due to the fact that when using a local service, labels are lost and the response to incoming packets goes to the interface and the default gateway configured on the balancer
Something like that. There will be questions - write.
Aborche 2013
Today we will talk about the publication of various services through several communication providers at the same time.
Due to the fact that over time almost all organizations have additional channels of communication for reservations or other needs, the question arises: “Is it possible to use these channels of communication for the simultaneous publication of corporate services?”
Some time ago this issue arose in our company, so it was decided to rebuild the outer perimeter.
In our case, there were 4 providers and the following list of services:
- HTTP services (about 60 sites)
- XMPP service
- HTTPS Corporate Mail Server (Exchange)
- VPN service
- Ssh
- IMAP, IMAPS, POP, POP3S, SMTP, SMTPS
- Owncloud
The initial wiring diagram looked like this.
I think the majority of those who publish services, it looks about the same.
The scheme is inconvenient because with the disappearance of communication with one provider, access to the service tied to this provider is lost.
I really wanted to get away from using a heap of external servers and preserve the convenience of configuring the system.
And build the following.
By trial and error, this is an interesting config for pf.
int_if="vlan420" ext1_if="vlan410" ext2_if="vlan400" ext3_if="vlan440" # External Gateways ext1_gw="83.0.0.33" ext2_gw="89.0.0.113" ext3_gw="212.0.0.241" out_gates="(vlan410 83.0.0.33), (vlan400 89.0.0.113), (vlan440 212.0.0.241)" #out_gates="(vlan400 89.0.0.113), (vlan440 212.0.0.241)" #out_gates="(vlan440 212.0.0.241), (vlan410 83.0.0.33)" # External IP for WWW,MAIL,XMPP ext_wan1="83.0.0.43" ext_wan2="89.0.0.126" ext_wan3="212.0.0.244" # WWW, Mail Frontend/Proxy server frontend="192.168.50.34" jabber="192.168.50.22" #VPN Server vpn="172.17.2.2" table <ournets> persist { 10.0.0.0/22, 192.168.50.0/24 } table <bruteforce> persist table <ossec_fwtable> persist # ossec_fwtable table <allowed_out> persist { } set state-policy floating set block-policy drop set optimization normal set require-order yes set timeout { interval 10, frag 30 } set timeout { tcp.first 120, tcp.opening 30, tcp.established 86400 } set timeout { tcp.closing 900, tcp.finwait 45, tcp.closed 90 } set timeout { udp.first 60, udp.single 30, udp.multiple 60 } set timeout { icmp.first 20, icmp.error 10 } set timeout { other.first 60, other.single 30, other.multiple 60 } set timeout { adaptive.start 0, adaptive.end 0 } set limit { states 100000, frags 50000 } set limit table-entries 500000 set fingerprints "/etc/pf.os" set loginterface $int_if set skip on lo0 # fragment reassemble scrub in all scrub out all fragment reassemble max-mss 1400 rdr on $ext1_if proto tcp from any to $ext_wan1 port ssh tag SSH_WAN1 -> $frontend port 2200 rdr on $ext2_if proto tcp from any to $ext_wan2 port ssh tag SSH_WAN2 -> $frontend port 2200 rdr on $ext3_if proto tcp from any to $ext_wan3 port ssh tag SSH_WAN3 -> $frontend port 2200 rdr on $ext1_if proto tcp from any to $ext_wan1 port { http, https } tag HTTP_WAN1 -> $frontend rdr on $ext2_if proto tcp from any to $ext_wan2 port { http, https } tag HTTP_WAN2 -> $frontend rdr on $ext3_if proto tcp from any to $ext_wan3 port { http, https } tag HTTP_WAN3 -> $frontend rdr on $ext1_if proto tcp from any to $ext_wan1 port 8083 tag HTTP_WAN1 -> $frontend rdr on $ext2_if proto tcp from any to $ext_wan2 port 8083 tag HTTP_WAN2 -> $frontend rdr on $ext3_if proto tcp from any to $ext_wan3 port 8083 tag HTTP_WAN3 -> $frontend rdr on $ext1_if proto tcp from any to $trade_wan1 port http tag HTTP_WAN1 -> $frontend rdr on $ext2_if proto tcp from any to $trade_wan2 port http tag HTTP_WAN2 -> $frontend rdr on $ext3_if proto tcp from any to $trade_wan3 port http tag HTTP_WAN3 -> $frontend rdr on $ext1_if proto tcp from any to $trade_wan1 port https tag HTTP_WAN1 -> $frontend port 8043 rdr on $ext2_if proto tcp from any to $trade_wan2 port https tag HTTP_WAN2 -> $frontend port 8043 rdr on $ext3_if proto tcp from any to $trade_wan3 port https tag HTTP_WAN3 -> $frontend port 8043 rdr on $ext1_if proto tcp from any to $ext_wan1 port { pop3, imap, pop3s, imaps, smtp, smtps, submission } tag MAIL_WAN1 -> $frontend rdr on $ext2_if proto tcp from any to $ext_wan2 port { pop3, imap, pop3s, imaps, smtp, smtps, submission } tag MAIL_WAN2 -> $frontend rdr on $ext3_if proto tcp from any to $ext_wan3 port { pop3, imap, pop3s, imaps, smtp, smtps, submission } tag MAIL_WAN3 -> $frontend rdr on $ext1_if proto tcp from any to $ext_wan1 port { xmpp-client, xmpp-server } tag JABBER_WAN1 -> $jabber rdr on $ext2_if proto tcp from any to $ext_wan2 port { xmpp-client, xmpp-server } tag JABBER_WAN2 -> $jabber rdr on $ext3_if proto tcp from any to $ext_wan3 port { xmpp-client, xmpp-server } tag JABBER_WAN3 -> $jabber #VPN access rdr on $ext1_if proto udp from any to $ext_wan1 port 1200 tag VPN_WAN1 -> $vpn port 1200 rdr on $ext2_if proto udp from any to $ext_wan2 port 1200 tag VPN_WAN2 -> $vpn port 1200 rdr on $ext3_if proto udp from any to $ext_wan3 port 1200 tag VPN_WAN3 -> $vpn port 1200 # Allow JABBER outgoing connections nat on $ext1_if from $jabber to any -> $ext_wan1 nat on $ext2_if from $jabber to any -> $ext_wan2 nat on $ext3_if from $jabber to any -> $ext_wan3 # Allow FRONTWAVE outgoing nat on $ext1_if from $frontend to any -> $ext_wan1 nat on $ext2_if from $frontend to any -> $ext_wan2 nat on $ext3_if from $frontend to any -> $ext_wan3 # Allow whitelisted hosts nat on $ext1_if from <allowed_out> to any -> $ext_wan1 nat on $ext2_if from <allowed_out> to any -> $ext_wan2 nat on $ext3_if from <allowed_out> to any -> $ext_wan3 # Allow VPNGW outgoing nat on $ext1_if from $vpn to any -> $ext_wan1 nat on $ext2_if from $vpn to any -> $ext_wan2 nat on $ext3_if from $vpn to any -> $ext_wan3 # block unwanted hosts block in quick from <bruteforce> block in quick from <ossec_fwtable> # block anything by default block in log block out log # Allow ICMP on external interfaces pass in quick on $int_if proto icmp from <ournets> to ($int_if) keep state # Allow SSH from LAN subnets pass in quick on $int_if proto tcp from <ournets> to ($int_if) port ssh keep state # Allow outgoing to trusted hosts <allowed_out> pass in quick on $int_if route-to { $out_gates } proto tcp from <allowed_out> to any flags S/SA modulate state pass in quick on $int_if route-to { $out_gates } proto { udp, icmp } from <allowed_out> to any keep state # Allow JABBER outgoing connections pass in quick on $int_if route-to { $out_gates } proto tcp from $jabber to any flags S/SA modulate state pass in quick on $int_if route-to { $out_gates } proto { udp, icmp } from $jabber to any keep state # Allow FRONTWAVE outgoing connections #pass in quick on $int_if route-to { $out_gates } proto tcp from $frontend to any flags S/SA modulate state #pass in quick on $int_if route-to { $out_gates } proto { udp, icmp } from $frontend to any keep state # Allow VPNGW port 1200 to any pass in quick on $int_if route-to { $out_gates } proto { tcp, udp } from $vpn port 1200 to any flags S/SA modulate state # Allow ICMP on external interfaces pass in quick on $ext1_if reply-to ($ext1_if $ext1_gw) proto icmp from any to ($ext1_if) keep state pass in quick on $ext2_if reply-to ($ext2_if $ext2_gw) proto icmp from any to ($ext2_if) keep state pass in quick on $ext3_if reply-to ($ext3_if $ext3_gw) proto icmp from any to ($ext3_if) keep state # Allow SSH/SFTP pass in on $ext1_if reply-to ($ext1_if $ext1_gw) proto tcp from any to $frontend port 2200 tagged SSH_WAN1 keep state pass in on $ext2_if reply-to ($ext2_if $ext2_gw) proto tcp from any to $frontend port 2200 tagged SSH_WAN2 keep state pass in on $ext3_if reply-to ($ext3_if $ext3_gw) proto tcp from any to $frontend port 2200 tagged SSH_WAN3 keep state pass out on $int_if proto tcp from any to $frontend port 2200 keep state # Allow HTTP/HTTPS pass in on $ext1_if reply-to ($ext1_if $ext1_gw) proto tcp from any to $frontend port { http, https } tagged HTTP_WAN1 keep state pass in on $ext2_if reply-to ($ext2_if $ext2_gw) proto tcp from any to $frontend port { http, https } tagged HTTP_WAN2 keep state pass in on $ext3_if reply-to ($ext3_if $ext3_gw) proto tcp from any to $frontend port { http, https } tagged HTTP_WAN3 keep state pass out on $int_if proto tcp from any to $frontend port { http, https } keep state # Allow HTTPS on owncloud pass in on $ext1_if reply-to ($ext1_if $ext1_gw) proto tcp from any to $frontend port 8083 tagged HTTP_WAN1 keep state pass in on $ext2_if reply-to ($ext2_if $ext2_gw) proto tcp from any to $frontend port 8083 tagged HTTP_WAN2 keep state pass in on $ext3_if reply-to ($ext3_if $ext3_gw) proto tcp from any to $frontend port 8083 tagged HTTP_WAN3 keep state pass out on $int_if proto tcp from any to $frontend port 8083 keep state # Alow HTTPS on mx pass in on $ext1_if reply-to ($ext1_if $ext1_gw) proto tcp from any to $frontend port 8043 tagged HTTP_WAN1 keep state pass in on $ext2_if reply-to ($ext2_if $ext2_gw) proto tcp from any to $frontend port 8043 tagged HTTP_WAN2 keep state pass in on $ext3_if reply-to ($ext3_if $ext3_gw) proto tcp from any to $frontend port 8043 tagged HTTP_WAN3 keep state pass out on $int_if proto tcp from any to $frontend port 8043 keep state # Allow IMAP/POP3/SMTP pass in on $ext1_if reply-to ($ext1_if $ext1_gw) proto tcp from any to $frontend port { pop3, imap, pop3s, imaps, smtp, smtps, submission } tagged MAIL_WAN1 keep state pass in on $ext2_if reply-to ($ext2_if $ext2_gw) proto tcp from any to $frontend port { pop3, imap, pop3s, imaps, smtp, smtps, submission } tagged MAIL_WAN2 keep state pass in on $ext3_if reply-to ($ext3_if $ext3_gw) proto tcp from any to $frontend port { pop3, imap, pop3s, imaps, smtp, smtps, submission } tagged MAIL_WAN3 keep state pass out on $int_if proto tcp from any to $frontend port { pop3, imap, pop3s, imaps, smtp, smtps, submission } keep state # Incoming VPN connection from any pass in on $ext1_if reply-to ($ext1_if $ext1_gw) proto { tcp, udp } from any to $vpn port 1200 tagged VPN_WAN1 keep state pass in on $ext2_if reply-to ($ext2_if $ext2_gw) proto { tcp, udp } from any to $vpn port 1200 tagged VPN_WAN2 keep state pass in on $ext3_if reply-to ($ext3_if $ext3_gw) proto { tcp, udp } from any to $vpn port 1200 tagged VPN_WAN3 keep state pass out on $int_if proto { tcp, udp } from any to $vpn port 1200 keep state # Allow JABBER/XMPP pass in on $ext1_if reply-to ($ext1_if $ext1_gw) proto tcp from any to $jabber port { xmpp-client, xmpp-server } tagged JABBER_WAN1 keep state pass in on $ext2_if reply-to ($ext2_if $ext2_gw) proto tcp from any to $jabber port { xmpp-client, xmpp-server } tagged JABBER_WAN2 keep state pass in on $ext3_if reply-to ($ext3_if $ext3_gw) proto tcp from any to $jabber port { xmpp-client, xmpp-server } tagged JABBER_WAN3 keep state pass out on $int_if proto tcp from any to $jabber port { xmpp-client, xmpp-server } keep state # Allow outbound pass out on $ext1_if route-to ($ext1_if $ext1_gw) from ($ext1_if) to any keep state pass out on $ext2_if route-to ($ext2_if $ext2_gw) from ($ext2_if) to any keep state pass out on $ext3_if route-to ($ext3_if $ext3_gw) from ($ext3_if) to any keep state pass out on $ext1_if route-to ($ext1_if $ext1_gw) from 83.0.0.43 to any keep state pass out on $ext2_if route-to ($ext1_if $ext1_gw) from 83.0.0.43 to any keep state pass out on $ext3_if route-to ($ext1_if $ext1_gw) from 83.0.0.43 to any keep state pass out on $ext1_if route-to ($ext2_if $ext2_gw) from 89.0.0.126 to any keep state pass out on $ext2_if route-to ($ext2_if $ext2_gw) from 89.0.0.126 to any keep state pass out on $ext3_if route-to ($ext2_if $ext2_gw) from 89.0.0.126 to any keep state pass out on $ext1_if route-to ($ext3_if $ext3_gw) from 212.0.0.244 to any keep state pass out on $ext2_if route-to ($ext3_if $ext3_gw) from 212.0.0.244 to any keep state pass out on $ext3_if route-to ($ext3_if $ext3_gw) from 212.0.0.244 to any keep state ')
Now a little explanation on config.
frontend - nginx server in http, https, smtp, imap, pop3 proxy mode. At the same time, multi-domain maintenance of IMAP, SMTP, POP3 services is provided.
jabber - XMPP messaging server
vpn - openvpn server
The principle of operation is quite simple and is based on tagging packets when passing pf rules.
A packet arriving at a published port receives a label when it passes through a specific incoming interface of the router. Then this tagged packet is forwarded to the recipient server, while the label of the departed packet is stored in the state table. When a packet is returned from the recipient's server, a label is calculated and the packet is sent back to the interface and the gateway from which it came. Thus, the integrity of the session is maintained.
The default gateway for published servers is the router on which the above rules are processed.
ATTENTION! This scheme does not imply the use of any services on the server balancer!
The server balancer should deal only with handling incoming and outgoing connections! Attempting to publish services located on the balancer server will cause problems with the unavailability of the published service. This is due to the fact that when using a local service, labels are lost and the response to incoming packets goes to the interface and the default gateway configured on the balancer
Something like that. There will be questions - write.
Aborche 2013
Source: https://habr.com/ru/post/176963/
All Articles