πŸ“œ ⬆️ ⬇️

Failsafe data transfer node

Each broadband operator is thinking about how to release users to the Internet and intelligently limit the speed of the network according to the existing tariff plans and have a reserve in case of equipment failure or work related to equipment disconnection. I will try to tell and show by example how it is implemented by us (more than 3 thousand users are connected to us and the option I described works very well)


For a start, we are given:
1) 1U rack mount server (Intel Xeon E5335, 1GB Ram, Intel PRO / 1000 EB dual port NIC), we use two of these. For what - I will tell in the text of the article
2) Border router (in my case Juniper j4350, 2 pieces)
3) L3 switch supporting BGP or OSPF
4) And also the task: A) Having all this household to release users to the Internet.
B) to make a reserve, which in the event of an accident will be involved
automatically and without our participation

Edge Router (recommendations)
')
Let's start with setting up the border router.
It can be any one, even a PC under FreeBSD / Linux with the well-known Quagga package, or it can be hardware from Cisco or Juniper. How to configure a border router on a PC with a quagga or cisco package there are a lot of articles on the Internet and a lot of detailed documentation (yes, the quag configuration is identical to the cisco configuration). There is information on the juniper, too, not so much, but there is. And the topic of the article is not about setting up a border router and BGP protocol on it, but a little about something else.

In my case, two Juniper j4350s are used as border routers, the hardware is not very powerful, but it suits the current tasks and budget.
Each of these routers is installed on its own node (nodes in different places) and its uplink is connected to each of them. On the uplink side of the bgp protocol, we accept full-view and announce our networks to them. And on the shapers, we give on the internal bgp, in the specially allocated for this vlan-e only the default route (default route)
For a juniper, these are lines in the config file:
protocols bgp
neighbor 195.xxx.xxx.226 { export [ default-originate reject ];

policy options
policy-statement default-originate {
from {
route-filter 0.0.0.0/0 exact;
}
then accept;
},
policy-statement reject {
then reject;
} , .


Border routers are configured, each accepts full-view from uplinks and gives away our networks. Now we configure vlan for internal bgp, inside it we raise communication on the bgp protocol between border routers for an exchange of routes between them. We select IP addresses with future shapers, but preferably not one, but several, for a pool of external IP addresses used for NAT. In my case, there are 4 addresses for each shaper, but you can do more or less. Depending on the size of the network and the number of subscribers.

Setting up a server for shaping and NAT on FreeBSD

We take the server out of the box, and begin to install FreeBSD on it, better than the latest stable version. It is advisable to install FreeBSD in the most minimal configuration, not forgetting to mark the ports and the kernel source tree (you can delete it after setup)

After we set up FreeBSD, we start setting up a network on it. In /etc/resolv.conf we set the addresses of the dns servers and set the default route and IP addresses in the /etc/rc.conf file (despite the presence of bgp, the default route will never be superfluous, in case of problems with quag which we will put Further)
------ / etc / rc.conf ----
defaultrouter="195.xxx.xxx.225" #
gateway_enable="YES" #
hostname="gw2.xxx.ru" #
# em0 – c bgp
ifconfig_em0="up"
ifconfig_em0="inet 195.xxx.xx.231 netmask 255.255.255.240" # IP
ifconfig_em0_alias0="inet 195.xxx.xxx.232 netmask 255.255.255.255" # IP.
ifconfig_em0_alias1="inet 195.xxx.xxx.233 netmask 255.255.255.255" #
ifconfig_em0_alias2="inet 195.xxx.xxx.234 netmask 255.255.255.255"
#em1 – , vlan l3
ifconfig_em1="up"
ifconfig_em1="inet 195.xxx.xxx.193 netmask 255.255.255.248"

- The network has risen, Yandex has faltered. Now we are updating the ports tree
The ports tree can be updated using the portsnap utility or csup
In our case, we will update via portsnap. To do this, write the command:
portsnap fetch, this will load the complete ports tree into the / var / db / portsnap directory
portsnap extract, with this command we will unpack the new ports tree into the / usr / ports directory

After upgrading the ports, it makes sense to build your own kernel, although it is not necessary to do this, ipfw, dummynet and pf can be loaded by modules, and the reassembled kernel does not particularly affect performance. In our case, kernel reassembly is needed because of the HZ option, this is the timer interval. By default it costs 100, which is not very suitable for high speeds, so we will change it to 2000

Copy the distributive kernel config to another file
cp / usr / src / sys / i386 / conf / GENERIC / usr / src / sys / i386 / conf / router-001
open the file router-001 with a text editor (vi, nano) and begin to edit it. After editing the kernel for your needs, you need to remember to add the following parameters to the kernel config:

#ipfw firewall ,
options IPFIREWALL
options IPFIREWALL_VERBOSE
options IPFIREWALL_VERBOSE_LIMIT=100
options IPFIREWALL_FORWARD
options DUMMYNET
options HZ=2000

# pf, NAT- IP
device pf
device pflog
device pfsync


Next, using the make buildkernel and make installkernel commands, we build and install the kernel

After building the kernel, go to setting up our ipfw firewall
Add the following lines to the /etc/rc.conf file:
firewall_enable="YES"
pf_enable="YES"
pf_rules="/etc/pf.conf"

we completely clear the /etc/rc.firewall file and start writing our own rules into it, there is nothing complicated so rc.firewall is the most common shell script that starts when ipfw starts

the lines that we add to the new /etc/rc.firewall
#!/bin/sh ,

/sbin/ipfw -q flush
/sbin/ipfw -q pipe flush , dummynet
fwcmd="/sbin/ipfw -q" , ipfw . –q ,

WAN_IP = 195.xxx.xxx.231, IP
IBGP_NET = 195.xx.xx.224/xx
LAN_IP = 10.249.0.0/16 ,

${fwcmd} add 10 allow ip from any to any via lo0 ,
loopback e

${fwcmd} add 11 allow ip from me to me ,

${fwcmd} add 12 allow icmp from any to me, icmp

${fwcmd} add 20 allow tcp from table(1) to me dst-port 22 ,
ssh IP 1.
ipfw table 1 add <ip_address>

${fwcmd} add 21 deny tcp from any to me dst-port 22 , ssh.
, 1

${fwcmd} add 30 pipe tablearg ip from any to table(2) out via em1
${fwcmd} add 31 pipe tablearg ip from table(3) to any in via em1 , 2 3 . , (2 , 3 – ). ipfw table 2 add <ip > < > ipfw table 2 add <ip > < >

,
- ( nginx)
${fwcmd} add 38 fwd 127.0.0.1,3128 tcp $LAN_NET to not me dst-port 80
${fwcmd} add 39 allow tcp from any $LAN_NET src-port 80

${fwcmd} add 40 deny all from $LAN_NET to not me
${fwcmd} add 41 deny all from not me to $LAN_NET , . 2 3

${fwcmd} add 50 allow ip from me to any keep-state , .

${fwcmd} add 51 allow tcp from $IBGP_NET to $WAN_IP dst-port 179 , vlan BGP 179 (bgp )

${fwcmd} add 52 allow ospf from 195.xxx.xxx.192/29 to any , ospf
,
:
#Speed 15Mbps,
${fwcmd} pipe 1 config mask dst-ip 0xffffffff bw 16000Kbit/s
${fwcmd} pipe 101 config mask src-ip 0xffffffff bw 16000Kbit/s

#Speed 20Mbps,
${fwcmd} pipe 2 config mask dst-ip 0xffffffff bw 21000Kbit/s
${fwcmd} pipe 102 config mask src-ip 0xffffffff bw 21000Kbit/s

#Speed 3Mbps,
${fwcmd} pipe 3 config mask dst-ip 0xffffffff bw 3500Kbit/s
${fwcmd} pipe 103 config mask src-ip 0xffffffff bw 3500Kbit/s

In any case, we govern everything for ourselves (do not forget that the packet will first be covered by ipfw, and only then it will go on to pf processing) in accordance with these recommendations, we consider this firewall configured. In principle, if you do not want to use tablearg, then you can not use it. There are more options for pipe per customer or a separate table for the tariff. I do not consider these options, due to the high load on the equipment (pipe per customer) and inconvenience (table for tariff)

For completeness, it remains only to configure the NAT, which we will do through pf.
Create the file /etc/pf.conf and enter the following lines into it

WAN_IF = "em0", this is our external interface in a vlan with an internal bgp. It is on it that we will produce NAT

LOCAL_NET = "10.249.0.0/16", this is the range of our local network which we will NAT

nat on $ WAN_IF from $ LOCAL_NET to! $ LOCAL_NET -> ($ WAN_IF) is a round-robin sticky-address, and this is the rule for NAT from the internal network to the next interface
the round-robin parameter means using the pool addresses in a circle.
The sticky-address parameter is used to ensure that the same source address is always assigned to the pool address.

Now we are setting up the quagga package.
From him we need bgp support and ospf support in my case

I will not publish Zebra.conf, everything is simple and of the same type. We describe interfaces and static routes, here is a clipping from it:

i nterface em0
ip address 195.xx.xxx.231/xxx
description ibgp
!
interface em1
ip address 195.xxx.xxx.193/29
description ospf
!
! Static routes.
!
ip route 10.0.0.0/8 Null0 254
ip route 79.142.80.132/30 195.xxx.xxx.238
ip route 94.124.180.57/30 195.xxx.xxx.225
nterface em0
ip address 195.xx.xxx.231/xxx
description ibgp
!
interface em1
ip address 195.xxx.xxx.193/29
description ospf
!
! Static routes.
!
ip route 10.0.0.0/8 Null0 254
ip route 79.142.80.132/30 195.xxx.xxx.238
ip route 94.124.180.57/30 195.xxx.xxx.225


We bring bgp config approximately here to such look. That is, we accept the default route from the border routers and give it any specific routes.

hostname gw2.xxx.ru
password rxxxx
enable password rxxxxx
log file /var/log/quagga/bgpd.log
!
router bgp 3333
bgp router-id 195.xxx.xxx.231
bgp log-neighbor-changes
network 195.xxx.xxx.192/29
neighbor 195.xxx.xxx.225 remote-as 3333
neighbor 195.xxx.xxx.225 description j4350-1
neighbor 195.xxx.xxx.225 next-hop-self
neighbor 195.xxx.xxx.238 remote-as 3333
neighbor 195.xxx.xxx.238 description j4350-2
neighbor 195.xxx.xxx.238 next-hop-self
!

Next, we configure the OSPF protocol (used in my case, since Dlink DGS-3612 is installed in parallel with the Cisco Catalyst 3560)

Hostname gw2.xxx.ru
password xxx
enable password xxxx
log file /var/log/quagga/ospfd.log
!
interface em1
!
router ospf
ospf router-id 195.xxx.xxx.193
network 195.xxx.xxx.192/29 area 0.0.0.0
default-information originate metric 100
!
line vty


If you have a switch that can work with BGP, then it is better to stop using OSPF and use BGP for this purpose. In this case,
bgpd.conf need to make the following lines

neighbor 195.xxx.xxx.195 remote-as 3333
neighbor 195.xxx.xxx.195 description sw-c3560-xxx.ru
neighbor 195.xxx.xxx.195 default-originate


It seems everything is set up, now the main thing is not to forget the small tuning sysctl
Add the following lines to the existing /etc/sysctl.conf
net.inet.ip.forwarding=1 #
net.inet.ip.fw.one_pass=1 # ipfw
net.inet.icmp.bmcastecho=0 #
net.inet.tcp.blackhole=2 # ( )
net.inet.udp.blackhole=1 # ( )
net.inet.ip.dummynet.io_fast=1 # dummynet( )
net.inet.icmp.drop_redirect=1 # icmp
net.inet.icmp.log_redirect=1 # icmp log,
net.inet.ip.redirect=0 # 0, ICMP REDIRECT
net.inet.ip.dummynet.expire=0 #
#
net.inet.ip.dummynet.hash_size=16384 # -, dummynet
# .
#dummynet # ,
#


At this, our shaper is considered complete, I have lowered its integration with the billing system in advance, since each has its own and its own methods
You can safely write the reboot command and if everything was correctly configured then you can proceed to configure the level 3 switch (l3)

And if we also want to redirect users who are not allowed to access the Internet on a specific page with information about blocking, then put the web server nginx and write to it in the config (nginx.conf) the following text

user nobody;
worker_processes 2;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 35;
server {
listen 3128;
server_name wkt_router;
charset windows-1251;
access_log /dev/null;
rewrite ^(.*) blocked.wktnet.ru/index.htm permanent;
}
}

Configuring Level 3 Switch

My network uses Cisco WS-C3560G-24-TS-S, everything is simple
Go to configuration mode: conf t
We assign an IP address to the vlan interface, within which ospf runs, (the same network as on the shaper on the em1 interface)
And set up ospf

router ospf 1
router-id 195.xxx.xxx.195
log-adjacency-changes
network 10.249.33.0 0.0.0.255 area 0.0.0.0
network 10.249.42.0 0.0.0.255 area 0.0.0.0
network 10.249.51.0 0.0.0.255 area 0.0.0.0
network 10.249.55.0 0.0.0.255 area 0.0.0.0


save the config with the wr command and enjoy the achieved result

if instead of the ospf protocol you decide to use bgp, then the following lines should be added to the switch config

router bgp 3333
bgp log-neighbor-changes
network 10.249.33.0 255.255.255.0
redistribute ospf 1
neighbor 195.xxx.xxx.193 remote-as 3333
neighbor 195.xxx.xxx.193 description gw


If you use a D-link l3 switch, then configure OSPF with the following commands:

enable ospf, enable ospf on the switch
config ospf router_id 195.xxx.xxx.196, specify the router id
config ospf ipif <interface> area 0.0.0.0 state enable, enable ospf on the interface

and save the config with the save command

Setting up a backup server

On the neighboring (backup shaper) we make the same configuration, except for the ospf config and IP addresses
The difference is that the default-information originate we set the metric 200, then it becomes a backup. Although we can leave the metric 100 and get load balancing between the shapers, but I don’t use this option in my network because with this setting, customers will get double speed (at the tariff rate on each shaper).

Conclusion

As a result, we get a system with dynamic routing (in case of failure of one of the shapers or border routers, we automatically begin to use the neighboring one, all the routes of the network specified on the level 3 switches appear automatically in the routing table on our shapers). If you are frightened by the thought of an idle server, then you can play around with the settings, change ospf to bgp, install another L3 switch and release part of the network through it.

Scheme of the resulting node:
image

For example, give your rc.firewall

cat /etc/rc.firewall
#!/bin/sh
#Flush all firewall rules
/sbin/ipfw -q flush
/sbin/ipfw -q pipe flush
#Setting firewall path and options for working with rules
fwcmd="/sbin/ipfw -q"
#variables
IBGP_NET="195.93.xxx.xxx/28" #iBGP network (vlan9)
WAN_IP="195.93.xxx.xxx" #Primary WAN ip address
LAN_IP="195.93.xxx.xxx" #Primary LAN ip address

###System rules
${fwcmd} add 10 allow ip from any to any via lo0 #do not filter loobpack
${fwcmd} add 11 allow ip from me to me #allow packets from this host to this host
${fwcmd} add 12 allow icmp from any to me #allow ICMP

#allow ssh connections
${fwcmd} add 20 allow tcp from table\(6\) to me dst-port 22 #Allow SSH connections (table 6)
${fwcmd} add 29 deny tcp from any to me dst-port 22

#allowing users and add his ip to shaping pipe
${fwcmd} add 30 pipe tablearg ip from any to table\(1\) out via em1
${fwcmd} add 31 pipe tablearg ip from table\(2\) to any in via em1

#Block spammerss
${fwcmd} add 34 deny ip from table\(3\) to any dst-port 25 #for auto block spammers

#allow connections to this networsk
${fwcmd} add 40 allow all from table\(7\) to any
${fwcmd} add 41 allow all from any to table\(7\)

#allow active users
${fwcmd} add 45 allow all from not me to table\(1\)
${fwcmd} add 46 allow all from table\(2\) to not me

#By default block users
${fwcmd} add 48 fwd 127.0.0.1,3128 tcp from table\(8\) to not me dst-port 80
${fwcmd} add 49 allow tcp from any to table\(8\) src-port 80
${fwcmd} add 50 deny all from table\(8\) to not me
${fwcmd} add 51 deny all from not me to table\(8\)

###Access rules
#allow outgoing connections
${fwcmd} add 60 allow ip from me to any keep-state #allow all ougoing packets and keep state

#Rules allowing SNMP
${fwcmd} add 61 allow udp from table\(6\) to $WAN_IP dst-port 161 #Allow SNMP

#Rules allowing bgp
${fwcmd} add 64 allow tcp from $IBGP_NET to $WAN_IP dst-port 179 #Allow BGP from iBGP network (vlan 9)

###############################################
# 6 - ssh,telnet snmp
${fwcmd} table 6 add 195.93.xxx.xxx #wkt office
${fwcmd} table 6 add 89.xxx.xxx.1 #
${fwcmd} table 6 add 93.xx.xxx.xxx #

# 7 - IP
${fwcmd} table 7 add 195.xxx.xx.0/25 #binat
${fwcmd} table 7 add 195.xx.xx.0/26 #servers dmz
${fwcmd} table 7 add 195.xx3.xx.192/29 #int net
${fwcmd} table 7 add 195.xx.xxx.xxx #c3560-b51
${fwcmd} table 7 add 10.88.88.1
# 8 -
${fwcmd} table 8 add 10.87.0.0/16
${fwcmd} table 8 add 10.88.0.0/16
${fwcmd} table 8 add 10.249.0.0/16
${fwcmd} table 8 add 10.90.0.0/16
${fwcmd} table 8 add 195.93.xxx.0/25
${fwcmd} table 8 add 195.93.xxx.128/25
${fwcmd} table 8 add 195.93.xxx.64/26
${fwcmd} table 8 add 195.93.xxx.200/28

# 1 2 -
##WKT tech
${fwcmd} table 1 add 10.87.xx.250/32 55
${fwcmd} table 2 add 10.87.xx.250/32 255
${fwcmd} table 1 add 10.87.xx.251/32 55
${fwcmd} table 2 add 10.87.xx.251/32 255


And bgpd.conf

hostname gw2.xxxx
password rxxxx
enable password xxx
log file /var/log/quagga/bgpd.log
!
router bgp 44xxx
bgp router-id 195.93.xxx.xxx
bgp log-neighbor-changes
network 195.93.2xx.xxx/29
neighbor 195.93.206.xx remote-as 44380
neighbor 195.93.206.xx description j4350-b51
neighbor 195.93.206.xx next-hop-self
neighbor 195.93.206.xx remote-as 44380
neighbor 195.93.206.xx description j4350-k18
neighbor 195.93.206.xx next-hop-self
neighbor 195.93.206.xx remote-as 65000
neighbor 195.93.206.xxdefault-originate
neighbor 195.93.206.xx description sw-c3560g-24ts-b51
neighbor 195.93.206.xx route-map c3560g-b51-in in
neighbor 195.93.206.xx remote-as 65001
neighbor 195.93.206.xx default-originate
neighbor 195.93.206.xx description sw-c3560g-24ts-k18
neighbor 195.93.206.xx route-map c3560g-k18-in in
!
route-map c3560g-k18-in permit 10
set local-preference 200
!
route-map c3560g-b51-in permit 10
set local-preference 300


UPD: in order to write kern.hz = β€œ2000” in / boot / loader.conf, it is not necessary to recompile the kernel.

Source: https://habr.com/ru/post/82656/


All Articles