Tuesday, August 18, 2009

Split Routing Over 2 DSL Lines

Its been a bit since I've blogged last. Most of it is due to moving. We moved from Raleigh NC to a small town in the western part of NC called Union Mills. The good thing is we are on a farm, we have lots of land, space, fresh air... however the Net connection just plain sucks.

I'm 2.3 miles from the CO, so the best I can get with DSL is a 3m/384k connection thru AT&T. So I called AT&T commercial and was assured I could be 2x bonded ADSL lines that would give me effectively a 6mb connection. Guess what? The fibbed. In the end I had to settle for 2x ADSL lines not bonded, and I elected for the non-commercial option since it was far cheaper.

Now came the big question how to best effectively use em'. After hitting up Google I found lots of interesting solutions. Most were very complex, routing various protocols out this interface or that.... I wanted something that would give me the closest to a load balanced connection as possible.

I'm using a old HP Desktop with 3 network cards as my gateway router running Jaunty 9.04 Server Edition.

Below is what I came up with using ip route and iptables. I put in bogus IPs to keep it as real as possible.

First I added two routing tables to the /etc/iproute2/rt_tables file:

1 line1
2 line2

Then I found a script on the net and used it as a template and hacked it up as so:

#!/bin/bash

# DSL Lines are IF0 & IF1, IF2 is local net.
IF0=eth0
IF1=eth1
IF2=eth2

# IP Addr on Gateway matching interfaces above
IP0=192.168.1.1
IP1=192.168.2.1
IP2=172.31.0.1

# DSL IPs
P0=192.168.1.254
P1=192.168.2.254

# Network addresses
P0_NET=192.168.1.0
P1_NET=192.168.2.0

# Routing table entries
T1=1
T2=2

# Set up routes
ip route add $P0_NET dev $IF0 src $IP0 table $T1
ip route add default via $P0 table $T1
ip route add $P1_NET dev $IF1 src $IP1 table $T2
ip route add default via $P1 table $T2

ip route add $P0_NET dev $IF0 src $IP0
ip route add $P1_NET dev $IF1 src $IP1

# Set up default route to balance between both interfaces
ip route add default scope global nexthop via $P0 dev $IF0 weight 1 \
nexthop via $P1 dev $IF1 weight 1

# Add the rules for the routing tables
ip rule add from $IP0 table $T1
ip rule add from $IP1 table $T2

# Now for the masq bits
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t filter -F
iptables -t filter -X
#iptables -t nat -A POSTROUTING -o $IF0 -j MASQUERADE
#iptables -t nat -A POSTROUTING -o $IF1 -j MASQUERADE
iptables -t nat -I POSTROUTING -s 172.31.0.0/24 -j MASQUERADE

# Turn on ip_forwarding
echo 1 > /proc/sys/net/ipv4/ip_forward

iptables -A FORWARD -i $IF2 -s 172.31.0.0/24 -j ACCEPT
iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

# Flush routing every four hours
echo 144000 > /proc/sys/net/ipv4/route/secret_interval

It works like this... Each time a connection is established outbound, the server will decide which interface to route it out of. This should be a rough 50/50 split. The server decides based on congestion and a few other factors. Only one connection can only ever be established over the same route.

If I was start a download, it will only ever utilize a single connection, another download from the same IP will also utilize the same connection. This is due to the kernel having already cached the route. If a 3rd download to a new server was to be started it will likely be established over the second connection due to the first route being congested and the second route is idle.

The route will be flushed 4 hours after the first download completes, then a new decision with be made on how to contact the original server.

I've been using this for a bit now and it seems to do what I need it to do... give me a faster response time when I have intensive net operations going on, like vonage & rsync's to offsite machines.

By no means am I a iptables or routing guru. If anyone has any other suggestions or better ways to do it I'd love to hear them.

~pete

4 comments:

agrover said...

Nothing can beat a Linux box like you're using for configurability, but I wanted to mention the Cisco RV042. I also have 3Mbps-capped DSL so we also subscribed to WiMAX (I'm in Portland OR.) for more throughput and redundancy. Our 2 links are not symmetrical - WiMAX has better throughput but worse latency. Using the 042 we can route port 80 traffic over the WiMAX and ssh, imap, games etc over the dsl, and if one of the links goes out, it will failover all traffic to the remaining link.

One gotcha with the 042 default settings is that some websites (such as online banking) really don't like it if pulling a webpage is done via multiple IPs. Your setup's use of periodically flushed routes handily avoids this, which is a pretty nice feature.

Anonymous said...

Here in south africa we have expensive internet, so we use tricks like this: http://tumbleweed.org.za/2008/09/19/split-routing-debianubuntu

StyXman said...

Is good to know how to do things by hand just to learn how they work, but then is better to use more high level tools. For that and any other router/firewall related tasks I suggest Shorewall. Its MultiISP support is very good, and I think it gets better in the perl version.

Robo47 said...

Hmm, sounds interesting, here were i life i only get 2mbit, but a second line (or some more attached via wifi from my neighbours who dont have/need/use internet) would really be nice. Especially the possible ability to set one line for ping-based things (like playing games online) and detecting this and then setting this line dedicated for playing while download/internet-traffic of all user people ... because often if 1 or 2 people do online-games they dont like if i upload or download stuff.

Nice to know that this is possible with a bit of linux magic :)