Monday, August 23, 2010

Source based policy routing on Centos (or any 2.6.x kernel Linux)

I've upgraded our BSD based firewall to Centos 5.4 and some newer hardware and as such migrated my skills also from IPFW to IPTABLES. This short tutorial shows what to do when you have shared resources (such as a mail server) and want to implement proper source based policy routing, without having to rely on having ONE gateway on your server.

Some Legends: - Local LAN on firewall eth0 - ISP1 on firewall eth2 - ISP2 on firewall eth1 - source host

Okay, so the standard routing on the firewall pushes all traffic destined for over eth1, with a gateway address of Our end-result here is to get to connect to SMTP running on (eth2) on the firewall. Normal routing will not work, packets will be recieved at eth2, but sent out over eth1 with a complete packet mixup.

We are going to implement a policy that will say:

If I get SMTP Traffic on, then make sure response traffic goes out over eth2 no matter what.


First, you have to make sure that iproute2 is installed:
    yum install kernel-devel
    tar xjf iproute2-2.6.26.tar.bz2
    cd iproute2-2.6.26 
    cp ip/ip /sbin

Check that IP is able to run:

  [root@firewall ~]# ip
   Usage: ip [ OPTIONS ] OBJECT { COMMAND | help }
        ip [ -force ] [-batch filename
        where  OBJECT := { link | addr | addrlabel | route | rule | neigh | ntable |
                   tunnel | maddr | mroute | monitor | xfrm }
       OPTIONS := { -V[ersion] | -s[tatistics] | -d[etails] | -r[esolve] |
                    -f[amily] { inet | inet6 | ipx | dnet | link } |
                    -o[neline] | -t[imestamp] }

and now we can begin:

First, mark the traffic you need dealt with in IPTABLES under the mangle table:

          -A FORWARD -t mangle -i eth2 -p tcp --dport 25 -j MARK --set-mark 0

meaning: Any incoming traffic on eth2, TCP based with a destination port of 25, mark it with 0 (which means 1 in non-computer binary)

         service iptables restart

to allow it to make the change.

Then, create a NEW routing table for this while exercise:

      echo 1 SMTP >> /etc/iproute2/rt_tables

now, create a new default route for our SMTP routing table:

      ip route add default via dev eth2 table SMTP

check that its enabled correctly:

     [root@firewall ~]# ip route show table SMTP
     default via dev eth2

Great, looking good so far, now we add the ip rule to process marked traffic with a new gateway:

     ip rule add fwmark 1 table SMTP

Remember its marked 0 in IPTABLES, but called 1 outside of it, 1 in IPTABLES then becomes 2, and so forth. These changes are not persistent, so ADD THEM TO /etc/rc.local

First a test without policy based routing from our server (I just disabled the IPTABLES line):

     support@ - ~>telnet 25
     telnet: Unable to connect to remote host: Connection timed out

And then a test with policy based routing enabled - working 100%

     support@ - ~>telnet 25
     Connected to
     Escape character is '^]'.

     221 2.0.0 closing connection
     Connection to closed by foreign host.


  1. Just to be clear here. What you're really doing in this example is port based routing not source based. At no point in your rules did you ever reference the source of the packet. You've only specified that any traffic on port 25 (regardless of the source or destination) will be directed to eth2.

    What I'm looking for is true source based routing. That is, if there is an incoming packet on one interface, the resulting outgoing packet will also go there. I'm not really sure that this is possible because either a) the packet would have to be marked by the application or b) there would have to be some sort of stateful routing going on.

  2. Actually what I'm doing here is "marking" traffic coming in on the secondary interface, and then forcing it back out over the same interface. I could have just specified all ports instead of just port 25 in the IPFW rule - so it is source based, insofar that the traffic travels across the same interface in and out.

    A more elegant solution (over time that I've been fine tuning this) is to use "netem" which is built into most 2.6 kernels: