Source Based Policy Routing & NAT (DNAT/SNAT) aka Multi WANs on CentOS 5

Source Based Policy Routing & NAT (DNAT/SNAT) aka Multi WANs on CentOS 5

My question is regarding Source Based Policy Routing on CentOS 5 with 2
WANs plus a LAN (NAT) port with Load Balancing, but first before anything
some remarks before starting to describe the problem...
I know this topic has been brought many times here at stack exchage and
seems that the top 5 answers are (ordered from the most to the least):
Disable rp_filter
Mark/Connmark based policy routing
IP Based policy routing (add more IPs)
Install pfSense, Shorewall, Ubuntu?, Etc...
Buy expensive Cisco/3com/Juniper/Etc... Router
Most of the times, some of this answers are correct but for me solutions 1
& 2 haven't workout (i don't discard at least point 2 cause i may have
some issues with my setup), point 3 is basically isolating a problem
rather than solving it (also adds complexity to the routing tables) and
solutions 4 & 5 are just out of the scope since i don't have resources to
buy specialized hardware nor can take offline the server since it is on
production so to summarize replacing the CentOS server with something
"better" is off the table.
Ok now back in the problem, lets first describe the current setup...
Interfaces:
eth1: IP: 10.0.0.1, GW: 10.0.0.1, NM: 255.255.255.0 (LAN)
eth0: IP: 10.0.1.1, GW: 10.0.1.254, NM: 255.255.255.0 (ISP1 - ADSL Router)
eth2: IP: 10.0.2.1, GW: 10.0.2.254, NM: 255.255.255.0 (ISP2 - ADSL Router)
/etc/sysctl.conf:
# Controls IP packet forwarding
net.ipv4.ip_forward = 1
# Controls source route verification
net.ipv4.conf.default.rp_filter = 0
# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0
# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0
# Controls whether core dumps will append the PID to the core filename
# Useful for debugging multi-threaded applications
kernel.core_uses_pid = 1
# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1
# Controls the maximum size of a message, in bytes
kernel.msgmnb = 65536
# Controls the default maxmimum size of a mesage queue
kernel.msgmax = 65536
# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736
# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296
# Ignoring broadcasts request
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_messages = 1
/etc/iproute2/rt_tables:
#
# reserved values
#
255 local
254 main
253 default
0 unspec
#
# local
#
#1 inr.ruhep
2 ISP1
3 ISP2
/etc/sysconfig/network-scripts/route-eth0:
10.0.1.0/24 dev eth0 src 10.0.1.1 table ISP1
default via 10.0.1.254 dev eth0 table ISP2
/etc/sysconfig/network-scripts/route-eth2:
10.0.2.0/24 dev eth2 src 10.0.2.1 table ISP2
default via 10.0.2.254 dev eth2 table ISP2
/etc/sysconfig/network-scripts/rule-eth0:
fwmark 2 table ISP1 prio 1024
from 10.0.1.1 table ISP1 prio 2024
/etc/sysconfig/network-scripts/rule-eth2:
fwmark 3 table ISP2 prio 1025
from 10.0.2.1 table ISP2 prio 2025
/etc/sysconfig/iptables:
*filter
:INPUT DROP [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
# Basic Rules
-A INPUT -i lo -j ACCEPT
-A INPUT -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth2 -m state --state RELATED,ESTABLISHED -j ACCEPT
# SSH
-A INPUT -i eth0 -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -i eth2 -m tcp -p tcp --dport 22 -j ACCEPT
# OpenVPN
-A INPUT -i eth0 -m udp -p udp --dport 1194 -j ACCEPT
-A INPUT -i eth2 -m udp -p udp --dport 1194 -j ACCEPT
# Allow everything from LAN
-A INPUT -i eth1 -j ACCEPT
# Allow everything from the VPN
-A INPUT -i tun0 -j ACCEPT
# Default Drop on everything else
-A INPUT -j DROP
# Allow forwarding from LAN and VPN
-A FORWARD -i eth1 -j ACCEPT
-A FORWARD -i tun0 -j ACCEPT
# Allow all outbound traffic
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -o eth1 -j ACCEPT
COMMIT
*nat
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
# DNAT to Developer Box (SSH Server)
-A PREROUTING -i eth0 -p tcp -m tcp --dport 2222 -j DNAT --to-destination
10.0.0.200:2222
-A PREROUTING -i eth2 -p tcp -m tcp --dport 2222 -j DNAT --to-destination
10.0.0.200:2222
# SNAT
-A POSTROUTING -o eth0 -j SNAT --to-source 10.0.1.1
-A POSTROUTING -o eth2 -j SNAT --to-source 10.0.2.1
COMMIT
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
# Mark Based Routing? (based on NerdBoys site)
-A PREROUTING -j CONNMARK --restore-mark
-A PREROUTING --match mark --mark 2 -j ACCEPT
-A PREROUTING -i eth0 -j MARK --set-mark 2
-A PREROUTING --match mark --mark 3 -j ACCEPT
-A PREROUTING -i eth2 -j MARK --set-mark 3
-A PREROUTING -j CONNMARK --save-mark
COMMIT
Load Balancing is being possible with the gwping bash script which
basically monitors the 2 wans (eth0 and eth2) and set up the default
routes and weights in the server like this (while in load balance or the 2
wans up and running):
ip route replace default scope global nexthop via 10.0.1.1 dev eth0 weight
1 nexthop via 10.0.2.1 dev eth1 weight 1
The problem i have is that even with this setup that lot of people seems
to agree is the correct one, m still having issues with accessing services
inside the network from the outside (specifically the ssh developer box
and the OpenVPN one) even that the packets are being "marked" and routed
accordingly the answer from the dev box goes always to the wrong path. I
don't know if m missing something in the mangle or nat area or m
misunderstanding source based routing at all, anyway if someone know how
to make this work accordingly it will be kindly appreciated.
My sources for this setup are:
lartc.org/lartc.html#LARTC.RPDB.MULTIPLE-LINKS
fatalsite.net/?p=90
nerdboys.com/2006/05/05/conning-the-mark-multiwan-connections-using-iptables-mark-connmark-and-iproute2/
policyrouting.org/PolicyRoutingBook/ONLINE/CH08.web.html
unix.stackexchange.com/questions/58635/iptables-set-mark-route-diferent-ports-through-different-interfaces
unix.stackexchange.com/questions/22770/two-interfaces-two-addresses-two-gateways
bulma.net/body.phtml?nIdNoticia=2145
Kindest Regards
PS1: I found a website that its saying that the marks in the routing table
should be + 1 different from the iptables marks
(kim.attr.ee/2010/08/source-based-policy-routing-on-centos.html) is this
true? or this website is super-incorrect.