What’s Asymmetric Routing?
In Asymmetric Routing, a packet traverses from a source to a destination in one path and takes a different path when it returns to the source. This is commonly seen in Layer-3 routed networks in internet. However, it may cause problem when it is related to security and firewalls.
A single server with more than two network interfaces may also encounter asymmetric routing.
For example, a server has two network interfaces, eth0 and eth1. The packets from a source received by eth1 may get replied from eth0 to the source.
How do asymmetric routing and Linux setting impact AWS EC2 Linux server?
If you attach two or more network interfaces (ENIs) from the same subnet to a non-Amazon Linux EC2 instance (for example, CentOS or RedHat Linux), you might encounter traffic flow issues caused by asymmetric routing and Linux settings. These issues occur because the primary and secondary network interfaces are in the same subnet, and there is one routing table with one gateway. Traffic that comes into the secondary network interface leaves the instance using the primary network interface, as described above. This isn’t allowed because the secondary IP address doesn’t belong to the MAC address of the primary network interface in Linux configurations.
AWS suggests customer to use a secondary private IPv4 address on the primary network interface instead of attaching more than one network interfaces or place multiple elastic network interfaces into non-overlapping subnets [1]. Advanced users may still want to attach the second network interface to the same subnet to a single instance.
The reason that Amazon Linux EC2 instances do not encounter traffic flow issue is that the ec2-net-utils package can take care of if [2].
Solution to avoid asymmetric routing on AWS EC2 Linux instance
Let’s do an experiment and observe how the asymmetric routing impacts traffic flow on AWS EC2 Linux with multiple network interfaces.
Environment setup:
- Launch two Centos7.2 instances in the same region.
- Two instances are in the same public subnet and using the same security group.
- One instance tagged as instance1 and the other tagged as instance2.
- SSH to the instances.
- Create ENI in the same subnet and security group.
- Attach the ENI to the instance1.
Setup 2nd ENI on instance1:
- Conifugre ifcfg-eth1
# cat >> /etc/sysconfig/network-scripts/ifcfg-eth1 <<EOF
DEVICE="eth1"
BOOTPROTO="dhcp"
ONBOOT="yes"
TYPE="Ethernet"
USERCTL="yes"
PEERDNS="yes"
IPV6INIT="no"
DEFROUTE="no"
EOF- Restart network
# systemctl restart network.service- Check IP addresses
# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc pfifo_fast state UP qlen 1000
link/ether 0e:1a:37:16:c5:22 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.124/24 brd 10.0.0.255 scope global dynamic eth0
valid_lft 3158sec preferred_lft 3158sec
inet6 fe80::c1a:37ff:fe16:c522/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc pfifo_fast state UP qlen 1000
link/ether 0e:3b:ae:ab:4e:84 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.254/24 brd 10.0.0.255 scope global dynamic eth1
valid_lft 3160sec preferred_lft 3160sec
inet6 fe80::c3b:aeff:feab:4e84/64 scope link
valid_lft forever preferred_lft forever- Install tcpdump for packet capturing in both instance1 and instance2.
# yum install tcpdump
Observation 1
- Can instance2 ping instance1’s 2nd IP (eth1) address successfully?
No.
- Observation in tcpdump on instance1:
Instance1 eth1 received ICMP echo request but does not reply.
- Why didn’t instance1 Linux reply ICMP echo request?
Let’s discuss Linux Reverse Path Filtering configuration before answering this question.
Linux Reverse Path Filtering
rp_filter – INTEGER
0 – No source validation.
1 – Strict mode as defined in RFC3704 Strict Reverse Path Each incoming packet is tested against the FIB and if the interface is not the best reverse path the packet check will fail. By default failed packets are discarded.
2 – Loose mode as defined in RFC3704 Loose Reverse Path Each incoming packet’s source address is also tested against the FIB and if the source address is not reachable via any interface the packet check will fail.
Current recommended practice in RFC3704 is to enable strict mode to prevent IP spoofing from DDos attacks. If using asymmetric routing or other complicated routing, then loose mode is recommended. The max value from conf/{all,interface}/rp_filter is used when doing source validation on the {interface}. Default value is 0. Note that some distributions enable it in startup scripts.
*Forwarding Information Base (FIB)
https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt
- Check rp_filter parameter on instance1 CentOS7.2
# sysctl -a |grep 'eth[0-1].rp_filter'
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.eth1.rp_filter = 1The rp_filter is configured as “1 – Strict node”.
- Enable log for martians packets to see if rp_filter configuration really cause packet drop.
# sysctl net.ipv4.conf.eth1.log_martians=1
net.ipv4.conf.eth1.log_martians = 1
# tail /var/log/messagesFrom /var/log/messages, we could see the packets are discarded due to rp_filter configuration. That’s why instance1 did not reply ICMP echo.
- Configure eth1 rp_filter as Loose Mode on instance1
# sysctl net.ipv4.conf.eth1.rp_filter=2
net.ipv4.conf.eth1.rp_filter = 2Observation 2
- After modifying instance1 eth1 rp_filter=2 (Loose Mode), can instance2 ping instance1 2nd IP address (eth1) successfully?
Still not.
- Observation in tcpdump on both instance1 and instance2.
Instance1 replies ICMP echo and sends out from eth0, but instance2 eth0 does not receive the ICMP echo reply.
- Why the ICMP echo reply did not reach instance2?
Before answering this question, let’s see what’s AWS EC2 Source/Destination Check.
AWS EC2 Source/Destination Check
You can enable or disable source/destination checks, which ensure that the instance is either the source or the destination of any traffic that it receives. Source/destination checks are enabled by default. You must disable source/destination checks if the instance if the instance runs services such as network address translation, routing, or firewalls.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html
Disable src/dst check for instance1.
EC2 console -> select instance -> Actions -> Networking -> Change Source/Dest. Check -> Yes, Disable.
Observation 3
- After disabling Source/Destination Check for instance1, Can instance2 ping the 2nd IP address successfully?
Yes!
- What did you see from tcpdump packet capture?
Instance1 received ICMP echo request from eth1.
Instance1 replies with ICMP echo reply and sends out from eth0.
Instance 2 received ICMP echo reply. Ping successful.
- Is this what we want?
Not exactly! Normally customers/users would like traffic in/out at the same interface for security and statistics reasons.
So this is not the right solution for asymmetric routing traffic flow issue on AWS EC2 instances.
How do we solve this problem? Let’s see how Linux network route traffic.
Linux Network Routing
Traditional Destination-based routing on Linux uses only some leading prefix of the packet’s destination IP address when selecting on which interface to send the packet out. Each entry in the routing table contains the IP address of the next-hop router (if a router is necessary) and the interface through which that packet should be sent. The entry also contains a variable length IP address prefix to match candidate packets against. That prefix could be as long as 32 bits for an IPv4 host route or as short as 0 bits for a default route that matches everything. If more than one routing table entry matches, the entry with the longest prefix is used.
A typical server has a simple routing table. It contains one entry for each interface on the server and one default route for reaching all the hosts not directly connected to its interfaces. This simple approach, which relies heavily on a single default route, results in a concentration of outgoing traffic through a single interface without regard to the interface through which the request originally was received [3].
A good way to address this imbalance is with the iproute2 package. iproute2 allows the administrator to throw away traditional network configuration tools, such as ifconfig, and tackle more complicated situations using a program named ip. iproute2 comes installed with most distributions [4].
The main idea of iproute2’s routing control is to separate routing decisions into two phases. The first phase Routing Policies/Rules point to Routing Tables. The second phase Routing Tables is a traditional destination-based routing table. Several rules may refer to one routing table and some routing tables may have no rules pointing to them.
Solution to asymmetric routing traffic flow issue on AWS EC2 Linxu instances.
Our solution to the asymmetric routing traffic flow problem is using the two-phase infrastructure. First, we need to create two routing rules that select the routing table that sends the server’s response traffic out the same interface on which the request arrived. Second, we create two routing tables; each table routes out through a different interface. The source address of the outgoing packets can be used for the route rules as the selector. In networking, this technique is known as source-based routing.
Check instance1 Linux default routing rules.
# ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup defaultCheck instance1 Linux default routing tables.
# ip route show table local
broadcast 10.0.0.0 dev eth0 proto kernel scope link src 10.0.0.124
broadcast 10.0.0.0 dev eth1 proto kernel scope link src 10.0.0.254
local 10.0.0.124 dev eth0 proto kernel scope host src 10.0.0.124
local 10.0.0.254 dev eth1 proto kernel scope host src 10.0.0.254
broadcast 10.0.0.255 dev eth0 proto kernel scope link src 10.0.0.124
broadcast 10.0.0.255 dev eth1 proto kernel scope link src 10.0.0.254
broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1
# ip route show table main
default via 10.0.0.1 dev eth0
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.124
10.0.0.0/24 dev eth1 proto kernel scope link src 10.0.0.254
# ip route show table default
(none)On instance1, add a route policy that checks if Source Address is from <eth0 IP>, then lookup route table 100.
# ip rule add from 10.0.0.100 lookup 100
# ip rule
0: from all lookup local
32765: from 10.0.0.100 lookup 100
32766: from all lookup main
32767: from all lookup default Add a default route in the route table 100.
# ip route add default via 10.0.0.1 dev eth0 table 100
# ip route show table 100
default via 10.0.0.1 dev eth0 Add a route policy that check if Source Address is from <eth1 IP>, then lookup route table 200.
# ip rule add from 10.0.0.200 lookup 200
# ip rule
0: from all lookup local
32764: from 10.0.0.200 lookup 200
32765: from 10.0.0.100 lookup 100
32766: from all lookup main
32767: from all lookup default Add a default route in the route table 200.
# ip route add default via 10.0.0.1 dev eth1 table 200
# ip route show table 200
default via 10.0.0.1 dev eth1 Observation 4
- Revert instance1 rp_filter back to 1 (Strict Mode) and enable source/destination check on instance1.
- Can instance2 ping instance1 successfully?
Yes!
- Observation from tcpdump packet capture.
Instance1 received ICMP echo request from eth1.
Instance1 replied with ICMP echo reply and sends out from eth1.
Instance 2 received ICMP echo reply. Ping successful.
- Are we finished? What’s the destination MAC address in ICMP echo reply? (Use tcpdump -e option to also show MAC layer)
The destination MAC address is the default gateway MAC address instead of the instance2 MAC address. However, the two instances are in the same subnet, the destination MAC address should be instance2 MAC address instead of going to default gateway.
Add one more route to the route tables which adding the subnet IP range to the route tables.
# ip route add 10.0.0.0/24 dev eth0 table 100
# ip route show table 200
default via 10.0.0.1 dev eth0
10.0.0.0/24 dev eth0 scope link
# ip route add 10.0.0.0/24 dev eth1 table 200
default via 10.0.0.1 dev eth1
10.0.0.0/24 dev eth1 scope linkObservation 5
- Can instance2 ping instance1 successfully? and What’s the destination MAC address of ICMP echo reply?
Yes, instance 2 can pin instance1 successfully and the destination MAC address of ICMP echo reply is instance2 MAC address.
- Are we finished? The “ip rule”, “ip route” commands could not survive a reboot.
Final solution for symmetric routing traffic flow problem on AWS EC2 instances.
To make the route changes permanent so that they can survive a reboot, add them to the interfaces file:
# echo "from 10.0.0.100 lookup 100" >> /etc/sysconfig/network-scripts/rule-eth0
# echo "10.0.0.0/24 dev eth0 table 100" >> /etc/sysconfig/network-scripts/route-eth0
# echo "default via 10.0.0.1 dev eth0 table 100" >> /etc/sysconfig/network-scripts/route-eth0
# echo "from 10.0.0.200 lookup 200" >> /etc/sysconfig/network-scripts/rule-eth1
# echo "10.0.0.0/24 dev eth1 table 200" >> /etc/sysconfig/network-scripts/route-eth1
# echo "default via 10.0.0.1 dev eth1 table 200" >> /etc/sysconfig/network-scripts/route-eth1Conclusion
Attaching two or more network interfaces (ENIs) from the same subnet to a non-Amazon Linux EC2 instance (for example, CentOS or RedHat Linux) might encounter traffic flow issues caused by asymmetric routing and Linux settings. The solution is to utilize iproute2 package two-phase routing infrastructure. First, we need to create two routing rules that select the routing table that sends the server’s response traffic out the same interface on which the request arrived. Second, we create two routing tables; each table routes out through a different interface. The source address of the outgoing packets can be used for the route rules as the selector.
Reference
[1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/MultipleIP.html
[2] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/best-practices-for-configuring-network-interfaces.html#ec2-net-utils
[3] https://www.linuxjournal.com/article/7291
[4] http://www.policyrouting.org/iproute2.doc.html
Extended Reading:
- Tuning Linux maximum TCP connections and test on Amazon Linux
- 在 Windows 10 的 WSL 2 Ubuntu 上安裝 WordPress
- 自架 WordPress 網站在 Google 上一直搜尋不到？自行提交 sitemap，讓 Google 建立索引
