2017年9月

以前我的线上某一台服务器有eth0eth1两个网卡,分别对应内外网不同网络。但是有一次用命令dstat -N eth0,eth1tcpdump监测网络的时候发现,发往eth0的IP的流量居然会从eth1网卡流入流出, 这两个ip的网段和路由没有任何联系。 当时震惊了一下,分析原因,因为当初为了节约交换机,这两个网卡插在相同的一台交换机上,而且没有VLAN。由于网络ip层不应该有问题, 所以应该是链路层造成的结果,可能是ARPMac寻址的结果,后来没有深究。
很久以后,出现了一个新的问题:配置iptables的时候,规则DROP无效,于是又找回这里。iptables精简配置如下:

# Generated by iptables-save v1.4.14 on Wed Sep 13 18:05:35 2017
*raw
:PREROUTING ACCEPT [69675:4914196]
:OUTPUT ACCEPT [54936:3566904]
-A PREROUTING -p tcp -m tcp --dport 13306 -j TRACE
COMMIT
# Completed on Wed Sep 13 18:05:35 2017
# Generated by iptables-save v1.4.14 on Wed Sep 13 18:05:35 2017
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [93762:6266296]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -i eth1 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 13306 -j DROP
-A INPUT -p icmp -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT

从上可知, 默认允许lo和eth1内网所有流量, 拒绝了外网eth0的6379, 但是事实上可以从外网从容访问13306端口。

打开iptables的调试:

sudo modprobe ipt_LOG
iptables  -t raw -A PREROUTING  -p tcp --dport 13306 -j TRACE
sudo tail /var/log/kern.log

得到访问日志:

Sep 13 17:49:21 host239 kernel: [141463786.397726] TRACE: raw:PREROUTING:policy:2 IN=eth1 OUT= MAC=90:b1:1c:36:34:2c:70:54:f5:9d:bc:7b:08:00 SRC=59.151.**.** DST=113.31.*.* LEN=52 TOS=0x00 PREC=0x00 TTL=48 ID=45702 DF PROTO=TCP SPT=49983 DPT=13306 SEQ=296903697 ACK=934588750 WINDOW=115 RES=0x00 ACK FIN URGP=0 OPT (0101080A4BD6D6963FB18493) 
Sep 13 17:49:21 host239 kernel: [141463786.397760] TRACE: filter:INPUT:rule:1 IN=eth1 OUT= MAC=90:b1:1c:36:34:2c:70:54:f5:9d:bc:7b:08:00 SRC=59.151.*.* DST=113.31.*.* LEN=52 TOS=0x00 PREC=0x00 TTL=48 ID=45702 DF PROTO=TCP SPT=49983 DPT=13306 SEQ=296903697 ACK=934588750 WINDOW=115 RES=0x00 ACK FIN URGP=0 OPT (0101080A4BD6D6963FB18493)

上面有神奇的是,日志中外部访问13306端口的流量指向的外网ip(属于eth0), 但是日志记录的是IN=eth1, 且mac address 90:b1:1c:36:34:2c 属于eth1。这简直是以前内外网混流量问题的翻版。研究了一下,虽然关键词贫乏google很久没有有效信息, 想起以前用vpn的时候arp_proxy相关配置似乎有些关联性, 用arp继续google,发现关键词arp_announce

arp_announce - INTEGER
    Define different restriction levels for announcing the local
    source IP address from IP packets in ARP requests sent on
    interface:
    0 - (default) Use any local address, configured on any interface
    1 - Try to avoid local addresses that are not in the target's
    subnet for this interface. This mode is useful when target
    hosts reachable via this interface require the source IP
    address in ARP requests to be part of their logical network
    configured on the receiving interface. When we generate the
    request we will check all our subnets that include the
    target IP and will preserve the source address if it is from
    such subnet. If there is no such subnet we select source
    address according to the rules for level 2.
    2 - Always use the best local address for this target.
    In this mode we ignore the source address in the IP packet
    and try to select local address that we prefer for talks with
    the target host. Such local address is selected by looking
    for primary IP addresses on all our subnets on the outgoing
    interface that include the target IP address. If no suitable
    local address is found we select the first local address
    we have on the outgoing interface or on all other interfaces,
    with the hope we will receive reply for our request and
    even sometimes no matter the source IP address we announce.

于是解决方法是:

echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce

或者把iptables的-A INPUT -i eth1 -j ACCEPT换成-A INPUT -s 192.168.0.0/16 -j ACCEPT

回顾一下,这个"bug"其实很隐蔽, 因为iptables的配置看起来完美无缺, 但是, 但是, 谁知道问题出现在链路层。