The following problem:
I have a server that is, in addition to other tasks not relevant for the problem at hand, supposed to act as an IPsec gateway. When I'm using the Ethernet interface pointing to the 'Net for handling IPsec traffic and sending out and receiving any forwarded traffic, everything works just fine: The traffic is sent through the IPsec tunnel, reaches the server, then gets decrypted and is sent out to its intended destination, and any replies are received, encrypted and passed through the IPsec tunnel.
However, once I set up an Ethernet bridge, attach aforementioned Ethernet card to it (I have duplicated its MAC address on the bridge, otherwise all outbound traffic would be dropped by the router to which the server is connected) and remap everything (both the entries in /etc/network/interfaces and the relevant iptables statements) and then try to use the server as a gateway, nothing is forwarded. By doing some checking I found that the gateway itself can be reached via IPsec, and other hosts can be reached from the gateway.
In the former case no additional stations are listed between the client and the gateway so they are definitely passing the tunnel (there would be more hops in between otherwise) when doing a traceroute, and in the latter case doing a traceroute shows that the packets leave the box as usual.
When placing a TRACE on both ipencap and icmp on both -t raw PREROUTING and -t raw OUTPUT, when the IPsec tunnel is up only the outbound packets (both the packets bound for the virtual IP of the client and the public IP address of the recipient, the latter being a firewall) are showing up in xtables-monitor; anything inbound isn't even noticed.
Also, the counters on the relevant firewall rules on the gateway keep increasing throughout the entire ruleset, however, nothing seems to be sent out (I've even logged things on ebtables's OUTPUT chain in filter to verify what SNAT is doing, and everything looked just fine: The source IP address had been mapped to the IP address of the bridge, the destination IP address had been what it was supposed to be, and the source MAC address was set to the one I had duplicated on the bridge. As far as forwarded traffic is concerned, no responses are ever received, and doing a traceroute from my IPsec client (the one that I'm working on right now) only elicits a response from the gateway. All other stations beyond that fail to respond, which in turn tells me that the router to which the sever is connected is dropping the forwarded packets.
Any ideas on how this could be fixed?
The relevant netfilter rules (iptables) - analogous for IPv6:
# Set default policies
iptables -P INPUT DROP
iptables -P FORWARD DROP
# Additional chains
/usr/sbin/iptables -N block_ext_in # Blocks unsolicited inbound traffic (Bogons, Spamhaus's DROP/EDROP)
/usr/sbin/iptables -N block_ext_out # Blocks undesired outbound traffic (Bogons, Spamhaus's DROP/EDROP)
/usr/sbin/iptables -N zone_ext_in # Accepts certain inbound traffic
/usr/sbin/iptables -N zone_ext_out # Permits certain outbound traffic (currently empty)
/usr/sbin/iptables -N ipsec_in # Handles inbound IPsec traffic
/usr/sbin/iptables -N drop_icmp # Kills deprecated ICMP messages
iptables -t raw -A PREROUTING -p ipencap -j TRACE
iptables -t raw -A PREROUTING -p icmp -j TRACE
iptables -t raw -A OUTPUT -p ipencap -j TRACE
iptables -t raw -A OUTPUT -p icmp -j TRACE
iptables -t mangle -A PREROUTING -p tcp -s <virtual-ip-range>/24 -m policy --pol ipsec --dir in -m tcp --tcp-flags SYN,RST SYN -m tcpmss --mss 1361:1536 -j TCPMSS --set-mss 1360
iptables -t mangle -A POSTROUTING -p tcp -d <virtual-ip-range>/24 -m policy --pol ipsec --dir out -m tcp --tcp-flags SYN,RST SYN -m tcpmss --mss 1361:1536 -j tCPMSS --set-mss 1360
iptables -A INPUT -i br0 -m policy --pol ipsec --dir in -j ipsec_in
iptables -A INPUT -i br0 -s <virtual-ip-range>/24 -m policy --pol ipsec --dir in -j ACCEPT
iptables -A INPUT -p tcp -m conntrack --ctstate INVALID -j DROP
iptables -A INPUT -p tcp -m tcp ! --tcp-flags SYN,ACK,FIN,RST SYN -m conntrack --ctstate NEW -j DROP
iptables -A INPUT -p tcp -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp -m conntrack --ctstate NEW -m tcpmss ! --mss 536:65535 -j DROP
iptables -A INPUT -i br0 -p icmp -j drop_icmp
iptables -A INPUT -i br0 -s <server-vip> -j ACCEPT
iptables -A INPUT -i eth0 -m policy --pol none --dir in -j block_ext_in
iptables -A INPUT -i br0 -p udp --dport isakmp -j ACCEPT
iptables -A INPUT -i br0 -p udp --dport 4500 -j ACCEPT
iptables -A INPUT -i br0 -p esp -j ACCEPT
iptables -A INPUT -p icmp -m conntrack --ctdir REPLY -j ACCEPT
iptables -A INPUT -i br0 -p icmp --icmp-type 8 -j ICMP_ECHO
iptables -A INPUT -i br0 -p icmp --icmp-type 3/4 -j ACCEPT
iptables -A INPUT -i br0 -j zone_ext_in
iptables -A INPUT -p icmp -j DROP
iptables -A INPUT -j REJECT --reject-with icmp-port-unreachable
iptables -A OUTPUT -d <virtual-ip-range>/24 -m policy --pol ipsec --dir out -j ACCEPT
iptables -A OUTPUT -o br0 ! -p icmp -j block_ext_out
# Safeguards against transmitting unencrypted traffic!
iptables -A OUTPUT -o br0 -d 10.0.0.0/8 -m policy --pol none --dir out -j REJECT --reject-with icmp-net-unreachable
iptables -A OUTPUT -o br0 -d 172.16.0.0/12 -m policy --pol none --dir out -j REJECT --reject-with icmp-net-unreachable
iptables -A OUTPUT -o br0 -d 192.168.0.0/16 -m policy --pol none --dir out -j REJECT --reject-with icmp-net-unreachable
iptables -A OUTPUT -o eth0 ! -p icmp -j zone_ext_out
iptables -A FORWARD -o br0 -s <virtual-ip-range>/24 -m policy --pol ipsec --dir in -j ACCEPT
iptables -A FORWARD -p tcp -m conntrack --ctstate INVALID -j DROP
iptables -A FORWARD -m physdev --physdev-is-bridged -j ACCEPT
iptables -A FORWARD -i br0 -p tcp -m physdev --physdev-in eth0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -i br0 -p icmp -m physdev --physdev-in eth0 -m conntrack --ctdir REPLY -j ACCEPT
iptables -A FORWARD -i br0 -p udp -m physdev --physdev-in eth0 -m conntrack --ctdir REPLY -j ACCEPT
# Safeguards against forwarding unencrypted traffic!
iptables -A FORWARD -o br0 -d 10.0.0.0/8 -m policy --pol none --dir out -j REJECT --reject-with icmp-net-unreachable
iptables -A FORWARD -o br0 -d 172.16.0.0/12 -m policy --pol none --dir out -j REJECT --reject-with icmp-net-unreachable
iptables -A FORWARD -o br0 -d 192.168.0.0/16 -m policy --pol none --dir out -j REJECT --reject-with icmp-net-unreachable
iptables -A FORWARD -j REJECT --reject-with icmp-net-unreachable
iptables -t nat -A POSTROUTING -s <server-ip> -j ACCEPT
iptables -t nat -A POSTROUTING -m policy --pol ipsec --dir out -j ACCEPT
iptables -t nat -A POSTROUTING -o br0 -j SNAT --to-source <server-ip> --persistent
From /etc/network/interfaces:
allow-hotplug eth0
iface eth0 inet manual
auto br0
iface br0 inet dhcp
hwaddress <mac-cloned-from-eth0>
bridge_ports eth0
bridge_stp off
bridge_fd 0
bridge_maxwait 0
iface br0 inet static
address <server-virtual-ipv4>/24
iface br0 inet6 static
address <public-ipv6>/128
gateway fe80::1
iface br0 inet6 static
address <server-virtual-ipv6>/64
Rules added by StrongSwan (show up first in their respective chains):
Chain FORWARD (policy DROP)
target prot opt source destination
ACCEPT all -- <client-virtual-ipv4> anywhere policy match dir in pol ipsec reqid 3 proto esp
ACCEPT all -- anywhere <client-virtual-ipv4> policy match dir out pol ipsec reqid 3 proto esp
Chain INPUT (policy DROP)
target prot opt source destination
ACCEPT ipencap-- <recipient-ip> <public-ipv4> policy match dir in pol ipsec reqid 3 proto esp
UPDATE
I have done some more digging since this behavior didn't go away in Debian Bookworm, and I have made the following discovery after setting the following log rules (new server so please note that the bridge port's identifier has changed from eth0 to enp2s0:
ebtables -t nat -A PREROUTING -p ip --ip-proto icmp -i enp2s0 --log-level warning --log-prefix "VPN-inbound: " -j CONTINUE
ebtables -t nat -A POSTROUTING -p ip --ip-proto icmp -o enp2s0 --log-level warning --log-prefix "VPN-outbound: " -j CONTINUE
ebtables -A INPUT -p ip --ip-proto icmp -i enp2s0 --log-level warning --log-prefix "VPN-reply: " -j CONTINUE
ebtables -A OUTPUT -p ip --ip-proto icmp -o enp2s0 --log-level warning --log-prefix "VPN-fwd: " -j CONTINUE
ebtables -A FORWARD -p ip --ip-proto icmp -i enp2s0 --log-level warning --log-prefix "VPN-bridged: " -j CONTINUE
iptables -I INPUT 1 -i br0 -p icmp -d 172.16.1.0/24 -j LOG --log-level warning --log-prefix "VPN-recv: "
hen, for testing purposes, I fired up the VPN and tried to ping www.google.de (some external server so that packets are forced out), and like before I got no reply.
Thanks to the log rules that I have set I discovered the following (sample from /var/log/kern.log):
2023-12-23T13:17:31.783622+01:00 h3013014 kernel: [48204.632320] VPN-fwd: IN= OUT=enp2s0 MAC source = xx:xx:xx:xx:xx:xx MAC dest = yy:yy:yy:yy:yy:yy proto = 0x
2023-12-23T13:17:31.783713+01:00 h3013014 kernel: [48204.642733] VPN-outbound: IN= OUT=enp2s0 MAC source = xx:xx:xx:xx:xx:xx MAC dest = yy:yy:yy:yy:yy:yy proto
2023-12-23T13:17:31.808457+01:00 h3013014 kernel: [48204.663290] VPN-inbound: IN=enp2s0 OUT= MAC source = zz:zz:zz:zz:zz:zz MAC dest = xx:xx:xx:xx:xx:xx proto
Going by this ebtable's OUTPUT and POSTROUTING rules are hit for leaving packets (as is expected) as is its PREROUTING rule (so the reply is actually received), but then things start to go haywire: Neither ebtables's INPUT nor its FORWARD rules are hit (the former would have been expected, but the latter is a check for packets gone astray), nor is iptables's INPUT rule. This means that everything before the bridging decision is made is accounted for, but beyond that things start to fall apart.
Since I have the bridging code also invoke ip(6)tables (I have set both /proc/sys/net/bridge/bridge-nf-call-iptables and /proc/sys/net/bridge/bridge-nf-call-ip6tables enabled) I decided to disable those.
echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables
echo 0 > /proc/sys/net/bridge/bridge-nf-call-ip6tables
A subsequent ping to an external server finally revealed this:
thorin@zanarkand:~$ ping www.google.de
PING www.google.de (142.250.185.227) 56(84) bytes of data.
64 bytes from fra16s53-in-f3.1e100.net (142.250.185.227): icmp_seq=1 ttl=57 time=65.3 ms
64 bytes from fra16s53-in-f3.1e100.net (142.250.185.227): icmp_seq=2 ttl=57 time=65.1 ms
64 bytes from fra16s53-in-f3.1e100.net (142.250.185.227): icmp_seq=3 ttl=57 time=89.8 ms
^C
--- www.google.de ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 65.107/73.397/89.763/11.572 ms
This in turn tells me that something is going terribly wrong when the bridging decision is made for IPsec traffic with ip(6)tables called for bridged packets. Setting the following rule with iptables is also hit and produces log entries:
iptables -t nat -A PREROUTING -i br0 -p icmp -d 172.16.1.0/24 -m physdev --physdev-in enp2s0 -j LOG --log-level warning --log-prefix "VPN-inbound2: "
However, since this rule is hit, but ebtables's INPUT rule is not, and, according to this image found on Wikipedia the only thing in between iptables's PREROUTING chain and ebtables's INPUT chain is the bridging decision. Since both chains are otherwise empty, they don't tamper with any incoming packets in any way. The only problem that I could come up with is that outbound IPsec traffic is NATed before it is leaving the box and that something could be going wrong when any replies are handled.
Interestingly enough, only IPsec traffic is adversely affected, and only if ip(6)tables is invoked as well. Any other routed outbound traffic is handled normally even though it is NATed.
