I established an IPsec VPN tunnel between two Juniper SRX routers across NAT, with the NAT being performed by the firewall (a Linux server). When attempting to configure the firewall rules to allow IPsec VPN traffic to pass through, I discovered that while IKE negotiation completes successfully with UDP ports 500 and 4500 allowed, VPN traffic communication does not function unless rules are added to allow ESP.
Here are the rules I applied on the firewall, but only IKE negotiation was completed:
iptables -F
iptables -P FORWARD DROP
# Inbound
iptables -I FORWARD -s <External IP address> -d <Internal IP address> -p udp --sport 500 -j ACCEPT
iptables -I FORWARD -s <External IP address> -d <Internal IP address> -p udp --sport 4500 -j ACCEPT
# Outbound
iptables -I FORWARD -s <Internal IP address> -d <External IP address> -p udp --sport 500 -j ACCEPT
iptables -I FORWARD -s <Internal IP address> -d <External IP address> -p udp --sport 4500 -j ACCEPT
And the following two rules should also be set in order to successfully transmit data through the VPN tunnel:
iptables -I FORWARD -s <External IP address> -d <Internal IP address> -p 50 -j ACCEPT
iptables -I FORWARD -s <Internal IP address> -d <External IP address> -p 50 -j ACCEPT
Since the ESP data is encapsulated by the UDP header (provided here are VPN communication packets captured on my firewall), why would the firewall block tunnel traffic without allowing ESP in rules? In my expectation, when the firewall sees that a packet contains a UDP port 4500 header, it should accept the packet accordingly.