Strongswan ipsec tunnel not establishing connection with AAA server

Question

I have two VM

VM-1 : I have installed Strongswan 5.9.

VM-2 : Installed Strongswan 5.9, Installed freeradius (radius server)

I have started Strongswan on both VM by systemctl start strongswan.

When I run radtest <username> <password> <ipaddress> <NAS Port> <secretkey> command from VM-1 request is not authenticated by aaa server, but when I stop strongswan.service then I am getting Received Access-Accept response from aaa server.

My question is why VM-1 is not able to communicate when strongswan.service is active. I guess IPsec tunnel is not established. Below are my swanctl.conf file. Can anyone please help here what I am missing. How to establish tunnel and access AAA server.

VM-1

connections {

   conn1 {
    local_addrs  = x.x.x.83
    remote_addrs = x.x.x.171

    local {
     auth = psk
     id = x.x.x.83
    }
    remote {
       auth = psk
       id = x.x.x.171
    }
    children {
       vm1-to-aaa {
          local_ts  = x.x.x.83
          remote_ts = x.x.x.171

          start_action = trap
          esp_proposals = aes192gcm16-aes128gcm16-prfsha256-ecp256-modp3072,aes192-sha256-ecp256-modp3072,default
          mode = transport
       }
    }
    version = 2
    mobike = no
    reauth_time = 10800
    proposals = aes192gcm16-aes128gcm16-prfsha256-ecp256-ecp521,aes192-sha256- modp3072,default
 }
}

secrets {
  ike-1 {
    id-1 = x.x.x.83
    id-2 = x.x.x.171
    secret = "thisissecret"
 }
 }

VM - 2

  connections {

    conn2 {
     local_addrs  = x.x.x.171
     remote_addrs = x.x.x.83

     local {
       auth = psk
       id = x.x.x.171
    }
    remote {
       auth = psk
       id = x.x.x.83
    }
    children {
       aaa-to-vm1 {
          local_ts  = x.x.x.171
          remote_ts = x.x.x.83

          start_action = trap
          esp_proposals = aes192gcm16-aes128gcm16-prfsha256-ecp256-modp3072,aes192-sha256-ecp256-modp3072,default
          mode = transport
       }
    }
    version = 2
    mobike = no
    reauth_time = 10800
    proposals = aes192gcm16-aes128gcm16-prfsha256-ecp256-ecp521,aes192-sha256-modp3072,default
 }
 }

secrets {
  ike-1 {
    id-1 = x.x.x.171
    id-2 = x.x.x.83
    secret = "thisissecret"
 }
}

Update

When I initiate a command swanctl -i conn1 -c vm1-to-aaa I am getting -

 [IKE] giving up after 5 retransmits
 [IKE] establishing IKE_SA failed, peer not responding
 initiate failed: establishing CHILD_SA 'vm1-to-aaa' failed

Below is my firewall settings -

 public (active)
   target: default
   icmp-block-inversion: no
   interfaces: ens32
   sources:
   services: cockpit dhcpv6-client http https ipsec ssh
   ports: 443/tcp 8765/tcp 8088/tcp 1812/udp 1813/udp 500/udp  4500/udp
   protocols:
   forward: no
   masquerade: yes
   forward-ports:
   source-ports:
   icmp-blocks:
   rich rules:
     rule protocol value="ah" accept
     rule protocol value="esp" accept

Answer 1

dropping IKE_SA_INIT response no matching IKE ISA

Some other service was also listening on 500/udp port

This sounds like you had two different IKE services running (my guess is strongswan-starter and strongswan-swanctl?)

how can I force freeradius to use ipsec tunnel? or any service installed on that host should validate by ipsec tunnel

The kernel's "transform" (xfrm) policies installed by strongSwan already force this for all outbound traffic. All packets matching the local_ts/remote_ts traffic selectors will be automatically transformed (in this case, encrypted in ESP) before being sent out. Either swanctl -l or ip xfrm will show what policies are active.

(As your local_ts/remote_ts traffic selectors currently specify only IP addresses, all traffic between those two hosts will be encrypted. If you want to limit protection to only specific ports, that needs to be added to the traffic selectors.)

You can use iptables or nftables rules to discard inbound non-IPsec packets (should they arrive by accident or from unwanted sources):

udp dport 1812 meta ipsec exists accept
udp dport 1812 reject

-p udp --dport 1812 -m policy --pol ipsec --dir in -j ACCEPT
-p udp --dport 1812 -j REJECT

It may be useful to make freeradius.service depend on strongswan, so that it won't even start until strongswan has started up (and installed at least the "trap" policies):

systemctl add-wants freeradius.service strongswan.service