IPsec Tunnels Routing Traffic Incorrectly

[zen_spartan]

As requested a seperate thread...
 
We have 6 Ipsec tunnels, all not routing traffic correctly. Using 19.7.1, the traffic seems to being Nat’d on the given the tunnel interface. No blocks on the firewall.
Issues only began after upgrade to 19.7

When pinging from the opnsense LAN interface to a host on the remote end of the tunnel we get:

# /sbin/ping -S '172.31.248.3' -c '3' '192.168.251.1'
PING 192.168.251.1 (192.168.251.1) from 172.31.248.3: 56 data bytes

--- 192.168.251.1 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
ping: sendto: Permission denied
ping: sendto: Permission denied
ping: sendto: Permission denied

Any ideas?

many thanks

Zen

[mimugmail]

At first, can you try to ping from an inside host and do a packet capture on LAN and the IPSEC Device?

[zen_spartan]

I've carried out a ping test as before and carried out a packet capture at each end of the ipsec tunnel. however only the local capture, captured any data going to 192.168.251.1, this is shown below:

ipsec3 Link
ipsec3000   11:13:22.907227 AF IPv4 (2), length 88: (tos 0x0, ttl 64, id 11116, offset 0, flags [none], proto ICMP (1), length 84)
    10.1.6.1 > 192.168.251.1: ICMP echo request, id 31059, seq 0, length 64
    10.1.6.1 > 192.168.251.1: ICMP echo request, id 31059, seq 1, length 64
    10.1.6.1 > 192.168.251.1: ICMP echo request, id 31059, seq 2, length 64
    10.1.6.1 > 192.168.251.1: ICMP echo request, id 38881, seq 0, length 64
    10.1.6.1 > 192.168.251.1: ICMP echo request, id 38881, seq 1, length 64
    10.1.6.1 > 192.168.251.1: ICMP echo request, id 38881, seq 2, length 64
ipsec3 Link
ipsec3000   11:13:23.917966 AF IPv4 (2), length 88: (tos 0x0, ttl 64, id 25230, offset 0, flags [none], proto ICMP (1), length 84)
ipsec3 Link
ipsec3000   11:13:24.959863 AF IPv4 (2), length 88: (tos 0x0, ttl 64, id 33184, offset 0, flags [none], proto ICMP (1), length 84)
ipsec3 Link
ipsec3000   11:13:45.071377 AF IPv4 (2), length 88: (tos 0x0, ttl 64, id 51072, offset 0, flags [none], proto ICMP (1), length 84)
ipsec3 Link
ipsec3000   11:13:46.082154 AF IPv4 (2), length 88: (tos 0x0, ttl 64, id 25305, offset 0, flags [none], proto ICMP (1), length 84)
ipsec3 Link
ipsec3000   11:13:47.146092 AF IPv4 (2), length 88: (tos 0x0, ttl 64, id 46855, offset 0, flags [none], proto ICMP (1), length 84)

the ping gave the same error:

# /sbin/ping -S '172.31.248.2' -c '3' '192.168.251.1'
PING 192.168.251.1 (192.168.251.1) from 172.31.248.2: 56 data bytes

--- 192.168.251.1 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
ping: sendto: Permission denied
ping: sendto: Permission denied
ping: sendto: Permission denied

does this help?

[mimugmail]

I don't really understand, can you do a ping from any device in 10.1.6 and not from the firewall itself (to rule out the problem with "permission denied").
And then what do you have in ipsec.log when pinging around, how does Status Overview looks like (screenshot)

[zen_spartan]

Hopefully this will make things clearer:


Remote LAN                  Remote ipsec GW                                                 Local ipsec GW           Local LAN

192.168.251.0/24 --->>> 10.1.6.2 =========IPsec Tunnel======== 10.1.6.1 <<<-----172.31.248.0/24

I have used a host on the Local LAN, IP 172.31.248.232, to ping a host on the Remote LAN, IP 192.168.251.1.
The ping fails.
The packet capture on the Local LAN interface of the opnsense router shows:

LAN
igb0   12:35:11.224685 aa:de:3d:a3:d8:c8 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 26729, offset 0, flags [DF], proto ICMP (1), length 84)
    172.31.248.232 > 192.168.251.1: ICMP echo request, id 56935, seq 1, length 64
    172.31.248.232 > 192.168.251.1: ICMP echo request, id 56935, seq 2, length 64
    172.31.248.232 > 192.168.251.1: ICMP echo request, id 56935, seq 3, length 64
    172.31.248.232 > 192.168.251.1: ICMP echo request, id 56935, seq 4, length 64
    172.31.248.232 > 192.168.251.1: ICMP echo request, id 56935, seq 5, length 64
    172.31.248.232 > 192.168.251.1: ICMP echo request, id 56935, seq 6, length 64
LAN
igb0   12:35:12.246508 aa:de:3d:a3:d8:c8 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 26983, offset 0, flags [DF], proto ICMP (1), length 84)
LAN
igb0   12:35:13.270404 aa:de:3d:a3:d8:c8 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 27197, offset 0, flags [DF], proto ICMP (1), length 84)
LAN
igb0   12:35:14.294417 aa:de:3d:a3:d8:c8 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 27374, offset 0, flags [DF], proto ICMP (1), length 84)
LAN
igb0   12:35:15.318361 aa:de:3d:a3:d8:c8 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 27463, offset 0, flags [DF], proto ICMP (1), length 84)
LAN
igb0   12:35:16.319273 aa:de:3d:a3:d8:c8 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 27542, offset 0, flags [DF], proto ICMP (1), length 84)

 

The packet capture of the local IPsec Interface is:

ipsec3 Link
ipsec3000   12:37:20.926293 AF IPv4 (2), length 88: (tos 0x0, ttl 63, id 43556, offset 0, flags [DF], proto ICMP (1), length 84)
    10.1.6.1 > 192.168.251.1: ICMP echo request, id 45508, seq 1, length 64
    10.1.6.1 > 192.168.251.1: ICMP echo request, id 45508, seq 2, length 64
    10.1.6.1 > 192.168.251.1: ICMP echo request, id 45508, seq 3, length 64
    10.1.6.1 > 192.168.251.1: ICMP echo request, id 45508, seq 4, length 64
    10.1.6.1 > 192.168.251.1: ICMP echo request, id 45508, seq 5, length 64
    10.1.6.1 > 192.168.251.1: ICMP echo request, id 45508, seq 6, length 64
ipsec3 Link
ipsec3000   12:37:21.939754 AF IPv4 (2), length 88: (tos 0x0, ttl 63, id 43800, offset 0, flags [DF], proto ICMP (1), length 84)
ipsec3 Link
ipsec3000   12:37:22.963773 AF IPv4 (2), length 88: (tos 0x0, ttl 63, id 43827, offset 0, flags [DF], proto ICMP (1), length 84)
ipsec3 Link
ipsec3000   12:37:23.987734 AF IPv4 (2), length 88: (tos 0x0, ttl 63, id 43928, offset 0, flags [DF], proto ICMP (1), length 84)
ipsec3 Link
ipsec3000   12:37:25.011707 AF IPv4 (2), length 88: (tos 0x0, ttl 63, id 44092, offset 0, flags [DF], proto ICMP (1), length 84)
ipsec3 Link
ipsec3000   12:37:26.035754 AF IPv4 (2), length 88: (tos 0x0, ttl 63, id 44246, offset 0, flags [DF], proto ICMP (1), length 84)

does this make things clearer?

[mimugmail]

Indeed, thanks!
Screenshot of outbound Nat please

[zen_spartan]

screenshot attached

[mimugmail]

so ... just kill the auto nat rule for IPSEC and you're good?

[zen_spartan]

How would you do that? Would you just have to take it out of hybrid mode to manual?
I dont see any settings within the IPsec setting section.

[mimugmail]

Normally I only use manual so I know what happens .. I think it was in general settings somewhere, but just guessing.

[zen_spartan]

Changed to manual and it magically worked! thanks for your help , odd thats its worked perfectly before update though.....

[mimugmail]

Maybe IPsec Nat was added to 19.7, no idea. Thats why I always do manual only

[mimugmail]

If you are brave and want to help the project, can you reenable auto-nat, check if the problem is there again, and if yes, apply this to CLI:

opnsense-patch a91babf

then reboot machine and it should also work with auto-nat

[arjen]

I'm facing the same issue here, disabling auto-nat helped.
But applying the patch did not fix the issue.