[SOLVED] IPsec tunnel breaks OpenVPN SIP/udp traffic

[Gauss23]

Hi,

I have a problem with an OPNsense bare metal box of a client.
Hardware is: ZBOX PRO CI329 nano with Realtek nics. OPNsense 20.7.4

Simple network:
LAN is 10.0.0.0/24
WAN is a static IP, direct ethernet connection
2 OpenVPN servers: TCP&UDP on WAN 1194, having 10.0.1.0/24 and 10.0.2.0/24 for their clients. LAN network is pushed to the client. Everything works great until I enable an IPsec tunnel.

This IPsec tunnel is IKEv1 in main mode and Phase2 is: local LAN 10.0.0.0/24, remote is 84.x.x.x/29. IPsec works like it should, so the LAN reaches the remote side.
 
My client uses a NetPhone Client to connect to the PBX in the LAN (10.0.0.70). No NAT involved. Plain routing.
If the clients are not inside of the office, they work by OpenVPN connection to connect to the PBX.

As soon as I enable the IPsec tunnel the problems start with the OpenVPN connections.

From the live log I can't see any blocked packets. The client is able to be called and is able to make outbound calls. But after max 1 minute the connection is dropped. And this client is not able to see the status of the other employees which are connected to the PBX.

Tried to connect to both OpenVPN (not at the same time of course), problem persists.

In the live log I see UDP connections without source and destination port. Is that maybe a hint? Never seen such connections in my logs before. See attached files.

When I disable the IPsec tunnel everything is running like it should with OpenVPN. Why is an IPsec tunnel disturbing an OpenVPN tunnel? IPsec is policy based. Any ideas?

Maybe we should move this thread to VPN section as it seem to be only VPN related.

And another note:
those creepy connections without source&destination port are also gone. But I need to re-enable the IPsec tunnel soon, so what can I do?

[mimugmail]

I didn't read everything there, but did this box ran 20.1 and was updated to 20.7?

[Gauss23]

Yes, that's possible. Why?

[mimugmail]

Forget it, seems yours is different. Please ifconfig via CLI before and enabling IPsec. Does MTU change on an interface?

Packets without ports are fragments which is a killer for voip, so it seems your mtu is changing somewhere.

Is this a new setup or did this start after an update?

[Gauss23]

The IPsec tunnel is new. It's the first IPsec tunnel on this box.

The box is running with OpenVPN for 10 months.

Without IPsec:

Code: [Select]

root@FW:~ # ifconfig
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=2018<VLAN_MTU,VLAN_HWTAGGING,WOL_MAGIC>
        ether 00:01:2e:92:86:c5
        inet6 fe80::X prefixlen 64 scopeid 0x1
        inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
re1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=2018<VLAN_MTU,VLAN_HWTAGGING,WOL_MAGIC>
        ether 00:01:2e:92:86:c6
        inet X.X.X.X netmask 0xfffffffc broadcast X.X.X.255
        inet6 fe80::X prefixlen 64 scopeid 0x2
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
        groups: enc
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=100<PROMISC> metric 0 mtu 33160
        groups: pflog
pfsync0: flags=0<> metric 0 mtu 1500
        syncpeer: 0.0.0.0 maxupd: 128 defer: off
        groups: pfsync
re0_vlan3000: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 00:01:2e:92:86:c5
        inet6 fe80::201:2eff:fe92:86c5%re0_vlan3000 prefixlen 64 scopeid 0x7
        inet X.X.X.X netmask 0xffffffe0 broadcast X.X.X.95
        groups: vlan
        vlan: 3000 vlanpcp: 0 parent interface: re0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
re0_vlan3901: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 00:01:2e:92:86:c5
        inet6 fe80::X prefixlen 64 scopeid 0x8
        inet 10.0.2.1 netmask 0xffffff00 broadcast 10.0.2.255
        groups: vlan
        vlan: 3901 vlanpcp: 0 parent interface: re0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ovpns1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet6 fe80::201:2eff:fe92:86c5%ovpns1 prefixlen 64 scopeid 0x9
        inet 10.0.1.1 --> 10.0.1.2 netmask 0xffffff00
        groups: tun openvpn
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 94022
ovpns2: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet6 fe80::201:2eff:fe92:86c5%ovpns2 prefixlen 64 scopeid 0xa
        inet 10.0.2.1 --> 10.0.2.2 netmask 0xffffff00
        groups: tun openvpn
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 63707
After IPsec is active:

Code: [Select]

re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=2018<VLAN_MTU,VLAN_HWTAGGING,WOL_MAGIC>
        ether 00:01:2e:92:86:c5
        inet6 fe80::X prefixlen 64 scopeid 0x1
        inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
re1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=2018<VLAN_MTU,VLAN_HWTAGGING,WOL_MAGIC>
        ether 00:01:2e:92:86:c6
        inet X.X.X.X netmask 0xfffffffc broadcast X.X.X.255
        inet6 fe80::X prefixlen 64 scopeid 0x2
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=41<UP,RUNNING> metric 0 mtu 1536
        groups: enc
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=100<PROMISC> metric 0 mtu 33160
        groups: pflog
pfsync0: flags=0<> metric 0 mtu 1500
        syncpeer: 0.0.0.0 maxupd: 128 defer: off
        groups: pfsync
re0_vlan3000: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 00:01:2e:92:86:c5
        inet6 fe80::X prefixlen 64 scopeid 0x7
        inet X.X.X.X netmask 0xffffffe0 broadcast X.X.X.95
        groups: vlan
        vlan: 3000 vlanpcp: 0 parent interface: re0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
re0_vlan3901: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 00:01:2e:92:86:c5
        inet6 fe80::X prefixlen 64 scopeid 0x8
        inet 10.0.2.1 netmask 0xffffff00 broadcast 10.0.2.255
        groups: vlan
        vlan: 3901 vlanpcp: 0 parent interface: re0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ovpns1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet6 fe80::201:2eff:fe92:86c5%ovpns1 prefixlen 64 scopeid 0x9
        inet 10.0.1.1 --> 10.0.1.2 netmask 0xffffff00
        groups: tun openvpn
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 94022
ovpns2: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet6 fe80::201:2eff:fe92:86c5%ovpns2 prefixlen 64 scopeid 0xa
        inet 10.0.2.1 --> 10.0.2.2 netmask 0xffffff00
        groups: tun openvpn
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 63707
Only difference is that the enc0 interface is UP, RUNNING with MTU 1536. Without IPsec it is not running. How can I lower the IPsec MTU?

Thanks for your help!

[mimugmail]

As the Screenshot shows an incoming fragmented packet on ovpnscthis means the client was the Sender.

Maybe you can set some mss-fix or mtu-fix in clients ovpn config? Dont have the correct syntax right now, just google it

[Gauss23]

Is it only needed in client config or do I need to lower the MTU for the OpenVPN interface on the OPNsense, too?

Tried it today with in client and server (TCP server):

Code: [Select]

tun-mtu 1440
mssfix

But still connection issues. Maybe not low enough?

I still don't understand why the active enc0 interface is creating trouble for other interfaces. This traffic is not flowing through enc0/IPsec. Do you have an explanation for it?

[mimugmail]

I would check via tcpdump at which point the packet gets fragmented

[Gauss23]

Ok, had a time window today to try some things.

I lowered the MTU values for the OpenVPN tunnel and added fragment and mssfix. Without IPsec enabled everything was running like with the higher default values.
When I enable IPsec it's just not working anymore.

Did some packet captures but didn't find any useful information there.

How is this possible?
I have multiple boxes with OpenVPN and IPsec running. Without a problem.

It's just UDP traffic which seems to be broken.

I even created an OpenVPN TCP server, same problem. IPsec enabled, UDP is broken, disable IPsec, UDP is flowing.

What can I do? Maybe it has something to do with the update 20.1->20.7 as you were previously assuming? What symptoms did you had there?

[Gauss23]

Ok, it seems to be solved.

Apparently you need to assign an interface to the OpenVPN server. Otherwise UDP traffic was not working correctly when having an IPsec site-to-site tunnel running.

Assigned an interface and now it works. Things can be so easy.

[mimugmail]

Hm, I never heard about this one 

[Gauss23]

Hm, I never heard about this one 

I would like to understand what happens if you assign an interface on kernel/os level. The interface exists as ovpnsX without assignment. What difference does it make to assign it?

Need to say that only one application is having those problems. It’s the NetPhone client from Deutsche Telekom for an Octopus PBX. Rest like SMB and HTTP was always running well.

I even tried it with a WireGuard tunnel yesterday evening. Same phenomenon:
IPsec disabled, WireGuard is working with NetPhone
IPsec enabled, NetPhone stops working
Assigned an interface for wg0, NetPhone starts working again over WireGuard

So some mechanism seems to change things when assigning interfaces.

[mimugmail]

Something with MTU assigning I'd guess.

Netphone with Dtag is real pain 

[Gauss23]

Dtag is real pain 

Let me correct that for you