16.7.14 -> upgrade to 17.1 - VPN multiwan broken

[manueljben]

Hi,

We have an opnsense running with two WANs, one of them has the default gateway for all services and internal machines. We configured it with the OpenVPN binding to "Localhost" and the Nat redirect trick as stated here:

https://doc.pfsense.org/index.php/Multi-WAN_OpenVPN

Before the upgrade users can connect to any WAN interface, but after the 17.1 one of the WANs doesn't work anymore.  We still can connect to the VPN via the WAN that has the default gateway, but if we try to connect via the another address/wan, the packet arrives to the interface (we can see with tcpdump inside opnsense), but the response never goes out, and we see that openvpn server in opnsense throws this error multiple times:

Feb 2 23:22:25   openvpn[38459]: xx.xx.xx.xx:yyyyy write UDPv4: Can't assign requested address (code=49)

where xx.xx.xx.xx:yyyyy is the client ip/port trying to connect.

Can you give me some advice?
Thanks in advance

[franco]

This is strange, OpenVPN gets the traffic, tries to bind the client but is rejected by the system.

Something similar happened with the move from MPD4 to MPD5, when the old way of binding to the system no longer worked, being more strict to be precise.

Is the xx.xx.xx.xx a local subnet? Is it properly configured in the secondary WAN?

EDIT: Did you guys try "any" instead of "localhost"?


Cheers,
Franco

[manueljben]

Thanks for your answer Franco!

In some proofs that we did trying to solve this issue, we tried "any" instead of "localhost" in Openvpn server, with no luck. But we will try also again, and resetting states/rebooting if it will help.

For clarification xx.xx.xx.xx is NOT a local subnet, is the client's IP trying to connect to opnsense (ovpn) (external one, with its own internet...).

Cheers,
    Manuel J.

[manueljben]

After some testing again with "any" instead of "localhost", the behaviour we get is that OpenVPN is not raising that error "write UDP", and responses goes out but using the wrong address / WAN.

If we reach VPN from WAN2, opnsense/ovpn answers using its address from wan1 (default gw) and sends it through this interface (even if we add a wan2 rule with the corresponding ovpn server udp port and wan2 gateway set, no policy routing working?).

This behaviour doesn't works too, because causes that the first "stateful" firewall in the client / inetpath throws away the responses that come from a different src address than the expected one.

[manueljben]

beside of this issue, we noticed that suddenly we have no internet, no traffic is going, so seems that opnsense hangs. We go to console and we see a "db>" prompt, causing that we need to reboot the fw to get it working again.  This happened at least twice since the upgrade, and we see this "hang" issue happened to other users also in the forum

here it is some db> info we extracted if it will help to you:
Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 02
fault virtual address   = 0x20
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80bfcd0c
stack pointer           = 0x28:0xfffffe0118bb5820
frame pointer           = 0x28:0xfffffe0118bb5850
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 66148 (openvpn)

Many thanks for your efforts and support!

[franco]

Hi Manuel,

We're missing the actual backtrace / function that the panic happened in. Did you also sent this via the crash reporter?

Also, can you try the following kernel?

# opnsense-update -kr 17.1-route
# /usr/local/etc/rc.reboot


Cheers,
Franco

[manueljben]

Sorry for not having more information about the crashes, i will give instructions to other colleagues so that if it happens again we will extract all important/available info.

Regarding the kernel you said if we can test, I inform you that we are currently running it:

FreeBSD OPN-FW 11.0-RELEASE-p7 FreeBSD 11.0-RELEASE-p7 #0 8fb06e15a(stable/17.1): Sun Feb  5 19:04:29 CET 2017     root@sensey64:/usr/obj/usr/src/sys/SMP  amd64

and as we can see in the openvpn logs, the previous issue remains (initial packet arrives, and openvpn complains in log):

Feb  6 14:36:37 OPN-FW openvpn[66338]: <external_client_ip>:<external_client_port> write UDPv4: Can't assign requested address (code=49)
Feb  6 14:36:52 OPN-FW openvpn[66338]: <external_client_ip>:<external_client_port> write UDPv4: Can't assign requested address (code=49)
Feb  6 14:37:22 OPN-FW openvpn[66338]: <external_client_ip>:<external_client_port> TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Feb  6 14:37:22 OPN-FW openvpn[66338]: <external_client_ip>:<external_client_port> TLS Error: TLS handshake failed

Cheers,

    Manuel

[manueljben]

Hi Franco,

Today OPN sense stopped working again (even with the new test kernel), in "db>" prompt.  Here you have the data we can get from there:

Tracing pid 58355 tid 100211 td 0xfffff80006459000
turnstile_broadcast() at turnstile_broadcast+0x9c/frame 0xfffffe0118ccc460
__rw_wunlock_hard() at __rw_wunlock_hard+0x8f/frame 0xfffffe0118ccc490
vm_map_delete() at vm_map_delete+0x3dc/frame 0xfffffe0118ccc510
vm_map_remove() at vm_map_remove+0x47/frame 0xfffffe0118ccc540
exec_new_vmspace() at exec_new_vmspace+0x225/frame 0xfffffe0118ccc5d0
exec_elf64_imgact() at exec_elf64_imgact+0xa50/frame 0xfffffe0118ccc6e0
kern_execve() at kern_execve+0x7f9/frame 0xfffffe0118ccca50
sys_execve() at sys_execve+0x4c/frame 0xfffffe0118cccad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0118cccbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0118cccbf0
--- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x3d9c28bcdfa, rsp = 0x632d2e7b49f8, rbp = 0x632d2e7b4b40 ---

I think that colleagues sent a crash report too.

[franco]

Hi Manuel,

Ok, we're tracking this via a FreeBSD bug report from October 2016: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

We may have to ask on the mailing list for help / advice. In the mean time, this may be hardware related. What are your system specs? (amd64 vs. i386, RAM, CPU type, etc.)

You can also send a crash report (System: Firmware: Reporter, Create report and send me a note, it will have the necessary details).


Thank you,
Franco

[manueljben]

due there is some confidential info, PM sent

[manueljben]

As we said before, opnsense 17.1.1 doesn't solve the VPN "localhost"/multiwan issue, but we have more info concerning about that, if it will help to trace/debug the issue, seems that:

 sysctl net.pf.share_forward=0

fixes the issue and we can connect to vpn using any of WAN1 or WAN2 external IP as we did in 16.7.14,  but we are not sure that line don't broke other thing, it can be used as a permanent solution or is a temporary one while nat/routing/xxxx is revised?

thanks in advance!

[franco]

Hi Manuel,

Are you still having the crash reports regarding turnstile_broadcast()?

If yes, a FreeBSD developer provided a test kernel:

# opnsense-update -kr 17.1.1-rwdebug
# /usr/local/etc/rc.reboot

It still panics, but will print vital debug information when doing so.

Running it and providing the output will hopefully help resolve this.


Cheers,
Franco

[franco]

PS: Please direct further info in the main thread: https://forum.opnsense.org/index.php?topic=4414.0