Messages

Hi,

I'm running opnSense on a china-box with intel 1Gbit/s NICs (igbn).

What's the CPU model exactly ?

PPPoE depends a lot on the Single Core Performance of the CPU when using FreeBSD based software !!

Yeah I know that pppoe on FreeBSD is limited by a single core - sadly. The CPU in question is a "Intel(R) Core(TM) i7-5550U CPU @ 2.00GHz". After tweaking around I hadn't had any issues on any of my opnSense installations to get speeds of >1 Gbit/s through a PPPoE link.

I was just wondering why the segmentation offloading seems to interact with using PPPoE.

Hi,

I'm running opnSense on a china-box with intel 1Gbit/s NICs (igbn). opnsense is virtualized using VMware as hypervisor. The opnSense VM NICs ar configured as VMXNET3.

The Box in question was using a "Deutsche Glasfaser" FTTH connection. WAN received its IP via DHCP. The throughput was 400/200 Mbps (the same speed the ISP offered in their contract).

The ISP was now changed to "1&1" using the same FTTH line. 1&1 requires a connection via PPPoE. From other PPPoE based connections I use with other opnSense gear I know that PPPoE and higher bandwidths on FreeBSD is a bit of an issue.

Im my case switching from DHCP to PPPoE reduced the downstream from 400Mbps to only ~30Mbps. I then remembered some tweak-settings within opnSense. The setting that provided the biggest improvement was "Hardware TSO" within the Interface Settings. Once I "unchecked" Disable hardware TCP segmentation offload, I instantly received 600Mbps in Downstream (the new bandwidth the ISP offers). The settings had no effect towards my upstream which always worked like the contract specified (200 old contract and 300Mbps new contract).

I wonder why moving the TCP sgementation towards "hardware" improves the throughput so much. And why it didn't matter when using DHCP instead of PPPoE. MTU size is 1492 (auto configured by opnSense).

Hi Franco,

Eventually I'd like to add some from of automatic AFTR setup and now I see that Uwe is offering a testbed :)

yeah some more user friendly option to setup AFTR configuration would help a lot of M-Net users who would like to use opnSense but are not willing to pay for Dual-Stack or do a manual GIF-tunnel setup.

thanks,
Fabian

So this also should fix this user's problem: https://forum.m-net.de/viewtopic.php?f=23&t=15580&p=141643&hilit=GIF#p141643 thanks for looking into this!

From what I know M-Net's AFTR also works when you have a dual-stack contract.

Hi,

I was able to reproduce the issue. When the firewall comes up and the configured DNS server is not responding, wireguard does some weird stuff like chosing a random port, see images:








recovery from that state was only possible by fixing the DNS first (obviously) and then either restarting the whole firewall or restating the wireguard instance via the "apply" button. Neither the stale connections nor the restart wireguard cron jobs do work in this case.

I'm sure there is some reason behind this behaviour, however it somewhat degrades the availability of the VPN connections ;-)

best,
Dark-Sider

Thanks for your replies thus far! However it's not that simple (see below)

If your connection via wireguard is inbound, it means that your devices resolve the DNS entry. On your OPNsense, I would not expect any hostname in the configuration, wireguard just binds to all interfaces (* 51820).

So I assume your clients resolve the wrong IP address and send packets to the wrong destination.

It's not a road-warrior-only setup. The wg0 instance has also one outbound connection which resolves a dynamic hostname. My clients don't have any issues with the dns resolution of my wg-server, as I can just restart the service on my clients, reboot them or whatever. Usually it immediately starts to work, once I rest wireguard on the server. The issue I observed most recently was, that according to the wg-status-page in opnsense it was listening on a port 33### while the tunnel configuraton shows port 51820.

I wrote the stale connection job and of course, it only does half of the job...

Thanks Uwe for contributing to the project - didn't realize you were the author of this little helper :-)

In your scenario, with a road warrior, the cron job on the "server" side does not help at all, because the road warrior peer has to initiate a new connection. If he fails to notice the stale connection, it will never restart and thus the DynDNS update will go unnoticed.

Yes I fully get that, and that is not my issue tbh, since I just could restart a road-warrior-client while travelling. However restarting the server without a "dial-in" option is not possible. More on my analysis below.

Luckily, for M-Net, there is a fix to that: They usually do not change their IPv6 prefix on reconnection, much unlike the IPv4. Thus, if you use IPv6 only or IPv6 and IPv4 in your DynDNS, it will effectively work as if you have a fixed IP(v6). In that case, your roadwarrior clients will regain contact within the same Wireguard session.

I know they say they only change the IPv6 prefix after a set amount of "offline" time. A simple restart of my opnsense VM, only lasting <60sec triggers a new IPv6 prefix. But that's also fine, since reconnects don't happen that often...

Call me any time if you have questions, you have my number!

Yeah we can have a chat on the weekend, I'm very busy until friday (business trip).


As mentioned above, I did a bit more tinkering and thinking last night. As you all have mentioned, the road-warriors shouldn't have an issue connecting to the server as they have their own DNS-resolution. However the problem was not that the clients couldn't resolve my server (packets were arriving, checked with packed capture) but nothig was sent back. As described earlier, the wg-status page reported wg to run on port 33### which is not what is configured and I don't know where it pulled that port from. After restarting the wg-tunnel on the server everything worked fine, clients were online immediatly.

So I went ahead and restarted my opnSense VM (multiple times) - the result was always the same: wireguard worked fine for the roadwarriors, the other site2site box also connected after some offline time and with the help of the stale cron job. So, problem solved? Not quite.

It still bugged me why the wg service "failed" after the reboot a couple of days ago... Then it occured to me that I rebooted the opnSense VM because of issues with my DNS resolution. DHCP offers opnSense as DNS, and Unbound on opnSense uses my local pihole as an upstream host. I initially thought Unbound failed (which it does once a year or so) but the restart did NOT fix my DNS issue this time. It actually came down to pihole needing an update and restart.

So opnsense was rebooted without having a working DNS reachable, as opnSense exclusively uses my pihole for all its queries.

Since my local wg0 instance also has an outbound site2site connection which uses a dynamic hostname as the target it was not able to resolve that hostname after the system booted. In some other thread I read, that if wg DNS fails during startup the service might get stuck. Without having tested this further, I beleive that this is what happened to me. The hung service also does not accept any inbound connections, as the peers were not shown on the status page and wg0 was listening on the wrong port. When I'm back home I'll simulate this with a missing DNS server after rebooting opnSense and see if I run into the same issue.

TL;DR
- Clients never were an issue, they can easily be restarted to "update" any outdated hostnames, but server was not responding
- opnSense was rebooted without any DNS available, maybe wg didn't fully initialize, which als didn't allow clients to connect
- Stale connection script was successfully tested with my site2site connection, after running the script, the connection came back up.

Hi,

since wireguard made its way into opnsense it works ok-ish for however its "stability" is not comparable to OpenVPN. However I like the concept behind wireguard therefore I'm putting up with some issues and still using it.

Last week I had to restart my opnSense box (25.1.12) and me being away on a business trip, wireguard failed me (again) after the restart. I usually solve this issue by de- and reactivating my one and only wg0 instance via the webgui. After restarting wg0 everything works as it is supposed to.

Since the issue caught me cold (again) I did some forum reading and found interesting threads regarding wg and DNS, stale connections etc:
https://forum.opnsense.org/index.php?topic=49432.0
https://forum.opnsense.org/index.php?topic=37905.0
https://forum.opnsense.org/index.php?topic=42648.0

Honestly I didn't know about the quirks of wg and DNS resolve issues after your dynamic IP refreshes or wg only doing DNS queries once on startup and not refreshing it. One might argue that using a static ip would solve such problems, however static IPs on consumer lines are hard to get these days. Even IPv6 is dynamic with my ISP.

While I think wg's behaviour is a severe design oversight in the protocol / moudule (nothing related to opnsense though) I appreciate the effort that a cron job exists that somewhat is supposed to fix the issue.

I activated the cron-job to run */5 * * * * however my issue was not resolved. My mobile phone was not able to connect via IPv6 or IPv4 (both usually works) to my opnSense box. I did a packet capture on 51820 and the packets from phone arrived but no response was sent back.

I then noticed there is another cron-job called "restart wireguard service" I also did setup this job */7 * * * * however after waiting for 14 minutes my wireguard log still showed that the service was started last week - no other log entries.

While looking at the logs I found that the wg status page was quite empty, only showing wg0 with my local endpoint at port 33###. Didn't notice this first, but my wg setup uses only port 51820. Also no peers were shown at all on the status page.

I have 3 road warrior peers configured ("dial in only") being my phone, my laptop and a mobile gl_inet travel router. I also have a site2site connection configured to a remote network.

Only after I deactivate my Instance and reactivate it, all 4 peers will be listed on the status page. When the peers are listed the connections start working again.

My openSense runs virtualized (yes it could need a firmware update which I will do later) and is on a dial up connection at a German ISP (M-Net) using both IPv4 and IPv6 connectivity via pppoe. Luckily my ISP-connection is hyper-stable so reboots and disconnects thus ip-changes happen very rarely.

I still wonder why wg needs a kick in the... after my box boots up? And shouldn't that restart wg cron-job also fix my issue?

thanks,
Dark-Sider

Hi again,

I meanwhile solved the problem with the help of mitmproxy to inspect the headers.

As it turns out nginx rewirtes several headers to lower-case. In my case the header field X-XSRF-TOKEN was changed to x-xsrf-token which caused the problem within the webapp.

I hot-fixed the problem by including

Code Select Expand

proxy_set_header X-XSRF-TOKEN $http_x_xsrf_token;
in the /usr/local/opnsense/service/templates/OPNsense/Nginx/location.conf template.

Is saw that the generated nginx.conf contains includes for each location in the form of:

Code Select Expand


include <guid>_post/*.conf;


Would you suggest to create that directory and put an include config there myself? Do you know of any other ways to have nginx not change the case of header-fields?

Edit: it appears that client <-> nginx is using http2 and nginx <-> webapp is using http 1.1. As http2 requires headers in lowercase format chrome actually sends them in lower case.... aye.

thx
Dark-Sider

Hi,

I have a web-application that up until now used a NAT port-forward. However I need URL-based filtering. As the application is "closed", my solution of choice was to setup a nginx reverse proxy in opnsense and add some ACL-based filtering. It all works great, except on small but important detail:

The web-app displays a logon-page. If I enter the correct username / password (while using nginx as reverse proxy) it displays an login error page. The web-app's log shows:

Code Select Expand


[ERROR] 2022-02-24 13:14:01,144 [qtp142733894-87857] Unauthorized access detected
com.appName.AuthenticationException: Invalid CSRF token


If I then press "reload" on the browser, I'm magically logged in and everything works. Since the web-app is also accessed by external users, I would like to get it 100% working though :)

The reverse proxy configuration is very basic at this stage:
Upstream, and Upstream server are configured with correct ssl certs.
I tried the Upstream configuration with Proxy Protocol enabled and disabled (no difference)

Location configuration is as basic as it can get (just enforce HTTPS) I also tried to enable and disable the response/request buffering (no idea what this actually does though)

The HTTP-Server configuration is also very basic. It just listens on a specific virtual IP on specific ports. Location is set and SSL-Cert is set.
I also tried enabling proxy protocol within the HTTP-Server options, and setting the real ip source to all options. Nothing worked (I restarted nginx after each configuration change)

I have not defined any security headers.

Any ideas what my configuration is missing?

regards
Dark-Sider

Here are some updates on this issue. I have had opened a ticket with AVM the manufacturer of the VOIP-Router. I sent them packet captures and a lot of details. They concluded (and I might have to agree) that issue has to be on the firewall. So I did some more testing tonight and the results are really strange. The MTU for my PPPoE device is set to 1492, the LAN interface MTU is 1500.

The SIP response going out to sipgate is too big fit in one packet, so the Fritz!Box already devides it into 2 packets: 1st packet has a length of 1502 and the 2nd packet a length of 237. Those lengths are the full physical ethernet frames reaching opnsense. So 18 bytes ethernet header and footer are deducted, yeilding a full IPv6 size of 1484 which is way below 1492. But opnsense replies to my Fritz!Box with ICMPv6 Packet Size too big.

I dumped the SIP-Payload of both fragments and recombined them to the total size of 1604 Bytes (just the SIP response), opened and piped it to netcat on a linux box in my LAN. I chose sipgates server as destination and used the 5060 port. The packet was again fragmented by nc/linux to the exact same 2 fragment sizes as the fritzBox did, but it went through (!!!). When I compared the headers I only found that the fritzBox chose a flow label of 0x0 and nc put in some (maybe not so) random hex.

When doing a packet capture on the pfsense (filtering to the packet sending machine), I always get those messages:

Code Select Expand


LAN
vmx2 20:44:11.309778 IP6 2001:a61:xxx > 2001:ab7::4: frag (0|1440) 38065 > 5060: UDP, bad length 1604 > 1432
LAN
vmx2 20:44:11.309787 IP6 2001:a61:xxx > 2001:ab7::4: frag (1440|172)


however when the Fritz!Box sends it, it gets dropped and that ICMPv6 message is returned. - the other numbers (except for the source ip) stay the same...

over at pfsense I found that there apparently existed a FreeBSD bug: https://forum.netgate.com/topic/123169/problems-with-mtu-and-dropped-packets/10 https://redmine.pfsense.org/issues/8165 but as opnsense is already at 12.x I think this should be fixed?!

Edit: The issue persists after upgrading to 21.1.3

Edit2: more testing:
This actually might be related to the bug I linked above. I did the following testing:
pfctl -d      The packets go through
pfctl -e      The packets get blocked again with the "too big" message


regards
Darky

Hi,

I'm running an AVM Fritz!Box router as a LAN client for my VOIP needs. It's a neat all-in-one PBX solution for internal ISDN, DECT and SIP phones. Internet is provided by the German ISP M-Net through FTTH/GPON. OPNsense connects via PPPoE to the Internet (MTU set to 1492 in the pppoe options).

To avoid NAT issues with SIP I configured all SIP-accounts as IPv6 only. Under Firewall settings I have allowed sipagte's servers so they can talk to my AVM Fritz!Box.

I'm using both, sipagte and my ISP's voip service. While my ISP's voip works fine, incoming sipagate calls cannot be answered (caller still hears the phone ringing although the call was picked up).

To troubleshoot the problem I did some packet capturing. As it turns out the SIP/SDP 200 OK packet that is sent  to sipgate is "rejected" by OPNsense with an ICMPv6 "Packet too big" and therefore never reaching my voip provider. The ICMPv6 Packet Too Big contains the correct MTU of 1484 (the value that is displayed und calculated MTU for PPP in the pppoe options), the Packet that is rejected has a packet size of 1494.

What would be the expected behavior there? Is the Fritz!Box expected to rentransmit the packet with a smaller packet size or is OPNsense expected to refragment the packet to fit the MTU? If so, what setting do I miss?

regards,
Fabian

Thanks mate, made the changes and at a first glance it works.

Is this the recommended setup to have one box as "master" and let the "client" (DG) initiate the connection?
https://docs.opnsense.org/manual/how-tos/wireguard-s2s.html -- I assumed that the connection settings have to be mutual. Giving it a second thought though, roadwarriors don't have reachable DNS entries well...

Hey guys,

I'm running a wireguard VPN between two dial-up sites in Germany. One provider in M-Net (local carrier in Munich) and the other provider is Deutsche Glasfaser (a FTTH provider, active in several parts of Germany).

For the ease of this post I'll refer to the two sites as MNET and DG. Both sites have installed identical hardware, opnsense is running in an ESXi 6.7 VM on a qotom headless PC (core i7). I have plenty of experience with that hardware and it gets the job done quite well.

Although both sites connect via FTTH (gpon) mnet requires a pppoe connection which featurese IPv4/IPv6 Dual Stack. The IPv6 prefix (a /56) is received via the IPv4 connectivity through the pppoe connection. The pppoe interface does not receive a public but a linklocal IPv6 address. DG just issues IPv4 and IPv6 addresses via DHCP (no pppoe required). The WAN interface also gets a public /128 IPv6 address. Since DG only offers cg NAT for IPv4 connectivity, I'm using wireguard with IPv6 to connect the two sites.

As the IPv6 subnets assigned by both providers are not static, I use dynv6 to update my IPv6 addresses on both sites. Since MNet does not provide a public v6 address on the pppoe/wan I selected the LAN-IPv6-adresses to be served on dynv6. This is checked, and both boxes are pingable with their respective hostnames from the internet using IPv6.

The wireguard part of the configuration was quite straight forward:
- install wirguard
- configure the local part on both boxes
- configure the remote-endpoint on both boxes with the public key of the other box
- activate the endpoints within the respective local config.

For the allowed IP-addresses I selected the corresponding remote network, and the remote tunnel address with /32

MNET uses 172.20.0.0/16, 172.19.0.0.2/32
DG uses 192.168.0.0/24, 172.19.0.0.1/32
as allowed nets. The allowed net is the remote LAN-Subnet of the other box and the other boxe's tunnell address.

I don't need IPv6 within the tunnel so it's only IPv4 for the tunnel.

For the sake of ease both sites have the firewall settings wg0 to IPv4+IPv6 any to any. (Protocol, sourc, dest, port...)

Both boxes run the latest 20.7 opnSense version.

Now the strange part: The tunnel sometimes works and sometimes doesn't work - although the IPv6 prefixes of both sites have not changed and DNS returns the correct IPv6 address. The DNS record also has only an AAAA record configured, to avoid IPv4 connections.

When I look at the wireguard "List Configuration" output on each site i see sth like this:

Code Select Expand


interface: wg0
  public key: XXX
  private key: (hidden)
  listening port: 51820

peer: XXX
  endpoint: [2a00:6020:1000:xxx]:51820
  allowed ips: 172.19.0.2/32, 172.20.0.0/16
  latest handshake: 19 minutes, 38 seconds ago
  transfer: 22.06 KiB received, 14.01 KiB sent
  persistent keepalive: every 1 minute


The other site displays similiar information. If the connection is working, then the received and sent counters go up as expected and traffic passes through the network. But I very often reach a state where both boxes just send packets and the other box won't receive any, or it receives the packet and even sends out an answer (check with tcpdump) but the sending box won't receive any packets.

I also noticed, that the IPv6 address noted in the "peer" section of the List Configuration view on the MNET box, does not match the configured address in the endpoint dialogue or the the resolved address from hostname. (tried both) but keeps displaying the WAN address of the DG box instead of it's LAN address. So it seems to me that wg tries to origniate traffic from DGs WAN instead of LAN IP. I updated the DYNV6 configuration to match the WAN - but no luck there either.

WG runs on 51820 on both boxes and the ports are opened in the Firewall (wan/pppoe) with "THIS FIREWALL" as target. To allow the ICMP-Echo for both boxes I use the same rule but with ICMP...

any ideas? any fundamental flaws in my thinking?

the odd thing is, that it sometimes works and sometimes doesn't, and I can't see whe :(

regards,
Fabian

@Dark-Sider:
So, you set the gateway IP to the real IP instead of "dynamic" and tick "disable routes" and have what exactly???

I set the gateway IP to the real IP so I'm not affected by that bug that won't let me add the gateway. I personally would prefer the dynamic solution. Since I control both endpoints of this setup the gateway's IP adress won't suddenly change so this setup works for me right now.

I have to tick disable routes since I have to add 0.0.0.0/0 in the wireguard's allowed list. If I don't tick it - a new route with highest priority is added to opnsense once the tunnel is established and all traffic goes down that new route. If this should not happen anymore when the other gateways have that "upstream" box ticked, than there is still a bug present, as it happens to me in 19.7.2

Without ticking the box and adding the route for the tunnel network manually on console my setup starts working as it is supposed to be.

So a option, what kind of routes should be added once the tunnel IF goes live would be cool. Usually this would be just the route for the tunnel-network.

regards,

Fabian