Messages

Hi,

I was able to reproduce the issue. When the firewall comes up and the configured DNS server is not responding, wireguard does some weird stuff like chosing a random port, see images:








recovery from that state was only possible by fixing the DNS first (obviously) and then either restarting the whole firewall or restating the wireguard instance via the "apply" button. Neither the stale connections nor the restart wireguard cron jobs do work in this case.

I'm sure there is some reason behind this behaviour, however it somewhat degrades the availability of the VPN connections ;-)

best,
Dark-Sider

Thanks for your replies thus far! However it's not that simple (see below)

If your connection via wireguard is inbound, it means that your devices resolve the DNS entry. On your OPNsense, I would not expect any hostname in the configuration, wireguard just binds to all interfaces (* 51820).

So I assume your clients resolve the wrong IP address and send packets to the wrong destination.

It's not a road-warrior-only setup. The wg0 instance has also one outbound connection which resolves a dynamic hostname. My clients don't have any issues with the dns resolution of my wg-server, as I can just restart the service on my clients, reboot them or whatever. Usually it immediately starts to work, once I rest wireguard on the server. The issue I observed most recently was, that according to the wg-status-page in opnsense it was listening on a port 33### while the tunnel configuraton shows port 51820.

I wrote the stale connection job and of course, it only does half of the job...

Thanks Uwe for contributing to the project - didn't realize you were the author of this little helper :-)

In your scenario, with a road warrior, the cron job on the "server" side does not help at all, because the road warrior peer has to initiate a new connection. If he fails to notice the stale connection, it will never restart and thus the DynDNS update will go unnoticed.

Yes I fully get that, and that is not my issue tbh, since I just could restart a road-warrior-client while travelling. However restarting the server without a "dial-in" option is not possible. More on my analysis below.

Luckily, for M-Net, there is a fix to that: They usually do not change their IPv6 prefix on reconnection, much unlike the IPv4. Thus, if you use IPv6 only or IPv6 and IPv4 in your DynDNS, it will effectively work as if you have a fixed IP(v6). In that case, your roadwarrior clients will regain contact within the same Wireguard session.

I know they say they only change the IPv6 prefix after a set amount of "offline" time. A simple restart of my opnsense VM, only lasting <60sec triggers a new IPv6 prefix. But that's also fine, since reconnects don't happen that often...

Call me any time if you have questions, you have my number!

Yeah we can have a chat on the weekend, I'm very busy until friday (business trip).


As mentioned above, I did a bit more tinkering and thinking last night. As you all have mentioned, the road-warriors shouldn't have an issue connecting to the server as they have their own DNS-resolution. However the problem was not that the clients couldn't resolve my server (packets were arriving, checked with packed capture) but nothig was sent back. As described earlier, the wg-status page reported wg to run on port 33### which is not what is configured and I don't know where it pulled that port from. After restarting the wg-tunnel on the server everything worked fine, clients were online immediatly.

So I went ahead and restarted my opnSense VM (multiple times) - the result was always the same: wireguard worked fine for the roadwarriors, the other site2site box also connected after some offline time and with the help of the stale cron job. So, problem solved? Not quite.

It still bugged me why the wg service "failed" after the reboot a couple of days ago... Then it occured to me that I rebooted the opnSense VM because of issues with my DNS resolution. DHCP offers opnSense as DNS, and Unbound on opnSense uses my local pihole as an upstream host. I initially thought Unbound failed (which it does once a year or so) but the restart did NOT fix my DNS issue this time. It actually came down to pihole needing an update and restart.

So opnsense was rebooted without having a working DNS reachable, as opnSense exclusively uses my pihole for all its queries.

Since my local wg0 instance also has an outbound site2site connection which uses a dynamic hostname as the target it was not able to resolve that hostname after the system booted. In some other thread I read, that if wg DNS fails during startup the service might get stuck. Without having tested this further, I beleive that this is what happened to me. The hung service also does not accept any inbound connections, as the peers were not shown on the status page and wg0 was listening on the wrong port. When I'm back home I'll simulate this with a missing DNS server after rebooting opnSense and see if I run into the same issue.

TL;DR
- Clients never were an issue, they can easily be restarted to "update" any outdated hostnames, but server was not responding
- opnSense was rebooted without any DNS available, maybe wg didn't fully initialize, which als didn't allow clients to connect
- Stale connection script was successfully tested with my site2site connection, after running the script, the connection came back up.

Hi,

since wireguard made its way into opnsense it works ok-ish for however its "stability" is not comparable to OpenVPN. However I like the concept behind wireguard therefore I'm putting up with some issues and still using it.

Last week I had to restart my opnSense box (25.1.12) and me being away on a business trip, wireguard failed me (again) after the restart. I usually solve this issue by de- and reactivating my one and only wg0 instance via the webgui. After restarting wg0 everything works as it is supposed to.

Since the issue caught me cold (again) I did some forum reading and found interesting threads regarding wg and DNS, stale connections etc:
https://forum.opnsense.org/index.php?topic=49432.0
https://forum.opnsense.org/index.php?topic=37905.0
https://forum.opnsense.org/index.php?topic=42648.0

Honestly I didn't know about the quirks of wg and DNS resolve issues after your dynamic IP refreshes or wg only doing DNS queries once on startup and not refreshing it. One might argue that using a static ip would solve such problems, however static IPs on consumer lines are hard to get these days. Even IPv6 is dynamic with my ISP.

While I think wg's behaviour is a severe design oversight in the protocol / moudule (nothing related to opnsense though) I appreciate the effort that a cron job exists that somewhat is supposed to fix the issue.

I activated the cron-job to run */5 * * * * however my issue was not resolved. My mobile phone was not able to connect via IPv6 or IPv4 (both usually works) to my opnSense box. I did a packet capture on 51820 and the packets from phone arrived but no response was sent back.

I then noticed there is another cron-job called "restart wireguard service" I also did setup this job */7 * * * * however after waiting for 14 minutes my wireguard log still showed that the service was started last week - no other log entries.

While looking at the logs I found that the wg status page was quite empty, only showing wg0 with my local endpoint at port 33###. Didn't notice this first, but my wg setup uses only port 51820. Also no peers were shown at all on the status page.

I have 3 road warrior peers configured ("dial in only") being my phone, my laptop and a mobile gl_inet travel router. I also have a site2site connection configured to a remote network.

Only after I deactivate my Instance and reactivate it, all 4 peers will be listed on the status page. When the peers are listed the connections start working again.

My openSense runs virtualized (yes it could need a firmware update which I will do later) and is on a dial up connection at a German ISP (M-Net) using both IPv4 and IPv6 connectivity via pppoe. Luckily my ISP-connection is hyper-stable so reboots and disconnects thus ip-changes happen very rarely.

I still wonder why wg needs a kick in the... after my box boots up? And shouldn't that restart wg cron-job also fix my issue?

thanks,
Dark-Sider

Hi again,

I meanwhile solved the problem with the help of mitmproxy to inspect the headers.

As it turns out nginx rewirtes several headers to lower-case. In my case the header field X-XSRF-TOKEN was changed to x-xsrf-token which caused the problem within the webapp.

I hot-fixed the problem by including

Code Select Expand

proxy_set_header X-XSRF-TOKEN $http_x_xsrf_token;
in the /usr/local/opnsense/service/templates/OPNsense/Nginx/location.conf template.

Is saw that the generated nginx.conf contains includes for each location in the form of:

Code Select Expand


include <guid>_post/*.conf;


Would you suggest to create that directory and put an include config there myself? Do you know of any other ways to have nginx not change the case of header-fields?

Edit: it appears that client <-> nginx is using http2 and nginx <-> webapp is using http 1.1. As http2 requires headers in lowercase format chrome actually sends them in lower case.... aye.

thx
Dark-Sider

Hi,

I have a web-application that up until now used a NAT port-forward. However I need URL-based filtering. As the application is "closed", my solution of choice was to setup a nginx reverse proxy in opnsense and add some ACL-based filtering. It all works great, except on small but important detail:

The web-app displays a logon-page. If I enter the correct username / password (while using nginx as reverse proxy) it displays an login error page. The web-app's log shows:

Code Select Expand


[ERROR] 2022-02-24 13:14:01,144 [qtp142733894-87857] Unauthorized access detected
com.appName.AuthenticationException: Invalid CSRF token


If I then press "reload" on the browser, I'm magically logged in and everything works. Since the web-app is also accessed by external users, I would like to get it 100% working though :)

The reverse proxy configuration is very basic at this stage:
Upstream, and Upstream server are configured with correct ssl certs.
I tried the Upstream configuration with Proxy Protocol enabled and disabled (no difference)

Location configuration is as basic as it can get (just enforce HTTPS) I also tried to enable and disable the response/request buffering (no idea what this actually does though)

The HTTP-Server configuration is also very basic. It just listens on a specific virtual IP on specific ports. Location is set and SSL-Cert is set.
I also tried enabling proxy protocol within the HTTP-Server options, and setting the real ip source to all options. Nothing worked (I restarted nginx after each configuration change)

I have not defined any security headers.

Any ideas what my configuration is missing?

regards
Dark-Sider

Here are some updates on this issue. I have had opened a ticket with AVM the manufacturer of the VOIP-Router. I sent them packet captures and a lot of details. They concluded (and I might have to agree) that issue has to be on the firewall. So I did some more testing tonight and the results are really strange. The MTU for my PPPoE device is set to 1492, the LAN interface MTU is 1500.

The SIP response going out to sipgate is too big fit in one packet, so the Fritz!Box already devides it into 2 packets: 1st packet has a length of 1502 and the 2nd packet a length of 237. Those lengths are the full physical ethernet frames reaching opnsense. So 18 bytes ethernet header and footer are deducted, yeilding a full IPv6 size of 1484 which is way below 1492. But opnsense replies to my Fritz!Box with ICMPv6 Packet Size too big.

I dumped the SIP-Payload of both fragments and recombined them to the total size of 1604 Bytes (just the SIP response), opened and piped it to netcat on a linux box in my LAN. I chose sipgates server as destination and used the 5060 port. The packet was again fragmented by nc/linux to the exact same 2 fragment sizes as the fritzBox did, but it went through (!!!). When I compared the headers I only found that the fritzBox chose a flow label of 0x0 and nc put in some (maybe not so) random hex.

When doing a packet capture on the pfsense (filtering to the packet sending machine), I always get those messages:

Code Select Expand


LAN
vmx2 20:44:11.309778 IP6 2001:a61:xxx > 2001:ab7::4: frag (0|1440) 38065 > 5060: UDP, bad length 1604 > 1432
LAN
vmx2 20:44:11.309787 IP6 2001:a61:xxx > 2001:ab7::4: frag (1440|172)


however when the Fritz!Box sends it, it gets dropped and that ICMPv6 message is returned. - the other numbers (except for the source ip) stay the same...

over at pfsense I found that there apparently existed a FreeBSD bug: https://forum.netgate.com/topic/123169/problems-with-mtu-and-dropped-packets/10 https://redmine.pfsense.org/issues/8165 but as opnsense is already at 12.x I think this should be fixed?!

Edit: The issue persists after upgrading to 21.1.3

Edit2: more testing:
This actually might be related to the bug I linked above. I did the following testing:
pfctl -d      The packets go through
pfctl -e      The packets get blocked again with the "too big" message


regards
Darky

Hi,

I'm running an AVM Fritz!Box router as a LAN client for my VOIP needs. It's a neat all-in-one PBX solution for internal ISDN, DECT and SIP phones. Internet is provided by the German ISP M-Net through FTTH/GPON. OPNsense connects via PPPoE to the Internet (MTU set to 1492 in the pppoe options).

To avoid NAT issues with SIP I configured all SIP-accounts as IPv6 only. Under Firewall settings I have allowed sipagte's servers so they can talk to my AVM Fritz!Box.

I'm using both, sipagte and my ISP's voip service. While my ISP's voip works fine, incoming sipagate calls cannot be answered (caller still hears the phone ringing although the call was picked up).

To troubleshoot the problem I did some packet capturing. As it turns out the SIP/SDP 200 OK packet that is sent  to sipgate is "rejected" by OPNsense with an ICMPv6 "Packet too big" and therefore never reaching my voip provider. The ICMPv6 Packet Too Big contains the correct MTU of 1484 (the value that is displayed und calculated MTU for PPP in the pppoe options), the Packet that is rejected has a packet size of 1494.

What would be the expected behavior there? Is the Fritz!Box expected to rentransmit the packet with a smaller packet size or is OPNsense expected to refragment the packet to fit the MTU? If so, what setting do I miss?

regards,
Fabian

Thanks mate, made the changes and at a first glance it works.

Is this the recommended setup to have one box as "master" and let the "client" (DG) initiate the connection?
https://docs.opnsense.org/manual/how-tos/wireguard-s2s.html -- I assumed that the connection settings have to be mutual. Giving it a second thought though, roadwarriors don't have reachable DNS entries well...

Hey guys,

I'm running a wireguard VPN between two dial-up sites in Germany. One provider in M-Net (local carrier in Munich) and the other provider is Deutsche Glasfaser (a FTTH provider, active in several parts of Germany).

For the ease of this post I'll refer to the two sites as MNET and DG. Both sites have installed identical hardware, opnsense is running in an ESXi 6.7 VM on a qotom headless PC (core i7). I have plenty of experience with that hardware and it gets the job done quite well.

Although both sites connect via FTTH (gpon) mnet requires a pppoe connection which featurese IPv4/IPv6 Dual Stack. The IPv6 prefix (a /56) is received via the IPv4 connectivity through the pppoe connection. The pppoe interface does not receive a public but a linklocal IPv6 address. DG just issues IPv4 and IPv6 addresses via DHCP (no pppoe required). The WAN interface also gets a public /128 IPv6 address. Since DG only offers cg NAT for IPv4 connectivity, I'm using wireguard with IPv6 to connect the two sites.

As the IPv6 subnets assigned by both providers are not static, I use dynv6 to update my IPv6 addresses on both sites. Since MNet does not provide a public v6 address on the pppoe/wan I selected the LAN-IPv6-adresses to be served on dynv6. This is checked, and both boxes are pingable with their respective hostnames from the internet using IPv6.

The wireguard part of the configuration was quite straight forward:
- install wirguard
- configure the local part on both boxes
- configure the remote-endpoint on both boxes with the public key of the other box
- activate the endpoints within the respective local config.

For the allowed IP-addresses I selected the corresponding remote network, and the remote tunnel address with /32

MNET uses 172.20.0.0/16, 172.19.0.0.2/32
DG uses 192.168.0.0/24, 172.19.0.0.1/32
as allowed nets. The allowed net is the remote LAN-Subnet of the other box and the other boxe's tunnell address.

I don't need IPv6 within the tunnel so it's only IPv4 for the tunnel.

For the sake of ease both sites have the firewall settings wg0 to IPv4+IPv6 any to any. (Protocol, sourc, dest, port...)

Both boxes run the latest 20.7 opnSense version.

Now the strange part: The tunnel sometimes works and sometimes doesn't work - although the IPv6 prefixes of both sites have not changed and DNS returns the correct IPv6 address. The DNS record also has only an AAAA record configured, to avoid IPv4 connections.

When I look at the wireguard "List Configuration" output on each site i see sth like this:

Code Select Expand


interface: wg0
  public key: XXX
  private key: (hidden)
  listening port: 51820

peer: XXX
  endpoint: [2a00:6020:1000:xxx]:51820
  allowed ips: 172.19.0.2/32, 172.20.0.0/16
  latest handshake: 19 minutes, 38 seconds ago
  transfer: 22.06 KiB received, 14.01 KiB sent
  persistent keepalive: every 1 minute


The other site displays similiar information. If the connection is working, then the received and sent counters go up as expected and traffic passes through the network. But I very often reach a state where both boxes just send packets and the other box won't receive any, or it receives the packet and even sends out an answer (check with tcpdump) but the sending box won't receive any packets.

I also noticed, that the IPv6 address noted in the "peer" section of the List Configuration view on the MNET box, does not match the configured address in the endpoint dialogue or the the resolved address from hostname. (tried both) but keeps displaying the WAN address of the DG box instead of it's LAN address. So it seems to me that wg tries to origniate traffic from DGs WAN instead of LAN IP. I updated the DYNV6 configuration to match the WAN - but no luck there either.

WG runs on 51820 on both boxes and the ports are opened in the Firewall (wan/pppoe) with "THIS FIREWALL" as target. To allow the ICMP-Echo for both boxes I use the same rule but with ICMP...

any ideas? any fundamental flaws in my thinking?

the odd thing is, that it sometimes works and sometimes doesn't, and I can't see whe :(

regards,
Fabian

@Dark-Sider:
So, you set the gateway IP to the real IP instead of "dynamic" and tick "disable routes" and have what exactly???

I set the gateway IP to the real IP so I'm not affected by that bug that won't let me add the gateway. I personally would prefer the dynamic solution. Since I control both endpoints of this setup the gateway's IP adress won't suddenly change so this setup works for me right now.

I have to tick disable routes since I have to add 0.0.0.0/0 in the wireguard's allowed list. If I don't tick it - a new route with highest priority is added to opnsense once the tunnel is established and all traffic goes down that new route. If this should not happen anymore when the other gateways have that "upstream" box ticked, than there is still a bug present, as it happens to me in 19.7.2

Without ticking the box and adding the route for the tunnel network manually on console my setup starts working as it is supposed to be.

So a option, what kind of routes should be added once the tunnel IF goes live would be cool. Usually this would be just the route for the tunnel-network.

regards,

Fabian

some additional input:

if I uncheck disable routes mit routing table looks like this:

Code Select Expand


Internet:
Destination        Gateway            Flags     Netif Expire
0.0.0.0/1          wg0                US          wg0
default            ppp-default.m-onli UGS      pppoe0
[...]


and all traffic is sent through wg0 - upstream flags are set correctly

Hi,

thanks for your swift reply. I changed the interface from DHCP to ipv4 none. I also tried to add a gateway like in the guide:

To do this, go to System ‣ Gateways ‣ Single and add a new gateway. Choose your WireGuard interface and set the Gateway to dynamic.

If I do so and put "dynamic" in the gateway's ip address field the gateway just won't appear in the list. Also there is no errormessage. If I put in the ip adress of my VPS server's tunnel interface (192.168.4.1) the gateway shows up. Maybe this is part of the new gateway code.

I already put my ISP's v6 and v4 gateway as upstream gateway, the vpn_gw does not have that option set.

As for the routing stuff, I'll check this once I'm home tonight. When I  check "disable routes" I now successfully created the route to the tunnel via commandline:

route add -inet 192.168.4.0/24 -link -iface wg0
Is there any way to make this persistant via the gui?

while I did so I noticed another strange thing:

netstat -r lists the following:
192.168.4.2        link#13            UH          wg0
While the .2 IP-Adress is the local ip of my wg interface it is routed out via wg0. If I ping 192.168.4.2 from my opnsense box, packets go into the tunnel, and come back after 180ms (from canada):

root@OPNsense:~ # ping 192.168.4.2
PING 192.168.4.2 (192.168.4.2): 56 data bytes
64 bytes from 192.168.4.2: icmp_seq=0 ttl=63 time=188.488 ms

traceroute to 192.168.4.2 (192.168.4.2), 3 hops max, 40 byte packets
1  192.168.4.1 (192.168.4.1)  94.379 ms  94.387 ms  94.121 ms

If I ping 192.168.4.2 from a device from my LAN it shows my local reply times <1ms

the local ip address of an openvpn tunnel however is bound to lo0
10.41.0.2          link#11            UHS         lo0

is this intended?`

With 19.7 gateway code changed (not Wireguard Code). You have to verify that your default gateway has "upstream" checked and your Wireguard gateaway not. Then this thing can't happen.

So you would say, that ticking the "disable routes" box is not necessary anymore with 19.7, when the ISP gateways have set the upstream flags and the vpn-gw does not?

regards,
Fabian

Hi,

to set the general picture: I installed wireguard devel on 19.1.x a couple of weeks ago. As I now had time play around with my planned wg VPN tunnel I finally did so yesterday.

I already have multiple ipsec and openvpn tunnels on my opnsense box running - so I thought this would be an easy and straight forward task :-)

The purpose for my setup is to route some hosts of my network through an VPS server in canada. This VPS runs ubuntu 18.04 with wireguard. I did manage to bring up the tunnel quite easily. The tunnel network is 192.168.4.0/24 with my wireguard box being .2 and the VPS .1

Without defining any other virtual interfaces (just setting the allow rule for the wirguard interface) I was able to ping the remote location - ping times and tcpdump also show that it actually is the VPS who answered.

My first try was to put 192.168.4.0/24 as "allowed ips" in the wireguard config on the opnsense box. Once the tunnel was established the routes were set and the tunnel worked.

The next step was to route actual internet upstream traffic through the vpn. I looked in the forum and found those threads:
https://forum.opnsense.org/index.php?topic=8998.0 (ovpn)
https://forum.opnsense.org/index.php?topic=4979.0 (ovpn)
https://forum.opnsense.org/index.php?topic=11737.0 (wg)

So I went ahead and did
- define an alias for one host in my network,
- setup the NAT rules (but not needed as NAT can be achieved on the VPS, just makes allowed ips config easier)
- created the virtual VPN interface with assigned wg0 to it and set it to ipv4 DHCP
- created the firewall rule with the alias and the gateway of the virtual VPN interface.

It did not work as I had expected. tcpdump on the VPS shows, that 1962.168.4.0/24 traffic is sent through the tunnel to the VPS but all other traffic just doesn't go through the tunnel - but also does not exit my opnsense on any other interface (ping shows a timeout).

I did a bit of thinking and replaced the allowed ips in the wg config with 0.0.0.0/0 as this might be the problem.

After restarting  the box my whole network was routed through the VPN not just this one host. However, the traffic actually went over the VPS in Canada. It took me some time to figure out that I probably had to check that disable routes checkbox in the wireguard config, das my default route was now set to 0.0.0.0/0 -> VPN

before doing so however I also read that 19.7 was now released and has wg as stable on board. So I reset my opnsense VM to a config state before my wireguard efforts and upgraded the box to 19.7.2 (uninstalled wg-devel and installed the stable wg package)

After setting up the wg tunnel again and verifying that the VPS is pingable from my LAN I again defined the virtual interface.

Then things changed.

First thing I noticed is, that there is no more gateway created automatically once you choose IPV4 DHCP for the virtual interface --- why?

I just stayed with ipv4 "none". Restarted  the tunnel and the VPN interface got the correct ip 192.168.4.2 - a minor change I thought.

I created the gateway manually leaving all IP fields blank just setting the Interface to VPN. After pressing OK and Applying there was no gateway added. I retried this a couple of times - no error message - no gateway.

I then created the gateway with the IP 192.168.4.1 and Interface VPN - which actually worked.

I thend changed the allowed ip again to 0.0.0.0/0 - and all traffic was again forced through the tunnel.

After that I ticked the "disable routes" option in the wg config and restarted the box (just to be safe). This is when things got really bad.

The default route was gone, however so was any route pointing 192.168.4.0/24 to go out through the VPN interface. Thus, 192.168.4.1 (my VPS) was no longer reachable. The routing table also shows only 192.168.4.2 (my opnsense box) on the VPN interface without any netmask.

I then tried to add the route manually as read on the forums, however this is not (not anymore?) possible. Whenn adding the route I can't chose my VPN interface. I only have choice of my PPPoE gws for my dial-up connection.

This is where I hit the roadblock, as I was not able to set
- 0.0.0.0/0 as allowed ips
- without setting a default route through the tunnel
- and I failed adding the 192.168.4.0/24 route manually

Were there changes in 19.7 how this is done?
Is there an obvious mistake I just don't see?

Thanks for helping out!

bye
Fabian

Hey guys,

thanks for bringing wireguard to opnsense! Maybe my question will become obsolete since a fix that should stop crashing is apparently on its way:

Is the crashing only related to configuration tasks while setting up the tunnels / config or is the whole setup unstable while in use?

regards,

Darky

Hi,

When I uncheck "only request a prefix" no ipv6 connectivity is established whatsoever. I also changed the prefix id to 1 on the lan interface if WAN would grab id 0. (also rebooted opnsense several times).

When I got home today ipv6 routing was working great, however half an hour ago or so it just stopped working, prefixes etc. stayed the same. I noticed that IPv6 routing died as several websited wouldn't load since the browser had used v6 for them apparently.

I checked some logs on the opnsense box and found this:

Code Select Expand


Apr 19 22:20:12 OPNsense apinger: No usable targets found, exiting
Apr 19 22:21:43 OPNsense apinger: Starting Alarm Pinger, apinger(40601)
Apr 19 22:21:43 OPNsense apinger: No usable targets found, exiting
Apr 19 22:21:50 OPNsense apinger: Starting Alarm Pinger, apinger(44357)
Apr 19 22:21:50 OPNsense apinger: No usable targets found, exiting
Apr 19 22:21:56 OPNsense apinger: Starting Alarm Pinger, apinger(33336)
Apr 19 22:21:56 OPNsense apinger: No usable targets found, exiting
Apr 19 22:21:56 OPNsense apinger: Starting Alarm Pinger, apinger(51798)
Apr 19 22:21:56 OPNsense apinger: No usable targets found, exiting
Apr 20 22:21:47 OPNsense apinger: Starting Alarm Pinger, apinger(65383)
Apr 20 22:21:47 OPNsense apinger: No usable targets found, exiting


so apinger was dead for like 24 hrs and when it started my v6 routing died - could this be related? Can I start this tool manually What actually does apinger do?

EDIT:
So I did more digging through the logs: ppps.log shows that my Internet-connection was reset after 24 hrs of online-time. This was (until recently) usual behaviour in Germany, however my plan should not have any timed disconnects. I'll have to open a ticket with my provider.

ppps.log, showing the 24hrs disconnect (look at the day):

Code Select Expand


Apr 19 22:21:38 OPNsense ppp: [wan]   93.104.101.XX -> 82.135.16.28
Apr 20 22:21:36 OPNsense ppp: [wan_link0] LCP: rec'd Terminate Request #133 (Opened)


However the pppoe connection is reestablished shorty after the disconnect and the same IPv4 address is issued.
dhcpd.log shows that a NEW v6 prefix was received.

The public v6 address on the LAN-Interface however won't update! It stayed the same and is still the old v6 address (as I'm typing). So there might be some kinks with the "track interface" once the pppoe connection get's reset.

DHCPD.log now looks like this, every few minutes a NEW /56 is received - however the LAN interface or the LAN clients won't get updated

Code Select Expand


Apr 20 23:24:56 OPNsense dhcp6c[61772]: Sending Solicit
Apr 20 23:24:56 OPNsense dhcp6c[61772]: set client ID (len 14)
Apr 20 23:24:56 OPNsense dhcp6c[61772]: set elapsed time (len 2)
Apr 20 23:24:56 OPNsense dhcp6c[61772]: set option request (len 4)
Apr 20 23:24:56 OPNsense dhcp6c[61772]: set IA_PD prefix
Apr 20 23:24:56 OPNsense dhcp6c[61772]: set IA_PD
Apr 20 23:24:56 OPNsense dhcp6c[61772]: send solicit to ff02::1:2%pppoe0
Apr 20 23:24:56 OPNsense dhcp6c[61772]: reset a timer on pppoe0, state=SOLICIT, timeo=38, retrans=126960
Apr 20 23:24:56 OPNsense dhcp6c[22061]: receive advertise from fe80::5a00:bbff:fe09:f8b0%pppoe0 on pppoe0
Apr 20 23:24:56 OPNsense dhcp6c[22061]: get DHCP option client ID, len 14
Apr 20 23:24:56 OPNsense dhcp6c[22061]:   DUID: 00:01:00:01:22:67:cd:3e:00:0c:29:8c:38:00
Apr 20 23:24:56 OPNsense dhcp6c[22061]: get DHCP option server ID, len 26
Apr 20 23:24:56 OPNsense dhcp6c[22061]:   DUID: 00:02:00:00:05:83:35:38:3a:30:30:3a:62:62:3a:30:39:3a:66:66:3a:63:63:00:00:00
Apr 20 23:24:56 OPNsense dhcp6c[22061]: get DHCP option IA_PD, len 41
Apr 20 23:24:56 OPNsense dhcp6c[22061]:   IA_PD: ID=0, T1=1800, T2=2880
Apr 20 23:24:56 OPNsense dhcp6c[22061]: get DHCP option IA_PD prefix, len 25
Apr 20 23:24:56 OPNsense dhcp6c[22061]:   IA_PD prefix: 2001:a61:47b:9200::/56 pltime=3600 vltime=8199546692136082464
Apr 20 23:24:56 OPNsense dhcp6c[22061]: get DHCP option DNS, len 32
Apr 20 23:24:56 OPNsense dhcp6c[22061]: XID mismatch
Apr 20 23:24:56 OPNsense dhcp6c[62386]: Sending Solicit
Apr 20 23:24:56 OPNsense dhcp6c[62386]: set client ID (len 14)
Apr 20 23:24:56 OPNsense dhcp6c[62386]: set elapsed time (len 2)
Apr 20 23:24:56 OPNsense dhcp6c[62386]: set option request (len 4)
Apr 20 23:24:56 OPNsense dhcp6c[62386]: set IA_PD prefix
Apr 20 23:24:56 OPNsense dhcp6c[62386]: set IA_PD
Apr 20 23:24:56 OPNsense dhcp6c[62386]: send solicit to ff02::1:2%pppoe0
Apr 20 23:24:56 OPNsense dhcp6c[62386]: reset a timer on pppoe0, state=SOLICIT, timeo=38, retrans=126960
Apr 20 23:24:56 OPNsense dhcp6c[22061]: receive advertise from fe80::5a00:bbff:fe09:f8b0%pppoe0 on pppoe0
Apr 20 23:24:56 OPNsense dhcp6c[22061]: get DHCP option client ID, len 14
Apr 20 23:24:56 OPNsense dhcp6c[22061]:   DUID: 00:01:00:01:22:67:cd:3e:00:0c:29:8c:38:00
Apr 20 23:24:56 OPNsense dhcp6c[22061]: get DHCP option server ID, len 26
Apr 20 23:24:56 OPNsense dhcp6c[22061]:   DUID: 00:02:00:00:05:83:35:38:3a:30:30:3a:62:62:3a:30:39:3a:66:66:3a:63:63:00:00:00
Apr 20 23:24:56 OPNsense dhcp6c[22061]: get DHCP option IA_PD, len 41
Apr 20 23:24:56 OPNsense dhcp6c[22061]:   IA_PD: ID=0, T1=1800, T2=2880
Apr 20 23:24:56 OPNsense dhcp6c[22061]: get DHCP option IA_PD prefix, len 25
Apr 20 23:24:56 OPNsense dhcp6c[22061]:   IA_PD prefix: 2001:a61:47b:9200::/56 pltime=3600 vltime=8199546692136082464
Apr 20 23:24:56 OPNsense dhcp6c[22061]: get DHCP option DNS, len 32
Apr 20 23:24:56 OPNsense dhcp6c[22061]: XID mismatch
Apr 20 23:25:38 OPNsense dhcp6c[40907]: Sending Solicit
Apr 20 23:25:38 OPNsense dhcp6c[40907]: set client ID (len 14)
Apr 20 23:25:38 OPNsense dhcp6c[40907]: set elapsed time (len 2)
Apr 20 23:25:38 OPNsense dhcp6c[40907]: set option request (len 4)
Apr 20 23:25:38 OPNsense dhcp6c[40907]: set IA_PD prefix
Apr 20 23:25:38 OPNsense dhcp6c[40907]: set IA_PD
Apr 20 23:25:38 OPNsense dhcp6c[40907]: send solicit to ff02::1:2%pppoe0
Apr 20 23:25:38 OPNsense dhcp6c[40907]: reset a timer on pppoe0, state=SOLICIT, timeo=518, retrans=127512
Apr 20 23:25:38 OPNsense dhcp6c[22061]: receive advertise from fe80::5a00:bbff:fe09:f8b0%pppoe0 on pppoe0
Apr 20 23:25:38 OPNsense dhcp6c[22061]: get DHCP option client ID, len 14
Apr 20 23:25:38 OPNsense dhcp6c[22061]:   DUID: 00:01:00:01:22:67:cd:3e:00:0c:29:8c:38:00
Apr 20 23:25:38 OPNsense dhcp6c[22061]: get DHCP option server ID, len 26
Apr 20 23:25:38 OPNsense dhcp6c[22061]:   DUID: 00:02:00:00:05:83:35:38:3a:30:30:3a:62:62:3a:30:39:3a:66:66:3a:63:63:00:00:00
Apr 20 23:25:38 OPNsense dhcp6c[22061]: get DHCP option IA_PD, len 41
Apr 20 23:25:38 OPNsense dhcp6c[22061]:   IA_PD: ID=0, T1=1800, T2=2880
Apr 20 23:25:38 OPNsense dhcp6c[22061]: get DHCP option IA_PD prefix, len 25
Apr 20 23:25:38 OPNsense dhcp6c[22061]:   IA_PD prefix: 2001:a61:47b:a400::/56 pltime=3600 vltime=8199546692136082464
Apr 20 23:25:38 OPNsense dhcp6c[22061]: get DHCP option DNS, len 32
Apr 20 23:25:38 OPNsense dhcp6c[22061]: XID mismatch
Apr 20 23:27:03 OPNsense dhcp6c[61772]: Sending Solicit
Apr 20 23:27:03 OPNsense dhcp6c[61772]: set client ID (len 14)
Apr 20 23:27:03 OPNsense dhcp6c[61772]: set elapsed time (len 2)
Apr 20 23:27:03 OPNsense dhcp6c[61772]: set option request (len 4)
Apr 20 23:27:03 OPNsense dhcp6c[61772]: set IA_PD prefix
Apr 20 23:27:03 OPNsense dhcp6c[61772]: set IA_PD
Apr 20 23:27:03 OPNsense dhcp6c[61772]: send solicit to ff02::1:2%pppoe0
Apr 20 23:27:03 OPNsense dhcp6c[61772]: reset a timer on pppoe0, state=SOLICIT, timeo=39, retrans=120360
Apr 20 23:27:03 OPNsense dhcp6c[22061]: receive advertise from fe80::5a00:bbff:fe09:f8b0%pppoe0 on pppoe0
Apr 20 23:27:03 OPNsense dhcp6c[22061]: get DHCP option client ID, len 14
Apr 20 23:27:03 OPNsense dhcp6c[22061]:   DUID: 00:01:00:01:22:67:cd:3e:00:0c:29:8c:38:00
Apr 20 23:27:03 OPNsense dhcp6c[22061]: get DHCP option server ID, len 26
Apr 20 23:27:03 OPNsense dhcp6c[22061]:   DUID: 00:02:00:00:05:83:35:38:3a:30:30:3a:62:62:3a:30:39:3a:66:66:3a:63:63:00:00:00
Apr 20 23:27:03 OPNsense dhcp6c[22061]: get DHCP option IA_PD, len 41
Apr 20 23:27:03 OPNsense dhcp6c[22061]:   IA_PD: ID=0, T1=1800, T2=2880
Apr 20 23:27:03 OPNsense dhcp6c[22061]: get DHCP option IA_PD prefix, len 25
Apr 20 23:27:03 OPNsense dhcp6c[22061]:   IA_PD prefix: 2001:a61:47b:a400::/56 pltime=3600 vltime=8199546692136082464
Apr 20 23:27:03 OPNsense dhcp6c[22061]: get DHCP option DNS, len 32
Apr 20 23:27:03 OPNsense dhcp6c[22061]: XID mismatch
Apr 20 23:27:03 OPNsense dhcp6c[62386]: Sending Solicit
Apr 20 23:27:03 OPNsense dhcp6c[62386]: set client ID (len 14)
Apr 20 23:27:03 OPNsense dhcp6c[62386]: set elapsed time (len 2)
Apr 20 23:27:03 OPNsense dhcp6c[62386]: set option request (len 4)
Apr 20 23:27:03 OPNsense dhcp6c[62386]: set IA_PD prefix
Apr 20 23:27:03 OPNsense dhcp6c[62386]: set IA_PD
Apr 20 23:27:03 OPNsense dhcp6c[62386]: send solicit to ff02::1:2%pppoe0
Apr 20 23:27:03 OPNsense dhcp6c[62386]: reset a timer on pppoe0, state=SOLICIT, timeo=39, retrans=120360
Apr 20 23:27:03 OPNsense dhcp6c[22061]: receive advertise from fe80::5a00:bbff:fe09:f8b0%pppoe0 on pppoe0
Apr 20 23:27:03 OPNsense dhcp6c[22061]: get DHCP option client ID, len 14
Apr 20 23:27:03 OPNsense dhcp6c[22061]:   DUID: 00:01:00:01:22:67:cd:3e:00:0c:29:8c:38:00
Apr 20 23:27:03 OPNsense dhcp6c[22061]: get DHCP option server ID, len 26
Apr 20 23:27:03 OPNsense dhcp6c[22061]:   DUID: 00:02:00:00:05:83:35:38:3a:30:30:3a:62:62:3a:30:39:3a:66:66:3a:63:63:00:00:00
Apr 20 23:27:03 OPNsense dhcp6c[22061]: get DHCP option IA_PD, len 41
Apr 20 23:27:03 OPNsense dhcp6c[22061]:   IA_PD: ID=0, T1=1800, T2=2880
Apr 20 23:27:03 OPNsense dhcp6c[22061]: get DHCP option IA_PD prefix, len 25
Apr 20 23:27:03 OPNsense dhcp6c[22061]:   IA_PD prefix: 2001:a61:47b:a400::/56 pltime=3600 vltime=8199546692136082464
Apr 20 23:27:03 OPNsense dhcp6c[22061]: get DHCP option DNS, len 32
Apr 20 23:27:03 OPNsense dhcp6c[22061]: XID mismatch
Apr 20 23:27:45 OPNsense dhcp6c[40907]: Sending Solicit
Apr 20 23:27:45 OPNsense dhcp6c[40907]: set client ID (len 14)
Apr 20 23:27:45 OPNsense dhcp6c[40907]: set elapsed time (len 2)
Apr 20 23:27:45 OPNsense dhcp6c[40907]: set option request (len 4)
Apr 20 23:27:45 OPNsense dhcp6c[40907]: set IA_PD prefix
Apr 20 23:27:45 OPNsense dhcp6c[40907]: set IA_PD
Apr 20 23:27:45 OPNsense dhcp6c[40907]: send solicit to ff02::1:2%pppoe0
Apr 20 23:27:45 OPNsense dhcp6c[40907]: reset a timer on pppoe0, state=SOLICIT, timeo=519, retrans=125580
Apr 20 23:27:45 OPNsense dhcp6c[22061]: receive advertise from fe80::5a00:bbff:fe09:f8b0%pppoe0 on pppoe0
Apr 20 23:27:45 OPNsense dhcp6c[22061]: get DHCP option client ID, len 14
Apr 20 23:27:45 OPNsense dhcp6c[22061]:   DUID: 00:01:00:01:22:67:cd:3e:00:0c:29:8c:38:00
Apr 20 23:27:45 OPNsense dhcp6c[22061]: get DHCP option server ID, len 26
Apr 20 23:27:45 OPNsense dhcp6c[22061]:   DUID: 00:02:00:00:05:83:35:38:3a:30:30:3a:62:62:3a:30:39:3a:66:66:3a:63:63:00:00:00
Apr 20 23:27:45 OPNsense dhcp6c[22061]: get DHCP option IA_PD, len 41
Apr 20 23:27:45 OPNsense dhcp6c[22061]:   IA_PD: ID=0, T1=1800, T2=2880
Apr 20 23:27:45 OPNsense dhcp6c[22061]: get DHCP option IA_PD prefix, len 25
Apr 20 23:27:45 OPNsense dhcp6c[22061]:   IA_PD prefix: 2001:a61:47b:b600::/56 pltime=3600 vltime=8199546692136082464
Apr 20 23:27:45 OPNsense dhcp6c[22061]: get DHCP option DNS, len 32
Apr 20 23:27:45 OPNsense dhcp6c[22061]: XID mismatch
Apr 20 23:29:03 OPNsense dhcp6c[61772]: Sending Solicit
Apr 20 23:29:03 OPNsense dhcp6c[61772]: set client ID (len 14)
Apr 20 23:29:03 OPNsense dhcp6c[61772]: set elapsed time (len 2)
Apr 20 23:29:03 OPNsense dhcp6c[61772]: set option request (len 4)
Apr 20 23:29:03 OPNsense dhcp6c[61772]: set IA_PD prefix
Apr 20 23:29:03 OPNsense dhcp6c[61772]: set IA_PD
Apr 20 23:29:03 OPNsense dhcp6c[61772]: send solicit to ff02::1:2%pppoe0
Apr 20 23:29:03 OPNsense dhcp6c[61772]: reset a timer on pppoe0, state=SOLICIT, timeo=40, retrans=129864
Apr 20 23:29:03 OPNsense dhcp6c[22061]: receive advertise from fe80::5a00:bbff:fe09:f8b0%pppoe0 on pppoe0
Apr 20 23:29:03 OPNsense dhcp6c[22061]: get DHCP option client ID, len 14
Apr 20 23:29:03 OPNsense dhcp6c[22061]:   DUID: 00:01:00:01:22:67:cd:3e:00:0c:29:8c:38:00
Apr 20 23:29:03 OPNsense dhcp6c[22061]: get DHCP option server ID, len 26
Apr 20 23:29:03 OPNsense dhcp6c[22061]:   DUID: 00:02:00:00:05:83:35:38:3a:30:30:3a:62:62:3a:30:39:3a:66:66:3a:63:63:00:00:00
Apr 20 23:29:03 OPNsense dhcp6c[22061]: get DHCP option IA_PD, len 41
Apr 20 23:29:03 OPNsense dhcp6c[22061]:   IA_PD: ID=0, T1=1800, T2=2880
Apr 20 23:29:03 OPNsense dhcp6c[22061]: get DHCP option IA_PD prefix, len 25
Apr 20 23:29:03 OPNsense dhcp6c[22061]:   IA_PD prefix: 2001:a61:47b:b600::/56 pltime=3600 vltime=8199546692136082464
Apr 20 23:29:03 OPNsense dhcp6c[22061]: get DHCP option DNS, len 32
Apr 20 23:29:03 OPNsense dhcp6c[22061]: XID mismatch


Edit2:
just now - a few minutes ago - a new ipv6 prefix was propagated to my LAN and LAN-IF. Here is the system log from the relavant time frame. Interestingly "rc.newwanipv6" was just executed before connectivity was restored:

Code Select Expand


Apr 20 22:21:44 OPNsense opnsense: /usr/local/etc/rc.newwanipv6: IP renewal is starting on 'pppoe0'
Apr 20 22:21:44 OPNsense opnsense: /usr/local/etc/rc.newwanipv6: On (IP address: fe80::2c40:3052:ceaf:5446) (interface: MNET[wan]) (real interface: pppoe0).
Apr 20 22:21:46 OPNsense opnsense: /usr/local/etc/rc.newwanip: IP renewal is starting on 'pppoe0'
Apr 20 22:21:46 OPNsense opnsense: /usr/local/etc/rc.newwanip: On (IP address: 93.104.101.XX) (interface: MNET[wan]) (real interface: pppoe0).
Apr 20 22:21:46 OPNsense opnsense: /usr/local/etc/rc.newwanip: Accept router advertisements on interface pppoe0
Apr 20 22:21:47 OPNsense opnsense: /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan'
Apr 20 22:21:47 OPNsense opnsense: /usr/local/etc/rc.newwanip: ROUTING: no IPv4 default gateway set, trying wan on 'pppoe0' (82.135.16.28)
Apr 20 22:21:47 OPNsense opnsense: /usr/local/etc/rc.newwanip: ROUTING: no IPv6 default gateway set, trying wan on 'pppoe0' (fe80::5a00:bbff:fe09:f8b0)
Apr 20 22:21:47 OPNsense opnsense: /usr/local/etc/rc.newwanip: ROUTING: setting IPv4 default route to 82.135.16.28
Apr 20 22:21:47 OPNsense opnsense: /usr/local/etc/rc.newwanip: ROUTING: setting IPv6 default route to fe80::5a00:bbff:fe09:f8b0
Apr 20 22:21:48 OPNsense ipsec_starter[79816]: configuration 'con1' unrouted
Apr 20 22:21:48 OPNsense ipsec_starter[79816]:
Apr 20 22:21:48 OPNsense opnsense: /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface MNET.
Apr 20 22:21:48 OPNsense kernel: ovpnc1: link state changed to DOWN
Apr 20 22:21:49 OPNsense ipsec_starter[79816]: 'con1' routed
Apr 20 22:21:49 OPNsense ipsec_starter[79816]:
Apr 20 22:21:52 OPNsense opnsense: /usr/local/etc/rc.newwanip: Dynamic DNS (xxx): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Apr 20 22:21:53 OPNsense kernel: ovpnc1: link state changed to UP
Apr 20 22:21:53 OPNsense opnsense: /usr/local/etc/rc.newwanip: IP renewal is starting on 'ovpnc1'
Apr 20 22:21:53 OPNsense opnsense: /usr/local/etc/rc.newwanip: Interface '' is disabled or empty, nothing to do.
Apr 20 22:32:16 OPNsense opnsense: /index.php: Session timed out for user 'root' from:
Apr 20 22:32:19 OPNsense opnsense: /index.php: Web GUI authentication error for 'root' from
Apr 20 22:32:23 OPNsense opnsense: /index.php: Successful login for user 'root' from:
Apr 20 22:43:18 OPNsense opnsense: user 'root' could not authenticate.
Apr 20 22:43:18 OPNsense sshd[7223]: error: PAM: authentication error for root from
Apr 20 22:43:18 OPNsense sshd[7223]: error: PAM: authentication error for root from
Apr 20 22:43:21 OPNsense opnsense: user 'root' authenticated successfully
Apr 20 22:43:21 OPNsense sshd[7223]: Accepted keyboard-interactive/pam for root from port 49181 ssh2
Apr 20 22:54:35 OPNsense kernel: pid 9128 (apinger), uid 65534: exited on signal 11
Apr 20 23:52:00 OPNsense opnsense: /usr/local/etc/rc.newwanipv6: IP renewal is starting on 'pppoe0'
Apr 20 23:52:00 OPNsense opnsense: /usr/local/etc/rc.newwanipv6: On (IP address: fe80::2c40:3052:ceaf:5446) (interface: MNET[wan]) (real interface: pppoe0).


regards,
Fabian