Messages

Hi, I have found a dumb solution for this problem.
Just put a basic switch between the starlink router and your opnsense, and you won't be annoyed anymore.

I think the starlink router has an ARP management problem or something related that just don't work with dpinger.

I tired this but the issue is still occurring.  Any other idea's?

Did you block bogon or private network on starlink interface configuration? Try to disable both.

Bump this post.

I am having the same issue.  OPNsense switches over from StarLink to Cox even though I never lose StarLink.  The logs I am seeing are the same as the OP.  At this point I have had to remove the GW Monitor for COX from the gateway group for my devices to stay on StarLink.

Just started happening this morning.  Noticed this when I was on 24.1.10_8.  Upgraded to 24.7_9 and still seeing the issue.

Has anyone found a solution for this.

Hi, I have found a dumb solution for this problem.
Just put a basic switch between the starlink router and your opnsense, and you won't be annoyed anymore.

I think the starlink router has an ARP management problem or something related that just don't work with dpinger.

A couple of things:

1. What thresholds do you have set to fail over with? i.e. Packetloss high/low or Latency high/low?
2. The graphs you showed looked like RRDtool graph? Is that smokeping you're using? The default for it is 20 pings every 300 seconds, so it not showing wouldn't be a surprise.  if it's not that, how often is the pinger used for those graphs going off?

1- Failover is using link down.
2. It's 5 pings every 60 secondes here.

The thing is that is not the stability of starlink.

I won't go at all the things I have at home, you can take a quick look here: https://wiki.abyssproject.net/en/infra/blog-infrastructure

The thing to remember, is that I monitor the starlink with other check than opnsense, both inside and outside the network, and I don't have any drops.

I also have a permanent wireguard tunnel that doesn't log any drop or any error.
+ all the checks from my internal LibreNMS that monitor my external services (+/- 70 checks, every 60 seconds) that include some pings and that don't fail or record any latencies neither.

Dpinger has a bug in OPNSense, or starlink changed something that make opnsense thinks the network is down, but it isn't.

I can confirm this issue on StarLink WAN: https://forum.opnsense.org/index.php?topic=40613.0

As temporary workaround I deactivated gateway monitoring for SL WAN

Thannks for the feedback.

Tier group works without gateway monitoring?
I will try this when I'm home.

Hello,
Since a couple of days I have a problem with Dpinger and the gateway moniroting.

I have OPNsense 24.1.7_4-amd64 and I have a gateway group with 2 tiers:
• My Starlink connection is in Tier 1
• My ADSL connection is in Tier 2 for backup purposes.

Since I've got back from the holidays, I've noticed that Opnsense disconnects my starlink every 10 to 20 minutes.

Code Select Expand

Date
Severity
Process
Line
2024-05-24T08:59:51 Warning dpinger send_interval 1000ms loss_interval 4000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 0ms loss_alarm 0% alarm_hold 10000ms dest_addr 100.64.0.1 bind_addr 100.109.13.234 identifier "WANGW "
2024-05-24T08:59:51 Warning dpinger exiting on signal 15
2024-05-24T08:59:51 Warning dpinger exiting on signal 15
2024-05-24T08:59:51 Warning dpinger exiting on signal 15
2024-05-24T08:50:04 Warning dpinger send_interval 1000ms loss_interval 4000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 0ms loss_alarm 0% alarm_hold 10000ms dest_addr 100.109.13.234 bind_addr 100.109.13.234 identifier "WANGW "
2024-05-24T08:50:04 Warning dpinger exiting on signal 15
2024-05-24T08:50:03 Warning dpinger WANGW 100.109.13.234: sendto error: 65
2024-05-24T08:50:02 Warning dpinger WANGW 100.109.13.234: sendto error: 65
2024-05-24T08:50:01 Warning dpinger WANGW 100.109.13.234: sendto error: 65
2024-05-24T08:35:47 Warning dpinger send_interval 1000ms loss_interval 4000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 0ms loss_alarm 0% alarm_hold 10000ms dest_addr 100.109.13.234 bind_addr 100.109.13.234 identifier "WANGW "
2024-05-24T08:35:47 Warning dpinger exiting on signal 15
2024-05-24T08:35:46 Warning dpinger WANGW 100.109.13.234: sendto error: 65
2024-05-24T08:35:45 Warning dpinger WANGW 100.109.13.234: sendto error: 65

The fact is that the starlink isn't disconnecting at all.

I also have monitoring of both my gateways on a LibreNMS instance locally, and it doesn't see these packet drops (Those showing are because a Starlink modem restart), see attachment.

And if I force all my internet to go through starlink, internal and external monitoring (I also self-host some things behind my starlink with some checks from OVH) doesn't see anything never ...

Same when I play online, I don't see any lag or get disconnected, but in the logs, the OPNSense says the gateway is lost.

Any idea of what may be going on?
Thank you

It's solved, it was unplugging it for the lunch and restart it, that's all...

But we have another bug leading to 100%cpu usage by Python when we check the logs of ipsec.


You should really get into a ticketing system for MSP and professionals, apart from the "anydesk" remote session like all the other vendors with full log analysis and everything.

The problem, even if we pay for support hours by default when buying your product, it that legally, I can't let you connect to our systems and customer's one without a NDA with each of your staff that may connect, which is a bit complicated you will agree.

I asked what else I could look 5th post ago and didn't get any feedback on this.
I didn't know there was another way beside "opnsense-revert" to actually revert to previous version, so I didn't write further indeed.

If I get through the professional service that we paid for this firewall (and others) will I get the same answers?

So, it won't change anything, I have already done this as written in my first post.

Where can I get the traffics logs and ipsec logs from command line? Web interface shows everything empty for ipsec (and as I said, I don't have any incoming traffic inside the tunnel on the watchguard so I can't see any error on it).

I will test this afternoon, as I don't have access to the firewall right now.
But this won't reverse kernel and all the packages?

The auto-update is suspected because it was working just before, you will agree that is has something to do with it.

Anyway, what could we check? To get rid of this?

Best regards,

The cron update the firewall every 1 hour, that why I said it was the release just before (this one is on our test labs to check if we are going to resell opnsense firewalls)

We already check the other side, I don't think it comes from here, as we have the same watchguard on another location, with same update and same tunnels settings (except for the networks and public ip of course) that works flawlessly, this where it gets strange.

Hello,

Last is: OPNsense 22.1.7_1-amd64
FreeBSD 13.0-STABLE
OpenSSL 1.1.1o 3 May 2022

The last working was the update before, we did the update yesterday (the firewall have a cron to auto-update, so I don't monitor the version number closely to be honest).

Hello,

We have updated the OPNSense at our office to the latest release yesterday (auto-update in fact) and one of our two IPSEC tunnel won't let data goes through.

Both IPSEC tunnel goes to two up-to-date Watchguard's firewall at two different locations.

We have the same configuration on both site-to-site tunnels:
IPv4 IKEv2
Phase 1:  main   AES (128 bits) + SHA256 + DH Group 20   Mutual PSK
Phase 2: ESP IPv4 tunnel   AES256 + SHA256+ DH Group 20

The first tunnel work as expected:

Code Select Expand

State
INSTALLED
Routed

Stats
Time : 459
Bytes in : 70596
Bytes out : 59200


But the second won't accept traffic anymore:

Code Select Expand

State
INSTALLED
Routed

Stats
Time : 459
Bytes in : 0
Bytes out : 19228

We restarted both firewall, deleted the IPSEC config completly then made it from scratch, rolled back to the previous version on the opnsense firewall but nothing works.
We don't see the traffic comming to the watchguard or being rejected at all, it seems that nothing is going outside the opnsense firewall.

Any idea?

Hello,

We have two OPNsense DEC3840 running the business edition
Here is the information on both of them:
OPNsense 21.4.3-amd64
FreeBSD 12.1-RELEASE-p19-HBSD
OpenSSL 1.1.1k 25 Mar 2021
AES-SNI enabled

We have an IPSec tunnel with the following settings:
PH1 : 128 bit AES-GCM with 128 bit ICV + SHA256 + DH Group 28
PH2 : aes128gcm16 + + 28 (Brainpool EC 256 bits)

I have tested different combination with and without hash and everything, and it doesn't seem to impact IPSEC performance.

We have 1Gbps professional connection on both side, and we only get 100mbps throughput on IPSEC (tested on smb copy, iperf).

Any idea of what is blocking IPSEC performance? The cpu usage doesn't move.


Thanks

Hello,

I have a small problem.

Here is the deal.
We have a firewall at our main office that is connected to our two datacenters via two ipsec tunnels that works fine.

We want to connect via OpenVPN to our main office (this part is actually working too) and we also want to have access to our two datacenter subnets via this vpn.

Is it possible? I tried via pushing route throught ovpn client config but I don't get it working.

Thanks and best regards,