Topics

This is not a bug or problem, this is basically just a FYI...

Careful, I made the mistake of thinking that this 24.1.4 release note was DHCP option 121 to send static routes to dhcp clients (I need this option before moving to kea) but it is not.

Code Select Expand

o kea-dhcp: add domain-search, time-servers and static-routes client options to subnet configuration


It's DHCP option 33, which is a single IP to a router IP. You cannot use this to route 192.168.30.0/24 to 192.168.31.1 for instance. You can only route 1 ip to a router, like 192.168.30.12 to 192.168.31.1 for instance.

This is basic and can still serve some purpose for some people but most of us use DHCP option 121 in ISC which is totally different than the less used DHCP option 33 which basically enforce /32 on your static route because you cannot define a subnet/CIDR on the IP/network you pass.

In other words, what we likely want, is the support for this KEA feature in OPNSense (DHCP option 121) which encompass and overrides when it is present (per RFC) option 33. Option 121 also enables you to do exactly the same as option 33 since you can specify /32 on a subnet if you want but the important part is that it allows you to specify subnets, not only individual IP addresses to be routed to a router.

https://gitlab.isc.org/isc-projects/kea/-/merge_requests/2135/diffs

Now, since this has been implemented like that. This will likely create a breaking change when option 121 is implemented unless the upgrade process converts what was specified as ip,router to ip/32,router so that we can keep the same field for the better DHCP option 121 without breaking the kea config for people that started using it already in the option 33 format. Unless someone decides to support both options in the UI and configs and then we would have 2 separate fields for static routes in the kea UI. One for single IPs (DHCP option 33) and another one for networks (DHCP option 121)

I'm very glad that a lot of work is being done on KEA and I'm almost to the point of being able to move away from ISC. Once Option 121 is implemented in OPNSense and that KEA also registers its DHCP leases hosts in Unbound, I'll be good to migrate.

Thanks everyone !

@franco FYI since we worked on this together last time including @xaxero which I suspect might be affected as well too since he was also using multi-wan and Starlink like me.

Seems a regression or similar of this old issue that was fixed until 24.1

https://forum.opnsense.org/index.php?topic=33831.msg163808#msg163808

After Starlink updates itself during the night, the gateway sometimes gets flagged as down and never comes back up by itself even though it is up and working

When Starlink goes down, it temporarily also assigns itself an ip of 192.168.100.1/24 range (and gives opnsense .100) and then when it comes back up it goes back on his normal IPs or 100.64.x.x

For some reasons, dpinger has a hard adjusting itself when that happens. Not sure if it is begin restarted properly on the gateway change or the temporarily network flap. It seems to remain stucked on the 192.168.100.1 gateway

You can also wee that the state of the dpinger process is kinda stuck to the 192.168.100.1 IP and that if I manually clear the state it will then change to a right gateway but that's not enough to bring the IP UP I need to reload dpinger.

Code Select Expand

root@opnsense:~ # pfctl -ss -vvv | grep "1\.1\.1\.1" -A 3
No ALTQ support in kernel
ALTQ related functions disabled
all icmp 100.79.101.92:924 -> 1.1.1.1:924       0:0
   age 14:00:37, expires in 00:00:09, 49623:0 pkts, 1439067:0 bytes, rule 102
   id: adb0c36500000002 creatorid: 5f0e2da3 gateway: 192.168.100.1
   origif: igb0

Code Select Expand

root@opnsense:~ # pfctl -k id -k adb0c36500000002
killed 1 states


Code Select Expand


root@opnsense:~ # pfctl -ss -vvv | grep "1\.1\.1\.1" -A 3
No ALTQ support in kernel
ALTQ related functions disabled
all icmp 100.79.101.92:924 -> 1.1.1.1:924       0:0
   age 00:00:06, expires in 00:00:09, 6:6 pkts, 174:174 bytes, rule 104
   id: 7518c56500000002 creatorid: 5f0e2da3 gateway: 100.64.0.1
   origif: igb0

igb0 is my Starlink interface, you can see before the state clear that packet are going out but not coming back

Code Select Expand

root@opnsense:~ # tcpdump -i igb0 icmp and host 1.1.1.1 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:45:36.973765 IP 100.79.101.92 > 1.1.1.1: ICMP echo request, id 924, seq 49523, length 9
15:45:37.974749 IP 100.79.101.92 > 1.1.1.1: ICMP echo request, id 924, seq 49524, length 9
15:45:38.994204 IP 100.79.101.92 > 1.1.1.1: ICMP echo request, id 924, seq 49525, length 9
15:45:40.033544 IP 100.79.101.92 > 1.1.1.1: ICMP echo request, id 924, seq 49526, length 9

After I clear the state, they are coming back to normal now.

Code Select Expand

root@opnsense:~ # tcpdump -i igb0 icmp and host 1.1.1.1 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:55:44.088583 IP 100.79.101.92 > 1.1.1.1: ICMP echo request, id 924, seq 50121, length 9
15:55:44.125365 IP 1.1.1.1 > 100.79.101.92: ICMP echo reply, id 924, seq 50121, length 9
15:55:45.092572 IP 100.79.101.92 > 1.1.1.1: ICMP echo request, id 924, seq 50122, length 9
15:55:45.146659 IP 1.1.1.1 > 100.79.101.92: ICMP echo reply, id 924, seq 50122, length 9
15:55:46.139495 IP 100.79.101.92 > 1.1.1.1: ICMP echo request, id 924, seq 50123, length 9
15:55:46.191996 IP 1.1.1.1 > 100.79.101.92: ICMP echo reply, id 924, seq 50123, length 9
15:55:47.157407 IP 100.79.101.92 > 1.1.1.1: ICMP echo request, id 924, seq 50124, length 9
15:55:47.210673 IP 1.1.1.1 > 100.79.101.92: ICMP echo reply, id 924, seq 50124, length 9

Logs of gateway/dpinger

Code Select Expand

root@opnsense:~ # tail -100 /var/log/gateways/latest.log | grep -v DHCP6
<12>1 2024-02-04T01:45:05-05:00 opnsense dpinger 57453 - [meta sequenceId="1"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:06-05:00 opnsense dpinger 57453 - [meta sequenceId="3"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:07-05:00 opnsense dpinger 57453 - [meta sequenceId="5"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:08-05:00 opnsense dpinger 57453 - [meta sequenceId="7"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:09-05:00 opnsense dpinger 57453 - [meta sequenceId="9"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:10-05:00 opnsense dpinger 57453 - [meta sequenceId="11"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:11-05:00 opnsense dpinger 57453 - [meta sequenceId="13"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:12-05:00 opnsense dpinger 57453 - [meta sequenceId="15"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:13-05:00 opnsense dpinger 57453 - [meta sequenceId="17"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:14-05:00 opnsense dpinger 57453 - [meta sequenceId="19"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:15-05:00 opnsense dpinger 57453 - [meta sequenceId="21"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:16-05:00 opnsense dpinger 57453 - [meta sequenceId="24"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:17-05:00 opnsense dpinger 57453 - [meta sequenceId="26"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:18-05:00 opnsense dpinger 57453 - [meta sequenceId="28"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:19-05:00 opnsense dpinger 57453 - [meta sequenceId="30"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:20-05:00 opnsense dpinger 57453 - [meta sequenceId="32"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:21-05:00 opnsense dpinger 57453 - [meta sequenceId="35"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:22-05:00 opnsense dpinger 57453 - [meta sequenceId="37"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:23-05:00 opnsense dpinger 57453 - [meta sequenceId="39"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:24-05:00 opnsense dpinger 57453 - [meta sequenceId="41"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:25-05:00 opnsense dpinger 57453 - [meta sequenceId="43"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:26-05:00 opnsense dpinger 57453 - [meta sequenceId="45"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:27-05:00 opnsense dpinger 57453 - [meta sequenceId="47"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:28-05:00 opnsense dpinger 57453 - [meta sequenceId="48"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:29-05:00 opnsense dpinger 57453 - [meta sequenceId="49"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:30-05:00 opnsense dpinger 57453 - [meta sequenceId="50"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:31-05:00 opnsense dpinger 57453 - [meta sequenceId="51"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<165>1 2024-02-04T01:45:31-05:00 opnsense dpinger 53072 - [meta sequenceId="52"] ALERT: STARLINK_DHCP (Addr: 1.1.1.1 Alarm: none -> loss RTT: 43.7 ms RTTd: 9.1 ms Loss: 42.0 %)
<12>1 2024-02-04T01:45:32-05:00 opnsense dpinger 57453 - [meta sequenceId="53"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:33-05:00 opnsense dpinger 57453 - [meta sequenceId="54"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:34-05:00 opnsense dpinger 57453 - [meta sequenceId="55"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:35-05:00 opnsense dpinger 57453 - [meta sequenceId="56"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:36-05:00 opnsense dpinger 57453 - [meta sequenceId="57"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:37-05:00 opnsense dpinger 57453 - [meta sequenceId="58"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:38-05:00 opnsense dpinger 57453 - [meta sequenceId="59"] STARLINK_DHCP 1.1.1.1: sendto error: 22
<12>1 2024-02-04T01:45:38-05:00 opnsense dpinger 57453 - [meta sequenceId="60"] exiting on signal 15
<12>1 2024-02-04T01:45:38-05:00 opnsense dpinger 1844 - [meta sequenceId="61"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 1.1.1.1  bind_addr 192.168.100.100  identifier "STARLINK_DHCP "
<12>1 2024-02-04T01:45:38-05:00 opnsense dpinger 59623 - [meta sequenceId="62"] exiting on signal 15
<165>1 2024-02-04T01:45:38-05:00 opnsense dpinger 53072 - [meta sequenceId="63"] Reloaded gateway watcher configuration on SIGHUP
<12>1 2024-02-04T01:45:38-05:00 opnsense dpinger 1844 - [meta sequenceId="64"] exiting on signal 15
<165>1 2024-02-04T01:45:38-05:00 opnsense dpinger 53072 - [meta sequenceId="65"] Reloaded gateway watcher configuration on SIGHUP
<12>1 2024-02-04T01:45:38-05:00 opnsense dpinger 11941 - [meta sequenceId="66"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 1.1.1.1  bind_addr 192.168.100.100  identifier "STARLINK_DHCP "
<165>1 2024-02-04T01:45:39-05:00 opnsense dpinger 53072 - [meta sequenceId="67"] Reloaded gateway watcher configuration on SIGHUP
<165>1 2024-02-04T01:45:43-05:00 opnsense dpinger 53072 - [meta sequenceId="68"] ALERT: STARLINK_DHCP (Addr: 1.1.1.1 Alarm: loss -> down RTT: 0.0 ms RTTd: 0.0 ms Loss: 100.0 %)
<12>1 2024-02-04T01:46:41-05:00 opnsense dpinger 11941 - [meta sequenceId="69"] exiting on signal 15
<12>1 2024-02-04T01:46:41-05:00 opnsense dpinger 66460 - [meta sequenceId="70"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 1.1.1.1  bind_addr 100.79.101.92  identifier "STARLINK_DHCP "
<165>1 2024-02-04T01:46:41-05:00 opnsense dpinger 53072 - [meta sequenceId="71"] Reloaded gateway watcher configuration on SIGHUP
<165>1 2024-02-04T01:46:44-05:00 opnsense dpinger 53072 - [meta sequenceId="73"] Reloaded gateway watcher configuration on SIGHUP
<12>1 2024-02-04T15:38:41-05:00 opnsense dpinger 36972 - [meta sequenceId="1"] send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr 1.1.1.1  bind_addr 100.79.101.92  identifier "STARLINK_DHCP "
<165>1 2024-02-04T15:38:43-05:00 opnsense dpinger 53072 - [meta sequenceId="2"] ALERT: STARLINK_DHCP (Addr: 1.1.1.1 Alarm: down -> none RTT: 52.6 ms RTTd: 8.4 ms Loss: 0.0 %)
<12>1 2024-02-04T15:38:44-05:00 opnsense dpinger 36972 - [meta sequenceId="3"] exiting on signal 2

The process before I reload it

Code Select Expand

root    66460   0.0  0.0  13340  2508  -  Is   01:46       0:02.54 /usr/local/bin/dpinger -f -S -r 0 -i STARLINK_DHCP -B 100.79.101.92 -p /var/run/dpinger_STARLINK_DHCP.pid -u /var/run/dpinger_STARLINK_DHCP.sock -s 1s -l 4s -t 60s -d 1 1.1.1.1

And after I reload it and it marks the interface as up now

Code Select Expand

root@opnsense:~ # ps aux | grep LINK_DHCP\
root    91462   0.0  0.0  13340  2512  -  Is   16:07       0:00.03 /usr/local/bin/dpinger -f -S -r 0 -i STARLINK_DHCP -B 100.79.101.92 -p /var/run/dpinger_STARLINK_DHCP.pid -u /var/run/dpinger_STARLINK_DHCP.sock -s 1s -l 4s -t 60s -d 1 1.1.1.1

They are the same...

And state after dpinger reload is still normal like the one after I manually forced state kill to reset it

Code Select Expand

root@opnsense:~ # pfctl -ss -vvv | grep "1\.1\.1\.1" -A 3
No ALTQ support in kernel
ALTQ related functions disabled
all icmp 100.79.101.92:25926 -> 1.1.1.1:25926       0:0
   age 00:01:53, expires in 00:00:10, 112:112 pkts, 3248:3248 bytes, rule 104
   id: ba21c56500000002 creatorid: 5f0e2da3 gateway: 100.64.0.1
   origif: igb0

Seems like the state isn't cleared properly and/or dpginger isn't resetting properly after interface flag or gateway change.

I forgot to take the output during the outage of

Code Select Expand

pluginctl -r host_routes

But here it is after everything is good. Next outage I'll take it before fixing it.

Code Select Expand

root@opnsense:~ # pluginctl -r host_routes
{
    "core": {
        "8.8.8.8": null,
        "8.8.4.4": null
    },
    "dpinger": {
        "8.8.4.4": "10.50.45.70",
        "1.1.1.1": "100.64.0.1",
        "2001:4860:4860::8844": "fe80::200:xxxx:xxxx:xxx%igb0",
        "149.112.112.112": "192.168.2.1",
        "192.168.170.2": "192.168.170.2",
        "192.168.171.2": "192.168.171.2",
        "2620:fe::9": "2001:470:xx:x::x
    }
}

Code Select Expand

/usr/local/etc/rc.d/nut: WARNING: failed precmd routine for nut


I'm trying to figure out what is happening. I removed/reinstalled the plugin without luck so far. I'll keep you posted.

I upgraded from 21.1.2 to 22.1.3 last night.

Since then, on reboot and every time I save/apply my IPSec configs I get those errors

Code Select Expand

[18-Mar-2022 00:29:44 America/Toronto] PHP Warning:  openssl_x509_read(): supplied parameter cannot be coerced into an X509 certificate! in /usr/local/etc/inc/plugins.inc.d/ipsec.inc on line 1257

[39675810-00da-4d28-b42b-1befc06e7261] Script action stderr returned "b"no files found matching '/usr/local/etc/strongswan.opnsense.d/*.conf'""

/usr/local/sbin/pluginctl: Error: Invalid certificate hash info for


Anybody has an idea of what's up ? I think it may be related to this release note but I'm not sure how to fix it...

o ipsec: clean up stale CA certificates on reconfigure

Also, I checked under: /usr/local/etc/strongswan.opnsense.d/*.conf and there is nothing (Except a README file) in there but this configuration line

Code Select Expand

include strongswan.opnsense.d/*.conf


is included in the /usr/local/etc/strongswan.conf anyways, this could explain one of the errors above (maybe all 3 actually).

Checking the code of

Code Select Expand

        $strongswan = generate_strongswan_conf($strongswanTree);
        $strongswan .= "\ninclude strongswan.opnsense.d/*.conf\n";
        @file_put_contents("/usr/local/etc/strongswan.conf", $strongswan);

        /* generate CA certificates files */
        $cafiles = [];
        foreach (isset($config['ca']) ? $config['ca'] : [] as $ca) {
            $cert = base64_decode($ca['crt']);
            $x509cert = openssl_x509_parse(openssl_x509_read($cert));
            if (is_array($x509cert) && isset($x509cert['hash'])) {
                $fname = "{$capath}/{$x509cert['hash']}.0.crt";
                $cafiles[] = $fname;
                if (!@file_put_contents($fname, $cert)) {
                    log_error(sprintf('Error: Cannot write IPsec CA file for %s', $ca['descr']));
                }
            } else {
                log_error(sprintf('Error: Invalid certificate hash info for %s', $ca['descr']));
            }
        }
        foreach (glob("{$capath}/*.0.crt") as $fname) {
            if (!in_array($fname, $cafiles)) {
                unlink($fname);
            }
        }


I see it seems to do something against the ipsec certs... I see 2 certs generated there: /usr/local/etc/ipsec.d/cacerts/ so I'm not sure what going on...

Hello,

I don't think this was the behaviour before 22.1 but I may be mistaken. Here's my issue.

I have "Allow DNS server list to be overridden by DHCP/PPP on WAN" unchecked (off) in Settings/General.

I have multiple interfaces in DHCP and I use multi-wan

- ISP1 IPV4 PPPoE (Original WAN)
- ISP1 IPTV DHCP

- ISP2 IPv4 DHCP
- ISP2 IPv6 DHCPv6

For some reasons, the IPv4 DNS servers that are learned from the DHCP interfaces are being added as static routes. Even if I put a static route for the DNS server to another interface, it will get overwritten after some time automatically.

When i check Interfaces/Overview, I see that the all the interfaces, except the ISP1 IPV4 (Original WAN), have learned DNS servers in them. They should all be empty since I've unchecked the "Allow DNS server list to be overridden by DHCP/PPP on WAN" setting. Or if they are learned they should not be acted upon at least.

Those learned DNS servers are not being pushed to clients so that part is actually working but opnsense itself seems to be creating static routes to those dns servers IPs (Only on IPv4 DNS servers from what I'm seeing) to the interface it learned them from.

Could it be that only the first, original, WAN interface DNS servers are being ignored ? If so, how can I prevent those automatically added static routes or how can I prevent the other interfaces from learning DNS servers through DHCP ?

Bottom line for me, my ISP2 is much slower (40ms vs 1ms) than my ISP1 and since opnsense seems to add static routes to the interface where it learned the DNS servers automatically, all DNS queries are going to ISP2 instead of ISP1 like I would prefer.

I tried setting the DNS servers in General to the ISP1 gateway, it works for some time and then ISP2 DHCP (I think it's the trigger) renewal happens and the DNS servers routes change back to ISP2. I tried adding a static route to ISP1 for the DNS IPs and again, works for some time then automatically gets back to ISP2.

Any suggestion or idea is welcome as it seems I have no control over the behaviour and cannot force it.