Messages

There are several aspects to this. here are a few that come to mind:

(1) The two test sites that you mention (waveform and cloudflare) do not use icmp/icmpv6 "echo requests" (i.e. classic ping) but measure the time difference between a request for data and the receipt of the data using tcp/udp. Cloudflare explicitely says so. To convince yourself, you could also do a wireshark capture - there will be no icmp/icmpv6 echo requests. I believe that it is also true that internet routers in general treat icmp/icmpv6 echo requests differently (in terms of priority and treatment) to other protocols.  So the high "ping" latency you see is not really relevant.

(2) Did you tune the pipe bandwidths of your configuration? At somepoint in the ISP infrastructure you will be sharing bandwidth with other users so it is important to test at different times of day. For example, I have a 2.5Gbps/1Gbps fibre connection but in order to get A+ at all times of day, I need to set my bandwidths to 1.5Gbps/600Mbps - at 60% this is way below the article's suggested 85% starting point. If I try to increase the bandwidths, my latency invariably rises quickly in the evening.

I am not using version 25.1 (I use 24-19-2) but I believe that the OPNsense NUT plugin can only take one of two roles: standalone or netclient. On my version I see this as the help text of 'Service Mode' on Services->Nut->Configuration->General Settings.

I believe that this means that nut on OPNsense can only be the Secondary and not the Primary (using the definitions from RFC 9271). Though of course it can be used in Standalone mode.

I don't believe that you can do what you ask.

Hi admins, sorry for the delay in replying.

By 'Create an interface for the raw underlying interface to WAN (I call it underWAN)' I mean do something like the following the following on Interfaces->Assignements->Add (Assign a new interface)

Device=igc3
Description=underWAN

[ensure that the Protectli pppoe optimisations are configured (and rebooted)]

I had a similar problem with my Protectli VP4670 on OPNsense 23.7.8-amd64. These are my notes on how I fixed my problem:

• ensure that powerd is active
• ensure that the Protectli pppoe optimisations are configured (and rebooted)
• create an interface for the raw underlying interface to WAN (I call it underWAN)
• assign your own MAC to underWAN (and NOT to WAN)
• remove promiscuous mode from WAN (the pppoe device). if you dont then flapping always occurs.
• set promiscuous mode on underWAN (the igc device)

I needed to use promiscuous mode.  I have highlighted what I now think is the key part.

I hope that helps.

Surely if the MTU of the WAN is 1492 and the LAN is 1500 then OPNSense should generate a ICMP type=3 code=4 response and send it to the LAN device.

It does.

Thanks to you both for your comments and to Patrick for confirming that OPNsense 25.1.2 does indeed respect the requirements of RFC 1191.

In the meantime I have carried out further tests after having:
* disabled Suricata
* disabled shaping
* inserted explicit fw rules to accept (and log) icmp destination unreachable messages (Should be pointless because such messages are associated with the original ping and so should automatically be accepted without logging)

In each case I did the following from my linux machine:

Code Select Expand

ping -c 1 -M do -D -s 1464 151.101.0.81 # Returns a echo response and so verify that the target host does honour pings
ping -c 1 -M do -D -s 1465 151.101.0.81 # Should receive icmp type 3 code 4 response from OPNsense, but doesnt.
In no case did the LAN and WAN interface captures show any icmp type 3 code 4 responses. The evidence seems to point to OPNsense 24.10 silently dropping the "oversize" DF marked icmp after being passed by the firewall on WAN but before being sent down the wire.

I have looked through the change logs of OPNsense releases since 24.10 and I see that 25.1.1 includes the following change:
  "src: pf: send ICMP destination unreachable fragmentation needed when appropriate"

So it looks as if the non-compliance with RFC 1911 is fixed in 25.1.1 - which would explain why neither of you have the problem.

Thanks again for your help.

Thanks for the fast response.

I had read those articles but my question regards what seems to me to be a bug in FreeBSD/OPNsense (or more likely a user error of course).

Surely if the MTU of the WAN is 1492 and the LAN is 1500 then OPNSense should generate a ICMP type=3 code=4 response and send it to the LAN device.

My evidence shows that this is not happening.

Or is my understanding incorrect?

My system:
*fully updated OPNsense 24.10.2-amd64 running on a Protecli VP4670.
*WAN configuration: pppoe->vlan->igc1 (connects to an ONT via ethernet)
*LAN configuration: vlan->igc0
*I do not manually set MTU parameters on any interface.
*I do use shaping on WAN and LAN

OPNsense correctly calculates the WAN MTU as 1492 (=1500-8 = 1464+28). Indeed, executing on the OPNsense box:
    (FP1) ping -D -s 1464 9.9.9.9 # receives a ping response
    (FP2) ping -D -s 1465 9.9.9.9 # is rejected by the interface with a "ping: sendto: Message too long" message. No message is actually sent down the wire.
   
On a linux machine connected to the LAN interface, executing:
    (LP1) ping -M do -D -s 1464 9.9.9.9 # receives a ping response
    (LP2) ping -M do -D -s 1465 9.9.9.9 # receives no response

From within OPNsense a packet capture of both the LAN interface and the WAN interface shows that the
echo request from ping LP2 is seen in the LAN interface capture but is NOT present in the WAN interface capture.

But the Firewall log for the WAN interface shows that the LP2 ping message is "passed" to the WAN interface for transmission.

I deduce that OPNsense (or FreeBSD) is discarding the LP2 ping without generating the ICMP type=3 code=4 response that should be sent to the device on the LAN.

Is there some setting (tunable?) that I can use to persuade OPNsense to send the ICMP type=3 code=4 responses to my LAN devices?

The three rules with syntax errors all reference the pf "alias" LOG_UDP. None of the other rules you quote contain that "alias".

I guess if $LOG_UDP were to be empty then that might lead to a syntax error?

The contents of LOG_UDP should be listed at the beginning of  /tmp/rules.debug.

Several years ago I had problems with aliases and I needed to increase the size of the OPNsense setting

    Firewall: Settings: Advanced->  Firewall Maximum Table Entries

So it looks as if it cannot be done.

I already successfully use the API for manipulating FreeRADIUS users and reading DHCPv4 leases.

Using the OPNSense API Reference, I have been unable to discover how to list the contents of any specific Firewall Alias. I have tried many commands for both the alias and alias-util controllers.

Can this be done?

If so, could somebody provide a one line hint using curl?

I am monitoring modifications to /var/dhcpd/etc/dhcpd.conf which is owned by the user dhcpd (uid=136).

When that file changes I would like to run a script as root.  However, it seems that the script is running as dhcpd which is not what I want and does not work

How can I configure monit to run this script as root?

I followed this guide, the only one I found of a user in the same ISP as mine, PFSense instead
https://forum.meo.pt/internet-fixa-e-movel-11/pfsense-ipv6-155245

Looking at a few other posts on that forum, it would seem that your provider could and does (or rather did)  change the delegated ipv6 prefix. It is not clear to me with what frequency. Though it seems that even small changes to the ISP router will cause the prefix to change.

Made a forced ISP Modem restart, IPv6 is back, without touching the OPNsense Router.

Does that mean that your system consists of:
   <ISP ONT> -> <ISP router> -> <OPNsense router>
If so how have you configured the ISP router for ipv6?

@Alec246: thanks

Why do you set "Request only an IPv6 prefix Checked "?  Is that required by your ISP?  Do you know what mechanism is used to get the ipv6 address of the gateway?

It would be good to know if your delegated ipv6 prefix is static or not. Maybe, try comparing it over a few days with a few WAN restarts in between?

When your ipv6 connectivity is functioning, could you share the first 8 hex digits of your WAN ipv6 address (no more than 8 since otherwise it might identify you). You can find it on the OPNsense Da.shboard

For the record here are my config details which are somewhat similar to yours, though I need to use pppoe and obtain my ipv6 prefix/address through dhcpv6 over the pppoe. I too have LANs based on vlan's over a lagg. Here are some details:

WANside interfaces:  WAN->pppoe->vlan->(interface on igc)

• interface on igc uses MAC spoofing and has Promiscuos mode set. It has no ipv4 or ipv6 configuration.
• vlan is required by my ISP
• pppoe is required by my ISP
• WAN:
  promiscuos mode unset and has DHCPv6 client configuration as follows:
  Request only an ipv6 prefix: No (otherwise the WAN has no ipv6 address of its own)
  Prefix delegation size: 48
  Send ipv6 prefix hint: yes
  Use ipv4 connectivity (required by my ISP - i.e. ipv6 trafic travels down the same pppoe connection as the ipv4.
  

LANside interfaces:  LAN->vlan->underLAN->lagg (lacp) ->(a pair of igc interfaces)

• underLAN has Promiscuos mode set and sets a MAC address. It has no ipv4 or ipv6 configuration.
• LAN has promiscuos mode unset. Static ipv4 and tracks the WAN interface for ipv6.
  

My ipv6 prefix is static and so will only change when I change ISP. The WAN ipv6 address is also static.

A few questions for you:

(1) Does your ISP give you a static or dynamic ipv6 prefix? By dynamic I mean one that changes every now and again

(2) When you say that "ipv6 goes out". What do you mean? For example, when "ipv6 goes out":

• does your WAN still have an ipv6 address (look in the Dashboard under interfaces panel)?
• does your LAN still have an ipv6 address? (as above)
• does your PC still have an ipv6 address?

(3) How do you get your ipv6 prefix from your ISP?  That is, how have you configured the WAN interface for ipv6

(4) How do you assign ipv6 prefixes to your devices on the LAN:

• SLAAC only (ie through Services: Router Advertisements: [LAN])? What type of " Router Advertisements" do you use?
• Do you use DHCPvc6?

(5) Have you configured the LAN interface to use "Track Interface" as the ipv6 configuration type?

Edit: One other question:
(6) I assume that ipv6 connectivity comes back when you reboot your OPNsense device. Correct?

By changing "invert LAN" to "any" you move from only blocking connections to outside your LAN to blocking everthing including to your DNS,  NTP etc. services on OPNsense. Not having DNS is certainly going to prevent access to the internet (unless the destinations are hardwired in the IoT devices)

BTW Do your IoT devices use a hub/gateway lying within your LAN? If so, is that MAC contained in the IoT Alias? Does communication to the internet always go through that?

Regarding the need to clear firewall states after a reboot. Maybe this relates to the nature of OPNsense's MAC Aliases: OPNsense obtains the ipv4/ipv6 addresses by periodically (?) checking the arp and ndp tables. Possibly,  your IoT devices are establishing a connection to the internet before the MAC Alias is populated.  In this case the firewall No 1 rule will never fire. Resetting firewall states is not going to clear out the MAC alias, this would explain why a reset after a reboot fixes the problem..

Have you thought about putting the IoT devices on a separate vlan? Would make it a lot easier.