Messages

I've now understood what happend and why.

OPNSense rejected the incoming responses on WAN, because there was no ARP entry for the VM's ip address so it really couldnt find the host. I checked because I noticed that the (ISC) DHCP lease table had a red plug symbol (offline) against it.

Why didnt the ARP table have and entry for the VM's ip address? Ah, well, in an attempt to dissuade the childrens' friends from connecting their PC's to the LAN (I prefer them to use wifi where they automatically get put onto their own vlan) I set to true within the ISC DHCP server the two flags:
* Deny unknown clients
* Enable Static ARP entries
It seems that something has changed within OPNSense because at one time, when I added a new ipv4, the static arp entry was created, now it isnt. So I unchecked the flags, restarted the DHCP server, checked the flags again, restarted the server and did a reboot for good measure. Hey presto, the arp entry was created and everything worked as should.

I have OPNsense 25.10.

This problem has me completely stumped. It only occurs when the the ipv4 request (eg TCP SYN, ICMP..) originates from any VM on my Fedora workstation. It does not happen when the request originates from my workstation or from a VM in a PRoxmox server.

I have taken captures from the workstation and the LAN and WAN interfaces on OPNsense.

This is what I see on the WAN interface:
* If I execute 'wget 142.251.209.46' from my workstation I see the traffic you would expect.
* If I execute 'wget 142.251.209.46' from the Debian VM I see an incoming SYN,ACK followed immediately by OPNSENSE sending out an icmp Host Unavailable.

I can see no discernible difference between the two ipv4 requests.

The issue is clearly being caused from within OPNSense, but where and why?

Why should OPNSense reject a protocol response when the firewall has already let the outgoing ipv4 message pass?

Anybody have some ideas on how I can diagnose this?

 

There are several aspects to this. here are a few that come to mind:

(1) The two test sites that you mention (waveform and cloudflare) do not use icmp/icmpv6 "echo requests" (i.e. classic ping) but measure the time difference between a request for data and the receipt of the data using tcp/udp. Cloudflare explicitely says so. To convince yourself, you could also do a wireshark capture - there will be no icmp/icmpv6 echo requests. I believe that it is also true that internet routers in general treat icmp/icmpv6 echo requests differently (in terms of priority and treatment) to other protocols.  So the high "ping" latency you see is not really relevant.

(2) Did you tune the pipe bandwidths of your configuration? At somepoint in the ISP infrastructure you will be sharing bandwidth with other users so it is important to test at different times of day. For example, I have a 2.5Gbps/1Gbps fibre connection but in order to get A+ at all times of day, I need to set my bandwidths to 1.5Gbps/600Mbps - at 60% this is way below the article's suggested 85% starting point. If I try to increase the bandwidths, my latency invariably rises quickly in the evening.

I am not using version 25.1 (I use 24-19-2) but I believe that the OPNsense NUT plugin can only take one of two roles: standalone or netclient. On my version I see this as the help text of 'Service Mode' on Services->Nut->Configuration->General Settings.

I believe that this means that nut on OPNsense can only be the Secondary and not the Primary (using the definitions from RFC 9271). Though of course it can be used in Standalone mode.

I don't believe that you can do what you ask.

Hi admins, sorry for the delay in replying.

By 'Create an interface for the raw underlying interface to WAN (I call it underWAN)' I mean do something like the following the following on Interfaces->Assignements->Add (Assign a new interface)

Device=igc3
Description=underWAN

[ensure that the Protectli pppoe optimisations are configured (and rebooted)]

I had a similar problem with my Protectli VP4670 on OPNsense 23.7.8-amd64. These are my notes on how I fixed my problem:

• ensure that powerd is active
• ensure that the Protectli pppoe optimisations are configured (and rebooted)
• create an interface for the raw underlying interface to WAN (I call it underWAN)
• assign your own MAC to underWAN (and NOT to WAN)
• remove promiscuous mode from WAN (the pppoe device). if you dont then flapping always occurs.
• set promiscuous mode on underWAN (the igc device)

I needed to use promiscuous mode.  I have highlighted what I now think is the key part.

I hope that helps.

Surely if the MTU of the WAN is 1492 and the LAN is 1500 then OPNSense should generate a ICMP type=3 code=4 response and send it to the LAN device.

It does.

Thanks to you both for your comments and to Patrick for confirming that OPNsense 25.1.2 does indeed respect the requirements of RFC 1191.

In the meantime I have carried out further tests after having:
* disabled Suricata
* disabled shaping
* inserted explicit fw rules to accept (and log) icmp destination unreachable messages (Should be pointless because such messages are associated with the original ping and so should automatically be accepted without logging)

In each case I did the following from my linux machine:

Code Select Expand

ping -c 1 -M do -D -s 1464 151.101.0.81 # Returns a echo response and so verify that the target host does honour pings
ping -c 1 -M do -D -s 1465 151.101.0.81 # Should receive icmp type 3 code 4 response from OPNsense, but doesnt.
In no case did the LAN and WAN interface captures show any icmp type 3 code 4 responses. The evidence seems to point to OPNsense 24.10 silently dropping the "oversize" DF marked icmp after being passed by the firewall on WAN but before being sent down the wire.

I have looked through the change logs of OPNsense releases since 24.10 and I see that 25.1.1 includes the following change:
  "src: pf: send ICMP destination unreachable fragmentation needed when appropriate"

So it looks as if the non-compliance with RFC 1911 is fixed in 25.1.1 - which would explain why neither of you have the problem.

Thanks again for your help.

Thanks for the fast response.

I had read those articles but my question regards what seems to me to be a bug in FreeBSD/OPNsense (or more likely a user error of course).

Surely if the MTU of the WAN is 1492 and the LAN is 1500 then OPNSense should generate a ICMP type=3 code=4 response and send it to the LAN device.

My evidence shows that this is not happening.

Or is my understanding incorrect?

My system:
*fully updated OPNsense 24.10.2-amd64 running on a Protecli VP4670.
*WAN configuration: pppoe->vlan->igc1 (connects to an ONT via ethernet)
*LAN configuration: vlan->igc0
*I do not manually set MTU parameters on any interface.
*I do use shaping on WAN and LAN

OPNsense correctly calculates the WAN MTU as 1492 (=1500-8 = 1464+28). Indeed, executing on the OPNsense box:
    (FP1) ping -D -s 1464 9.9.9.9 # receives a ping response
    (FP2) ping -D -s 1465 9.9.9.9 # is rejected by the interface with a "ping: sendto: Message too long" message. No message is actually sent down the wire.
   
On a linux machine connected to the LAN interface, executing:
    (LP1) ping -M do -D -s 1464 9.9.9.9 # receives a ping response
    (LP2) ping -M do -D -s 1465 9.9.9.9 # receives no response

From within OPNsense a packet capture of both the LAN interface and the WAN interface shows that the
echo request from ping LP2 is seen in the LAN interface capture but is NOT present in the WAN interface capture.

But the Firewall log for the WAN interface shows that the LP2 ping message is "passed" to the WAN interface for transmission.

I deduce that OPNsense (or FreeBSD) is discarding the LP2 ping without generating the ICMP type=3 code=4 response that should be sent to the device on the LAN.

Is there some setting (tunable?) that I can use to persuade OPNsense to send the ICMP type=3 code=4 responses to my LAN devices?

The three rules with syntax errors all reference the pf "alias" LOG_UDP. None of the other rules you quote contain that "alias".

I guess if $LOG_UDP were to be empty then that might lead to a syntax error?

The contents of LOG_UDP should be listed at the beginning of  /tmp/rules.debug.

Several years ago I had problems with aliases and I needed to increase the size of the OPNsense setting

    Firewall: Settings: Advanced->  Firewall Maximum Table Entries

So it looks as if it cannot be done.

I already successfully use the API for manipulating FreeRADIUS users and reading DHCPv4 leases.

Using the OPNSense API Reference, I have been unable to discover how to list the contents of any specific Firewall Alias. I have tried many commands for both the alias and alias-util controllers.

Can this be done?

If so, could somebody provide a one line hint using curl?

I am monitoring modifications to /var/dhcpd/etc/dhcpd.conf which is owned by the user dhcpd (uid=136).

When that file changes I would like to run a script as root.  However, it seems that the script is running as dhcpd which is not what I want and does not work

How can I configure monit to run this script as root?

I followed this guide, the only one I found of a user in the same ISP as mine, PFSense instead
https://forum.meo.pt/internet-fixa-e-movel-11/pfsense-ipv6-155245

Looking at a few other posts on that forum, it would seem that your provider could and does (or rather did)  change the delegated ipv6 prefix. It is not clear to me with what frequency. Though it seems that even small changes to the ISP router will cause the prefix to change.

Made a forced ISP Modem restart, IPv6 is back, without touching the OPNsense Router.

Does that mean that your system consists of:
   <ISP ONT> -> <ISP router> -> <OPNsense router>
If so how have you configured the ISP router for ipv6?

@Alec246: thanks

Why do you set "Request only an IPv6 prefix Checked "?  Is that required by your ISP?  Do you know what mechanism is used to get the ipv6 address of the gateway?

It would be good to know if your delegated ipv6 prefix is static or not. Maybe, try comparing it over a few days with a few WAN restarts in between?

When your ipv6 connectivity is functioning, could you share the first 8 hex digits of your WAN ipv6 address (no more than 8 since otherwise it might identify you). You can find it on the OPNsense Da.shboard

For the record here are my config details which are somewhat similar to yours, though I need to use pppoe and obtain my ipv6 prefix/address through dhcpv6 over the pppoe. I too have LANs based on vlan's over a lagg. Here are some details:

WANside interfaces:  WAN->pppoe->vlan->(interface on igc)

• interface on igc uses MAC spoofing and has Promiscuos mode set. It has no ipv4 or ipv6 configuration.
• vlan is required by my ISP
• pppoe is required by my ISP
• WAN:
  promiscuos mode unset and has DHCPv6 client configuration as follows:
  Request only an ipv6 prefix: No (otherwise the WAN has no ipv6 address of its own)
  Prefix delegation size: 48
  Send ipv6 prefix hint: yes
  Use ipv4 connectivity (required by my ISP - i.e. ipv6 trafic travels down the same pppoe connection as the ipv4.
  

LANside interfaces:  LAN->vlan->underLAN->lagg (lacp) ->(a pair of igc interfaces)

• underLAN has Promiscuos mode set and sets a MAC address. It has no ipv4 or ipv6 configuration.
• LAN has promiscuos mode unset. Static ipv4 and tracks the WAN interface for ipv6.
  

My ipv6 prefix is static and so will only change when I change ISP. The WAN ipv6 address is also static.