Messages

I am seeing "weird" stuff in dhclient.leases for at least one of my WAN interfaces as well. It's AT&T Fiber, and I'm bypassing their "BGW" device so I'm getting DHCP directly from their network (there's no "IP Passthrough" happening here). I'm not on v25.x yet, either - this is on 24.7.12, so I don't think this is a recent change.

My leases file has 11 separate "lease { }" stanzas in it, each of them with different renew, rebind, and expire dates and times. My dates and times appear to be relatively correct, but it seems to have a bunch of leases expiring several cycles "in the future," which I don't understand.

For example, it's currently 18:53. I have leases with a renew at 18:56 (this makes sense), but then also 19:26, 19:56, 20:26, 20:56, 21:26, 21:56, etc, all the way up to 23:56. I don't get why I have "future" leases, since I didn't think DHCP worked that way? The lease time is 3600 seconds (1hr), and I know that default behavior is to renew at the midpoint... I just don't get why I have "lease { }" entries for leases that shouldn't have been issued yet.

I don't seem to have any actual problems here, FWIW, just mentioning what seemed like odd behavior in my leases DB file.

(I was just looking through the file because I wanted to know the lease time, because I want to change my WAN MAC and was wondering how long I'd be stuck without service, because AT&T gives each customer an IP that doesn't really change, and will just ignore requests until the previous lease expires.)

[best]

I think subject says most of it.

I have an ONT/ONU device on one of my WAN interfaces that has a management IP, but after a while it stops answering ARP requests for that IP (I suspect this might be a "security" feature but I'm not positive, but I don't think it's something I can easily change).

Easy fix is to just add a static ARP entry on my WAN interface, and an IP Alias to that interface in the same subnet, and I can have permanent management access to the device.

What is the best way to add a static ARP entry that will survive reboots as well as upgrades? There doesn't seem to be a way to do it in the GUI. I found this ~4 year old post that suggested adding some lines to /etc/rc.conf but I assume this won't survive an upgrade and isn't really the "right" way to go.

If it was on the LAN side, I know I could add a DHCPv4 static entry with the "ARP Table Static Entry" option enabled, but I don't know if this will work for something in a different subnet that isn't actually being served by the DHCPv4 daemon...

[only]

This was happening on 22.1.8 and is still happening after upgrading to 22.1.9. Unfortunately I'm not exactly sure when it started happening. I noticed it after I changed the router's hostname (while still running 22.1.8 ).

I have dual-WAN, plus a LAN interface, and an OPT/secondary LAN that is disabled most of the time (but does have a static IP).

I'm using a subdomain of a public domain (that I own) for my LAN, and my opnsense machine is named "router.bh.example.net"  (example.net is standing in for my public domain here).

If I use dig against the router's hostname from my LAN, it is returning not only the LAN IP, but also the OPT, and both WAN IPs.

This is obviously somewhat of a problem because then hosts on the LAN aren't sure which IP to try to access for HTTPS / ping / etc.

I'm able to "fix" this by changing Services -> Unbound DNS -> General -> "Network Interfaces" to only list my LAN interface... but this is not desirable if I should ever want to use the OPT/secondary LAN. I don't want both interfaces to have the same hostname, and I definitely don't want public WAN IPs being resolved (internally) to the LAN hostname of the system.

I'm fairly sure Unbound / OPNsense didn't have this behavior in the past, as I think I would've run into problems trying to ping router.bh.example.net otherwise. Anybody else noticing this?

This has been fixed.  See https://forum.opnsense.org/index.php?topic=28695.0 for information on how to configure USB quirks to make this work.

As the person who figured things out on that other thread... I wouldn't call this "fixed," more like "worked around the problem." I don't know enough about the internals of FBSD and Nut to fully understand why this worked correctly in the past and doesn't now. It would be much better for someone with that knowledge to fix it correctly, instead of requiring a system tunable / tweak to get the kernel to step out of the way.

OPNsense tunables do end up in loader.conf.local and are activated during kernel initialisation.

I appreciate the clarification, I wish this was documented more clearly in the official docs. I knew that "tunables" were just a way to set sysctl flags, but it wasn't obvious to me what the load order is. I assumed (incorrectly) that the process by which they were loaded happened after initial kernel load...

FWIW, I ended up moving to a "netclient" Nut config - I was passing through the USB device (running OPNsense virtualized on Proxmox VE). I just setup Nut on the PVE server directly and am connecting the OPNsense VM to it via Nut's "netclient" mode now instead.

It's probably all redundant in my case, since Proxmox should gracefully shutdown the OPNsense guest anyway... but I'd rather the guest shut itself down on its own terms.

[confirm]

Conversing with myself, but this felt like it should be a new post.

https://www.freebsd.org/cgi/man.cgi?query=usb_quirk&sektion=4&apropos=0&manpath=FreeBSD+13.1-RELEASE+and+Ports

You'll need to use usbcontrol to find your device's descriptor info, first.

Code Select Expand

usbconfig -d ugen0.2 dump_device_desc
ugen0.2: <American Power Conversion Smart-UPS 1500 FW:UPS 09.4 / ID18> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (2mA)

  bLength = 0x0012
  bDescriptorType = 0x0001
  bcdUSB = 0x0200
  bDeviceClass = 0x0000  <Probed by interface class>
  bDeviceSubClass = 0x0000
  bDeviceProtocol = 0x0000
  bMaxPacketSize0 = 0x0040
  idVendor = 0x051d
  idProduct = 0x0003
  bcdDevice = 0x0106
  iManufacturer = 0x0001  <American Power Conversion >
  iProduct = 0x0002  <Smart-UPS 1500 FW:UPS 09.4 / ID=18>
  iSerialNumber = 0x0003  <xxxxx redacted  >
  bNumConfigurations = 0x0001

Note the "idVendor" and "idProduct" as you'll need these.

in /boot/loader.conf.local, you'll need to add, where x is a number starting at 0 (you may have other quirsk already)
hw.usb.quirk.x = "vendorID productID minprodVer maxprodver QUIRKNAME"

MinprodVer / maxProdVer are only important if you have multiple UPSes attached and are trying to be specific... otherwise, they can be "0" and "0xffff" for min and max (should catch all versions).

In our case, the problem is that the kernel is attaching the USB HID driver to the UPS, which Nut is having problems detaching. There's a "quirk" to tell the HID driver to ignore it... UQ_HID_IGNORE

Note that normally OPNsense would want us to set this "hw.usb.quirk.x" value in the GUI as a "System tunable," but I suspect that doing this won't work because tunables aren't loaded until after the kernel and modules are situated... which is too late. Option B would be to hack one of the nut startup scripts and call usbconfig and tell it to detach the kernel driver from the device (as I discovered in my earlier post)... but I like just telling the kernel to gtfo, especially since loader.conf.local should survive reboots and upgrades.

So, in my example, the final line added to /boot/loader.conf.local (which didn't exist on my system, so I created it) should be:
hw.usb.quirk.0 = "0x051d 0x0003 0 0xffff UQ_HID_IGNORE"

I can confirm that this has worked around my problem with nut + usbhid-ups (an APC) on 22.1.8_1, after a reboot. Nut loads correctly at boot and everything is working.

How do we fix this in OPNsense "correctly" instead of hacking around it? I dunno... maybe have nut run as root again instead of the uucp user? (I'm not sure why it's running as uucp in the first place?) Alternatively, figure out how to add the correct kernel-level permission to allow nut's user account to detach the kernel device driver without being UID 0. But my little work-around here is functional... just a pain in the butt since it requires manual intervention and is hard to script.  :P

[as root]

I've been having problems with this since 22.1.7, possibly a little earlier...

My first question / concern - is nut supposed to be running as the "uucp" user?

What I'm noticing, using a usbhid-ups driver, is that it chokes on access to a socket file that is supposed to live in /var/db/nut.

If I manually start that driver as root (/usr/local/libexec/nut/usbhid-ups -a <upsname> -DD -u root), it throws a complaint that the permissions on this socket file are wrong and that it has fixed them.

From that point forward, I can kill the debug copy, and then "restart" nut in the GUI and it works OK... but this doesn't survive a reboot.

So then I tried rm'ing the entire contents of /var/db/nut, and then restarting again... and this seems to work OK, but I suspect it too won't survive a reboot.

I've seen discussion elsewhere that the problem is permissions on the /dev tree (and specifically the USB entry for the UPS), but that did not need to be modified on my system.

I'm trying to figure out now how to permanently correct the permission issue so it survives a reboot...

[edit]
OK, it's not the /var/db/nut thing, permissions are fine by default.

But trying to run the usbhid-ups "nut driver" as root first is key. That forces the kernel USB HID driver to detach from the device, which then allows nut to bind to it, and everything works OK. Then you can kill off that process, get the GUI to start the driver as normal, and it all works.

Work-around:

Code Select Expand

usbconfig -d <usb device entry - mine is ugen8.2> detach_kernel_driver

After running that, you can kick the nut service in the webUI and everything works.
So, this is something weird going on with FBSD13 and the HID driver not wanting to let go of the device when nut asks for it; presumably somehow combined with permissions (maybe a change in how 13 handles this?)

Now I'm trying to see if there's a way to tell the kernel to just ignore the UPS device so I don't have to get it to release first...

[LAN]

I know this thread is over a month old now, but I found a solution and wanted to report it. :)

I believe my problem stemmed from the fact that I'm using Multi-WAN, which then requires rules on the LAN side to set a gateway group for the outgoing traffic.

I have a rule above my GW Group rule that is just a generic "allow to firewall LAN IP"... but that overlooks multicast.

Had to add this rule, prior to my GW group selection rule:
source network: <LAN net>
destination host: 239.255.255.250  [multicast IP used for UPnP discovery]
Protocol: UDP
Port: 1900

This got UPnP functional again, at least with the handy Mac app "Port Map". I haven't tried an Xbox yet, but I suspect it will be OK too.

[did]

Hi all,

I have a LOT of experience with this.  Are both the OPNSense box & the other device on the same switch?  Are they on the same vlan?  Can they ping each other directly without going through a router?

Well, yes, sort-of. :)

all of my devices are on a single VLAN. I have a Ubiquiti EdgeSwitch 24-PoE as my "core", and in two locations I have Ubiquiti Unifi switches (an 8 port and a 16 port PoE). I have IGMP snooping disabled on the EdgeSwitch, and have not enabled it in the Unifi controller in the "Network" section, so everything should be getting multicast fllood. All systems that would normally be using UPnP are hardwired.

I did just realize that I had some level of IGMP filtering happening on my Unifi APs ("Multicast Enhancement" was enabled on the wireless network), which would partially explain why some of my testing was failing (laptop is on wifi)... but it doesn't explain why the Xbox wasn't working (which is hardwired).

However, there is one other piece in the game I often forget about... I have a "SamKnows" bandwidth testing box on my LAN at home. It's basically a small linux box that sits behind my opnsense box and it just bridges all traffic through it, and then runs internet speed tests when it detects enough of an "idle window" in the traffic flow. It's supposed to be a transparent bridge, although I've discovered that it filters 802.1q (i.e. it chokes on vlan tagging). Multicast had been working fine in the past though (UPnP has not been a problem before.)

I wouldn't be entirely surprised if something changed and it started filtering multicast though.  That said, after fixing the filtering on the wifi, I now can use the Windows UPnP test app that thecodemonk mentioned, and I'm getting responses from opnsense. So it doesn't seem like multicast is entirely blocked, at least. More testing needed to see if it will allow me to actually map a port though...

Win10 (in a VMware VM, using bridged networking on my Mac) with this UPnPTest app can map a port. 

The Mac itself cannot map a port or even get a response from opnsense via UPnP (although NAT-PMP works). I even pulled down the "upnpc" client  (which is part of the miniupnpd project, same thing used on opnsense) with homebrew, and it isn't even getting an answer from opnsense... which makes me wonder if Apple is doing something with multicast on recent versions of MacOS. I don't really care much about the Mac though, the xboxes are what need to work.

I need to give an xbox another try and see if it magically starts working now... I definitely have IGMP snooping disabled on the Unifi switch it is uplinked to, so it should be getting flooded, but it's probably worth verifying that with wireshark because Unifi gear is known to often do things you don't expect.

This is pfSense documentation, but AFAIK since OPNsense is still based on the same guts, the order of operations are the same...

https://docs.netgate.com/pfsense/en/latest/nat/process-order.html

I think you'd be able to feed the various loader.conf variables to the "loader" prompt, but you wouldn't be able to edit /etc/ttys from there (AFAIK).

So I think it would boot and give you all of the kernel boot output at that point, but the second it passed to userland and where you'd normally get a screen or login prompt, you'd have nothing (if /etc/ttys isn't setup correctly).

I could be wrong though...

So I think I've come to the conclusion for me that it's Call of Duty that isn't working right, plus I think there might be a bug in the gui UPNP status.

The app I am using to test with will also display current port forwards and I have not been using it to check for them.. I've just been using the status page.

Curious, what app are you using to create the forwards (presumably on Windows)? I have a Win VM and a work laptop running Windows that I can break out to try a different app.

If I create a forward without a description, it does not display in the list. According to the specs, a description isn't required. But without a description it doesn't show in the GUI status list. I've been creating them without this whole time and seeing them not show up. Also, when testing these forwards, I haven't been closing the browser window or clearing states. I think FreshTomato must be doing that behind the scenes without telling you it is..

How is FreshTomato part of the scenario here, I thought you were running OPNsense?
You shouldn't need to clear states in the router to make a new port forward work.

Anyway. My test is by running nginx on my local PC (same one that runs Warzone/Cold War) with it's default web page. I create a forward using upnp for external port 8080 and internal port 80 to my local PC. Then on my phone, I turn off Wifi, and go to http://myexternalip:8080 and see if the page comes up. Without the forward, it doesn't come up. I then close that tab, clear all the states, then create the forward. I then go to that address on my phone and the page shows up immediately. Close the tab, remove the forward using the utility, clear all states, and open a tab and go to that address and it times out.

Yeah, so this proves that you are able to get a port to forward... if the opnsense GUI isn't showing that port mapping, that definitely seems like a bug.

I've tried that both with a description and without, and it works. Now, this was before reading about nat-pmp. So mine is on right now. I will test again with that turned off, but for now, it does look like this is working on my config (without changing any settings like xpendable has. I will most likely try his settings as well.

I don't know what utility you are using to create the mappings from your PC, but NAT-PMP is originally an Apple thing, so unless your utility explicitly has NAT-PMP support in it, it probably is just using UPnP.

I'm having problems with an XBox One X that is failing to get a port mapping via UPnP (and it has worked in the past with OPNsense), so there's definitely something wrong.  I have the same problem as you - wife will be upset if I break the internet to downgrade the router unless she's asleep, so I have to work around that. I'm also not exactly sure how to rollback OPNsense to an older release... Do I need to make a config backup and then fully reinstall from an older ISO?

[but]

I had some time to play around with this some more... from a Mac, using the free/open source "Port Map" app, and the Mac creates a port mapping successfully.

but, by default the Mac app is using NAT-PMP and not UPnP.

When I disable NAT-PMP in OPNsense's daemon, the mac app wants to fall back to UPnP, but it's not working. It sits at "searching" forever and things are not happy. I'm not seeing anything revealing popping up in the log on opnsense though...

I can confirm you're not crazy, something is wrong with UPnP for me too (on 21.1.4).

I'd be happy to provide more info if I knew what would help.

And in outbound nat, I have it in hybrid mode, and a rule for source lan net with the wan address at the nat address and static port enabled.

ZPrime he already has his outbound NAT using hybride mode with a single rule with static-ports for his entire LAN network.

Sorry about that... I shouldn't be replying at 4am local time. :p

Make sure you reboot your consoles or PC. I've noticed my Xbox or PS5 won't send AddPortMapping requests after they are up and running so no port mappings will show up in OPNsense until you reboot. But this isn't always the case.

I can confirm that with Xbox at least, it sometimes doesn't want to map. What sometimes helps (and avoids a lengthy reboot) is to get into the networking section and check/test the NAT status. If it still doesn't map after that, there's a secret code (IIRC it's hold both triggers and then hit both bumpers simultaneously) that gives you a "more details" screen, and after bringing that up and then checking NAT status again, it usually seems to force the mapping.

UPnP is so great in theory, but the practice/execution is very janky at times...