Messages

Its /22, but I have a static IP so I'll always get the same IP from the ISP

Is that /22 an rfc1918 block, or ISP public? Do you also get DHCP?

No, it's not. When I initially went with this ISP I had them disable CG-NAT (In case that's what you were thinking it might be).

I've had 5 days uptime with zero problems since installing it as a VM (Under Proxmox), so it seems it's an issue with BSD drivers.

The fw WAN is likely not in a /30.
So let's ask.... OP, what subnet is your FW WAN getting from dhcp, or now whatever OS is connecting to the ISP?

Its /22, but I have a static IP so I'll always get the same IP from the ISP

Not sure what version of OPNsense you are running, but duly noted freeBSD 14.3-RELEASE has a noted fix for igc driver.
https://www.freebsd.org/releases/14.3R/relnotes/

Its on the latest.



I'll have a look at those links you've sent. So far the switch to using proxmox with Opnsense as a VM has been flawless.

Hey all, I've done a full reinstall of Opnsense and have been learning the difference between Dnsmasq and ISC. Previous in ISC I could put in static leases with basically just the hostname, ip address, and the mac address. It would also automatically register leases into Unbound.

As far as i'm aware i've setup Dnsmasq as required - I've created DHCP Ranges for each interface and then created a bunch of Hosts entries, I found I needed to put in a Domain in order for it to resolvem, Is that correct or did I do something wrong? One thing I've also noticed is that under Hosts when I select a filter say for the LAN interface nothing actually shows up, does that mean i've set the Hosts incorrectly? I couldn't see an interface assignment or something when I created the host overrides.

I've set Dnsmasq to listen to 53053, enabled "Do not forward to system defined DNS servers", DHCP FQDN, DHCP local domain, and DHCP register firewall rules. In unbound i've set Register DHCP Static Mappings, and " Do not register system A/AAAA records" with an override to only return the LAN Ip for opnsense.

I've been unable to get to the bottom of the issues unfortunately so i've but it in the VM under Proxmox. Took a bit of doing because Unbound and dnsmasq are the defaults now - I didn't want to just restore from backup so I didn't bring across any weird nonsense I had done on my previous install when trying to get stuff working.

I'll let everybody know how things go - I really wish I could have figured it out but it was getting on my nerves constantly having to restart stuff.

Do you have a /var/log/messages file? If so you can cat or grep that file looking for entries related to igc or interfaces. State changes should be logged.

I also suspect not related to any power or sleep settings, the WAN iface is always active just from fw itself doing stuff, and, the fw never actually drops off into a power state of sleep.

Interface hardware seems ok, need to look elsewhere. DHCP issues is a DHCP issue, not a hardware issue, etc. I don't suspect DHCP either.
I did mean to ask earlier, in your DHCP clinet file, is the provided IP the same or did it change?

When you say "100% packet loss", what tool is used to derive that? Ping using IP? Other?

Another thing to look at is "arp -a" , make note of the igc value, keep running the command, watch the timer go down, make note of the MAC address, when the timer gets to zero just keep watching for the arp renew, right after zero timer keep watching that you get a IP and MAC address quickly, any delay here would cause 100% packet loss. Your Intel WAN iface should be the MAC that starts with 00:e0:b4, so you want to look at the other one with the timer (usually at the op of the list), this is your DFG, aka ISP IP and MAC on WAN side.

There was no /var/log/messages file unfortunately. Under Gateways configuration it would have Loss: 100%.
The provided IP Address is always the same. I'll keep looking at that 'arp -a' command, I had a look a a few times and it seemed to refresh always in the last 5 seconds or so

For brevities sake, here are the tunables i've added so far:

Code Select Expand

hw.pci.enable_aspm = 0
hw.em.smart_pwr_down = 0
hw.pci.do_power_nodriver = 0
hw.pci.do_power_suspend = 0
net.link.ether.inet.max_age = 120
dev.igc.0.fc = 0
dev.igc.1.fc = 0
hw.igc.eee_setting = 0


Also, Just wanted to quickly thanks everybody for your help so far - it's been fantastic, i'm learning a lot. Hopefully we can get to the bottom of it as there are a few others that also have issues.

Code Select Expand

$ kldstat
Id Refs Address                Size Name
 1   71 0xffffffff80200000  216dad8 kernel
 2    1 0xffffffff8236e000    16650 if_lagg.ko
 3    2 0xffffffff82385000     3558 if_infiniband.ko
 4    1 0xffffffff82389000     ed60 if_bridge.ko
 5    2 0xffffffff82398000     8990 bridgestp.ko
 6    1 0xffffffff823a2000    1e280 opensolaris.ko
 7    1 0xffffffff823c1000    11a78 pfsync.ko
 8    3 0xffffffff823d3000    908a0 pf.ko
 9    1 0xffffffff82464000     3c10 pflog.ko
10    1 0xffffffff832ce000     aa30 if_gre.ko
11    1 0xffffffff832d9000     4be0 if_enc.ko
12    1 0xffffffff832de000     fb90 carp.ko
13    1 0xffffffff832ee000   5e9300 zfs.ko
14    1 0xffffffff84510000    b4270 if_iwlwifi.ko
15    1 0xffffffff845c5000     3378 lindebugfs.ko
16    1 0xffffffff845c9000     d200 rtsx.ko
17    1 0xffffffff845d7000     4250 ichsmb.ko
18    1 0xffffffff845dc000     2178 smbus.ko
19    1 0xffffffff845df000     3390 acpi_wmi.ko
20    1 0xffffffff845e3000     5640 ng_ubt.ko
21    4 0xffffffff845e9000     abb8 netgraph.ko
22    3 0xffffffff845f4000     a250 ng_hci.ko
23    2 0xffffffff845ff000     2670 ng_bluetooth.ko
24    1 0xffffffff84602000    2f5c0 if_wg.ko
25    1 0xffffffff84632000     4850 nullfs.ko

Yep, I don't think it's a driver issue specifically. I have already disabled "Allow DNS server list to be overridden by DHCP/PPP on WAN" as well.

When it drops out, it's just 100% packet loss. Next time it happens, i'll try and capture as many different types of logs as I can.

What sort of logs should I be capturing to try and help us identify the root cause?

Quite edit: I've set up a ping on Opnsense to my remote VPS, and I have it pinging back as well so I can monitor traffic in both directions

How do I confirm that the igc driver is loaded correctly? I think I read somewhere there should be a kernel module present and loaded. I can't find it now though. I'd have thought that because it's identified the device that the driver wouldn't be the issue.

I'm new to BSD so I don't know how to really troubleshoot this stuff unfortunately.

[aspm: 1]

Hmmm, well, my i226v N150 has the aspm disabled on igc, but I don't see where the setting that disables it, seems like my settings are set to "1".
Are you running powerd?

sysctl -a |grep hw.pci.enable
hw.pci.enable_pcie_ei: 0
hw.pci.enable_pcie_hp: 1
hw.pci.enable_mps_tune: 1
hw.pci.enable_aspm: 1
hw.pci.enable_ari: 1
hw.pci.enable_msix: 1
hw.pci.enable_msi: 1
hw.pci.enable_io_modes: 1

pciconf -lbcevV igc1
cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
                max read 512
                link x1(x1) speed 5.0(5.0) ASPM disabled(L1)


You have WAN dhcp? What does the lease time look like?
in "/var/db/dhclient.leases.igcX" , X being your WAN iface number

option dhcp-lease-time

Not sure if this is normal, but there are a lot of leases:

Code Select Expand

root@OPNsense:~ # cat /var/db/dhclient.leases.igc1
lease {
  interface "igc1";
  fixed-address AAA.BBB.CC1.132;
  option subnet-mask 255.255.252.0;
  option routers AAA.BBB.CC0.1;
  option domain-name-servers XXX.YYY.ZZZ.142,XXX.YYY.ZZZ.242;
  option host-name "opnsense";
  option dhcp-lease-time 1800;
  option dhcp-message-type 5;
  option dhcp-server-identifier AAA.BBB.CC0.1;
  renew 3 2025/8/27 07:24:18;
  rebind 3 2025/8/27 07:35:33;
  expire 3 2025/8/27 07:39:18;
}
lease {
  interface "igc1";
  fixed-address AAA.BBB.CC1.132;
  option subnet-mask 255.255.252.0;
  option routers AAA.BBB.CC0.1;
  option domain-name-servers XXX.YYY.ZZZ.142,XXX.YYY.ZZZ.242;
  option host-name "opnsense";
  option dhcp-lease-time 1800;
  option dhcp-message-type 5;
  option dhcp-server-identifier AAA.BBB.CC0.1;
  option dhcp-renewal-time 900;
  option dhcp-rebinding-time 1575;
  renew 3 2025/8/27 07:32:05;
  rebind 3 2025/8/27 07:43:20;
  expire 3 2025/8/27 07:47:05;
}
lease {
  interface "igc1";
  fixed-address AAA.BBB.CC1.132;
  option subnet-mask 255.255.252.0;
  option routers AAA.BBB.CC0.1;
  option domain-name-servers XXX.YYY.ZZZ.142,XXX.YYY.ZZZ.242;
  option host-name "opnsense";
  option dhcp-lease-time 1800;
  option dhcp-message-type 5;
  option dhcp-server-identifier AAA.BBB.CC0.1;
  renew 3 2025/8/27 07:47:05;
  rebind 3 2025/8/27 07:58:20;
  expire 3 2025/8/27 08:02:05;
}
lease {
  interface "igc1";
  fixed-address AAA.BBB.CC1.132;
  option subnet-mask 255.255.252.0;
  option routers AAA.BBB.CC0.1;
  option domain-name-servers XXX.YYY.ZZZ.142,XXX.YYY.ZZZ.242;
  option host-name "opnsense";
  option dhcp-lease-time 1800;
  option dhcp-message-type 5;
  option dhcp-server-identifier AAA.BBB.CC0.1;
  option dhcp-renewal-time 900;
  option dhcp-rebinding-time 1575;
  renew 3 2025/8/27 08:02:05;
  rebind 3 2025/8/27 08:13:20;
  expire 3 2025/8/27 08:17:05;
}
lease {
  interface "igc1";
  fixed-address AAA.BBB.CC1.132;
  option subnet-mask 255.255.252.0;
  option routers AAA.BBB.CC0.1;
  option domain-name-servers XXX.YYY.ZZZ.142,XXX.YYY.ZZZ.242;
  option host-name "opnsense";
  option dhcp-lease-time 1800;
  option dhcp-message-type 5;
  option dhcp-server-identifier AAA.BBB.CC0.1;
  renew 3 2025/8/27 08:17:05;
  rebind 3 2025/8/27 08:28:20;
  expire 3 2025/8/27 08:32:05;
}
lease {
  interface "igc1";
  fixed-address AAA.BBB.CC1.132;
  option subnet-mask 255.255.252.0;
  option routers AAA.BBB.CC0.1;
  option domain-name-servers XXX.YYY.ZZZ.142,XXX.YYY.ZZZ.242;
  option host-name "opnsense";
  option dhcp-lease-time 1800;
  option dhcp-message-type 5;
  option dhcp-server-identifier AAA.BBB.CC0.1;
  option dhcp-renewal-time 900;
  option dhcp-rebinding-time 1575;
  renew 3 2025/8/27 08:32:05;
  rebind 3 2025/8/27 08:43:20;
  expire 3 2025/8/27 08:47:05;
}
lease {
  interface "igc1";
  fixed-address AAA.BBB.CC1.132;
  option subnet-mask 255.255.252.0;
  option routers AAA.BBB.CC0.1;
  option domain-name-servers XXX.YYY.ZZZ.142,XXX.YYY.ZZZ.242;
  option host-name "opnsense";
  option dhcp-lease-time 1800;
  option dhcp-message-type 5;
  option dhcp-server-identifier AAA.BBB.CC0.1;
  option dhcp-renewal-time 900;
  option dhcp-rebinding-time 1575;
  renew 3 2025/8/27 08:47:06;
  rebind 3 2025/8/27 08:58:21;
  expire 3 2025/8/27 09:02:06;
}
lease {
  interface "igc1";
  fixed-address AAA.BBB.CC1.132;
  option subnet-mask 255.255.252.0;
  option routers AAA.BBB.CC0.1;
  option domain-name-servers XXX.YYY.ZZZ.142,XXX.YYY.ZZZ.242;
  option host-name "opnsense";
  option dhcp-lease-time 1800;
  option dhcp-message-type 5;
  option dhcp-server-identifier AAA.BBB.CC0.1;
  renew 3 2025/8/27 09:02:06;
  rebind 3 2025/8/27 09:13:21;
  expire 3 2025/8/27 09:17:06;
}
lease {
  interface "igc1";
  fixed-address AAA.BBB.CC1.132;
  option subnet-mask 255.255.252.0;
  option routers AAA.BBB.CC0.1;
  option domain-name-servers XXX.YYY.ZZZ.142,XXX.YYY.ZZZ.242;
  option host-name "opnsense";
  option dhcp-lease-time 1800;
  option dhcp-message-type 5;
  option dhcp-server-identifier AAA.BBB.CC0.1;
  option dhcp-renewal-time 900;
  option dhcp-rebinding-time 1575;
  renew 3 2025/8/27 09:17:06;
  rebind 3 2025/8/27 09:28:21;
  expire 3 2025/8/27 09:32:06;
}
lease {
  interface "igc1";
  fixed-address AAA.BBB.CC1.132;
  option subnet-mask 255.255.252.0;
  option routers AAA.BBB.CC0.1;
  option domain-name-servers XXX.YYY.ZZZ.142,XXX.YYY.ZZZ.242;
  option host-name "opnsense";
  option dhcp-lease-time 1800;
  option dhcp-message-type 5;
  option dhcp-server-identifier AAA.BBB.CC0.1;
  renew 3 2025/8/27 09:32:06;
  rebind 3 2025/8/27 09:43:21;
  expire 3 2025/8/27 09:47:06;
}


@jstarta: Please show the output of "sysctl hw.pci" - I do not believe that the ASPM setting was applied correctly.

Code Select Expand

root@OPNsense:~ # sysctl hw.pci
hw.pci.mcfg: 1
hw.pci.host_mem_start: 2147483648
hw.pci.default_vgapci_unit: 0
hw.pci.enable_pcie_ei: 0
hw.pci.pcie_hp_detach_timeout: 5000
hw.pci.enable_pcie_hp: 1
hw.pci.clear_pcib: 0
hw.pci.iov_max_config: 1048576
hw.pci.intx_reroute: 1
hw.pci.enable_mps_tune: 1
hw.pci.clear_aer_on_attach: 0
hw.pci.enable_aspm: 0
hw.pci.enable_ari: 1
hw.pci.clear_buses: 0
hw.pci.clear_bars: 0
hw.pci.usb_early_takeover: 1
hw.pci.honor_msi_blacklist: 1
hw.pci.msix_rewrite_table: 0
hw.pci.enable_msix: 1
hw.pci.enable_msi: 1
hw.pci.do_power_suspend: 0
hw.pci.do_power_resume: 1
hw.pci.do_power_nodriver: 0
hw.pci.realloc_bars: 1
hw.pci.enable_io_modes: 1
root@OPNsense:~ # pciconf -lbcevV igc1
igc1@pci0:89:0:0:       class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x125c subvendor=0x1462 subdevice=0xb0b1
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller I226-V'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 32, base 0x6a300000, size 1048576, enabled
    bar   [1c] = type Memory, range 32, base 0x6a400000, size 16384, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 5 messages, enabled
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
    cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
                 max read 512
                 link x1(x1) speed 5.0(5.0) ASPM L1(L1)
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 0003[140] = Serial 1 d843aeffffbc6cac
    ecap 0018[1c0] = LTR 1
    ecap 001f[1f0] = Precision Time Measurement 1
    ecap 001e[1e0] = L1 PM Substates 1


I have a brand new MSI Cubi 1M with 2 x Intel I226V Nics
I installed OPNSense on it (last week)
I have a 1000/50 internet connection.

With OPNSense, the download speed varies depending on what program is trying to do the download.

From my Windows PC I can run speedtest.net and get full download speed.
I can run SABnzbd and get full download speeds.

On my Ubuntu box I can run speedtest.net and get full download speed.
I can run SABnzbd and the download speed stop out good but then drops and drops.
The Ubuntu box also has docker containers.
From docker, SABnzbd download speed stop out good but then drops and drops.
From docker, speedtest-tracker shows slow download speeds.

Prior to using OPNSense (last week), I was using a Netgate 6100 which never has these slow download speeds.
I also tested with a Netgear XR500 which also had full download speeds.

I will see if the " disable PCI-E ASPM in BIOS power settings" makes an improvement over the next few days.

root@firewall:~ # pciconf -llcvVBa igc0
drv    selector        class    rev  hdr  vendor device subven subdev
igc0@pci0:88:0:0:      020000  04  00  8086  125c  1462  b0b1
    vendor    = 'Intel Corporation'
    device    = 'Ethernet Controller I226-V'
    class      = network
    subclass  = ethernet
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 5 messages, enabled
                Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
    cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
                max read 512
                link x1(x1) speed 5.0(5.0) ASPM L1(L1)
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 0003[140] = Serial 1 d843aeffffb16e95
    ecap 0018[1c0] = LTR 1
    ecap 001f[1f0] = Precision Time Measurement 1
    ecap 001e[1e0] = L1 PM Substates 1

 

I have exactly the same machine - I don't have speed issues on my WAN but I do on the LAN. However the WAN connection drops out daily. As far as I can tell, there is no way to disable ASPM in the BIOS that i've been able to find. Did you have any luck with that?

[ASPM L1(L1)]

I had a look at the pciconf for bother interfaces, and it looks like the tunable didn't take effect 'hw.pci.enable_aspm=0', because it states ASPM is still enabled in the output:


root@OPNsense:~ # pciconf -lbcevV igc1
igc1@pci0:89:0:0:       class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x125c subvendor=0x1462 subdevice=0xb0b1
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller I226-V'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 32, base 0x6a300000, size 1048576, enabled
    bar   [1c] = type Memory, range 32, base 0x6a400000, size 16384, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 5 messages, enabled
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
    cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
                 max read 512
                 link x1(x1) speed 5.0(5.0) ASPM L1(L1)
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 0003[140] = Serial 1 d843aeffffbc6cac
    ecap 0018[1c0] = LTR 1
    ecap 001f[1f0] = Precision Time Measurement 1
    ecap 001e[1e0] = L1 PM Substates 1
root@OPNsense:~ # pciconf -lbcevV igc0
igc0@pci0:88:0:0:       class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x125c subvendor=0x1462 subdevice=0xb0b1
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller I226-V'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 32, base 0x6a600000, size 1048576, enabled
    bar   [1c] = type Memory, range 32, base 0x6a700000, size 16384, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 5 messages, enabled
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
    cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
                 max read 512
                 link x1(x1) speed 5.0(5.0) ASPM L1(L1)
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 0003[140] = Serial 1 d843aeffffbc6cab
    ecap 0018[1c0] = LTR 1
    ecap 001f[1f0] = Precision Time Measurement 1
    ecap 001e[1e0] = L1 PM Substates 1

It's set as a IPv4 DHCP connection, though I guess technically it's static IPv4 because my ISP gives me a static ip

Cable, Ethernet or fiberoptics?

Ethernet. Setting that tunable didn't seem to fix things unfortunately.

Just a quick add - I checked the BIOS for any ASPM stuff but couldn't see anything. I did see an ErP Ready setting which i've just disabled now (Seemed to have something to do with limiting power).

Added the tunable "hw.pci.enable_aspm" and set it to 0. I'll give it a reboot at some point and then see how it all goes. This BIOS is definitely lacking a lot of advanced features :(

[...]
What logs should I be looking at to help me figure out what the issue is? [...]

I'd look at ARP. One of the logs (General, I believe) may log ARP changes, but that's usually only when ARP moves between bridge member interfaces. You'll probably have to look when you lose connectivity. It could also be the (apparent) i226 ASPM issue.

I only see two entries for the WAN interface - i'll take a look at my bios for the ASPM settings (Thanks for the hint).

The most important bit in this mystery is the type of your WAN connection.

It's set as a IPv4 DHCP connection, though I guess technically it's static IPv4 because my ISP gives me a static ip