OPNsense Forum
Archive => 21.7 Legacy Series => Topic started by: MacLemon on July 09, 2021, 05:17:04 pm
-
UPDATE
This thread was started with 21.7-RC1 but applies to the 21.7-RELEAS (2021-07-29) version as well.
Not only new installs, but also upgrades with VLANs on LAGG interfaces seem to be affected.
Summary:
When importing an existing OPNsense 21.1.7_1 config into a freshly installed 21.7-RC1 config (on different Hardware) the system hangs during reboot at “Configuring VLAN interfaces...” forever.
Steps to Reproduce:
- Production firewall: OPNsense 21.1.7_1 on intel Atom C2758 with 8 x i350 (igb0-3 on PCIe NIC, igb4-7 onboard) Only igb0-3 are in use!
- Export Config file as usual with all sections.
- Install 21.7-RC1 on new Hardware (Xeon-D 2143, 4 x i350 (igb0-3), 4 x X772(ixl0-3))
- Import config from the old firewall
- reboot
- See boot hang at configuring VLANs.
Expected Results:
I'd expect the import to automatically reassign the interfaces according to their names.
Match old igb0 to new igb0. (Same name, different MAC) and so on.
This is in fact the behaviour I actually see happen flawlessly when trying the same migration from 21.1.7_1 on the old hardware to 21.1 on the new hardware. It just works, and works as I had hoped for.
Actual Results:
When importing the same file exported on 21.1.7_1 on the old hardware into 21.7-RC1 on the new hardware the system hangs at the first reboot (and all subsequent reboots).
No errors are shown.
Regression:
The hardware change works *perfectly* fine (it's almost boring) when importing that same file from 21.1.7_1 into 21.1 on the new hardware.
System boots up as expected and automagically assigns all the NICs correctly.
Notes:
I've also tried to import the config into 21.1 on the new hardware, export it again into a fresh file. Which would basically resemble having the same hardware reinstalled and reimporting an existing config dump from that exact hardware.
This results in the same problem.
Version Information:
Old hardware: 21.1.7_1
New hardware: 21.1 and 21.7-RC1 tested
Is there anything else I could have missed during these tests? Any obvious mistakes I've overlooked while covering my ears from the fan noise on my desk?
You input is much appreciated.
MacLemon
-
Do you use third-party repositories? E.g. SunnyValley.
If yes, and there is no binary available, Opnsense setup chokes to death.
-
Thanks for the input.
Nope, Sunny Valley is not in use.
After importing the config the system hangs before it has fully booted up. So it doesn't even get to a point where it could fail to download/install any plugins.
-
On a config reimport it's very unlikely third party repositories play a role, especially when their code is not there the settings in the config.xml won't be executed.
The question is what is hanging there. Looking at the code legacy_interface_listget() is executed first which also checks for WLAN capable cards. Is there such a thing plugged into the new HW?
Cheers,
Franco
-
No WiFi hardware here.
NICs:
- 4 x intel i350 Gigabit/s (igb)
- 2 x intel X772 10GE Copper (ixl)
- 2 x intel X772 10GE SFP+ (ixl)
The new hardware used is a Thomas-Krenn RI1102D-F (v2.1) (https://www.thomas-krenn.com/en/products/rack-server/1u-servers/intel-single-cpu/intel-single-ri-1102d-f.html) which is basically a SUPERMICRO X11SDV-4C-TP8F (https://www.supermicro.com/en/products/motherboard/X11SDV-4C-TP8F) motherboard preassembled in a chassis with 2x8GB RAM and an NVMe SSD as boot drive.
The console output up to the hang reads like this:
Configuring Kernel Modules...done.
Setting up extended sysctls...done.
Setting timezone...done.
Writing firmware setting...done
Writing trust files...done.
Settings hostname: <opnsense.example.org>
Generating /etc/hosts...done.
Configuring system logging...done.
Configuring loopback interface...done.
Creating wireless clone interfaces...done.
Configuring LAGG interfaces...done.
Configuring VLAN interfaces...
This is where it never continues any further.
Importing the very same file into 21.1 works flawlessly.
Importing it on 27.1-RC1 shows this symptom of a non-booting firewall.
To me that points to a difference in how the config is parsed for the VLAN section, or in the way the VLAN interfaces are getting configured.
We do use a LAGG (LACP) and all the VLANs are on that LAGG if this is of any help.
-
> Importing the very same file into 21.1 works flawlessly. (1)
> Importing it on 27.1-RC1 shows this symptom of a non-booting firewall. (2)
The question is if (1) was confirmed on the new hardware as well and if the upgrade actually makes it stuck. I'm not convinced it's the code that wasn't considerably changed.
Cheers,
Franco
-
I did test:
- export config on 21.1.7_1 on the existing hardware
- import into 21.1 on the new hardware
Imports all the settings correctly, maps the igb0-3 interfaces correctly, reboots completely, just works.
I'd say, the common OPNsense experience with updates. :-)
I also did test
- export config on 21.1.7_1 on the existing hardware
- import into 21.7 on the new hardware
which results on the mentioned hang while configuring VLAN interfaces.
I've also tested
- export config on 21.1.7_1 on the existing hardware
- import into 21.1 on the new hardware
- export config again from 21.1 on the new hardware
- import the new export into 21.7 on the new hardware
which results in the same hang.
Is there anything else I could test? I do have the new hardware at my disposal for tests. :-)
-
When you put it like that maybe something in the RC is causing this. The first suspect would be
https://github.com/opnsense/core/commit/a98d776fa4ff0
Can you try to patch it and see if the hang is still there?
# opnsense-patch a98d776fa4ff0
(patching actually un-patches it, but it works splendid for testing)
Cheers,
Franco
-
- Installed OPNsense 21.7-RC1
- Applied Patch as instructed (output below)
- rebooted for good measure (came back just fine)
- Imported unencrypted config file that had been exported on the old hardware with 21.1.7_1
- auto-reboot after Config import.
Same result so far. The reboot after the import hangs at
Configuring VLAN interfaces...
Full output emitted when applying the patch.
# opnsense-patch a98d776fa4ff0
Fetched a98d776fa4ff0 via https://github.com/opnsense/core
Hmm... Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|From a98d776fa4ff04d616e46b45a3bc60f8c1407269 Mon Sep 17 00:00:00 2001
|From: Ad Schellevis <ad@opnsense.org>
|Date: Wed, 16 Jun 2021 16:18:50 +0200
|Subject: [PATCH] Interfaces / Hardware settings - Overwite global settings,
| closes https://github.com/opnsense/core/issues/5050
|
|---
| src/etc/inc/interfaces.lib.inc | 32 +++++++----
| src/www/interfaces.php | 102 +++++++++++++++++++++++++++++++++
| 2 files changed, 123 insertions(+), 11 deletions(-)
|
|diff --git a/src/etc/inc/interfaces.lib.inc b/src/etc/inc/interfaces.lib.inc
|index 9ca22ab996..cc4c59470b 100644
|--- a/src/etc/inc/interfaces.lib.inc
|+++ b/src/etc/inc/interfaces.lib.inc
--------------------------
Patching file etc/inc/interfaces.lib.inc using Plan A...
Reversed (or previously applied) patch detected! Assuming -R.Hunk #1 succeeded at 386 (offset 1 line).
Hunk #2 succeeded at 399 (offset 1 line).
Hunk #3 succeeded at 439 (offset 1 line).
Hmm... The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|diff --git a/src/www/interfaces.php b/src/www/interfaces.php
|index 30a95ff65e..c078c1a4de 100644
|--- a/src/www/interfaces.php
|+++ b/src/www/interfaces.php
--------------------------
Patching file www/interfaces.php using Plan A...
Reversed (or previously applied) patch detected! Assuming -R.Hunk #1 succeeded at 388.
Hunk #2 succeeded at 1307.
Hunk #3 succeeded at 1699.
Hunk #4 succeeded at 1913.
done
All patches have been applied successfully. Have a nice day.
Here's the VLAN config section extracted from the config file. I've replaced the customer's name with “customer” and some other vendor's we use with “vendor”. The general description structure stays identical. (The only “special” characters in the description fields are spaces and a “-”.)
All VLANs are assigned to the same single lagg0 interface.
0 <vlans>
1 <vlan>
2 <if>lagg0</if>
3 <tag>1104</tag>
4 <pcp>1</pcp>
5 <descr>Customer Studio</descr>
6 <vlanif>lagg0_vlan1104</vlanif>
7 </vlan>
8 <vlan>
9 <if>lagg0</if>
10 <tag>1254</tag>
11 <pcp>7</pcp>
12 <descr>AdminVLAN</descr>
13 <vlanif>lagg0_vlan1254</vlanif>
14 </vlan>
15 <vlan>
16 <if>lagg0</if>
17 <tag>1251</tag>
18 <pcp>0</pcp>
19 <descr>XXX DMZ</descr>
20 <vlanif>lagg0_vlan1251</vlanif>
21 </vlan>
22 <vlan>
23 <if>lagg0</if>
24 <tag>1250</tag>
25 <pcp>3</pcp>
26 <descr>YYY DMZ</descr>
27 <vlanif>lagg0_vlan1250</vlanif>
28 </vlan>
29 <vlan>
30 <if>lagg0</if>
31 <tag>1105</tag>
32 <pcp>1</pcp>
33 <descr>CustomerPublic</descr>
34 <vlanif>lagg0_vlan1105</vlanif>
35 </vlan>
36 <vlan>
37 <if>lagg0</if>
38 <tag>1109</tag>
39 <pcp>2</pcp>
40 <descr>Customer labels</descr>
41 <vlanif>lagg0_vlan1109</vlanif>
42 </vlan>
43 <vlan>
44 <if>lagg0</if>
45 <tag>1106</tag>
46 <pcp>2</pcp>
47 <descr>VendorService</descr>
48 <vlanif>lagg0_vlan1106</vlanif>
49 </vlan>
50 <vlan>
51 <if>lagg0</if>
52 <tag>1107</tag>
53 <pcp>1</pcp>
54 <descr>VendorDemo</descr>
55 <vlanif>lagg0_vlan1107</vlanif>
56 </vlan>
57 <vlan>
58 <if>lagg0</if>
59 <tag>1108</tag>
60 <pcp>1</pcp>
61 <descr>CustomerDemo</descr>
62 <vlanif>lagg0_vlan1108</vlanif>
63 </vlan>
64 </vlans>
-
I tried to reproduce with a lagg0 and the vlans on top and boot went fine. I half-suspect the config isn't the underlying issue, but it would be good to rule that out. Would you mind sending it over to franco@opnsense.org with the necessary redactions?
If that were true it might still be tied to specific hardware behaviour in which case we can only try to find the command that hangs the system.
Cheers,
Franco
-
I just had this exact issue happen to me when trying to upgrade from 21.1 to 21.7. After updating the we web UI and rebooting it hangs at "Configuring VLAN interfaces". Never proceeds further. No hardware change for me. Running on a supermicro X10SDV-TP8F
I also have a lagg interface that has vlans associated with it.
-
Getting the exact same issue. Upgraded this morning to 21.1.9, then an in-pace upgrade to 21.7. Hardware is a QOTOM Q555G6-S05. I used the in-place upgrade method rather than a config import. This is the 2nd node of a 2-node HA cluster.
If I can help troubleshoot in any way please let me know what is needed.
-
Same isssue with a ASUS AMD 350 Zacate Mini ITX Board.
-
In an attempt to bring my network back up I tried using my config with the 21.7 install config importer. That failed and hung configuring the VLANs. Went back a release to 21.1 and the installer successfully imported the config and booted fully.
-
@franco, I just realized this is a post for RC1. Should we start a new thread since this is now the GA code?
-
same issue here with the final 21.7.
After upgrade it stucks at the "vlan changing name to..."
I use miniPC with intel CPU and NIC's (nrg-systems.de)
what i tested:
Clean install 21.7(works) and restore with 21.1.9 backup
-> freeze on vlan config
Clean install 21.1.9, restore config 21.1.9 and upgrade to 21.7
-> freeze on vlan config
Clean install 21.1.9 and restore config 21.1.9
-> everything works fine
i also use LAGG interfaces with VLAN's
Thanks for Help!
Best Regards
-
I'm not sure if it's related to this but there's definitely something related to VLANs in 21.7. The system is fine until I began adding VLANs. Using the same hardware that failed to upgrade, here's what I did and what I ran into. Apologies if it's a ramble or is missing relevant data, it's 2 AM, I've been testing this for the last few hours because OCD. The issue is 100% repeatable.
Steps to reproduce:
- Clean install of 21.7. Mirrored ZFS (Old 21.1 setup was GEOM Mirror)
- Minor changes. Change theme, point logging to syslog server, etc.
- Create lagg0 from igb0 and igb1.
- Create lagg1 from idb2 and igb3.
- Assign lagg0 to WAN interface
- Assign lagg1 to LAN interface
- Create VLAN 20, parent interface lagg0. No problems at all up to this point.
- Here's where the trouble begins. Create VLAN 30, parent interface lagg0
As soon as I created VLAN 30 on lagg0 the system started having trouble.
The first time I tried to add VLAN 30, the system froze up and I started getting scrolling drive alerts (ATA_IDENTIFY, CAM status, etc) on the console. Had to hard power cycle the system to get it back.
After the power cycle, the second time I tried to add VLAN 30, the system froze up and the GUI is just sitting there with a dot moving on the browser tab like the browser is waiting, but there are no drive alerts on the console. The system shutdown gracefully via power button this time.
The third time I tried to add VLAN 40 and had pretty much the same results.
I was sending my logs to a syslog server during this and captured this is the configd.py.log file during the 1st attempt.
2021-07-29T00:39:27-04:00 inner-fw2.lan.thejeffcoats.net configd.py[84073]: [8d7456bd-a164-4aab-b572-0893ce42a42c] Linkup stopping igb0
2021-07-29T00:39:27-04:00 inner-fw2.lan.thejeffcoats.net configd.py[84073]: [e0604275-642d-4c72-a2e4-0c8c8a70bb27] Linkup stopping igb1
2021-07-29T00:39:28-04:00 inner-fw2.lan.thejeffcoats.net configd.py[84073]: [08bc2d5f-8eb8-4dbd-925b-4e2ef22476be] Linkup stopping lagg0
2021-07-29T00:39:28-04:00 inner-fw2.lan.thejeffcoats.net configd.py[84073]: [fa84fb2a-16f0-4b3e-b20c-ee3f3a789bd6] Linkup stopping lagg0_vlan20
2021-07-29T00:39:28-04:00 inner-fw2.lan.thejeffcoats.net configd.py[84073]: [1ce28a8d-8096-48bc-afc2-567dda5db8c2] trigger config changed event
2021-07-29T00:39:32-04:00 inner-fw2.lan.thejeffcoats.net configd.py[84073]: [742fcbb1-5fc2-422f-b61a-c90904ea3e33] Linkup starting igb0
2021-07-29T00:39:32-04:00 inner-fw2.lan.thejeffcoats.net configd.py[84073]: [d4daa58f-dbbb-45b6-b39e-f4dfecaa9e02] Linkup starting lagg0
2021-07-29T00:39:32-04:00 inner-fw2.lan.thejeffcoats.net configd.py[84073]: [b3a9ba7a-5fed-4cc8-8343-a2ac6ff3d9e8] New IPv4 on lagg0
2021-07-29T00:39:33-04:00 inner-fw2.lan.thejeffcoats.net configd.py[84073]: [04145dfb-5625-461b-b94d-ea82d39ea3fc] generate template OPNsense/Filter
2021-07-29T00:39:33-04:00 inner-fw2.lan.thejeffcoats.net configd.py[84073]: generate template container OPNsense/Filter
2021-07-29T00:39:33-04:00 inner-fw2.lan.thejeffcoats.net configd.py[84073]: [3b49d59a-fa71-4f63-b6e5-9ceb2e936307] refresh url table aliases
2021-07-29T00:39:33-04:00 inner-fw2.lan.thejeffcoats.net configd.py[84073]: [898420e5-6f67-441e-8b55-0710f9fad03a] Linkup starting lagg0_vlan20
2021-07-29T00:39:33-04:00 inner-fw2.lan.thejeffcoats.net configd.py[84073]: [a73be435-5bb7-4e6c-aea6-f009c7a96986] Linkup starting igb1
2021-07-29T00:39:33-04:00 inner-fw2.lan.thejeffcoats.net configd.py[84073]: message 3b49d59a-fa71-4f63-b6e5-9ceb2e936307 [filter.refresh_aliases] returned {"status": "ok"}
2021-07-29T00:39:35-04:00 inner-fw2.lan.thejeffcoats.net configd.py[84073]: [4dab0966-c2a9-4e31-b4df-971ef18750ad] Reloading filter
2021-07-29T00:39:35-04:00 inner-fw2.lan.thejeffcoats.net configd.py[84073]: [81acf0e3-927e-4ad0-a090-e19f66f43802] generate template OPNsense/Filter
2021-07-29T00:39:35-04:00 inner-fw2.lan.thejeffcoats.net configd.py[84073]: generate template container OPNsense/Filter
2021-07-29T00:39:35-04:00 inner-fw2.lan.thejeffcoats.net configd.py[84073]: [29824610-759d-45d3-ad3e-14289162d97b] refresh url table aliases
2021-07-29T00:39:35-04:00 inner-fw2.lan.thejeffcoats.net configd.py[84073]: message 29824610-759d-45d3-ad3e-14289162d97b [filter.refresh_aliases] returned {"status": "ok"}
2021-07-29T00:39:44-04:00 inner-fw2.lan.thejeffcoats.net configd.py[84073]: [d9cf2ad2-0fa5-41b8-aaab-09ad4c2df34f] Linkup stopping igb0
The link stopping igb0 is the last entry before the system powered back up.
Not sure if there's a real drive problem, smart status shows errors on the drive being called out (ada1) as having no problems. I ran a short test, it returned no errors. I do have spares that I can swap it out to eliminate this as the potential source of any problems. The fact that it loads 21.1 after without issue leads me to think it might not be the root cause or related.
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.1-RELEASE-p19-HBSD amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Silicon Motion based SSDs
Device Model: TS256GMSA370
Serial Number: F915720124
LU WWN Device Id: 5 7c3548 19c3583bc
Firmware Version: P1225CH1
User Capacity: 256,060,514,304 bytes [256 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
TRIM Command: Available, deterministic, zeroed
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Jul 29 02:00:13 2021 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x02) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x71) SMART execute Offline immediate.
No Auto Offline data collection support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0002) Does not save SMART data before
entering power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 1) minutes.
Conveyance self-test routine
recommended polling time: ( 1) minutes.
SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x0000 100 100 000 Old_age Offline - 0
5 Reallocated_Sector_Ct 0x0000 100 100 000 Old_age Offline - 0
9 Power_On_Hours 0x0000 100 100 000 Old_age Offline - 343
12 Power_Cycle_Count 0x0000 100 100 000 Old_age Offline - 56
160 Uncorrectable_Error_Cnt 0x0000 100 100 000 Old_age Offline - 0
161 Valid_Spare_Block_Cnt 0x0000 100 100 000 Old_age Offline - 155
163 Initial_Bad_Block_Count 0x0000 100 100 000 Old_age Offline - 10
164 Total_Erase_Count 0x0000 100 100 000 Old_age Offline - 411497
165 Max_Erase_Count 0x0000 100 100 000 Old_age Offline - 251
166 Min_Erase_Count 0x0000 100 100 000 Old_age Offline - 148
167 Average_Erase_Count 0x0000 100 100 000 Old_age Offline - 197
168 Max_Erase_Count_of_Spec 0x0000 100 100 000 Old_age Offline - 3000
169 Remaining_Lifetime_Perc 0x0000 100 100 000 Old_age Offline - 94
175 Program_Fail_Count_Chip 0x0000 100 100 000 Old_age Offline - 0
176 Erase_Fail_Count_Chip 0x0000 100 100 000 Old_age Offline - 0
177 Wear_Leveling_Count 0x0000 100 100 050 Old_age Offline - 2157
178 Runtime_Invalid_Blk_Cnt 0x0000 100 100 000 Old_age Offline - 0
181 Program_Fail_Cnt_Total 0x0000 100 100 000 Old_age Offline - 0
182 Erase_Fail_Count_Total 0x0000 100 100 000 Old_age Offline - 0
192 Power-Off_Retract_Count 0x0000 100 100 000 Old_age Offline - 11
194 Temperature_Celsius 0x0000 100 100 000 Old_age Offline - 60
195 Hardware_ECC_Recovered 0x0000 100 100 000 Old_age Offline - 0
196 Reallocated_Event_Count 0x0000 100 100 016 Old_age Offline - 0
197 Current_Pending_Sector 0x0000 100 100 000 Old_age Offline - 0
198 Offline_Uncorrectable 0x0000 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0000 100 100 050 Old_age Offline - 0
232 Available_Reservd_Space 0x0000 100 100 000 Old_age Offline - 100
241 Host_Writes_32MiB 0x0000 100 100 000 Old_age Offline - 303133
242 Host_Reads_32MiB 0x0000 100 100 000 Old_age Offline - 24735
245 TLC_Writes_32MiB 0x0000 100 100 000 Old_age Offline - 1645988
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 87 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
7 0 65535 Read_scanning was completed without error
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
ZFS shows now errors.
root@inner-fw2:~ # zpool status
pool: zroot
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada0p2 ONLINE 0 0 0
ada1p2 ONLINE 0 0 0
errors: No known data errors
At the end of all this I reinstalled 21.1, restored my config from backup, and was back online with no errors or issues.
Not sure this will help but may give some clues. I can test the upgrade again on this firewall at will if necessary to try to help capture this, it's the standby firewall in my HA setup.
Thank you!
Al
-
Please keep responses in this thread. I am starting to suspect that this is a (igb?) driver issue with the latest kernel code? We did not add any code related to LAGG or VLAN in any case... the only code updates related with VLAN are these of iflib, which is also used by igb driver.
If you replace the kernel with the old one does it still hang?
# opnsense-update -zkr 21.1.8
# opnsense-shell reboot
Cheers,
Franco
-
Tried that, here's the steps and results.
- Clean Install
- Assign interfaces igb4 to WAN and igb5 to LAN temporarily from CLI
- Assign LAN IP Address from CLI
- Connect to GUI, run wizard for minimal config.
- Create lagg0 from igb0 and igb1.
- Assign lagg0 to WAN interface.
- Create lagg1 from idb2 and igb3.
- Assign lagg1 to LAN interface.
- Set ipv6 to none on WAN interface.
- Create upstream gateway and assign to WAN interface.
- Reboot, just because.
- Run command to replace kernel with old one: opnsense-update -zkr 21.1.8
- Run command to reboot: opnsense-shell reboot
- Assign VLAN ID 20 to lagg0: Success
- Assign VLAN ID 30 to lagg0: Success
- Assign VLAN ID 40 to lagg0: Success
YAY! So that seemed to do the trick. No hangs and no disk errors with that downgrade of the kernel. Now I really am going to bed, it's 3:15 ;) Thanks and have a good day!
Al
-
Thanks for confirming. Now comes the hard part figuring out what change in the kernel actually causes this... *sigh*
Cheers,
Franco
-
I do have the same issue:
Update to 21.7, then the opnsense hangs on Configuring Vlan interfaces
Hardware:
Intel Celeron G3900 2-Core 2,80GHz 2MB
8 GB (1x 8GB) ECC DDR4 2666 RAM
Supermicro X11SSH-LN4F with a Onboard Quad LAN with Intel® Ethernet Controller I210-AT
If i select the Kernel.old image in the boot screen, the opnsense starts fine. But I have to do this on every startup/reboot.
-
Thanks for confirming. Now comes the hard part figuring out what change in the kernel actually causes this... *sigh*
Cheers,
Franco
If I can help in any way, I'm at your disposal. Unfortunately I'm not a developer so cannot help in the way I think you need. :(
-
Ok one more thing to try:
Add two tunables "hint.ahci.0.msi" and "hint.ahci.1.msi" to "0" and try booting the new kernel.
These can also be set from the loader prompt (3. escape to loader prompt)
set hint.ahci.0.msi=0
set hint.ahci.1.msi=0
boot
Cheers,
Franco
-
Ok one more thing to try:
Add two tunables "hint.ahci.0.msi" and "hint.ahci.1.msi" to "0" and try booting the new kernel.
These can also be set from the loader prompt (3. escape to loader prompt)
set hint.ahci.0.msi=0
set hint.ahci.1.msi=0
boot
Cheers,
Franco
I've applied them, how to I upgrade the kernel on the last install / downgrade I just ran through?
-
I rebooted and selected old kernel, and it's hung at "Configuring VLAN interfaces...". It looks like it successfully changes the name of vlan0_vlan20 to lagg0_vlan20, then hangs at changing vlan1 to lagg0_vlan30, and I start getting achich1 timeout errors and CAM status timeouts. Took a screenshot, will attach.
-
I already tried that, and it didnt work :(
Same error message and stuck on Vlan configuration
-
Not sure if it is related, but with Sensei it seems to be a problem using igb with netmap:
https://forum.opnsense.org/index.php?topic=24133.0
As soon as I use igb with netmap (emulated or native), my OPNsense becomes unresponsive.
-
I do have the same issue:
Update to 21.7, then the opnsense hangs on Configuring Vlan interfaces
Hardware:
Intel Celeron G3900 2-Core 2,80GHz 2MB
8 GB (1x 8GB) ECC DDR4 2666 RAM
Supermicro X11SSH-LN4F with a Onboard Quad LAN with Intel® Ethernet Controller I210-AT
If i select the Kernel.old image in the boot screen, the opnsense starts fine. But I have to do this on every startup/reboot.
Same problem here (and also a friend of mine) after upgrading to 21.7. Booting kernel.old works but not the new kernel.
CPU: Intel(R) Pentium(R) CPU G4560 @ 3.50GHz (3504.14-MHz K8-class CPU)
Quad Intel(R) PRO/1000 PCI-Express
VLANs on LAGG configured
My friend has a Qotom box. Not sure which model but with intel nics and VLANs on LAGG configured.
-
Do custom added tunables survive updates between point releases?
I've got my system back to 21.1.9_1, I want to try setting those tunables again. I think earlier this morning I added them to my primary firewall instead of the secondary I had been testing with, and want to try it again.
-
Tried the upgrade again.
- Restored back to 21.1.9_1
- Added the tunables, rebooted
- Upgraded to 21.7 via GUI
- Failed to boot, stuck at "Configuring VLAN interfaces". Had to power cycle.
- Booted, at loader selected option 3 and added the tunables, boot. System is stuck at "Configuring VLAN interfaces.".
Figured it was worth a shot when I didn't have keyboard rash on my face ;)
-
I found the time to retry with
- Export on 21.1.9_1 on the existing firewall
- Import into fresh 21.7 release intall (with ZFS) on the new hardware
Same procedure as described already, same result where it just hangs at configuring VLANs.
Thanks to everyone else who chimed in to help resolve this!
@franco
Would it still be of any value to send you the whole config (redacted) that I'm using? Or is it already certain, that the underlying issue is elsewhere?
Unsure what more I could contribute at this point.
-
This sounds concerning, I also have a 3558 based system with a LAGG using a chelsio 10G sfp+ nic and two "backup" igb interfaces.
Take-home message here from Franco is to wait? Is thi caused by the new kernel?
-
Those who are experiencing this problem:
Any chances that you can disable msix on the igb interfaces and see if that helps?
root@fw_i5:~ # sysctl -a | grep dev.igb | grep msix
dev.igb.3.iflib.disable_msix: 1
dev.igb.2.iflib.disable_msix: 1
dev.igb.1.iflib.disable_msix: 1
dev.igb.0.iflib.disable_msix: 1
root@fw_i5:~ #
You can set these from System -> Settings -> Tunables and you need a reboot.
-
I can give it a shot. If I set them in 21.1.9_1, will the tunables survive a reboot, or do I need to set them in the boot menu option 3 after the upgrade?
And I'm assuming there's a # for each interface, so on my 6-port qotom, there'd be dev.igb.[0-5].iflib.disable_msix:1?
Answered my own question. Currently the setting on one of my firewalls:
root@inner-fw2:~ # sysctl -a | grep dev.igb | grep msix
dev.igb.5.iflib.disable_msix: 0
dev.igb.4.iflib.disable_msix: 0
dev.igb.3.iflib.disable_msix: 0
dev.igb.2.iflib.disable_msix: 0
dev.igb.1.iflib.disable_msix: 0
dev.igb.0.iflib.disable_msix: 0
-
If you can set these from System -> Settings -> Tunables and they should survive a reboot.
-
Yep, knew they survived a reboot ( I wrote that wrong in my previous post), don't know (yet) if they survive an upgrade.
So, I set the tunables:
root@inner-fw2:~ # sysctl -a | grep dev.igb | grep msix
dev.igb.5.iflib.disable_msix: 1
dev.igb.4.iflib.disable_msix: 1
dev.igb.3.iflib.disable_msix: 1
dev.igb.2.iflib.disable_msix: 1
dev.igb.1.iflib.disable_msix: 1
dev.igb.0.iflib.disable_msix: 1
Rebooted, and tested a sync from inner-fw1, everything looked good.
Tried to update and the "update repository" step was just hanging (gave up after 5 minutes, rebooted again)
Tried to update again.
And now, I have an upgraded 21.7 system that is not hanging at the configuring VLANs step!!! (Yay! :) )
SSH Login banner:
Last login: Fri Jul 30 02:26:46 2021
----------------------------------------------
| Hello, this is OPNsense 21.7 | @@@@@@@@@@@@@@@
| | @@@@ @@@@
The tunables did indeed survive the upgrade:
root@inner-fw2:~ # sysctl -a | grep dev.igb | grep msix
dev.igb.5.iflib.disable_msix: 1
dev.igb.4.iflib.disable_msix: 1
dev.igb.3.iflib.disable_msix: 1
dev.igb.2.iflib.disable_msix: 1
dev.igb.1.iflib.disable_msix: 1
dev.igb.0.iflib.disable_msix: 1
And this:
root@inner-fw2:~ # opnsense-version
OPNsense 21.7 (amd64/OpenSSL)
I tested an HA Sync (Successful) and a failover (Successful) (still running on the secondary as I write this).
So, this is a driver issue for the igb devices? For sure going to set these on the primary HA firewall, probably in a day or so, gonna let this cook. I have a third non-HA firewall that has no VLAN settings (Not that it's a VLAN issue, but that seemed to have been the trigger), should I disable the dev.igb.#.iflib.disable_msix on that firewall as well?
I tried finding info on the that tunable, wasn't able to, can you explain what it is? Found it: Disables MSI-X interrupts for the device.
And, finally, THANK YOU to you, franco and anyone else who helped on this!
-
Here is a test kernel that takes out a FreeBSD patch as per Murat's suggestion:
# opnsense-update -zkr 21.7.r_6
# opnsense-shell reboot
If this one works without the tunables set that would be the kernel we can push out today without much delay.
Thanks,
Franco
-
Alrighty ;)
On the same system I've been using all along, standby firewall in HA setup:
Updated the kernel: opnsense-update -zkr 21.7.r_6
Rebooted: opnsense-shell reboot (tunables still in place).
Remove tunables.
Reboot via GUI.
System booted successfully! Not sure how to check the running kernel, opnsense-version still returns 21.7.
-
# uname -v
FreeBSD 12.1-RELEASE-p19-HBSD 79ea2ec061b(master) SMP
Alternatively
# opnsense-version kernel
21.7.r_6
But opnsense-version can only say which package is installed, not if the kernel is booted.
Cheers,
Franco
-
Not sure if it's the right way, in the GUI Firmware->updates panel, it shows this now:
Current Version: 21.7.r_6.
New Version: 21.7
And it's wanting to upgrade :)
-
That's correct, but you don't want the stock 21.7 kernel at the moment. ;)
uname -v is the best bet to verify.
-
That's correct, but you don't want the stock 21.7 kernel at the moment. ;)
uname -v is the best bet to verify.
yea, I figured it might not go well ;)
root@inner-fw2:~ # opnsense-version kernel
21.7.r_6
root@inner-fw2:~ # uname -v
FreeBSD 12.1-RELEASE-p19-HBSD 79ea2ec061b(master) SMP
-
Looks good. Did you also have the hang at VLAN with LAGG? It's been a bit confusing...
Cheers,
Franco
-
Looks good. Did you also have the hang at VLAN with LAGG? It's been a bit confusing...
Cheers,
Franco
Yes, without the tunables mb posted, or the kernel downgrade/upgrade you have given, this machine hangs at the configuring VLANs boot message.
-
Ok, perfect. Thanks a lot! :)
More feedback from others welcome. Now we have to wait and see if stability issues reported around igb are also addressed by this change.
Cheers,
Franco
-
Thanks much Franco, the effort is much appreciated!
I'm going to leave the primary VIPs in maintenance mode and let the updated secondary server stay online as the active firewall until the fix is officially released. Once that happens I'll use the as yet untouched primary firewall to test the upgrade from 21.1.9 -> 21.7fixed.
Have a great day!
Al
-
For reference the change was: https://github.com/opnsense/src/commit/79ea2ec061b
Cheers,
Franco
-
I've just tested the 21.7.r_6 version of kernel on a previously upgraded (in-place) custom build with J3160-ITX + I350-T4 and it booted correctly.
Thanks a lot!
-
I was having the AHCI timeouts/errors, and with the new test kernel they seem to be gone. Uptime is now over an hour or so. Sensei running fine with native netmap driver.
Thanks to all involved in getting this fixed!
-
Is it now safe to update? Do I need to do anything specific to get the right kernel? ex. either before or after the GUI update process.
-
Hi @blblblb, things are going in the right direction; however, I'd advise waiting for a while and tune in for more updates from the OPNsense team.
-
Hi @blblblb, things are going in the right direction; however, I'd advise waiting for a while and tune in for more updates from the OPNsense team.
Thank you! So, better to hold on to 21.1.8? I have the system in question operating through a lagg with vlans and 2 out of 3 interfaces there are ix based/igb, and they are the fallback ones in case the main 10g link falls apart. It's a critical point of that network and I don't mind dealing with manual kernel installs but I definitely need to confirm what works. I'm atm quite far from a system lookalike that uses the same HW so I can't just test and try my luck ;P
-
21.1.8 should be all ok; and expect updates from @franco or @adschellevis :)
-
Is the following OK to upgrade?
-
I’m not sure, unless they’ve replaced the package in the stock 21.7 repository, this would get you back the default 21.7 kernel with the bug still in it.
-
Wait till Monday. We did replace the previous "21.7" kernel but depending on your mirror it may not be there yet.
In any case new images with 21.7.1 will be out soon to avoid this problem.
Cheers,
Franco
-
Wait till Monday. We did replace the previous "21.7" kernel but depending on your mirror it may not be there yet.
In any case new images with 21.7.1 will be out soon to avoid this problem.
Cheers,
Franco
Hello Franco,
I've never paid much attention to the updrade screen until now. I'm assuming when the System -> Firmware -> Updates: New Version = 21.7.1 (currently shows 21.7), it will then be safe to update?
Thank you,
Al
-
Hi Al,
We did replace the 21.7 kernel and it should be safe to upgrade (although I can't guarantee that a third party mirror will not catch the old bad kernel). The default mirror will definitely have the corrected version.
The 21.7.1 kernel should follow next week and will likely be the same kernel as the replacement currently for 21.7 (which is also the same as 21.7.r_6). And then it will be super-safe to upgrade.
Cheers,
Franco
-
@franco can you link to the correct mirror with the replaced 21.7 with the fixes?
-
It's the "(default)" mirror setting.
Cheers,
Franco
-
Thanks Franco!
Just updated fine using the default repo. Not using VLANs but Sensei is also not affected by the igb anymore problem it seems.
No more hangs on my Proxmox KVM with pass-through igb interface.
:)
-
Thanks for the feedback. I will take this to FreeBSD next week so the changes can either be reworked or pulled from their stable branches.
Cheers,
Franco
-
Same for me, I've updated the fw1 device without any issue using the default repository. This machine has been untouched during all the prior testing. I failed services from fw2 -> fw1 without any problems.
Thank you, and have a great day!
Al
-
Using the default and dns-root repos all I can see is:
(https://i.imgur.com/RlPr1qd.png)
-
> Using the default and dns-root repos all I can see is:
Yes, correct.
Cheers,
Franco
-
> Using the default and dns-root repos all I can see is:
Yes, correct.
Cheers,
Franco
Should it reflect 21.7.1 for the kernel instead once the updated package is propagated?
-
Only after 21.7.1 was released. For now we replaced the 21.7 kernel for triage. Every mirror should have the good copy now.
Cheers,
Franco
-
Please disregard, read a few posts back and I'm good.
-
With the new kernel, just to confirm beyond any reasonable level of doubt, the tunables for MSI-X interrupts are NOT needed, right?
-
Yes, reinstall 21.7 kernel and check from shell after reboot:
# uname -v
FreeBSD 12.1-RELEASE-p19-HBSD f692db99f3c(stable/21.7) SMP
If you see this it's safe to remove the tunables and reboot again.
Cheers,
Franco
-
Can the updated 21.7 kernel be installed with:
opnsense-update -krf 21.7
I have currently booted with old_kernel and have not applied the msi-x tunables.
-
Can confirm:
# uname -v
FreeBSD 12.1-RELEASE-p19-HBSD f692db99f3c(stable/21.7) SMP
Is fully functional and without any issues so far, on a system with a Chelsio SFP+ adapter and ix interfaces ("new" igbe driver, Denverton built-in NICs).
I just updated through the UI after Franco confirmed the mirrors have the updated kernel package propagated.
Excellent work and thank you for all the help!
-
opnsense-update -krf 21.7
Yes that works. Note that -r 21.7 is the default so you only really need -kf
Cheers,
Franco
-
Anyone willing to deliver a CTRL+T on the VLAN configuration hang during boot? We may need more elaborate information still but don't yet know how to do that remotely.
Thanks,
Franco
-
Anyone willing to deliver a CTRL+T on the VLAN configuration hang during boot? We may need more elaborate information still but don't yet know how to do that remotely.
Thanks,
Franco
I don't mind, but now that my kernels are updated from repo, you'd have to tell me how to downgrade to reboot and catch it.
-
opnsense-update -krf 21.7
Yes that works. Note that -r 21.7 is the default so you only really need -kf
Cheers,
Franco
Thanks! Will try :)
-
Get a dmesg from the working system:
Go to System: Firmware: Reporter, click "report an issue". Copy contents of dmesg.boot text. No need to submit, just close the page.
bad kernel:
# opnsense-update -zkr 21.7_bad
# opnsense-shell reboot
Hit CTRL+T on the spot where it hangs and make a screenshot or photo of the debug information that appears.
Get a dmesg from the non-working system:
Boot from 21.7 image without any VLAN/LAGG configuration. Let it boot and log into web GUI. Same firmware reporter dmesg.boot text for reference. We really
Restore working kernel:
Boot from kernel.old, simply install update to 21.7.1 or:
# opnsense-update -fk
# opnsense-shell reboot
Please send the info to franco@opnsense.org
Thank you,
Franco
-
I hope there will be a new image out soon, because I explicitly don't want to upgrade but reinstall, because I want my systems to be on ZFS.
-
I'm not sure 21.7.1 is soon enough, but is our only next best option anyway like we said many times already. ;)
Cheers,
Franco