Messages

Second Update, and I hope this finally solves the issue.


A Google search lead me to this thread. Checked my settings, and I found that I had configured Zenarmor to use the native nmap driver. Changed it to emulated namp driver, and all tests which were failing are passing now. Fingers crossed it stays that way.

I hope this helps anyone else who might be running into the same problem.

As a comment, your additional tests changed the machine but not the NIC which was the more likely culprit, if ZenArmor were not intruding.

I did change the NIC(s) when I changed the machine. em0 is onboard, and igb0 was another m.2 NIC I had as a spare. That eliminates the NIC or any hardware as the root cause.

I'll move this thread to the Zenarmour section and see if folks have encountered this problem before.

Thanks for the assistance and inputs.

[none of them fixed the issue]

@passeri / @pfry thanks for your inputs.

Here's what I tried, and none of them fixed the issue.

• Opnsense on a different mini pc. (Lenovo m910 with the same Intel(R) I219-LM onboard NIC as the M900)
• Static ARP on both sides
• MTU of 1300 on both sides
• Restarting the WAN interface on the Sophos
• Static IP and DHCP IP for the Sophos WAN interface
• Was using KEA DCHP on Opnsense, tried using ISC DHCP

Observations:
netstat -i on the Opnsense box shows outpackets incrementing. inpackets stays at 0
The Sophos works just fine when connected to a Ubiquity ER-X, but not with OPNSense. I'll test with pfsense sometime this weekend to see if that fixes the problem.


Update: I think I have isolated the problem to ZenArmour which is enabled on the OPNSense LAN interface. The issues I described go away if I stop Zenarmour and disrupt the OPNSense LAN connection. Tested again with ZenArmour running and connectivity is not restored until I restart the OPNSense LAN Interface.

[or issue the ifconfig em0 down | up command]

I did not imply I thought it was the Sophos box. The switch was from Lenovo M900 with its LAN ports to Ubiquiti ER-X, was it not? If not, what exactly are you swapping please? A labelled network diagram may be helpful

I swapped the OPNSense box with the ER-X for testing. OPNSense - fail, ER-X - Pass. The Sophos device and the unmanaged switch are unchanged. Diagram attached.

The em0 interface is up when I check via console, but does not respond to ping unless I power off the box and power it back up; or issue the ifconfig em0 down | up command

I need to completely power down the OPNSense box after a reboot, or a LAN cable change; before the OPNSense box starts passing traffic on the LAN interface. The interface is up, but does not do anything.

The bold parts of the statements are in conflict. What statement is both true and complete please?

My apologies, this statement is true and complete

"The em0 interface is up when I check via console, but does not respond to ping unless I power off the box and power it back up; or issue the ifconfig em0 down | up command"

Why, when you have also changed the hardware?

Since the Sophos + ER-X combo does not exhibit the same issues as Sophos + OPNSense; I came to the conclusion that the issue is not on the Sophos Box. And this was validated again by putting a unmanaged switch between the devices. By doing this, I ensured that there was no disruption to the Sophos WAN interface while the ER-X / OPNSense box were unplugged / rebooted.

Your problem appears to be that after a reboot of the Opnsense box you need to reboot it, or the LAN interface, a second time before it will respond on em0. Is that correct? I ask because your description is a little ambiguous.

I need to completely power down the OPNSense box after a reboot, or a LAN cable change; before the OPNSense box starts passing traffic on the LAN interface. The interface is up, but does not do anything.

The behaviour of em0 contrasts with igb0 which continues to operate normally.

That is correct, and both NICs are based on Intel Chipset. Only difference is that em0 is onboard, while igb0 is in a m.2 slot. I will also check the BIOS settings to make sure there's nothing in there causing the issue.

Realtek aside, I know of no other cases of similar behaviour under Opnsense, certainly not on several different boxes which I have used. Your own case shows that Opnsense has no difficulty on igb0.

Correct again.

I would be looking at the NIC. Perhaps a workaround could be to add a script that issues ifconfig commands to cycle em0 after startup completes. I have not thought about how to do that.

The em0 NIC is onboard, I could try swapping boxes and see how that goes. Will share an update after I do further testing with a different box.

Update: I replaced the OPNSense box with a spare Ubiquiti ER-X with exactly the same IP address combination and the same topology.

I did the following tests, and the connection was restored as soon as the ER-X device LAN interface was available.

• Unplug ER-X LAN interface
• Unplug Sophos WAN interface
• Reboot ER-X

I beleive this is definitely an issue with OPNSense.

[|]

Hardware: Lenovo M900 mini PC 16 GB RAM | LAN interface- em0- Intel(R) I219-LM SPT-H(2) | WAN interface - igb0 - Intel(R) I210 (Copper)
OPNsense version: 25.1.7_4-amd64 running on bare metal

Topology: Internet --> OPNsense --> unmanaged switch --> Sophos XG135 running SFOS 21.0.1  (Static IP) --> LAN devices


Issue:

Hi folks
The LAN interface is not reachable from any of the LAN devices, or the Sophos box whenever there is an interruption on the OPNSense LAN interface. It could be a reboot after a OPNSense software upgrade, or the OPNSense LAN cable being unpatched and repatched again. The em0 interface is up when I check via console, but does not respond to ping unless I power off the box and power it back up; or issue the ifconfig em0 down | up command. The WAN interface operates normally when this happens.
 
I tested by connecting the Sophos box directly, and using a unmanaged switch; but the result is the same. MTU is set to 1500 on both sides, and the BIOS on the Lenovo Box is the most recent one.

This issue has persisted for quite some time, will appreciate some tips on how to resolve this issue.

[Intel AMT works only with Intel 'LM' series NICs]

*Update in case someone else has a similar requirement.*

It turns out that the Lenovo M900 Tiny PC I am using is capable of Intel AMT, which provides remote manageability at the BIOS level. After enabling the feature in BIOS, I am able to manage the device using opensource software called Mesh Commander.

The caveat is that Intel AMT works only with Intel 'LM' series NICs which is usually the onobard NIC on these little corporate PCs.

Thanks for your responses everyone.

I was mostly commenting on the picture shown...

If we talk USB to serial with USB being plugged into OPNsense and the serial end somewhere else ti view the OPNsense

Two USB-Serial cables connected to each other, so USB at either end, and it works well. Console becomes available
after the login message appears on screen. Would it be possible to get console before the login screen appears so I can view the boot up messages for troubleshooting.

I meant USB working at the OS level before it fully boots up. The Opnsense image for serial install provides output over serial while reading off a USB storage device? That would mean both serial and USB are recognised by the installer?

Would the serial image be able to provide console output to serial during boot up after I complete the full installation? Or is it limited to the install process only?

If this will not work, I'll have to install a physical serial port. Bit the problem is that the spare port on the rear panel is taken up by the second NIC and I'll have to rig an extension cable and install the serial port outside the case.

I have attached a photo of the com port header can someone tell me what type of connector it is?

I meant USB working at the OS level before it fully boots up. The Opnsense image for serial install provides output over serial while reading off a USB storage device? That would mean both serial and USB are recognised by the installer?

Would the serial image be able to provide console output to serial during boot up after I complete the full installation? Or is it limited to the install process only?

Well, using a USB based serial console needs OS support, so no wonder you get output only once it is booted. Your system's BIOS as well as the boot loader do not support USB serial devices as console, so there is no way to access these phases of the boot process.

You would need a builtin serial port and BIOS support to see anything before the OS is up and running.

Pardon my ignorance, but USB boot and install from USB storage is possible; and I guess that means USB is enabled at power up? And I believe Serial being bi-directional it doesnt matter which end (USB or RS-232) is plugged into the opnsense box?

System - Settings - Administration - Console: Primary console

Already set to Serial Console and "Use USB-based serial ports" is ticked. Do I need to change any other settings?

Thanks everyone, I bought the USB to serial cable and am able to get console access to the Opnsense box. However, there's no console output until the OS fully boots up.

The link below seems to indicate that serial can be active during boot. I'm using a VGA image, will switching to the serial image help to get console access during boot?

https://serverfault.com/questions/918972/serial-over-usb-during-boot

Thank you