[SOLVED] Weird network behaviour

[crazywolf13]

Hi

I'm experiencing some weird network issues, a part of my homelab behind opnsense becomes incassible almost daily.
I'm trying to wrap my head around this for weeks.

Nearly daily everything in my VLAN 1 (so default network) behind opnsense becomes inacessible / behaving very weirdly, a reboot of opnsene or a reboot of any device between like switch/ISP router fixes every single problem magically.

Tough everything boils down to opnsense, as soon as opnsense becomes inaccessible to PING my network begins to behave weirdly.

This probably explains everthing the best:



- Ping is working to everything except opnsense. (most of the time ping is also not working and looks fully red on uptime-kuma.)
- OPNsense is reachable through IP:Port, however not through reverseproxy
- physical hosts are inaccessible through reverseproxy
- every single web-service running in vlan20 is fully accessible through the same reverseproxy

My current network layout:

- Physical Hosts: 10.50.20.x Subnet, VLAN 1
- VM/LXC: 10.10.20.x Subnet, VLAN 20

On my Switch I tagged each port for a physical host as VLAN 1 and 20 TAGGED

I dug through every log avaiable on opnsense but sadly have not yet found anything roughly matching with the timeframe opnsense/network becoming offline.

I'm really at the end of my knowledge on how to go on in this error/ how to troubleshoot, and really really hope for some help.

THANKS!

EDIT: Problems were also before upgrade to 24.7
EDIT: Fixed images

[cookiemonster]

Have you checked that the interface into OPN i.e. the trunk is carrying _only_ tagged frames?

Ideally but not mandatory is to move away from VLAN1 and replace with something else. I.e. you end up with ports on either VLAN 20 or VLAN 10 for instance. This is to avoid -if necessary- oddities between vendors as to the behaviour of their "native VLAN".
Also can you describe your OPN setup in this.

[crazywolf13]

Hi Cookiemonster, thanks for your reply!

What's the best view to see that?

I've wanted to do that as well, however due to the nature of proxmox, that runs on the same Port as the VM's / LXC's will run, I had issues in getting Proxmox itself work in a VLAN with only 1 LAN connection.

Also I'm not really sure if it has to do with the VLAN, I just noticed, that the devices in VLAN 1 are getting somewhat inaccessible.

To my opnsense node:
https://www.shuttle.eu/de/products/slim/dl30n
Intel N100
16 GB DDR5
2x 2.5GB Intel Ports

[cookiemonster]

On OPN, Interfaces > Assignments. There the interfaces should have only tagged devices.

Then on your managed switch, you have to define the trunk port that will carry only tagged traffic

And then you would setup your access ports with the vlan tags you want to be carried on the trunk.

[crazywolf13]

Thanks!

Yes that's exactly how I set it up.

Also there are no unassigned interfaces.

Any other ideas, how I could further troubleshoot these weird issues ?

[cookiemonster]

sounds like OPN side is fine then.

[crazywolf13]

Well what else could it be then?

Can't really imagine of the hosts themselves bening the problem here.

I imagined the switch, but could not get a single idea, even with the help of cisco support/forum.

[cookiemonster]

maybe expanding on the layout and setup would help. For instance you have virtualisation/hypervisor that can do vlans in play. Also some reverse proxy in play it seems. A schematic would help too but that's just me.

[crazywolf13]

Hi

Yeah you are right!

Sorry about that, the images embed di not work correctly due to me not using the correct RAW iamge url, just fixed it.

Edit: altough the uptime-kuma board shows they are partually online, they are inaccessible for a normal browser, during that time.

For the reverse proxy, it's run inside a LXC inside Proxmox with an IP inside VLAN 20, it's nginxreverseproxy.

Feel free to let me know what else I shall provide.

[crazywolf13]

So I've just further looked into this and even physically replaced the suspected broken switch.

After replacing the switch, the issue kep occuring, so I can now definitely say that opnsense is the problem.

Anyone got ideas on how to further troubleshoot here?

[crazywolf13]

Issue definitely keep occurring, I now have to physically reboot my opnsense around every 5 hours.

Most likely it's always the opnsense that becomes offline first, then my other Proxmox Nodes all become really sluggish, reachable/the other second not, login through proxmox webgui is then mostly not possible etc.

This seems really odd to me, where could I start to debug this?

Is that some package/ip conflict?
Hardware fault? (Opnsense has Intel NIC's and manufacturer labeled as opnsense good)
Config Error?
Problems with other Networking gear?
Problem with VLANs?

I can't seem to find any real trace here, not even any of the debug logs on opnsense tell me anything.
Also weirdly enough, all my web-applications, that are proxied through my reverseproxy are always online and fully accessible (except opnsense).

has anyone some idea where I could start?

[cookiemonster]

sorry, been away. But no ideas how to go about this one. Seems like a few moving parts there and some I don't  even dabble with i.e. LXC. And the only reverse proxies I use are on the router/firewall i.e. the edge. and not within a network, nor within a DMZ. No idea, sorry.

[Gauss23]

Maybe it’s something on the network. Do you use STP or similar? That explains things getting sluggish over time and not reacting anymore at some point in time. I would check in that direction.
Are you able to access the OPNsense via console, when it’s no longer reachable via network?

[imk82]

Hi crazywolf13,

just as a shot into the blue, maybe you can check this and have a look at your Bios settings:
https://forum.opnsense.org/index.php?topic=42240.0

Best regards
Robert

[crazywolf13]

Thanks imk82

That is fitting my exact scenario very well, for me it also occured after switching to a Intel N100 based Mini-PC, with Intel NIC's, also for me it surprisingly often occurs when there is high load on the LAN interface. Also funnily enough I also have Intel-i226-LM NIC's.

I'll update as soon as I could look at the bios.

Hi Gauss23
Thanks for the suggestion with SPanning-Tree-Protocol, I don't have anything particularly set up in this direction.
As I don't have a console cable, I can only guess here, but mostly the opnsense seems online, just really sluggish, and sometimes not on all interfaces.