I upgraded vom 24.7.7 to 24.7.10_2 yesterday evening and let it run, since our Internet connection was a bit slow yesterday evening.
This Morning my familiy complained about broken connectivity. Tried to login to my OpnSense installation, but failed. Finally I could login over VPN / Mobile.
I could not figure out what the source of the Problem was. Symptoms: No untagged VLAN Traffic over the LAN Interface (igc0), however all tagged traffic worked like a charm over the same interface. No Config change by the Update nor on the Switch on the other Side. It is definitively not a Firewall issue, since there was no traffic arriving on the LAN Interface.
Since I did not know how to fix, I reverted the installation to 24.7.9 and the Kernel to 24.7.8. Again everything works as expected.
Interfaces that do not have mixed traffic (Tagged and Untagged) worked flawless.
I have some more info on this.
Tagged Traffic on LAN Interface -> WAN worked (ie. my Solar Converters on VLAN reached Internet)
Untagged Traffic on LAN Interface -> WAN did not work (ie. Browser on LAN did not reach Internet)
Untagged Traffic on LAN Interface -> Tagged LAN worked (ie. Browser reached Shelly on restricted Net)
Untagged Traffic on LAN Interface -> LAN untagged did not work (ie. Browser to NAS did not work)
Tagged Traffic on LAN Interface -> LAN untagged did not work (ie. Shelly Actions to NAS Docker Image)
Physical Link was UP all the time. DHCP did sometimes work, and sometimes not (Probably the ACK was not received) according to the Managed Switch that is directly connected to OpenSense on LAN Interface. NTP could not be synced.
I have the same problem.
Can you ping the router? Not me, only briefly during the boot phase. Do you use wiregard ?
Ping got lost , when one of this logs where displayed
>>> Invoking start script 'freebsd'
setup igb0_vlan5
setup igb1 [egress only]
Starting flowd_aggregate.
setup igb0_vlan5, only for one of the six Vlans. No setup igb0_vlan4 while booting
No I could not Ping anything at all. Luckily I have had a VPN that I could use to reach the GUI.
As of now, at least 3 People reported this Problem when using 24.7.10 and having an Interface that has tagged and untagged VLAN Traffic. Reverting to 24.7.8 Kernel with 24.7.9 Base did solve the Problem.
I'm no expert in Kernels, but maybe the latest Bugfix introduced this Problem?
24.7.9 works for me too.
I have problem to get access to my UniFi ports, and a really need to lock my playstation5 to one of them for the speed, now it jumps between the the ports and when the playstation get connected to the port one floor up the internet gets very slow. Can somebody help with this, this problem started arter the new system update
If this was all working before then there's very likely a new issue here, but
just in case-
I have seen multiple posts from senior members warning against mixing tagged and untagged traffic in FreeBSD. Is this one of the symptoms of that?
This is one of the very first threads I came across when first setting up my home network: https://forum.opnsense.org/index.php?topic=36530.0
There's a warning in the OPNsense docs against mixing: https://docs.opnsense.org/manual/how-tos/vlan_and_lagg.html
To better understand the reasons and potential issues, I asked ChatGPT and am providing the conversation below for reference.
If this is indeed a common issue then I'm surprised OPNsense doesn't have a built-in warning in the application, as most people probably only go to the docs when they get stuck and are likely to miss it. There's nothing that prevents people from setting things up like this and not realizing that there could be routing problems.
------
Prompt:
QuoteWe are discussing networking in FreeBSD, particularly firewall systems based on pf such as OPNsense and pfSense.
There are some opinions that tagged and untagged traffic should not be mixed on the same interface. Why? What is the technical explanation and what problems does it cause?
Answer by ChatGPT 4o mini, 17 December 2024:
Quote
In FreeBSD, particularly with firewall systems like pfSense and OPNsense, the issue of mixing tagged and untagged traffic on the same interface arises when dealing with VLANs (Virtual Local Area Networks) and standard Ethernet traffic.
Explanation of Tagged and Untagged Traffic
- Tagged traffic refers to Ethernet frames that include a VLAN tag (also known as a 802.1Q tag). This tag carries information about which VLAN the frame belongs to, allowing a network switch or other network devices to properly segregate traffic into different logical networks (VLANs) over the same physical interface.
- Untagged traffic, on the other hand, does not include any VLAN tag. It is treated as being part of the default VLAN (usually VLAN 1 on most switches), which is typically used for normal, unsegmented network traffic.
Why Should Tagged and Untagged Traffic Not Be Mixed?
The technical issue with mixing tagged and untagged traffic on the same interface involves how network devices process Ethernet frames. Here are some of the key reasons:
1. Ambiguity in Frame Processing
- If both tagged and untagged traffic is sent through the same interface, it can cause ambiguity in how frames are handled. When an Ethernet frame arrives at the interface, the device needs to know if the frame should be processed as part of a specific VLAN (tagged) or as part of the default network (untagged). This confusion can lead to misrouting or mishandling of traffic.
- For example, if the interface receives a frame without a VLAN tag (untagged) but is also configured to handle tagged traffic for several VLANs, it might not be clear which VLAN the untagged frame should belong to, potentially causing traffic to be misdirected.
2. Security Risks and Traffic Leakage
- Mixing tagged and untagged traffic on the same interface can lead to security vulnerabilities. If traffic that is not meant to be part of a VLAN is accidentally tagged or handled as part of another VLAN, sensitive traffic could be exposed to devices that should not have access to it.
- Additionally, malicious users or misconfigured devices might be able to inject tagged traffic into an untagged VLAN, leading to VLAN hopping, where traffic is inadvertently allowed to access networks it shouldn't.
3. Operational Complexity
- Having both tagged and untagged traffic on the same interface complicates network administration. It can be more difficult to troubleshoot network issues because traffic might behave unpredictably. Misconfigurations are harder to detect when the network doesn't clearly separate VLAN-tagged and untagged traffic.
- Network configurations such as DHCP (Dynamic Host Configuration Protocol) can also become more complicated. If an interface serves both tagged and untagged traffic, it may be difficult for DHCP servers to assign IP addresses properly to hosts in different VLANs, leading to IP address conflicts or other networking issues.
4. Performance Considerations
- Mixing tagged and untagged traffic can also lead to inefficiencies in processing, especially in larger networks. The network stack must continuously check if traffic is tagged, and this overhead can become problematic, particularly in high-traffic environments. Furthermore, switches and routers may handle this situation less efficiently when performing VLAN tagging or untagging operations.
Specific Problems that Can Occur
- Misrouting of Traffic: A frame may be incorrectly assigned to the wrong VLAN, leading to communication issues and potentially leaking traffic to unintended segments of the network.
- Broadcast Traffic: Broadcast traffic, typically sent untagged, could be inadvertently forwarded to the wrong VLAN if the interface is improperly configured.
- DHCP Problems: If a device requests an IP address on an untagged VLAN but the interface is processing multiple VLANs, the DHCP process may fail to assign the correct IP, as the VLAN tag information may not be properly applied or recognized.
- VLAN Hopping: In some cases, a misconfigured interface could allow tagged traffic from one VLAN to enter another VLAN, bypassing segmentation and causing unauthorized access.
Best Practices
To avoid these issues, it is generally recommended to separate tagged and untagged traffic on different interfaces. This can be done by:
- Using different physical interfaces for tagged and untagged traffic.
- Using VLAN-aware interfaces where tagged traffic is explicitly separated from untagged traffic through proper VLAN configuration.
- Ensuring that the network device's configuration clearly distinguishes between traffic that should be tagged and that which should remain untagged.
In summary, mixing tagged and untagged traffic on the same interface can lead to ambiguity, security risks, and operational complications, making it a practice that should generally be avoided in network design. Separating tagged and untagged traffic allows for clearer traffic handling, better security, and simpler network management.
Quote from: tops4u on December 09, 2024, 08:04:59 AMI upgraded vom 24.7.7 to 24.7.10_2 yesterday evening and let it run, since our Internet connection was a bit slow yesterday evening.
This Morning my familiy complained about broken connectivity. Tried to login to my OpnSense installation, but failed. Finally I could login over VPN / Mobile.
I could not figure out what the source of the Problem was. Symptoms: No untagged VLAN Traffic over the LAN Interface (igc0), however all tagged traffic worked like a charm over the same interface. No Config change by the Update nor on the Switch on the other Side. It is definitively not a Firewall issue, since there was no traffic arriving on the LAN Interface.
Since I did not know how to fix, I reverted the installation to 24.7.9 and the Kernel to 24.7.8. Again everything works as expected.
Interfaces that do not have mixed traffic (Tagged and Untagged) worked flawless.
@tops4u have you learned anything more about this condition? I have a similar problem, and also fixed with the same reversion you used. But I'd like to know when it's safe to attempt update again.
I now think that it's not just only a tag/untag problem. My first attempt to update from 7.9 to 7.10_2 ended with the LAN interface no longer being accessible. Well, I had one untagged and 4 tagged interfaces on same hardware interface. Now I have reworked this and there is no more untagged. I now have VLAN 2,3,5,6 and 10 on my LAN interfaces. Everything runs with 24.7.9_1-amd64. I have now carried out the update. After the restart, exactly the same. During the boot phase you can briefly ping the LAN (VLAN 10). After approx. 10-15 pings the connection is lost and the router can no longer be reached via the LAN interface IP (previously untagged, now VLAN 10).
I use Wiregard and IPsec ?!?
If you can take the downtime, I'd try to characterize the firewall configuration (make sure it boots into the expected configuration), traffic flow (particularly actual entrance interfaces), and ARP table - those were/are my problems. I filter pretty explicitly and have my own default deny rules (per-interface); I also set rule descriptions in the form of "[interface]: [action] [protocol] from [source] to [destination]" (e.g. "TRUST: Pass DNS from TRUST net to Public_v4"), so I can see exactly what rules are being hit in the logs. I can't say if it'll help characterize your issue, but the actual flow path through the firewall may surprise you. Or not. But what you see (or don't see) may be a data point. As for ARP, verify that the entries (MAC and interface) are as you expect, on the firewall and on your connected equipment. A basic element, but unless you're a super-sniffer, ARP issues can be hard to spot.
Good luck.
I have no deny rule on the Lan interface. There are only the standard 2 "allow all" rules ( IPV4 , IPV6 ).
And why does it work for X years with the rules only with the update not. Additionally others have similar problems ?
Quote from: vigeland on January 04, 2025, 05:42:08 PMI have no deny rule on the Lan interface. There are only the standard 2 "allow all" rules ( IPV4 , IPV6 ).
And why does it work for X years with the rules only with the update not. Additionally others have similar problems ?
I've wondered for some time about OPNSense and if it is reliable to work with in all environments.
I'm using it primarily as a VM firewall. In that it seems to be 'not great but works'.
One culprit is now using an OPNSense VM with a hypervisor bridge which has a physical interface with multiple vlan assigned.
The hypervisor sets the PVID egress as untagged on the bridge and tagged for the vlans, which is as it is. However, the tagged vlan are visible as untagged inside OPNSense VM. And that's that. No docu pointing out what to do or not to do.
When using multiple vlan-id on a single bridge the only solution seems to be to create a bridge per vlan, which doesn't really make sense, but works.
Quote from: JL on January 06, 2025, 09:41:50 PMQuote from: vigeland on January 04, 2025, 05:42:08 PMI have no deny rule on the Lan interface. There are only the standard 2 "allow all" rules ( IPV4 , IPV6 ).
And why does it work for X years with the rules only with the update not. Additionally others have similar problems ?
I've wondered for some time about OPNSense and if it is reliable to work with in all environments.
I'm using it primarily as a VM firewall. In that it seems to be 'not great but works'.
One culprit is now using an OPNSense VM with a hypervisor bridge which has a physical interface with multiple vlan assigned.
The hypervisor sets the PVID egress as untagged on the bridge and tagged for the vlans, which is as it is. However, the tagged vlan are visible as untagged inside OPNSense VM. And that's that. No docu pointing out what to do or not to do.
When using multiple vlan-id on a single bridge the only solution seems to be to create a bridge per vlan, which doesn't really make sense, but works.
Quote from: vigeland on January 04, 2025, 05:42:08 PMI have no deny rule on the Lan interface. There are only the standard 2 "allow all" rules ( IPV4 , IPV6 ).
And why does it work for X years with the rules only with the update not. Additionally others have similar problems ?
Ermm. https://docs.opnsense.org/manual/how-tos/vlan_and_lagg.html
The manual for VLANs takes you to this URL which explains not to mix tagged and untagged traffic.
From this thread what I gather is that we have users who have not heeded the advice and have mixed traffic anyways. So far it has "worked" but not after an update. What might have happened is that the update has exposed the misconfiguration only.
p.s. I was one of those users for some time and it was also not giving me any trouble. Until I realised that one day (like with this update - if that is what it is) it was bound to catch me. So I fixed it, by reconfiguring to the recommended setup.
So the ideal course of action is to correct any remaining installations with mixed tagged and untagged traffic in an interface used with OPN, and then move to diagnose any other problems and thinking OPN is "reliable to work with in all environments."
For info:
In my particular case, it was not due to tagged and untagged mix. The problem persisted, even though I only use tagged interfaces. During the update something was changed in wiregard, without wiregard everything works. If I switch back to tagged and untagged, everything also works, just without Wiregard.
I suspect that Wirecard handles the route differently than before.
Fair enough. A different scenario.
Meanwhile there is 24.7.12 out... did anybody try?
Unfortunately the problem still exists in the current version. a rollback to version 24.7.9_1 works. However, that is not a solution either.
Quote from: tsense on January 20, 2025, 09:59:19 AMUnfortunately the problem still exists in the current version. a rollback to version 24.7.9_1 works. However, that is not a solution either.
And repeating...
Quote from: cookiemonster on January 07, 2025, 11:59:54 AMQuote from: JL on January 06, 2025, 09:41:50 PMQuote from: vigeland on January 04, 2025, 05:42:08 PMI have no deny rule on the Lan interface. There are only the standard 2 "allow all" rules ( IPV4 , IPV6 ).
And why does it work for X years with the rules only with the update not. Additionally others have similar problems ?
I've wondered for some time about OPNSense and if it is reliable to work with in all environments.
I'm using it primarily as a VM firewall. In that it seems to be 'not great but works'.
One culprit is now using an OPNSense VM with a hypervisor bridge which has a physical interface with multiple vlan assigned.
The hypervisor sets the PVID egress as untagged on the bridge and tagged for the vlans, which is as it is. However, the tagged vlan are visible as untagged inside OPNSense VM. And that's that. No docu pointing out what to do or not to do.
When using multiple vlan-id on a single bridge the only solution seems to be to create a bridge per vlan, which doesn't really make sense, but works.
Quote from: vigeland on January 04, 2025, 05:42:08 PMI have no deny rule on the Lan interface. There are only the standard 2 "allow all" rules ( IPV4 , IPV6 ).
And why does it work for X years with the rules only with the update not. Additionally others have similar problems ?
Ermm. https://docs.opnsense.org/manual/how-tos/vlan_and_lagg.html
The manual for VLANs takes you to this URL which explains not to mix tagged and untagged traffic.
From this thread what I gather is that we have users who have not heeded the advice and have mixed traffic anyways. So far it has "worked" but not after an update. What might have happened is that the update has exposed the misconfiguration only.
p.s. I was one of those users for some time and it was also not giving me any trouble. Until I realised that one day (like with this update - if that is what it is) it was bound to catch me. So I fixed it, by reconfiguring to the recommended setup.
So the ideal course of action is to correct any remaining installations with mixed tagged and untagged traffic in an interface used with OPN, and then move to diagnose any other problems and thinking OPN is "reliable to work with in all environments."
The TLD. Mixing tagged and untagged has never been a supported configuration. Some setups will expose it in form of problems, others won't.
Expecting new versions to make it go away is an exercise in futility.
Correct your setup.
I don't have any mixed vlan traffic in my setup. But I also noticed that I no longer have internet connectivity when I update to a newer version. My setup is a CARP HA cluster with two machines. The backup machine has internet connectivity. However, the master node is not. When I roll back to version 24.7.9 it works without any problems
Currently running OPNsense 24.7.12 (amd64) at Mon Jan 20 11:08:24 CET 2025
Fetching changelog information, please wait... fetch: https://pkg.opnsense.org/FreeBSD:14:amd64/24.7/sets/changelog.txz: Permission denied
Updating OPNsense repository catalogue...
pkg: https://pkg.opnsense.org/FreeBSD:14:amd64/24.7/latest/meta.txz: Network is unreachable
repository OPNsense has no meta file, using default settings
pkg: https://pkg.opnsense.org/FreeBSD:14:amd64/24.7/latest/packagesite.pkg: Network is unreachable
pkg: https://pkg.opnsense.org/FreeBSD:14:amd64/24.7/latest/packagesite.txz: Network is unreachable
Unable to update repository OPNsense
Error updating repositories!
Checking integrity... done (0 conflicting)
Your packages are up to date.
Quote from: tsense on January 20, 2025, 11:13:41 AMI don't have any mixed vlan traffic in my setup. But I also noticed that I no longer have internet connectivity when I update to a newer version. My setup is a CARP HA cluster with two machines. The backup machine has internet connectivity. However, the master node is not. When I roll back to version 24.7.9 it works without any problems
Currently running OPNsense 24.7.12 (amd64) at Mon Jan 20 11:08:24 CET 2025
Fetching changelog information, please wait... fetch: https://pkg.opnsense.org/FreeBSD:14:amd64/24.7/sets/changelog.txz: Permission denied
Updating OPNsense repository catalogue...
pkg: https://pkg.opnsense.org/FreeBSD:14:amd64/24.7/latest/meta.txz: Network is unreachable
repository OPNsense has no meta file, using default settings
pkg: https://pkg.opnsense.org/FreeBSD:14:amd64/24.7/latest/packagesite.pkg: Network is unreachable
pkg: https://pkg.opnsense.org/FreeBSD:14:amd64/24.7/latest/packagesite.txz: Network is unreachable
Unable to update repository OPNsense
Error updating repositories!
Checking integrity... done (0 conflicting)
Your packages are up to date.
So why are you posting on this thread that is/was about tagged/untagged traffic?
I saw that there were several problems starting with the subsequent versions 24.7.9. I have a similar problem in that I no longer have internet connectivity... Although the WAN traffic is not tagged. Internet access works permanently on the slave node. So it can't be a misconfiguration. Do you have an idea what I can check?
Quote from: tsense on January 20, 2025, 12:02:20 PMI saw that there were several problems starting with the subsequent versions 24.7.9. I have a similar problem in that I no longer have internet connectivity... Although the WAN traffic is not tagged. Internet access works permanently on the slave node. So it can't be a misconfiguration. Do you have an idea what I can check?
Instead of latching to any thread with a reported problem, do a search and for your symptoms in case it has been posted already. If nothing found, create your own new thread with your setup, hardware, current version, what worked before, what changed and what is now not working. Add your diagnostics so far.
Try to avoid problem statements lacking andy technical detail. "internet is not working" is an example of what not to do as it just delays the time to resolve whatever problem exists.
I have now opened an issue on Github to get a fix or a statement on this Problem: GitHub (https://github.com/opnsense/core/issues/8231)
Until then I will currently not upgrade my System.
After it was recommended in GitHub that having tagged and untagged Traffic on an Interface is not recommended However I tried various things by tagging upstream Traffic from the Switch that was untagged, moving to another physical interface with the tagged traffic... etc. However no improvement could be observed.
So I assume there is something else that causes the Problems. Maybe due to the age of my configuration and subsequent migrations of the configuration even from other HW.
Some things I noted, some I already fixed (hopefully) others not yet.
- My old VLAN Names are staring with the interface name ie. igc0_vlan01 however at least 24.7.x enforces names start with vlan0...
- All my Interfaces physical or VLAN Identifier are are opt<x> except two WAN and LAN - not sure if this is a problem (currently LAN has all tagged and untagged traffic).
Next I'll try to figure if this is a Kernal or a Distribution issue. However I'll fix the VLAN Names beforehand.
This thread started with an unclear problem and a setup with mixed tagged and untagged traffic on the same interface. You were advised to rectify that as a general starting point.
Now the latest post is unclear to me what the problem is and whether you have set up tagged only on the trunk.
@cookiemonster
I'll try to explain my setup. As noted it has a certain age and the Hardware has been exchanged and the configuration copied over. So it might have "age" issues.
I have a Cable Modem connected to the WAN interface in transparent mode, I have a DMZ with a Webserver and there is a LAN Interface for the internal Network. Over time I wanted to segment my internal Network and started creating VLANs for this reason. Some on WLAN AP (Ubiquity directly issuing the needed Tags) some Port Based on a Netgear Switch. All are routed to OPNSense for Firewall rules defining wich traffic may pass to what other networks. Tagging is enforced and checked on the Switch.
As such the "trusted" Traffic remained untagged on the LAN Interface, while all other (more specialized) Traffic get a VLAN (for example IoT Traffic that is not allowed Internet but Access to one IP on the LAN where IOBroker IoT Integration is running).
This all ran fine up until 24.7.10 where the trusted untagged LAN Traffic ceased to work correctly.
So this week I thought I'll try to clean up and migrate the until now untagged Traffic to a VLAN. But for some unknown reason it behaved identical to the update before when I used mixed traffic on the interface. So I suspect some other Problem with my config.
The Problem is clearly on the IP Layer as some Clients were able to get a DHCP IP but then were not pingable from Opnsense. As far as I understood there is no egress on this network from Opensense anymore.
As noted above I have observed some issues for example with vlan naming where older vlans obivously violate naming conventions - however I don't know if this is a problem. I was thinking to check if it is a kernel or opnsense core related issue next. As not many people reported similar problems I assume it is bound to my setup/config.
PS: Config File available on request
The new naming scheme applies only to newly created VLANs, existing ones work fine with old names. I have that, too, so this should be out of the question.
Although it is advisable to avoid mixing tagged and untagged traffic on the same interface, it usually works fine, too. It seems to be depending on network hardware, because some hardware has idiosyncrasies with that.
I have mixed traffic, too, probably for the same reasons as you: Ubiquiti equipment could handle tagged-VLANs-only, but adopting a new device would become a nightmare with this kind of setup.
That being said, I have not seen any problems with OPNsense 24.7.10 and above, so I would assume that there is no principal new problem.
It could be driver or hardware-dependent, as 24.7.10 had a new kernel (at least after a hotfix) or a configuration issue.
Exactly. @top4u as I refer you back to my earlier post #17 on this thread https://forum.opnsense.org/index.php?msg=226441 .
TLD: mixed traffic works until it doesn't, due to as meyergru says a variety of reasons. Usually hardware and configuration & the combination. Hence it is not advised to be done. You were told the same thing on github.
If however you say you want to move to the not mixed scenario and that is not working for you, then that is something that you could be advised on.
You then need to explain that new setup in detail and explain the problem. Don't expect anyone to go back to the beginning and figure out what the "same behaviour" is. Treat it as a brand new setup and problem statement, as that is what that is.
Thanks for your fast reply. I have probably a similar HW as you. N100 from Aliexpress equiped with 4 Intel I226-V Ports. I have disabled all HW Support in Opensense for the NICs.
@cookiemonster: Yes I'll try to cleanup my config first then I'll try to go to a new Version and see how it behaves. Maybe I'll first try it out on my old HW.