[Question/Bug?] Different MAC-addresses per VLAN on one NIC.

[mweesenaar]

Hi all,

I am dumping my question here, since it seems to be the right location... I suppose. I have tried searching this forum for my question, but could not find it. So if there is any similar question - please let me know!

First my setup: I got OPNsense (21.1) on a dedicated machine with one on-board ethernet interface and I got a PCI-card with 4 ethernet ports: Broadcom Inc NetXtreme BCM5719 Gigabit Ethernet PCIe. (according to `pciconf -l -v`).

I got glass fiber from Telfort (daughter organization of the Dutch ISP KPN), via a fiber to ethernet-unit in one of the ports of the Broadcom. For getting both IPTV and Internet working, I need to setup two VLANs (34 for internet and 4 for IPTV) on this physical network interface. Both require a DHCP-client to get their IP address. So far, so good.
However, using the simple setup as described above, resulted in getting an IP for IPTV but not for internet. Search results on this topic lead me to the solution that both VLAN interfaces require to have a different MAC address. The original modem/router showed that the IPTV-connection (VLAN) has a MAC address which is a +1 of the internet-connection (VLAN). Here comes my problem. When I do that using the WebUI (interfaces -> IPTV -> field MAC address), I notice that both VLANs get the new MAC address - where I expected that only the changed interface would have an updated MAC address. Can someone explain me why this occurs and if this is OPNsense-expected behaviour?

I was able to fix this problem, in my situation, by going to the shell of OPNsense and running ifconfig bge0_vlan4 ether 00:00:00:00:01 (fake MAC address here...) This forces the interface to use the provided MAC address; which I expect when setting this on the WebUI. However - then the following problem is that after a reboot, these settings are lost - since I just executed them on command line. So another question from that perspective for me is: How would I be able to persist these changes?

I hope my situation and questions are clear (enough) for being able to follow my thoughts and challenges.
Thanks for reading and I hope one can share its thoughts with me :-)

~Mikki

[mweesenaar]

-sorry for the bump-

[elcocoloco]

What happens if you set a fake mac on both interfaces

[mweesenaar]

Hello elcocoloco,

If I do that in the UI or on console, depending on which I set first, will get an IP and the other won't. That is how this ISP works(/seem to work)... The original modem/router does exactly the same: it has 1 physical NIC for internet, but it has 2 MAC addresses, one for internet and one for IPTV.

[Maurice]

This is indeed how it's currently implemented. If you configure a spoofed MAC address for a physical interface or VLAN, it is applied to the physical interface and all of its VLANs. I agree this is not ideal and doesn't cover all use cases.

Could you test this patch:

opnsense-patch 55f54f8

This should allow setting individual MAC addresses for VLANs on the same physical interface.

Cheers

Maurice

[mweesenaar]

Hello Maurice,

Thanks for your reply and your effort!
Since I work from home, and I require an OK from my userbase (aka my wife), I will try to apply the patch on my machine as soon as possible; preferably before end of this weekend.

Would you maybe also be able to point me to some documentation or anything related to the possibility to 'permanently' configure interface-related stuff (e.g. promisc mode) on OS-level, rather than from the UI? Permanently meaning, in a file, that is not overwritten before or after a reboot or upgrade Some sort of include file...?

I hope to quickly get feedback for you!
Mikki

[mweesenaar]

Hello Maurice,

Bit later than expected, however I do have feedback :-)
After some searching where to find your patch and where to apply it to, I was able to do so, but it did not yet give me the expected result. After some more trying, rebooting, failing, digging, Googling and so on, I found out that I seem to be required to set `promisc` on the vlan interface too.

So, in fact, your patch works perfectly for the MAC-address change; however I am now still a bit stuck for the promisc part. Is there already a way I could follow to make that work, too?

Thanks for your effort so far, and hope to hear from you!
Mikki

[Maurice]

Thanks for the feedback Mikki, interesting observation. Promiscuous mode indeed seems to be a known requirement to make this work.
(I only tested that the MAC addresses were applied correctly, not whether traffic actually passes.)

I'm not sure what approach would be best here. Automatically enable promiscuous mode if multiple MAC addresses are configured on the same physical interface? Or add an option to manually enable promiscuous mode? This should probably be discussed in the PR: https://github.com/opnsense/core/pull/4774

Cheers

Maurice

[mweesenaar]

Hello Maurice,

I would say that to allow to set promiscuous mode manually would be best, since it might introduce unexpected behaviour if it is set automatically, as Ad also stated.
I am not entirely sure how this exactly works, but thanks to a friend of mine, we were able to figure out that setting promiscuous actually fixed it. Strangely enough this worked somewhat easier when I had pfSense installed. So I am not entirely sure how the OPNsense code base differs from pfSense on this particular topic.

If more feedback, investigation or further case description from my side is helpful, please let me know!

Thanks for your effort in helping me!
Mikki

[mweesenaar]

Hello Maurice,

Can I be of help to get the pull request more in shape to get it merged?

Thanks in advance,
Mikki

[Maurice]

Hello Mikki,

Unfortunately, the promiscuous mode requirement makes this much more complex than expected. Also see my latest comment on the pull request. I think we first need to find consensus if and how to go forward with this. Suggestions welcome.

Cheers

Maurice

[Patrick M. Hausen]

There's a bug in upstream in bnxt(4) that leads to the "promiscuous" requirement, or VLANs won't work at all. I'm experiencing the same on TrueNAS. Really annoying.

https://jira.ixsystems.com/browse/NAS-109330
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236983

Kind regards,
Patrick

[mweesenaar]

Thanks both for replying and getting this topic some momentum again.
For me it is quite a thingy since I am glad I got it working now, but only one last step is required to completely solve it for me. I understand that there is some discussion required where to best fit the button and/or some other topics - but until then, OPNsense does not work (sufficiently) for me...

Maybe approaching from another course, to at least fix it 'on my machine': is there a possibility for me to persist the promisc mode, so that after a reboot, my internet actually works again .. without having to figure out what I needed to do to get it going again? And for now I am totally fine if that would mean that I cannot change my VLAN/NIC config(s).

Hope to hear from you!
Mikki

[Patrick M. Hausen]

You could create a file named /etc/rc.local with the ifconfig commands in there.
If I am not completely mistaken, OPNsense still contains the hooks to read and execute that file.

[lordjakoby]

Want to say thank you both for this discussion. I spent this weekend troubleshooting my home network eventually realizing that the lan sub interfaces shared the physical NICs/parent mac address once I saw the same mac address for each gateway in each vlan's mac-address table. This was causing strange issues like allowing me to initially ssh into a machine and then disconnecting me 5-8 seconds later with cli saying "Operation timed out.", and fw logs saying blocked ssh via default deny rule after it had already correctly hit the allow rules setup for the vlan interface. The lan networks were setup for allow any/any for troubleshooting and was perplexed at how the rules could work, then my ssh traffic no longer match the allow and start hitting the default deny.

Wanted to consolidate the steps mentioned above as I had a hard time finding anything useful till this lovely post.

Steps that fixed my router on a stick via a single physical port opnsense implementation. Downlink is a cisco trunk port

- initial setup found here: https://www.youtube.com/watch?v=dv13d6rfQPI
- enabled ssh access via: https://techexpert.tips/opnsense/opnsense-remote-access-ssh/
(i had disabled ssh initially, but wasn't easily able to get my admin non-root user to work so temporarily setup root ssh.)

- ran: opnsense-patch 55f54f8
output:
root@eri-hq-opnsense:~ # opnsense-patch 55f54f8
Fetched 55f54f8 via https://github.com/opnsense/core
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|From 55f54f878e88fcd67d462f7b78b151067f48296a Mon Sep 17 00:00:00 2001
|From: Maurice Walker <maurice@walker.earth>
|Date: Wed, 3 Mar 2021 22:22:37 +0100
|Subject: [PATCH] Fine-grained MAC address spoofing for VLANs
|
|- If a spoofed MAC address is configured for a VLAN, apply it to the VLAN only, not the parent.
|- If a spoofed MAC address is configured for a parent interface, restore any spoofed VLAN MAC addresses.
|---
| src/etc/inc/interfaces.inc | 13 ++++++++-----
| 1 file changed, 8 insertions(+), 5 deletions(-)
|
|diff --git a/src/etc/inc/interfaces.inc b/src/etc/inc/interfaces.inc
|index b7567e958e..6f6d08ba4f 100644
|--- a/src/etc/inc/interfaces.inc
|+++ b/src/etc/inc/interfaces.inc
--------------------------
Patching file etc/inc/interfaces.inc using Plan A...
Hunk #1 succeeded at 2399 (offset -2 lines).
done
All patches have been applied successfully.  Have a nice day.
#####################################

- ran these commands in cli, will remain until reboot.

ifconfig igb0 promisc; ifconfig igb0_vlan2 promisc; ifconfig igb0_vlan3 promisc; ifconfig igb0_vlan4 promisc; ifconfig igb0_vlan5 promisc; ifconfig igb0_vlan6 promisc;


For a reboot persistent config, I originally added the commands to /etc/rc.local but for w/e reason it wasn't working for me after reboot.

What did work however was to add the commands to the rc file, located at /usr/local/etc/rc
between the last "exit 0" line and after "/usr/local/etc/rc.subr.d/livemode" see below live example.

I found it was personally easier to scp the file to my machine (vi default in opnsense) > edit with nano > delete the opnsense /usr/local/etc/rc  and re-upload the rc file.

#copy from opnsense
scp root@10.11.254.10:/usr/local/etc/rc rc

#copy back to opnsense
scp rc root@10.11.254.10:/usr/local/etc/rc

-----------------------> Example of: /usr/local/etc/rc <---------------------------

/usr/local/etc/rc.subr.d/livemode

#CUSTOM

#Router on a stick requires promiscous mode enable on all participating #interfaces. Including parent.
ifconfig igb0 promisc; ifconfig igb0_vlan2 promisc; ifconfig igb0_vlan3 promisc; ifconfig igb0_vlan4 promisc; ifconfig igb0_vlan5 promisc; ifconfig igb0_vlan6 promisc;

exit 0
root@eri-hq-opnsense:~ # cat /usr/local/etc/rc

----------------------------------> Example end <----------------------------------

- changed sub vlan interfaces to unique mac addresses in the gui.

- confirmed each gateways new mac address was only showing in the associated vlans mac address table.

#########################################
Previous:
eri-hq-sw1#sh mac add dynamic int g1/0/1
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
   2    4062.3114.0cf4    DYNAMIC     Gi1/0/1
   3    4062.3114.0cf4    DYNAMIC     Gi1/0/1
   4    4062.3114.0cf4    DYNAMIC     Gi1/0/1
   5    4062.3114.0cf4    DYNAMIC     Gi1/0/1
   6    4062.3114.0cf4    DYNAMIC     Gi1/0/1
   7    4062.3114.0cf4    DYNAMIC     Gi1/0/1

After fix:
eri-hq-sw1#sh mac add int g1/0/1
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
   2    4062.3114.0cc2    DYNAMIC     Gi1/0/1
   3    4062.3114.0cc3    DYNAMIC     Gi1/0/1
   4    4062.3114.0cc4    DYNAMIC     Gi1/0/1
   5    4062.3114.0cc5    DYNAMIC     Gi1/0/1
   6    4062.3114.0cc6    DYNAMIC     Gi1/0/1
   7    4062.3114.0cf4    DYNAMIC     Gi1/0/1