multivlan, multiwan strange behavior

[mitchskis]

I recently attempted to lab a router-on-a-stick scenario.

Switch port 10, untagged vlan10, WAN/DHCP
Switch port 11, untagged vlan11, down
opnsense, tagged em1_vlan10, opt1
opnsense, tagged em1_vlan11, opt2
opnsense, untagged em0, lan

In this scenario, an internal packet capture on opt1 showed unbelievable amount of arp traffic on opt1 before an IP address was even assigned. Perhaps this is more than just undesirable. I've verified that the switch is not generating any traffic.

This seems like something worth troubleshooting. Thoughts?

[mimugmail]

Just post the capture

[Oxygen61]

Switch port 10, untagged vlan10, WAN/DHCP
Switch port 11, untagged vlan11, down

untagged VLAN10 - access Port - PVID 10 - check?
untagged VLAN11 - access Port - PVID 11 - check?

Your Uplink to OPNsense needs to be a Trunk-Port, which transports Tagged Traffic for VLAN 10 and 11

opnsense, tagged em1_vlan10, opt1
opnsense, tagged em1_vlan11, opt2

Okay, so em1 is your physical Interface carrier of the VLAN interfaces.
Remember to not assign em1 in any way, since there is no need for any other Interface except Tagged VLAN Interfaces.

opnsense, untagged em0, lan

Why did you assign em0 in the first place and why is it untagged? untagged for which vlan?
If you want to do Inter-VLAN-Routing with a Router-on-a-stick setup and you want both em0 and em1 to carry tagged traffic through many vlan Interfaces you will then need to bind them together by enabling LACP.

For the Firewall do the following:
1. Enable LACP with em0 and em1
2. create VLAN 10 and 11
3. Assign VLAN Interface 10 on the Lagg0 Interface group
4. Assign VLAN Interface 11 on the Lagg0 Interface group
5. Give both of them IP Addresses and subnets and configure them like you would configure "real" interfaces.
DHCP / NAT /Rules and so on.....
6. DO NOT make the mistake to assign physical interfaces just by themselve and let them send traffic if you have already configured VLAN interfaces on this physical interface, since this will end up in VLAN mismatch errors.

For the Switch do the following:
1. Enable Link Aggregation/LACP on two of your uplink ports and make them as "Active Dynamic LACP Bundle"
2. Create VLAN 10 and 11 on your Switch and name them something cool
3. Assign both VLANs to your LACP group Ports and make this Port-group a "Trunk-port" and let it forward your Tagged VLAN 10 and 11 traffic.
4. Do NOT Tagg the Default vlan 1. This VLAN stays untagged on every Switchport not configured already.
5. decide which client-PC or whatever should reside on which vlan and make these Switchports untagged VLAN 10 or VLAN 11 whatever VLAN 10 and VLAN 11 means to you.
6. Connect your LACP Bundle Switchports with your em0 and em1 LACP Bundle Lagg Group at your firewall and let them loadbalance all the incoming traffic.
7. Connect the rest of your PC's and Clients to the Switchports configured als untagged.

Finish.

Best regards,
Oxy