Looking for some guidance on an issue I've experienced in both the pfsense and opnsense worlds. I was a long time user of pfsense and switched ~2 years ago when the feature parity I cared about became available and love it. However, I just experienced an issue that I've experienced in pfsense multiple times over the years which makes me think its FreeBSD related (this is the first time I've tried making this type of change in opnsense).
I seem to always run into an issues where traffic stops flowing through an interface when making changes to that interface configuration. Two specific examples with diagrams attached:
1.
As-is state: DMZ interface is configured for 192.168.1.1/24. Layer 2 adjacent client is 192.168.1.12/24.
Future: I change the mask to /29 on the firewall and change the mask and IP client to be within the /29. Post change the client can only ping the DMZ firewall interface, nothing behind it. Firewall can ping the client from the DMZ interface AND from any other interface on the firewall (Client can NOT ping any other interface on the firewall or any device behind those interfaces). Similar to the firewal I even created a floating/quick permit any any on all interfaces to make sure I'm not crazy.
2.
As-is state: One of the physical firewall interfaces has two sub-interfaces created (the physical "untagged" interface is not being used, just sub-interfaces 10 and 11).
Future: Per the diagram, I try adding a new sub-interface and all three basically stop working (similar behavior to #1). Nothing is changed on the switch or to the trunk. I've also tried removing one sub-interface and creating a new sub-interface with a different vlan and the behavior is basically the same. This is what I experienced in pfsense as well all the time. Only way I've solved it in the past is to rebuild the box which is why I feel it might be a FreeBSD related issue.
Anyone experience anything similar with fixes?
Can anyone tell me where in FreeBSD and/or opnsense config file is with this info so I can see what the GUI is applying to the config? Don't mind updating via CLI in the event the GUI is jumbling the data up.
1. I assume your DMZ interface is the only one you show vtnet6(opt5). If you want to make the network on that interface a /29, that's all you need to do on your interface settings on OPN, from the dropdown selector. I suggest strongly not to make changes directly, only via GUI. The settings are stored in /conf/config.xml
if after making the change then your client hanging from the switch can't ping anymore any other client on another interface, it suggest a problem with the switch configuration perhaps. Since you mention tags I presume the switch is a managed one, so look there and I would verify that the baeaviour is correct when the netmask is not /29 (presumably /24). That will validate your interface rules.
2. More information please. Managed switch, how are you configuring the trunk between OPN and switch.
In both cases you need to provide more of the basic information of the setups.
For #1, you need to adjust the DHCP range accordingly if you have a DHCP server on the interface.
Unplugging/replugging the client is a quick way to entice it to get a new IP.
If you manually set IP on the client, double-check that they match. Which OS?
#2 is just adding a new VLAN?
And that breaks the existing ones?
Thanks for the replies. After some digging I found the root of the problem but I don't know why it's happening or how to fix it. Suggestions would be appreciated.
The problem is related to FRR not removing the original IP address information for the interface and installing two connected routes for the interface in it's RIB (the original subnet and the new one). Below is the specific example I can replicate:
Current State:
Interface vtnet2(opt1) = 10.1.1.1/24
FRR RIB: Contains one entry for vtnet2
Connected route 10.1.1.0/24 via "Directly Attached" interface "vtnet2"
Future state:
I change the IP address and/or mask of vtnet2 to something different.
Interface vtnet2(opt1) = 10.10.10.1/29
FRR RIB: Contains two entries for vtnet2
Connected route 10.1.1.0/24 via "Directly Attached" interface "vtnet2"
Connected route 10.10.10.0/29 via "Directly Attached" interface "vtnet2"
FRR is not dropping the old IP address information and essentially blackholes traffic destined for 10.1.1.0/24.
I've tried rebooting, deleting vtnet2 and re-adding it (with FRR stopped or running), deleting vtnet2 then rebooting then re-adding, etc, etc. Only way FRR removes the subnet from the RIB is if I shutdown the interface. Even removing the IP address info (setting IPv4 Configuration Type to none) doesn't remove the route.
Got it solved.
I had a virtual IP enabled for the interfaces. Looks like FRR uses this information for route table population as well.
Thanks again for the input!