Messages

@My_Network - interesting, I didn't think of this way with these things you pointed out.

Let me try these out and see if this helps.

OPNsense is def different than any other firewall I've setup in the past; all it takes is one setting to break it.

In my current setup, my Sophos XG is an OSPF ABR, so figured the OPNsense would be able to be an ABR too.

I have another Sophos XG at my parent's place and at my sisters; I currently have two IPSEC tunnels to these locations over identical Sophos XG firewalls with a local NAS and solo ESXi 8.0U3 host backed to a 3 bay QNAP in RAID 5. I use the local NAS as a Veeam remote backup repository and both locations have a pair of DC's to extend the development domain.

Hopefully I'll still be able to propagate my Server vLAN to the OPNsense to funnel over the IPSEC tunnel. Only time shall tell.

Here is a question for people.

I stepped back and looked at this with some perspective.

What is different between the Sophos XG and the OPNsense, I asked myself.

Right now, all of the VLAN's are being "trunked" over from the Netgear M4300 to the OPNsense on the LAN port which is one of the VP6670's Intel X710 10GE CNA SFP+ ports. All of the vLAN's (1, 100, 200) - 1 being the native vlan, are allowed on the Netgear side.

On the OPNsense the port is assigned the LAN tag for the port, but vlan1, vlan100, and vlan200 are assigned in the 'Other Type' and then associated as sub-interfaces of LAN. They each have a x.x.x.1 assignment for the /24 subnet. The Netgear assignment is x.x.x.3 for the /24 subnet.

Questions for community:

1. Do I maybe need to define in the CLI on the OPNsense a mirror trunk config on the VLAN interface?
2. Do I maybe need to manually hardcode a MTU of 1500 in the CLI on all connecting parts, maybe the GUI is buggy?

I ask all of this cause, the Netgear does acquire the OSPF info from the OPNsense, but I cant ping the other side of the connect to x.x.x.3 from the OPNsense; so this to me leads me to think normal vlan traffic isn't flowing even with OSPF configured.

I do have an inbound and outbound ICMP rule right below the rules mentioned below, so in theory ping should work. My thinking is vlan traffic should flow 1st and then ICMP would work after that, so the ICMP rules are rule 3 & 4 respectfully.

I currently have an inbound and outbound rule for each zone (vlan 1, 100, and 200) on the OPNsense with the interface being the zone's "net" designation with a source "any" and destination "any" - I know this isn't the most secure but for testing and flow it was my thinking it would be good to start. This rule is at the top of the user-defined rules as I'm using the routing-enabled OSPF rules, so they are predefined automatically.

I'm also pondering if maybe the default deny is at play here.

anyways; going for a walk to think on this.

just curious if others have some thoughts... trying to think of this programmatically.

[Protectli V1211 (2) 2.5G ports with 8 GB's of RAM running OPNsense with only Suricata IPS on WAN interface]

I was really hoping this would just work for OSPF.

I'm planning in the future to acquire another VP6670 and do a inside and outside firewall deployment.

Protectli recently sent me a V1211 2 port vault to deploy either OPNsense or pFsense as an inline pass-thru IPS-only box that sits like this:

Internet ISP (Coxial)  ---> Arris DOC 3.1 2.5G modem ---> Protectli V1211 (2) 2.5G ports with 8 GB's of RAM running OPNsense with only Suricata IPS on WAN interface ---> OPNsense firewall on Protectli VP6670 ---> Netgear M4300 16X16F (Core) ---> Netgear M4300 16X16F (Distro) --> VMware ESXi 8.0U3 Datacenter (6 host) w/ vSAN (VMware vExpert licensing)


My future goal was this:

Internet ISP (Coxial)  ---> Arris DOC 3.1 2.5G modem ---> Protectli V1211 (2) 2.5G ports with 8 GB's of RAM running OPNsense with only Suricata IPS on WAN interface ---> OPNsense (outside) firewall on Protectli VP6670 ---> Mitrok 2.5G/QNAP 2.5G with dual 10G uplinks (DMZ switch for AT&T OSSIM or Security Onion Mirrored port for SPAN monitoring) ---> OPNsense (inside) firewall on Protectli VP6670 --->  Netgear M4300 16X16F (Core) ---> Netgear M4300 16X16F (Distro) --> VMware ESXi 8.0U3 Datacenter (6 host) w/ vSAN (VMware vExpert licensing)

Note: the links between inside and outside firewalls would be over a bonded 10GE backbone since the Protectli VP 6670's have Dual 10G SFP+ ports.

My primary interest in OPNsense was using the Suricata on the 1st line of defense Protectli v1211 IPS. Then on the Outside firewall use ZenArmor on the WAN interface and on all zoned edge services like IoT and other wireless setup. Then on the Inside firewall again use Suricata in IDS mode while using ZenArmor on all internal "trusted" vLAN's for ingress/egress traffic.

This would allow my edge to be protected by 3 layers of defense and two SIEM platforms. Meanwhile, Defender for Endpoint and Defender for Identity would protect the internal assets along with M365 and Azure security services protecting all assets via various mechanisms.

Should be noted in my VMware cluster I'm running a Server 2025 forest following a CIS Level 1 & Level 2 framework with an ESAE "red" forest, (2) Tier 4 PKI's, and a fully deployed vCenter with vROPS and vSAN deployed with a mix of linux, Windows 10/11 VM's. Microsoft licensing is via a yearly Visual Studio Pro subscription for all my server licensing.

I also pay for (3) licenses of ME5 for MDE, MDI, Azure Sentinel, and a fully functioning Azure DevOps with Azure PIM deployed and Yubikey used on nearly everything. I also make us of Purview for sensitivity labeling and DLP policies. 

But if I cant get OSPF to work correctly on the OPNsense then this makes me have to rethink this whole setup and pray pFsense can do this. But I realize pFsense and OPNsense both use the FRR plugin. So I may be screwed.

Really hoping someone has some ideas. 

I've spent the whole day reading all previous posts on this forum with regards to OSPF and I'm working to try all the things others have tried. I'll report back in a day or two if any of these workaround others tried works.

you are 100% right. I've only have OPNsense for about 2 weeks now. I was looking at the plugin version of 1.42, after looking at the running config I see its frr 8.

Asking the gents on the FRR slack if they have seen the latest version of frr 10 has fixed the ospf peering problems since frr 8.

I'm learning this new firewall slowly but surely.

Its very different from my Fortigate 100F and Sophos XG firewalls in form and function. It reminds me of a ASA back in the day.

[Here is the primary reason for this posting]

Hello all,

Introduction: I've been in IT for 15 years, but doing tinkering with networking since I was 8 years old in the 1990's. I'm a MCSE x4, CCNA, CCNP, VCAP5-DCA, VCP5-DCV/NV, VCP6-DCV/NV, VCP7-DCV, CompTIA: A+, N+, Sec+, Stor+, Linux+, CySA+, CASP+; AZ-103/104, AZ-305, AZ-500, AZ-700, MS-203, MS-100/101, MS-500, SC-200, SC-300, SC-400, SC-100, CISSP, CISM. I have an extensive 42U server rack for all my hobbies and toys. I admit to being a life learner that knows nothing and even with all that I know, I know that I still know nothing. I will happily admit to not knowing something. I'm hoping one of you can find an error in my configs that I'm missing for this problem. Thank you again for helping me. ^_^

The Problem:

I'm running into a problem with OPNsense 25.1 on a Protectli VP6670 firewall w/ 32 GB of DDR5 and a 1 TB NVMe SSD. This is a new deployment, less than 2 weeks old.

Some general network layout info (old):

Internet ISP (Coxial)  ---> Arris DOC 3.1 2.5G modem ---> Sophos XG Home (Qotom) ---> Netgear M4300 16X16F (Core) ---> Netgear M4300 16X16F (Distro) --> VMware ESXi 8.0U3 Datacenter (6 hosts) w/ vSAN (VMware vExpert licensing)
 
Starting facts:

OSPF works perfectly fine with Sophos XG Home for 5+ years.


Some general network layout info (new):

Internet ISP (Coxial)  ---> Arris DOC 3.1 2.5G modem ---> OPNsense firewall on VP6670 ---> Netgear M4300 16X16F (Core) ---> Netgear M4300 16X16F (Distro) --> VMware ESXi 8.0U3 Datacenter (6 host) w/ vSAN (VMware vExpert licensing)

Starting facts:

Gateways defined for all interfaces (System: Gateways: Configuration) to the IP address of all vLAN's in question to the downstream Netgear M4300 static IP (next hop)
Routes defined under System: Routes: Configuration
default OPNsense NAT
autoconfigured OSPF firewall rules.
Routing: OSPF: Interfaces for OSPF set to Broadcast for all 3 vLAN's, AAA is blank, Hello set to 10, Dead set to 40, retrans interval 5, restrans delay 1 (this is mirrored on Netgear, as shown below)
Area 0.0.0.0 defined only once in Routing: OSPF: Networks
Router ID is the native vlan's IP on OPNsense
Again, Hello Interval set to 10 on OPNsense and Netgear switch
Again, Dead Interval set to 40 on OPNsense and Netgear switch
I tried to set Netgear to IGMP (though it wasn't needed for Sophos XG) - but it made no change in OSPF relationship
I tried to set Netgear to RSTP (though it wasn't needed for Sophos XG) - but it made no change in OSPF relationship
MTU is hardcoded on the interfaces on the Netgear, and the vLAN's, and also on the interfaces on the OPNsense


Here is the primary reason for this posting, the Netgear M4300 cant establish an OSPF peering relationship (again, with the Sophos XG Home - it peers perfectly fine with no config changes on the Netgear M4300 16X16F - merely just an ethernet swap between firewalls):

(M4300-16X16F) #show ip ospf ne

Router ID       Priority IP Address      Neighbor    State              Dead
                                         Interface                      Time
--------------- -------- --------------- ----------- ------------------ ----
30.30.30.30     1        172.16.100.5    vlan 400    Full/BACKUP-DR     38
30.30.30.30     1        172.16.110.5    vlan 410    Full/BACKUP-DR     38
192.168.115.1   1        192.168.115.1   vlan 1      Init/BACKUP-DR     37
30.30.30.30     1        192.168.115.5   vlan 1      Full/BACKUP-DR     38
192.168.115.1   1        192.168.120.1   vlan 100    Ex Start/DR-OTHER  37
30.30.30.30     1        192.168.120.5   vlan 100    Ex Start/DR-OTHER  31
192.168.115.1   1        192.168.130.1   vlan 200    Ex Start/DR-OTHER  37
30.30.30.30     1        192.168.130.5   vlan 200    Loading/DR-OTHER   31

For comparison, here is the OSPF relationship with the Sophos XG Home (13.13.13.13 is the Sophos):

(M4300-16X16F) #show ip ospf ne

Router ID       Priority IP Address      Neighbor    State              Dead
                                         Interface                      Time
--------------- -------- --------------- ----------- ------------------ ----
30.30.30.30     1        172.16.100.5    vlan 400    Full/BACKUP-DR     37
30.30.30.30     1        172.16.110.5    vlan 410    Full/BACKUP-DR     37
13.13.13.13     1        192.168.115.1   vlan 1      Full/DR-OTHER      33
30.30.30.30     1        192.168.115.5   vlan 1      Full/DR-OTHER      37
13.13.13.13     1        192.168.120.1   vlan 100    Full/DR-OTHER      33
30.30.30.30     1        192.168.120.5   vlan 100    Full/DR-OTHER      30
13.13.13.13     1        192.168.130.1   vlan 200    Full/DR-OTHER      33
30.30.30.30     1        192.168.130.5   vlan 200    Full/DR-OTHER      30


Here is the config from within OPNsense from inside the GUI:


Current configuration:
!
frr version 8.5.6
frr defaults traditional
hostname base1.maxdomain.local
log syslog notifications
!
interface enc0
 ip ospf passive
exit
!
interface igc0
 ip ospf passive
exit
!
interface igc1
 ip ospf passive
exit
!
interface igc2
 ip ospf passive
exit
!
interface igc3
 ip ospf passive
exit
!
interface lo0
 ip ospf passive
exit
!
interface vlan01
 ip ospf dead-interval 40
 ip ospf network broadcast
exit
!
interface vlan02
 ip ospf dead-interval 40
 ip ospf network broadcast
exit
!
interface vlan03
 ip ospf dead-interval 40
 ip ospf network broadcast
exit
!
router ospf
 ospf router-id 192.168.115.1
 redistribute kernel
 redistribute connected
 redistribute static
 network 192.168.115.0/24 area 0.0.0.0
 network 192.168.120.0/24 area 0.0.0.0
 network 192.168.130.0/24 area 0.0.0.0
 area 0.0.0.0 range 192.168.115.0/24
 area 0.0.0.0 range 192.168.120.0/24
 area 0.0.0.0 range 192.168.130.0/24
 default-information originate metric 1
exit
!
end

For context, here is the config from the Netgear M4300 16X16F (all ports operate at 10GE):

This is the interface upstream into the OPNsense from the M4300:

interface 1/0/15
description 'Uplink to firewall'
mtu 1500
switchport mode trunk
switchport trunk allowed vlan 1,100,200
ip ospf area 0
exit

Here are the vLAN configs on Netgear:

interface vlan 1
description 'Native vLAN Network'
routing
ip address 192.168.115.3 255.255.255.0
ip ospf area 0
ip mtu 1500
exit



interface vlan 100
description 'LAB Server vLAN'
routing
ip address 192.168.120.3 255.255.255.0
ip ospf area 0
ip mtu 1500
exit



interface vlan 200
description 'Lab Desktop vLAN'
routing
ip address 192.168.130.3 255.255.255.0
ip ospf area 0
ip mtu 1500
exit

Here is the OSPF config on the Netgear:

router ospf
router-id 1.1.1.1
no 1583compatibility
network 192.168.115.0 0.0.0.255 area 0
network 192.168.116.0 0.0.0.255 area 0
network 192.168.120.0 0.0.0.255 area 0
network 192.168.130.0 0.0.0.255 area 0
network 192.168.190.0 0.0.0.255 area 1
network 172.16.100.0 0.0.0.255 area 0
network 172.16.110.0 0.0.0.255 area 0
default-metric 2
default-information originate always metric 11
redistribute connected subnets
exit

Here are the OSPF timer settings on the Netgear:

(M4300-16X16F) #show ip ospf interface br

                                               Hello Dead  Retrax        LSA
             Admin                Router       Int.  Int.  Int.   Tranx  Ack
Interface    Mode     Area ID     Prior. Cost  Val.  Val.  Val.   Delay  Intval
------------ -------- ----------- ------ ----- ----- ----- ------ ------ ------
vlan 1       Enable   0           1      10    10    40    5      1      1
vlan 100     Enable   0           1      10    10    40    5      1      1
vlan 200     Enable   0           1      10    10    40    5      1      1


If anyone can provide insight into this problem, it would be helpful.

Thank you in advance. ^_^

roger roger, i'll create a new posting then. Thank you kindly Cedrik.

@cedrik - doing that it trails off the screen inside of putty, how do I make it scrollable in the console?

Should be noted, the OPNsense is running on a Protectli VP6670: https://protectli.com/product/vp6670/ with 32 GB of DDR5.

I've been switching between P2P and broadcast. Also strangely the OSPF interfaces have a hello set at 10, but it doesn't show in the running config. I've been switching between enabling different STP on my Netgear M4300 16X16F, and also enabling igmp. But prior to the OPNsense I didn't need igmp/STP for my previous Sophos XG Home firewall.

Below is the FFR config, I'm using default OPNsense NAT, and I'm using the autoconfigured OSPF firewall rules.

Right now my downstream Netgear M4300 16X16F is stuck in an INIT/DR-Backup state for native vlan, and for the other vlans they are stuck in EXSTART/DROther.

MTU is hardcoded end to end with 1500.

Here is the config from within OPNsense from inside the
GUI.




Current configuration:
!
frr version 8.5.6
frr defaults traditional
hostname base1.maxdomain.local
log syslog notifications
!
interface enc0
 ip ospf passive
exit
!
interface igc0
 ip ospf passive
exit
!
interface igc1
 ip ospf passive
exit
!
interface igc2
 ip ospf passive
exit
!
interface igc3
 ip ospf passive
exit
!
interface lo0
 ip ospf passive
exit
!
interface vlan01
 ip ospf dead-interval 40
 ip ospf network broadcast
exit
!
interface vlan02
 ip ospf dead-interval 40
 ip ospf network broadcast
exit
!
interface vlan03
 ip ospf dead-interval 40
 ip ospf network broadcast
exit
!
router ospf
 ospf router-id 192.168.115.1
 redistribute kernel
 redistribute connected
 redistribute static
 network 192.168.115.0/24 area 0.0.0.0
 network 192.168.120.0/24 area 0.0.0.0
 network 192.168.130.0/24 area 0.0.0.0
 area 0.0.0.0 range 192.168.115.0/24
 area 0.0.0.0 range 192.168.120.0/24
 area 0.0.0.0 range 192.168.130.0/24
 default-information originate metric 1
exit
!
end

For context, here is the config from the Netgear M4300 16X16F (all ports operate at 10GE):

interface 1/0/15
description 'Uplink to firewall'
mtu 1500
switchport mode trunk
switchport trunk allowed vlan 1,100,200
ip ospf area 0
exit

interface vlan 1
description 'Native vLAN Network'
routing
ip address 192.168.115.3 255.255.255.0
ip ospf area 0
ip mtu 1500
exit



interface vlan 100
description 'LAB Server vLAN'
routing
ip address 192.168.120.3 255.255.255.0
ip ospf area 0
ip mtu 1500
exit



interface vlan 200
description 'Lab Desktop vLAN'
routing
ip address 192.168.130.3 255.255.255.0
ip ospf area 0
ip mtu 1500
exit

router ospf
router-id 1.1.1.1
no 1583compatibility
network 192.168.115.0 0.0.0.255 area 0
network 192.168.116.0 0.0.0.255 area 0
network 192.168.120.0 0.0.0.255 area 0
network 192.168.130.0 0.0.0.255 area 0
network 192.168.190.0 0.0.0.255 area 1
network 172.16.100.0 0.0.0.255 area 0
network 172.16.110.0 0.0.0.255 area 0
default-metric 2
default-information originate always metric 11
redistribute connected subnets
exit

Here is what "Show IP OSPF Neighbor" looks like on the Netgear M4300 (192.168.115.1 is the vlan 1 IP on the OPNsense):
30.30.30.30 is another Netgear M4300 12X12F (as you can see OSPF is fully up to that switch) in series below my 16X16F. My 16X is my Core switch, my 12X is my distro layer, below that and out of scope is another 16X for a VMware ESXi 8.0U3 datacenter with 6 ESXi hosts. I'm a VMware vExpert, hence free Enterprise Plus licensing.

(M4300-16X16F) #show ip ospf ne

Router ID       Priority IP Address      Neighbor    State              Dead
                                         Interface                      Time
--------------- -------- --------------- ----------- ------------------ ----
30.30.30.30     1        172.16.100.5    vlan 400    Full/BACKUP-DR     38
30.30.30.30     1        172.16.110.5    vlan 410    Full/BACKUP-DR     38
192.168.115.1   1        192.168.115.1   vlan 1      Init/BACKUP-DR     37
30.30.30.30     1        192.168.115.5   vlan 1      Full/BACKUP-DR     38
192.168.115.1   1        192.168.120.1   vlan 100    Ex Start/DR-OTHER  37
30.30.30.30     1        192.168.120.5   vlan 100    Ex Start/DR-OTHER  31
192.168.115.1   1        192.168.130.1   vlan 200    Ex Start/DR-OTHER  37
30.30.30.30     1        192.168.130.5   vlan 200    Loading/DR-OTHER   31

Not sure if it's correlated but since upgrading to 25.1 FRR doesn't establish OSPF relationships.

My downstream switches are stuck in an INIT or EXSTART stance and won't form a connection.

[FRR website]

Hello,

New to OPNsense.

I've been running into OPSF neighbor relationship problems for over a week and been hammering at this right to get this to work with a Netgear and Cisco switch topology.

I've been a CCNA since 2005, and nearly all of my networking at home is based on the Cisco-CLI.

I presently have a Sophos XG firewall as my edge connected to a Netgear M4300 switch that is dishing out OSFP for layer 3. Right now its working correctly and has been for the past 5+ years.

I recently purchased a Protectli VP6630 firewall and deployed OPNsense on the vault.

Right now, I can not for the death of me get the OPNsense to form a OSPF neighbor relationship with the Netgear M4300. I know its not the switch cause it forms correctly to the Sophos XG firewall.

My M4300 on all of the interfaces sent from the OPNsense show up, but they settle on 'Init/BACKUP-DR' for the native vLAN and 'Ex Start/DR-OTHER' on the other vlans.

I've set the MTU on the OPNsense side to 1500 and also forced a MTU of 1500 on the Netgear side via CLI. The Netgear is presenting the vLAN's to the OPNsense via "switchport mode trunk | switchport trunk allow vlan 1, 200, 300 | ip mtu 1500 | ip ospf area 0".

The Hello intervals are 10, an dead intervals are 40 on both ends. The interface for OSPF is set to broadcast, and i've tried all of the AAA types, and just using none for AAA right now.

I'm using the auto-deployed OSPF rules, but I did trying to disable the auto rules and manually create a OSPF Multicast, UDP, IGMP Multicast (all 3 of them as inbound/bound) so 6 rules in total for all OSPF-enable interfaces on the OPNsense.

I'm honestly stumped right now and not sure why the FRR is not working.

If anyone has had this problem and knows how to get around it, please let me know.

Note: I tried to disable the firewall under the Advanced setting as I found in a OPNsense forum article, but it didn't fix the problem under the latest build.

I noticed on the FRR website they manage a Github, the latest stable build of FRR is 10.2.1 but the plugin the OPNsense can fetch is 1.42 from March 4th 2017; is there any reason the OPNsense is using an 8 year old plugin for routing?