OSPF not running

[Layer8]

Hello everyone,

we would like to setup a OSPF network in our VMware Environment. Because these are our first steps with opnsense and ospf/frr, we need some help please.

Problem 1:

All Sensense are VMs with one vmx0-NIC and VLAN interfaces. We get the following error in the debug log of frr in every of the three opnsense routers:

2022-02-09T16:36:48   Error   ospfd   [EC 100663299] setsockopt_so_recvbuf: fd 13: SO_RCVBUF set to 2097152 (requested 8388608)

All we found is this thread: https://forum.opnsense.org/index.php?topic=23187.0

Problem 2:

Not sure if problem 1 is a complete show stopper, but we also need some help to get our setup running.

This is our network:

[CORESENSE] .1 <- 10.90.10.0/24 VLAN 910 -> .2 [TRANSFERSENSE] .1 <- 10.90.11.0/24 .2 VLAN911 -> [PROJECT1SENSE]

CORESENSE has many VLANs attached, also WAN with Internet-Access and a default route.
TRANSFERSENSE is only to transfer the traffic between CORSESENSE and PROJECT1SENSE.
PROJECT1SENSE has some Project VLANs configured for different types of clients. These VLANs are configured like 10.101.0.0/24 - 10.101.9/24.

There is more than one PROJECT*SENS. We also have a PROJECT2SENSE filewall and so one attached to TRANSFERSENSE, but with local networks like 10.101.10.0/24 - 10.101.19.0/24 for example. All attached networks to the PROJECT firewalls can be summarized under 10.101.0.0/16.



Here are our running configs of the free firewalls:

Building configuration...

Current configuration:
!
frr version 7.5.1
frr defaults traditional
hostname OPNsense-CORE.localdomain
log syslog notifications
!
router ospf
 redistribute connected
 redistribute static
 passive-interface vmx0
 passive-interface vmx0_vlan900
!
line vty
!
end

Building configuration...

Current configuration:
!
frr version 7.5.1
frr defaults traditional
hostname OPNsense-Projecttransfer.localdomain
log syslog notifications
!
interface vmx0_vlan910
 ip ospf area 0.0.0.0
!
interface vmx0_vlan911
 ip ospf area 0.0.0.0
!
router ospf
 redistribute connected
 redistribute static
 passive-interface vmx0
 passive-interface vmx0_vlan900
!
line vty
!
end

    IPv4 Routes
    IPv6 Routes
    Running Configuration

Building configuration...

Current configuration:
!
frr version 7.5.1
frr defaults traditional
hostname OPNsense-001_Project1.localdomain
log syslog notifications
!
interface vmx0_vlan911
 ip ospf area 0.0.0.0
!
interface vmx0_vlan1010
 ip ospf area 0.0.0.0
!
router ospf
 redistribute connected
 redistribute static
 passive-interface vmx0_vlan900
!
line vty
!
end

Can you please help us?

All senses are opnsense v22.1 with latest plugins.

Thanks



                                                                     

[mimugmail]

Dont use area in interface section, only in networks Tab. Then this will work

[marcquark]

Regarding the error message i think the 2nd "warning" box from the docs should be the fix you're after: https://docs.opnsense.org/manual/dynamic_routing.html

[Layer8]

Thanks for help.

We removed the Area from Interface-Configuration and we increased the buffer size.

We cant get all senses to see theire neighbours. At the moment, only the PROJECT1SENSE can see the TRANSFERSENSE. TRANSFERSENSE and CORESENSE do not see any OSPF neighbours.

All interfaces which are involved into OSPF are configured with *allow all* firewall rules. We also tested the network with static route setup, which is working, so we think that we dont have any general network issues.

Any ideas?

Thanks.

[mimugmail]

I need a quick drawing of the network and all running configs please

[Layer8]

Please take a look in the initial post, there is a quick network description.

Here are the running configs:

CORESENSE:
Current configuration:
!
frr version 7.5.1
frr defaults traditional
hostname OPNsense-core.localdomain
log syslog notifications
!
router ospf
 redistribute connected
 redistribute static
 passive-interface vmx0
 passive-interface vmx0_vlan900
 network 10.90.10.0/24 area 0.0.0.0
!
line vty
!
end

TRANSFERSENSE:
Current configuration:
!
frr version 7.5.1
frr defaults traditional
hostname OPNsense-Projecttransfer.localdomain
log syslog notifications
!
router ospf
 redistribute connected
 redistribute static
 passive-interface vmx0
 passive-interface vmx0_vlan900
 network 10.90.10.0/24 area 0.0.0.0
 network 10.90.11.0/24 area 0.0.0.0
!
line vty
!
end

PROJECT1SENSE:
Current configuration:
!
frr version 7.5.1
frr defaults traditional
hostname OPNsense-001_Project1.localdomain
log syslog notifications
!
router ospf
 redistribute connected
 redistribute static
 passive-interface vmx0_vlan900
 network 10.90.11.0/24 area 0.0.0.0
!
line vty
!
end

[mimugmail]

And a drawing including networks

[Layer8]

Here it is.

BTW: We need OSPF not for backuproutes at the moment (maybe later), but to minimize the administration impact of static routes.

[mimugmail]

Transfersense needs 10.90.12 as area 0 too

[lilsense]

I do not recommend using OSPF for WAN. If one of the ifWAN starts to bounce constantly your entire network is dead. Traffic will only be forwarded once the table ha been complete.

I recommend using BGP for your WAN network.

[Layer8]

Thanks for this hint, but we dont use the default WAN interface, and they are on the passive list on every sense ( passive-interface vmx0_vlan900 ).

mimugmail: Will add it to area 0 tomorrow at work. Thanks.

[mimugmail]

Just keep in mind that for the interfaces who will have neighbors, those networks need to be defined in area. Rest is done via its magic

[Layer8]

mimugmail i did not follow your suggestion to add 10.90.12.0 to area 0, because this network is for future use and not yet available. So, currently no need to add it.



We tried to get frr working today, but we failed. Even RIP is not working. We get the following error under System -> Routes -> Log file:

PROJECTTRANSFERSENSE

2022-02-11T15:17:57   Debug   ospfd   interface 10.90.11.1 [8] join AllDRouters Multicast group.   
2022-02-11T15:17:57   Debug   ospfd   DR-Election[2nd]: DR 10.90.11.1   
2022-02-11T15:17:57   Debug   ospfd   DR-Election[2nd]: Backup 0.0.0.0   
2022-02-11T15:17:57   Debug   ospfd   DR-Election[1st]: DR 10.90.11.1   
2022-02-11T15:17:57   Debug   ospfd   DR-Election[1st]: Backup 10.90.11.1   
2022-02-11T15:17:17   Notice   frr_carp   FRR trigger OspfdEventHandler event.   
2022-02-11T15:17:17   Notice   frr_carp   FRR received carp configuration event.   
2022-02-11T15:17:17   Error   ospfd   [EC 100663299] buffer_flush_available: write error on fd 2: Bad file descriptor   
2022-02-11T15:17:17   Error   ospfd   [EC 100663304] ERROR: Command returned Warning Config Failed on config line 22: network 10.90.11.0/24 area 0.0.0.0   
2022-02-11T15:17:17   Error   ospfd   [EC 100663299] buffer_flush_available: write error on fd 2: Bad file descriptor   
2022-02-11T15:17:17   Informational   ospfd   ASBR[default:Status:2]: Already ASBR   
2022-02-11T15:17:17   Informational   ospfd   ASBR[default:Status:2]: Update   
2022-02-11T15:17:17   Informational   ospfd   ASBR[default:Status:1]: Update   
2022-02-11T15:17:17   Notice   zebra   client 23 says hello and bids fair to announce only ospf routes vrf=0   
2022-02-11T15:17:17   Error   ospfd   [EC 100663304] ERROR: Command returned Warning Config Failed on config line 22: network 10.90.11.0/24 area 0.0.0.0   
2022-02-11T15:17:17   Informational   ospfd   ASBR[default:Status:2]: Already ASBR   
2022-02-11T15:17:17   Informational   ospfd   ASBR[default:Status:2]: Update   
2022-02-11T15:17:17   Informational   ospfd   ASBR[default:Status:1]: Update   
2022-02-11T15:17:17   Notice   frr_carp   FRR received carp configuration event.   
2022-02-11T15:17:17   Informational   zebra   Zebra final shutdown   
2022-02-11T15:17:17   Notice   zebra   Terminating on signal   
2022-02-11T15:17:16   Notice   zebra   client 11 disconnected 1 ospf routes removed from the rib   
2022-02-11T15:17:16   Debug   zebra   release_daemon_table_chunks: Released 0 table chunks   
2022-02-11T15:17:16   Warning   zebra   [EC 4043309122] Client 'ospf' encountered an error and is shutting down.   
2022-02-11T15:17:16   Notice   ospfd   Terminating on signal

PROJECT1SENSE

2022-02-11T15:17:52   Debug   ospfd   interface 10.90.11.2 [7] join AllDRouters Multicast group.   
2022-02-11T15:17:52   Debug   ospfd   DR-Election[2nd]: DR 10.90.11.2   
2022-02-11T15:17:52   Debug   ospfd   DR-Election[2nd]: Backup 0.0.0.0   
2022-02-11T15:17:52   Debug   ospfd   DR-Election[1st]: DR 10.90.11.2   
2022-02-11T15:17:52   Debug   ospfd   DR-Election[1st]: Backup 10.90.11.2   
2022-02-11T15:17:12   Notice   frr_carp   FRR trigger OspfdEventHandler event.   
2022-02-11T15:17:12   Notice   frr_carp   FRR received carp configuration event.   
2022-02-11T15:17:12   Informational   ospfd   ASBR[default:Status:2]: Already ASBR   
2022-02-11T15:17:12   Informational   ospfd   ASBR[default:Status:2]: Update   
2022-02-11T15:17:12   Informational   ospfd   ASBR[default:Status:1]: Update   
2022-02-11T15:17:12   Notice   zebra   client 11 says hello and bids fair to announce only ospf routes vrf=0   
2022-02-11T15:17:12   Informational   ospfd   ASBR[default:Status:2]: Already ASBR   
2022-02-11T15:17:12   Informational   ospfd   ASBR[default:Status:2]: Update   
2022-02-11T15:17:12   Informational   ospfd   ASBR[default:Status:1]: Update   
2022-02-11T15:17:12   Notice   frr_carp   FRR received carp configuration event.   
2022-02-11T15:17:12   Informational   zebra   Zebra final shutdown   
2022-02-11T15:17:12   Notice   zebra   Terminating on signal   
2022-02-11T15:17:12   Notice   zebra   client 11 disconnected 1 ospf routes removed from the rib   
2022-02-11T15:17:12   Debug   zebra   release_daemon_table_chunks: Released 0 table chunks   
2022-02-11T15:17:12   Warning   zebra   [EC 4043309122] Client 'ospf' encountered an error and is shutting down.   
2022-02-11T15:17:12   Notice   ospfd   Terminating on signal

Here are the current running configs:

PROJECTTRANSFERSENSE

Building configuration...

Current configuration:
!
frr version 7.5.1
frr defaults traditional
hostname OPNsense-Projekttransfer.localdomain
log syslog
!
interface vmx0_vlan911
 ip ospf area 0.0.0.0
 ip ospf authentication message-digest
 ip ospf message-digest-key 1 md5 test
!
router ospf
 ospf router-id 10.90.11.1
 redistribute connected
 redistribute static
 passive-interface vmx0
 passive-interface vmx0_vlan900
!
line vty
!
end

PROJECT1SENSE

Building configuration...

Current configuration:
!
frr version 7.5.1
frr defaults traditional
hostname OPNsense-001_Autoinspect.localdomain
log syslog
!
interface vmx0_vlan911
 ip ospf authentication message-digest
 ip ospf message-digest-key 1 md5 test
!
router ospf
 ospf router-id 10.90.11.2
 redistribute connected
 redistribute static
 passive-interface lo0
 passive-interface vmx0
 passive-interface vmx0_vlan900
 passive-interface vmx0_vlan1010
 network 10.90.11.0/24 area 0.0.0.0
!
line vty
!
end

Following the settings on both senses:

Routing -> General
1 Enabled
Profile Traditional
0 Enable CARP Failover
0 Enable SNMP Agent X Support
1 Event Loggin
Log Level Debugging

Routing -> OSPF -> General
1 Enable
0 CARP demote
Router ID 10.90.11.1 and 10.90.11.2
Reference Cost empty
Passiv Interfaces see running config
Route Distribution: Connected routes + statically configured routes
Redistribution Map none
0 Advertise Default Geteway
0 Always Advertise Default Gateway
Advertise Default Gateway Metci empty

Routing -> OSPF -> Networks
1 Enabled
Network Address 10.90.11.0
Network Mask 24
Area 0.0.0.0
Area Range empty
Prefix-List In none
Prefix-List Out none

Routing -> OSPF -> Interfaces
1 Enabled
Interface see running config
AuthenticatioN Type see running config
Authentication Key see running config
Authentication Key ID 1
Area empty
Cost empty
Cost when demoted 65535
Depent on carp none
hHlloe interval empty
Dead Intervalt empty
Retransmission Interval empty
Priority empty
Network Type none

Routing -> OSPF -> Prefix List
empty

Routing -> OSPF -> Route Maps
empty



oh and again: allow all rules on every interface.

[mimugmail]

Sorry, I have No idea how to help, your drawing tells a different story

[Layer8]

Sorry, whats different between the drawing and the config?

There is 10.90.11.0/24 in VLAN 911 between the transfer and the project1 sense and this is the network which is definted to exchange routes over OSPF?

I said in the last post, that we disabled OSPF on all other interfaces/networks to reduce the complexity to only two routers until this minimal setup is running.