Hello everyone,
we would like to setup a OSPF network in our VMware Environment. Because these are our first steps with opnsense and ospf/frr, we need some help please.
Problem 1:
All Sensense are VMs with one vmx0-NIC and VLAN interfaces. We get the following error in the debug log of frr in every of the three opnsense routers:
2022-02-09T16:36:48 Error ospfd [EC 100663299] setsockopt_so_recvbuf: fd 13: SO_RCVBUF set to 2097152 (requested 8388608)
All we found is this thread: https://forum.opnsense.org/index.php?topic=23187.0
Problem 2:
Not sure if problem 1 is a complete show stopper, but we also need some help to get our setup running.
This is our network:
[CORESENSE] .1 <- 10.90.10.0/24 VLAN 910 -> .2 [TRANSFERSENSE] .1 <- 10.90.11.0/24 .2 VLAN911 -> [PROJECT1SENSE]
CORESENSE has many VLANs attached, also WAN with Internet-Access and a default route.
TRANSFERSENSE is only to transfer the traffic between CORSESENSE and PROJECT1SENSE.
PROJECT1SENSE has some Project VLANs configured for different types of clients. These VLANs are configured like 10.101.0.0/24 - 10.101.9/24.
There is more than one PROJECT*SENS. We also have a PROJECT2SENSE filewall and so one attached to TRANSFERSENSE, but with local networks like 10.101.10.0/24 - 10.101.19.0/24 for example. All attached networks to the PROJECT firewalls can be summarized under 10.101.0.0/16.
Here are our running configs of the free firewalls:
QuoteBuilding configuration...
Current configuration:
!
frr version 7.5.1
frr defaults traditional
hostname OPNsense-CORE.localdomain
log syslog notifications
!
router ospf
redistribute connected
redistribute static
passive-interface vmx0
passive-interface vmx0_vlan900
!
line vty
!
end
QuoteBuilding configuration...
Current configuration:
!
frr version 7.5.1
frr defaults traditional
hostname OPNsense-Projecttransfer.localdomain
log syslog notifications
!
interface vmx0_vlan910
ip ospf area 0.0.0.0
!
interface vmx0_vlan911
ip ospf area 0.0.0.0
!
router ospf
redistribute connected
redistribute static
passive-interface vmx0
passive-interface vmx0_vlan900
!
line vty
!
end
Quote
IPv4 Routes
IPv6 Routes
Running Configuration
Building configuration...
Current configuration:
!
frr version 7.5.1
frr defaults traditional
hostname OPNsense-001_Project1.localdomain
log syslog notifications
!
interface vmx0_vlan911
ip ospf area 0.0.0.0
!
interface vmx0_vlan1010
ip ospf area 0.0.0.0
!
router ospf
redistribute connected
redistribute static
passive-interface vmx0_vlan900
!
line vty
!
end
Can you please help us?
All senses are opnsense v22.1 with latest plugins.
Thanks
Dont use area in interface section, only in networks Tab. Then this will work
Regarding the error message i think the 2nd "warning" box from the docs should be the fix you're after: https://docs.opnsense.org/manual/dynamic_routing.html
Thanks for help.
We removed the Area from Interface-Configuration and we increased the buffer size.
We cant get all senses to see theire neighbours. At the moment, only the PROJECT1SENSE can see the TRANSFERSENSE. TRANSFERSENSE and CORESENSE do not see any OSPF neighbours.
All interfaces which are involved into OSPF are configured with *allow all* firewall rules. We also tested the network with static route setup, which is working, so we think that we dont have any general network issues.
Any ideas?
Thanks.
I need a quick drawing of the network and all running configs please
Please take a look in the initial post, there is a quick network description.
Here are the running configs:
CORESENSE:
Current configuration:
!
frr version 7.5.1
frr defaults traditional
hostname OPNsense-core.localdomain
log syslog notifications
!
router ospf
redistribute connected
redistribute static
passive-interface vmx0
passive-interface vmx0_vlan900
network 10.90.10.0/24 area 0.0.0.0
!
line vty
!
end
TRANSFERSENSE:
Current configuration:
!
frr version 7.5.1
frr defaults traditional
hostname OPNsense-Projecttransfer.localdomain
log syslog notifications
!
router ospf
redistribute connected
redistribute static
passive-interface vmx0
passive-interface vmx0_vlan900
network 10.90.10.0/24 area 0.0.0.0
network 10.90.11.0/24 area 0.0.0.0
!
line vty
!
end
PROJECT1SENSE:
Current configuration:
!
frr version 7.5.1
frr defaults traditional
hostname OPNsense-001_Project1.localdomain
log syslog notifications
!
router ospf
redistribute connected
redistribute static
passive-interface vmx0_vlan900
network 10.90.11.0/24 area 0.0.0.0
!
line vty
!
end
And a drawing including networks
Here it is.
BTW: We need OSPF not for backuproutes at the moment (maybe later), but to minimize the administration impact of static routes.
Transfersense needs 10.90.12 as area 0 too
I do not recommend using OSPF for WAN. If one of the ifWAN starts to bounce constantly your entire network is dead. Traffic will only be forwarded once the table ha been complete.
I recommend using BGP for your WAN network.
Thanks for this hint, but we dont use the default WAN interface, and they are on the passive list on every sense ( passive-interface vmx0_vlan900 ).
mimugmail: Will add it to area 0 tomorrow at work. Thanks.
Just keep in mind that for the interfaces who will have neighbors, those networks need to be defined in area. Rest is done via its magic :)
mimugmail i did not follow your suggestion to add 10.90.12.0 to area 0, because this network is for future use and not yet available. So, currently no need to add it.
We tried to get frr working today, but we failed. Even RIP is not working. We get the following error under System -> Routes -> Log file:
PROJECTTRANSFERSENSEQuote2022-02-11T15:17:57 Debug ospfd interface 10.90.11.1 [8] join AllDRouters Multicast group.
2022-02-11T15:17:57 Debug ospfd DR-Election[2nd]: DR 10.90.11.1
2022-02-11T15:17:57 Debug ospfd DR-Election[2nd]: Backup 0.0.0.0
2022-02-11T15:17:57 Debug ospfd DR-Election[1st]: DR 10.90.11.1
2022-02-11T15:17:57 Debug ospfd DR-Election[1st]: Backup 10.90.11.1
2022-02-11T15:17:17 Notice frr_carp FRR trigger OspfdEventHandler event.
2022-02-11T15:17:17 Notice frr_carp FRR received carp configuration event.
2022-02-11T15:17:17 Error ospfd [EC 100663299] buffer_flush_available: write error on fd 2: Bad file descriptor
2022-02-11T15:17:17 Error ospfd [EC 100663304] ERROR: Command returned Warning Config Failed on config line 22: network 10.90.11.0/24 area 0.0.0.0
2022-02-11T15:17:17 Error ospfd [EC 100663299] buffer_flush_available: write error on fd 2: Bad file descriptor
2022-02-11T15:17:17 Informational ospfd ASBR[default:Status:2]: Already ASBR
2022-02-11T15:17:17 Informational ospfd ASBR[default:Status:2]: Update
2022-02-11T15:17:17 Informational ospfd ASBR[default:Status:1]: Update
2022-02-11T15:17:17 Notice zebra client 23 says hello and bids fair to announce only ospf routes vrf=0
2022-02-11T15:17:17 Error ospfd [EC 100663304] ERROR: Command returned Warning Config Failed on config line 22: network 10.90.11.0/24 area 0.0.0.0
2022-02-11T15:17:17 Informational ospfd ASBR[default:Status:2]: Already ASBR
2022-02-11T15:17:17 Informational ospfd ASBR[default:Status:2]: Update
2022-02-11T15:17:17 Informational ospfd ASBR[default:Status:1]: Update
2022-02-11T15:17:17 Notice frr_carp FRR received carp configuration event.
2022-02-11T15:17:17 Informational zebra Zebra final shutdown
2022-02-11T15:17:17 Notice zebra Terminating on signal
2022-02-11T15:17:16 Notice zebra client 11 disconnected 1 ospf routes removed from the rib
2022-02-11T15:17:16 Debug zebra release_daemon_table_chunks: Released 0 table chunks
2022-02-11T15:17:16 Warning zebra [EC 4043309122] Client 'ospf' encountered an error and is shutting down.
2022-02-11T15:17:16 Notice ospfd Terminating on signal
PROJECT1SENSEQuote2022-02-11T15:17:52 Debug ospfd interface 10.90.11.2 [7] join AllDRouters Multicast group.
2022-02-11T15:17:52 Debug ospfd DR-Election[2nd]: DR 10.90.11.2
2022-02-11T15:17:52 Debug ospfd DR-Election[2nd]: Backup 0.0.0.0
2022-02-11T15:17:52 Debug ospfd DR-Election[1st]: DR 10.90.11.2
2022-02-11T15:17:52 Debug ospfd DR-Election[1st]: Backup 10.90.11.2
2022-02-11T15:17:12 Notice frr_carp FRR trigger OspfdEventHandler event.
2022-02-11T15:17:12 Notice frr_carp FRR received carp configuration event.
2022-02-11T15:17:12 Informational ospfd ASBR[default:Status:2]: Already ASBR
2022-02-11T15:17:12 Informational ospfd ASBR[default:Status:2]: Update
2022-02-11T15:17:12 Informational ospfd ASBR[default:Status:1]: Update
2022-02-11T15:17:12 Notice zebra client 11 says hello and bids fair to announce only ospf routes vrf=0
2022-02-11T15:17:12 Informational ospfd ASBR[default:Status:2]: Already ASBR
2022-02-11T15:17:12 Informational ospfd ASBR[default:Status:2]: Update
2022-02-11T15:17:12 Informational ospfd ASBR[default:Status:1]: Update
2022-02-11T15:17:12 Notice frr_carp FRR received carp configuration event.
2022-02-11T15:17:12 Informational zebra Zebra final shutdown
2022-02-11T15:17:12 Notice zebra Terminating on signal
2022-02-11T15:17:12 Notice zebra client 11 disconnected 1 ospf routes removed from the rib
2022-02-11T15:17:12 Debug zebra release_daemon_table_chunks: Released 0 table chunks
2022-02-11T15:17:12 Warning zebra [EC 4043309122] Client 'ospf' encountered an error and is shutting down.
2022-02-11T15:17:12 Notice ospfd Terminating on signal
Here are the current running configs:
PROJECTTRANSFERSENSEQuote
Building configuration...
Current configuration:
!
frr version 7.5.1
frr defaults traditional
hostname OPNsense-Projekttransfer.localdomain
log syslog
!
interface vmx0_vlan911
ip ospf area 0.0.0.0
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 test
!
router ospf
ospf router-id 10.90.11.1
redistribute connected
redistribute static
passive-interface vmx0
passive-interface vmx0_vlan900
!
line vty
!
end
PROJECT1SENSEQuoteBuilding configuration...
Current configuration:
!
frr version 7.5.1
frr defaults traditional
hostname OPNsense-001_Autoinspect.localdomain
log syslog
!
interface vmx0_vlan911
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 test
!
router ospf
ospf router-id 10.90.11.2
redistribute connected
redistribute static
passive-interface lo0
passive-interface vmx0
passive-interface vmx0_vlan900
passive-interface vmx0_vlan1010
network 10.90.11.0/24 area 0.0.0.0
!
line vty
!
end
Following the settings on both senses:
Routing -> General1 Enabled
Profile Traditional
0 Enable CARP Failover
0 Enable SNMP Agent X Support
1 Event Loggin
Log Level Debugging
Routing -> OSPF -> General1 Enable
0 CARP demote
Router ID 10.90.11.1 and 10.90.11.2
Reference Cost empty
Passiv Interfaces see running config
Route Distribution: Connected routes + statically configured routes
Redistribution Map none
0 Advertise Default Geteway
0 Always Advertise Default Gateway
Advertise Default Gateway Metci empty
Routing -> OSPF -> Networks1 Enabled
Network Address 10.90.11.0
Network Mask 24
Area 0.0.0.0
Area Range empty
Prefix-List In none
Prefix-List Out none
Routing -> OSPF -> Interfaces1 Enabled
Interface see running config
AuthenticatioN Type see running config
Authentication Key see running config
Authentication Key ID 1
Area empty
Cost empty
Cost when demoted 65535
Depent on carp none
hHlloe interval empty
Dead Intervalt empty
Retransmission Interval empty
Priority empty
Network Type none
Routing -> OSPF -> Prefix Listempty
Routing -> OSPF -> Route Mapsempty
oh and again: allow all rules on every interface.
Sorry, I have No idea how to help, your drawing tells a different story
Sorry, whats different between the drawing and the config?
There is 10.90.11.0/24 in VLAN 911 between the transfer and the project1 sense and this is the network which is definted to exchange routes over OSPF?
I said in the last post, that we disabled OSPF on all other interfaces/networks to reduce the complexity to only two routers until this minimal setup is running.
Transfersense has no networks defined, check your running config
core does not have
interface vmx0_vlan910
ip ospf area 0.0.0.0
Also, get rid of the ospf auth to make sure they work.