Messages

have two firewalls/routers, one in each datacenter.

Each router has 3 interfaces
Router A
0- layer 3 lan subnet(2)
1- WAN single public ip address
2- layer 2 lan subnet(1) (vxlan over ipsec vpn over the wan interface)

Router B
0- layer 3 lan subnet(3)
1- WAN single public ip address
2- layer 2 lan subnet(1) (vxlan over ipsec vpn over the wan interface)

The layer 2 subnet(1) uses carp as the default gateway and spans both datacenters
I am using ospf for route redistribution.
There is a site to site vpn tunnel between routers.


In normal operation.
On router A carp Master
OSPF sees the subnets(1) and (2) as Directly Attached.  (GOOD)
OSPF sees the subnets(3) as available over the vpn to router B (GOOD)
system routing table sees subnets (1) and (2) as    Directly Attached (GOOD)
system routing table subnets (3) as available over the vpn to router B (GOOD)

On router B carp backup
OSPF sees the subnets (3) as Directly Attached (GOOD)
OSPF sees the subnets(1) an (2) as available over the vpn to router A (GOOD)
system routing table sees subnets (3) is Directly Attached (GOOD)
system routing table subnets (2) as available over the vpn to router A (BAD)

The carp subnet is not listed in the system routing table :(
This stops traffic from subnet (3) for accessing subnet (1)

It the minimum I would have expected the interface ip addresses to be in the system routing table. but the carp interfaces are not listed in the system routing table. Should they be listed?

Why is the route from ospf not making its way to the system routing table?

In backup operation (CARP Maintenance mode) on router A
On router A carp BACKUP
OSPF sees the subnets(2) as Directly Attached.  (GOOD)
OSPF sees the subnets(1) and (3) as available over the vpn to router B (GOOD)
system routing table sees subnets (1) and (2) as    Directly Attached (BAD) i think as CARP interface is in BACKUP state?
System routing table subnets(3) as available over the vpn to router B (GOOD)

On router B carp MASTER
OSPF sees the subnets (1) and (3) as Directly Attached (GOOD)
OSPF sees the subnets an (2) as available over the vpn to router A (GOOD)
system routing table sees subnets (1) and (3) is Directly Attached (GOOD)
system routing table subnets (2) as available over the vpn to router A (GOOD)


If we can fix the ospf route missing in normal operation i think we will be good?

any suggestions?
Running on 24.1.9

1. If doing this on VMware check the port security on the ports connecting to the firewall.
2. OPT1 Physical interface will be for the connection of VXLAN


Router A
Add interface>Other types>VXLAN
VNI=1
Source address= local L3 Interface facing Router B
Remote address= remote L3 Interface on Router B

Interface> Assignments
Add OPT1( where the l2 network will connect)
Add new vxlan interface.

Interface > VXLAN
Enable Interface
No IP address

Interface > OPT1
Enable Interface
No IP address

Add interface>Other types>Bridge
members= OPT1 + vxlan

Interface> Assignments
Add Bridge

Interface > Bridge
Enable Interface
Add the l3 network gateway IP address here for the l2 subnet

System > Tunables
net.link.bridge.pfil_bridge   (Set to 1 to enable filtering on the bridge interface) = 1
net.link.bridge.pfil_member (Set to 0 to disable filtering on the incoming and outgoing member interfaces.   ) = 0

REBOOT!!!!!!!

Firewall Rules > Bridge
Do the firewall rules here :)

Repeat for Router B
swap the IP address on the vxlan device

If it not working check the device that you are plugging the firewall into for security at layer 2 eg vmware port security

think iv have cracked it.
will wright up the notes. but lets just blame vmware port security stuff in the meantime.

I have looked at the that post numerus times. :(

Dose the vxlan need an ip address, if so in what subnet? Im assuming not as it should be a L2 tunnel ?
How is the vxlan connected to the physical port on the firewall? Do I use a bridge?

At some some point there will need at be an interface with an address to allow external connectivity in/out of the  L2 vxlan network. 
would carp be available?

I can use as a know starter position i can use https://docs.opnsense.org/manual/how-tos/ipsec-s2s-route.html
How can the network have Vxlan overlaid to have Site B PC be in the same L2 network as site A PC

OK I have tried and not got very far. have not found any documentation on how to implement on opnsense.

As a starter I have working L3 Can ping between PC
[PC 192.168.1.2]-192.168.1.1/24-LAN[opnsese A]-{10.1.1.1 ipsec tunnel}-INTERNET-{10.1.1.2 ipsec tunnel}-[opnsense B]LAN-192.168.2.1/24-[PC 192.168.2.2]

Im looking to use vxlan to extend a layer 2 network from Site A to Site B
[PC 192.168.1.2]-192.168.1.0/24-VxLAN[opnsense A]-{10.1.1.1 ipsec tunnel}-INTERNET-{10.1.1.2 ipsec tunnel}-[opnsense B]VxLAN-192.168.1.0/24-[PC 192.168.1.3]

I am using a bridge to  [lan and vxlan]
im using the ip address of the  ipsec tunnel for vxlan.

Has anyone got a guide on setup ?

thanks

Have updated Site B madness.

(_______Site A Opnsense_________)                      (________Site B other firewall___________)
10.1.99.0/24LAN---198.51.100.5WAN---INTERNET---198.51.100.15WAN A---192.168.66.0/24LAN
                                                                         ---198.51.100.25WAN B---192.168.66.0/24LAN
                            [_________10.1.1.1 vpn-A tunnel 10.1.1.2_________]
                            [_________10.1.2.1 vpn-B tunnel 10.1.2.2_________]

Site B has two independent wan connections. so I have run a vpn tunnel to each at site B.
Can we not have 2 static routes with preference to VPN-A when it is up, then fullback to VPN-B?
In cisco world I believe this to be a floating static route at configured at site A

OPNsense 23.7.8_1-amd64
FreeBSD 13.2-RELEASE-p5
OpenSSL 1.1.1w 11 Sep 2023

Having an issue with routing over routed ipsec site-to-site vpn.

Site A Opnsense                                                          Site B (other firewall)
10.1.99.0/24------[10.1.1.1 vpn-A tunnel 10.1.1.2]---- 192.168.0.0/16
                           [10.1.2.1 vpn-B tunnel 10.1.2.2]---- 192.168.0.0/16

Tunnels are up and I can ping the tunnel ips of site B
Im trying to configured failover from primary vpn-A to backup vpn-B using static routes.

System/Gateway/Single detects the vpn-tunnels are up or down.
I have configured System/routes have 2 entry's for 192.168.0.0/16
1. Pointing to tunnel ip vpn-a
2. Pointing to tunnel ip vpn-b

When checking System/Routes/Status only vpn-b appears to be listed even if system/gateway/single shows this gateway to be down.

Im expecting the Priority and Status to be taken into consideration when the system makes routing decisions.
Can have more than one static route to the same destination?
Have I got this wrong?
Whatever route was configured last appears to win
System/Routes/configuration do not appear to be able to use Gateway Groups.

I can route via firewall rules with gateway pointing to the Gateway groups but If opnsenese is hosting internal dns how do I route the responses back to site B correctly ? Im assuming it would be using the system routing table?

Any pointers welcome
Thanks

         

have similar issues.
hangs on configuring firewall and
ipsec0: changing name to ipsec1
ipsec2: changing name to ipsec3
ipsec4: changing name to ipsec2
configuring firewall

only have 3 ipsec trunks only 1 is actually in use.

have it runing on vmware 2cpu 8Gb hard drive, 2G ram

looking at + 5minutes boot time ay ideas whare to look.

Versions   OPNsense 22.7.2-amd64
FreeBSD 13.1-RELEASE-p1
OpenSSL 1.1.1q 5 Jul 2022
Updates   Click to check for updates.
CPU type   Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz (2 cores, 2 threads)

Well that was easy when you know how  :)

Spent about a week looking at this and wondering why I could not ping the firewall lan interfaces, then yesterday found out it was the ipsec vpn.

now it is all working as required.

Thankyou for the pointer.

version 21.1.7

Lan lost of connections to networks all behind 10.0.0.0/8
Head office (Draytek 2962)
Wan
Internet
|
IPSEC vpn Lan to Lan 10.0.0.0/8 - 10.14.182.0/24
|
Internet
WAN
opnsense (21.1.7 in the cloud on esxi)
Lan1 10.14.182.1/28 (Firewall ipv4 any to any)
Lan2 10.14.182.128/28 (Firewall ipv4 any to any)
Lan3 10.14.182.144/28 (Firewall ipv4 any to any)
IPSEC (Firewall ipv4 any to any)

As soon as the vpn comes up I lose connectivity between between the lan interfaces on the opnsense 
(Testing from a pc on lan 1 , lose ping to lan2 and lan 3 interfaces)

My expectation is the routing table takes priority, routing out local interfaces first before sending out the wan  where it is vpned back to head office.
10.0.0.0/8 should be lower priority than a local 10.141.182.1/28 Interface

Is the vpn capturing the traffic before it hits the routing table ?
I have tried with individual phase 2 for each Lan interface which did not help.

Have I done something incorrect.
Hopefully I don't need to create individual phase 2 for all the network hiding behind the head office 10.0.0.0/8

Any guidance welcomed.

Thanks

Nat port Forward
LAN   UDP   *   *   *   69 (TFTP)   127.0.0.1   69 (TFTP)   TFTP-Proxy out system default gateway.

Using carp ip I think will be a non starter might work if tftp-proxy was a newer version. more options

Just need to find out how to get inetd to start automatically on a restart?

Hi All

Is anyone able to provide guidance on getting the TFTP-proxy to work.

If tried to follow https://github.com/opnsense/core/issues/1810

I have created /usr/local/etc/inc/plugins.inc.d/tftpproxy.inc
I have added "tftp   dgram   udp   wait   root   /usr/libexec/tftp-proxy   tftp-proxy -v" to /etc/inetd.conf

Stuck on "the rest is done by generic Portforward rules"
I think Iv tried passing it to 127.0.0.1:69

This is the only thing stopping me from switching over from the other firewall.

TFTP is a show stopper if this is not available. Unfortunately some of the kit we lookafter is all managed by tftp transfers.
The TFTP servers will be on the internet we are all behind the hopefully new opnsense  firewall.

In an ideal world we would be using a carp address, to source tftp requests from however I would settle for any address that works.

Any help would be great.

thanks
dkeith