Migrating OpenBSD pf to OpnSense 26.7 - routing issues

[mbrogies]

Hi Folks,

my first post here on the forum.

In my past, I have installed already several OpnSense, often as something completely for a new environment.

This time it's more complex and somehow I have a interesting problem with routing.

The old setup:

Code Select Expand

#                    +----------+
#                    |  router  | ISP Router (Fiber)
#                    +----+-----+
#                          |
#                          |
#                    +----+-----+
#                    |          |
#                    |          |
#  +-----------------+  Hub     +------------------+                   
#  |WANIP            +----+-----+                  |  WANIP VPN
# +-+----------+          |                +-------+------+       
# | fortinet  |           |bge1            |  OpenBSD/IKED|
# +-+----------+          |WAN IP         +-------+------+     
#  |                 +-----+------+                | 10.2.1.254
#  +-----------------|  firewall  +-----------------+         
#  em0/192.168.200.1 |            | bge3/10.2.1.1             
#                    |            |                         
#                    |            |                         
#  DMZ  -------------+            +---------------------------+10.4.1.254 - WLAN
#  em1/172.32.16.1   |            | bge2/10.4.1.1           
#                    |            |                         
#  DMZ2 -------------+            |                         
#  em2/10.3.1.1      |            |                         
#                    |            |
#                    +-----+------+
#                          |bge0
#                          |192.168.10.254
#                          |
#                          |
#                          |
#                          |
#                          | 192.168.10.1
#                          | 192.168.100.1
#                          | 192.168.140.1
#                    +-----+----+
#                    | Layer3SW +---------------192.168.10.0/24
#                    +-----+----+
#                          |
#                         / \
# 192.168.140.0/24--------   -------------- 192.168.100.0/24                           

The firewall in this case is an OpenBSD with pf

Now I try to replace "firewall" with an OPNsense in the first step.
Afterwards there will be some additional improvents (VLAN / true segmentation etc.)
Currently I am just focusing replaceing the old stuff with a current HW/SW.


And here some intresting routing problem begins for me.

Server1:
IP 192.168.10.33/24
GW: 192.168.10.1

Server2:
IP: 172.32.16.20/24
GW: 172.32.16.1

Example packet flow: 192.168.10.33 -> 172.32.16.20

192.168.10.33 -> 192.168.10.1 -> 192.168.10.254 -> 172.32.16.1 -> 172.32.16.20
Both IPs (192.168.10.254,172.32.16.1) are Interfaces on the OPNsense.

This works like a charm. The Server 1 can access everything needed on 172.32.16.20 (Server 2).

The other way around is currently making me some headache.

172.32.16.20 -> 172.32.16.20 -> 192.168.10.254 -> 192.168.10.33

I am stepping here in some sort of asymetric routing, since the OPNSense Interface is on the same Network as the Server 1 it try's to send the traffic directly back to Server 1 instead over the Layer3SW. This somehow let the statetable not match the traffic and the packages are silently dropped.

In the Livelog GUI i just see the "pass" packets and nothing else, even after enabled logging of "everything". NAT, auto generated rules, default block, etc.


Do you guys have any clue how I can solve that?

With the old OpenBSD this is running since years.


Greetings,

Matthias

[newsense]

I'll take a guess here that one is your default LAN and the other just another network on another port.

While the default lan allows you to go anywhere the additional ones will need specific firewall rules

[pfry]

[...]I am stepping here in some sort of asymetric routing, since the OPNSense Interface is on the same Network as the Server 1 it try's to send the traffic directly back to Server 1 instead over the Layer3SW. This somehow let the statetable not match the traffic and the packages are silently dropped.

In the Livelog GUI i just see the "pass" packets and nothing else, even after enabled logging of "everything". NAT, auto generated rules, default block, etc.[...]

That's not a layer 3 (IP) asymmetry. I wouldn't expect OPNsense to care about the source MAC, but I haven't had a similar routing arrangement in... a couple decades (prior to OPNsense). Packet flow should be visible in the session counters ("Firewall: Diagnostics: Sessions"). I don't think packet capture (on the firewall) would be more enlightening; you could check the switch. I can't think of a mechanism (on OPNsense) that would discard silently under this condition. To test, you could kill the L2 asymmetry with a route on Server1 for 172.32.16.0/24 (to 192.168.10.254).

[mbrogies]

[...]I am stepping here in some sort of asymetric routing, since the OPNSense Interface is on the same Network as the Server 1 it try's to send the traffic directly back to Server 1 instead over the Layer3SW. This somehow let the statetable not match the traffic and the packages are silently dropped.

In the Livelog GUI i just see the "pass" packets and nothing else, even after enabled logging of "everything". NAT, auto generated rules, default block, etc.[...]

That's not a layer 3 (IP) asymmetry. I wouldn't expect OPNsense to care about the source MAC, but I haven't had a similar routing arrangement in... a couple decades (prior to OPNsense). Packet flow should be visible in the session counters ("Firewall: Diagnostics: Sessions"). I don't think packet capture (on the firewall) would be more enlightening; you could check the switch. I can't think of a mechanism (on OPNsense) that would discard silently under this condition. To test, you could kill the L2 asymmetry with a route on Server1 for 172.32.16.0/24 (to 192.168.10.254).

As said, with the OpenBSD6.8 I don't get any errors like that.

My plan was to place proper Network segmentations with vlans on the Opnsense in the next step, but I think I will just do it a lot further in the step.

I will eliminate the switch as default gateway for the complete 192.168.x.x block and replace it with the Opnsense as default gateway.

[mbrogies]

I'll take a guess here that one is your default LAN and the other just another network on another port.

While the default lan allows you to go anywhere the additional ones will need specific firewall rules

Default LAN is in my case "VLAN 10" where also the server sits....
The rules can be ruled out, since I have have currently an  "pass in lan interface from any to any"
But as said in the other post, I think I will eliminate next the layer3 switch as an router.

[nero355]

Do you guys have any clue how I can solve that?

Policy Based Routing usually solves A-Symmetric Routing issues if that's really the case :)

I had something like this :
Host A :
- eth0 192.168.1.x/24
- eth0.10 10.0.0.x/24

Host B :
- eth0 10.0.0.x/24

And whenever I was connecting to Host A to it's 192.168.1.x/24 IP Address the replied traffic would go back via 10.0.0.x/24 instead of 192.168.1.x/24 and time-out after something like 30 to 60 seconds.

A small Policy Routing Rule @ Host A telling it to reply via the Gateway of the Interface the traffic came from instead of a Directly Connected Interface solved this :)

[pfry]

As said, with the OpenBSD6.8 I don't get any errors like that.[...]

Yup. I'd just like to know what the issue ends up being, when you find it. Out of curiosity.

[...]I will eliminate the switch as default gateway for the complete 192.168.x.x block and replace it with the Opnsense as default gateway.

Heh. I wouldn't do that, as you'll send traffic for the other two segments (192.168.100 and .140) to the firewall (presuming no specific routing on stations on the 192.168.10 segment, and a potential performance impact). You could create a stub VLAN with a /30 between the firewall and the switch to eliminate any asymmetry... if necessary. I assume you already have static routes in the firewall for the other segments, so another isn't an unusual addition.

[mbrogies]

Do you guys have any clue how I can solve that?

Policy Based Routing usually solves A-Symmetric Routing issues if that's really the case :)

I had something like this :
Host A :
- eth0 192.168.1.x/24
- eth0.10 10.0.0.x/24

Host B :
- eth0 10.0.0.x/24

And whenever I was connecting to Host A to it's 192.168.1.x/24 IP Address the replied traffic would go back via 10.0.0.x/24 instead of 192.168.1.x/24 and time-out after something like 30 to 60 seconds.

A small Policy Routing Rule @ Host A telling it to reply via the Gateway of the Interface the traffic came from instead of a Directly Connected Interface solved this :)

In the end I have to do it for the complete 192.168.10.0/24 to let it work.

Anyway, this primary network design was not created by me nor would I place an Layer3 Switch as GW into a SMB network where besides subnets, no other segmentation is done.

My plan was to introduce segmentation and VLAN routing over the OPNsense later in the process ... but now, I think I will just do it this friday.

 

[mbrogies]

Heh. I wouldn't do that, as you'll send traffic for the other two segments (192.168.100 and .140) to the firewall (presuming no specific routing on stations on the 192.168.10 segment, and a potential performance impact). You could create a stub VLAN with a /30 between the firewall and the switch to eliminate any asymmetry... if necessary. I assume you already have static routes in the firewall for the other segments, so another isn't an unusual addition.

I have to deal with it anyway later...
The hardware which is in use should be fast enough to handle 10Gbit Firewalling.
HPE ProLiant DL360 Gen10
2x 8-Core 3,3GHz / 4 GHz Turbo with 16 Threads
1x 4x 1GbE LOM
1x HPE FlexFabric 536FLR-T (4x 10GbE)
32GB RAM
4x 480GB 6G SATA SSD
So the Hardware should not be the bottleneck and there should be no performance issue.

But back to topic...

The complete 192.168.x.0/24 Subnets are quite sensitve

192.168.10.0/24 -> Server
192.168.20.0/24 -> Backup
192.168.40.0/24 -> ILO / Management
192.168.100.0/24 -> Clients
192.168.140.0/24 -> Wifi Clients
192.168.140.0/24 -> Printer
They are already seperated by VLANs, so every Network has a ".1" as default GW. They are handled by the Layer3 Switch. Just routing, no firewalling, nothing...
Security is technically not existend with that setup.

You could create a stub VLAN with a /30 between the firewall and the switch to eliminate any asymmetry... if necessary.

It's an intresting idea, but tbh. it sounds just like a whakey hack to cover the networks design flaw.

[pfry]

[...]They are handled by the Layer3 Switch. Just routing, no firewalling, nothing...
Security is technically not existend with that setup.[...]

Is your intent to add security? I ditched my L3 switch(es) to go with a flat network where everything runs through the firewall. I even use my (current) switches as port expanders only (each end-station port with a unique VLAN). I have a small network with a relatively large firewall, so it works for me. Your Xeon Gold 6234s should give you decent throughput... One way to find out.