Possible asymmetric route printing issue?

[WatchIt]

I installed a new OPNSense router and I am trying to print to a different inside VLAN.  The printer just flashes "Printing Document" and hangs.  (It is clearly getting partially there)  There are multiple printers from various vendors on the same VLAN, and the same issue happens on all of them.

The OPN router only has two interfaces.  WAN and LAN.   Router LAN IP is 192.168.25.4/24.  Layer 3 switch with multiple vlans at 192.168.25.1/24.  Printer is at 192.168.26.10/24.

I added a static route to the OPN LAN interface 192.168.26.0/24 192.168.25.1.  Also, checked "Bypass firewall rules for traffic on the same interface" and this resolved most problems.  Only issue remaining is I can not print.  I can ping and access the printers management page without issue.  I can even print small text jobs, but anything of much size will hang. 

My workstation is IP:192.168.25.10/24 GW:192.168.25.4 (OPNSense Rtr)  If I add a manual route to my workstation "route add 192.168.26.0 mask 255.255.255.0 192.168.25.1" (bypass opnssense rtr) it prints immediately.  If select "Disable all packet filtering" on the OPNSense Router (convert into a routing-only platform) it prints immediately.  I can print from other vlans without issue, and the only difference is this route is asymmetrical and it is going through the OPNsense router. 

I am pretty sure this is not a NAT issue. (not 100%, but maybe 99%)  My gut instinct is the OPNSense is still filtering somehow even though set to "Bypass firewall rules for traffic on the same interface".  Any idea what I am missing? 

[CJ]

Can you post a network diagram?  I'm a bit unclear as to what you have configured and why.

What switch are you using and how are the VLANs configured on it?

[WatchIt]

We presently are a Cisco house, but are looking for a replacement router for our smaller clients.  Asymmetric routing is certainly not a common installation, but we do have a few clients that require them.  The test lab originally had a Cisco ASA 5506 installed that worked perfectly until lightning cooked it a few months back.  We replaced it with a home user ASUS model in the recycle pile to get back on-line short term.  It also works fine.  We are merely trying to replace a working router and not having any luck. 

[WatchIt]

The lab workstation sends to the printer
lab workstation -> opnsense -> switch -> printer

The printer packets return to the workstation
printer -> switch -> lab workstation (return traffic does not go through the opnsense since it is local to the switch)

I am no expert, but from experience it is typically one of two things.  A NAT issue, or one of the routers is filtering packets.  I was able to disable the opnsense firewall by selecting the below option and used it as a router, and when I do that everything prints perfectly.  I would guess the opnsense is still filtering somehow, but not sure how to disable the filtering.

[CJ]

I have to bow out of this.  It's further down the networking rabbit hole than I'm familiar with.  Hopefully another poster can help you.

[zan]

This really is an asymmetric routing issue, OPNsense would block TCP-SA from your workstation because it didn't see the TCP-Sync coming from your printer.
You can fix it by using 'sloppy state' on your vlan25 TCP pass rules. But if it were me, I'd redo the vlan25 or create a new vlan so all traffic must go through OPNsense.

[Seimus]

I agree with Zan.

Please do not do it like this. You tried to stretch a VLAN on L3 using a FW which is networking nightmare. I would not even advice to do it if there would be a pure simple router.

A FW must be a single point for ingress and egress traffic, asymmetric routing/traffic is to be advised to avoid of all costs.

Separate the segments properly, and configure a relay on OPNsense for the DHCP if you have a separate DHCP server in a different segment.

Regards,
S.

[WatchIt]

This really is an asymmetric routing issue, OPNsense would block TCP-SA from your workstation because it didn't see the TCP-Sync coming from your printer.
You can fix it by using 'sloppy state' on your vlan25 TCP pass rules. But if it were me, I'd redo the vlan25 or create a new vlan so all traffic must go through OPNsense.

Thank you for the reply.  According to the documentation... checking "Bypass firewall rules for traffic on the same interface" should be doing this automatically.  I did also look at the sloppy state which is the manual procedure to disable this, but I did not have any luck there either.  It generically suggests to add this as a rule with a matching floating rule, but it did not go into detail and I certainly could be doing something wrong.

Understood there are cleaner ways to do this.  However. this is this a lab environment with special circumstances and the documentation suggests this is should be possible.  The only issue I am having is with printing. 

[WatchIt]

I would not even advice to do it if there would be a pure simple router.

Thank you for your reply.  Asymmetric routing is very common in general.  It would be very common for an internet device not follow the same route paths.  Why would you not recommend this?

[Seimus]

On Internet yes, but here you are dealing with your LAN, a network you have control over it.

Asymmetric routing brings a lot of pain. Sub-optimal routing, performance issues, troubleshooting difficulties and other issues as you could see on this issue you try to resolve yourself.

You always want to go the path of optimal routing and optimal switching (L2 forwarding).

Regards,
S.

[Patrick M. Hausen]

Asymmetric routing brings a lot of pain.

Like breaking stateful firewalls 

[zan]

Thank you for your reply.  Asymmetric routing is very common in general.  It would be very common for an internet device not follow the same route paths.  Why would you not recommend this?

Ok for routers, not firewalls.

Anyway if you prefer not to redesign your network you may try this guide https://docs.netgate.com/pfsense/en/latest/troubleshooting/asymmetric-routing.html#manual-fix
It worked for me few years ago when I had to deal with asymmetric.
Or you can also try disable state tracking (state type: none) - I've never tried this before.
Or you can try pushing static route to your clients via dhcp option 121. You can follow a guide here https://forum.opnsense.org/index.php?topic=1972.0

[WatchIt]

Thank you for your reply.  Asymmetric routing is very common in general.  It would be very common for an internet device not follow the same route paths.  Why would you not recommend this?

Ok for routers, not firewalls.

Anyway if you prefer not to redesign your network you may try this guide https://docs.netgate.com/pfsense/en/latest/troubleshooting/asymmetric-routing.html#manual-fix
It worked for me few years ago when I had to deal with asymmetric.
Or you can also try disable state tracking (state type: none) - I've never tried this before.
Or you can try pushing static route to your clients via dhcp option 121. You can follow a guide here https://forum.opnsense.org/index.php?topic=1972.0

Thank You Zan,

I had actually seen both of the above links.   Adding the DHCP 121 option was the first thing I tried.  It worked well other than it does not release the route when you disconnect from the network.   Due to the test lab nature they are flipping networks all the time and it was not ideal.  I also tried the manual option from the PFSense link and that is one of them that did not resolve the issue, but as mentioned it just generically suggests what needs to be done, and I certainly could not have set it up correctly.

Thanks for your feedback!

[WatchIt]

Asymmetric routing brings a lot of pain. Sub-optimal routing, performance issues, troubleshooting difficulties and other issues as you could see on this issue you try to resolve yourself.

You always want to go the path of optimal routing and optimal switching (L2 forwarding).

Understood it is never ideal to use a FW as a router, but the FW is a couple of firewalls behind the front end, and security and performance is not my primary concern.  You seem to be offering generic best practice advice, which does not really apply to our test lab condition.  I appreciate your feedback, but I am really just looking how to configure it to use asymmetrical routing as the documentation suggests it will do.

[WatchIt]

We have worked around the issue another way.

Thanks for everyone's feedback!