OPNsense Forum

I've done this loads of times on a pfSense without issue, thought I'd give OPNsense a try for a change, and am hitting a brick wall.

I've got an IPSec tunnel set up between my OPNsense router and a Cisco ASA. Tunnel is good, and users behind the router can reach hosts on the remote network without issue. The problem is traffic originating on the router itself, which does not get sent through the tunnel as it should. I've set up the requisite hack (https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/accessing-firewall-services-over-ipsec-vpns.html) of a static route pointing to the LAN interface so the routing is good, but traffic gets dropped by the firewall for some reason. Naturally I have an allow all rule on the IPSec interface.

I've narrowed this down to the firewall because if I disable it with pfctl -d my traffic is sent without issue. Looking at the logs while trying to ping a host on the remote network, I noticed this:


The 96.51.x.x is my local WAN address, and 162.212.x.x is the tunnel endpoint. That traffic definitely should not be getting blocked. There's an automatically generated rule that should pass it:


pass out log on igb2 route-to (igb2 96.51.y.y) inet proto esp from any to 162.212.x.x keep state label "IPsec: NOC"
pass in log on igb2 reply-to (igb2 96.51.y.y) inet proto esp from 162.212..x.x to any keep state label "IPsec: NOC"


The fact that it's showing up as coming from the LAN interface is concerning. So I've put in floating rules to allow ESP traffic to and from the remote router but still the traffic originating from the local router does not pass.


Any thoughts on what could be causing this?

Here are my rules that work.
The 176.16.99.0/24 network is th one i configured in  VPN --> IPsec --> Mobile Clients .

Quote from: steffda on February 26, 2019, 08:18:09 PM
Here are my rules that work.
The 176.16.99.0/24 network is th one i configured in  VPN --> IPsec --> Mobile Clients .

I'm using a site-to-site VPN, so not really the same thing. Also, you shouldn't be leaving your management interfaces wide open for the world to see!

This issue of not being able to pass traffic mysteriously resolved itself after a reboot, despite identical firewall rules. But this misclassification of packets as coming from wrong interface continues to cause problems.

Observe the following firewall log excerpt, igb2 is the WAN port, igb3 is the LAN port.


Feb 28 20:14:33 calgary filterlog: 98,,,0,igb2,match,pass,out,4,0x0,,64,15585,0,none,17,udp,492,96.51.y.y,162.212.x.x,500,500,472
Feb 28 20:14:33 calgary filterlog: 92,,,0,igb2,match,pass,out,4,0x0,,64,54613,0,none,17,udp,432,96.51.y.y,162.212.x.x,4500,4500,412
# ESP traffic is listed as outbound from WAN igb2 as expected
Feb 28 20:14:37 calgary filterlog: 100,,,0,igb2,match,pass,out,4,0x0,,64,17677,0,none,50,esp,148,96.51.y.y,162.212.x.x,datalength=128
# suddenly changes to outbound from LAN igb3
Feb 28 20:17:18 calgary filterlog: 92,,,0,igb3,match,pass,out,4,0x0,,64,24955,0,none,50,esp,140,96.51.y.y,162.212.x.x,datalength=120
# and then changes to inbound to LAN igb3, resulting in a block
Feb 28 20:17:18 calgary filterlog: 10,,,0,igb3,match,block,in,4,0x0,,64,27452,0,none,50,esp,140,96.51.y.y,162.212.x.x,datalength=120
Feb 28 20:17:18 calgary filterlog: 10,,,0,igb3,match,block,in,4,0x0,,64,4042,0,none,50,esp,140,96.51.y.y,162.212.x.x,datalength=120
Feb 28 20:17:18 calgary filterlog: 10,,,0,igb3,match,block,in,4,0x0,,64,17528,0,none,50,esp,140,96.51.y.y,162.212.x.x,datalength=120
Feb 28 20:17:19 calgary filterlog: 10,,,0,igb3,match,block,in,4,0x0,,64,10157,0,none,50,esp,140,96.51.y.y,162.212.x.x,datalength=120
Feb 28 20:17:19 calgary filterlog: 10,,,0,igb3,match,block,in,4,0x0,,64,14787,0,none,50,esp,140,96.51.y.y,162.212.x.x,datalength=120
Feb 28 20:17:19 calgary filterlog: 10,,,0,igb3,match,block,in,4,0x0,,64,25526,0,none,50,esp,140,96.51.y.y,162.212.x.x,datalength=120
Feb 28 20:17:20 calgary filterlog: 10,,,0,igb3,match,block,in,4,0x0,,64,13062,0,none,50,esp,140,96.51.y.y,162.212.x.x,datalength=120
Feb 28 20:17:20 calgary filterlog: 10,,,0,igb3,match,block,in,4,0x0,,64,11938,0,none,50,esp,140,96.51.y.y,162.212.x.x,datalength=120
Feb 28 20:17:20 calgary filterlog: 10,,,0,igb3,match,block,in,4,0x0,,64,23065,0,none,50,esp,140,96.51.y.y,162.212.x.x,datalength=120
Feb 28 20:17:21 calgary filterlog: 10,,,0,igb3,match,block,in,4,0x0,,64,37367,0,none,50,esp,140,96.51.y.y,162.212.x.x,datalength=120
Feb 28 20:17:22 calgary filterlog: 10,,,0,igb3,match,block,in,4,0x0,,64,57219,0,none,50,esp,140,96.51.y.y,162.212.x.x,datalength=120
Feb 28 20:17:22 calgary filterlog: 10,,,0,igb3,match,block,in,4,0x0,,64,45356,0,none,50,esp,140,96.51.y.y,162.212.x.x,datalength=120
Feb 28 20:17:23 calgary filterlog: 10,,,0,igb3,match,block,in,4,0x0,,64,7877,0,none,50,esp,140,96.51.y.y,162.212.x.x,datalength=120
Feb 28 20:17:23 calgary filterlog: 10,,,0,igb3,match,block,in,4,0x0,,64,492,0,none,50,esp,140,96.51.y.y,162.212.x.x,datalength=120
Feb 28 20:17:23 calgary filterlog: 10,,,0,igb3,match,block,in,4,0x0,,64,52354,0,none,50,esp,140,96.51.y.y,162.212.x.x,datalength=120
Feb 28 20:17:27 calgary filterlog: 10,,,0,igb3,match,block,in,4,0x0,,64,53851,0,none,50,esp,140,96.51.y.y,162.212.x.x,datalength=120
Feb 28 20:17:27 calgary filterlog: 10,,,0,igb3,match,block,in,4,0x0,,64,57452,0,none,50,esp,140,96.51.y.y,162.212.x.x,datalength=120
Feb 28 20:17:27 calgary filterlog: 10,,,0,igb3,match,block,in,4,0x0,,64,51372,0,none,50,esp,140,96.51.y.y,162.212.x.x,datalength=120
Feb 28 20:17:30 calgary filterlog: 10,,,0,igb3,match,block,in,4,0x0,,64,38156,0,none,50,esp,140,96.51.y.y,162.212.x.x,datalength=120
Feb 28 20:17:30 calgary filterlog: 10,,,0,igb3,match,block,in,4,0x0,,64,39704,0,none,50,esp,140,96.51.y.y,162.212.x.x,datalength=120
Feb 28 20:17:30 calgary filterlog: 10,,,0,igb3,match,block,in,4,0x0,,64,34262,0,none,50,esp,140,96.51.y.y,162.212.x.x,datalength=120
Feb 28 20:17:36 calgary filterlog: 10,,,0,igb3,match,block,in,4,0x0,,64,760,0,none,50,esp,140,96.51.y.y,162.212.x.x,datalength=120
Feb 28 20:18:04 calgary filterlog: 98,,,0,igb2,match,pass,out,4,0x0,,64,46766,0,none,17,udp,492,96.51.y.y,162.212.x.x,500,500,472


I'm not certain but I think this intermittent packet blocking is dropping my tunnel regularly and making things unusable. Sometimes it will only be up for a couple of minutes, sometimes a couple of hours. Can anyone explain why traffic that is going out the WAN port all of a sudden gets classified as coming in the LAN port?

Complete IPSec log of the session corresponding to the above filter log, from manual start to unexpected drop: https://pastebin.com/ynEw5Q2s

Out of curiosity, how was pfS handling this error ? o_0

Feb 28 20:18:03 calgary charon: 09[ENC] <con1|2> parsed IKE_AUTH response 1 [ N(AUTH_FAILED) ]
Feb 28 20:18:03 calgary charon: 09[IKE] <con1|2> received AUTHENTICATION_FAILED notify error


Could it be something to look into ?