1. DISCLAIMER:
SCRAPPED - I CANT GET IT TO WONT WORK (without help)
DONT USE THIS ANYWHERE ELSE THAN IN TEST ENVIRONMENTS AT THE MOMENT.
IF YOU USE THIS IN PRODUCTION ITS AT YOUR OWN PERSONAL RISK.
This is a proof of concept to see if there's a way to improve current ipsec handling in the OPNsense in some way to have Filtering and NAT at the same time with ipsec and enc interfaces. Don't use this in production. It's here for discussion.
When I talk about NAT, I always mean SNAT here. NATD only supports SNAT.
===========================================================================================================================
2. What's the POC about?
When using IPsec with PF, there are limitations with FILTER and NAT when there are POLICY BASED and ROUTED (VTI) IPsec Tunnels on the same system.
I wanted to find out if there is a way forward that improves and resolves this behavior once and for all.
Since then I found out there is PF, IPFW and NATD available on the OPNsense at the same time, I wondered if I could just:
- Remove IPsec NAT and FILTER from PF
- Add IPsec NAT and FILTER to IPFW and NATD
It seems like OPNsense already uses this concept to some degree to share forwarding between PF and captive portal and traffic shaping IPFW:
Firewall: Settings: Advanced
Shared forwarding - Use shared forwarding between packet filter, traffic shaper and captive portal
Using policy routing in the packet filter rules causes packets to skip processing for the traffic shaper and captive portal tasks. Using this option enables the sharing of such forwarding decisions between all components to accommodate complex setups.
The following POC demonstrates that it looks like it is indeed possible to do that. Yet the side effects are UNCLEAR.
===========================================================================================================================
3. POC Test Setup explained:
+-------------------+ +------------------+
| OPNsense Site A | IPsec VTI | OPNsense Site B |
|-------------------| |------------------|
| LAN 192.168.1.1 | | LAN 192.168.2.1 |
| OPT1 192.168.101.1| | |
| WAN 172.16.0.189 |==================| WAN 172.16.0.190|
| ipsec2 10.20.30.2 | | ipsec2 10.20.30.3|
+-------------------+ +------------------+
||
|| Policy-based Tunnel
||
+-------------------+
| OPNsense Site C |
|-------------------|
| LAN 192.168.3.1 |
| WAN 172.16.0.191 |
+-------------------+
All of the below modifications have been made on OPNsense Site A. OPNsense Site B simulates an IPsec VTI implementation I have no control over, and OPNsense Site C simulates an IPSec Policy Based implementation I have no control over. OPNsense B and C are stock out of the box with no hacks.
My goal is to SNAT all requests from OPT1 to OPNsense A LAN IP address, since I assume that its the only accepted IP on Site B and Site C. They both don't know about my OPT1 net.
I have set the tunables to disable all filtering on enc and ipsec interfaces:
net.enc.in.ipsec_filter_mask IPsec input firewall filter mask runtime 0
net.enc.out.ipsec_filter_mask IPsec output firewall filter mask runtime 0
net.inet.ipsec.filtertunnel If set, filter packets from an IPsec tunnel. runtime 0
net.inet6.ipsec6.filtertunnel If set, filter packets from an IPsec tunnel. runtime 0
This will pass any traffic past the PF firewall and PF NAT, but it will still hit the IPFW firewall as well as the NATD daemon.
Afterwards I created:
- an IPsec VTI tunnel between OPNsense Site A and OPNsense Site B, but without any gateway or routes set on OPNsense Site A. The IPsec kernel routes (SPD) now take precedence over the routing decisions that PF would impose. This means we have to use IPFW to policy route policy based tunnel traffic to the right destination. And we also have to manage routes manually to the routed VPN tunnels.
route add -net 192.168.2.0/24 10.20.30.3
- a Policy Based IPsec Tunnel between OPNsense Site A and OPNsense Site C, with the traffic selectors being the LAN Site A and LAN Site C.
Now I load the IPFW kernel module in the opnsense:
kldload ipfw
Afterwards I restart ipfw:
service ipfw onerestart
Careful with further ipfw restarts, the rules we add are only in memory and will be lost after restarts.
Now you can see that there seem to be default rules that have been implemented by the already existing "/usr/local/etc/ipfw.rules"
ipfw list
00100 allow pfsync from any to any
00110 allow carp from any to any
00120 allow layer2 mac-type 0x0806,0x8035
00130 allow layer2 mac-type 0x888e,0x88c7
00140 allow layer2 mac-type 0x8863,0x8864
00150 deny layer2 not mac-type 0x0800,0x86dd
00200 skipto 60000 ip6 from ::1 to any
00201 skipto 60000 ip4 from 127.0.0.0/8 to any
00202 skipto 60000 ip6 from any to ::1
00203 skipto 60000 ip4 from any to 127.0.0.0/8
06000 skipto 60000 tcp from any to any out
06199 skipto 60000 ip from any to any
60000 return
65533 allow ip from any to any
65534 deny ip from any to any
65535 allow ip from any to any
Because there is "65533 allow ip from any to any" and ipfw comes after pf in the packet flow, all packets go through.
Now we can implement our divert rule so that we NAT traffic, and additionally our routes so that the traffic is routed through the right IPsec tunnels:
ipfw add 00005 divert 8668 ip from 192.168.101.0/24 to 192.168.2.0/24
ipfw add 00006 divert 8668 ip from 192.168.101.0/24 to 192.168.3.0/24
route add -net 192.168.2.0/24 10.20.30.3
route add -net 192.168.3.0/24 172.16.0.191
If you want the rules ipfw restart persistent you have to add them to the /usr/local/etc/ipfw.rules
Afterwards we have to configure "natd" to listen on 8668 and SNAT the packets for us on 192.168.1.1 if they match the above divert rule:
/etc/rc.conf
keymap="de.noacc.kbd"
natd_enable="YES"
natd_flags="-a 192.168.1.1 -redirect_address 192.168.101.1 192.168.1.1"
Here NATD SNATs source IP address 192.168.101.1 to source IP address 192.168.1.1 when it is diverted by IPFW to the socket 8668 (The default divert socket.) You can also create additional divert sockets on different ports and have multiple different NATs this way when adjusting the IPFW divert rules to them.
Now we just have to restart natd to take this in effect
service natd restart
===========================================================================================================================
4. Proof that it works:
Now I can prove with packet captures that the NATing works, on the Routed IPsec tunnel, as well as the Policy Based tunnel at the same time, without any tunables set and without pf in the way:
OPNsense SITE A TO OPNsense SITE B
root@OPNsenseA:~ # ping -S 192.168.101.1 192.168.2.1
PING 192.168.2.1 (192.168.2.1) from 192.168.101.1: 56 data bytes
64 bytes from 192.168.2.1: icmp_seq=0 ttl=64 time=0.755 ms
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.732 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=0.760 ms
root@OPNsenseB:~ # tcpdump -i enc0 -n host 192.168.2.1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enc0, link-type ENC (OpenBSD encapsulated IP), capture size 262144 bytes
08:48:41.013508 (authentic,confidential): SPI 0xc9014910: IP 192.168.1.1 > 192.168.2.1: ICMP echo request, id 61125, seq 0, length 64
08:48:41.013609 (authentic,confidential): SPI 0xc453dc2a: IP 192.168.2.1 > 192.168.1.1: ICMP echo reply, id 61125, seq 0, length 64
08:48:42.064250 (authentic,confidential): SPI 0xc9014910: IP 192.168.1.1 > 192.168.2.1: ICMP echo request, id 61125, seq 1, length 64
08:48:42.064342 (authentic,confidential): SPI 0xc453dc2a: IP 192.168.2.1 > 192.168.1.1: ICMP echo reply, id 61125, seq 1, length 64
08:48:43.136055 (authentic,confidential): SPI 0xc9014910: IP 192.168.1.1 > 192.168.2.1: ICMP echo request, id 61125, seq 2, length 64
08:48:43.136145 (authentic,confidential): SPI 0xc453dc2a: IP 192.168.2.1 > 192.168.1.1: ICMP echo reply, id 61125, seq 2, length 64
I can prove that filtering works too, by denying ICMP before the divert rule at 00050:
ipfw add 00040 deny icmp from any to any
Opnsense A cant ping Opnsense B anymore.
root@OPNsenseA:~ # ping 192.168.2.1
PING 192.168.2.1 (192.168.2.1): 56 data bytes
ping: sendto: Permission denied
ping: sendto: Permission denied
Opnsense B cant ping Opnsense A anymore.
root@OPNsenseB:~ # ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 56 data bytes
^C
--- 192.168.1.1 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
OPNsense SITE A TO OPNsense SITE C
root@OPNsenseA:~ # ping -S 192.168.101.1 192.168.3.1
PING 192.168.3.1 (192.168.3.1) from 192.168.101.1: 56 data bytes
64 bytes from 192.168.3.1: icmp_seq=0 ttl=64 time=1.231 ms
64 bytes from 192.168.3.1: icmp_seq=1 ttl=64 time=0.754 ms
64 bytes from 192.168.3.1: icmp_seq=2 ttl=64 time=0.920 ms
^C
--- 192.168.3.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.754/0.969/1.231/0.198 ms
root@OPNsenseA:~ # tcpdump -i enc0 -n not net 10.20.30.0/24
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enc0, link-type ENC (OpenBSD encapsulated IP), capture size 262144 bytes
10:55:37.713685 (authentic,confidential): SPI 0xc96a6d76: IP 192.168.1.1 > 192.168.3.1: ICMP echo request, id 53976, seq 0, length 64
10:55:37.714290 (authentic,confidential): SPI 0xc843e965: IP 192.168.3.1 > 192.168.1.1: ICMP echo reply, id 53976, seq 0, length 64
10:55:38.783108 (authentic,confidential): SPI 0xc96a6d76: IP 192.168.1.1 > 192.168.3.1: ICMP echo request, id 53976, seq 1, length 64
10:55:38.783573 (authentic,confidential): SPI 0xc843e965: IP 192.168.3.1 > 192.168.1.1: ICMP echo reply, id 53976, seq 1, length 64
10:55:39.854198 (authentic,confidential): SPI 0xc96a6d76: IP 192.168.1.1 > 192.168.3.1: ICMP echo request, id 53976, seq 2, length 64
10:55:39.854896 (authentic,confidential): SPI 0xc843e965: IP 192.168.3.1 > 192.168.1.1: ICMP echo reply, id 53976, seq 2, length 64
root@OPNsense:~ # tcpdump -i enc0 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enc0, link-type ENC (OpenBSD encapsulated IP), capture size 262144 bytes
10:55:37.713917 (authentic,confidential): SPI 0xc96a6d76: IP 192.168.1.1 > 192.168.3.1: ICMP echo request, id 53976, seq 0, length 64
10:55:37.714029 (authentic,confidential): SPI 0xc843e965: IP 192.168.3.1 > 192.168.1.1: ICMP echo reply, id 53976, seq 0, length 64
10:55:38.783298 (authentic,confidential): SPI 0xc96a6d76: IP 192.168.1.1 > 192.168.3.1: ICMP echo request, id 53976, seq 1, length 64
10:55:38.783381 (authentic,confidential): SPI 0xc843e965: IP 192.168.3.1 > 192.168.1.1: ICMP echo reply, id 53976, seq 1, length 64
10:55:39.854391 (authentic,confidential): SPI 0xc96a6d76: IP 192.168.1.1 > 192.168.3.1: ICMP echo request, id 53976, seq 2, length 64
10:55:39.854513 (authentic,confidential): SPI 0xc843e965: IP 192.168.3.1 > 192.168.1.1: ICMP echo reply, id 53976, seq 2, length 64
As reference, here's all established IPsec SAs of all firewalls:
root@OPNsenseA:~ # swanctl --list-sas
no files found matching '/usr/local/etc/strongswan.opnsense.d/*.conf'
99e0e1c1-285e-45cd-b4ef-61dcc914c879: #10, ESTABLISHED, IKEv2, 64e822915de2c77c_i* f9322b2b8d08c3f9_r
local '172.16.0.189' @ 172.16.0.189[4500]
remote '172.16.0.190' @ 172.16.0.190[4500]
AES_CBC-128/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/ECP_256
established 4904s ago, rekeying in 8431s
c24cc267-6c7e-4993-8bca-d55a1a36fc92: #6, reqid 2, INSTALLED, TUNNEL, ESP:AES_CBC-128/HMAC_SHA2_256_128
installed 1419s ago, rekeying in 1863s, expires in 2541s
in cc6a1148, 32241 bytes, 445 packets
out ca9c2e5c, 94340 bytes, 647 packets
local 172.16.0.189/32
remote 172.16.0.190/32
2cb5596b-8866-4675-bfe3-6743851fa95e: #29, ESTABLISHED, IKEv2, 3c5f14c4aff43214_i 9fb73ebfee8e3639_r*
local '172.16.0.189' @ 172.16.0.189[4500]
remote '172.16.0.191' @ 172.16.0.191[4500]
AES_CBC-128/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/ECP_256
established 716s ago, rekeying in 12374s
46f91ed0-3152-4fce-9ec5-2644e60ad11d: #7, reqid 1, INSTALLED, TUNNEL, ESP:AES_CBC-128/HMAC_SHA2_256_128
installed 716s ago, rekeying in 2561s, expires in 3244s
in c843e965, 504 bytes, 6 packets, 679s ago
out c96a6d76, 936 bytes, 6 packets, 679s ago
local 192.168.1.0/24
remote 192.168.3.0/24
root@OPNsenseB:~ # swanctl --list-sas
no files found matching '/usr/local/etc/strongswan.opnsense.d/*.conf'
bc96997e-6b9e-4b81-805d-48a1135d4560: #17, ESTABLISHED, IKEv2, 64e822915de2c77c_i f9322b2b8d08c3f9_r*
local '172.16.0.190' @ 172.16.0.190[4500]
remote '172.16.0.189' @ 172.16.0.189[4500]
AES_CBC-128/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/ECP_256
established 4903s ago, rekeying in 8853s
bad2434e-38cd-4d07-9908-8a73e60e1e21: #35, reqid 2, INSTALLED, TUNNEL, ESP:AES_CBC-128/HMAC_SHA2_256_128
installed 1418s ago, rekeying in 1862s, expires in 2542s
in ca9c2e5c, 47253 bytes, 645 packets
out cc6a1148, 64272 bytes, 444 packets
local 172.16.0.190/32
remote 172.16.0.189/32
root@OPNsenseC:~ # swanctl --list-sas
no files found matching '/usr/local/etc/strongswan.opnsense.d/*.conf'
29a73eff-323d-458d-9203-62f64dff2275: #15, ESTABLISHED, IKEv2, 3c5f14c4aff43214_i* 9fb73ebfee8e3639_r
local '172.16.0.191' @ 172.16.0.191[4500]
remote '172.16.0.189' @ 172.16.0.189[4500]
AES_CBC-128/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/ECP_256
established 717s ago, rekeying in 12424s
2f5a11bf-9f31-444a-9986-e5f3b694624d: #5, reqid 1, INSTALLED, TUNNEL, ESP:AES_CBC-128/HMAC_SHA2_256_128
installed 717s ago, rekeying in 2566s, expires in 3243s
in c96a6d76, 504 bytes, 6 packets, 680s ago
out c843e965, 936 bytes, 6 packets, 680s ago
local 192.168.3.0/24
remote 192.168.1.0/24
===========================================================================================================================
5. Things to check:
5.1 Can PF use the NATD socket with a divert rule like IPFW?
RESULT: PF + NATD doesn't work as expected. The packets get diverted and NATed but they're not returned back to the host. With this out of the way, it looks like IPFW with NATD is the only choice for Source NATing both policy and vti tunnels.
https://man.openbsd.org/divert.4
It seems like PF can also use "divert-packet" to send pakets to a NATD socket in userspace.
Testing this soon. If it works there could be a solution by implementing NATD into the opnsense GUI and expanding the PF Gui to allow setting a "divert-packet" parameter that selects the NATD socket.
It seems like the divert-packet options is exclusive to OpenBSDs implementation of PF. But there seems to be a "divert-to" parameter in FreeBSDs PF.
https://man.freebsd.org/cgi/man.cgi?pf.conf
divert-to <host> port <port>
Used to redirect packets to a local socket bound to host and
port. The packets will not be modified, so getsockname(2) on the
socket will return the original destination address of the
packet.
pass in log quick on hn2 from 192.168.101.1 to {any}
pass out log quick on ipsec2 from 192.168.101.1 to 192.168.2.1 divert-to localhost port 8668
===========================================================================================================================
6. Conclusions:
(Careful, so far these conclusions are only verified for firewall to firewall traffic, using ping -S)
PF + PF_NAT = Not working in this setup (either if_ipsec or if_enc)
PF + NATD = Not working in this setup (at all)
IPFW + IPFW_NAT = Not working in this setup (either if_ipsec or if_enc)
IPFW + NATD = Working for if_enc (policy based ipsec) and if_ipsec (route based ipsec) with FILTER and SNAT, but only from OPNsense to OPNsense directly. Hosts behind it don't work with NAT.
Current known drawbacks (to me):
- There is no pfsync equivalent for ipfw, that means if you use this hack you shouldn't use it on HA if you want seamless failover.
- You can't use the GUI to configure any of this. You have to use the shell.
- Traffic only works between OPNsense with NAT, I can't get it to work on hosts...
-So far, all traffic between the firewalls works great (which is a step up), but as soon as an outside host is introduced, things seem to fall apart and I don't know why. Following this alone with nobody to talk to won't result in anything. If anybody is interested to help me to follow this concept through, please PM.
It seems like this is related to this thread:
https://forums.freebsd.org/threads/share-ipsec-vpn-between-gateway-and-clients-using-ipfw-and-nat.87178/
7. Explanation why it doesn't work:
https://man.freebsd.org/cgi/man.cgi?query=ipsec&sektion=4&format=html
enc(4) is a virtual IPsec interface that is used to pass raw IP packets before encapsulation, and after decapsulation of ESP. That means, that the packets can be filtered and nated on it, before they're processed by IPsec. That is also the reason, why you need to set manual SPD (Security Policy Database) entries, when you NAT. Because you NAT before IPsec encapsulates the ESP packet. And if there's no security policy for the source IP address, IPsec will drop the packet before its encapsulated.
ipsec(4) is a virtual tunnel interface, that is used to pass already encapsulated packets. That means, you can't filter and NAT them, because the original IP header was already encapsulated into an ESP packet.
Now, that should mean, that you should be able to filter and nat all packets that come in and out of enc(4), because they all have their original IP header and are raw packets. In practice, this somehow does not work if "ipsec" is also introduced. With VTI, the packets first go into enc(4), and after they have been encapsulated they go into ipsec(4). Something about this process makes filter and NAT impossible, due to an implementation bug.
With the tunable "net.inet.ipsec.filtertunnel" you can change the filter behavior. Per default, you can perform packet filtering ant NAT before and after ESP on the enc(4) interface. With the tunable set, you can filter "inside" the ESP packet, but because enc(4) doesn't have any encapsulated packets, the filter won't work on it anymore. It will then work on the ipsec(4) interface where the encapsulated ESP packets are. The filter is set on the IP header inside the ESP packets.
Because Policy Based tunnel traffic only passes through the enc(4) interface, and not also through an ipsec(4) interface like VTI tunnels, you can either filter on the RAW IP packets, or on the ESP packets, but not on both (at least in FreeBSDs ipsec(4) and enc(4) implementation)
===========================================================================================================================
Let's start a discussion. 8)
Don't use divert to natd in 2023. There's in kernel NAT builtin in ipfw ...
@Patrick
I have seen that theres an ipfw_nat kernel module.
It exists on the system:
root@OPNsense:~ # ls /boot/kernel/*ipfw_nat*
/boot/kernel/ipfw_nat.ko /boot/kernel/ipfw_nat64.ko
But it's not loaded:
kldstat | grep ipfw_nat
I was able to load it with:
kldload ipfw_nat
The problem is the documentation. I found multiple sources that verified natd working correctly with kernel IPsec implementations. But I found nothing to none about ipfw and ipfw_nat being able to archieve the same. I will conduct further testing though.
I think that using in kernel nat with ipfw will have all the same problems as in kernel nat with pf, when in kernel ipsec is used at the same time.
EDIT:
As I imagined, using ipfw_nat has the same problems as using pf nat. This means the old technology of natd which works outside the kernel is probably still the right way to go.
pfw add 00050 nat 1 all from 192.168.101.0/24 to 192.168.2.0/24 via ipsec2
ipfw nat 1 config if ipsec2 ip 192.168.1.1
ipfw list
00050 nat 1 ip from 192.168.101.0/24 to 192.168.2.0/24 via ipsec2
00100 allow pfsync from any to any
00110 allow carp from any to any
00120 allow layer2 mac-type 0x0806,0x8035
00130 allow layer2 mac-type 0x888e,0x88c7
00140 allow layer2 mac-type 0x8863,0x8864
00150 deny layer2 not mac-type 0x0800,0x86dd
00200 skipto 60000 ip6 from ::1 to any
00201 skipto 60000 ip4 from 127.0.0.0/8 to any
00202 skipto 60000 ip6 from any to ::1
00203 skipto 60000 ip4 from any to 127.0.0.0/8
06000 skipto 60000 tcp from any to any out
06199 skipto 60000 ip from any to any
60000 return
65533 allow ip from any to any
65535 allow ip from any to any
root@OPNsense:~ # ipfw nat show config
ipfw nat 1 config if ipsec2
The ping is translated, goes to OPNsense B, is sent back, but doesn't come out of the enc0 or ipsec2 interface. That means the main culprit is "in kernel nat" together with "in kernel ipsec".
There is a known thread with most limitations about it at FreeBSD where I also added comments to:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248474
The problem with sysctl tuning is that it only works in one way, only vti or only policy based.
I'm not sure if I tested the whole thing with ipfw-only.
@mimugmail
I have seen that bug report and studied it thoroughly. Thats how I got the idea to turn all tunables to zero and let ipfw with natd take over. And with it, all the nat tricks and filtering seem to work on both enc0 and ipsec at the same time.
It would be great if there could be further testing to evaluate if maybe the ipsec traffic could be outsourced from PF to IPFW and NATD. (Just like the dummynet.ko kernel module outsourced traffic shaping from pf to ipfw)
It would make the ipsec implementation even more superior. But I dont know what the caveats or the scope of such a project would be, and if its even wanted.
Implementing this would also solve problems like using the webproxy over ipsec and probably any service the firewall has on localhost where traffic needs to be nat or rdr after going through an ipsec tunnel.
But my knowledge here is rather limited. I dont even know if pf and ipfw can coexist peacefully. So far it looks like pf filters first and then ipfw comes after it.
The hools between pf and ipfw have their limitations. There is a ticket in core about shared forwarding and its drawbacks.
I need to recheck your guide when family sleeps :)
Great, I'm looking forward to your opinion on it. :)
Updated the POC with an additional policy based tunnel, and IPFW policy based routing rules.
Now the test setup is complete and shows that with IPFW and NATD the IPsec implementation can indeed Filter and NAT on Route Based and Policy Based IPsec Tunnels at the same time.