So, I tried to move about 30 IPSEC running tunnels from a PFSense to a new OPNSense, using the new "connections" config, and it simply does not work (legacy tunnel setting works well).
If someone can help me I will try to solve, otherwise I wasted so much time that I will remain with PFSense.
The other end are Fortiagates.
I configured (last running test) as:
1- connections
-- local address, remote address
-- local auth: type = psk
-- remote auth: type = PSK, ID = remote endpoint's IP
-- children: local and remote = 0.0.0.0/0
2: pre shared keys
-- local id = remote endpoint's IP
-- PSK = the PSK
3: virtual tunnel interface
-- local address and remote address as in point 1
-- tunnel local address = 10.77.36.53, remote address = 10.77.36.54
FG has simmetric config.
Tunnel goes up. FG reports up both phase1 and 2. OPNSense shows phase1 up, and phase2 installed. SA database reports two SAs, one from remote to local, second from local to remote.
Routing tables shows 10.77.36.53 and 54 assigned to the link, one local other remote, as expected.
If FG pings from .54 to .53, OPNSense packet capture sees incoming packets, and reply packets entering to the tunnel. SA associations and IPSEC status report bytes in growing, but ZERO BYTES OUT. FG does not receive anything.
So packets from OPNSense are not entering the tunnel. I cannot understand why. Logs does not reports anything useful.
I configure VPN tunnels from 20 years, and it's never been so difficult. This is a basic setup, so where do I loose it?
Just for understanding the setup:
Local Nets: 0.0.0.0/0
Remote Nets: 0.0.0.0/0
Did you try to speficy the traffic selectors more tightly to see if it solves the routing problem?
For example:
Local Nets: 192.168.1.0/24
Remote Nets: 192.168.2.0/24
EDIT: I just saw you use VTI, disregard what I posted.
yes :(
and however routing table is ok and test is to the IP on the other site of the point to point
Did you use this tutorial and made sure policies are unchecked in the child?
https://docs.opnsense.org/manual/how-tos/ipsec-s2s-conn-route.html
Yes, I used that tutorial as reference.
My child section: Policies UNchecked, mode = tunnel, start action = start, DPD = clear, reqiq = empty, ESP = default, Local and Remote = 0.0.0.0/0
Did you create the firewall filter rules to allow traffic to enter the IPsec VTI Interface?
Firewall: Rules: VTI INTERFACE NAME, create an any any rule for troubleshooting.
Yes, otherwise I think the I will not see the ping response generate by OPNSense in packet capture.
I've the rule on "IPSEC" interface to catch all IPSECs, and also on a Firewall::Group that contains all VPNs (this is where I hope will define production rules).
Now I've added an "IPv4 all-all" on the specific VTI interface but still nothing. I tried this in the past, because I found same rule disabled and commented as "test"
From the firewall log:
action [pass]
anchorname
dir [in]
dst 10.77.36.53
ecn
id 10078
interface enc0
interface_name IPsec
ipflags none
ipversion 4
label tessssssssssstttt
length 84
offset 0
protoname icmp
protonum 1
reason match
rid dee6f5a05275dca20d8758de7ddafd31
rulenr 278
src 10.77.36.54
subrulenr
tos 0x0
ttl 64
There don't seem to be any obvious mistakes.
I would suggest you create a small isolated test network with two opnsense VMs and verify your VTI setup working between them. Maybe you're going to find out whats going wrong that way, and if there's an obvious step missing you could write it here so the docs can be improved.
Small Edit:
https://docs.opnsense.org/manual/vpnet.html#route-based-vti
Look at those tunables.
my opnsense is in fact a "dedicated test" instance, because I configured VPNs but I wasn't able to start nothing, so I'm using it only as "test instance on public network"
On the other side I have a Fortigate, but the problem is on the opnsense side: it is here where packets do not enter the tunnel.
So I can try a second opnsense, but I think it will not help :(
Yeah it might be a misconfiguration in the opnsense. Look at the link I provided above maybe that helps to identify the issue.
I'm not that firm with VTI tunnels since NAT won't match on them in the Opnsense due to FreeBSD limitations. Its either enc0, OR, vti. Never both. So I only use policy based enc0 tunnels.
Was just looking at the S2S doc last night and as is it doesn't work, and it's because of the auto generated rules.
To fix this do the following:
1) Disable auto generated rules in VPN - IPSec - Advanced Settings - Save
2) Firewall Rules - WAN - add the following
a) Pass Proto ESP source_any dest wan_address
b) Pass Proto TCP/UDP source_any dest wan_address dest_Port ISAKMP
c) Pass Proto TCP/UDP source_any dest wan_address dest_Port IPSEC NAT-T
After completing the two steps above restart IPSec and you'll have Phase 2 in Status Overview
I got stuck trying to further lock it down by adding an Alias-Hostname and replacing source_any with it - as I'm expecting WAN IPs to change without warning and ddclient is now operational on both ends. Using an alias doesn't work and I'm not sure if it's a bug or by design.
I'll try with the current IPs and see if that makes a difference.
My settings was exactly as you wrote. I'm sure I've created rules, and I don't know if I enabled "Disable all auto-added VPN rules" or if this was by default.
Can this be involved in my problem? I suppose that without them I will not able to establish connection at all, but I've not this kind of problem.
In status settings I can see phase 2 up. With "Bytes in" growing, "bytes out" fixed to empty (0).
I have created two opnsense VMs and two Ubuntu VMs as hosts and followed the tutorial in the docs, and I added the VTI tunables above.
I had trouble getting it to work... The doc was a bit hard to follow for me (maybe I was tired). But now I revised this setup:
I got it running I could ping the VTI interfaces from one opnsense to the other and back, and also I could ping from one ubuntu VM through the VTI IPsec tunnel to the other Ubuntu VM.
I'm going to post my settings here:
EDIT2: Revised the test setup. Now it always works without hikkups. I enabled Gateway Monitoring in the gateway so they always ping each other. I also set the WAN Test network to only internal IP addresses. And I changed the tunnel local and remote addresses to .2 and .3 so they reside in the same /31 network.
OPNsense 1:
LAN: 192.168.1.1/24
WAN: 172.16.11.1/24 (Private and Bogon nets are allowed)
OPT1: 192.168.101.1/24 (For SNAT tests)
VPN: IPsec: Virtual Tunnel Interfaces
Reqid: 10
Local address: 172.16.11.1
Remote address: 172.16.11.2
Tunnel local address: 10.10.1.2
Tunnel remote address: 10.10.1.3
Name: vti10
Interfaces: Assignments
Assign a New interface: ipsec10 (vti10)
Description: vti10
(make sure that the assigned interface is really on ipsec10 and not on an opt)
(Only assign the interface if it isn't automatically assigned)
Interfaces: [ipsec10]
Enable interface
(dont set any ip addresses)
(Set the description to vti10)
System: Gateways: Single:
Name: IPSEC10_GW
Interface: vti10
IP address: 10.10.1.3
Disable Gateway Monitoring: Unchecked
System: Routes: Configuration
Network Address: 192.168.2.0/24
Gateway: IPSEC10_GW
VPN: IPsec: Connections
Proposals: aes256-sha256-modp2048
Version: IKEv2
Local addresses: 172.16.11.1
Remote addresses: 172.16.11.2
Description: vti10p1
Local Authentication
Authentication: Pre-Shared Key
Id: 172.16.11.1
Remote Authentication
Authentication: Pre-Shared Key
Id: 172.16.11.2
Children:
Mode: Tunnel
Policies: UNCHECKED
Reqid: 10
Proposals: aes256-sha256-modp2048
Local: 0.0.0.0/0
Remote: 0.0.0.0/0
VPN: IPsec: Pre-Shared Keys:
Local Identifier: 172.16.11.1
Remote Identifier: 172.16.11.2
Pre-Shared Key: hallo
Type: PSK
System: Settings: Tunables:
Name: net.enc.in.ipsec_filter_mask
Value: 0
Name: net.enc.out.ipsec_filter_mask
Value: 0
Name: net.inet.ipsec.filtertunnel
Value: 1
Name: net.inet6.ipsec6.filtertunnel
Value: 1
Firewall:Rules: WAN:
IPv4 ICMP * * WAN net * * *
IPv4 ESP * * WAN net * * *
IPv4 UDP * * WAN net 500 (ISAKMP) * *
IPv4 UDP * * WAN net 4500 (IPsec NAT-T) * *
Firewall: Rules vti10
IPv4 * * * * * * *
Firewall: Rules LAN
IPv4 * * * * * * *
OPNsense 2:
LAN: 192.168.2.1/24
WAN: 172.16.11.2/24 (Private and Bogon nets are allowed)
VPN: IPsec: Virtual Tunnel Interfaces
Reqid: 10
Local address: 172.16.11.2
Remote address: 172.16.11.1
Tunnel local address: 10.10.1.3
Tunnel remote address: 10.10.1.2
Name: vti10
Interfaces: Assignments
Assign a New interface: ipsec10 (vti10)
Description: vti10
(make sure that the assigned interface is really on ipsec10 and not on an opt)
(Only assign the interface if it isn't automatically assigned)
Interfaces: [ipsec10]
Enable interface
(dont set any ip addresses)
(Set the description to vti10)
System: Gateways: Single:
Name: IPSEC10_GW
Interface: vti10
IP address: 10.10.1.3
Disable Gateway Monitoring: Unchecked
System: Routes: Configuration
Network Address: 192.168.1.0/24
Gateway: IPSEC10_GW
VPN: IPsec: Connections
Proposals: aes256-sha256-modp2048
Version: IKEv2
Local addresses: 172.16.11.2
Remote addresses: 172.16.11.1
Description: vti10p1
Local Authentication
Authentication: Pre-Shared Key
Id: 172.16.11.2
Remote Authentication
Authentication: Pre-Shared Key
Id: 172.16.11.1
Children:
Mode: Tunnel
Policies: UNCHECKED
Reqid: 10
Proposals: aes256-sha256-modp2048
Local: 0.0.0.0/0
Remote: 0.0.0.0/0
VPN: IPsec: Pre-Shared Keys:
Local Identifier: 172.16.11.2
Remote Identifier: 172.16.11.1
Pre-Shared Key: hallo
Type: PSK
System: Settings: Tunables:
Name: net.enc.in.ipsec_filter_mask
Value: 0
Name: net.enc.out.ipsec_filter_mask
Value: 0
Name: net.inet.ipsec.filtertunnel
Value: 1
Name: net.inet6.ipsec6.filtertunnel
Value: 1
Firewall:Rules: WAN:
IPv4 ICMP * * WAN net * * *
IPv4 ESP * * WAN net * * *
IPv4 UDP * * WAN net 500 (ISAKMP) * *
IPv4 UDP * * WAN net 4500 (IPsec NAT-T) * *
Firewall: Rules vti10
IPv4 * * * * * * *
Firewall: Rules LAN
IPv4 * * * * * * *
ATTENTION:
- With this setup, all filter rules (firewall rules) will match on the ipsecXX interfaces. NOT on the enc0 interface. All filtering on the enc0 was disabled, so policy based tunnels won't have firewall anymore.
- Please either use only VTI or only ENC0 tunnels, mixing them will leave one of them unable to filter in the firewall.
Update about NAT with VTI:
When those tunables are set, Source NAT matches on VTI (ipsecXX) interfaces, but will stop to match on encX interfaces.
System: Settings: Tunables:
Name: net.enc.in.ipsec_filter_mask
Value: 0
Name: net.enc.out.ipsec_filter_mask
Value: 0
Name: net.inet.ipsec.filtertunnel
Value: 1
Name: net.inet6.ipsec6.filtertunnel
Value: 1
I have tested Source NAT on VTI interface with the following rule:
Firewall:NAT:Outbound
Interface: vti10
Source address: OPT1 net
Destination address: 192.168.2.0/24
Translation/target: LAN address
@1 nat on ipsec10 inet from (hn2:network:1) to 192.168.2.0/24 -> (hn0:0) port 1024:65535
evaluations: 653 packets
It matches and works, so the information in the doc about it not working is outdated.
PACKET CAPTURES:
OPNsense 1:
LAN (hn0) -> v4: 192.168.1.1/24
OPT1 (hn2) -> v4: 192.168.101.1/24
WAN (hn1) -> v4: 172.16.11.1/24
vti10 (ipsec10) -> v4: 10.10.1.2/31
OPNsense 2:
LAN (hn0) -> v4: 192.168.2.1/24
WAN (hn1) -> v4: 172.16.11.2/24
vti10 (ipsec10) -> v4: 10.10.1.3/31
Packet Capture:
root@OPNsense1:~ # ping -S 192.168.101.1 192.168.2.1
PING 192.168.2.1 (192.168.2.1) from 192.168.101.1: 56 data bytes
64 bytes from 192.168.2.1: icmp_seq=0 ttl=64 time=0.963 ms
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.828 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=0.969 ms
root@OPNsense1:~ # tcpdump -i ipsec10 not net 10.10.1.0/24 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ipsec10, link-type NULL (BSD loopback), capture size 262144 bytes
12:38:55.558640 IP 192.168.1.1 > 192.168.2.1: ICMP echo request, id 35578, seq 0, length 64
12:38:55.559417 IP 192.168.2.1 > 192.168.1.1: ICMP echo reply, id 35578, seq 0, length 64
12:38:56.601865 IP 192.168.1.1 > 192.168.2.1: ICMP echo request, id 35578, seq 1, length 64
12:38:56.602516 IP 192.168.2.1 > 192.168.1.1: ICMP echo reply, id 35578, seq 1, length 64
12:38:57.644241 IP 192.168.1.1 > 192.168.2.1: ICMP echo request, id 35578, seq 2, length 64
12:38:57.644970 IP 192.168.2.1 > 192.168.1.1: ICMP echo reply, id 35578, seq 2, length 64
root@OPNsense2:~ # tcpdump -i ipsec10 not net 10.10.1.0/24 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ipsec10, link-type NULL (BSD loopback), capture size 262144 bytes
12:38:55.529523 IP 192.168.1.1 > 192.168.2.1: ICMP echo request, id 35578, seq 0, length 64
12:38:55.529624 IP 192.168.2.1 > 192.168.1.1: ICMP echo reply, id 35578, seq 0, length 64
12:38:56.572862 IP 192.168.1.1 > 192.168.2.1: ICMP echo request, id 35578, seq 1, length 64
12:38:56.572945 IP 192.168.2.1 > 192.168.1.1: ICMP echo reply, id 35578, seq 1, length 64
12:38:57.615212 IP 192.168.1.1 > 192.168.2.1: ICMP echo request, id 35578, seq 2, length 64
12:38:57.615309 IP 192.168.2.1 > 192.168.1.1: ICMP echo reply, id 35578, seq 2, length 64
that's great! SNAT was the only drownback with VTI.
The only problem is that I cannot start data flow over my tunnels :(
@bazbaz
When you have time, could you replicate the test setup I did above? If it works for you too, maybe you can find out what the problem is.
~Thank you for your time :)
What I did today:
- started a new opnsense on a different datacenter, connected to a public network with a public IP address (quicker this that a full private env)
- created only firewall rules for ipsec, nothing else
- created a tunnel on boh opnsenses with quick ipsec settings I'm using. All-all rules on IPSEC interfaces.
Now on the NEW opnsense, in status overview, I can see in phase 2 some "bytes out" but zero "bytes in". On the other (main) opnsense, I can see some "bytes in" but zero "bytes out".
It seems that connecting this OpnSense to the other, I have the same problem I'm trying to solve. Packets are not entering the tunnel.
Addendum: on the main opnsense, I was able to start ONE tunnel. Same as other: same kind of settings, I've an other FG on the other side, etc. This is working. Tried to compare/align every single settings on one other tunnel but I was unable to start it.
I'm disheartened
I found something.
On the NEW opnsense, in virtual tunnel interfaces, I assigned reqid = 1 to the interface. With this settings I can see bytes out. If I change it with anything else, also this opnsense stops sending packets to the tunnel.
My knowledge of reqid is that it may be unique for every tunnel interface. Not connected to a specific value. But I thing I miss something here, so on the main opnsense I have something similar.
in fact the only tunnel that is working (see my prev post) is the one where interface has reqid=1
Did you look at the test setup I described.
The VTI tunnel interface has a REQID set (10)
The child has the same REQID set (10)
Each tunnel interface and each child needs a unique matching requid.
Yes, and they were bold! Sorry, I didn't understand that that value was the link between interface and encryption!
I also needed a full reboot of routers to make them running, but after them now seems they are working!
Really thank you!
I'm happy you got it sorted out.
The REQID caught me off guard too, thats why I bolded it yesterday.
Sometimes its the small things.
Just make sure each of your 30 tunnels gets its own REQID :)
of course :)
I moved all tunnels and all is fine now, thanks
I think that the doc may be improved here:
https://docs.opnsense.org/manual/how-tos/ipsec-s2s-conn-route.html#children
where nothing says to assign reqid parameter to children, so I forgot the point. The table may have an additional row with "reqid = 10" and maybe a reminder that 10 is the value selected before.
Great that it worked.
Its on my list to improve it:
https://github.com/opnsense/docs/issues/502
Quote from: Monviech (Cedrik) on October 03, 2023, 11:37:04 AM...
ATTENTION:
- With this setup, all filter rules (firewall rules) will match on the ipsecXX interfaces. NOT on the enc0 interface. All filtering on the enc0 was disabled, so policy based tunnels won't have firewall anymore.
The instructions (https://docs.opnsense.org/manual/how-tos/ipsec-s2s-conn-route.html#install-firewall-policies) say to add rules to "Firewall -> Rules -> IPsec", but I think they mean "Firewall -> Rules -> IPSECnn", which makes sense.
Quote- Please either use only VTI or only ENC0 tunnels, mixing them will leave one of them unable to filter in the firewall.
Is this still a thing? If so, then it's simply not possible to migrate your tunnels one at a time?