[SOLVED] new IPSEC simply does not work

[bazbaz]

So, I tried to move about 30 IPSEC running tunnels from a PFSense to a new OPNSense, using the new "connections" config, and it simply does not work (legacy tunnel setting works well).

If someone can help me I will try to solve, otherwise I wasted so much time that I will remain with PFSense.

The other end are Fortiagates.
I configured (last running test) as:

1- connections
-- local address, remote address
-- local auth: type = psk
-- remote auth: type = PSK, ID = remote endpoint's IP
-- children: local and remote = 0.0.0.0/0

2: pre shared keys
-- local id = remote endpoint's IP
-- PSK = the PSK

3: virtual tunnel interface
-- local address and remote address as in point 1
-- tunnel local address = 10.77.36.53, remote address = 10.77.36.54

FG has simmetric config.

Tunnel goes up. FG reports up both phase1 and 2. OPNSense shows phase1 up, and phase2 installed. SA database reports two SAs, one from remote to local, second from local to remote.
Routing tables shows 10.77.36.53 and 54 assigned to the link, one local other remote, as expected.

If FG pings from .54 to .53, OPNSense packet capture sees incoming packets, and reply packets entering to the tunnel. SA associations and IPSEC status report bytes in growing, but ZERO BYTES OUT. FG does not receive anything.

So packets from OPNSense are not entering the tunnel. I cannot understand why. Logs does not reports anything useful.

I configure VPN tunnels from 20 years, and it's never been so difficult. This is a basic setup, so where do I loose it?

[Monviech (Cedrik)]

Just for understanding the setup:

Local Nets: 0.0.0.0/0
Remote Nets: 0.0.0.0/0

Did you try to speficy the traffic selectors more tightly to see if it solves the routing problem?

For example:
Local Nets: 192.168.1.0/24
Remote Nets: 192.168.2.0/24

EDIT: I just saw you use VTI, disregard what I posted.

[bazbaz]

yes :(
and however routing table is ok and test is to the IP on the other site of the point to point

[Monviech (Cedrik)]

Did you use this tutorial and made sure policies are unchecked in the child?

https://docs.opnsense.org/manual/how-tos/ipsec-s2s-conn-route.html

[bazbaz]

Yes, I used that tutorial as reference.

My child section: Policies UNchecked, mode = tunnel, start action = start, DPD = clear, reqiq = empty, ESP = default, Local and Remote = 0.0.0.0/0

[Monviech (Cedrik)]

Did you create the firewall filter rules to allow traffic to enter the IPsec VTI Interface?

Firewall: Rules: VTI INTERFACE NAME, create an any any rule for troubleshooting.

[bazbaz]

Yes, otherwise I think the I will not see the ping response generate by OPNSense in packet capture.

I've the rule on "IPSEC" interface to catch all IPSECs, and also on a Firewall::Group that contains all VPNs (this is where I hope will define production rules).

Now I've added an "IPv4 all-all" on the specific VTI interface but still nothing. I tried this in the past, because I found same rule disabled and commented as "test"

From the firewall log:

action   [pass]
anchorname   
dir   [in]
dst   10.77.36.53
ecn   
id   10078
interface   enc0
interface_name   IPsec
ipflags   none
ipversion   4
label   tessssssssssstttt
length   84
offset   0
protoname   icmp
protonum   1
reason   match
rid   dee6f5a05275dca20d8758de7ddafd31
rulenr   278
src   10.77.36.54
subrulenr   
tos   0x0
ttl   64

[Monviech (Cedrik)]

There don't seem to be any obvious mistakes.

I would suggest you create a small isolated test network with two opnsense VMs and verify your VTI setup working between them. Maybe you're going to find out whats going wrong that way, and if there's an obvious step missing you could write it here so the docs can be improved.

Small Edit:
https://docs.opnsense.org/manual/vpnet.html#route-based-vti

Look at those tunables.

[bazbaz]

my opnsense is in fact a "dedicated test" instance, because I configured VPNs but I wasn't able to start nothing, so I'm using it only as "test instance on public network"

On the other side I have a Fortigate, but the problem is on the opnsense side: it is here where packets do not enter the tunnel.

So I can try a second opnsense, but I think it will not help :(

[Monviech (Cedrik)]

Yeah it might be a misconfiguration in the opnsense. Look at the link I provided above maybe that helps to identify the issue.

I'm not that firm with VTI tunnels since NAT won't match on them in the Opnsense due to FreeBSD limitations. Its either enc0, OR, vti. Never both. So I only use policy based enc0 tunnels.

[newsense]

Was just looking at the S2S doc last night and as is it doesn't work, and it's because of the auto generated rules.


To fix this do the following:

1) Disable auto generated rules in  VPN - IPSec - Advanced Settings - Save


2) Firewall Rules - WAN - add the following

   a) Pass Proto ESP source_any dest wan_address

   b) Pass Proto TCP/UDP  source_any dest wan_address dest_Port ISAKMP

   c) Pass Proto TCP/UDP  source_any dest wan_address dest_Port IPSEC NAT-T


After completing the two steps above restart IPSec and you'll have Phase 2 in Status Overview



I got stuck trying to further lock it down by adding an Alias-Hostname and replacing source_any with it - as I'm expecting WAN IPs to change without warning and ddclient is now operational on both ends. Using an alias doesn't work and I'm not sure if it's a bug or by design.

I'll try with the current IPs and see if that makes a difference.

[bazbaz]

My settings was exactly as you wrote. I'm sure I've created rules, and I don't know if I enabled "Disable all auto-added VPN rules" or if this was by default.

Can this be involved in my problem? I suppose that without them I will not able to establish connection at all, but I've not this kind of problem.

In status settings I can see phase 2 up. With "Bytes in" growing, "bytes out" fixed to empty (0).

[Monviech (Cedrik)]

I have created two opnsense VMs and two Ubuntu VMs as hosts and followed the tutorial in the docs, and I added the VTI tunables above.

I had trouble getting it to work... The doc was a bit hard to follow for me (maybe I was tired). But now I revised this setup:

I got it running I could ping the VTI interfaces from one opnsense to the other and back, and also I could ping from one ubuntu VM through the VTI IPsec tunnel to the other Ubuntu VM.

I'm going to post my settings here:

EDIT2: Revised the test setup. Now it always works without hikkups. I enabled Gateway Monitoring in the gateway so they always ping each other. I also set the WAN Test network to only internal IP addresses. And I changed the tunnel local and remote addresses to .2 and .3 so they reside in the same /31 network.

OPNsense 1:
LAN: 192.168.1.1/24
WAN: 172.16.11.1/24  (Private and Bogon nets are allowed)
OPT1: 192.168.101.1/24  (For SNAT tests)

VPN: IPsec: Virtual Tunnel Interfaces
Reqid: 10
Local address: 172.16.11.1
Remote address: 172.16.11.2
Tunnel local address: 10.10.1.2
Tunnel remote address: 10.10.1.3
Name: vti10

Interfaces: Assignments
Assign a New interface: ipsec10 (vti10)
Description: vti10
(make sure that the assigned interface is really on ipsec10 and not on an opt)
(Only assign the interface if it isn't automatically assigned)

Interfaces: [ipsec10]
Enable interface
(dont set any ip addresses)
(Set the description to vti10)

System: Gateways: Single:
Name: IPSEC10_GW
Interface: vti10
IP address: 10.10.1.3
Disable Gateway Monitoring: Unchecked

System: Routes: Configuration
Network Address: 192.168.2.0/24
Gateway: IPSEC10_GW

VPN: IPsec: Connections
Proposals: aes256-sha256-modp2048
Version: IKEv2
Local addresses: 172.16.11.1
Remote addresses: 172.16.11.2
Description: vti10p1

Local Authentication
Authentication: Pre-Shared Key
Id: 172.16.11.1

Remote Authentication
Authentication: Pre-Shared Key
Id: 172.16.11.2

Children:
Mode: Tunnel
Policies: UNCHECKED
Reqid: 10
Proposals: aes256-sha256-modp2048
Local: 0.0.0.0/0
Remote: 0.0.0.0/0

VPN: IPsec: Pre-Shared Keys:
Local Identifier: 172.16.11.1
Remote Identifier: 172.16.11.2
Pre-Shared Key: hallo
Type: PSK

Code Select Expand

System: Settings: Tunables:
Name: net.enc.in.ipsec_filter_mask
Value: 0
Name: net.enc.out.ipsec_filter_mask
Value: 0
Name: net.inet.ipsec.filtertunnel
Value: 1
Name: net.inet6.ipsec6.filtertunnel
Value: 1
Firewall:Rules: WAN:
  IPv4 ICMP    *    *    WAN net    *    *    *             
  IPv4 ESP    *    *    WAN net    *    *    *             
  IPv4 UDP    *    *    WAN net    500 (ISAKMP)    *    *             
  IPv4 UDP    *    *    WAN net    4500 (IPsec NAT-T)    *    *       
Firewall: Rules vti10
  IPv4 *    *    *    *    *    *    *
Firewall: Rules LAN
  IPv4 *    *    *    *    *    *    *


OPNsense 2:
LAN: 192.168.2.1/24
WAN: 172.16.11.2/24 (Private and Bogon nets are allowed)

VPN: IPsec: Virtual Tunnel Interfaces
Reqid: 10
Local address: 172.16.11.2
Remote address: 172.16.11.1
Tunnel local address: 10.10.1.3
Tunnel remote address: 10.10.1.2
Name: vti10

Interfaces: Assignments
Assign a New interface: ipsec10 (vti10)
Description: vti10
(make sure that the assigned interface is really on ipsec10 and not on an opt)
(Only assign the interface if it isn't automatically assigned)

Interfaces: [ipsec10]
Enable interface
(dont set any ip addresses)
(Set the description to vti10)

System: Gateways: Single:
Name: IPSEC10_GW
Interface: vti10
IP address: 10.10.1.3
Disable Gateway Monitoring: Unchecked

System: Routes: Configuration
Network Address: 192.168.1.0/24
Gateway: IPSEC10_GW

VPN: IPsec: Connections
Proposals: aes256-sha256-modp2048
Version: IKEv2
Local addresses: 172.16.11.2
Remote addresses: 172.16.11.1
Description: vti10p1

Local Authentication
Authentication: Pre-Shared Key
Id: 172.16.11.2

Remote Authentication
Authentication: Pre-Shared Key
Id: 172.16.11.1

Children:
Mode: Tunnel
Policies: UNCHECKED
Reqid: 10
Proposals: aes256-sha256-modp2048
Local: 0.0.0.0/0
Remote: 0.0.0.0/0

VPN: IPsec: Pre-Shared Keys:
Local Identifier: 172.16.11.2
Remote Identifier: 172.16.11.1
Pre-Shared Key: hallo
Type: PSK

Code Select Expand

System: Settings: Tunables:
Name: net.enc.in.ipsec_filter_mask
Value: 0
Name: net.enc.out.ipsec_filter_mask
Value: 0
Name: net.inet.ipsec.filtertunnel
Value: 1
Name: net.inet6.ipsec6.filtertunnel
Value: 1
Firewall:Rules: WAN:
  IPv4 ICMP    *    *    WAN net    *    *    *             
  IPv4 ESP    *    *    WAN net    *    *    *             
  IPv4 UDP    *    *    WAN net    500 (ISAKMP)    *    *             
  IPv4 UDP    *    *    WAN net    4500 (IPsec NAT-T)    *    *       
Firewall: Rules vti10
  IPv4 *    *    *    *    *    *    *
Firewall: Rules LAN
  IPv4 *    *    *    *    *    *    *


ATTENTION:
- With this setup, all filter rules (firewall rules) will match on the ipsecXX interfaces. NOT on the enc0 interface. All filtering on the enc0 was disabled, so policy based tunnels won't have firewall anymore.

- Please either use only VTI or only ENC0 tunnels, mixing them will leave one of them unable to filter in the firewall.

[Monviech (Cedrik)]

Update about NAT with VTI:

When those tunables are set, Source NAT matches on VTI (ipsecXX) interfaces, but will stop to match on encX interfaces.

Code Select Expand

System: Settings: Tunables:
Name: net.enc.in.ipsec_filter_mask
Value: 0
Name: net.enc.out.ipsec_filter_mask
Value: 0
Name: net.inet.ipsec.filtertunnel
Value: 1
Name: net.inet6.ipsec6.filtertunnel
Value: 1
I have tested Source NAT on VTI interface with the following rule:

Firewall:NAT:Outbound
Interface: vti10
Source address: OPT1 net
Destination address: 192.168.2.0/24
Translation/target: LAN address

Code Select Expand

@1 nat on ipsec10 inet from (hn2:network:1) to 192.168.2.0/24 -> (hn0:0) port 1024:65535
evaluations: 653 packets
 
It matches and works, so the information in the doc about it not working is outdated.

PACKET CAPTURES:

Code Select Expand

OPNsense 1:

 LAN (hn0)       -> v4: 192.168.1.1/24
 OPT1 (hn2)      -> v4: 192.168.101.1/24
 WAN (hn1)       -> v4: 172.16.11.1/24
 vti10 (ipsec10) -> v4: 10.10.1.2/31

OPNsense 2:
 LAN (hn0)       -> v4: 192.168.2.1/24
 WAN (hn1)       -> v4: 172.16.11.2/24
 vti10 (ipsec10) -> v4: 10.10.1.3/31

Packet Capture:

Code Select Expand

root@OPNsense1:~ # ping -S 192.168.101.1 192.168.2.1
PING 192.168.2.1 (192.168.2.1) from 192.168.101.1: 56 data bytes
64 bytes from 192.168.2.1: icmp_seq=0 ttl=64 time=0.963 ms
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.828 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=0.969 ms


Code Select Expand

root@OPNsense1:~ # tcpdump -i ipsec10 not net 10.10.1.0/24 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ipsec10, link-type NULL (BSD loopback), capture size 262144 bytes
12:38:55.558640 IP 192.168.1.1 > 192.168.2.1: ICMP echo request, id 35578, seq 0, length 64
12:38:55.559417 IP 192.168.2.1 > 192.168.1.1: ICMP echo reply, id 35578, seq 0, length 64
12:38:56.601865 IP 192.168.1.1 > 192.168.2.1: ICMP echo request, id 35578, seq 1, length 64
12:38:56.602516 IP 192.168.2.1 > 192.168.1.1: ICMP echo reply, id 35578, seq 1, length 64
12:38:57.644241 IP 192.168.1.1 > 192.168.2.1: ICMP echo request, id 35578, seq 2, length 64
12:38:57.644970 IP 192.168.2.1 > 192.168.1.1: ICMP echo reply, id 35578, seq 2, length 64


Code Select Expand

root@OPNsense2:~ #  tcpdump -i ipsec10 not net 10.10.1.0/24 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ipsec10, link-type NULL (BSD loopback), capture size 262144 bytes
12:38:55.529523 IP 192.168.1.1 > 192.168.2.1: ICMP echo request, id 35578, seq 0, length 64
12:38:55.529624 IP 192.168.2.1 > 192.168.1.1: ICMP echo reply, id 35578, seq 0, length 64
12:38:56.572862 IP 192.168.1.1 > 192.168.2.1: ICMP echo request, id 35578, seq 1, length 64
12:38:56.572945 IP 192.168.2.1 > 192.168.1.1: ICMP echo reply, id 35578, seq 1, length 64
12:38:57.615212 IP 192.168.1.1 > 192.168.2.1: ICMP echo request, id 35578, seq 2, length 64
12:38:57.615309 IP 192.168.2.1 > 192.168.1.1: ICMP echo reply, id 35578, seq 2, length 64


[bazbaz]

that's great! SNAT was the only drownback with VTI.

The only problem is that I cannot start data flow over my tunnels :(