Routing internet traffic through a site-to-site IPsec tunnel

[Strykar]

Hi,

This is my setup, both sites are using OPNsense:

Site A
ADSL dynamic IPv4 WAN which is also a gateway to 2 LANs - LAN A(192.168.1.0/24) + LAN B(192.168.2.0/24)

Site B
Static public IPv4 + global IPv6. 2nd NIC connects a 10.x LAN but this machine does no NAT for its LAN.
As suggested by the howto doc linked below, it does do outbound NAT for Site A's LAN A+B.


I wish to tunnel all Internet traffic from both LAN subnets at Site A via Site B.
I've used https://goo.gl/0YriHL as a reference and it works using IKEv1 PSK.

My issue is that all inter-LAN access at Site A is lost when the tunnel goes up.
Machines in LAN A cannot ping or access machines in the same LAN. Same for LAN B.
While none of the LAN machines can ping each other, all traffic is correctly sent out via Site B.

These are the NAT rules at Site A.
A diff before and after the tunnel is established shows no change in them.

# pfctl -sn
no nat proto carp all
nat-anchor "natearly/*" all
nat-anchor "natrules/*" all
nat on pppoe0 inet from 127.0.0.0/8 to any port = isakmp -> 182.70.11.137 static-port
nat on pppoe0 inet from 192.168.1.0/24 to any port = isakmp -> 182.70.11.137 static-port
nat on pppoe0 inet from 192.168.2.0/24 to any port = isakmp -> 182.70.11.137 static-port
nat on pppoe0 inet from 127.0.0.0/8 to any -> 182.70.11.137 port 1024:65535
nat on pppoe0 inet from 192.168.1.0/24 to any -> 182.70.11.137 port 1024:65535
nat on pppoe0 inet from 192.168.2.0/24 to any -> 182.70.11.137 port 1024:65535
no rdr proto carp all
rdr-anchor "relayd/*" all
no rdr on igb0 proto tcp from any to (igb0) port = http
no rdr on igb0 proto tcp from any to (igb0) port = ssh
rdr-anchor "miniupnpd" all

The rather large diff between firewall rules (pfctl -sr) before and after the tunnel is established is at https://dl.dropboxusercontent.com/u/314525/rules.diff

• Where do I start to troubleshoot this?
• Is there any documentation to migrating this setup from IKEv1 PSK to IKEv2 RSA pubkey?

Perhaps of useful note is the fact that road warrior RSA pubkey auth works fine at both sites.

[reep]

My issue is that all inter-LAN access at Site A is lost when the tunnel goes up.
Machines in LAN A cannot ping or access machines in the same LAN. Same for LAN B.
While none of the LAN machines can ping each other, all traffic is correctly sent out via Site B.

At a very vague guess that's probably because once the link is up all and the new routing for it kicks in your traffic is then getting pushed over it, including LAN traffic...  the IPSEC connection is really 'just another interface'

Probably need to look at your rules/routing and figure what goes where.

Tried traceroute to see where packets are going ?

• Where do I start to troubleshoot this?

At the beginning ;-)

• Is there any documentation to migrating this setup from IKEv1 PSK to IKEv2 RSA pubkey?

I believe with Libreswan (which I use a lot elsewhere on Linux) you can use PSK passwords/RSA key pairs/Certificates but I am not sure that you can use RSA key pairs with Opnsense. Looks like PSK or Certs only (shame). Not sure why though, and I could be wrong.

Perhaps of useful note is the fact that road warrior RSA pubkey auth works fine at both sites.

Sorry  - can you explain that ?

B. Rgds
John

[Strykar]

It totally seems like LAN traffic is sent out via the IPsec interface, although this can't be the intended behavior?

Sorry I should've posted basic network test results.
192.168.1.2 is directly connected to LAN on the router (igb0) so it is just 1 hop away.

Here it's working without any IPsec tunnel established:

root@opnsense:~ # ipsec status
Security Associations (0 up, 0 connecting):
  none

root@opnsense:~ # ping -qc4 192.168.1.2
PING 192.168.1.2 (192.168.1.2): 56 data bytes

--- 192.168.1.2 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.300/0.369/0.445/0.052 ms

root@opnsense:~ # traceroute -m 5 -i igb0 -I 192.168.1.2
traceroute to 192.168.1.2 (192.168.1.2), 5 hops max, 48 byte packets
 1  192.168.1.2 (192.168.1.2)  0.485 ms  0.254 ms  0.204 ms
root@opnsense:~ #

And after the tunnel is established:

root@opnsense:~ # ipsec status
Routed Connections:
        con1{1}:  ROUTED, TUNNEL, reqid 1
        con1{1}:   192.168.1.0/24 === 0.0.0.0/0
Security Associations (1 up, 1 connecting):
   (unnamed)[2]: CONNECTING, 111.120.36.160[%any]...155.99.25.7[%any]
        con1[1]: ESTABLISHED 18 seconds ago, 111.120.36.160[C=US, ST=NYC, L=ALB, O=PSTO PKI/emailAddress=xxx@xxxxx.xx, CN=xxx.xxxxx.xx]...155.99.25.7[155.99.25.7]
        con1{2}:  INSTALLED, TUNNEL, reqid 1, ESP SPIs: xxxx519a_i xxxxd40b_o
        con1{2}:   192.168.1.0/24 === 0.0.0.0/0

root@opnsense:~ # ping -qc4 192.168.1.2
PING 192.168.1.2 (192.168.1.2): 56 data bytes

--- 192.168.1.2 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss

root@opnsense:~ # traceroute -m 5 -i igb0 -I 192.168.1.2
traceroute to 192.168.1.2 (192.168.1.2), 5 hops max, 48 byte packets
 1  * * *
 2  * * *
 3  * * *
 4  * * *
 5  * * *
root@opnsense:~ #

tcpdump at Site B shows no ICMP traffic so it'd appear to not reach there.

[reep]

You need a long hard look at your firewall rules, routes, gateways etc. Something in there somewhere is passing all traffic down the IPSEC connection. No easy answer without seeing more.

A normal connection will only route traffic for the far end network.

You have obviously added some rules somewhere - from the looks of the pfsense documentation for which you pasted a link I noticed this :

"Remote Network    0.0.0.0/0    This tells pfSense to route everything over this interface."

But you don't want ALL traffic to go over the interface, only some, or everything BUT your local traffic. So every time that the link comes up, all traffic goes down.

I think that is where your issue lies. How you solve it is another question beyond my pay grade :-)

B. Rgds
John

[Strykar]

Right, except we can't specify 'route all traffic except 192.168.1.0/24'.
I had reviewed my firewall rules before posting here and did not find anything related to the issue. I'd even tried disabling all rules to make troubleshooting easier.

And if the IPsec routing is done via kernel policy, pf isn't going to show it.
Thanks for your insights, I appreciate it.

[reep]

There is undoubtedly a solution.... but my knowledge hits a brick wall at this juncture.

I guess the first question is what firewall rules do you have set up ?

How do you manage the two subnets on Opnsense ?

[coom]

Hi,

I have exactly the same problem !

I've got 1 OPNSense at home which handle 5 LANs (10.0.1.0/24 to 10.0.5.0/24). Got another OPNSense on another box that handle only 1 LAN (192.168.1.0/24).

As Strykar, I had set up an IPSec VPN that permit all LAN to be viewed from the other side.

When I try to route all the WAN traffic, the first OPNSense route ALL traffic (even the one that is destinated to local LAN, like 10.0.1.0/24) through the IPSec tunnel, instead of routing the LAN he known.

Is there any workaround to route all WAN traffic through Site B while routing all local LAN through Site A?

Thanks in advance,

coom

PS: BTW, I'm on the latest release (17.7.1_2-amd64)

[horus]

hi,

have anyone a sollution for this issue?

need it very asap. 

kind regards