Messages

Hi, I am trying to set up a route-based (VTI) ipsec site-to-site tunnel between OPNsense (the A site) and VyOS (well, a Unifi USG-3P, the B site) using the "new > 23.1" setup using this guide: https://docs.opnsense.org/manual/how-tos/ipsec-s2s-conn-route.html

I have previously had a policy-based ipsec vpn working since about 19.1 but I think the problem referenced here hindered me (https://forum.opnsense.org/index.php?topic=30525.0) in that after 22.7 only one pair of subnets at each site could communicate at any one time and I had to manually take CHILD_SAs up and down.

I'm finally deciding to move to a route-based setup in the hope that with only one CHILD_SA for 0.0.0.0/0 -> 0.0.0.0/0 communication that it will work properly. I'd move to wireguard but there is no Unifi support for it on the USG-3P.

So, I have followed the guide for OPNsense for site A, and set up the other site B according to some VyOS/Unifi instructions.

I see what looks to me like a good SA setup (yes, both sides WAN interfaces are behind NAT on a 192* "DMZ" network, but the IKE and ESP seems to work):

Code Select Expand

# swanctl --list-sas
no files found matching '/usr/local/etc/strongswan.opnsense.d/*.conf'
ca24ede1-a222-4a6e-91dc-2c5720143fe6: #1, ESTABLISHED, IKEv2, 7d53711e7fea6ef4_i* 62a0c1809bc25095_r
  local  'A.A.A.A' @ 192.168.178.20[4500]
  remote '192.168.188.20' @ B.B.B.B[4500]
  AES_CBC-256/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_2048
  established 2172s ago, rekeying in 11323s
  afbe2c6c-fb46-4a1d-a9f4-facecc5ff2f9: #1, reqid 10, INSTALLED, TUNNEL-in-UDP, ESP:AES_CBC-256/HMAC_SHA1_96
    installed 2172s ago, rekeying in 1169s, expires in 1788s
    in  cc44dc3d,      0 bytes,     0 packets
    out c033ca54,      0 bytes,     0 packets
    local  0.0.0.0/0
    remote 0.0.0.0/0
Here is my OPNsense ipsec10 interface (do I have my 'tunnel inet' and 'inet' addresses mixed up?):

Code Select Expand

# ifconfig ipsec10
ipsec10: flags=1008011<UP,POINTOPOINT,MULTICAST,LOWER_UP> metric 0 mtu 1400
        options=0
        tunnel inet A.A.A.A --> 192.168.188.20
        inet 10.111.0.1 --> 10.111.0.2 netmask 0xfffffffc
        inet6 fe80::dead:beef:dead:beef%ipsec10 prefixlen 64 tentative scopeid 0x1a
        groups: ipsec
        reqid: 10
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
My A site's local networks are /24s contained in a 10.1.0.0/16 range and the B site is likewise in 10.2.0.0/16

Here's my A site routing table:

Code Select Expand

# netstat -rn4
Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
default            192.168.178.1      UGS            igb3
10.1.2.0/24        link#12            U      lagg0_vlan10
10.1.2.1           link#5             UHS             lo0
10.1.4.0/24        link#14            U      lagg0_vlan10
10.1.4.1           link#5             UHS             lo0
10.1.5.0/24        link#15            U      lagg0_vlan10
10.1.5.1           link#5             UHS             lo0
10.1.6.0/24        link#16            U      lagg0_vlan10
10.1.6.1           link#5             UHS             lo0
10.1.8.0/24        link#18            U      lagg0_vlan10
10.1.8.1           link#5             UHS             lo0
10.1.9.0/24        link#19            U      lagg0_vlan10
10.1.9.1           link#5             UHS             lo0
10.1.46.1          link#5             UHS             lo0
10.1.64.0/24       link#27            US            nat64
10.1.64.1          link#27            UH            nat64
10.2.0.0/16        10.111.0.2         UGS         ipsec10
10.111.0.1         link#5             UHS             lo0
10.111.0.2         link#26            UH          ipsec10
127.0.0.1          link#5             UH              lo0
192.168.1.0/24     link#1             U              igb0
192.168.1.10       link#5             UHS             lo0
192.168.178.0/24   link#4             U              igb3
192.168.178.20     link#5             UHS             lo0
So it looks as though I have set up the gateway and route to send 10.2.0.0/16 to the far side of the ipsec10 tunnel.

But it's not working.

I try a packet capture on ipsec10 and a ping to the far network (at the same time) and I get this:

Code Select Expand

$ ping -4 -S 10.1.8.1 10.2.4.1
PING 10.2.4.1 (10.2.4.1) from 10.1.8.1: 56 data bytes
ping: sendto: Network is down
ping: sendto: Network is down
ping: sendto: Network is down
^C
--- 10.2.4.1 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

# tcpdump -i ipsec10
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ipsec10, link-type NULL (BSD loopback), snapshot length 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel

I'm not sure how this should behave, but I get this:

Code Select Expand

# ping 10.111.0.1
PING 10.111.0.1 (10.111.0.1): 56 data bytes
64 bytes from 10.111.0.1: icmp_seq=0 ttl=64 time=0.329 ms
64 bytes from 10.111.0.1: icmp_seq=1 ttl=64 time=0.158 ms
^C
--- 10.111.0.1 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.158/0.243/0.329/0.086 ms
# ping 10.111.0.2
PING 10.111.0.2 (10.111.0.2): 56 data bytes
ping: sendto: Network is down
ping: sendto: Network is down
^C
--- 10.111.0.2 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss
When using policy-based ipsec I used to be able to tcpdump on the enc0 interface and see the encapsulated ESP traffic, but that also shows nothing.

Perhaps the far end needs to be configured with that tunnel address, but it looks like VyOS doesn't do it like that and their vti interface is simply unnumbered (see this old unaswered post https://forum.opnsense.org/index.php?topic=38062.0)

What am I doing wrongly?

Does anyone have such an OPNsense <-> VyOS setup working?

Place hosts that share an outbound policy in a common network/VLAN and ignore host addresses. Filtering by address does not scale and is easily spoofed.

This seems to be the most straightforward way, thanks for confirming.

Also, please do not use /57 as LAN networks, you must (almost) always use /64 for IPv6 networks.

My bad, it was late at night when I posted. I have a number of /64s in the /57 that's been delegated to my OPNsense router by my CPE router (that has the /48).

Hi,

I am migrating to IPv6 only. I have a /48 from my ISP which I have created a number of /57 local prefixes in which I am hosting various vms and physical machines.

All these hosts I have set to use SOII (the OpenBSD name for RFC7217 addresses). In short each host has a static listening address and fairly rapidly cycles through random(?) addresses in the /57 for outgoing traffic. I think Windows hosts do something similar so what I am asking here is I guess a fairly common use case.

Any incoming traffic through OPNsense is easy to add to allowlists in firewall rules as the addresses is static, but the outgoing traffic is causing me issues.

I would like to, on a host-by-host basis create allowlists and so firewall rules for specific outgoing traffic. So far I have tried allowing by src MAC address (even though it was in an "IPv6" rule); this worked for a while but then started blocking the traffic some hours later*. I have settled on allowing the entire /57 (I basically have a single host in each /57 I have created so far) but this seems unsatisfactory and not a long term solution.

Does anyone have any advice/war stories regarding the same? I thought I'd check here before I head upstream.

*I had a quick read around and filtering by MAC does seem a bad idea:
- Still true?: https://forum.opnsense.org/index.php?topic=2790.0
- Also seems like it could get bad performance: https://forums.freebsd.org/threads/filtering-by-mac-address.32841/

Hi,

I have dhcp active on all my subnets. This works well and provides the dns nameserver option for my dhcp clients to point to a couple of non-opnsense nameservers I use internally.

I am now configuring unbound to listen on just a single vlan/subnet.

I spotted this at the bottom of the plugin config page:

If the DNS Resolver is enabled, the DHCP service (if enabled) will automatically serve the LAN IP address as a DNS server to DHCP clients so they will use the DNS Resolver.

This is a pity because it doesn't just provide the nameservers for the vlan I'm targeting (also through ipv6 RAs, not just dhcp as mentioned in the quote) but it overrides my custom dhcp dns nameserver settings for _all_ other scopes.

Is this really necessary? Is it possible to change this behaviour?

Thanks

Hi,

I have ipv6 working well (I believe). I get a prefix on my opnsense host from my ISP (XS4ALL) and distribute it to downstream ipv6-only vlans using radvd.

I am trying to set up a non-transparent (opaque?) proxy in each of these vlans using the built-in squid proxy from opnsense. Here I have a problem.

I set the proxy to use the ipv6-only vlan interface as a "Proxy interface" but the squid.conf doesn't get generated correctly. I get:

Code Select Expand

# grep -E 'https_port|http_port' /usr/local/etc/squid/squid.conf
http_port 127.0.0.1:3128 intercept
http_port [::1]:3128 intercept
http_port 192.168.1.10:3128 
http_port 10.1.8.1:3128 
http_port 10.1.9.1:3128 
http_port 10.1.6.1:3128 
http_port 10.1.2.1:3128 
http_port 10.1.4.1:3128
http_port :3128   <--- problem line
http_port 10.1.5.1:3128 

You'll see some other ipv4-only vlan interface addresses there.

My ipv6-only vlan interface looks like this:

Code Select Expand


# ifconfig lagg0_vlan640
lagg0_vlan640: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 00:02:b0:1a:68:39
inet6 2001:dead:beef:a8:202:b0ff:fe1a:6839 prefixlen 64
inet6 fe80::1:1%lagg0_vlan640 prefixlen 64 scopeid 0x14
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
vlan: 640 vlanpcp: 0 parent interface: lagg0
groups: vlan


If I put that address in the squid.conf replacing the 'problem line' above it seems to work correctly, clients in that vlan can use the proxy just fine:

Code Select Expand


# netstat -an|grep 3128
tcp4       0      0 10.1.2.1.3128          10.1.2.23.33264        ESTABLISHED
tcp4       0      0 10.1.5.1.3128          *.*                    LISTEN
tcp6       0      0 2001:dead:beef:a8.3128 *.*                    LISTEN
tcp4       0      0 10.1.4.1.3128          *.*                    LISTEN
tcp4       0      0 10.1.2.1.3128          *.*                    LISTEN
tcp4       0      0 10.1.6.1.3128          *.*                    LISTEN
tcp4       0      0 10.1.9.1.3128          *.*                    LISTEN
tcp4       0      0 10.1.8.1.3128          *.*                    LISTEN
tcp4       0      0 192.168.1.10.3128      *.*                    LISTEN
tcp6       0      0 ::1.3128               *.*                    LISTEN
tcp4       0      0 127.0.0.1.3128         *.*                    LISTEN


Could it be that the squid.conf generator just isn't inserting that address correctly? I use the unbound plugin too and it puts that ipv6 address in its config file just fine.

Thanks

Excellent, looks like it will soon be all sorted. I will just make a bug report next time.

As you say, I had selected Basic with a prefix of /60 before I chose the Config File Override, so yes, I only got the choice of 0 to f. Selecting Basic and /56 and applying it prior to selecting Config File Override allowed me to "Enter a hexadecimal value between 0 and ff here," for the prefix id in the track interface section.

Thanks for the quick response.

Hi,

I'm successfully running a a wan interface with a DHCPv6 Config File Override that gets a /57 prefix from an upstream router and configures some internal vlan interfaces.

This works nicely.

My override file, /var/etc/dhcp6c_wan.override.conf looks like this:

Code Select Expand

interface igb3 {
  send ia-na 0; # request stateful address
  send ia-pd 0; # request prefix delegation
  request domain-name-servers;
  request domain-name;
  script "/var/etc/dhcp6c_wan_script.sh"; # we'd like some nameservers please
};
id-assoc na 0 { };
id-assoc pd 0 {
  prefix ::/57 infinity;
  prefix-interface lagg0_vlan620 {
    sla-id 20;
    sla-len 7;
  };
  prefix-interface lagg0_vlan640 {
    sla-id 40;
    sla-len 7;
  };
  prefix-interface lagg0_vlan660 {
    sla-id 60;
    sla-len 7;
  };
  prefix-interface lagg0_vlan680 {
    sla-id 80;
    sla-len 7;
  };
};

This is basically an extension from what gets created from the "Basic" mode, only I have:
1. A /57 prefix. My upstream router gets a /48 from my ISP but then will only hand out a /57 and no other. This 57 is not a chooseable option in the "Basic" interface, it goes from 56 to 60. Can this be fixed?
2. sla-ids that are more than 0xf, which is the maximum "allowed" in the "Track Interface" part of each interface's config. Can this be fixed too?

Thanks,
Ben

So, I gave up and used the HAProxy plugin instead. Nice to have learned something new.

Hi,

I'm having an intermittent problem with (I think) my Opnsense 17.7.7 router/firewall. Apologies up front if I've missed some other post describing exactly this problem.

So, I have this Opnsense firewall (192.168.178.20) and a rpi3 (192.168.178.21) sitting on my flat network behind my ISP's CPE (192.168.178.1). I then have my real home network behind the Opnsense firewall.

I'm trying to terminal a little bit of HTTPS traffic from the outside world on the rpi3 and send it through my firewall as plain HTTP via a port forward from 192.168.178.20:1644/tcp to my Nextcloud server (10.1.6.44:80/tcp) in my real home network.

This works half the time. I'll show you what I mean.

Here's a successful wget from my docker instance on the rpi3:

Code Select Expand

root@b3f74cc5c974:/$ wget --header 'Host: myserver.com' http://192.168.178.20:1644/blah1.php
Connecting to 192.168.178.20:1644 (192.168.178.20:1644)
wget: server returned error: HTTP/1.1 404 Not Found
root@b3f74cc5c974:/$ netstat -tupa
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 localhost:9000          0.0.0.0:*               LISTEN      313/php-fpm.conf)
tcp        0      0 0.0.0.0:11211           0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:http            0.0.0.0:*               LISTEN      309/nginx.conf
tcp        0      0 0.0.0.0:https           0.0.0.0:*               LISTEN      309/nginx.conf
tcp        0      0 b3f74cc5c974:37972      myopnsensebox.com:1644  TIME_WAIT   -
tcp        0      0 :::11211                :::*                    LISTEN      -
udp        0      0 0.0.0.0:11211           0.0.0.0:*                           -
udp        0      0 :::11211                :::*                                -

And the Nextcloud server's access.log:

Code Select Expand

myserver.com 192.168.178.21 - - [30/Oct/2017:19:46:43 +0100] "GET /blah1.php HTTP/1.1" 404 0

So, the page isn't there, but they're at least they're on speaking terms.

Weird thing is that just a few seconds later I try again and get this:

Code Select Expand

root@b3f74cc5c974:/$ wget --header 'Host: myserver.com' http://192.168.178.20:1644/blah2.php
Connecting to 192.168.178.20:1644 (192.168.178.20:1644)
wget: can't connect to remote host (192.168.178.20): Connection refused

What's happening?

I did a traffic capture on the outside (192.168.178.20) and inside (10.1.6.1) interfaces of the firewall (simple text captures attached), but the crux of it is that the outside interface is resetting the second connection immediately like so:

Code Select Expand

19:47:40.630906 IP 192.168.178.21.37974 > 192.168.178.20.1644: Flags [S], seq 4153658341, win 29200, options [mss 1460,sackOK,TS val 10306995 ecr 0,nop,wscale 7], length 0
19:47:40.631670 IP 192.168.178.20.1644 > 192.168.178.21.37974: Flags [R.], seq 0, ack 4153658342, win 0, length 0

This is not showing up in any logs I can see, where can I look for what is doing this??

Many thanks,
Ben

Hi Franco,

I will look into it further when I can, but I think for now I will give in and buy a managed switch and do the plumbing downstream of my opnsense host.

I'm guessing there is no analogue of a Cisco BVI for opnsense/freebsd because bridging vlans on 2 physical interfaces (say igb1_vlan1016 and igb2_vlan1016) will get the kernel involved in layer 2 switching and that will never be very nice unless there's some hardware integration like on Cisco (and other) devices. Right?

Still, it's something that would be nice to have working in the end.

Many thanks for your time.

Hi, I wonder if there are any further thoughts from anyone about this one.

Just to summarize, my problems are like so:

Code Select Expand

physical = fine
physical ---> vlan = fine
physical ---> bridge = fine
physical -|-> vlan ---> bridge = not fine

In that everything works apart from certain traffic going missing (at least according to tcpdump) between physical and vlan interfaces in the last configuration. I say 'going missing' instead of 'being dropped' as I can't find any trace of missing packets with netstat, ifconfig, pf logs or any other tools I can think of.

This testing is all done on a single physical interface (although the whole point is to incorporate others).

I turned off all hardware offloards (vlanhwfilter etc.) without luck.

Where can I look for documentation/hints to go forward?

Worth trying the same setup in pfsense or vanilla freebsd to see if I can recreate it?

Thanks

I thought I would also try for fun to take the vlans out of the question, and the bridged interface does indeed work correctly with a device plugged into igb2.

Code Select Expand

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:7e:e3:17:05:00
inet 10.1.6.1 netmask 0xffffff00 broadcast 10.1.6.255
nd6 options=1<PERFORMNUD>
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: igb2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
        ifmaxaddr 0 port 3 priority 128 path cost 55

Ok, I have given it a go.

Unfortunately my packet captures are pretty much the same, with all the traffic from the host sitting on vlan1016 hitting igb1 but getting no further. As before traffic out to the host (an icmp ping) reaches the host, and the host replies, but again, that traffic gets lost somewhere after it hits igb1.

I am pretty sure the firewall is not to blame here (although my pf-log-reading-fu might need some sharpening). Default drop rule logging is enabled and I have no other drop rules.

I also tried setting the bridge sysctl parameters back to 'default' but still the same.

Am I conceptually doing something wrong by setting up these bridges as my only addressed interfaces (apart from WAN and LAN)?

Sorry, vga thanks.