Topics

Hi, I am trying to set up a route-based (VTI) ipsec site-to-site tunnel between OPNsense (the A site) and VyOS (well, a Unifi USG-3P, the B site) using the "new > 23.1" setup using this guide: https://docs.opnsense.org/manual/how-tos/ipsec-s2s-conn-route.html

I have previously had a policy-based ipsec vpn working since about 19.1 but I think the problem referenced here hindered me (https://forum.opnsense.org/index.php?topic=30525.0) in that after 22.7 only one pair of subnets at each site could communicate at any one time and I had to manually take CHILD_SAs up and down.

I'm finally deciding to move to a route-based setup in the hope that with only one CHILD_SA for 0.0.0.0/0 -> 0.0.0.0/0 communication that it will work properly. I'd move to wireguard but there is no Unifi support for it on the USG-3P.

So, I have followed the guide for OPNsense for site A, and set up the other site B according to some VyOS/Unifi instructions.

I see what looks to me like a good SA setup (yes, both sides WAN interfaces are behind NAT on a 192* "DMZ" network, but the IKE and ESP seems to work):

Code Select Expand

# swanctl --list-sas
no files found matching '/usr/local/etc/strongswan.opnsense.d/*.conf'
ca24ede1-a222-4a6e-91dc-2c5720143fe6: #1, ESTABLISHED, IKEv2, 7d53711e7fea6ef4_i* 62a0c1809bc25095_r
  local  'A.A.A.A' @ 192.168.178.20[4500]
  remote '192.168.188.20' @ B.B.B.B[4500]
  AES_CBC-256/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_2048
  established 2172s ago, rekeying in 11323s
  afbe2c6c-fb46-4a1d-a9f4-facecc5ff2f9: #1, reqid 10, INSTALLED, TUNNEL-in-UDP, ESP:AES_CBC-256/HMAC_SHA1_96
    installed 2172s ago, rekeying in 1169s, expires in 1788s
    in  cc44dc3d,      0 bytes,     0 packets
    out c033ca54,      0 bytes,     0 packets
    local  0.0.0.0/0
    remote 0.0.0.0/0
Here is my OPNsense ipsec10 interface (do I have my 'tunnel inet' and 'inet' addresses mixed up?):

Code Select Expand

# ifconfig ipsec10
ipsec10: flags=1008011<UP,POINTOPOINT,MULTICAST,LOWER_UP> metric 0 mtu 1400
        options=0
        tunnel inet A.A.A.A --> 192.168.188.20
        inet 10.111.0.1 --> 10.111.0.2 netmask 0xfffffffc
        inet6 fe80::dead:beef:dead:beef%ipsec10 prefixlen 64 tentative scopeid 0x1a
        groups: ipsec
        reqid: 10
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
My A site's local networks are /24s contained in a 10.1.0.0/16 range and the B site is likewise in 10.2.0.0/16

Here's my A site routing table:

Code Select Expand

# netstat -rn4
Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
default            192.168.178.1      UGS            igb3
10.1.2.0/24        link#12            U      lagg0_vlan10
10.1.2.1           link#5             UHS             lo0
10.1.4.0/24        link#14            U      lagg0_vlan10
10.1.4.1           link#5             UHS             lo0
10.1.5.0/24        link#15            U      lagg0_vlan10
10.1.5.1           link#5             UHS             lo0
10.1.6.0/24        link#16            U      lagg0_vlan10
10.1.6.1           link#5             UHS             lo0
10.1.8.0/24        link#18            U      lagg0_vlan10
10.1.8.1           link#5             UHS             lo0
10.1.9.0/24        link#19            U      lagg0_vlan10
10.1.9.1           link#5             UHS             lo0
10.1.46.1          link#5             UHS             lo0
10.1.64.0/24       link#27            US            nat64
10.1.64.1          link#27            UH            nat64
10.2.0.0/16        10.111.0.2         UGS         ipsec10
10.111.0.1         link#5             UHS             lo0
10.111.0.2         link#26            UH          ipsec10
127.0.0.1          link#5             UH              lo0
192.168.1.0/24     link#1             U              igb0
192.168.1.10       link#5             UHS             lo0
192.168.178.0/24   link#4             U              igb3
192.168.178.20     link#5             UHS             lo0
So it looks as though I have set up the gateway and route to send 10.2.0.0/16 to the far side of the ipsec10 tunnel.

But it's not working.

I try a packet capture on ipsec10 and a ping to the far network (at the same time) and I get this:

Code Select Expand

$ ping -4 -S 10.1.8.1 10.2.4.1
PING 10.2.4.1 (10.2.4.1) from 10.1.8.1: 56 data bytes
ping: sendto: Network is down
ping: sendto: Network is down
ping: sendto: Network is down
^C
--- 10.2.4.1 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

# tcpdump -i ipsec10
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ipsec10, link-type NULL (BSD loopback), snapshot length 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel

I'm not sure how this should behave, but I get this:

Code Select Expand

# ping 10.111.0.1
PING 10.111.0.1 (10.111.0.1): 56 data bytes
64 bytes from 10.111.0.1: icmp_seq=0 ttl=64 time=0.329 ms
64 bytes from 10.111.0.1: icmp_seq=1 ttl=64 time=0.158 ms
^C
--- 10.111.0.1 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.158/0.243/0.329/0.086 ms
# ping 10.111.0.2
PING 10.111.0.2 (10.111.0.2): 56 data bytes
ping: sendto: Network is down
ping: sendto: Network is down
^C
--- 10.111.0.2 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss
When using policy-based ipsec I used to be able to tcpdump on the enc0 interface and see the encapsulated ESP traffic, but that also shows nothing.

Perhaps the far end needs to be configured with that tunnel address, but it looks like VyOS doesn't do it like that and their vti interface is simply unnumbered (see this old unaswered post https://forum.opnsense.org/index.php?topic=38062.0)

What am I doing wrongly?

Does anyone have such an OPNsense <-> VyOS setup working?

Hi,

I am migrating to IPv6 only. I have a /48 from my ISP which I have created a number of /57 local prefixes in which I am hosting various vms and physical machines.

All these hosts I have set to use SOII (the OpenBSD name for RFC7217 addresses). In short each host has a static listening address and fairly rapidly cycles through random(?) addresses in the /57 for outgoing traffic. I think Windows hosts do something similar so what I am asking here is I guess a fairly common use case.

Any incoming traffic through OPNsense is easy to add to allowlists in firewall rules as the addresses is static, but the outgoing traffic is causing me issues.

I would like to, on a host-by-host basis create allowlists and so firewall rules for specific outgoing traffic. So far I have tried allowing by src MAC address (even though it was in an "IPv6" rule); this worked for a while but then started blocking the traffic some hours later*. I have settled on allowing the entire /57 (I basically have a single host in each /57 I have created so far) but this seems unsatisfactory and not a long term solution.

Does anyone have any advice/war stories regarding the same? I thought I'd check here before I head upstream.

*I had a quick read around and filtering by MAC does seem a bad idea:
- Still true?: https://forum.opnsense.org/index.php?topic=2790.0
- Also seems like it could get bad performance: https://forums.freebsd.org/threads/filtering-by-mac-address.32841/

Hi,

I have dhcp active on all my subnets. This works well and provides the dns nameserver option for my dhcp clients to point to a couple of non-opnsense nameservers I use internally.

I am now configuring unbound to listen on just a single vlan/subnet.

I spotted this at the bottom of the plugin config page:

If the DNS Resolver is enabled, the DHCP service (if enabled) will automatically serve the LAN IP address as a DNS server to DHCP clients so they will use the DNS Resolver.

This is a pity because it doesn't just provide the nameservers for the vlan I'm targeting (also through ipv6 RAs, not just dhcp as mentioned in the quote) but it overrides my custom dhcp dns nameserver settings for _all_ other scopes.

Is this really necessary? Is it possible to change this behaviour?

Thanks

Hi,

I have ipv6 working well (I believe). I get a prefix on my opnsense host from my ISP (XS4ALL) and distribute it to downstream ipv6-only vlans using radvd.

I am trying to set up a non-transparent (opaque?) proxy in each of these vlans using the built-in squid proxy from opnsense. Here I have a problem.

I set the proxy to use the ipv6-only vlan interface as a "Proxy interface" but the squid.conf doesn't get generated correctly. I get:

Code Select Expand

# grep -E 'https_port|http_port' /usr/local/etc/squid/squid.conf
http_port 127.0.0.1:3128 intercept
http_port [::1]:3128 intercept
http_port 192.168.1.10:3128 
http_port 10.1.8.1:3128 
http_port 10.1.9.1:3128 
http_port 10.1.6.1:3128 
http_port 10.1.2.1:3128 
http_port 10.1.4.1:3128
http_port :3128   <--- problem line
http_port 10.1.5.1:3128 

You'll see some other ipv4-only vlan interface addresses there.

My ipv6-only vlan interface looks like this:

Code Select Expand


# ifconfig lagg0_vlan640
lagg0_vlan640: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 00:02:b0:1a:68:39
inet6 2001:dead:beef:a8:202:b0ff:fe1a:6839 prefixlen 64
inet6 fe80::1:1%lagg0_vlan640 prefixlen 64 scopeid 0x14
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
vlan: 640 vlanpcp: 0 parent interface: lagg0
groups: vlan


If I put that address in the squid.conf replacing the 'problem line' above it seems to work correctly, clients in that vlan can use the proxy just fine:

Code Select Expand


# netstat -an|grep 3128
tcp4       0      0 10.1.2.1.3128          10.1.2.23.33264        ESTABLISHED
tcp4       0      0 10.1.5.1.3128          *.*                    LISTEN
tcp6       0      0 2001:dead:beef:a8.3128 *.*                    LISTEN
tcp4       0      0 10.1.4.1.3128          *.*                    LISTEN
tcp4       0      0 10.1.2.1.3128          *.*                    LISTEN
tcp4       0      0 10.1.6.1.3128          *.*                    LISTEN
tcp4       0      0 10.1.9.1.3128          *.*                    LISTEN
tcp4       0      0 10.1.8.1.3128          *.*                    LISTEN
tcp4       0      0 192.168.1.10.3128      *.*                    LISTEN
tcp6       0      0 ::1.3128               *.*                    LISTEN
tcp4       0      0 127.0.0.1.3128         *.*                    LISTEN


Could it be that the squid.conf generator just isn't inserting that address correctly? I use the unbound plugin too and it puts that ipv6 address in its config file just fine.

Thanks

Hi,

I'm successfully running a a wan interface with a DHCPv6 Config File Override that gets a /57 prefix from an upstream router and configures some internal vlan interfaces.

This works nicely.

My override file, /var/etc/dhcp6c_wan.override.conf looks like this:

Code Select Expand

interface igb3 {
  send ia-na 0; # request stateful address
  send ia-pd 0; # request prefix delegation
  request domain-name-servers;
  request domain-name;
  script "/var/etc/dhcp6c_wan_script.sh"; # we'd like some nameservers please
};
id-assoc na 0 { };
id-assoc pd 0 {
  prefix ::/57 infinity;
  prefix-interface lagg0_vlan620 {
    sla-id 20;
    sla-len 7;
  };
  prefix-interface lagg0_vlan640 {
    sla-id 40;
    sla-len 7;
  };
  prefix-interface lagg0_vlan660 {
    sla-id 60;
    sla-len 7;
  };
  prefix-interface lagg0_vlan680 {
    sla-id 80;
    sla-len 7;
  };
};

This is basically an extension from what gets created from the "Basic" mode, only I have:
1. A /57 prefix. My upstream router gets a /48 from my ISP but then will only hand out a /57 and no other. This 57 is not a chooseable option in the "Basic" interface, it goes from 56 to 60. Can this be fixed?
2. sla-ids that are more than 0xf, which is the maximum "allowed" in the "Track Interface" part of each interface's config. Can this be fixed too?

Thanks,
Ben

Hi,

I'm having an intermittent problem with (I think) my Opnsense 17.7.7 router/firewall. Apologies up front if I've missed some other post describing exactly this problem.

So, I have this Opnsense firewall (192.168.178.20) and a rpi3 (192.168.178.21) sitting on my flat network behind my ISP's CPE (192.168.178.1). I then have my real home network behind the Opnsense firewall.

I'm trying to terminal a little bit of HTTPS traffic from the outside world on the rpi3 and send it through my firewall as plain HTTP via a port forward from 192.168.178.20:1644/tcp to my Nextcloud server (10.1.6.44:80/tcp) in my real home network.

This works half the time. I'll show you what I mean.

Here's a successful wget from my docker instance on the rpi3:

Code Select Expand

root@b3f74cc5c974:/$ wget --header 'Host: myserver.com' http://192.168.178.20:1644/blah1.php
Connecting to 192.168.178.20:1644 (192.168.178.20:1644)
wget: server returned error: HTTP/1.1 404 Not Found
root@b3f74cc5c974:/$ netstat -tupa
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 localhost:9000          0.0.0.0:*               LISTEN      313/php-fpm.conf)
tcp        0      0 0.0.0.0:11211           0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:http            0.0.0.0:*               LISTEN      309/nginx.conf
tcp        0      0 0.0.0.0:https           0.0.0.0:*               LISTEN      309/nginx.conf
tcp        0      0 b3f74cc5c974:37972      myopnsensebox.com:1644  TIME_WAIT   -
tcp        0      0 :::11211                :::*                    LISTEN      -
udp        0      0 0.0.0.0:11211           0.0.0.0:*                           -
udp        0      0 :::11211                :::*                                -

And the Nextcloud server's access.log:

Code Select Expand

myserver.com 192.168.178.21 - - [30/Oct/2017:19:46:43 +0100] "GET /blah1.php HTTP/1.1" 404 0

So, the page isn't there, but they're at least they're on speaking terms.

Weird thing is that just a few seconds later I try again and get this:

Code Select Expand

root@b3f74cc5c974:/$ wget --header 'Host: myserver.com' http://192.168.178.20:1644/blah2.php
Connecting to 192.168.178.20:1644 (192.168.178.20:1644)
wget: can't connect to remote host (192.168.178.20): Connection refused

What's happening?

I did a traffic capture on the outside (192.168.178.20) and inside (10.1.6.1) interfaces of the firewall (simple text captures attached), but the crux of it is that the outside interface is resetting the second connection immediately like so:

Code Select Expand

19:47:40.630906 IP 192.168.178.21.37974 > 192.168.178.20.1644: Flags [S], seq 4153658341, win 29200, options [mss 1460,sackOK,TS val 10306995 ecr 0,nop,wscale 7], length 0
19:47:40.631670 IP 192.168.178.20.1644 > 192.168.178.21.37974: Flags [R.], seq 0, ack 4153658342, win 0, length 0

This is not showing up in any logs I can see, where can I look for what is doing this??

Many thanks,
Ben

Hi,

I'm pretty new to OPNsense in particular and FreeBSD in general. I decided to have a go at it since a) I heard it was a great router distribution and b) it uses a strong host model and I am sick and tired of dealing with traffic leaking in/out the wrong interfaces in Linux.

Although at the moment though I wish it would leak a little more :)

Let me describe my problem a little...

I have a host with 4 physical interfaces: http://www.fit-pc.com/web/products/fitlet/fitlet-x/

I would like to keep it simple and have the default LAN and WAN interfaces plumbed to 2 of the physical interfaces, igb0 and igb3. This works well.

The problem comes in when I try to set up bridge interfaces that bridge together both physical and VLAN interfaces (to act as a sort of Cisco-style BVI). My issue is that VLAN tagged traffic is going missing somewhere after it enters a physical interface and doesn't appear in a tcpdump on the bridge interface or the VLAN interface in question. The problem seems similar to the one mentioned here: https://redmine.pfsense.org/issues/2613

Let me show you part of my setup:

Code Select Expand

root@OPNsense:~ # ifconfig bridge0
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:92:47:45:cc:00
        inet 10.1.6.1 netmask 0xffffff00 broadcast 10.1.6.255
        nd6 options=1<PERFORMNUD>
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: igb2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 55
        member: igb1_vlan1016 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 15 priority 128 path cost 20000
root@OPNsense:~ # ifconfig igb2
igb2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=500bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO>
        ether 00:01:c0:1a:67:3a
        inet6 fe80::201:c0ff:fe1a:673a%igb2 prefixlen 64 scopeid 0x3
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
root@OPNsense:~ # ifconfig igb1_vlan1016
igb1_vlan1016: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=3<RXCSUM,TXCSUM>
        ether 00:01:c0:1a:67:39
        inet6 fe80::201:c0ff:fe1a:6739%igb1_vlan1016 prefixlen 64 scopeid 0xf
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        vlan: 1016 parent interface: igb1
root@OPNsense:~ # ifconfig bridge3
bridge3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:92:47:45:cc:03
        inet 10.1.1.1 netmask 0xffffff00 broadcast 10.1.1.255
        nd6 options=1<PERFORMNUD>
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: igb1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 55
root@OPNsense:~ # ifconfig igb1
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=500bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO>
        ether 00:01:c0:1a:67:39
        inet6 fe80::201:c0ff:fe1a:6739%igb1 prefixlen 64 scopeid 0x2
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active


To help diagnose this, I have performed some packet captures on the igb1, bridge0 and igb1_vlan1016 interfaces. These are attached (I had to filter the igb1 trace of other irrelevant traffic, I think all the relevant packets are included).

You'll see that the client host on vlan1016 is able to perform DHCP correctly (getting the 10.1.6.101 address), ARP works, an ICMP ping request gets sent out correctly, but the ICMP ping reply never makes it back to bridge0 or igb1_vlan1016. For further information the client is a guest VM running on a Linux KVM hypervisor patched into igb1 via a dumb/unmanaged switch. I believe the hypervisor is set up correctly and the ethernet frames are being correctly 802.1q labelled by it for vlan 1016 (take a look at the attached packet traces for details).

I have created allow all IPv4 type rules on all interfaces, and even a floating rule in case pf might be filtering this traffic out.. I can't see any blocked traffic in the logs, so I don't think that pf is dropping the traffic.

I guessed that there may be some ethernet filtering going on, so I altered some tunables, specifically

Code Select Expand

net.link.bridge.pfil_member=0
net.link.bridge.pfil_bridge=1
but this didn't seem to help.

Does anyone have an explanation for this behaviour or any advice on where I can look next?