Messages

Red herring. Zenarmor was the culprit.

Hi,

I've set up an openvpn server, and it dishes out both IPv4 and IPv6 addresses. In the Advanced section of the server I've added 'push "redirect-gateway ipv6"', and so far all seems fine. Clients connect, they get both IPv4 and IPv6 addresses assigned, and on IPv4 things are all good. I set up a NAT to the WAN, they can browse the Internet, and connect to both the LAN and DMZ networks internally.
On IPv6: not so much. I can see traffic coming in from the clients using tcpdump, but it's dropped on the firewall without a trace in the logs.
The Firewall rule under OpenVPN has 1 simple rule, to allow everything that comes in from the clients: (IPv4+6, pass). Logging is enabled, and I can see log-entries for IPv4 traffic, just nothing for IPv6.

Where oh where does one go to analyse further?

Thanks for any help in advance.

Ferry.

PS. The networks are all dual stack, including the WAN connection.

I hacked it temporarily by running a script on Gateway events, so that when the Gateway of the underlying WireGuard tunnel is up, I manually add the right ifconfig tunnel command... Not really ideal but it works.

No, not a peep.

Made an identical setup using IPSec instead of WireGuard, and after a little while the GRE tunnel does come up automatically, from time to time. (Works after some reboots, not after others)

Hi,

I did yes, see attached.

Hi,

I run a couple of GRE tunnels over WireGuard VPNs. Works fine, except after a reboot. When I look at one of the tunnels after a boot, it looks like this:

root@OPNsense:~ # ifconfig gre1
gre1: flags=8011<UP,POINTOPOINT,MULTICAST> metric 0 mtu 1396
        options=80000<LINKSTATE>
        inet 10.1.11.6 --> 10.1.11.5 netmask 0xfffffffc
        inet6 fe80::20c:29ff:fea3:3bc8%gre1 prefixlen 64 tentative scopeid 0xa
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: gre

i.e. Link not set up. When, in the OPNsense interface, I go to this GRE tunnel, edit it, hit Save and Apply (without changing any settings), things start working and ifconfig shows me the below:

root@OPNsense:~ # ifconfig gre1
gre1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1396
        options=80000<LINKSTATE>
        tunnel inet 10.1.9.2 --> 10.1.9.1
        inet 10.1.11.6 --> 10.1.11.5 netmask 0xfffffffc
        inet6 fe80::20c:29ff:fea3:3bc8%gre1 prefixlen 64 scopeid 0xa
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: gre

I'm not entirely sure what is happening here. Could it be a timing issue, in that the GRE tunnel can't be set up until routing entries exist after the WireGuard tunnel has negotiated? If so, is there any way I can trigger things after WireGuard has established its link?

Chees,
Ferry.

PS, further to this, some dmesg output from the boot:

Code Select Expand

Trying to mount root from ufs:/dev/gpt/rootfs [rw,noatime]...
random: unblocking device.
VMware memory control driver initialized
aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard
em2: promiscuous mode enabled
carp: 6@em2: INIT -> BACKUP (initialization complete)
em3: promiscuous mode enabled
carp: 7@em3: INIT -> BACKUP (initialization complete)
em1: promiscuous mode enabled
carp: 5@em1: INIT -> BACKUP (initialization complete)
ifa_maintain_loopback_route: deletion failed for interface em1: 3
ifa_maintain_loopback_route: deletion failed for interface em1: 3
ifa_maintain_loopback_route: deletion failed for interface em1: 3
carp: 5@em1: BACKUP -> INIT (hardware interface up)
carp: 5@em1: INIT -> BACKUP (initialization complete)
ifa_maintain_loopback_route: deletion failed for interface em2: 3
ifa_maintain_loopback_route: deletion failed for interface em2: 3
ifa_maintain_loopback_route: deletion failed for interface em2: 3
carp: 6@em2: BACKUP -> INIT (hardware interface up)
carp: 6@em2: INIT -> BACKUP (initialization complete)
ifa_maintain_loopback_route: deletion failed for interface em3: 3
ifa_maintain_loopback_route: deletion failed for interface em3: 3
ifa_maintain_loopback_route: deletion failed for interface em3: 3
carp: 7@em3: BACKUP -> INIT (hardware interface up)
carp: 7@em3: INIT -> BACKUP (initialization complete)
gre0: link state changed to DOWN
gre1: link state changed to DOWN
pflog0: promiscuous mode enabled
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
tun0: link state changed to UP
tun0: changing name to 'wg0'
tun1: link state changed to UP
tun1: changing name to 'wg1'
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
carp: 7@em3: BACKUP -> MASTER (preempting a slower master)
carp: 6@em2: BACKUP -> MASTER (preempting a slower master)
carp: 5@em1: BACKUP -> MASTER (preempting a slower master)
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
Showing both gre0 and gre1 being down.

Then, when I edit, save and apply both the gre tunnels, the following appears:

Code Select Expand

gre0: link state changed to DOWN
gre0: link state changed to UP
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
gre1: link state changed to DOWN
gre1: link state changed to UP
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled

Ok, I'll try. Attached is a quick drawing of the different layers:

Internet obviously public IPs, on top of that I have the WireGuard tunnel between the two public IPs. It tunnels 10.1.10.0/30, where the OPNsense box has 10.1.10.2 and the Linux server 10.1.10.1. These are also the IPs of the interfaces (WG on OPNsense and wg0 on Linux).
On top of that I created a GRE tunnel between 10.1.10.1 and 10.1.10.2, and it routes 10.1.11.0/30. Linux gets 10.1.11.1 and OPNSense gets 10.1.11.2. These are also the IPs of the respective interfaces.
On the Linux server, I have a static route that points the LAN (192.168.1.0/24 for instance) to 10.1.11.2 (GRE endpoint on OPNsense) (ip route add 192.168.1.0/24 via 10.1.11.2 dev gre0)

The the last trick in the toolbox was to create a Gateway on OPNsense. I called it BogusGW, Interface WGGRE, IP is 8.8.8.8, checked 'Far Gateway' and 'Disable Gateway Monitoring'.
Then in the Interface section, on the WGGRE Interface, I use BogusGW as the IPv4 Upstream Gateway. This will enable the reply-to statements in the rules and then things magically start to work.

Hope this helps!

Update: Don't forget to drop the MTU to something like 1396 on the GRE interface to prevent fragmentation.

No problem, I keep going at this.

So, I read that in order for an interface to be considered a WAN interface, there needs to be a gateway assigned to the interface. To do this, I created a dummy gateway with IP 8.8.8.8 (Google must be sick at this stage of people using this IP for random stuff), and in the Interface settings for my gre0 tunnel I assigned this as the gateway.
Now OPNsense considers this a WAN interface, and the rules include the reply-to tags.

Things work....

Going to attempt a reboot now and see if the world explodes...

Update: Different problem, but I'll fix that.
So the reply-to issue works after a reboot, that's great. The GRE tunnel doesn't come up after a reboot though, that's something I'll need to look at. I have to go into the GRE tunnel settings, change nothing, and hit Save and Apply, and then it works. Must have something to do with the timing of the WireGuard VPN and the tunnel coming up at boot time. Could try with an IPSec tunnel as that's natively supported.
But the good news is that once the GRE tunnel is up, we're in business. The gateway on the Interface was the trick apparently.

Well, getting there. I manually inserted "reply-to ( gre0 10.1.11.1 )" in the forwarding rule, and reloaded the rules with "pfctl -f /tmp/rules.debug". And that does the trick.

Code Select Expand

pass in log quick on gre0 reply-to ( gre0 10.1.11.1 ) inet proto tcp from {any} to $IP_Lab port {25} keep state label "d96d282773c24f3269134267e65aad05"

So I think my question is going to become a lot simpler. Am I correct in assuming that these reply-to tags are only inserted on WAN-type interfaces? If that's correct, how do I get OPNsense to think that my gre0 interface is a WAN-type interface?

Ok, I think I just might have to give up on this and go create a Linux router just for the GRE/WG tunnel.
I did find this in the pf.conf man page, but not sure how this is handled by OPNsense:
(The reply-to is interesting)

Code Select Expand


ROUTING
     If a packet matches a rule with a route option set, the packet filter
     will route the packet according to the type of route option.  When such a
     rule creates state, the route option is also applied to all packets
     matching the same connection.

     route-to
   The route-to option routes the packet to the specified interface
   with an optional address for the next hop.  When a route-to rule
   creates state, only packets that pass in the same direction as the
   filter rule specifies will be routed in this way.  Packets passing
   in the opposite direction (replies) are not affected and are routed
   normally.

     reply-to
   The reply-to option is similar to route-to, but routes packets that
   pass in the opposite direction (replies) to the specified inter-
   face.  Opposite direction is only defined in the context of a state
   entry, and reply-to is useful only in rules that create state.  It
   can be used on systems with multiple external connections to route
   all outgoing packets of a connection through the interface the in-
   coming connection arrived through (symmetric routing enforcement).

     dup-to
   The dup-to option creates a duplicate of the packet and routes it
   like route-to.  The original packet gets routed as it normally
   would.


If I run a pfctl -s rules on the CLI, I don't see any reply-to. Could this be the problem?

Ok, definitely some strange behaviour here. Possibly my fault, but I wouldn't know what/how. I added a few more tunnels, didn't change any of the NAT rules, and now OPNsense decided to send return traffic out on the WAN instead of the GRE tunnel again.
I simply must be doing something wrong, but I have no idea what...

I'm not sure if it's a bug or by design. PfSense acts the same way, which makes me think it's a kernel 'thing'.  I'm just not sure why it does work with OpenVPN, at least on pfSense.

I'm going to continue with the double NAT thing for now, hoping that sometime soon I can get rid of that second step. It feels like a dirty hack this.

Well, I found a way of getting it to work and keeping the source IP. It involves two NATs so I feel very dirty....

It's a workaround, but it will have to do as the performance gains of WireGuard over OpenVPN are too large to ignore.

So, instead of doing a NAT on the Linux server straight to the LAN server, I do a NAT to the GRE tunnel's IP on the OPNsense box. On the OPNsense box I add a second Inbound NAT rule to take that packet and forward it to the server on the LAN. This way the return traffic finds its way back on the GRE tunnel back to the Linux server and from there to the Internet. I preserve the source IP, and things now work.

Not an ideal solution by any stretch of the imagination, but again, it will have to do.

I'd love for OPNsense to start using the multiple routing table feature in FreeBSD. I use it extensively on Linux and it creates for a lot of flexibility.

Hope this helps your case a little.

Will do. I'll try playing around with a few combinations to see if this is in anyway possible. The strange thing is that it works with OpenVPN on pfSense (so I assume it works with OpenVPN on OPNsense as well, but perhaps I should try that too...)