Messages

I turned on my debug logs for the WG instance and I can see the following message when the peer connects.

wg1: Packet has unallowed src IP from peer 1

My Opnsense instance is 172.16.0.1/24
My Opnsense peer for the mobile allowed IP is set to 172.16.0.101/32
My peer side config allowed IP is set to 0.0.0.0/0
I have a firewall rule from the 172.16.0.0/24 network to an internal server. I can ping it and connect to it fine from the mobile peer.
Packet capture on the WAN interface shows the wireguard connection.
I have an interface assigned to WG.

The peer connects fine and passes traffic when connected to 5G connection.

Why would this message show in the debug?

I'm finding when switching from 5g to wifi, I need to turn off/on wifi and wg off/on and then it works correctly again. As if the routing needs to be reset.

Do you ever need to do this?

Other than brief delays when the phone switches from wifi to mobile, no. I never need to restart android client or toggle wifi/mobile. Having said that I am not a heavy mobile user. But the only times I experience VPN connection issues is if I am out and lose mobile signal.

No prob, thanks for sharing. Will keep at it and see if I can figure it out. I'm pretty sure it's something on the client (android) side as no traffic is routing until I reconnect it.

I also use a cloudflare A record for my vpn client conf.
wg client conf specifies a local DNS server (pihole) and search domain (local.lan).
Opnsense's unbound is the upstream server for the pihole.
Wireguard interface wg0 is assigned to a specific interface (VPN) and rules on there permit access to the local pihole instance amongst other things.

When using openvpn, prior to wireguard, I tried various configs to get the smoothest roaming experience. I ended up with port forwarding to localhost (for both WAN and LAN) because I found the openvpn server not reliably listening on all interfaces. When I switched to wireguard, all I did was modify port aliases and of course configure wireguard.

I tried wireguard for first time very recently, after using openvpn for a long time. After setting up the FQ Codel scheduler as per the buffer bloat recipe, I began to see increased openvpn warnings re out of sequence packets. So, thought it a good time to try Wireguard.

I am not sure why you see intermittent connectivity when connected to WG on the LAN. I notice the WG android client log is not very detailed.

Thanks for the further info and insights on your config.

Do you have any issues when roaming from 5g to wifi, or does it seamlessly roam without a problem?

I'm finding when switching from 5g to wifi, I need to turn off/on wifi and wg off/on and then it works correctly again. As if the routing needs to be reset.

Do you ever need to do this?

...

When I am on the internal wi-fi, I connect to wireguard, and can see the tunnel is connected in the WG status page. The IP address of the client is from the internal wifi subnet.

However, when I ping the wireguard gateway IP address, I get patchy responses, one reply, followed by timeouts.

Sometimes I ping, and I don't even get one reply back.

For minute I thought I had a asymmetric route, but after taking a packet capture on all interfaces, I can see that when the ping traffic fails, there is not tcmpdump entry.

It's as if the traffic from the mobile client is not being routed down the wireguard tunnel at all.

My wireguard config states to route 0.0.0.0/0 (all) traffic via the wireguard tunnel connected, so I'm not sure why it's not doing that.

Any ideas?

Not sure where your issue lies but the way I have done this (for both openvpn and wireguard) is one destination NAT rule on the WAN, and another on the relevant lan interface. Both forwarding wireguard port to 127.0.0.1. I found that to be the most reliable way to get mobile/wifi roaming whilst only using WAN IP in any vpn client config. The WAN version of the port forward and fw rule filters src by my mobile provider's ASN.

Thanks for sharing and explaining your setup Keeka.

One question - How is your DNS configured within wireguard on the client and in Unbound DNS and external DNS server (e.g. cloudflare)?

I'm use a DNS A record in Cloudflare that points to my WAN IP. Internally, when on the Wireless LAN, the mobile client resolves the Wireguard DNS name (e.g. vpn.domain.com) to my WAN IP address, I have a rule to allow the traffic from the wireless network to the WAN interface.

I don't have any destination NAT rules configured for Wireguard, because Wireguard is listenting on the WAN IP on UDP 51821 already.

Can you please help me understand how you have your DNS configured?

Will give your configuration a try.

p.s. Really like how you've filtered on ASN for the WAN rule

Hi all,

I'm having issues with an Always-On-VPN setup I'm trying to get working for mobile clients using Wireguard VPN.

My goal is to always leave the Wireguard VPN On for all mobile phones, and iand for them to roam between the internal wi-fi and their 5g connection, all traffic for the mobile client should route through the Opnsense firewall.

Here's the toplogy.

192.168.1.x = Internal Wireless Subnet
172.16.1.x = Wireguard Subnet
10.0.1.x = File server Subnet

Currently if the mobile phone is connected to the 5G connection, everything is working fine.

However, when the mobile phone is connected to internal wifi, and the Wireguard connection is sucessfully established, I try to connect apps (file server), and I receive a "TIME_WAIT:TIME_WAIT" message in the session logs.

After reviewing the firewall traffic logs I can see that the traffic is allowed and "pass" status.

However, when comparing a trace route from the mobile to the file server when on the wifi and wg connected, it does not hit the wireguard gateway first, instead I see * * * .. for each hop.

Does anyone have a configuration like this working properly or know how to resolve?

Thanks!

You can use the external IP address of the OPNsense, split DNS is not necessary.

Would a reflection or hairpin NAT be needed so that various internal LAN subnet clients can connect back to the external IP address?

Bump.

Any advice would be appreciated.

I was thinking to perhaps create a new VLAN  and use VIP's for any service hosted on the OPNsense itself.

Is this the right design approach or should a different design be used?

Hi all,

If I have multiple LAN subnets, and I want my clients in each subnet to be able to resolve/route to NGINX running on OPNSense, and then NGINX forwards to a server IP running in a DMZ subnet, what is the correct way to configure the DNS.

Do you setup a single Unbound DNS override entry to point to a single LAN gateway that you designate for NGINX, or do you somehow setup each LAN to have the DNS name of the server resolve to their respective LAN Gateway interfaces? 

Do you actually see blocked websites or are these just random log entries? One that you posted is from Google and it has a FIN-ACK state.

Therefore, potentially, you see artifacts from QUIC traffic - I see those, too.

You can test if you allow HTTP3/QUIC traffic and see if the test triggers those log entries. Wait a bit, it may be that the TCP stream must be closed to cause a log entry.

Thanks for your suggestion. I was able to browse to the cloudflare page without issues. Looking in the live view log, I don't see a blocked entry for it either.

It's prodominately tcp 443, but have noticed other ports too; 5223 and 6159.

I haven't put my finger on anything that is not working that I can consistently use as a test. Everything seems to be working.

I'm seeing the blocked traffic originate from multiple different clients.

Just checked the details of the log file, and can see it's for protocol 6, which is IPv6.

I added in a specific interface rule for IPv6, however I'm still seeing traffic being blocked in the Live Logs.

Do I need to setup IPv6 somewhere else as well?

What's strange is that the source address is in IPv4 format, not IPv6.

Click on the details for one of those blocking incidents and post them.

The most common cause of unexpected "default deny" is asymmetric routing. So a diagram of your network would also be helpful.

Here's a basic network diagram:

Please attach here. I categorically block so called "image hosting services".

Hi,

I'm looking in the Live Firewall logs for my LAN interface, and I can see that the default deny rule is periodically blocking 443 traffic, even though my firewall policy for that interface has a rule that allows all ports and protocols.

How can I track down what is causing this?
Is there a way to run a debug trace, and see what object in OPNsense is triggering this and determine why it's not being allowed out based on the interface rule?

Most tcp 443 traffic is being allowed, just randomly some of it is being blocked. It's also happening on other ports too, not just tcp 443.

Thanks!