Messages

I'm not sure how to proceed with setting up a simple VLAN setup and whether to use vlan aware Linux Bridges, or Openvswitches?

I don't want to do any vlan configuration on my physical switches at all.
It should all be done on Proxmox and/or OPNsense.

I have 2 proxmox servers. One is at the edge of my network and it hosts an OPNsense firewall, and the other is in the core.

I want to be able to run some vlans to keep traffic isolated on my internal networks.
I want to add a vlan tagged interface to a guest vm on my core proxmox, and it should be able to communicate with OPNsense.

So far I have just used a vmbr0 Linux Bridge that is set to vlan aware connected to my physical 10G interface.
Added standard vmbr0 / vmbr1 / etc interfaces to my OPNsense VM. No tags from the Proxmox UI.
Created vlan interfaces in my OPNsense firewall. Tagged in the OPNsense UI.
Created VM guests with the network card set to specific vlans. Tagged on the Proxmox UI

This is working, but is there a better way to approach it?

Any advice on how to set this up would be greatly appreciated

Tried your approach using the new connections method and got a little closer.
The firewall can now ping the inside /31 of the other end, but clients can't pass traffic.

Seems like maybe a NAT issue, or some weird setting somewhere.

I have not added a route, as I want to use PBR to send all traffic from specific hosts over the tunnel, so ideally I want to use a firewall rule, and specify the VTI interface as the gateway

Single gateway pointing at the VTI interface looks good and health monitoring is working.
If I try and set the VTI interface as the gateway in a firewall rule I get these errors immediately, but I'm not sure if it's cosmetic or a show stopper?
You can't set an IP address on the interface because it's a tunnel interface

Error   firewall   There were error(s) loading the rules: no IP address found for ipsec10ip   
Error   firewall   /usr/local/etc/rc.reload_all: The command '/sbin/pfctl -f /tmp/rules.debug.old' returned exit code '1', the output was 'no IP address found for ipsec10ip /tmp/rules.debug.old:50: could not parse host specification pfctl: Syntax error in config file: pf rules not loaded'   
Error   firewall   There were error(s) loading the rules: no IP address found for ipsec10ip   
Error   firewall   /usr/local/etc/rc.newwanip: The command '/sbin/pfctl -f /tmp/rules.debug.old' returned exit code '1', the output was 'no IP address found for ipsec10ip /tmp/rules.debug.old:50: could not parse host specification pfctl: Syntax error in config file: pf rules not loaded'

I can't quite seem to work out how to get my VTI based IPSEC tunnel working, and need another set of eyes.
The frustrating thing is it works on pfsense, and I'll be damned if I use that!

This is from an OPNsense firewall to a cloud based IPSEC termination point

I'm using Legacy mode and the tunnel appears to come up just fine and is shown in VPN status overview.
I am using a /31 as the inside tunnel addresses and they show in the routing table.

But, when I try and add a gateway and ping the inside /31 at the other end it does not work.
The gateway always shows as offline / defunct, and also doesn't work if I turn off monitoring

No traffic flows over the tunnel whether it is sourced from the firewall or an internal host.
I have tried messing with outbound NAT rules, and doing policy based routing.

Not sure where to go from here as this should be straightforward and it works in pfsense and doesn't work in opnsense.

What logs can I delve in to, or provide to try and fix it?

Great instructions! Thank you :)

The only issue I'm facing is getting the firewall redirect rule for dns just won't work for me.
I've tried using the "LAN address" object, and also specifying my LAN IP address and my VirtualIP's, but it just doesn't seem to want to redirect the dns traffic :(

I did notice when setting up Adguard it chose my Virtual IP, instead of my LAN address.

I feel like I'm missing something really simple, but I'm not sure what?

Interface: LAN
Protocol: TCP/UDP
Destination / Invert: Ticked
Destination: LAN address
Destination port range: From: DNS - To: DNS
Redirect target IP: 127.0.0.1
Redirect target port: DNS
Description: Forward DNS to AdGuard
NAT Reflection: Disable

So...I like Zerotier, but I really like Nebula a LOT more :D

https://slack.engineering/introducing-nebula-the-open-source-global-overlay-network-from-slack-884110a5579

Full mesh
Auto-learning
Much better licensing model!
Fully self hosted private architecture

I'd love to see it become a native OPNsense module :)

That 2nd peer public key needs to be filled in :)

You only need them if one side is behind nat

It would be awesome if the OPNsense documentation mentioned that :D

OK...finally worked it out :D

It seems if you want to get a site to site VPN working you need to enable keepalives.
If one side doesn't have that enabled they both just sit there and never try and make the connection.

It would be lovely if that was documented somewhere. haha :D

So you should be able to ping 10.7.7.0/24 network via remoteFW ... can you check with tcpdump on wg0 if packets are going through the tunnel?

I'd love to ping something on the 10.7.x network, but the tunnel never comes up, so there is no way to ping through it.

I'm not actually seeing the remote fw ever make an attemp to connect to the central fw.
Once I get the tunnel up, I'm very familiar with opnsense rules/nat/gateways/pbr's etc. :)

Why disable routes at endpoint?
Also bump port above 1024

Just been playing around with all sorts of settings and port numbers.
No matter what I do I never see a connection attempt at all with tcpdump :(

Perhaps this is an issue with opnsense/wireguard running on esxi 6.7u2?

I've tried opnsense firewalls at 2 different remote locations to rule out a location based issue.
Also, wireguard clients on macos/windows can connect perfectly from those locations

netstat shows the wg service listening on the ports I've assigned
tcpdump shows udp connections on that port if I scan it using nmap
tcpdump shows wireguard connections if I use a wireguard client
tcpdump never sees a connection attempt from the remote fw

wireguard command looks good to me from the cli

Code Select Expand


root@remoteFW:~ # /usr/local/etc/rc.d/wireguard restart
[#] rm -f /var/run/wireguard/wg0.sock
[#] wireguard-go wg0
INFO: (wg0) 2019/12/04 11:52:03 Starting wireguard-go version 0.0.20191012
[#] wg setconf wg0 /tmp/tmp.PhMa1gs5/sh-np.L1XuNv
[#] ifconfig wg0 inet 10.11.0.2/24 10.11.0.2 alias
[#] ifconfig wg0 mtu 1420
[#] ifconfig wg0 up
[#] route -q -n add -inet 10.11.0.1/32 -interface wg0
[#] route -q -n add -inet 10.7.7.0/24 -interface wg0
[+] Backgrounding route monitor


[Central FW:]

No known issues ... need screenshots :)

Hmm, something weird is going on then, and I'm not sure what I'm missing?

I am expecting the remote FW to at least try and make the connection, and that I would be able to see that running tcpdump on the central FW.
I have tried different ports, and different remote networks, etc. with no luck.
Firing up a client on windows or macos works instantly.

Screenshots aren't very exciting really...

Central FW:






Remote FW:

I have a very basic configuration and I'm just not seeing the remote OPNsense fw trying to initiate the connection :(
Remote wireguard clients on windows/macos can all connect just fine, so I know the central fw is listening and functional.

All FW's are running on ESXi 6.7 and are on OPNsense version 19.7.7-amd64 with os-wireguard 1.1 plugin

When I run tcpdump on my central fw I don't even see remote OPNsense/WG client trying to reach it, but I can test manually and can confirm there is end to end connectivity

Are there any known issues at this time?

Seems it's all on the VPN provider side of things.

They are handing out a gateway IP address as the same one they are giving my client :(
Weird stuff, because it works with a windows client flawlessly.

Manual gateway settings make things work just fine with OPNsense, but I'm trying to get their support to fix things so that the correct assignments are set dynamically

This seems like it should be really straight forward and it's just not working for me :(
OpenVPN tunnel is through Usenetserver.com and is up and appears to be working as expected.

VPN interface is assigned to opt2
Gateway is assigned and looks good
Outbound NAT is for a single host on the VPN interface
Outbound LAN rule has the gateway set as the VPN interface

Straight forward VPN setup with a cert and creds, using these advanced settings:
persist-key;
persist-tun;
persist-remote-ip;
tls-client;
remote-cert-tls server;
comp-lzo;
verb 3;
auth SHA256;
cipher AES-256-CBC;
auth-retry nointeract;

One weird thing is the if I try and ping an external host the firewall replies from the lan IP?

It might just be my VPN provider causing the issue, and I'll test it with another one soon.

I'm at a loss and I was wondering if anyone might have some hints