Topics

As of relatively recently my HA setting has stopped working.  When I try to access "High Availability" -> "Status" I get an error message:

    The backup firewall is not accessible or not configured.

CARP is still working fine, which I why I hadn't noticed and can't say when this begun.  There are no firewall rules preventing traffic on the direct ethernet link between the two firewalls.   Can anyone suggest how I might investigate / fix this?

Thanks

[pciconf -l -v]

I have been experiencing some problems with the router becoming unresponsive overnight due to some sort of leak which I have found impossible to diagnose.   As such I have been trying to enable FreeBSD's built in watchdog functionality to qemu to reboot my OpnSense VM if it becomes unresponsive.   Unfortunately I cannot get it to work at all.

I am running a couple of OpnSense (based on FreeBSD 12.1) routers in qemu virtual machines in Proxmox 6.4. 

Promox has its own non-libvert way of configuring VMs, but to add a watchdog device, one adds the following line to the VM config file:

Code Select Expand

    watchdog: model=i6300esb,action=reset

I can see that this is working and the virtual device is present in the VM because pciconf -l -v in the guest includes the following output:

Code Select Expand

    none0@pci0:0:4:0:       class=0x088000 card=0x11001af4 chip=0x25ab8086 rev=0x00 hdr=0x00
        vendor     = 'Intel Corporation'
        device     = '6300ESB Watchdog Timer'
        class      = base peripheral

Unfortunately, although the ichwd driver supports the emulated chipset, it is not being detected, and there are is nothing in the dmesg logs about it.

I have enabled watchdogd by adding the line watchdog_enable=yes to /etc/rc.conf.  This is working, but it will default to using a software watchdog, and appears to produce no debugging information, so is not helpful.

I can also force the kernel to load the ichwd driver by adding ichwd_load="YES" to boot/loader.conf (actually in OpnSense this is done by adding it to System -> Settings -> Tuneables in the GUI, but the final effect is identical).  Output from 'kldstat` shows

Code Select Expand

    Id Refs Address                Size Name
    12    1 0xffffffff82959000     70c8 ichwd.ko


I am thus reasonably certain that the virtual device is present on the system and the correct device driver loaded, but am unable to coax any further debugging information out of qemu, ichwd, or watchdogd.  What should I try next?

Hello, I have two routers connected to different uplinks in a redundant CARP setup, one configured as primary CARP, the other as backup.  I would like the primary machine to demote itself when when its uplink fails and to re-establish itself as primary when the link comes back up.  Is there a way to do this with OpnSense? Thanks.

Hello all.  I am running into a nasty problem with current OpnSense which probably started sometime during the 20.7 release cycle (I certainly don't remember this happening before the 20.7 upgrade).  It is difficult to debug as the symptoms are that the router becomes unresponsive, and throws up errors due to killing processes because it is out of swap.  It seems like it might be some sort of memory leak, but that does not make much sense.  I am able to monitor memory usage easily because the router runs in a VM on a Dell R710 1u server, and I can see that of the 2G I have allocated only about 1.2G is in use by the VM even during this issue.   Normal memory usage as reported by OpnSense (not directly comparable to the usage reported outside the VM by by qemu) itself is about 500G in normal operation, 1.9G just after boot.  CARP is enabled, but does not kick in, presumably because the kernel level CARP interface remains functional even though the router itself does not.

I have checked the logs, and unfortunately the out of swap errors do not get logged.  In fact there seems to be nothing in the way of logging for at least 30 minutes before the router became unresponsive.  The only reference to swap I have in the logs is as follows:

Code Select Expand

2020-12-13T00:41:01 kernel warning: increase kern.maxswzone or reduce amount of swap.
2020-12-13T00:41:01 kernel warning: total configured swap (2097152 pages) exceeds maximum recommended amount (2002392 pages).

I thought that there was no swap configured on OpnSense, so I am confused by this.

The unresponsiveness usually occurs overnight, so if there is an intensive system job running overnight, this could be the cause.

Can anyone give me any pointers as to how to work out what is going wrong.  I have not tried to login on the console when the router is unresponsive, but the web interface does not respond at all.

I have finally managed to get Wireguard VPN set up.  While I do think it's a lot simpler than the old staples like OpenVPN, it's a little bit of a frustrating experience, and the OPNSense instructions, while good as far as they go, do (like much of OPNSense's documentation) rather assume that you already know what you are doing.    So, for people who want to get an easyish VPN up and running but don't necessarily know a lot about Wireguard, here are some tips from my experience.

1) Pre-reading is a really good idea.

Start at wireguard.com for a quick conceptual overview - https://www.wireguard.com/

Read the Wireguard OPNSense instructions carefully.  A lot of of the frustration I experienced was due to not understanding these instructions properly when I started out.  When I came back later, a lot of the answers to the problems I was having were actually there.   There are two example configs provided.  The first, site to site setup, at https://docs.opnsense.org/manual/how-tos/wireguard-s2s.html is for linking the networks at two remote sites.  My usage case focused on connecting individual remote client machines to my local network, so I relied upon the slightly unhelpfully named "road warrior" setup at https://docs.opnsense.org/manual/how-tos/wireguard-client.html.

2) Useful Wireguard concepts

Wireguard is designed with a "general case" and peer-to-peer philosophy in mind, that makes it slightly confusing in a client/server context.  Wireguard nodes are peers, the basic idea is that each peer has a private key and a public key.  If you know your own private key and the other end's public key, and the other end knows its private key and your public key, then you will be able to talk.  But you will also need to understand how your network topology fits into Wireguard's model.

Because nodes are peers and not clients and servers, some of the configuration you would usually do on a server actually takes place on the "client".   On the "server", you can (and should) restrict the addresses a "client" is allowed to present as, but access control is all done outside of Wireguard in the router itself - Wireguard only really handles the creation of the network interface.   If you are reliant on a non-public DNS server, your clients will need to be configured individually to use it.

DHCP is not really a thing with Wireguard.  The remote end decides what IP address it's asking for and the local end gets to say yes or no.  This is true for both ends.

One end (likely the server) is going to need a resolvable DNS name or a static IP.  The other end can be non-fixed and behind a firewall (although you will need to configure keepalive in the latter case), but without one "anchored" endpoint there is no way for the peers to find each other.

3) Shared Secrets

Wireguard provides a pre-shared secret key or PSK (referred to as "shared secret" in OPNSense) as an added layer of security.  This is another key that is known to both clients.  This is optional, and you can ignore it if you like.  Explanations as to how it improves security are sparse at best, but it appears to work by creating a hash with the public keys sent across the network, so your actual keys are never transmitted.   The OPNSense documentation doesn't really refer to this, but if you want to generate one, you can do it from the command line using "wg genpsk".  This key needs to be configured on both ends of the connection.

4) OPNSense configuration - server

Read the OPNSense docs first.

Some settings are not relevant in a client/server setup.  You do not need to set a DNS server on the server end, as this setting is for DNS servers on the remote end of the tunnel.  The "tunnel address" setting is slightly confusing, but you want it to be the IP address and netmask of the local end of the tunnel.  You pick the address, and it needs to be on a different logical network from the other networks on your router.  If you decide on 10.80.0.1 with netmask 255.255.255.0 (so allowing for up to 253 clients), this setting would be 10.80.0.1/24.  OPNSense will generate public and private keys for you when you save the local configuration.

You can use any port number not in use on the local machine for your server.  You will need access to this UDP port through your firewall so that "clients" can talk to it.

5) Client configuration

I found it easiest to create the client configurations outside of OPNSense and then copy the relevant details into the OPNSense GUI.  Wireguard is still relatively new, and there is not much in the way of GUI configuration, so you will likely have to become familiar with its config file format in any event.   Here is a sample client config.

Code Select Expand

[Interface]
Address = 10.80.0.5/24
ListenPort = 51870
PrivateKey = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx=
DNS = 10.10.0.1

[Peer]
PublicKey = yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy=
PresharedKey = zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz=
AllowedIPs = 0.0.0.0/0
Endpoint = my.fqdn.com:51820
PersistentKeepalive = 25


Running through these in order

(I) Interface section - configuration of the local end of the tunnel.
a) Address - the local tunnel endpoint address and network used by the client.  Again this must be otherwise unused on the client. 
b) Listenport - can be anything that's not used locally - if you're behind a firewall, the connection will be on a different port anyway.
c) PrivateKey - Wireguard generates these.  On Unix, the command "wg genkey | tee privatekey | wg pubkey > publickey" will put your key in the file "privatekey".  Remember to keep these secret - do not make these files world readable.
d) DNS - the name of your DNS server on the server network.  You need to set this if you are routing all traffic via the VPN.  You probably don't want to try to run a DNS server directly on the Wireguard interface, because it may not be able to bind to the interface at boot time.  Note that there is no current way to set a default DNS search prefix on Windows, so your Windows users will have to live with typing full domain names for your services.

(II) Peer section - information about the remote end
(a) PublicKey - the server's public key - you can take this from the OPNSense GUI - just edit the "local endpoint" and copy from there
(b) PresharedKey - must be the same for both ends of the tunnel.  Generate using "wg genpsk"
(c) AllowedIPs - 0.0.0.0/0 is the setting to tell this machine to route all of its traffic over the VPN when it is up.  You can alternatively set this only to route traffic to your secure network using e.g. "10.10.0.0/24,10.20.0.0/24"
(d) Endpoint - this is the public name and port of your server
(e) PersistentKeepalive - if your client will never be behind NAT you can leave this out, but if NAT is ever likely to be involved, you will need this to make sure that the firewall on the client end does not close the connection between client and server.

Confguring mobile clients can be painful.  There does not appear to be a qrencode package for OPNSense, but on a Unix system with qrencode installed, you can convert a config file into scanable format using "qrencode -t ansiutf8 -r client.conf" (make sure your terminal window is big enough).  This can then be scanned in in the Android (and possibly iOS) application.  (It may be that the OPNSense maintainers might want to include  graphics/libqrencode in their available packages so that people could install it if they wanted to, but it is rather a niche use case.)

(https://github.com/burghardt/easy-wg-quick is a script used to quickly generate config files for this type of installation which may be worth investigating if you have a lot of clients to set up, but plumbing its configuration in to OPNSense may take some thought.)

6) OPNSense configuration - client

Again, not all settings are relevant in this sort of setup.  Add the relevant ones from the "Endpoints" tab.   "Name" can be whatever you want to call the link.  "Public key" is the client's key as generated above, and "shared secret" is likewise your generated PSK.   "Allowed IPs" should be set to the IP address you want this client to connect as, so in the above example, you would set it to 10.80.0.5/32.  Note that you use a single host netmask, as there is only one IP address that this client is allowed to connect as and this link is not a route to anything other than this specific endpoint.  "Endpoint address" and "endpoint port" are likely to change and do not need to be configured.  You are unlikely to need "keepalive" here, as your server port is accessible via the firewall at all times.  The clients will set keepalive if they need it.

Don't forget to go back to add this endpoint to "peers" in your local endpoint to allow the connections to talk to each other.

6) Routing

This will depend hugely on your setup.  The official docs say that to allow routing to the Internet, you need to add the Wireguard interface to the OPNSense interfaces.   I wasn't able to get this to work, and didn't need to, so it may be that this is out of date.  For me, it was enough to add a firewall rule allowing traffic on the Wireguard interface to pass.   For outbound NAT rules I use a hybrid configuration that turns off NAT on the Wireguard interface (not sure if I need that or not) and after that I created duplicates of the auto-created rules, substituting the "WireGuard" net for my local networks.  In all honestly I find the OPNSense firewall to be a bit of a black art, and there may be a better way to do this, but it seems to work for me.

7) Troubleshooting

Wireguard's diagnostics are, frankly, utterly rubbish, but OPNSense does provide you with most of the info available.  You may also find it helpful to use "tcpdump -i wg0" from the router on the command line interface to monitor the traffic on the VPN link.

OPNSense's "list configuration" tab shows your currently configured tunnels.  If a client is not showing up here, then you probably forgot to add it as a peer to the local endpoint.  If "allowed IPs" is blank, that may be a symptom of an incorrectly configured tunnel.  "Latest handshake" shows when a client last connected - if it's not there, then your tunnel is not working, but bear in mind it can take a short while for the tunnel to come up after you restart it.  Similarly the "Handshakes" tab will show you if a client has connected since you last restarted Wireguard on the server (if the number on the right corresponding to the public key of the link is not 0) so may also be useful.  You can stop and restart Wireguard on the router to reset these if you need to.

Hopefully this info will be useful to someone who, like me, was having trouble getting things to work.

I have two OpnSense routers set up, router1 is connected to Virgin, router 2 to Vodafone. Router1 and router2 are in a CARP group, and router2 takes over from router1 as DNS and DHCP server in the event it goes down.   In addition to LAN links, there is a direct back to back link between the two machines.  The setup for router1 is thus

Code Select Expand

LAN: 192.168.37.5/24
Backbone: 192.168.66.5/24 - point to point with 192.168.66.4
WAN: <wan address>
CARP: 192.168.37.1/24

And router 2 is on .4 with a mirror configuration

I have manually configured a gateway, BACKBONE_STATIC with address 192.168.66.4, it is set as an upstream gateway, but not as a far gateway, and in practice, neither of those settings seems to make a difference.  Priority is the default 255.

The external interface and the backbone link are in a WAN group together, with the external uplink as Tier 1 and the backbone as Tier 2.  The backbone interface has a default firewall rule to allow all traffic, and also, there is a custom outbound NAT rule saying not to NAT the backbone ahead of the auto-generated rules.

This configuration works, and as expected there is failover to the backbone link if the Virgin connection goes down.  The problem is that monitoring seems extremely flakey.   If the WAN on router2 is up when I configure the backbone link, it comes up and continues to show as online even if I take the WAN on router2 down.  If the WAN on router2 is down when I configure monitoring, the backbone link shows as offline and does not come back.

I thought this might be an asymmetric routing issue i.e. that router2 does not send the returning packets back via 192.168.66.5, but enabling the "bind states to interface" firewall option on both routers does not change this behaviour.  Adding a static route to 0.0.0.0/0 via 192.168.66.4 similarly makes no difference.   How do I get this setup to work properly?

As a side note, using Cloudflare DNS (i.e. 1.1.1.1) as a remote target for dpinger does not seem to work.  The OPNsense examples all use Google's Public DNS as a suggested endpoint, but this is only helpful if you only have two WAN links.   Are there any other good IPs to use with gateway monitoring?

My current OPNSense config involves 2 opnsense routers - call them "router1" and "router2".  Router1 connects to a Virgin uplink.  Router2 connects to a Vodafone uplink.   Both routers share a CARP virtual IP, 192.168.5.1, with router1 having higher priority and being the master.  They do not share firewall config because they have different interfaces attached, but dhcp and unbound are synced across from router1 to router2 using the OPNsense HA features on a dedicated direct static ethernet link between them.  The network is configured to use the CARP address for DNS and routing.

What works OK - when router1 goes down, router2 transparently takes over, and there is no interruption in network service.

What doesn't work OK - when the Virgin router goes down, router1 keeps hold of the CARP link and does not allow router2 to step in.

I've played about with trying to create a multi-WAN gateway to route traffic to the internet via the dedicated link from router1 to router2 on router1, but I can't even get the gateway to come up.  Adding a static route to 0.0.0.0/0 via the dedicated interface didn't help, nor did enabling the "static route filtering" option on router1.

What I'd like to be able to do is one of the following

1) Have router2 become the CARP master when router1 has no internet access; or
2) Allow router1 to send internet traffic via router2 when its own  uplink is down

What is the best way to achieve this, please?

I'd like my local network to be a bit more robust on internal DNS if the router goes down.  I could create a second OpnSense machine, but that seems like overkill.  All I really want to do is have a second DNS server, which copies all the configuration on the OpnSense box, and can be referred to if the primary server goes down.

Is there a good way to accomplish this?

I have a router that is accessible as router.domain.org and router.subdomain.domain.org.  I am using dehydrated to generate an SSL certificate valid for both of these domains, which is then installed on opnsense using a script called opnsense-import-certificate.php that I found on this forum.

This certificate shows as valid for both domains.  I have "alternate hostnames" under "Systems -> Settings -> Administration" configured as "router.domain.org router.subdomain.domain.org" in the GUI.  SSL is enabled also.

Unfortunately, "router.subdomain.domain.org" is still always throwing a DNS rebind attack warning.   I'm loathe to turn DNS rebind attack warnings off, but I would like to be able to access from both domains.  Does anyone have any idea how to fix this?  Thanks.

I have set up my Opnsense 20.1.7-AMD64 router (running on a Dell R220 in a vm under qemu) with multiple WAN connections.  I have an IP modem on vtnet2 connected to Virgin Media UK using DHCP and an VDSL2 modem on vtnet1 connected to Vodafone UK (because Vodafone unhelpfully don't let you run their supplied router in modem mode).   I used the guide at https://docs.opnsense.org/manual/how-tos/multiwan.html to set these two links up in fallover mode with the Voda connection being used when the Virgin one falls over (since the Virgin one is twice the speed).   

Both gateways are assigned to WAN_GROUP, and the default rule for LAN (on vtnet0) is configured to route via this as a gateway.  All the firewall NAT holes on the router point to the Virgin link - my dynamic DNS provider (Mythic Beasts) doesn't offer an easy way to assign multiple IPs to a single hostname - the previous A record will simply be overwritten, which is of course the behaviour one usually wants, so it makes sense to keep all the external services on a single WAN interface, although it would be nice if there were a way to fallover these too.

Anyway, if I set the Virgin link to Priority 1 in the gateway group and Vodafone to 2, everything works like a dream.  The local internet transparently flips back and forth if I disconnect Virgin, so I have redundancy and can carry on working in the event of an ISP outage so long as both Virgin and Vodafone aren't down.

Ideally though, I'd like to aggregate the links.   I previously had this working in a test configuration using Wingate with a direct connection of the Virgin link to my PC.   This would show download speeds of 150Mbit/s and upload of 20Mbit/s on speedtest.

If I set both routers to priority 1 on OPNSense, however, things just get weird.  Some sites become unreachable, or have massively long timeouts - Speedtest will only ever use one WAN connection at a time, regardless of whether or not "bind states to interface" is set in firewall rules. 

Link speeds at 110Mbit/s for Virgin, 50-55Mbit/s Vodafone.  Trigger level is set to "Member Down".    Weight is set to 2 for Virgin, 1 for Vodafone.

I'm genuinely not sure where to start in diagnosing the issues I am seeing, and some pointers as to what to start looking at would be much appreciated.

Hi,

I simply can't get PPPoE to work on OPNsense 20.1.7-amd64 with an EchoLife HG612 (patched) mpdem connecting to Vodafone UK.  I have configured the link, and can see from the modem's admin interface that there is a physical connection.   OpnSense simply will not connect though.  The logs show the following:

Code Select Expand


2020-06-07T14:02:04 ppp: [opt3_link0] Link: reconnection attempt 99 in 2 seconds
2020-06-07T14:02:04 ppp: [opt3_link0] LCP: Down event
2020-06-07T14:02:04 ppp: [opt3_link0] Link: DOWN event
2020-06-07T14:02:04 ppp: [opt3_link0] PPPoE connection timeout after 9 seconds
2020-06-07T14:01:55 ppp: [opt3_link0] PPPoE: Connecting to ''
2020-06-07T14:01:55 ppp: [opt3_link0] Link: reconnection attempt 98


The "connecting to" line in particular suggests some sort of misconfiguration, but I have followed guides from people who apparently have the same sort of connection working wihtout difficulty.  Can anyone suggest where I should be looking?  Thanks.

I have an opnsense router on 192.168.10.1 and several NAT rules set up similar to the following

Interface: WAN
Destination: WAN IP
Port: 2222
Target: 192.168.10.100
Target Port: 2222

When I try to connect to port 2222 on the router from inside the network (i.e from 192.168.10.50 to 192.168.10.1), then the request is redirected to the NAT machine 192.168.10.100

Port reflection and automatic outbound NAT for reflection is enabled.   Turning these off stops this behaviour.

This is not desired behaviour.  Is it an inevitable consequence of port refection.  If so I will just put all of my external names in the unbound overrides and turn off reflection, as this will largely achieve what I want, but for testing and consistency purposes I would prefer that DNS resolve to the same values inside and outside of the LAN.

So I have a NATed LAN behind an opnsense firewall.  Port forwarding (with reflection) is in place for my nextcloud server on TCP port 8443.  All works well in that I can access the sever using https://www.internet.name.com:8443/ from both inside and outside of the network.

However, the config file backup option in opnsense does not work in that opnsense times out trying to contact the server.  This does not happen when the internal name of the server is used, but for security reasons, nextcloud will not serve files as the server's SSL certificate only covers its public facing name.  If I want to back up my config from the opnsense router to the internal server, it therefore has to be done via using the public host name and NAT reflection.

NAT reflection does not appear to be working from the router itself.  I force an update from the user interface, but the page waits for a long term before returning with an error. The firewall logs are unhelpful, and do not show any block rules being matched or any traffic being allowed from the router.  tcpdump confirms no traffic to or from port 8443 (the port I am using) on either the LAN or WAN interfaces related to the request to back up the config file.

The error from the logs is:

2020-04-19T15:47:53   config[43811]: {"url":"https:\/\/<server_url>\/","content_type":null,"http_code":0,"header_size":0,"request_size":0,"filetime":-1,"ssl_verify_result":0,"redirect_count":0,"total_time":60.030287,"namelookup_time":0.022502,"connect_time":0,"pretransfer_time":0,"size_upload":0,"size_download":0,"speed_download":0,"speed_upload":0,"download_content_length":-1,"upload_content_length":-1,"starttransfer_time":0,"redirect_time":0,"redirect_url":"","primary_ip":"","certinfo":[],"primary_port":0,"local_ip":"","local_port":0}
2020-04-19T15:47:53   config[43811]: Error while fetching filelist from Nextcloud

(DNS on the router is resolving the public host name.)

Does anyone have any suggestions as to what is going on, and how I might fix it?  Thanks.

I am having a bit of a tricky configuration issue that I hope someone here might be able to shed some light on

Network topology is as follows

Code Select Expand


Public                        NATed
IP v4                         IP v4 (192.168.11.x)
WAN <-> opnsrouter <-> sshhost
                               <-> cliient
                                NATed
                                IP v4 VLAN (192.168.22.x)
                                <-> vlanhost


(I don't think the VLAN is relevant but I include it for completeness - it's using tagged 802.11Q on opnsrouter.  Firewall rules stop it sending traffic to local machines or opnsrouter.)

Automatic outbound NAT for reflection and reflection for port forwards are enabled

I set up a rule to port ssh to sshhost as follows

Code Select Expand


Interface: WAN
Destination: WAN IP
Destination port range: SSH
Redirect target IP: sshhost (via a firewall alias containing sshhost's IP)
Redirect target port: SSH
NAT reflection: enable

All other options are left at default

With this rule set up ssh to opnsrouter's public name works from both inside and outside the router, as expected.  However, if I try to ssh to opnsrouter itself from inside the network, the connection also gets forwarded to sshhost rather than being picked up by opnsrouter as I would like.

What am I doing wrong?  Is there a way to fix this?  Thanks.