Topics

Hi there,

I've got following setup: https://prnt.sc/uCpJIatM6ZQc

I'm trying to get connection between the 10.0.10.25 and the 10.11.104.5 hosts. The latter is a host behind a firewall I do not control.
There's a static route set on Site2's OPNsense to allow traffic to the 10.11.104.0/24 network via the 3rd party IP address in the 10.11.10.0/24 LAN. Pinging from the 10.11.10.0/24 LAN to 10.11.104.5 works without problems.

However when I try to ping 10.11.104.5 either through the VPN tunnel, or even just on Site2's OPNsense using the (local, generated) OVPN_TS interface (setting 'Source Address'), I am unable to do so.

Working from LAN:

# /usr/sbin/traceroute -w 2 -n  -m '10' -s '10.11.10.1'   '10.11.104.5'
traceroute to 10.11.104.5 (10.11.104.5) from 10.11.10.1, 10 hops max, 40 byte packets
1  10.11.10.2  0.118 ms  0.049 ms  0.034 ms
2  10.11.104.5  0.192 ms  0.094 ms  0.083 ms

Failing from the local tunnel interface:

# /usr/sbin/traceroute -w 2 -n  -m '4' -s '10.9.1.2'   '10.11.104.5'
traceroute to 10.11.104.5 (10.11.104.5) from 10.9.1.2, 4 hops max, 40 byte packets
1  * * *
2  * * *
3  * * *
4  * * *

The static route that was added is reflected in the routing table: Site2's OPNsense should know where to reach the 10.11.10.0/24 network:

Proto Destination       Gateway    Flags Use MTU Netif Netif (name)
ipv4   default         77.22.44.1   UGS   NaN   1500   vmx0   WANlink       
ipv4   10.0.10.0/24   10.9.1.1   UGS   NaN   1500   ovpnc1   OVPN_TS
ipv4   10.9.1.1      link#8      UH   NaN   1500   ovpnc1   OVPN_TS
ipv4   10.9.1.2      link#8      UHS   NaN   16384   lo0      Loopback
ipv4   10.11.104.0/24   10.11.10.2   UGS   NaN   1500   vmx1   LAN
(...)

I added 'allow all' rules on the OVPN_TS interface but I'm unable to get ping to work.

Any idea what I'm doing wrong?

[On site2 I can reach site1 just fine (I can ping 10.7.1.0/24 addresses).However I cannot reach any IP address from site1 to site2 (e.g. 10.8.1.2).]

Hi all,

I've set up an OpenVPN Site2Site over the public internet. The VPN connects well, no problems there.

Name   Remote Host   Virtual Addr   Connected Since   Bytes Sent   Bytes Received   Status   
My_OpenVPN UDP:1194   79.12.15.170   10.9.1.1   2023-01-19 20:58:26   46 KB   35 KB   up

I use 10.9.1.0/24 as tunnel network (per the above) and the subnets at each site are 10.7.1.0/24 (LAN site1) and 10.8.1.0/24 (LAN site2). Site1 is runs the OpenVPN server; site2 runs the client.

Firewall rules were set to allow all traffic on the OpenVPN tunnel (regardless of source), at both sides:

   Protocol   Source   Port   Destination   Port   Gateway   Schedule   Description       
        IPv4 *   *   *   *   *   *   *   Allow_OpenVPN_traffic

On the OPNsense shell of site1, I can ping 10.9.1.1 (local IP address of the tunnel) and 10.9.1.2 (which is the other side/site).

On site2 I can reach site1 just fine (I can ping 10.7.1.0/24 addresses).
However I cannot reach any IP address from site1 to site2 (e.g. 10.8.1.2).

An extract of the routing table of site2 (https://10.8.1.1/ui/diagnostics/interface/routes) shows entries of site1's 10.7.1.0 network:

ipv4   default   79.12.15.1   UGS   NaN   1500   vmx0   My_WAN
ipv4   10.7.1.0/24   10.9.1.1   UGS   NaN   1500   ovpnc1           
ipv4   10.9.1.1   link#8   UH   NaN   1500   ovpnc1           
ipv4   10.9.1.2   link#8   UHS   NaN   16384   lo0   Loopback       
ipv4   10.8.1.0/24   link#2   U   NaN   1500   vmx1   lan   

The same applies for site1 (i.e. site2 routes exist).

I rebooted both sides just in case to no avail.

Any idea where things are going wrong?

Hi,

I'm getting following error when trying to delete a Single Gateway:

The following input errors were detected:

• Gateway X cannot be deleted because it is in use on Gateway Group Y
• Gateway X cannot be deleted because it is in use on Gateway Group Z

I checked GWG Y and Z though and it can be confirmed that these are NOT using this Single Gateway X in any of the tiers.

Any idea on how to properly delete the Single Gateway?
I had already disabled it but I keep on getting "Status failed gateway_alert" messages from Monit.

Since OPNsense 20.1, the plugin munin-node is available. No need to use the FreeBSD repo any longer.

The below is a mini-howto to install and configure munin-node on OPNsense 20.7.

Steps to take:
- Install the plugin "os-munin-node" through the GUI on System: Firmware: Plugins
- On the GUI, enable munin-node and set the proper server/network that may connect to munin: Services: Munin-Node: General
- Ensure your firewall rules allow access to munin-node (TCP port 4949)
- Enable SSH access to configure munin-node further
- Run "munin-node-configure --suggest" to get a suggestion on which plugins can be activated. NOTE: this command can take a *long* time to execute (4 minutes or so on my 4 CPU/6GB RAM system) without any output... Do " munin-node-configure --debug --suggest" to get more verbose output.
- Inspect what the suggestions are; if agreed run "munin-node-configure --debug --shell" to generate the shell commands. For me, these were:

Code Select Expand

ln -s '/usr/local/share/munin/plugins/cpu' '/usr/local/etc/munin/plugins/cpu'
ln -s '/usr/local/share/munin/plugins/df' '/usr/local/etc/munin/plugins/df'
ln -s '/usr/local/share/munin/plugins/df_inode' '/usr/local/etc/munin/plugins/df_inode'
ln -s '/usr/local/share/munin/plugins/if_' '/usr/local/etc/munin/plugins/if_em0'
ln -s '/usr/local/share/munin/plugins/if_' '/usr/local/etc/munin/plugins/if_pppoe2'
ln -s '/usr/local/share/munin/plugins/if_' '/usr/local/etc/munin/plugins/if_pppoe3'
ln -s '/usr/local/share/munin/plugins/if_' '/usr/local/etc/munin/plugins/if_vmx0'
ln -s '/usr/local/share/munin/plugins/if_' '/usr/local/etc/munin/plugins/if_vmx1'
ln -s '/usr/local/share/munin/plugins/if_' '/usr/local/etc/munin/plugins/if_vmx2'
ln -s '/usr/local/share/munin/plugins/if_' '/usr/local/etc/munin/plugins/if_vmx3'
ln -s '/usr/local/share/munin/plugins/if_' '/usr/local/etc/munin/plugins/if_vmx4'
ln -s '/usr/local/share/munin/plugins/if_' '/usr/local/etc/munin/plugins/if_vmx5'
ln -s '/usr/local/share/munin/plugins/if_' '/usr/local/etc/munin/plugins/if_vmx6'
ln -s '/usr/local/share/munin/plugins/if_' '/usr/local/etc/munin/plugins/if_vmx7'
ln -s '/usr/local/share/munin/plugins/if_' '/usr/local/etc/munin/plugins/if_vmx8'
ln -s '/usr/local/share/munin/plugins/if_errcoll_' '/usr/local/etc/munin/plugins/if_errcoll_em0'
ln -s '/usr/local/share/munin/plugins/if_errcoll_' '/usr/local/etc/munin/plugins/if_errcoll_pppoe2'
ln -s '/usr/local/share/munin/plugins/if_errcoll_' '/usr/local/etc/munin/plugins/if_errcoll_pppoe3'
ln -s '/usr/local/share/munin/plugins/if_errcoll_' '/usr/local/etc/munin/plugins/if_errcoll_vmx0'
ln -s '/usr/local/share/munin/plugins/if_errcoll_' '/usr/local/etc/munin/plugins/if_errcoll_vmx1'
ln -s '/usr/local/share/munin/plugins/if_errcoll_' '/usr/local/etc/munin/plugins/if_errcoll_vmx2'
ln -s '/usr/local/share/munin/plugins/if_errcoll_' '/usr/local/etc/munin/plugins/if_errcoll_vmx3'
ln -s '/usr/local/share/munin/plugins/if_errcoll_' '/usr/local/etc/munin/plugins/if_errcoll_vmx4'
ln -s '/usr/local/share/munin/plugins/if_errcoll_' '/usr/local/etc/munin/plugins/if_errcoll_vmx5'
ln -s '/usr/local/share/munin/plugins/if_errcoll_' '/usr/local/etc/munin/plugins/if_errcoll_vmx6'
ln -s '/usr/local/share/munin/plugins/if_errcoll_' '/usr/local/etc/munin/plugins/if_errcoll_vmx7'
ln -s '/usr/local/share/munin/plugins/if_errcoll_' '/usr/local/etc/munin/plugins/if_errcoll_vmx8'
ln -s '/usr/local/share/munin/plugins/if_packets_' '/usr/local/etc/munin/plugins/if_packets_em0'
ln -s '/usr/local/share/munin/plugins/if_packets_' '/usr/local/etc/munin/plugins/if_packets_pppoe2'
ln -s '/usr/local/share/munin/plugins/if_packets_' '/usr/local/etc/munin/plugins/if_packets_pppoe3'
ln -s '/usr/local/share/munin/plugins/if_packets_' '/usr/local/etc/munin/plugins/if_packets_vmx0'
ln -s '/usr/local/share/munin/plugins/if_packets_' '/usr/local/etc/munin/plugins/if_packets_vmx1'
ln -s '/usr/local/share/munin/plugins/if_packets_' '/usr/local/etc/munin/plugins/if_packets_vmx2'
ln -s '/usr/local/share/munin/plugins/if_packets_' '/usr/local/etc/munin/plugins/if_packets_vmx3'
ln -s '/usr/local/share/munin/plugins/if_packets_' '/usr/local/etc/munin/plugins/if_packets_vmx4'
ln -s '/usr/local/share/munin/plugins/if_packets_' '/usr/local/etc/munin/plugins/if_packets_vmx5'
ln -s '/usr/local/share/munin/plugins/if_packets_' '/usr/local/etc/munin/plugins/if_packets_vmx6'
ln -s '/usr/local/share/munin/plugins/if_packets_' '/usr/local/etc/munin/plugins/if_packets_vmx7'
ln -s '/usr/local/share/munin/plugins/if_packets_' '/usr/local/etc/munin/plugins/if_packets_vmx8'
ln -s '/usr/local/share/munin/plugins/iostat' '/usr/local/etc/munin/plugins/iostat'
ln -s '/usr/local/share/munin/plugins/load' '/usr/local/etc/munin/plugins/load'
ln -s '/usr/local/share/munin/plugins/memory' '/usr/local/etc/munin/plugins/memory'
ln -s '/usr/local/share/munin/plugins/netstat' '/usr/local/etc/munin/plugins/netstat'
ln -s '/usr/local/share/munin/plugins/ntp_' '/usr/local/etc/munin/plugins/ntp_45.87.76.3'
ln -s '/usr/local/share/munin/plugins/ntp_kernel_err' '/usr/local/etc/munin/plugins/ntp_kernel_err'
ln -s '/usr/local/share/munin/plugins/ntp_kernel_pll_freq' '/usr/local/etc/munin/plugins/ntp_kernel_pll_freq'
ln -s '/usr/local/share/munin/plugins/ntp_kernel_pll_off' '/usr/local/etc/munin/plugins/ntp_kernel_pll_off'
ln -s '/usr/local/share/munin/plugins/ntp_offset' '/usr/local/etc/munin/plugins/ntp_offset'
ln -s '/usr/local/share/munin/plugins/ntp_states' '/usr/local/etc/munin/plugins/ntp_states'
ln -s '/usr/local/share/munin/plugins/open_files' '/usr/local/etc/munin/plugins/open_files'
ln -s '/usr/local/share/munin/plugins/swap' '/usr/local/etc/munin/plugins/swap'
ln -s '/usr/local/share/munin/plugins/systat' '/usr/local/etc/munin/plugins/systat'
ln -s '/usr/local/share/munin/plugins/uptime' '/usr/local/etc/munin/plugins/uptime'
ln -s '/usr/local/share/munin/plugins/users' '/usr/local/etc/munin/plugins/users'

- Paste the listed shell commands so they get executed
- Restart munin-node: # service munin-node restart

Code Select Expand

# service munin-node restart
Stopping munin_node.
Waiting for PIDS: 46787.
Starting munin_node.

- (Optional) Verify whether connectivity from the munin master to OPNsense is OK and whether all plugins are listed:

Code Select Expand

root@srv-8:~# nc 10.0.10.1 4949
# munin node at OPNsense-v1
list
cpu df df_inode if_em0 if_errcoll_em0 if_errcoll_pppoe2 if_errcoll_pppoe3 if_errcoll_vmx0 if_errcoll_vmx1 if_errcoll_vmx2 if_errcoll_vmx3 if_errcoll_vmx4 if_errcoll_vmx5 if_errcoll_vmx6 if_errcoll_vmx7 if_errcoll_vmx8 if_packets_em0 if_packets_pppoe2 if_packets_pppoe3 if_packets_vmx0 if_packets_vmx1 if_packets_vmx2 if_packets_vmx3 if_packets_vmx4 if_packets_vmx5 if_packets_vmx6 if_packets_vmx7 if_packets_vmx8 if_pppoe2 if_pppoe3 if_vmx0 if_vmx1 if_vmx2 if_vmx3 if_vmx4 if_vmx5 if_vmx6 if_vmx7 if_vmx8 iostat load memory netstat ntp_45.87.76.3 ntp_kernel_err ntp_kernel_pll_freq ntp_kernel_pll_off ntp_offset ntp_states open_files swap systat uptime users

- Configure the OPNsense host on the munin master:

Code Select Expand

[OPNsense.mydomain.local]
        address 10.0.10.1
        use_node_name yes

- Restart munin on master:

Code Select Expand

# service munin restart
stop: Unknown instance:
munin stop/waiting

- Wait for your munin interval (default 5 mins) and enjoy initial OPNsense statistics generated on munin master

Hi there,

One of my WAN connections had its connection type changed from DHCP to PPPoE.
However on the Gateways listing, I still get "ISP1_DHCP" and cannot get rid of the DHCP naming: when editing the GW, I get:

The following input errors were detected:
Changing name on a gateway is not allowed.

Just to avoid any misunderstandings when I take a look at this page in a year or so from now... is there any way to change the GW name manually?

Thanks.

Hi,

I'm trying to move my firewall from pfSense to OPNsense but am a bit puzzled regarding Gateway Groups.
In the MultiWAN guide I read that when picking a GW group for a particular network, traffic intended for the firewall itself will be routed wrongly:

This rule will utilize the gateway group for all traffic coming from our LAN network. This also means that traffic intended for the firewall itself will be routed in this (wrong) direction. That is why Step 5 is needed for our DNS traffic going to and coming from our DNS forwarder on the firewall itself.

My pfSense is at 10.0.0.1 and has GW groups configured for the interface.
My OPNsense is at 10.0.10.18 currently and has similar GW groups configured.
No particular firewall rules are added for DNS traffic on either system.

Code Select Expand

C:\>nslookup
Default Server:  UnKnown
Address:  10.0.10.22

> server 10.0.10.1
Default Server:  [10.0.10.1]
Address:  10.0.10.1

> google.com
Server:  [10.0.10.1]
Address:  10.0.10.1

Non-authoritative answer:
Name:    google.com
Addresses:  2a00:1450:400e:809::200e
          216.58.211.110

> server 10.0.10.18
Default Server:  [10.0.10.18]
Address:  10.0.10.18

> google.com
Server:  [10.0.10.18]
Address:  10.0.10.18

DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
*** Request to [10.0.10.18] timed-out

---> Adding a rule to allow DNS traffic on OPNsense <---

> google.com
Server:  [10.0.10.18]
Address:  10.0.10.18

Non-authoritative answer:
Name:    google.com
Addresses:  2a00:1450:400e:80d::200e
          172.217.168.238

>

Can anyone explain why this is the case? What are the advantages of doing it the OPNsense way?
As it comes with annoyances: all services that need to be available on OPNsense need to have a firewall rule added. Not just DNS, but also SSH, Munin-node, ..., and this for all the interfaces/VLANs that exist.

Hi there,

I've got two WANs with IP addresses that might change (although they don't do so often).
I'd like to be able to connect to my network from wherever I am, through either of the WANs, so I have two dynamic addresses (isp1-myname.mydnsprovider.tld and isp2-myname.mydnsprovider.tld).

However in Services: Dynamic DNS I cannot pick which interface to use for DynDNS. So both addresses are set to the same (default) WAN IP address.
Is there another way to configure this somehow?

Hi all,

Coming from pfSense, I'm looking for an equivalent setting of "Flush all states when a gateway goes down" (pfSense: System -> Advanced -> Miscellaneous).
I needed to use this setting as my SIP trunk would stop working periodically.

Is there some equivalent setting in OPNsense to this? Or is this included in the "Allow default gateway switching" setting ("If the link where the default gateway resides fails switch the default gateway to another available one."), and hence should this one be set instead?

Thanks!

Hi guys,

I'm trying to configure two Gateway groups (dual-WAN) but both are stuck at "Pending" state.
What I did:
- Add a Tier 1 and a Tier 2 for the GW group
- Trigger level: member down
- Apply changes

When Googling I understand this could be related to the fact that a "monitor" is missing. Well, that might be true, but:
1. Where can this monitor be added?
2. Coming from pfSense, no such monitor was required (for "member down" at least) - does anyone know the reason why OPNsense would require these?

Thanks!

Hi,

I'm setting up OPNsense on VMware with 6GB RAM and 90GB (virtual) Hard Drive.
During the installation I'm asked whether I want to continue with a "recommended swap partition of 8192M".

I do not think this is necessarily 'recommended' in the above setup, and hence should be set zero? Any thoughts?

From the installation docs on https://docs.opnsense.org/manual/install.html, I see disabling swap is typically done for embedded systems.

Hi all,

I'm running OPNsense on VMware ESXi 6.7 and have about 10 VLANs.

What is the most recommended way of working?
A/ Define the VLANs on the VMware VM definition (so a unique interface is presented to OPNsense)
B/ Apart from the mandatory LAN and a WAN, provide a trunk interface to OPNsense with all VLANs and define the other VLANs in OPNsense (subinterface of the trunk)

What are the advantages/disadvantages of each approach?

Advantages of A:
- Security: in case OPNsense gets breached, the VLANs that are not defined to the VM will not be visible

Advantages of B:
- When adding a VLAN, OPNsense doesn't need to be restarted

I'm sure there are more - any ideas?
E.g. is there an expected CPU overhead or speed drop with one approach vs. the other?
Or expected issues when moving OPNsense to a different system?

Hi guys,

My situation:
- ISP A: n=1 Public IP address, bound to a certain MAC address
- ISP B: n=1 Public IP address (though PPPoE)
- Latest OPNsense using different VLANs with WAN failover (i.e. VLAN 1 using ISP A by default, if not available then ISP B; VLAN 2 using ISP B by default, if not available then ISP A)

I'd like to use OPNsense High Availability so I can reboot my host easily.
Is that elegantly possible with just n=1 IP address per WAN link (out of which one is bound to a MAC address, which I can choose (once))?

Thanks!

Hi there,

When I try to install OPNsense 20.7 on VMware ESXi 6.7, I get following error message when I do either the Guided or the Manual install:

The installer could not find any disks suitable for installation (IDE or SCSI) attached to this computer. If you wish to install OPNsense on an unorthodox storage device, you will have to exit to a LiveCD command prompt and install it manually, using the file /README as a guide.

The installation medium I'm using is OPNsense-20.7-OpenSSL-dvd-amd64.iso.
The VMware guest was created with default settings for Other -> "FreeBSD 12 or later versions (64-bit)".

Any idea what's going on/wrong?