Topics

I've written a (quick and dirty) script to handle a specific scenario for my OPNSense VM.

I'm hosting OPNSense on a KVM hypervisor with 10Gb NICs (named ixl0..9).
I backup that VM every night, and restore it on a second much smaller hypervisor which has 1Gb NICs (named igb0..9).
This is done because I don't want to handle a high availability scenario which is a quite insane amount of work (every interface needs a VRRP, and some of the plugins aren't HA aware, meaning more work).

Anyway, once I've restored the VM on my spare device, it won't work unless I rename my interfaces from ixl to igb, which makes sense.
In order to make things more smooth, I came up with (perhaps a bad) idea:

1. Mount restored disk image locally on backup KVM hypervisor
2. Replace interface names in config.xml
3. Unmount disk image
4. Profit !

So far, I came up with a script to handle this.
The script supposes that the host hypervisor has a running ZFS implementation, and can mount qemu disk images via qemu-nbd.

I'm pretty sure there might be better solutions, perhaps you could tell me how you manage cross hypervisor OPNsense backup/restore scenarios ?

Anyway, please find the script at https://github.com/deajan/linuxscripts/blob/master/virsh/offline_rename_opnsense_interfaces.sh

Any feedback is appreciated.

Hello,

I'm an OPNsense user since almost 10 years now, and use it at home & work.
I've setup a couple of home ipv6 networks, where I had to use NPTv6 since ISP wouldn't hand out a bigger than /64 prefix delegation, so I have basic IPv6 knowledge.

My problem today is quite different.
I've setup bgp routers which have IPv4 and IPv6 sessions.
My bgp routers are bridged to my OPNSense, which has a DMZ vlan interface in which I have a couple of servers.

So far, my setup looks like:


(BGP-Router(s))--------[bridge]----------(OPNSense WAN_________OPNSense DMZ)----------[bridge]-------------[VM]
2001:X:Y:0::254/48 (VRRP)               2001:X:Y:0:1/48        2001:X:Y:FF:254/64                          2001:X:Y:FF::1/64     
2001:X:Y:0::253/48 (RTR1)               GW 2001:X:Y:0:254                                                  GW 2001:X:Y:FF:254
2001:X:Y:0::252/48 (RTR2)                   

I've setup the wan address as static since I don't plan on running a dhcp server on bgp routers.
I've setup the dmz address as static too, as well as the VM.

I can't ping the BGP routers from the VM (traceroute shows that it stops at opnsense). Ping to OPNsense works.
I can ping both the BGP routers and the VM from OPNSense.
I can ping OPNSense from the BGP routers.

I came to the conclusion that OPNSense doesn't route IPv6 from DMZ to WAN interface.
I did of course setup a ipv4/ipv6 any to any rule on DMZ interface for my tests.
I've also checked that ipv6 forwarding is enabled via:
```
# sysctl net.inet6.ip6.forwarding
net.inet6.ip6.forwarding: 1
```

My IPv6 routing table looks sane to me:
```
netstat -nr

[ipv4...]

Internet6:
Destination                       Gateway                       Flags         Netif Expire
default                           2001:X:Y:0::254   UGS          vtnet4
::1                               link#6                        UHS             lo0
2001:X:Y::/48                 link#5                        U            vtnet4
2001:X:Y:0::1         link#6                        UHS             lo0
2001:X:Y:FF::/64            link#25                       U            vlan04
2001:X:Y:FF::254            link#6                        UHS             lo0
fe80::%vtnet4/64                  link#5                        U            vtnet4
fe80::5054:ff:feb9:2fc7%lo0       link#6                        UHS             lo0
fe80::%lo0/64                     link#6                        U               lo0
fe80::1%lo0                       link#6                        UHS             lo0
fe80::%vlan04/64                  link#25                       U            vlan04
fe80::5054:ff:fe32:b0eb%lo0       link#6                        UHS             lo0
```

All my IPv4 networking works, so my problem looks really IPv6-only.
Since I've setup a /48 on WAN and a /64 on DMZ, is there anything else I should have configured except the firewall rule ?

Also, side question, since my VM in DMZ interface will be publicly accessible and have an AAAA record, I configured it with a static ipv6.
Is that a "good practice", or should I go a SLAAC / DHCP6 way ? If so, doesn't that make things more complicated when trying to find it's IP to setup AAAA records ?

Thanks for any insight.

Hello,

I'm facing a real WTF situation and really could need some advice here.
I have a OPNSense FW (25.1.10), and cannot wrap my head around my issue.

Randomly, pings to 8.8.8.8 or 1.1.1.1 get blocked by the firewall.
From the same LAN subnet, some computers can ping, others can't.
After a while, other computers can't ping, and some that couldn't now can.

When looking at the firewall logs, the ICMP packets that get blocked are processed by the same rule as that passes other ICMP packets.
When clicking on the rule link, the window closes since the rule isn't found (see the attached screenshots).

In order to outrule some factors, I've:
- Disabled Zenarmor
- Disabled Suricata
- Disabled Crowdsec
- Marked my default gateway as always on (no monitoring)

When I ping 1.1.1.1 or 8.8.8.8 from the firewall itself, of course both respond.

I've also tried to check the Disable reply-to wan setting in firewall>settings>advanced.
I've checked that I don't have any special routes to 1.1.1.1 nor 8.8.8.8

This happened already with 24.7 series as far as I can remember.
I've just updated 25.1.10 to 25.1.11 and a computer that could ping 8.8.8.8 but couldn't ping 1.1.1.1 can now ping 1.1.1.1 but can't ping 8.8.8.8 anymore.
I've also exported my opnsense config file and searched for those IPs in order to make sure I didn't forget anything.
The only entry I found for 1.1.1.1 is for query forwarding in unbound.

I'm totally puzzled as why this is random.
I'd be grateful for any clue where to search for.

Thank you.

Hello,

I've setup a Wireguard site to site tunnel between two OPNSense 24.1.2_1 instances.
So far so good, tunnel is up, firewall rules allow any IPv4 traffic on "Wireguard (Group)" interfaces.

From any computer on site A (10.0.0.0/24) I can ping any computer on site B (10.0.1.0/24) and from B to A, so everything looks good.

But, ping (and others protocols) doesn't work from the firewall itself, eg OPNSense A (10.0.0.1) to OPNSense B (10.0.1.1), neither does it work from OPNSese B to OPNSense A.

Now the strange part is, if I happen to add the OPNSense source IP to the ping, eg `ping -S 10.0.0.1 10.0.1.1`, the ping works.

I'm a bit puzzled here.
The routing tables look good (10.0.1.0/24 via wg0 on OPNSense A and 10.0.0.0/24 via wg0 on OPNSense B).
It looks like the originating IP isn't good when running ping from OPNSense.

So basically, from OPNSense A:
`ping 10.0.1.1` does not work
`ping -S 10.0.0.1 10.0.1.1` works

Why do I need to specify the source IP when trying to ping the other firewall ?
I need the firewalls to be able to speak with eachother (for DNS resolution), how can I achieve this ?

Looks like a bug to me.

Best regards.

PS: I've verified (multiple times) my config according to the docs.
Any idea is welcome ^^

PS2: Shall I configure an outgoing NAT rule ? Doesn't ring right to me.

So, I've got a couple of OPNSenses running with Zenarmor (23.7 to 24.1.2).
I've installed the qemu-guest-agent plugin into OPNSense.

When I try to make a snapshot with quiescing eg `virsh snapshot-create myvm.local --disky-only --atomic --quiesce`, I get the following error:

error: internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to freeze /usr/local/zenarmor/output/active/temp: Resource deadlock avoided

As far as I understand, the qemu-agent freeze / thaw scripts would need to exclude that path, which is basically a RAM disk.

I've searched for the freeze / thaw scripts in OPNsense but couldn't find them.
Any idea where to configure this exception perhaps ?

Best regards.

Hello,

I have a couple of linux VM (Almalinux/CentOS/Debian/SLES) behind an OPNSense and would like to limit internet acces from those VMs for security reasons.
The problem is that I'd still need those machines to fetch their security updates from various mirrors.

If I setup firewall rules to allow things like mirrors.distro.tld, nothing works great since mirrors.distro.tld will redirect to some mirror, and I cannot just add all various mirrors to fqdn aliases.

So far I've investigated:
- Using squid proxy with whitelists (doesn't work well because for the same reasons above)
- Setting up local repositories... Way too much work and space used
- Making maintenance windows where internet is available via schedules (doesn't right right in terms of security)

Has anyone ever found an elegant solution for this scenario ?
Thanks.

Hello,

I'm running anOPNsense 23.1.6 virtualized via KVM on AlmaLinux 9.1, on a Xeon Gold 6312U.
All nics are virtio, all hardware accelerations are disabled in OPNsense.
So far so good, without using Suricata, I get about 850Mbits in/out speeds:

[root@virtual_guest_client ~]# ./speedtest -s 20411

   Speedtest by Ookla

      Server: Quantcom, a.s. - Prague (id: 20411)
         ISP: Ouiheberg Sarl
Idle Latency:    29.01 ms   (jitter: 0.12ms, low: 28.84ms, high: 29.11ms)
    Download:   849.25 Mbps (data used: 1.2 GB)
                 28.96 ms   (jitter: 1.31ms, low: 27.73ms, high: 38.89ms)
      Upload:   903.78 Mbps (data used: 1.3 GB)
                 29.80 ms   (jitter: 0.86ms, low: 28.38ms, high: 36.39ms)
Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/29c7f928-abc2-41c8-ba13-59359d3b18c7

If I happen to enable suricata, using ET Telemetry ruleset, IPS mode with Hyperscan engine, upload speed are terrible:

[root@virtual_guest_client ~]# ./speedtest -s 20411

   Speedtest by Ookla

      Server: Quantcom, a.s. - Prague (id: 20411)
         ISP: Ouiheberg Sarl
Idle Latency:    29.72 ms   (jitter: 0.17ms, low: 29.45ms, high: 29.85ms)
    Download:   875.42 Mbps (data used: 1.6 GB)
                 28.11 ms   (jitter: 3.05ms, low: 27.52ms, high: 291.24ms)
      Upload:    67.82 Mbps (data used: 55.4 MB)
                 29.38 ms   (jitter: 0.48ms, low: 28.12ms, high: 30.38ms)
Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/9f80a999-38de-4e4e-bf5d-31d3352d0726

I've played with various things in order to find what the culprit could be:

- Using E1000 nics was disastrous, got 700% cpu usage on host when OPNsense was idle, instead of roughly 5%, also got some packet loss
- Using a Q35 machine instead of a 440fx with qemu: Did not change alot
- Playing with tuned profiles: Got 30% better upload speed when using performance cpu governor instead of conservative
- Isolated CPUs from host and made the OPNsense cpu affinity use those cpus: Did not change alot

So it looks indeed that my problem is CPU bound, since using a better cpu governor helps getting better speeds.
Nevertheless, my host has a 2.4Ghz Xeon CPU, with 3.6Ghz turbo frequency.
The CPU cores were indeed clocked at 3.6Ghz while performing the tests.

I cannot imagine Suricata being so hungry that 8 x 3.6Ghz CPUs won't suffice for having 1GB throughput.

Can someone confirm if this has something todo with KVM ?
If so, any advices perhaps ?

Best regards.

Hello,

I have a couple of OPNSense VMs that I usually run in Linux KVM hosts.
I usually pass the PCIE Intel I211AT cards directly to the VM via pci passthrough, thus avoiding virt driver issues.

When downloading "big" files via curl or wget, I noticed that I get 20-50Mbps which quickly fall down to 1-2Mbps.
Downloading in front of an OPNSense will give me about 80Mbps.

I've fooled a bit around, trying to disable suricata, zenarmor, etc...
I've also played with hardware offloading on ethernet cards. No better results.

Finally, I've changed the timer settings in libvirtd to something like

Code Select Expand


<clock offset='utc'>
    <timer name='hpet' present='yes'/>
    <timer name='hypervclock' present='yes'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='kvmclock' present='yes'/>
  </clock>


I've also replaced the pc440fx-rhel7.6.0 emulation type with pc-q35-rhel9.0.0.
Having done these, I now get "correct speeds", about 20-40Mbps.

Anyone has experienced same issues ?

PS: setup is a VM with 4 vCPU, 8Gb RAM, 30Gb disk on a i5-8365U 4core computer which doesn't use 10% of it's CPU.

Best regards.

Hello,

I am currently seeking a way to create cheap layer 2 tunnels accros WAN links.
My primarty goal is to interconnect two sites (A and B) just like using a (very long) ethernet cable.

The scenario is the following:
- Site A and site B are in different countries, both have quite okay WAN links (RTT between sites is 14ms)
- Site A has some industrial machines which are operated by some specific industrial computers
- Site B has perfect clones of the industrial computers of site A

Whenever one/more computers of site A fails, I'd like the clones from site B to interact directly with the industrial machines from site A (as disaster plan).

Most of the traffic is layer 3 (TCP/UDP/ICMP), but some traffic is layer 2 (ARP, VLAN, DHCP).
The layer 2 traffic is mandatory for that setup to work, so I am heading for a layer 2 tunnel.

So far I've looked at the following routes:

- L2TP over IPsec: looks like a big overhead to me, ie 128 bytes of headers
- OpenVPN tap: well OpenVPN is very slow compared to IPSec / Wireguard, and I would like to achieve as much bandwidth and low latency as I may get. So OpenVPN is the fallback if I don't get anything to work
- VxLAN (or GENEVE, or GRETAP) over Wireguard: looks promising ?
- Tinc ?
- Zerotier ?

So here are my questions:
What's the best solution (the most performance oriented one, without sacrificing security) ?
Has anyone achieved a good performing layer 2 tunnel setup with OPNsense yet ?

Any feedback is appreciated.

Thanks.

PS: This is my first post on the OPNsense forum (former pfSense user), so I probably don't know OPNsense good enough yet ;)