Topics

[it seems wireguard doesn't tolerate more peers with allowed ips of 0.0.0.0/0 set against the same local listener.]

I have 3 sites, each site has multi WAN.

Site A: WAN1 & WAN2 plus HA firewall pair
Site B: WAN1 & WAN2 plus HA firewall pair
Site C: WAN1 & WAN2 plus HA firewall pair

I'm using FRR with BGP for dynamic routing and got it working great with 2 sites and excellent WAN failover, only losing 2 pings during WAN failover. As soon as I added a third site, I get a strange "allowed ips: (none)" and routing problems.

The allowed ips set for the peer is 0.0.0.0/0, but, it seems wireguard doesn't tolerate more peers with allowed ips of 0.0.0.0/0 set against the same local listener.

The reason I want to set 0.0.0.0/0 is I want to do all my routing using FRR, so, I don't want to have to set the peer allowed IP addresses in wireguard plus then control the IP addresses in FRR BGP.

See the attachment, it shows that the running config for the peer smPI... has allowed ips: (none) (but I can assure you, it has allowed ips of 0.0.0.0/0 set) and routing doesn't work. As soon as I put a list of allowed ips as expected from that peer, volia, it works.

The local endpoint (listener) of course has disable routes set.

FRR and BGP and BFD all working great.

Environment

• OPNsense 22.7.2
• Wireguard
• Wireguard-kmod
• 10-wireguard CARP hook script

OPNsense 22.7.1 (and earlier versions I have discovered)

IP aliases set on the primary firewall do not sync across to the backup firewall.

[Trouble shooting steps I have tried]

Code Select Expand

2022-07-23T18:51:24 Error opnsense /usr/local/etc/rc.filter_configure: There were error(s) loading the rules: /tmp/rules.debug:806: sticky-address cannot be redefined - The line in question reads [806]: pass in quick on vlan01 route-to {( vlan02 202.202.202.202 )} sticky-address inet proto {tcp udp} from $groveseg to $Marshal_updates port $http_https keep state label "9e64a311a494a21cfdbefcba91dad3a5" # : Allow ServerSEG license check


As soon as a add WAN fail-over capability to rules, this break badly.
I can't seem to pin down exactly what is going on, my best guess is the WAN fail-over "WAN1_failover_WAN2" gateway group is just not working.
Often, I can get the issue to go away by moving the rule to the top of the interface rules, or to the end. But that doesn't always work either.

Trouble shooting steps I have tried

• Deleted the offending rule and made it again (doesn't always fix it)
• Moving the rule around for rule order (also doesn't always fix it)
• Rebooting OPNsense (definitely doesn't fix it)
• Exported the config, looked the config by hand (seems fine) - re-import the config and reboot (doesn't fix it)
• Also occurs when I take an existing rule and change the default gateway to the new WAN fail-over gateway

The WAN fail-over group looks perfect.

Any ideas anyone?

[Instead, Mobile IPsec fails:]

It appears that OPNsense incorrectly requires the client certificate to be installed inside OPNsense. This should NOT be required. If OPNsense has a server certificate issued from an external CA, and, a copy installed of that external CA (just the public cert, no private key), then OPNsense should be able to correctly verify the authenticity of the remote Mobile IPsec client presented client certificate.

Instead, Mobile IPsec fails:
(IP addresses and FQDN's changed for privacy)

Code Select Expand

022-07-21T23:37:23 Informational charon 14[NET] <con9|38> sending packet: from 202.202.202.202[4500] to 101.101.101.101[10673] (80 bytes)
2022-07-21T23:37:23 Informational charon 14[ENC] <con9|38> generating IKE_AUTH response 9 [ EAP/FAIL ]
2022-07-21T23:37:23 Informational charon 14[IKE] <con9|38> EAP method EAP_TLS failed for peer GregsiPhone.domain.local
2022-07-21T23:37:23 Informational charon 14[ENC] <con9|38> parsed IKE_AUTH request 9 [ EAP/RES/TLS ]
2022-07-21T23:37:23 Informational charon 14[NET] <con9|38> received packet: from 101.101.101.101[10673] to 202.202.202.202[4500] (112 bytes)
2022-07-21T23:37:23 Informational charon 14[NET] <con9|38> sending packet: from 202.202.202.202[4500] to 101.101.101.101[10673] (96 bytes)
2022-07-21T23:37:23 Informational charon 14[ENC] <con9|38> generating IKE_AUTH response 8 [ EAP/REQ/TLS ]
2022-07-21T23:37:23 Informational charon 14[TLS] <con9|38> sending fatal TLS alert 'certificate unknown'
2022-07-21T23:37:23 Informational charon 14[TLS] <con9|38> no trusted certificate found for 'GregsiPhone.domain.local' to verify TLS peer
2022-07-21T23:37:23 Informational charon 14[TLS] <con9|38> received TLS intermediate certificate 'DC=local, DC=domain, CN=domain-domainECA-CA'
2022-07-21T23:37:23 Informational charon 14[TLS] <con9|38> received TLS peer certificate 'serialNumber=8714.21901, DC=local, DC=domain, CN=GregsiPhone'
2022-07-21T23:37:23 Informational charon 14[ENC] <con9|38> parsed IKE_AUTH request 8 [ EAP/RES/TLS ]
2022-07-21T23:37:23 Informational charon 14[ENC] <con9|38> received fragment #3 of 3, reassembled fragmented IKE message (1056 bytes)
2022-07-21T23:37:23 Informational charon 14[ENC] <con9|38> parsed IKE_AUTH request 8 [ EF(3/3) ]
2022-07-21T23:37:23 Informational charon 14[NET] <con9|38> received packet: from 101.101.101.101[10673] to 202.202.202.202[4500] (132 bytes)

Consider this case:

• Large installation, many hundreds of mobile phones
• Mobile phones are managed using a Mobile Device Manager (MDM) system
• Using that MDM, the phones obtain a client certificate without intervention using SCEP and the VPN configuration
• VPN auth is EAP-TLS and auto starts for any traffic to "domain.local", requiring no user action

This works with pfSense, but, on migration to OPNsense, we see OPNsense incorrectly appears to require the client certificate to be installed locally within OPNsense which just cannot happen at scale.

[Whats going on?]

I have noticed, that when you create a new site to site IPsec VPN tunnel, it simply will not become active.

I have done a reasonable amount of diagnostics and my finding is this:
If you create or modify a firewall rule or alias and save, thereby reloading the firewall rules, the site to site VPN tunnel will then come up.

Whats going on?
I think when you press save on the new IPsec tunnel, the OPNsense is not immediately updating the hidden IPsec allow rules on the WAN interface and/or not reloading the firewall rules and therefore IPsec traffic is blocked until a firewall rules reload is manually done.

["Related item not found"]

22.1.7_1
FRR version 7.5.1

I can get OSFP to talk to the neighbor or BGP just fine.
The issue is pruning routes.

What I want to do prune the routes sent via OSFP or BGP from site A to site B and back from site B to site A.
But, I need to send a series of routes.

Route Maps using multiple prefix lists seem to be the logical way to do this, but, as soon as I try and save a Route Map with more than 1 Prefix List it will not save stating:
"Related item not found"

What am I doing wrong?

How does anyone else prune OSPF or BGP routing between sites?

Could be quite useful.

Thanks.

Since upgrade from 22.1.6, all users cannot authenticate on OpenVPN using "remote Access (SSL/TLS + User Auth) and the backend for auth is local user and TOTP.

Nothing has changed but 22.1.6 upgrade to 22.1.7_1.

Tried:

• Rebooting
• Checking settings (but nothing has changed)
• Reset local user passwords

Code Select Expand

2022-05-16T17:00:05 Error openvpn 101.100.xxx.xxx:55438 TLS Auth Error: Auth Username/Password verification failed for peer
2022-05-16T17:00:05 Warning openvpn 101.100.xxx.xxx:55438 WARNING: Failed running command (--auth-user-pass-verify): external program exited with error status: 255
2022-05-16T17:00:05 Warning openvpn user 'username' could not authenticate.


If you want to enable MSS clamping on all IPSEC VPN tunnels, then, am I right, you set it here:

Firewall: Settings: Normalization

And, under detailed settings, you can then make a specific rule to enable MSS clamping on the IPSEC interface.

New "Firewall scrub rule"
Select Interface "IPSEC"
Max mss "1400"

See my screenshot.

Is that correct?
Is that all I need to do?

• I've setup some site to site VPN tunnels using WG for a migration project from another firewall using IPSEC tunnels
• I have build specific fw rules on the "Wireguard (Group" fw rules tab, including rules for TCP/UDP
• Citrix users, running an older Citrix client can logon, but, newer client including thin client OS couldn't logon
  

After a bit of work, I figured out that OPNsense is blocking UDP traffic. TCP and ICMP is passing just fine, but all UDP traffic is getting blocked.

Somehow, TCP and ICMP are routing up and down the WG tunnels and passing correctly through the firewall rules, but, not UDP.

See the screen capture showing blocked UDP. I guarantee 100% there IS a firewall rule on the "Wireguard (Group)" fw rules tab to allow this UDP traffic, but, somehow TCP and ICMP are being treated differently.

Questions
I don't have a "wg0" interface setup - do I need to add that "wg0" interface?
If I add that, do have to give it an IP address?

Any help appreciated.

[States incorrectly:]

Reference:
https://docs.opnsense.org/manual/how-tos/carp.html

States incorrectly:
Note
IP Alias is not synchronized to slave, be sure to also add it to your second machine.


Update:
IP Alias are sync'd to the backup firewall.

Code Select Expand

/boot/kernel/kernel text=0x184dd54 data=0x1e3968+0x76ed60 / elf64_loadimage: read failed

Tried USB DVD image and USB VGA installer.

Tried disabling UEFI, booting legacy.

If I set the server BIOS to normal booting via UEFI;

Code Select Expand

/boot/kernel/kernel text=0x184dd54 /

If I set the server BIOS to booting via legacy;

Code Select Expand

/boot/kernel/kernel text=0x184dd54 data=0x1e3968+0x76ed60 / elf64_loadimage: read failed


Any ideas?

[Services > Intrusion Detection > Administration > Download]

My custom-rules.xml was working fine.

After upgrade, my custom-rules.xml is still present in

Code Select Expand

/usr/local/opnsense/scripts/suricata/metadata/rules/custom-rules.xml but it's no longer appearing in the GUI:

Services > Intrusion Detection > Administration > Download

Interestingly it's still in the policy section.

It looks like the 21.7.3 upgrade to suricata 6.0.3_2 has lost the ability to load custom.xml rules.

HAPROXY reports September/2021 – CVE-2021-40346: Duplicate 'Content-Length' Header Fixed
https://www.haproxy.com/blog/september-2021-duplicate-content-length-header-fixed/

Can we pleas update the HAPROXY package to v2.2.17

Thanks!

[WireGuard needs an option in the package ;]

All that is needed to get it going properly, is have WireGuard (WG) follow the CARP master and STOP on the backup firewall and only start if it is the CARP master.

I don't care that WG doesn't send packets out the carp interface or I can't control which interface IP it uses, because the other end doesn't care either. It make NO difference because WG is stateless. It doesn't matter if the remote firewall sends from a different IP address to what the local firewall sends from. It just make no difference. Nobody cares. That actually makes it quite awesome.

I spent the last few days working on a Wireguard multiWAN with HA site to site setup.

SiteA
2 x firewalls in HA with CARP and WAN1 and WAN2

SiteB
2 x firewalls in HA with CARP and WAN1 and WAN2

FRR already has lots of cool features to follow carp, so it's no problem to get routing only running on the primary firewall.

All that is needed to get multi site, primary/backup HA WireGuard running in an active/passive is to have WG stopped on the backup firewall.

That's it!

I LOVE the stateless nature of WG and how fast it sets up a VPN tunnel compared to IPSEC. It's awesome. But, that does make a nightmare if WG is running on both the primary and backup firewall.
Because of the stateless design of WG it's likely that the local primary and local backup firewall both try to have the same VPN tunnel up to the remote firewall.
Without having WG as active/passive, for a pair of HA firewalls each end site to site you need unique 8 tunnels and the problem  is;
1. The complexity
2. You can't HA sync WG nor FRR because it all needs to be different
3. What about the WG interfaces needed which need to be different on primary/backup firewall?

WireGuard needs an option in the package ;

• Enable CARP Failover
• Follow this (drop down box) CARP VHID (user select which CARP to follow, probably the LAN CARP)

With that simple change, WG becomes instantly ready for multiWAN HA

• the config becomes the same on the primary and the backup firewall
• you can HA sync the FRR and Wireguard config
• 8 VPN tunnels and very complex routing become a far simpler design of 2

Please please please please.

[usr/local/opnsense/scripts/suricata/metadata/rules/spamhausBCL.xml]

I can't get my custom IDS rules to load. I've rebooted, waited a day, etc. Perhaps when using Proofpoint ET ruleset it won't add custom rules??

Here my file "spamhausBCL.xml" and it's placed in usr/local/opnsense/scripts/suricata/metadata/rules/spamhausBCL.xml

Code Select Expand


<?xml version="1.0"?>
<ruleset>
    <location url="https://pub-api.spamhaus.org/api/snort/" prefix="spamhausBCL"/>
    <files>
        <file url="https://pub-api.spamhaus.org/api/snort/?account=xxxxxxxxxxxxxxx&key=yyyyyyyyyyyyy"
              description="Spamhaus Botnet Controller List"
              documentation_url="https://www.spamhaus.org/bcl/"
        >spamhausBCL.rules</file>   
    </files>
</ruleset>


Any ideas?

[VPN: IPsec: Status Overview]

21.1.4
Trying to make a site to site IPSEC tunnel from a HA opnSense cluster using the CARP address to a single opnSense fw.

Remote site, single fw: VPN: IPsec: Status Overview
I noticed the Stats the Bytes out was counting up but never any Bytes in.

HA clustered fw: VPN: IPsec: Status Overview
I noticed the Stats the Bytes in and out was zero.

That got me thinking that the ESP traffic was getting dropped and then I discovered that on the HA site;
GUI> Firewall: Rules: WAN

The Automatically generated rules had nothing for the IPsec tunnel that had the CARP address set in P1. If I changed P1 from the CARP address to the actual WAN interface IP, then the auto rules get created.

Looks like a bug with CARP and IPSEC.

Could that be why the tunnel got created but no traffic would pass?

I need to accept the newer ip bgg-community list format from my upstream provider.

Can we please add bgp community-list support?

This is all well supported in FRR, just missing from the OPNsense GUI configuration.

[noatime]

Could we change the default installation to set the ufs filesystem to not update the last file access time (noatime) to reduce SSD writes and a slight boost to disk performance.

I notice on my new installation noatime is not present;

Code Select Expand

# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/gpt/rootfs /               ufs     rw              1       1
/dev/gpt/swapfs         none            swap    sw              0       0
fdesc /dev/fd fdescfs rw 0 0
proc /proc procfs rw 0 0
root@gateway:/etc #



I manually edited fstab and added notime;

Code Select Expand

# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/gpt/rootfs /               ufs     rw,noatime              1       1
/dev/gpt/swapfs         none            swap    sw              0       0
fdesc /dev/fd fdescfs rw 0 0
proc /proc procfs rw 0 0
root@gateway:/etc #


[Dnsmasq]

With the awesome BIND plugin and unbound, probably a great time to remove dnsmasq entirely.

Surely nobody needs Dnsmasq anymore, unbound or BIND surely will do.

If really necessary to somehow still keep Dnsmasq, then don't auto install it and offer it as an optional plugin, but, I'd be happy to see it removed entirely.