Messages

interested to know what more you're running on the crashing instances. Mine has been doing fine and I've tinkered with my zenarmor installation, putting the db in ram etc. Aside from zenarmor I'm only using suricata, so not much that can interfere.

Do you know if there's a way to keep this working while simultaneously running a home VPN for my own devices?  I had a road warrior wireguard setup for my devices, but every configuration for Mullvad seems incompatible with it.  I'm not entirely sure why, but I think it's because the outbound NAT rule for the road warrior setup interferes with the outbound NAT rule for the Mullvad setup.  I'm not entirely sure how to work around this though.

I am running this and my own roadwarrior setup side by side.

Not sure how the NAT rules could collide TBH. They won't be running on the same source interfaces.
The NAT rule for road warrior setup to access other internal networks doesn't seem to be needed if you assigned it an interface. My rule below is disabled for WGHOME and I can still access my internal resources.

Code Select Expand

Interface Source Source Port Destination Destination Port NAT Address NAT Port Static Port Description    
WAN WGHOME net udp/ * * udp/ * Interface address         * NO NAT wgHome Internal     
WGSTO1 selectiveRoute          * * * Interface address          * NO NAT wgSTO-1 SelectiveRoute 


Big thank you first!

Now a question: what is the best practice in case of multiple (Mullvad) WiregGuard instances?

There are more than one DNS server you can use. Check out the first link in the original post, I believe he went thru the steps to make more than one tunnel 🤞😎

Nope, not even one VLAN, besides Unbound I have pretty much nothing going on

It does sound pretty much exactly like the problem I'm having though and I still have the problem but the initial super long load times are for the most part gone.

Does it happen if you don't have the webgui open at all? Mine never stalls when I'm SSH'd into it but as soon as I open the GUI (dashboard) it 100% all cores immediately with PHP.

Mind checking with SSH and TOP -P? Start top, download a steam game, start a speed test etc, and then open the dashboard and see if you can reproduce it that way. Assuming it's not fully borked without even doing anything.

I can use the GUI fine as long as I'm using spotify / youtube and browsing the net but any heavy load and its game over.

You didn't answer my question about power mgmt features enabled in the BIOS...

Sorry, my bad, there are no power saving related settings in the BIOS, the only one I would consider close is auto boot when power is supplied.

Are you running plenty of VLANS?

Could be completely unrelated problems, but mine seems to be semi-remediated.

Noticed my webgui log was LOADED with dead/dying sessions. Running plenty of vlans and using my normal FQDN for access, guessing it chose different IPs or similar which could've been the reason gui loaded so damn slow (everywhere).

• I put a 10 minute session timeout in settings > administration. Default 240 min.
• Added a dns host override entry outside of the search domain.

It still completely shit the bed when working with a lot of traffic, but it seems to only happen on the dashboard now. Doesn't really seem to be related to netflow either as it happens without traffic charts running. But I'm much to stupid to find the actual cause. The GUI is snappy again in most other areas even during higher load.

[(attach 1)]

Same problem on my intel 305 running proxmox, since 1 or 2 versions before the webgui rebuild (and before that a few years hiatus from opnsense).

Sometimes SSH and HTTPS can take minute(s) to load. Without any traffic to speak of being routed / blocked / scanned. It does seem to be faster browsing to the IP address in general.

1. Saturating the 1gpbs with steam / web tests @3-7% cpu usage.  (attach 1)

2. OPNsense practically dying as soon as I open the webgui when steam / webtests running (attach 2)

3. Nearly saturating the 1gpbs with linus ISO's over wireguard / selective routing @ around 40% cpu

Mine always work fine if I don't connect to the webgui. Start a download with webgui open and 100% cpu usage about the time the page shows. Which again can take minutes.

Ran opnsense on a fitlet2 and qotom g7i7 for years without any problems to speak of. This device should handle the same network with ease compared to those two.

Another opnsense instance /w DNS and certificates running for years on another proxmox server never had this problem. Altough firewall is shut down on there, it does have unbound with blocklist - so probably thats not related(?).

Have reinstalled the VM more than once. Tried different tunables, all yield the same problem. WAN is DHCP. I'm not finding information on wether I'm supposed to assign and enable the vtnet parent interfaces on this version, maybe thats the problem?

I have disabled everything except the DNS service, geoblock (1 rule w/ 1 country in it), selective routing (but problem happens on non-routed hosts). I initially thought it was due to netflow but disabling it made no difference. Most firewall rules are not logging anything. Reinstalled webgui and netflow.

The same prox that hosts opnsense has an untangle setup with rulesets that allow me to "switch" rather seamlessly. There are no issues whatsoever when untangle is running.

An Idea here, maybe its stupid maybe not but...

What if this is included into the Official OPNsense docs?

Currently the docs do not have any Guide how to deploy OPNsense into Proxmox. Its easy to spin off OPNsense in Proxmox but "best practices" are another thing.

Would it be beneficial for the people to have something like that in the Official docs?

Regards,
S.

This is well above the know how of most people. Doubt many people run a datacenter-level opnsense with the VMs on the same server at home to this degree.

Good dive though, much appreciated 👌 Now I have to rebuild everything... again 😒

Can I ask why your routing rule is different than the one in the OPNsense docs? They have a floating rule, direction out.

This guide is using the "Step 8 - Create a Firewall rule" rule: https://docs.opnsense.org/manual/how-tos/wireguard-selective-routing.html#step-8-create-a-firewall-rule

I don't have my opnsense active at the moment. You'll be able to confirm whether it's working or not with the last step. You can easily add the rules from the wiki and see if it behaves differently after you've confirmed that selected routing is working.

Didn't manage to find any problems with this despite not using all the rules. All traffic I tested hopped the correct routes and was blocked where I wanted. Possibly something I missed that the other rules fixed. Going to need someone smarter than me to confirm.

If I understood properly, there is no need of rules within the actual Wireguard/VPN interface, but only in the interface where the hosts live, is that correct?

Tia.

The NAT and Floating rule should cover that.

[1. Install WireGuard]

Additional information and inspiration:
schnerring.net
OPNsense Docs


0. About

Consolidated the available information for us who get cross eyed by walls of text. Don't expect to learn why it's working here.

The bare minimum needed to get it up and running on a "clean" OPNsense.

Hopefully, it can be of help to someone and lets hope I never have to do BBcode formatting ever again 🤦


1. Install WireGuard
NOTE: Not required on newer versions

Code Select Expand

Navigate to: System > Firmware > Plugin- Install WireGuard


2. Download Mullvad config - I'll call it .conf

1. Login mullvad.net & go to wireguard-config
2. Generate Key
3. Scroll down and select server
4. Select IPv4
5. Select Only IPv4
6. Configure Content Blocking
  - Irrelevant here, changes the DNS server provided in .conf
7. Download .conf

Additional Mullvad info

These can be used as monitoring IP for gateway(s):
- Mullvad - How to set up ad-blocking in our app
  - 100.64.0.1 for Ad-blocking
  - 100.64.0.2 for Tracker-blocking
  - 100.64.0.3 for Ad- + Tracker-blocking.

- Mullvad - Adding another layer: malware DNS blocking
  - 100.64.0.4 Malware blocking only
  - 100.64.0.5 Ad and malware blocking, no tracker blocking
  - 100.64.0.6 Tracker and malware blocking, no ad blocking
  - 100.64.0.7 Ad, tracker and malware blocking ("everything")


3. WireGuard Configuration

3.1 WireGuard INSTANCE - [interface] in .conf

Code Select Expand

Navigate to: VPN > WireGuard > Settings > Instances
Fields not mentioned = BLANK / Default

- ADD

Code Select Expand

| Field           | Value                          |
| --------------- | ------------------------------ |
| Name            | Instance Name                  |
| Pub Key         | From Mullvad .conf creation*   |
| Priv Key        | Value of "PrivateKey" in .conf |
| Port            | 51820                          | * leave empty for random or specify manually
| Tunnel Address  | Value of "Address" in .conf    |
| Disable Routes  | CHECKED                        |
| Gateway         | Tunnel_Address (-1)**          |


* You cannot view this attachment.
edit: no need, see comment  https://forum.opnsense.org/index.php?msg=229060

** See note: OPnsense Docs - wireguard-selective-routing

- SAVE but don't apply

3.2 WireGuard PEER - [peer] in .conf

Code Select Expand

Navigate to: VPN > WireGuard > Settings > Peers
- ADD

Code Select Expand

| Field               | Value                        |
| ------------------- | ---------------------------- |
| Name                | Peer Name                    |
| Pub Key             | Value of "PublicKey" in .conf|
| Allowed IPs         | 0.0.0.0/0                    |
| Endpoint Address    | Value of "Endpoint" in .conf |
| Endpoint Port       | 51820                        |
| Instance            | The one you set up earlier   |
| Keepalive internal  | 25                           |

- SAVE
- APPLY

Code Select Expand

Navigate to: VPN > WireGuard > Settings > General- Enable WireGuard
- Verify tunnel is UP in VPN > WireGuard > Diagnostics


4.  Add an interface

Code Select Expand

Navigate to: Interfaces > Assignments > Assign
 a new interface
- Expand list and select the WireGuard interface
- Device wg1
  - ADD
  - SAVE (above)

- Click on the new interface (above)
  - Enable Interface: CHECKED
  - SAVE


5. Add a gateway

Code Select Expand

Navigate to: System > Gateways > Configuration
- ADD

Code Select Expand

| Field                          | Value                                            |
| ------------------------------- | -------------------------------------------------|
| Name                            | GW name                                          |
| Interface                      | wg1                                              |
| Address Family                  | IPv4                                            |
| IP Address                      | Value of "Address"(-1) in .conf*            |
| Far Gateway                    | CHECKED                                          |
| Disable Gateway Monitoring      | UNCHECKED                                        |
| Monitor IP                      | 10.64.0.1 (An internal Mullvad DNS)              |

* If .conf address is xx.xx.xx.10/32 you can use xx.xx.xx.9 - i.e. remove the subnet mask and subtract one from the last segment.

-  SAVE
-  APPLY


6. Firewall configuration
This configuration is as barebones as they come, modify it to your liking

Code Select Expand

Navigate to: Firewall > Aliases
- ADD

Code Select Expand

| Field            | Value                                          |
| ----------------- | ---------------------------------------------- |
| Name              | [selected hosts] - any name you want          |
| Type              | Host(s)                                        |
| Content          | Add the IP of each device to Selectively Route |

- SAVE
- APPLY

6.1 FIRST rule: Route [selected hosts] traffic through the tunnel

Code Select Expand

Navigate to: Firewall > Rules > Floating
- ADD

Code Select Expand

| Field                | Value                        |
| -------------------- | ------------------------------|
| Action              | Pass                          |
| Quick                | CHECKED                      |
| Interface            | Interface(s) where your [selected hosts] live
| Direction            | In                            |
| TCP/IP Version      | IPv4                          |
| Protocol            | Any                          |
| Source              | [selected hosts]              |
| Destination          | Any                          |
| Gateway              | WG Gateway                    |
|              Show Advanced Features                  |
| SET local tag        | NO_WAN_EGRESS                | * for kill switch

- SAVE

6.2 SECOND rule: Kill switch
May not be needed depending on your configuration, better safe than sorry?

- OPNsense Docs: Kill Switch


6.3 NAT Rule: NAT WireGuard for [selected hosts]

Code Select Expand

Navigate to: Firewall > NAT > Outbound
- Change mode to Hybrid outbound NAT rule generation

- ADD

Code Select Expand

| Field                  | Value                                          |
| ------------------------| ---------------------------------------------- |
| Interface              | WG interface                                  |
| TCP/IP Version          | IPv4                                          |
| Protocol                | Any                                            |
| Source                  | [selected hosts]                              |
| Src Port                | Any                                            |
| Destination            | Any                                            |
| Dst Port                | Any                                            |
| Translation / Target    | Interface Address                              |

- SAVE
- APPLY to save all the firewall rules


7. Verify it's working as intended

- Add a device IP to the [selected hosts] Alias
- Use Mullvad Check
  - All three should be green

- API, Powershell

Code Select Expand

(curl https://am.i.mullvad.net/json).Content | ConvertFrom-Json

[glow=cyan,2,300]Thanks for reading![/glow]
Please educate me where there are misstakes!

Hi,

We need more info for the higher Mem usage issue. Please share a report by following the instructions in the below link.


https://www.zenarmor.com/docs/support/reporting-bug

My fresh install of opnsense + sensei also used 85% (8GB) of available RAM but it settled after a while and is now using 57%. Using elastisearch locally.

Back to 75-83%. :-)

I can't SSH to my servers on a different VLAN than my Desktop.
I can SSH between my servers on the same interface. (LAN/untagged)

Any fix for this, or do I have have to remove myself from sensei to manage my servers?

I can move them onto a vlan if that's a solution.

How are you people going about excluding OPNsense traffic?
I have a bunch of vlans, all going to a Pihole (for now) then back to unbound and out from there.
IIRC i lose the lookup if u exclude all but one interface in Unbound aswell.

Sensei doesnt seem to take 127.0.0.1 / self as an exclusion.

Looking for ideas on how to set this up the best way :)

I'm getting 500mpbs without sensei and 0,5mbps with sensei.

I have a dogshit CPU, E3950 @ 1.60GHz (4 cores).

However the CPU is barely breaking a sweat and memory utilization is ~20% (8gb).

is this what I can expect performance-wise out of this hardware?
I was thinking abut upgrading but I'm doubting the Hades Canyon or similar can pull it if this isn't working out right now?

Or could there be some configuration error at play here?

Edit: Hardware offloading was the problem. now it's around 300 :)

Keep gettin this error even after a clean reinstall.

[29-Jan-2020 07:02:25 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'mongodb.so' (tried: /usr/local/lib/php/20170718/mongodb.so (Shared object "libcrypto.so.9" not found, required by "mongodb.so"), /usr/local/lib/php/20170718/mongodb.so.so (Cannot open "/usr/local/lib/php/20170718/mongodb.so.so")) in Unknown on line 0

In /user/local/lib there is a libcrypto.so.11 ..

Isn't it supposed to run internally and not on the WAN port?

I never get hits when it's activated on WAN, but from traffic on my LAN...