31
This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.
32
High availability / Managed Switch between WAN (ONT) & 2 x OPNsense -- seems to be working
« on: May 11, 2022, 04:17:10 am »
My internet provider (Verizon FiOS) is only giving me a single WAN IP, so I was looking for a way to implement full HA within that limitation. From all my reading, I understood that the traditional CARP tutorials (https://docs.opnsense.org/manual/how-tos/carp.html and https://www.thomas-krenn.com/en/wiki/OPNsense_HA_Cluster_configuration) require one to have three static IP addresses on the WAN -- one for each OPNsense box and another one for the VIP that ties both of them together.
I then came across this article (https://www.thegeekpub.com/5688/ethernet-switch-between-the-ont-and-the-router-fios/) that describes a scenario (option 2) where a managed switch can be put between the ONT and the two OPNsense boxes, and an untagged VLAN needs to be created across three physical Ethernet ports on that switch -- one for each OPNsense box and another for the cable that goes into the Ethernet port of the ONT.
On the surface, I didn't understand how that should work, but somehow it does, provided that both WAN ports of the OPNsense boxes share the same MAC address.
Somehow can perhaps educate me what the managed switch does that:
1) Enables it to handle two connected devices that have the same MAC address
2) Pretends towards the ONT that only a single device is attached
Both OPNsense boxes show an active WAN connection with the same externally assigned dynamic IP address. I've run several tests to confirm that connectivity is positively there, both through Master and Backup. I also shut down either one and the other keeps on working as it should in a proper HA configuration.
The switch I bought is the D-Link DGS-1100-08V2. I configured it to have a dedicated management port so that it can only be administered from a directly attached Ethernet cable.
Note: I do have CARP/pfSync/XMLRPC properly set up on the LAN, but not at all on the WAN side. The latter appears to be fully taken care off by the managed switch.
I then came across this article (https://www.thegeekpub.com/5688/ethernet-switch-between-the-ont-and-the-router-fios/) that describes a scenario (option 2) where a managed switch can be put between the ONT and the two OPNsense boxes, and an untagged VLAN needs to be created across three physical Ethernet ports on that switch -- one for each OPNsense box and another for the cable that goes into the Ethernet port of the ONT.
On the surface, I didn't understand how that should work, but somehow it does, provided that both WAN ports of the OPNsense boxes share the same MAC address.
Somehow can perhaps educate me what the managed switch does that:
1) Enables it to handle two connected devices that have the same MAC address
2) Pretends towards the ONT that only a single device is attached
Both OPNsense boxes show an active WAN connection with the same externally assigned dynamic IP address. I've run several tests to confirm that connectivity is positively there, both through Master and Backup. I also shut down either one and the other keeps on working as it should in a proper HA configuration.
The switch I bought is the D-Link DGS-1100-08V2. I configured it to have a dedicated management port so that it can only be administered from a directly attached Ethernet cable.
Note: I do have CARP/pfSync/XMLRPC properly set up on the LAN, but not at all on the WAN side. The latter appears to be fully taken care off by the managed switch.
33
High availability / PSA: If Master can't reach Backup, double check "Listen Interfaces" on Backup
« on: May 10, 2022, 01:38:18 am »
Took me a while to figure out why my Master OPNsense box couldn't reach the Backup box via XMLRPC. I made the following mistakes:
Hope this helps someone in the future.
- The WebUI of my OPNsense boxes are on a non-standard port (because I am using the standard 80 and 443 ports for Nginx). Under System > High Availability > Settings, I had to put the full URL into the "Synchronize Config to IP" field on the primary (i.e. https://<Backup pfSync IP Address>:4444)
- On the backup box, under System > Settings > Administration, I had to add the pfSync Interface into the "Listen Interfaces" selection (it was previously only set to "LAN").
Hope this helps someone in the future.
34
22.1 Legacy Series / Re: NUC-C3L4 (aka. FW4B) -- stuck on "Loading /boot/loader.conf"
« on: May 06, 2022, 08:23:36 pm »
So really the only way for me to restore the configuration between two different device models was to *not* use the graphical config importer at installation time, but to restore the config.xml from the OPNsense Web UI after installation. I found no other way to do the config import until after a fresh installation.
Also, if migrating between different models, it seems important to edit the config.xml file to adjust the interface names beforehand.
EDIT: All of the above is wrong. I thought it wasn't booting correctly but the real issue was that that configuration was set to only have serial output, no VGA output. I thought it was stuck booting, but it wasn't -- it booted fine, I just didn't see it on the attached screen. Duh.
Also, if migrating between different models, it seems important to edit the config.xml file to adjust the interface names beforehand.
EDIT: All of the above is wrong. I thought it wasn't booting correctly but the real issue was that that configuration was set to only have serial output, no VGA output. I thought it was stuck booting, but it wasn't -- it booted fine, I just didn't see it on the attached screen. Duh.
35
22.1 Legacy Series / Re: NUC-C3L4 (aka. FW4B) -- stuck on "Loading /boot/loader.conf"
« on: May 06, 2022, 06:00:09 pm »
I tested a lot more permutations of different settings during installation, but in the end concluded that the issue is the import of the old settings. If I don't do a config import, the device boots up fine. Otherwise it gets stuck on the boot screen as per above screenshot.
The device I am importing the config from had onboard SSD storage, so I couldn't take it out. Instead, I did an rsync of the entire root "/" to a FAT32 USB stick. So that stick has more than just the "/conf" folder on it. And that's what I did the config import from, which always concluded with "success". But apparently there's something on there that throws off the installation enough for it to not boot.
The device I am importing the config from had onboard SSD storage, so I couldn't take it out. Instead, I did an rsync of the entire root "/" to a FAT32 USB stick. So that stick has more than just the "/conf" folder on it. And that's what I did the config import from, which always concluded with "success". But apparently there's something on there that throws off the installation enough for it to not boot.
36
22.1 Legacy Series / Re: NUC-C3L4 (aka. FW4B) -- stuck on "Loading /boot/loader.conf"
« on: May 06, 2022, 04:13:13 pm »
Point of clarification: I followed the instructions here to change to UEFI only mode in AMI Bios (see here: https://protectli.com/kb/how-to-install-opnsense-on-the-vault/).
I can boot from a USB stick fine (i.e. OPNsense installer or Ubuntu), for what it's worth.
I can boot from a USB stick fine (i.e. OPNsense installer or Ubuntu), for what it's worth.
37
22.1 Legacy Series / [Solved] Config migration btw diff devices:Stuck on "Loading /boot/loader.conf"
« on: May 06, 2022, 06:24:37 am »
I received a brand new NUC-C3L4 (a clone of the Protectli FW4B).
Transferred my 32GB mSATA SSD from my old Protectli FW4A to it and it booted into 22.1.6 fine.
Then decided to start with a clean slate, installed OPNsense from USB stick, imported old config during the graphical portion of the installer, and at the end told the device to reboot. Now it's stuck at the below screen.
I repeated the install multiple times, each time with the same result.
I find it puzzling that the device was able to boot from the old installation before I decided to reinstall from scratch.
Thoughts?
P.S.: UFS with the guided installation method
Transferred my 32GB mSATA SSD from my old Protectli FW4A to it and it booted into 22.1.6 fine.
Then decided to start with a clean slate, installed OPNsense from USB stick, imported old config during the graphical portion of the installer, and at the end told the device to reboot. Now it's stuck at the below screen.
I repeated the install multiple times, each time with the same result.
I find it puzzling that the device was able to boot from the old installation before I decided to reinstall from scratch.
Thoughts?
P.S.: UFS with the guided installation method
38
22.1 Legacy Series / Re: HA failover interrupts streams
« on: May 03, 2022, 01:32:08 pm »
I am not using CARP on my end but am fully expecting that all active connections (inbound & outbound) would be impacted in one way or another during a failover. But it really is more up to the client how well it recovers from such temporary disruptions. For example, Zoom connections recover pretty well, even if your connection is going down for a few seconds (the client reconnects automatically, but you still experience a temporary disruption -- it's just that it gets resolved by itself).
39
22.1 Legacy Series / ACME client not starting (22.1.6 w/ /var on tmpfs)
« on: May 02, 2022, 05:37:47 pm »
I noticed that the ACME client doesn't start when I change /var to be on tmpfs. The only way to get it going again is to disable, then enable the ACME client in its settings. However, on the next OPNsense reboot, it's red again and I cannot start it until I go through the same sequence described above. This doesn't seem to happen when /var is not in RAM.
I see the following in the system log:
The file it's complaining about not being able to read has the following permissions:
When I do get it going again, even though it's green, it still shows the following in the system log:
There is, however, an active cron job scheduled for ACME in the OPNsense web UI cron settings.
I know there's a broader discussion underway about which folders under /var should be allowed to be on tmpfs and which ones shouldn't. I assume this is part of that conversation (i.e. whether /var/etc should be on tmpfs or not).
I see the following in the system log:
Code: [Select]
<13>1 2022-05-02T03:01:43-04:00 OPNsense.lan root 65690 - [meta sequenceId="26"] /usr/local/etc/rc.d/acme_http_challenge: WARNING: /var/etc/lighttpd-acme-challenge.conf is not readable.
<13>1 2022-05-02T03:01:43-04:00 OPNsense.lan root 66935 - [meta sequenceId="27"] /usr/local/etc/rc.d/acme_http_challenge: WARNING: failed precmd routine for acme_http_challenge
The file it's complaining about not being able to read has the following permissions:
Code: [Select]
# ll /var/etc/lighttpd-acme-challenge.conf
-rw-r--r-- 1 root wheel 2522 May 2 11:34 /var/etc/lighttpd-acme-challenge.conf
When I do get it going again, even though it's green, it still shows the following in the system log:
Code: [Select]
<147>1 2022-05-02T11:34:53-04:00 OPNsense.lan config 12736 - [meta sequenceId="7"] [2022-05-02T11:34:53-04:00][error] [OPNsense\AcmeClient\AcmeClient:settings.UpdateCron] Related cron not found.{23b7041f-6697-456f-9a42-4bfff087f806}
There is, however, an active cron job scheduled for ACME in the OPNsense web UI cron settings.
I know there's a broader discussion underway about which folders under /var should be allowed to be on tmpfs and which ones shouldn't. I assume this is part of that conversation (i.e. whether /var/etc should be on tmpfs or not).
41
22.1 Legacy Series / Re: Did 22.x change how DNS resolution works?
« on: May 01, 2022, 10:36:13 pm »
I'm not an OPNsense developer. Just a guy trying to help.
Good hunting.
P.S.: Lots of Unbound related changes in the release notes: https://docs.opnsense.org/releases/CE_22.1.html
Good hunting.
P.S.: Lots of Unbound related changes in the release notes: https://docs.opnsense.org/releases/CE_22.1.html
42
22.1 Legacy Series / Re: Did 22.x change how DNS resolution works?
« on: May 01, 2022, 09:44:14 pm »
I would not use ".local" overrides at all. I would switch all those overrides to a different name, i.e. ".lan".
I have never had reliable luck using ".local" overrides and switched to a different name, as local is sort of a reserved domain.
Edit: The .local domain is is a special-use domain primarily used for zeroconf purposes, more info here: https://en.wikipedia.org/wiki/.local
I have never had reliable luck using ".local" overrides and switched to a different name, as local is sort of a reserved domain.
Edit: The .local domain is is a special-use domain primarily used for zeroconf purposes, more info here: https://en.wikipedia.org/wiki/.local
43
22.1 Legacy Series / Re: Did 22.x change how DNS resolution works?
« on: May 01, 2022, 07:49:25 pm »
Are you overriding ".local" in your Unbound config?
In my experience, ".local" should be left alone and if you want to introduce an override for your LAN, use something else, i.e. ".lan" instead (which is what I am doing).
In my experience, ".local" should be left alone and if you want to introduce an override for your LAN, use something else, i.e. ".lan" instead (which is what I am doing).
44
22.1 Legacy Series / Re: OPNsense loosing WAN IP several times daily
« on: April 30, 2022, 03:55:07 pm »Same config but different hardware so interactions between hardware, drivers and config might be at play.
I would disable IPS to begin with and see if that makes it more stable, especially with re interfaces.
Disabling IPS completely indeed mitigated my WAN disconnect issue completely. Not really a good long term solution, as I'd like to turn IPS on again, but you were right, it's clearly related. Where to go from here?
45
Tutorials and FAQs / Re: TUTORIAL Nginx + Let's Encrypt for Plex / Emby / Jellyfin 100% A+ Rating
« on: April 30, 2022, 06:19:15 am »
Came here to say that this is a really nice write-up! I adapted it to my needs for my Plex instance, but you got me 99% there. This really is the only tutorial I found that talks about Plex/Nginx/OPNsense. There are a few other tutorials about just general Nginx & Plex, but it's always difficult to adapt raw Nginx config files to how it needs to be configured on the OPNsense Nginx GUI. So... thank you!
I now also disabled remote access from the Plex server settings and removed my OPNsense firewall rule to close that hole as well. And... drum roll... yes, I also got an A+ rating on the Qualys site. However, Mozilla Observatory reported a less than stellar score, so I had to tweak the security headers a bit more.
Anyway, thank you!
I now also disabled remote access from the Plex server settings and removed my OPNsense firewall rule to close that hole as well. And... drum roll... yes, I also got an A+ rating on the Qualys site. However, Mozilla Observatory reported a less than stellar score, so I had to tweak the security headers a bit more.
Anyway, thank you!