Menu

Show posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Show posts Menu

Messages - ProximusAl

#1
I have an OPNsense DEC750v2 which is a truly headless appliance. There are no HDMI, DisplayPort or VGA outputs - local console access is via the Mini-USB serial console port.

Normally this isn't a problem, but it becomes more important when something goes wrong during an upgrade. If OPNsense won't boot properly and both the WebGUI and SSH are unavailable, the serial console may be the only way to see what is happening and interact with the boot loader.

This is particularly relevant now that OPNsense supports ZFS snapshots/boot environments. I can create a snapshot before an upgrade, but if the upgrade goes badly enough that OPNsense is inaccessible, I still need some way of reaching the console to select and boot the previous environment.

I wanted a way of accessing that serial console without having to physically connect a laptop to the DEC750v2 every time.

I already had a QNAP NAS sitting next to the firewall, so I used Container Station/Docker to turn it into a simple network-accessible serial console server.

The setup is:

DEC750v2 Mini-USB console
        |
        v
QNAP /dev/ttyACM0
        |
        v
Docker container (Alpine)
        |
        +--- ser2net TCP 7001 ---> PuTTY (Raw TCP)
        |
        +--- ttyd TCP 7681 ------> Web browser


OPNsense configuration

In OPNsense I went to:

System -> Settings -> Administration -> Console

and configured:

Primary Console: Serial Console
Secondary Console: EFI Console
Serial Speed: 115200
USB-based serial: Disabled
Console menu: Password protect enabled

Once saved, the normal OPNsense console menu became available over the DEC750v2 serial connection.


QNAP configuration

I connected the DEC750v2 Mini-USB console port to a USB port on the QNAP.

On the QNAP this appeared as:

/dev/ttyACM0

I created the following directory:

/share/Container/opnsense-console

In that directory I created a file called:

Dockerfile

Containing:

FROM alpine:3.24

RUN apk add --no-cache ser2net ttyd netcat-openbsd

COPY ser2net.yaml /etc/ser2net.yaml

CMD sh -c 'ser2net -n -c /etc/ser2net.yaml & exec ttyd -W -p 7681 nc 127.0.0.1 7001'


I then created:

ser2net.yaml

Containing:

connection: &opnsense
  accepter: tcp,7001
  enable: on
  connector: serialdev,/dev/ttyACM0,115200n81,local


I built the Docker image with:

cd /share/Container/opnsense-console

sudo docker build --pull --no-cache -t opnsense-console:latest .


Then created the container:

sudo docker run -d \
  --name opnsense-console \
  --restart unless-stopped \
  --device=/dev/ttyACM0:/dev/ttyACM0 \
  -p 7001:7001 \
  -p 7681:7681 \
  opnsense-console:latest

The "--restart unless-stopped" option means the console container automatically comes back after the QNAP reboots.


PuTTY access

I can connect using PuTTY with:

Host: QNAP-IP
Port: 7001
Connection type: Raw

This gives me the normal interactive OPNsense serial console through PuTTY.


Web console

I also added ttyd to provide a proper browser-based terminal.

This is accessed using:

http://QNAP-IP:7681

I always use IP because OPNSense is my DNS server too...

This works particularly well from Safari on an iPad/iPhone and gives me a proper interactive terminal with the normal on-screen keyboard.

Both methods ultimately connect to the same serial console:

PuTTY -> TCP 7001 -> ser2net -> /dev/ttyACM0 -> DEC750v2

or:

Browser -> ttyd -> netcat -> ser2net -> /dev/ttyACM0 -> DEC750v2


Why I wanted this

My main reason for doing this is safer OPNsense upgrades.

Before upgrading I can create a ZFS snapshot/boot environment.

If an upgrade then breaks OPNsense badly enough that the WebGUI and SSH aren't available, I still have access to the physical serial console.

The particularly useful part is that the serial connection is available throughout a reboot. This means I can see the DEC/FreeBSD boot process and interrupt the boot loader if necessary.

I can therefore reboot the DEC, interrupt the FreeBSD loader and select the previous ZFS boot environment without having to physically connect another machine to the firewall.

Obviously this isn't true out-of-band management because the QNAP is still accessed over my normal LAN. However, my network is a flat LAN and devices retain their IP addresses while the firewall is rebooting or unavailable.

This means I can still access another machine on the LAN by IP and from there access the QNAP console server even if OPNsense itself isn't working.


Security

There are a couple of important security considerations.

TCP 7001 is raw, unencrypted and unauthenticated.

The basic ttyd configuration on TCP 7681 is also HTTP and unauthenticated.

I only expose these on my trusted internal LAN.

I would absolutely not port-forward either of these ports or expose them directly to the Internet.

I also have password protection enabled on the OPNsense console menu itself.


Other hardware

Although I happened to use a QNAP NAS, there's nothing particularly QNAP-specific about the actual solution.

Any always-on machine capable of running Docker and accessing the USB serial device should be able to do essentially the same thing.

For me it's effectively turned an existing NAS sitting next to the firewall into a small dedicated OPNsense console server, with both PuTTY and browser access.

I hope this might help or inspire someone else...
#2
I upgraded from OPNsense 26.1 to 26.7.1 (in-place upgrade, followed by updating the bootloaders).

After the upgrade, outbound traffic continued to work normally, but all unsolicited inbound traffic stopped working.

After troubleshooting, I traced the issue to two WAN firewall rules (one IPv4 and one IPv6) that used a custom Network(s) alias as the destination.

The alias was of type Network(s) and contained an IP address range in the format:

a.b.c.d - w.x.y.z

These rules had worked correctly prior to the upgrade.

Symptoms:

* Inbound packets reached the WAN interface.
* The packets never reached the LAN interface.
* pflog0 showed the packets hitting the default block rule instead of the WAN pass rule.
* Packet captures confirmed the traffic reached the WAN interface but was never forwarded.

The interface configuration was correct, and the built-in interface network object contained the expected IPv4 subnet and IPv6 prefix.

The fix was simply to change the firewall rule destination from the custom alias to the built-in  network object. As soon as I did this, both IPv4 and IPv6 immediately started working again.

Has there been a change in 26.7 regarding Network(s) aliases containing IP address ranges (a.b.c.d - w.x.y.z) when used as firewall rule destinations? This worked correctly before the upgrade, but afterwards the alias no longer matched traffic, whereas the built-in interface network object worked immediately.

Is this an intentional behavioural change or a regression?

I should add, this router is not using NAT and purely routes public IP addresses
#3
Quote from: franco on July 02, 2026, 10:43:29 AMIf you have manual rules to migrate then you will set hybrid or manual, but that is already set for your use case.

If you don't have manual rules nothing needs to be migrated.


Cheers,
Franco

Sorry Franco.....

I *already* migrated my rules from Outbound NAT to SNAT (At which point Outbound NAT was set to Hybrid)

Once I migrated them to SNAT, I changed outbound NAT back to Automatic.

Is this what I should be doing, or should I change Outbound NAT to something else before upgrading?

EDIT: Figured it out. Set it back to Hybrid after reading the release notes properly :)
#4
Quote from: keeka on July 02, 2026, 07:46:32 AMAm I right in thinking, after upgrading to 26.1.11, we should switch back to hybrid mode under Outbound NAT?
I had turned this off following migration of my rules to SNAT as I had noticed SNAT rules got generated regardless of the selected mode.

EDIT: I just upgraded and see the mode option also appears on the SNAT page, and is in sync with selection in legacy Outbound page. So question answered.

So what did you leave it on?  Mine is on Automatic after I migrated to SNAT.
Should I change this to Hybrid or manual or leave it as is?
#5
I use this
https://certifytheweb.com/

with powershell/ssh scripts etc to deploy to OPNSense, NAS's, nginx etc...all sorts.
#6
Why dont you sanitise your config.xml of passwords/secrets etc, and upload it to chatgpt and describe your issue.

I'm not an advocate of AI, but it has actually resolved an issue for me in the past with asymmetric routing, and inevitably, it was caused by me....and identified by AI.
#7
26.1, 26,4 Series / Re: 26.1.6 - Health Check
April 09, 2026, 11:56:19 PM
Quote from: nero355 on April 09, 2026, 11:08:08 PM
Quote from: ProximusAl on April 09, 2026, 05:16:08 PMI have zero idea what the heck happened.
How old was the OPNsense installation ?

Sounds like ZFS issues that sometimes happen when the FreeBSD Bootloader is too old...

4 months...Device was new in December 25 and fresh installed with 25.7 then.
#8
26.1, 26,4 Series / Re: 26.1.6 - Health Check
April 09, 2026, 05:16:08 PM
So that's 90 minutes I'm not getting back.

All I did was upgrade to 26.1.6 from .5 on a DEC750v2.
It rebooted and everything was working, but out of habit, after an upgrade I always run a health check.
It stalled on the 4 dots for ages, so I rebooted. Never came back.

I quickly put in a replacement 750v2, imported my config and we're back.

So I then put the original one on console, and it's weird.  I have a screenshot I'll try and link later but I saw something about "no pool to import". Leaving it for about 25 minutes, it does sort of begin to boot, but slow.
It hangs at "Initializing.........done." forever, then does a bit more 20 mins later.

Rather than waste time, I reinstalled 26.1.2 serial iso, upgraded to 26.1.6 and it's fine.

I have zero idea what the heck happened.
#9
26.1, 26,4 Series / Re: 26.1.6 - Health Check
April 09, 2026, 04:22:28 PM
I may have bigger issues.

Whilst it was hung, I decided to reboot. Never came up again.

I've just swapped out the DEC750v2 with another and imported config and we're back.

Time to put a console port on this one..
#10
26.1, 26,4 Series / 26.1.6 - Health Check
April 09, 2026, 03:25:05 PM
Just upgraded to 26.1.6, but health check sticks.....

***GOT REQUEST TO AUDIT HEALTH***
Currently running OPNsense 26.1.6 (amd64) at Thu Apr  9 14:21:30 BST 2026
>>> Root file system: zroot/ROOT/default
>>> Check installed kernel version
Version 26.1.6 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 26.1.6 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check installed repositories
OPNsense (Priority: 11)
>>> Check installed plugins
os-chrony 1.5_3
os-ddclient 1.30_2
os-theme-vicuna 1.51
>>> Check locked packages
No locks found.
>>> Check for missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: ....

Can anyone else confirm?
#11
The OPNSense docs state:

For legacy compatibility WAN interfaces set to type DHCP or interfaces with a Gateway Rules selection send reply packets to the corresponding gateway directly, also when the sender is on the same interface. This will break connectivity in some rare scenarios and can be disabled via Firewall->Settings->Advanced->Disable reply-to.

With Multi-WAN you generally want to ensure traffic leaves the same interface it arrives on, hence reply-to is added automatically by default. When using bridging, you must disable this behavior if the WAN gateway IP is different from the gateway IP of the hosts behind the bridged interface.

In my case, I have "Disable reply-to on WAN interface" selected, and my firewall rules have the reply-to explicitly set.
My secondary WAN is DHCP, and my primary is PPPoE, so this felt safest.

That works fine.

EDIT: I should add, I have migrated to the NEW rules....
#12
I can tell you it works fine on 26.1.5 so you must have something misconfigured.

You havent really given us enough information.

Have you checked "Disable Reply-To on WAN rules" on Firewall/Settings/Advanced?
Have you set the "Reply-To" on the actual firewall rule? (advanced mode)
#13
I am using PPPoE and get a /48 prefix from my ISP.

On 26.1.5, I have no issues and it works exactly as it should.

Identity association on the LAN and I get a /64 with an Assign prefix ID of 0

I do "Request Prefix Only" on the WAN interface, and nothing else. (Apart from Prefix delegation size of 48)

Doesn't help you, but I just wanted to say it works perfectly for me.....
#14
Now that makes a lot more sense :)

In which case, I change my answer to I never get a /128 :)

Perhaps a poorly setup at the ZEN ISP end then....

For now, because I am a *clean log* man, I just request a PD only.  It works for me.
#15
I do get a non temporary address yes, but it seems to come via SLAAC. (If I untick request prefix only)

So requesting a prefix only gives fe80::1%pppoe0/64 on the WAN, unticking Request Prefix only gives an address in my ND, e.g. 2a02:X0XX:XX01:XX6a::1/64, but then I get the entries in the log no address/prefix.

I think dhcp6c is trying to get the WAN IPv6 via DHCPv6 and thats why it throws no addresses.

The PD comes via DHCPv6, and that always works.

What is very interesting though, is even though I request prefix only, under Interfaces/Overview, I do still see the above ND address, but not on the dashboard interfaces widget, just fe80::1%pppoe0/64

Odd....

It's not a major issue for me as *both* options work, I was just interested in why the log was displaying those messages.

Keep up the good work Franco :)