Messages

I don't think it hangs, but serial is definitely not active after init takes over booting.

A thread for you: https://forum.opnsense.org/index.php?topic=27432.0



Cheers,
Franco

Thank you for the link.

Disabling legacy UART as per the following solution from this thread worked for me: https://forum.opnsense.org/index.php?msg=134528

I'm having this issue with my Deciso DEC740 trying to re-install fresh with OPNsense 26.1.

The following is the procedure I'm performing:

1. Download OPNsense-26.1.6-serial-amd64.img.bz2 (sha256 = 60E698B7F935B647B72A424C78ACAA1772A050BB4BD593CA001EA9A1634D5643)
2. Unpack the bz2 into an IMG file. (sha256 = 5971BDDBC1CA2A1CE09DF5A5675FA782B34963E8C7A2ADC5FE11B8523A0F1C60)
3. Use BalenaEtcher to write IMG file to USB stick.
4. Put USB stick in firewall.
5. Connect serial console to firewall (115200 baud) with PuTTY.
6. Power cycle firewall and hit ESC to enter boot menu.
7. Select to boot from USB stick.

The boot process gets this far, and then hangs:

Code Select Expand

uart0: <8250 or 16450 or compatible> port 0x3f8-0x3ff irq 3 flags 0x10 on acpi0
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
cpufreq0: <CPU frequency control> on cpu0
Timecounter "TSC-low" frequency 1097937717 Hz quality 1000
Timecounters tick every 1.000 msec
ugen0.1: <AMD XHCI root HUB> at usbus0
ugen1.1: <AMD XHCI root HUB> at usbus1
uhub0 on usbus0
uhub1 on usbus1
uhub1: <AMD XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1
uhub0: <AMD XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
nvme0: Allocated 64MB host memory buffer
Trying to mount root from ufs:/dev/ufs/OPNsense_Install [ro,noatime]...
uhub1: 3 ports with 3 removable, self powered
uhub0: 8 ports with 8 removable, self powered
ugen0.2: <USB SanDisk 3.2Gen1> at usbus0
umass0 on uhub0
umass0: <USB SanDisk 3.2Gen1, class 0/0, rev 3.00/1.00, addr 1> on usbus0
umass0:  SCSI over Bulk-Only; quirks = 0x4001
umass0:1:0: Attached to scbus1
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
nda0 at nvme0 bus 0 scbus0 target 0 lun 1
nda0: <TS128GMTE110S T1102A0L G959916399>
nda0: Serial Number G959916399
nda0: nvme version 1.3
nda0: 122104MB (250069680 512 byte sectors)
da0 at umass-sim0 bus 0 scbus1 target 0 lun 0
da0: <USB SanDisk 3.2Gen1 1.00> Removable Direct Access SPC-4 SCSI device
da0: Serial Number 0501281716877c101ddabe03038786894b7289b153ffa8ff759e95a0ec3d
da0: 400.000MB/s transfers
da0: 14664MB (30031872 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>
GEOM: da0: the secondary GPT header is not in the last LBA.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Troubleshooting steps I've tried to far:

- Try writing image with BalenaEtcher and Rufus (neither seem to work)
- Try using 2 different USB sticks
- Try changing console option from Serial to VGA on boot screen
- Connect Ethernet to see if there's any signs of life using wireshark - nothing
- Just wait

I've had this same issue with two seperate DEC740 firewalls.

What else could I try?

Thank you, that makes sense. I thought that this was the purpose of the pre-shared key, but now that you mention it, of course a symmetric key cannot be used for this purpose because all clients have access to this PSK and they could then impersonate the server.

But then what about the "Peer Certificate Authority"? Since we're not using client keys, the peers shouldn't be presenting any client certificates at all, or am I misunderstanding something?

Hello!

I have a scenario I'm working with where it is not feasible to do a PKI infrastructure for the purpose of VPN client authentication. Instead, we want to use Username/Password authentication through RADIUS.

In this scenario, I am not expecting a certificate authority to be neccessary to configure, since the authentication happens through:

1. A pre-shared key for TLS auth (to protect the initial exchange and provide some protection from casual password bruteforce attacks, because you cannot even try a password unless you have this PSK)
2. Username and passsword

In this scenario, I don't see the purpose of a certificate authority, yet it is forced for me to configure one for this scenario. Am I misunderstanding something about how this is supposed to work? Also, a "server certificate" is required, I'm not sure where this is used?

More generally, my concern is, I want to be aware of any "time bombs" in the system. For example, what if this (useless?) CA certificate expires.

OK, sorry for the forum spam, but I think I figured out what's going on.

"NAT" will not be usedul because it's only doing SOURCE NAT when what I actually want is DESTINATION NAT. (I think?)

BINAT is not neccessary, only DNAT. Which I can do using a PORT FORWARD (a bit misleading name in this case)

But I noticed that in this case, I run into an edge case where I cannot ping both the old IP and the new IP at the same time because both packets belong to the same "state"... so pfsense does not know what to do with the ICMP echo response packets coming from the pinged machine, if bothp ings ar running it doesn't know where to send the responses... so that's why I can't ping both at the same time. I can ping one, or the other. Theoretically I think the same should be true also for UDP. Hopefully, though, TCP only should work for what they need to do in this transition period so this is probably fine.

No, I spoke too soon. BINAT doesn't work properly, because if a client tries to talk to the NEW IP, the ICMP ping responses get translated to have the old IP as its source IP....

So I think I need only NAT in one direction

I was able to make this scenario work correctly using BINAT, but I'm confused as to why I needed to use BINAT and not just NAT. It's not clear to me what exactly just a 1:1 NAT (not BINAT) does...

Hello!

I have the following scenario. A firewall with two LAN interfaces, one for printers, and one for the PC LAN.

Printers will be receiving new IP addresses, but some clients are still configured with old IP addresses. For this reason, we want to set up NAT so that if a device on the LAN tries to reach one of the old printer IP's, NAT is used to translate the destination address to the new printer ip.

As such the following 1:1 NAT rule has been created:

Disabled: OFF
Interface: LAN
Type: NAT
External network: <old printer IP>/32
Source/Invert: OFF
Source: LAN net
Destination/Invert: OFF
Destination: Single host or network (<new printer IP>/32)
NAT reflection: <use system default>

This results in the following entry in /tmp/rules.debug:

nat on lagg0 from (lagg0:network) to <new printer IP> -> <old printer IP>/32

But this to me looks like it's backwards somehow... I would expect instead it being old->new? But it doesn't work even if I external network and destination... whatever I do I just get the untranslated packet egressing on WAN (default route) instead...

Am I missing something?

Oh! So the way it's supposed to work is that radvd is supposed to be only running on the primary node.

That doesn't seem to be the case on my cluster at least.

I tried again with HIGH and LOW. Still broken. At least from my windows machine on the LAN. Request packet goes to the secondary firewall, goes out, reply packet is handled by primary firewall.

At this point I have no idea how you're supposed to get CARP to work with IPv6 in the current state of OPNsense, other than that it's apparently possible. At least not if you actually want to use strict state.

What do other vendors do here? I mean I would expect that the resonable thing would be to work analogously to how it works with IPv4, only sending RA for the CARP MAC and VIP.

If we want to classify this as a bug, the bug exists in FreeBSD since forever. pfSense merely added a patch and we are no fans of non-standard OS modification unless they serve a higher purpose.

Any sane gateway will bounce the packet back to the destination, otherwise a checkbox to fix it is really not too much to ask from a user perspective especially since talking about the behaviour is proof that the solution has already been found.


Cheers,
Franco

In that case OPNsense itself is not a sane gateway by your own definition. :-)

Consider the scenario where another OPNsense box is your gateway on your WAN. It receives a "reply" to a packet there there's on state. The stateful firewall will drop that packet on the floor rather than route it anywhere.

Also This is not an OS/FreeBSD bug. The OS is doing exactly what the firewall rule says. Any replies are sent back to the gateway on the WAN subnet. OPNsense is the one generating this firewall rule, and FreeBSD is simply doing exactly what the pf rule says to do (even if it doesn't make sense), so if the bug is anywhere, it's in OPNsense, not in FreeBSD.

Okay, but then I run into a problem. Because my upstream router will route to the primary firewall using its CARP address (I have control over it and I can change the configuration need be), there's a possibility of triangular routing, where for example the secondary firewall receives a packet to be sent outwards, and the nteh priamry firewall receives the response, which it will then drop because there's no corresponding state in the firewall.

pfsync is a thing, yes (and I have it implemented), but because it's asynchronous, the way it works in opnsense it's not usable for active-active scenarios like that, rather useful as a mechanism to stop long-running connections from dropping on failover.

Or at least the above is what I understand. Is any of this wrong?

Due to changes in how MaxMind provides the GeoIP database, you need your own API key. That's what the documentation is there to show you.

Earlier versions of OPNsense do not have the corresponding settings needed to put in your own API key.

If you try to use GeoIP with these older versions of OPNsense, the feature will not work. Any GeoIP alias will act as if it were empty.

Hi.

I'm trying to set up a pair of OPNsense boxes in a CARP HA setup, with IPv6, and I'm stuck on how to get router advertisements to work correctly.

Router advertisements are set up as follows:

Router Advertisements: Stateless
Router Priority: Normal (Low on the secondary unit)
RA Interface: LAN_VIP11 (corresponds to the CARP address)
Advertize default gateway: Yes
DNS Servers: Set to the CARP address
Minimum interval: 200
Maximum interval: 600

Rest is left blank.

At this point I'm expecting router advertisements to advertise the CARP address as a default gateway to IPv6 clients on the lan, but instead, the link local addresses as well as the individual IP addresses of each router are all advertised as defautl gateways, causing CARP not to operate correctly. Sometimes it doesn't work right and traffic goes out of the secondary firewall, which means return traffic is never passed because triangular routing.

Am I missing some secret sauce to get CARP addresses to work properly with router advertisements?

[upstream gateway]

Thanks for showing me the checkbox.

I wasn't aware this was ever since pfSense 1.2. I'm surprised this hasn't bitten me yet, but then again I rarely deploy an OPNsense box behind a stateful firewall on the WAN interface. But I can definitely say that a lot of traffic has been needlessly flowing through the default gateway on some of my installed sites without me ever realizing, leaving performance on the table due to this misfeature. It just never broke anything because typically the gateway on a WAN is not a stateful firewall.

I would though also point out that the documentation text is misleading. To quote:

With Multi-WAN you generally want to ensure traffic leaves the same interface it arrives on, hence reply-to is added automatically by default. When using bridging, you must disable this behavior if the WAN gateway IP is different from the gateway IP of the hosts behind the bridged interface.

This does not mention that it also forces reply traffic bound for directly for another host on the WAN subnet is then also forced through the default gateway, which is especially surprising given the documentation on the "gateway" setting on creating a new firewall rule:

Leave as 'default' to use the system routing table. Or choose a gateway to utilize policy based routing.

In my opinion, either the help text needs to be updated to reflect the case that all traffic is forced back to the upstream gateway on the interface the packet came in (as opposed to back to the same interface, which is not exactly the same thing) or the rule generation code needs to be updated to not apply reply-to when the source IP address is on the same interface (for example using the technique I suggested in the OP). This would also avoid sending reply traffic bound for another host on the WAN network through the default gateway, and would not be likely to break many setups.

It's also not clear from the docs what's considered a "WAN" interface. Because if I make another interface for my second WAN it's going to be an OPT interface, and I'm not sure how/if reply-to is going to be applied to those rules. Based on a quick reading on the source code it seems to depend on whether you've set an upstream gateway on the interface or not, which makes sense, although it could be more explicit.

https://github.com/opnsense/core/blob/master/src/opnsense/mvc/app/library/OPNsense/Firewall/FilterRule.php#L144-L147