Messages

I was using the floating rule as a lazy shortcut to not have to change switch ports but now I see that it doesn't even work except in specific circumstances.

It helps me to write this out and make sure I got the full picture-

I'm not seeing the issue in Proxmox NOT because it is designed differently than TrueNAS.  It's because there is no non-default return path for management traffic.  Management requests always go to the configured interface with the default IP and gateway and reply from the same, because I don't have any interface with IPs on direct connected networks.

The VM/container bridge, even though it's on a separate interface, cannot respond anyway as it's unconfigured.  Because it's VLAN-aware, VMs/LXCs that are tagged on the bridge are isolated to the respective networks.

Similarly, TrueNAS VMs/LXCs can go on a bridge with VLANs as long as the bridge, its VLAN parents, and the parent IF are all unconfigured.  It might need to be separate bridges in TN, I think, because there is no concept of a VLAN-aware bridge but I'm not certain.  (The Aquantia NIC falls down in a layered scenario in my limited testing so far; seems like immature firmware/drivers from other reports).

---

Up to this point things make sense.

Where I'm getting tripped up still is that TrueNAS gives options for binding the UI and the services separately, and there are videos showing that TrueNAS can have different networks for iSCSI (such as for a SAN for Windows hypervisors) and for shares for office clients, for example.

I can imagine that they have a NIC on each network and bind the respective service to an IP on it, so in that case the service is directly connected on the respective subnet.  There should be no routing for Hypervisor<->iSCSI or for client<->SMB.

Where does the management interface go?  Is it available to all and locked down only with strong authentication? 

Is the purpose of allowing the UI to be bound on a specific IP just so that people can't find it by typing in https://truenas.local?

---

I found one such thread on the TN forum that was funny.  After you explained the limitation, the guy was upset and came up with an elaborate scheme involving intermediate switches and routing / firewall tricks to get his isolated management.  He never updated to say if he had success, lol.

Could you show the NAT rule as well?  I think this should be:

Interface: LAN
TCP/IP version: 4
Protocol: TCP/UDP
Source: LAN net
Source port range: any
Destination: any
Destination port range: 53 (DNS)
Redirect target IP: 192.168.1.254
Redirect target port: 53 (DNS)

This will redirect all plain DNS requests on LAN, not just from the google devices, to the AdGuard server and will break local resolution.  You'd have to make host overrides in AGH or forward back to Unbound for local zones without creating a loop.

If you need to you can create an Alias with the source IPs of just the google devices and use that in place of 'LAN net' for Source.

AdGuard Home is also available as a plugin in OPNsense, but requires that you enable the repo.  I'm assuming your AGH VM is able to receive requests and doesn't have a host firewall block (e.g. default deny on incoming with no exception for DNS).

When you created the NAT rule, which option did you use for 'Filter rule association'?

The default is to create an associated allow rule and you should see that it got added to the interface rules.  Make sure that rule has precedence over any blocking rules, especially any quick rules before it.

Ayyy, this is a painful lesson.  I spent a couple days. :)

Thank you, Patrick.

Per Quad9, turn it off.

https://docs.quad9.net/Quad9_For_Organizations/DNS_Forwarder_Best_Practices/

Disable DNSSEC Validation

Since Quad9 already performs DNSSEC validation, DNSSEC being enabled in the forwarder will cause a duplication of the DNSSEC process, significantly reducing performance and potentially causing false BOGUS responses.

Hello all, I'm wrestling with a first TrueNAS deployment and this issue is beyond my skillset.  Asking here because I think it's relevant to pf/OPNsense and maybe my particular rules causing the issue.

The NAS has two integrated NICs, an Intel I226-V (top) and an Aquantia AQC113 (bottom):

Code Select Expand

truenas_admin@truenas[~]$ ethtool -i enp6s0
driver: igc
version: 6.12.33-production+truenas
firmware-version: 2014:8877
expansion-rom-version:
bus-info: 0000:06:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes


Code Select Expand

truenas_admin@truenas[~]$ ethtool -i enp3s0
driver: atlantic
version: 6.12.33-production+truenas
firmware-version: 1.3.31
expansion-rom-version:
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no

At first I configured the management interface on enp6s0 with a static IP address, as per TrueNAS guidelines.  This interface has IP 192.168.1.118 on LAN.  It has the default gateway 192.168.1.1, DNS also 192.168.1.1, and the web UI is bound to this IP only.

Up to this point everything is working with a stable connection.  I'm able to access the web UI from a client on VLAN 30 ("CLEAR net") using a Floating rule with direction IN on the CLEAR interface that allows the client to "any" on LAN.

The problem comes when I configure the second interface.  I assign it a static IP 172.21.30.118 which is from the CLEAR subnet as this will be my data interface for SMB shares (the SMB service will bind to it).  I don't assign any additional gateway or DNS, and no static routes.  Just the IP on the interface.  IPv6 auto-config is still disabled in TrueNAS.

After this step, my TrueNAS web UI becomes unstable.  I'm still able to log in, but after some seconds I'm kicked off.  This process repeats itself ad-infinitum and I'm not able to stay in the UI long enough to get work done.  This dialog appears in between when I'm kicked off and the login prompt follows it.

You cannot view this attachment.

If I remove the IP on the second interface, then everything becomes stable again.

Checking the firewall logs while this happens I can see the initial request is allowed as it's matching the Floating rule, and it gets a response.  I don't know why the DNS lookups in between are logged at all as those are within the same subnet (this is a group rule).  Finally another interface group rule is matching instead which is rejecting and causing me to get kicked off.

You cannot view this attachment.

This happens in a loop as mentioned above.  I can see that the source ports changed in between the login and the rejected packets.  This also repeats and is reflected in the state table with new states being generated each cycle.

Also, there is a group 'pass' rule which is supposed to be overriding this 'reject' rule in any case, but that's not happening.

Floating:
You cannot view this attachment.

Group rules (from the CLEAR interface's view):
You cannot view this attachment.

The "PORTS_OUT_LAN" alias already has port 443 included.

Did I set up some kind of asymmetric routing condition when I configured the second NIC?  Is this a TrueNAS quirk? 

I do have Proxmox management on the same LAN network and with a separate VM bridge, but no problems at all there.  That's been stable for a while now.

TrueNAS ip/route info:

Code Select Expand

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 6c:xx:xx:xx:41:0a brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.118/24 brd 192.168.1.255 scope global enp6s0
       valid_lft forever preferred_lft forever
    inet6 fe80::xxxx:xxxx:xxxx:410a/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 6c:xx:xx:xx:41:09 brd ff:ff:ff:ff:ff:ff
    inet 172.21.30.118/24 brd 172.21.30.255 scope global enp3s0
       valid_lft forever preferred_lft forever
    inet6 fe80::xxxx:xxxx:xxxx:4109/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever


Code Select Expand

truenas_admin@truenas[~]$ route   
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         firewall        0.0.0.0         UG    0      0        0 enp6s0
172.21.30.0     0.0.0.0         255.255.255.0   U     0      0        0 enp3s0
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 enp6s0


The only caveat is that static configured addresses are not resolved by Dnsmasq. So I had to add them manually in Unbound overrides.
If the device is configured to get the IP via DHCP the name resolution work with both dynamic and reserved addresses, but if the IP is fixed on the device and it doesn't get it from DHCP, the name resolution doesn't work.
With ISC-DHCP, the name resolution worked both in the cases of static IPs configured on the devices and in IPs assigned via DHCP.

I'm not sure which approach is better (appreciate advice from others), but I have been adding Host entries in Dnsmasq even for statically configured ones.  They don't show up in Leases, but they do get added to DNS and I presume also marks that IP as reserved from the pool.

I make sure all the static IPs I use fall within the Dnsmasq range, which is a difference from ISC.  This can obviously leave unused IPs if your Dnsmasq pool is, for example, .100 to .254.  Then the entire range .2 to .99 is wasted.

I can think of a couple solutions:

1) Define the Dnsmasq range as .2 to .254 (.1 being for the gateway in this example)
2) Define two ranges:  192.168.1.2 - 192.168.1.99 (static pool) and 192.168.1.100 - 192.168.1.254 (DHCP pool) for the same domain

Question for @Monviech/@Maurice or others - is #2 viable and a good idea?  I haven't tested.

If you're saying that legitimate websites like this forum are hosting malicious payloads, then please show us.  You have the burden of proof.

Cool, you have a trusted network of some kind.  You use segmentation.

I presume most are and I think that it is much more effective than apparmor on the client.

I challenge the idea that the backend is "open and everything is allowed," because there shouldn't be access to the backend.  Only if there is an exploitable vulnerability in one of the services, which is in the realm of possibility (however unlikely).  I want to give OP some credit, despite the other wild and alarmist claims.  I don't buy that their OPNsense is getting hacked like this without receipts.

Good catch :) and yes I meant without a password because clients on the same LAN can't be blocked by rules.

So you don't think that most are using VLANs then?  I don't have any metrics but it seemed like that was the main reason to use OPNsense vs. the ISP-provided or retail router.

I have connections through the browser attacking my system trying to break through apparmor. It shows up in auditd log file. Its a mile long. Using this as your endpoint protection or a similar app protects your operating system and  the LAN side backend of opnsense which is open and everything is allowed, thats how they were breaking my separate opnsense router.

How many OPNsense users are on a flat LAN with open access to their management interfaces?

Merry Christmas! Happy holidays all

... annnnd, scratch my theory.  I missed that you said you have DNS over TLS, lol.

ECS or not makes no difference in how you configure Quad9 for Unbound.  It's just a forwarding address with a different IP than the non-ECS version:  9.9.9.11 vs. 9.9.9.9.  Unbound doesn't care :)

Quad9's TLS forwarding guide for OPNsense: https://docs.quad9.net/Setup_Guides/Open-Source_Routers/OPNsense_%28Encrypted%29/

Curious- why do you prefer the ECS version?  Do you get an appreciable performance boost from CDNs?

If Unbound is doing your DNS resolution then Dnsmasq should not be reached by your clients.  You need to configure it as per the examples in https://docs.opnsense.org/manual/dnsmasq.html#configuration-examples and make sure Unbound is forwarding to Dnsmasq for your internal domains.  Dnsmasq doesn't need to know anything about Quad9 in this case.  It should never be answering queries for any domain except those configured on your network.