[SOLVED] LAN blocking certain requests internall to OPNsense

[Sam of Ham]

Hi all! Sorry if this is a dumb question, I'm on the newer site to networking/firewall stuff, but learning quite quickly.

I have a new OPNsense firewall online, and, apart from Tailscale, and Zenarmor, I want it pretty default. I'm replacing a decent-enough ISP router for the purposes of better security and control, and because that device I suspect has a dodgy WAP.

I run Proxmox on a device inside the network which hosts things like HomeAssistant and a Pi.hole. Since deploying OPNsense, I noticed that neither PvE nor Pi.hole can reach out for updates - updating PvE returns the below, and updating Pihole gravity returns the normal DNS not found error. Internet is still up generally - I haen't run into a blocked website or app yet, though it's only been in place for about a week.

Code Select Expand

starting apt-get update
Ign:1 https://pkgs.tailscale.com/stable/debian bookworm InRelease
. [More PvE sites get blocked here - cut out so this post isn't a mile long]
Err:2 http://security.debian.org/debian-security bookworm-security InRelease
  Temporary failure resolving 'security.debian.org'
Err:1 https://pkgs.tailscale.com/stable/debian bookworm InRelease
  Temporary failure resolving 'pkgs.tailscale.com'
Ign:4 http://download.proxmox.com/debian/ceph-quincy bookworm InRelease
Ign:3 http://ftp.debian.org/debian bookworm InRelease
Ign:6 http://ftp.debian.org/debian bookworm-updates InRelease
Ign:5 http://download.proxmox.com/debian/pve bookworm InRelease
Err:4 http://download.proxmox.com/debian/ceph-quincy bookworm InRelease
  Temporary failure resolving 'download.proxmox.com'
Err:3 http://ftp.debian.org/debian bookworm InRelease
  Temporary failure resolving 'ftp.debian.org'
Err:5 http://download.proxmox.com/debian/pve bookworm InRelease
  Temporary failure resolving 'download.proxmox.com'
Err:6 http://ftp.debian.org/debian bookworm-updates InRelease
  Temporary failure resolving 'ftp.debian.org'
Reading package lists...
W: Failed to fetch http://ftp.debian.org/debian/dists/bookworm/InRelease  Temporary failure resolving 'ftp.debian.org'
W: Failed to fetch http://ftp.debian.org/debian/dists/bookworm-updates/InRelease  Temporary failure resolving 'ftp.debian.org'
. [More get blocked here - cut out so this post isn't a mile long]
W: Failed to fetch https://pkgs.tailscale.com/stable/debian/dists/bookworm/InRelease  Temporary failure resolving 'pkgs.tailscale.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.
TASK OK

When I look at a live capture in OPNsense, I can see a few key suspects being blocked at what looks like the LAN side; for reference, PvE is 192.168.1.190, and the Pi.hole is .192. The OPNsense firewall is at 192.168.1.1, acting as the router and gateway as well. 192.168.1.169 is my laptop, which I'm working on this from.

Code Select Expand

lan 2024-06-25T12:19:05 192.168.1.190:41641 172.29.80.1:41641 udp Default deny / state violation rule
lan 2024-06-25T12:19:05 192.168.1.190:41641 192.168.56.1:41641 udp Default deny / state violation rule
lan 2024-06-25T12:19:05 192.168.1.190:41641 192.168.77.75:41641 udp Default deny / state violation rule
lan 2024-06-25T12:19:05 192.168.1.190:41641 172.29.144.1:41641 udp Default deny / state violation rule
lan 2024-06-25T12:19:05 192.168.1.190:41641 172.21.112.1:41641 udp Default deny / state violation rule
lan 2024-06-25T12:19:05 192.168.1.190:41641 172.23.48.1:41641 udp Default deny / state violation rule

lan 2024-06-26T11:51:09 192.168.1.169:50811 192.168.1.1:1900 udp Default deny / state violation rule
lan 2024-06-26T11:51:09 192.168.1.169:50811 192.168.1.1:5351 udp Default deny / state violation rule
lan 2024-06-26T11:51:09 192.168.1.169:50811 192.168.1.1:5351 udp Default deny / state violation rule

lan 2024-06-26T11:51:55 192.168.1.192:34320 192.168.1.1:1900 udp Default deny / state violation rule
lan 2024-06-26T11:51:55 192.168.1.192:34320 192.168.1.1:5351 udp Default deny / state violation rule
lan 2024-06-26T11:51:55 192.168.1.192:34320 192.168.1.1:5351 udp Default deny / state violation rule
lan 2024-06-26T11:51:55 192.168.1.192:43553 192.168.1.1:5351 udp Default deny / state violation rule
lan 2024-06-26T11:51:55 192.168.1.192:43553 192.168.1.1:5351 udp Default deny / state violation rule

Lastly, here are my LAN and WAN rules.

LAN:

Code Select Expand

IPv4 TCP/UDP LAN net * LAN address 53 (DNS) * * Allow access to DNS server on LAN interface
IPv4 * LAN net * ! PrivateNetworks  * * * Allow access to the internet, but block access to private networks
NOTE: I have disabled BOTH rules in testing. These degrade internet functionality further and do not resolve this issue.

WAN:

Code Select Expand

IPv4 * Blocked_Devices  * * * * * Blocks the Blocked_Network Alias List from OUT Traffic.
IPv4 * Special_Allowed  * * * * * Rule that allows Special_Allowed group through WAN.
This ^ rule, Special_Allowed, I created to try to resolve this - it has .190 and .192 in it and allows HTTP. The block rule above it only blocks one IP, .103, which is a cheap smart device that doesn't need to communicate with HQ in who-knows-where.

Thanks in advance for your help, again apologies if this seems nooby, but please feel free to educate me in the interest of learning as well!

[Sam of Ham]

Oh, and, just FYI, Zenarmor isn't blocking anything at all. It's blocklist is completely empty.

[Seimus]

Code Select Expand


IPv4 TCP/UDP LAN net * LAN address 53 (DNS) * * Allow access to DNS server on LAN interface
IPv4 *         LAN net * ! PrivateNetworks  * * * Allow access to the internet, but block access to private networks


1st rule has wrongly defined port, When a host does a DNS query the DNS port (53) will be the destination port. The DNS server replies back with source port (53). If you want to exclusively allow DNS to be resolved for a host you need to have the destination port as 53. PiHole needs as well access the internet (Destination port 53) to upstream DNS servers or NS if you have it configured for unbound.

2nd Rule (same as well the 1st rule), does nothing for host in the same broadcast domain network, meaning device in the same /24 etc. subnet still can communicate to each other, just keep that in mind.

LAN needs to be direction IN on the rules

What is the direction of the rules you have set on the WAN?
Anyway do you see on you Pihole, that it can resolve the queries it receives?

Regards,
S.

[Sam of Ham]

Howdy! Thank you for your help! Apologies, I should have uploaded screenshots to begin with.

I'm definitely missing something. It looks like that's how this is set - destination is 53, and LAN net is the source (so, in my mind, anything from LAN is allowed to speak to .192, the DNS server)

In those logs, isn't it the PvE and Pihole IPs being blocked talking to the gateway, the firewall at 1.1?

Pihole is definitely working and my network is up as far as normal usage goes - I can get to internet from devices, Pihole is filtering as it should. However, the Pihole can't update it's own gravity, which times out with the usual error in Pihole:

Code Select Expand

  [✗] DNS resolution is currently unavailable
  [i] Time until retry: 120

The second rule, the 'private networks' rule, was from a video turorial seen HERE () that I was following. That guide has a trusted/untrusted network setup, which I want to one day also do, but for now just want to focus on replicating and tightening down the same network setup as under the ol' ISP router.

Here's WAN and a screenshot attached:

Code Select Expand

IPv4 * * * * * * *
IPv4 * Blocked_Devices  * * * * * Blocks the Blocked_Network Alias List from OUT Traffic.
IPv4 * Special_Allowed  * * * * * Rule that allows Special_Allowed group through WAN.


And a log screenshot attached for reference.

Thank you!!

[Sam of Ham]

Log here: (Size limitations in og reply)

[cookiemonster]

blocks seem to only be for out-of-state packets i.e. normal. Nothing there for DNS blocks.
What I think is missing is a bit of clarity. You have a pi-hole and block/pass rules to port 53 in the thread along others.
What I don't see is
a) any NAT or port forward rules to figure out who is to be the recursive resolver and
b) what these blocked_devices and special_allowed are for. The names are clear enough, how they might be interfering are not.
So, who do you want your LAN clients and Pi-hole to use as their respective recursors? This will inform what forward, nat rules might be needed. Also does dnsmasq and/or Unbound play a part, if so how is it configured?

[Sam of Ham]

Hey Cookie!

I didn't think NAT would necessarily matter here, but you might be onto something - and I'm so here for the education.

Okay, here's the network layout.
OPNsense router/firewall sits at 192.168.1.1. It's the gateway here and WAN is an ISP termination device. LAN goes to a dumb switch.
Pihole (1.192) is on Proxmox (1.190) and is the DNS and DHCP server(s). I don't believe unbound is in use and I'm not sure how to tell!
The firewall is set to forward DHCP and DNS to 1.192. No DHCP is running on it (though I'm thinking of switching soon.) Pihole has a defined IP range and gateway (1.100-240, gateway 1.1) so the gateway device should only be shifting DNS and DHCP sideways to the Pihole, then back again for public internet.

To answer your questions:
A) I'm not doing any port forwarding that I know of. I didn't think I'd need to to replicate a typical flat network, honestly! A definition for this might be good, actually, as I know for a fact that some devices (eg., Hue Bridge) have hard-coded DNS servers and bypass the Pihole even though it should be forwarded by the gateway.
B) Blocked_Devices has one IP in it, 1.103, and is the Hue Bridge. It's my test for seeing if a block there works, and thus, where I can put devices that should be locked to LAN only. Stuff like cheap smart switches.
Special allowed is what I created to allow HTTP for 1.190 (PvE) and 1.192 (Pihole), as a test to see if that would let them start communicating. It didn't! (I mentioned these earlier, also.)

Lastly, DNSMASQ is running on the Pihole and occasionally chirps up in Pihole logs but has never been problematic.

I generated a debug log in the Pihole. Here are a few line extracts from the DNSMASQ section:

Code Select Expand

Jun 27 00:00:02 dnsmasq[9114]: query[A] myipv4.p1.opendns.com from 192.168.1.191
Jun 27 00:00:04 dnsmasq[9114]: query[A] pkg.opnsense.org from 192.168.1.1
Jun 27 00:00:04 dnsmasq[9114]: reply pkg.opnsense.org is 89.149.222.99


And this is from the Pihole Networking diagnosis:

Code Select Expand

*** [ DIAGNOSING ]: Network routing table
   default via 192.168.1.1 dev eth0 onlink
   192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.192

*** [ DIAGNOSING ]: Networking
[✓] IPv4 address(es) bound to the eth0 interface:
    192.168.1.192/24

[✓] IPv6 address(es) bound to the eth0 interface:
    fe80::be24:11ff:febd:cc00/64

[i] Default IPv4 gateway(s):
     192.168.1.1
   * Pinging first gateway 192.168.1.1...
[✗] Gateway did not respond. (https://discourse.pi-hole.net/t/why-is-a-default-gateway-important-for-pi-hole/3546)

[i] Default IPv6 gateway(s):

*** [ DIAGNOSING ]: Ports in use
    udp:0.0.0.0:55030 is in use by tailscaled
[✓] udp:0.0.0.0:53 is in use by pihole-FTL
    udp:0.0.0.0:67 is in use by pihole-FTL
    udp:100.113.249.128:123 is in use by ntpd
    udp:192.168.1.192:123 is in use by ntpd
    udp:127.0.0.1:123 is in use by ntpd
    udp:0.0.0.0:123 is in use by ntpd
    udp:0.0.0.0:41641 is in use by tailscaled
[✓] udp:[::]:53 is in use by pihole-FTL
    udp:[fe80::2387:b1cb:1db9:1f5b]%tailscale0:123 is in use by ntpd
    udp:[fd7a:115c:a1e0::c871:f980]:123 is in use by ntpd
    udp:[fe80::be24:11ff:febd:cc00]%eth0:123 is in use by ntpd
    udp:[::1]:123 is in use by ntpd
    udp:[::]:123 is in use by ntpd
    udp:[::]:547 is in use by pihole-FTL
    udp:[::]:41641 is in use by tailscaled
[✓] tcp:0.0.0.0:53 is in use by pihole-FTL
[✓] tcp:0.0.0.0:80 is in use by lighttpd
    tcp:127.0.0.1:25 is in use by master
    tcp:100.113.249.128:33441 is in use by tailscaled
[✓] tcp:127.0.0.1:4711 is in use by pihole-FTL
    tcp:*:22 is in use by sshd
[✓] tcp:[::]:53 is in use by pihole-FTL
[✓] tcp:[::]:80 is in use by lighttpd
    tcp:[::1]:25 is in use by master
[✓] tcp:[::1]:4711 is in use by pihole-FTL
    tcp:[fd7a:115c:a1e0::c871:f980]:33441 is in use by tailscaled

*** [ DIAGNOSING ]: Name resolution (IPv4) using a random blocked domain and a known ad-serving domain
[✓] cdn.imhd.io is 0.0.0.0 on lo (127.0.0.1)
[✓] cdn.imhd.io is 0.0.0.0 on eth0 (192.168.1.192)
[✗] Failed to resolve cdn.imhd.io on tailscale0 (100.113.249.128)
[✓] doubleclick.com is 142.250.66.206 via a remote, public DNS server (8.8.8.8)

*** [ DIAGNOSING ]: Name resolution (IPv6) using a random blocked domain and a known ad-serving domain
[✓] support-customer-security-10898276365.netlify.app is :: on lo (::1)
[✗] Failed to resolve support-customer-security-10898276365.netlify.app on eth0 (fe80::be24:11ff:febd:cc00)
[✗] Failed to resolve support-customer-security-10898276365.netlify.app on tailscale0 (fd7a:115c:a1e0::c871:f980)
[✗] Failed to resolve support-customer-security-10898276365.netlify.app on tailscale0 (fe80::2387:b1cb:1db9:1f5b)
[✗] Failed to resolve doubleclick.com via a remote, public DNS server (2001:4860:4860::8888)

So ultimately, I suppose (and I think), everything should talk either to the gateway (.1.1) or the Pihole (1.192) directly, and that traffic should be forwarded by the Pihole to the DNS servers listed in it, through the gateway at 1.1. It should... I think!... be that simple?

Thanks again all for the help. I'm mostly self-taught so I could well be missing something stupid obvious, please abuse me gently (but welcome to abuse me nonetheless! ;D )

[Seimus]

Oh yes you are right that port is for destination, sorry about that.

But lets review those rules again on LAN >

Code Select Expand

IPv4 TCP/UDP LAN net * LAN address 53 (DNS) * * Allow access to DNS server on LAN interface
This rule allows only traffic of source from LAN subnet to LAN address of OPNsense port 53
Why? You have Pihole as your DNS why you need this rule?

Code Select Expand

IPv4 * LAN net * ! PrivateNetworks  * * * Allow access to the internet, but block access to private networks
This rule allows only traffic of source from LAN subnet to Public ranges (if that alias contains all subnets of private ranges), LAN subnet any source port to any destination IP/source port

Effectively as mentioned if you have a /24 in which are all your devices, this rules does not prevent them to talk to each other. Traffic in same broadcast domain wont hit the FW.

The Traffic that is being blocked in LAN is UDP, the only UDP traffic you allow on your GW is the DNS port 53 (1st rule).

Looks like all devices are able to get replies for queries expect the ones that PiHole tries itself. What DNS do you have set for the Pihole? For the machine that runs Pihole. Is it the OPN? If so why not target it to the Pihole itself.

For test log into the machine where Pihole is running and from konzole run nslookup opnsense.org, you will see who is the DNS for it and if it can resolve.

https://discourse.pi-hole.net/t/solved-dns-resolution-is-currently-unavailable/33725/5
https://github.com/pi-hole/pi-hole/issues/1801

P.S. Those rules on WAN are IN direction it will not block any Private IP, as OUT default permit any any will be hit. If you want to block something block it on IN on the LAN.

Regards,
S.

[Sam of Ham]

Hey and sorry I didn't reply yesterday!

This rule allows only traffic of source from LAN subnet to LAN address of OPNsense port 53
Why? You have Pihole as your DNS why you need this rule?

Ah, I see - honestly, I was following a guide. It wasn't quite this one, but at the end you can see he sets up a similar DNS rule. I'll reply again if I can find the one I followed. https://www.youtube.com/watch?v=jiiQUTQTNtk
I've also disabled both rules in testing and not had a change.

I suspect part of the issue could be Tailscale and my Pihole setup. I think there are a few holes in my know-how that have resulted in some sub-optimal setup.

Here's a bit of a look at how PvE is set up, and after that, how I've changed it:

- Proxmox (192.168.1.190)
  - DNS:
    - Search domain: tail[identifier].ts.net
    - DNS Server 1:   100.100.100.100
↳ Pihole (192.168.1.192)
  - DNS:
    - DNS Server 1:   Use host settings
    - DNS Server 2:   Use host settings
↳ Unifi (192.168.1.194)
  - DNS:
    - DNS Server 1:   Use host settings
    - DNS Server 2:   Use host settings
And so on for the other VMs/LXCs.

Here's what I've changed it to:
- Proxmox (192.168.1.190)
  - DNS:
    - Search domain: local
    - DNS Server 1:   192.168.1.192
    - DNS Server 2:   1.1.1.1
    - DNS Server 3:   100.100.100.100
↳ Pihole (192.168.1.192)
  - DNS:
    - DNS Server 1:   Use host settings
    - DNS Server 1:   Use host settings
↳ Unifi (192.168.1.194)
  - DNS:
    - DNS Server 1:   Use host settings
    - DNS Server 1:   Use host settings
And so on for the other VMs/LXCs.

I changed the above to point PvE specifically to the Pihole. Now, the reason I hadn't changed this so far was twofold; first, because everything was working fine, and second, because last time I tried to change the DNS for PvE to the Pihole and ended up flooding it recursively with over 100,000 requests/second. That's obviously my bad of course, but since everything was working, I figured it must be up OK!

Also, I edited

Code Select Expand

nano  /etc/resolv.conf and noticed that only Tailscale was in there as DNS servers/NS - I've added the below and I think that's helped here.
nameserver 1.1.1.1
nameserver 9.9.9.9
nameserver 100.100.100.100
search taile[ident].ts.net proxmox.local

I figure I must have had to change something in Tailscale and didn't - so some machines were actually talking through Tailscale the whole time. Again... My bad.

FWIW, now I'm getting updates for both devices.

Now, that doesn't solve the problem here in that communication to the gateway is still being blocked, but it does solve my initial reason for asking as I can now update both systems. Hopefully we can troubleshoot the issues with talking to 192.168.1.1 and then we're good!

FWIW, I can of course now look up something like opnsense via console on Pihole:

Code Select Expand

root@pihole:~# nslookup opnsense.org
Server:         1.1.1.1
Address:        1.1.1.1#53

Non-authoritative answer:
Name:   opnsense.org
Address: 178.162.131.118
Name:   opnsense.org
Address: 2001:1af8:4700:a1fa:3::2

I guess - should I care beyond this point if those queries to 1.1 are being blocked now that the only (as far as I can see) two devices not getting proper internet access are resolved? I don't know why anything should be blocked from talking to .1.1, but, I also don't see any other internet problems, so... Yeah!

Again, thank you so much to all of you, deeply.

[Seimus]

Yea so you see, thats why I had questions about your Pihole and how is the system configured that runs the Pihole. Because Pihole is the DNS lets say so it receives requests, but the machine on which the Pihole is running if you dont set local loopback or the IP of the Pihole itself will go where you tell him 1.1.1.1 8.8.8.8 etc. That was fishy in your Pihole setup cause all could resolve but Pihole system couldn't.


In regards of your other issue. Sorry but we were too focusing on the DNS stuff. Can you describe what is the other issue from scratch? Or even better if your DNS problem is resolved open a new thread so it will not get mixed.

Regards,
S.

[Sam of Ham]

Howdy Seimus! Apologies, had a packed weekend ahead of me.

Honestly, I figured that, because the Pihole was working fine at the time, it probably wasn't that - but (clearly!) I was wrong! That's a case of "assuming only makes an ass out of U and ME" on my part, so I apologise for wasting that time with you guys!

Okay, so, all that aside, this leaves us with what might not even be a problem in the end.

In short, I'm seeing two things that raise queries, rather than concerns, in my mind:

FIRST; that some devices are querying the OPNsense gateway (.1.1) and being blocked. See attachment 2.

SECOND; that some queries from my work office IP are being blocked coming into the public IP for this device, yet I'm currently accessing it via Tailscale without issue. See attachment 1.

Again everything seems to be working fine, but, I'd guess I'd see fairly few red entries in the logs here rather than so many - I get that scans are naturally going on online all the time, but my guess is most traffic would be simple legitimate LAN and WAN in/out. I'd say perhaps 50%~ of my log when viewed in 'live view' is blocked traffic.

Thanks all!!

[Seimus]

FIRST; that some devices are querying the OPNsense gateway (.1.1) and being blocked. See attachment 2.

So lets not use querying word here, queries as such are understood as DNS lookup from a host. Lets call it traffic/packets hitting OPNsense gateway. And this is normal, as you can see by the protocol and Port UDP/RandomPort.
This is can be fairly normal depending on what application is doing it as mentioned previously your Rule on LAN allows only port UDP/53 towards the GW so everything else is dropped because you are not allowing it.

SECOND; that some queries from my work office IP are being blocked coming into the public IP for this device, yet I'm currently accessing it via Tailscale without issue. See attachment 1.

Similiar as to FIRST, on your WAN you block INgress traffic, meaning what was not triggered by devices in LAN, all new Sessions that are originating from WAN will be blocked. This is how it should be, you do not want to open your FW WAN if you dont have a specific reason otherwise.

That source port you see 41641, is specific to TailScale, Its used for direct UDP tunnel initialization, its used in specific cases, you mostly do no need to open this port on your FW. If Tailscale works you dont need to bother with it>

https://tailscale.com/kb/1082/firewall-ports


If all works, there is no reason to be worried. Those reds you see is what you should see, specifically on WAN, and on LAN depending on your ruleset. Thats what a FW does, block what you tell him to block, and protects you from actors on the Internet that try to reach your Network.

[Sam of Ham]

Ah I see!! That makes sense! Sorry, I suppose I should have looked up those ports to see if Tailscale came back as a result - I suppose I assumed that so many red entries--especially between LAN IPs or known IPs like my office public IP--meant that something was probably going wrong, especially compared to what I saw (or more accurately, what I didn't see under the old ISP router - when, in reality, these would likely have been happening anyway and just invisible to me. I guess while rolling out a new system that i've set up myself - a la not a professional - I lack trust in the config and reliability, at least until those months go by without issues.

Thank you so, so much for your help and education, and for being kind and respectful with me in this thread! You're a true legend, both yourself and Cookie earlier on.

[Seimus]

You are welcome.

In regards of your old router, if you had same rules there as you had on OPNsense definitely same behavior was happening. Firewalls by default use "implicit deny" meaning what you dont allow is by default blocked.

So when you do Rules, and you want to allow only specific traffic for a connection or an APP, on a L4 FW you do it per Source/Destination, IP/Port, Protocol - this is called as well the 5-tuple.

If you configure your rules you need to think of what/who can with what/whom communicate across two different networks, example LAN to WAN or WAN to LAN. Usually you are considered only to allow LAN IN towards other networks such as Internet, you do not need to configure WAN IN in order to have Internet access. IN rules on WAN are used to have access to your resources on LAN network such as Webserver, Media, etc. basicaly per need.

But my advice, if you are new to this, do not do IN WAN rules until you dont understand the implications of it.

Anyway if your issues are resolved please adjusted your topic subject with [SOLVED] in the beginning ;)

Regards.
S.

[Sam of Ham]

Thanks again Seimus! I was away for a few days but have updated this post now.

My old router was just an ISP device so very little to customise there other than, effectively, "on or off". I couldn't see what it was allowing or blocking, but I figured fairly clean internet like my home site would be mostly green.

May you always have a cold spot to the pillow, always have enough spread on your knife to perfectly cover the toast, and your lettuce always be crunchy and crisp.  ;D :D 8)