Firewall issues - blocked access to 1.1.1.1

[GreenMatter]

OPNsense 23.7.12. Whole LAN (vlans) and all hosts are connected through Unifi switch.I keep having issues with either firewall setup or firewall itself.
One story is with blocked TCP communication between one vlan's interface and vlan's hosts [size=78%]https://forum.opnsense.org/index.php?topic=37602.msg184311#msg184311[/size]
And this one is about LAN wide blocked access to 1.1.1.1 DNS.
It is accessible from opnsense/firewall itself but not from within my whole LAN.
How to troubleshoot firewall - to examine all rules for presence of anything related to those 2 issues?
I've done it visually, I mean checking rules and I couldn't find anything what sticks out. Is there any tool or smarter than myself cli command?

[cookiemonster]

Maybe something inspecting traffic like Zenarmor, blocklists in Unbound or other services,  set to block DoT or DoH ?

[GreenMatter]

Maybe something inspecting traffic like Zenarmor, blocklists in Unbound or other services,  set to block DoT or DoH ?

I stopped Zenarmor service and disabled blocking in Unbound and it didn't help.
Is there any tool to allow me "query" firewall for blocked hosts/addresses?

[chemlud]

FW rules?

[GreenMatter]

FW rules?

I've done "visual" check but I might have something overlooked. I'm asking for more automated check / command letting me to verify presence of 1.1.1.1 address across ALL rules...

[chemlud]

it's not just about the target address, it could be port, protocoll... ;-)

[Fright]

@GreenMatter
may be you can try to find the 1.1.1.1 references in aliases (Firewall: Diagnostics: Aliases -> Find references)?
if that doesn't give any hint, it will probably be necessary to enable logging of default blocking rules, enable logging of other suitable blocking rules and look at the (live) log

[GreenMatter]

@GreenMatter
may be you can try to find the 1.1.1.1 references in aliases (Firewall: Diagnostics: Aliases -> Find references)?
if that doesn't give any hint, it will probably be necessary to enable logging of default blocking rules, enable logging of other suitable blocking rules and look at the (live) log

Finally I've found the reason, but I don't understand WHY...
I have created 3 VPN gateways (they use interfaces created by OpenVPN clients). Only one gateway has assigned vlan (rule) and outbound NAT; 2 other aren't in use.
These VPN gateways have monitor IP configured and one of them was 1.1.1.1. Long story short: any IP configured as "Monitor IP" (in any of VPN gateways) is not accessible from LAN hosts. It's applicable also for gateways without and FW rules assigned.
So, why it's like that?

[GreenMatter]

Anybody, anything?

[Fright]

anything?

adding "Monitor IP" adds a records in routing table

[GreenMatter]

@Fright
Thanks, I didn't know that.
It means it is added in such a way that excludes LAN?


EDIT:
In such a case, what hosts are the best to serve as monitor IP? External DNS sometimes might be useful inside LAN...

[Fright]

can't tell without full understaing your setup/rules (how host route can interfere with pf-rules etc), sorry
you asked for a hint - i was try to give one )
another hint - there is a "Disable Host Route" checkbox above Montor IP setting 

[GreenMatter]

can't tell without full understaing your setup/rules (how host route can interfere with pf-rules etc), sorry
you asked for a hint - i was try to give one )
another hint - there is a "Disable Host Route" checkbox above Montor IP setting 

I don't have any FW rules for "Monitor IPs". So, I guess I can exclude FW itself. That's why it confused me so much.
Activation of "Disable Host Route" doesn't help. Maybe because of [size=78%]https://github.com/opnsense/core/issues/6342[/size][/size] - I don't know how valid it is in 23.7.12 and now in 24.1 (I'm afraid to upgrade since I'm away of router location).


Anyway now I understand it a bit more: since monitor IPs have routing through gateways which are not in use in LAN - LAN hosts won't be able to contact them...