I have a rule to forward all DNS queries that are targeted outside my LAN net to my piHole, it's in Firewall>NAT>PortForward and it is like this :
Interface : LAN
Protocol : TCP/UDP
Source invert : ticked
Source : DNS_allowed (an alias of mine, the IP of my client is NOT in this alias, it will not match for this alias, with the invert it should match)
Source Port : any to any
Destination invert : ticked
Destination : LAN net
Destination port range : DNS to DNS
Redirect target IP : 10.x.x.x (ip of my piHole VM)
Redirect port : DNS
Pool Option : Default
Log : ticked (where are they ?)
NAT reflection : Disable
Filter Rule association : none
I'm testing it on a client using nslookup or dig directly to the IP 10.x.x.x of my piHole :
nslookup something.com 10.x.x.x.x
dig something.com @10.x.x.x
LAN is a bridge, my client is using WIFI, the wifiAP is plugged on the Switch, the switch has a LOADBALANCE LAGG to opnSense.
My piHole is a VM running on the same host tat is running opnSense and it's using a vtnet virtual interface.
Both the LAGG and the vtnet are in the LAN bridge.
And while doing this nslookup or dig, I am looking at the logs of piHole that says :
Oct 14 23:44:14 dnsmasq[4056]: query[A] something.com from 10.y.y.y
Oct 14 23:44:14 dnsmasq[4056]: forwarded something.com to 10.y.y.y
Oct 14 23:44:14 dnsmasq[4056]: reply something.com is <CNAME>
Sadly, 10.y.y.y is the IP of my opnSense box, and not the ip of the client that is performing the DNS query, in fact, on line 2 you can even see that piHole is setup to forward DNS queries to the opnSense box and that is exactly what it does.
It's not an issue with piHole : when I disable this NAT PortForward and query again from the same client to the same piHole, piHole has the proper IP of the client in its logs.
So for some reason, I'm doing UDP/53 to a destination that is in LAN net and yet the NAT rule applies despite it saying Destination = Lan_net and destination/invert ticked...
I'm out of ideas here, please help.
Edit : It's a very strange behaviour since disabling this rule does make the proper IP show up on piHole meaning the trafic can traverse the bridge properly, but with this rule enabled, the wrong IP is shown to piHole, but if the client makes the DNS query to some other IP in LAN_net where nothing is responding, it does not get any response, whereas if it does query any random public IP it does get an answer that is in fact coming from my piHole...
So it's probably not the NAT portForward alone that is doing this, maybe it's in combination with something else, but I don't know what...
Anyone able to help here ?
I really need to have no NAT on LAN since otherwise my reverse proxy will see both lan and wan client with the same IP of the router...
Thanks in advance
Please provide a graphical network plan.
is 10.x.x.x and 10.y.y.y the same subnet?
Quote from: toxic on October 17, 2020, 02:40:48 PM
Destination invert : ticked
Destination : LAN net
So if I understand that right your rule won't be applied if client and PiHole are in the same subnet. Maybe the client you´re testing with is asking your OPNsense and the OPNsense is then asking the PiHole? That would explain why you see the OPNsense IP.
Thanks for your kind help.
Indeed, 10.x.x.x and 10.y.y.y are both on the same subnet, actually 10.0.0.0/24 which happens to be the "LAN net".
On the client I'm testing with, I'm explicitely telling it to query the piHole directly, that the @10.0.0.y part in the dig command. That's what really baffeling me...
I can´t see why a network packet destined for the local network should be addressed by your OPNsense. Doesn´t make sense to me.
Please provide more info about your network, a graphical network plan with all relevant participants would be needed to find out why those packets are sent to the OPNsense instead of the piHole directly.
Thanks again for you kind answer, and sorry for the late response (did not have notify on for this thread somehow... fixed now ;))
I'll come around to making a graphical view asap, meanwhile, physically the signal has to go through opnSense even if it's all on a bridge.
For the client :
client ==> wifi AP ==> switch ==> LAGG(loadbalance igb0,igb1) ==> bridge0 wichi is assignet to my LAN interface on opnSense
The piHole is a VM on proxmox, proxmox is also the host that virtualizes my opnSense box passing it all 6 physical NIC (pci passthrough), and also attaching a virtual NIC as a 7th NIC.
So if you go to the other direction to the piHole : on opnSense bridge0 also contains a vtnet0 NIC which is a virtualNIC that proxmox gives to the opnSense VM, in proxmox this net0 virtual NIC is attached to a linux bridge called vmbr0, and when starting the piHome VM on proxmox, I also give it a virtual NIC attached to the same vmbr0 linux bridge.
Maybe of not here, at the beginning I was trying to use VLANs but I gave up (since it did not play nice with my LAGG to my switch and to my NA, and my Wifi AP refused to work on several VLANs one for each SSID...(got to learn more about VLANs still I believe)
But I believe it was for that purpose that I changed these 2 settings in tunables that are still like this :
net.link.bridge.pfil_member=0
net.link.bridge.pfil_bridge=1
Reading the description for these, it still makes sense to me to leave those like this (as I have some rules that will apply on some IP aliases only in my LAN subnet, like blocking my IP CCTV camera from going to internet...), but I wanted to mention it just in case.
And for the sake of completeness, on bridge0 in opnSense I have 1 other interface in the bridge, that is a LAGG LACP to my Synology NAS. No VLAN or VXLAN defined anywhere.
So opnSense will definitely have to thee the UDP packets on port 53 from my wifi client to the piHole, but as it is coming from LAGG LOADBALANCE and should be routed to vtnet0 NIC in the same bridge0, I don't understand why it's doing any kind of NAT on it. When I disable this NAT rule, the packets do get "routed" (actually switched) properly and the source IP is not touched by opnSense, only with this NAT rule enabled suddenly it changes...
Thanks in advance for your kind help,
Ok, why do you use bridges? From my current knowledge of your network I think it's way too complicated without a real need for it.
What is the LAGG for? Is your OPNsense able to route and firewall 2 Gbit/s?
You should really use VLANs if you want to separate networks. Of course you need hardware which is VLAN capable (switches and APs).
I'll wait for a graphical network plan to understand why you're doing it this way.
Hello, took me a while but I got the picture ;)
First a few answers :
QuoteOk, why do you use bridges? From my current knowledge of your network I think it's way too complicated without a real need for it.
Because I would like to use opnSense as a simple switch on several physical NICs but still be able to use LACP LAGG, and also enforce some intra-network rules to keep my smart-home things from going to internet or anywhere that's not there dedicated management server...
QuoteWhat is the LAGG for? Is your OPNsense able to route and firewall 2 Gbit/s?
I had troubles with the bandwidth to my NAS, not all client devices are shown on the picture, when backups and media streaming are in progress, bandwidth comes tight, my NAS and opnSence can do LACP, finally my switch can do LOADBALANCE LAGG, so I tested it and yes, throughput went to 90% of these 2GB/s on my NAS with several clients hitting it ;) nice ! and well above my prior limit of 1Gb/s !
QuoteYou should really use VLANs if you want to separate networks. Of course you need hardware which is VLAN capable (switches and APs).
Seems to be my issue as of now, not all HW is capable and it caused me some issues. Will try again after some upgrades (physical ;)) but essentially this does not affect my current issue, in the end, my client and my dns server will live in the same vlan, and there should be no NAT I believe.
Finally here is the picture attached to the post.
Remebmer the issue : with the NAT rule disabled, pihole (C) does see the proper IP of client (F) that is doing DNS request directly to the private IP of piHole (C).
When enabling the NAT rule, despite the source beeing setup as "not LAN Net", it seems to match and piHole still recieves the traffic but source IP of requester is then the IP of the opnSense router (B).
The host is quite powerfull (i5 8th gen) so it made sense for me to have several VMs runnign on it but not use physical NICs since I "only" have 6 of them. Plus having a FW between my NAS and the rest gives me peace of mind that traffic won't be "switched" directly to it without the chance of applying my FW rules. Also, as I said, the NAS can do LACP and only pfSense can do LACP for me in my devices...
Hope this helps you understand and you get an idea as to why this NAT rule still gets applied. Actually, I've looked at the pfStates and I get the feeling that other flows are NATed from LAN Net to LAN Net for some reasons, but let's focus on this UDP 53 DNS query since it's already identified...
Also, is there some user-frienly way to look at what is currently beeing NATed, I guess somewhere there's a table for that. pfStates shows originin parenthesis, but that's quite hard to read. What does the "log" checkbox do in the NAT rules ? I have ticed it but cannot find any place where it logs that the NAT rule has matched or not...
Just checked your opening post again.
And now it's clear to me. Because you're in the same network as the destination host OPNsense needs to put its own IP address as source because otherwise the pi-hole would send its answer directly to your client which asked let's say 1.1.1.1 to its gateway and needs to get an answer from that source, otherwise the packet will be discarded. If you put the piHole to its own VLAN you'll have no problem with that.
For VLANs not everything needs to understand VLAN. Just the switches. All devices without VLAN capability are marked on your switch to a static VLAN if packets are untagged and packets will leave untagged to that port just for one VLAN.
Thanks a lot for your reply !
I'm still not clear thought...
QuoteOPNsense needs to put its own IP address as source because otherwise the pi-hole would send its answer directly to your client which asked let's say 1.1.1.1 to its gateway and needs to get an answer from that source, otherwise the packet will be discarded
Since they are all on the same network, I don't see why opnSense would need to add it's IP, the response should indeed go directly to the client, and that is indeed what is happening properly when this NAT rule is disabled. Therefore, when this NAT rule says it should not match when source and destination are on LAN net, the NAT should not be applied and the query should arrive at piHole from the real client...
I must be missing something here...
I've tried many combinations, it seems the NAT rule will always match too widely, I've tried to use single host in the source or destination, unchecking the invert source or destination (adapting the IPs each time), I cannot get a situation where this NAT rule is enabled, wher it does spoof DNS trafic that should go out of my LAN by redirecting it to my piHole, WITHOUT touching the normal LAN trafic between LAN clients and the piHole... It always tries to NAT LAN to LAN and puts itself inbetween lient and piHole instead of letting trafic go through.
even when the sourceIP is the single IP of my client, and the destination is 8.8.8.8, no invert involved, simplest rule I could make. With that, dns query to 8.8.8.8 is properly NATed to piHole, but dns query to piHole directly also gets NATed to piHole... :'(
I was thinking of creating a FW rule to mark the LAN_net to piHole trafic and telle the NAT not to touch it but I believe that NAT is applied before FW rules...
I will try to apply this NAT outgoing rule to my WAN interfaces, we'll see if I can catch DNS traffic that is on it's way out only...
Example:
- Client F sends a DNS request to 1.1.1.1, this packet needs to be sent to the default gateway, which is your OPNsense.
- OPNsense then should rewrite the destination of this packet to your piHole which is in the same subnet as the client
- OPNsense then knows that it needs to rewrite the source to be able to send the answer back to Client F
- if OPNsense would rewrite the source (which would happen, if your piHole would be on another subnet) your piHole would send the answer directly to Client F, which doesn't know that it has sent a packet to the piHole. Client F would discard this packet because it doesn't know anything about this connection.
If Client F is asking the piHole directly traffic is not passing the OPNsense (at least not the packet filtet), source and destination know about their connection and talk happily with each other.
If the piHole or the client would be on another subnet that would work because the answer packet is running through the OPNsense as a gateway.
Creating a rule on the WAN interface won't solve that.
I understand that, thanks.
QuoteIf Client F is asking the piHole directly traffic is not passing the OPNsense (at least not the packet filtet), source and destination know about their connection and talk happily with each other.
That is what I expect, and actually what is happening when this NAT rule is disabled.
I had no luck with NAT Port Forward on the WAN interfaces as you predicted.
Now, let's start at how I setup things right now with my current non-uderstanding of the NAT PortForward.
Right now I hve disabled the NAT portForward for UDP/53.
DNS queries inside my LAN net between client and piHole directly are working fine and piHole sees the proper IP.
What I would like to add, is if the same client tries to use 1.1.1.1 as DNS server, then and only then, the router ould NAT this request to my piHole instead. For these packets, of course piHole would see sourceIP as the router. But when I manually query the piHole from a client on the same subnet, then the piHole would see the proper client IP as no NAT would be involved.
I'm at a loss on how to achieve that. I want to act on DNS queries only if they are tring to leave my network.
I have added a rule to match this DNS trafic, it works : I can block it, I can add a local tag to it, I can even set the gateway to a manually added LAN gateway (which as expected cause the Query to fail since noone on my LAN is responding to 1.1.1.1....). Allthewhile this rule does not match the client to piHole direct queries that pass properly on the bridge without beeing touched.
So my FW rule can match this trafic and only this, but I'm unaware on how to NAT only this trafic to my piHole.
Any idea ?
I thought it already works as you describe it.
piHole then just sees the wrong source ip. If this is a problem create a new vswitch in proxmox for the piHole and connect the piHole and the OPNsense to it. Then create a new subnet for this vswitch. Add some firewall rules to allow traffic to the piHole and allow the piHole to connect to the Internet.
Then update your NAT rule with the new IP of the piHole and you'll see the correct source IP i.e. from Client F.
Edit: just to be clear: this is nothing you can OPNsense blame for. It's just a networking "problem" you would have with every other solution.
Thanks again for your quick reply.
I get that it's most definitely my not beeing good enough at networking and not an issue of opnSense, and thanks for your kind networking support !
I get from your answer that my piHole should be on another subnet than the client, but I sadly don't understand why.
To correct you for a minute, it's not a problem for me that piHole is not seeing the correct IP of a client when the query in question has ben spoofed by opnSense. It is a problem for me only when the query is directly fom LAN clien to piHole.
The reason is that piHole has some statistics to show you which devices are using your piHole or not, and if it never sees the proper source IP it's useless. I also have the situation where the google-mini is using the DNS advertised by DHCP sometimes (that's my piHole in this case), and sometimes also uses google's own DNS. Now that is something I would have liked to fix by having this client always query piHole even not knowingly : sometimes directly and reported properly by piHole, and sometime spoofed where piHole responds instead of 8.8.8.8
I don't understand how having piHole on a separate subnet would help, sorry if I'm a little slow to understand :(
Isn't there a way for me to redirect trafic that only match my FW rule that I have setup to match the outgoing DNS queries only and not internal between clients and piHole directly ?
also, sorry, I think what works or not has not been made clear :
Works :
- client query piHole direcly and pihole reports proper client IP
- client query 1.1.1.1 and gets blocked
Does not work :
- client query 1.1.1.1 and piHole responds instead
In this last case that does not work, I really don't care much about what IP piHole sees, I'll find some log files somewhere to report on this trafic later finding out who is refusing to use the DNS provided by my DHCP ;) I'd just like to not break the working points :D.
Quote from: toxic on October 24, 2020, 04:45:30 PM
also, sorry, I think what works or not has not been made clear :
Works :
- client query piHole direcly and pihole reports proper client IP
- client query 1.1.1.1 and gets blocked
Does not work :
- client query 1.1.1.1 and piHole responds instead
In this last case that does not work, I really don't care much about what IP piHole sees, I'll find some log files somewhere to report on this trafic later finding out who is refusing to use the DNS provided by my DHCP ;) I'd just like to not break the working points :D.
Sorry if I'm not getting your point here. In your opening post you wrote, if you enable that rule, clients that try to connect to an outside DNS (like 1.1.1.1) are getting port forwarded to the piHole but the piHole sees the wrong source IP (OPNsense IP). Correct?
In your last post you wrote this is not working. If your client is trying to reach 1.1.1.1@853 (DoT) you cannot port forward this connection.
Client F (10.0.0.50) ----> 1.1.1.1@53 (DNS) ---> OPNsense (default gateway) --> rule says to rewrite that destination to 10.0.0.6 WITH source rewritten OPNsene IP
piHole gets the query and sends the packet back to OPNsense (source) and OPNsense sends the answer back to Client F
Without source rewrite:
Client F (10.0.0.50) ----> 1.1.1.1@53 (DNS) ---> OPNsense (default gateway) --> rule says to rewrite that destination to 10.0.0.6 WITHOUT source rewritten OPNsene IP
piHole gets the query and sends the packet directly to Client F (because that is the source piHole was told) --> Client F discards that packet, because Client F doesn't know anything about that query towards piHole
With piHole or client in a different subnet:
Client F (10.0.0.50) ----> 1.1.1.1@53 (DNS) ---> OPNsense (default gateway) --> rule says to rewrite that destination to 10.0.
1.6 source would not be rewritten
piHole gets the query sees the correct IP of Client F and sends the packet back to OPNsense (gateway) and OPNsense forwards the answer to Client F
Thanks a lot, i'll give that a try !
Not sure yet how to go around with 2 subnets 10.0.0.0/24 and 10.0.1.0/24, have LAN be on 10.0.0.0/16 or leave LAN on 10.0.0.0/24 and create another LAN2 for 10.0.1.0/24... Will try out a few things, starting simple with an extra virtual interface.
Sorry for all the going back and forth, I did try out lots of things and changed the status from my first post.
So while I'm trying out the new subnet, maybe you can still help me understand, because I'm still in the dark...
Let say current situation is with no special NAT rule enabled :
1/ - 10.0.0.50 queries DNS 10.0.0.6 directly : normal net routing, 10.0.0.6 sees the request from 10.0.0.50
2/- 10.0.0.50 queried DSN 1.1.1.1 directly : beeing NATed to my WAN interface, got the proper reply from CloudFlare
Now I'd like to understand why I cannot find a NAT rule that I would add and would allow : this :
3/ - 10.0.0.50 queries DNS 10.0.0.6 directly : normal net routing, 10.0.0.6 sees the request from 10.0.0.50
4/ - 10.0.0.50 queried DSN 1.1.1.1 directly , go through 10.0.0.1 that rewrites to 10.0.0.6 and NAT it putting itself as origin. 10.0.0.6 sees query from 10.0.0.1, reply, and opnSense re-NAT the reply to the client
Any single NAT rule I've tried has only allowed me to do half of it : either I do a NAT rule that never matches and 3/ keeps working as it was in 1/ but 4/ never works. And if I try harder and do a NAT rule that matches, it always matches "too widely" and I do get 4/ working, but 3/ is not working, instead I see this 5/ :
5/ - 10.0.0.50 queries DNS 10.0.0.6 directly : go through 10.0.0.1 anyway due to physical needs, and despite the NAT rule saying "source is NOT in LAN net" it still rewrites to 10.0.0.6 and NAT it putting itself as origin. 10.0.0.6 sees query from 10.0.0.1, reply, and opnSense re-NAT the reply to the client. So the client gets the answer, but the DNS has seen a NATed packet where there was no need to NAT it and actually the rule said it should not be a match :(
Still that baffles me... I will definitely give it a try with another subnet, but if I put anything else than my DNS on this subnet, the same issue will arise... For now I understand that it might simplify my setup and I will give a subnet to my DNS alone, maybe would like to use it for my servers in the future and avoiding the same issue again would be nice ;)
damn it ! you were right !
I still don't understand why my first attempt won't work, but with a second subnet, I have it working, and the piHole sees the source IP regardless of if it was spoofed or not, the NAT rule actually does not rewrite the source adress at all...
I mean, with my old NAT port forward rule and piHole on the same subnet as the client, when client asks pihole directly, I understand that if opnSense rewrites the target IP from 1.1.1.1 to 10.0.0.6 without NATing itself in the src address, it will probably not see the response come back, but it is realy not an issue for me, it should not put itself in the src...
So, first and foremost : thanks for showing me how it's done properly that is with different subnets. It pains me to have to dedicate one subnet to my DNS only sadly... Going through the troubles of creating yet another interface, subnet and the FW rules to go with it... I would have liked to isolate my servers from the clients as well. But if I put my servers on the same subnet as the DNS, I'll have the same issue if I want to spoof the DNS for my servers...
This turns into new questions for me now that I'm considering having different subnets... Is there a way for me to have several subnets on the same bridge ? Or can I combine things like my LAN10 that is "only" subnet 10.0.10.1/24 and have some other LAN interface that is on 10.0.0.1/16 to which I connect hosts that are on 10.0.0.0/24 for some of them and others on 10.0.10.0/24 or even another on 10.0.123.0/24 ?
Again, thanks for your kind help and patience in helping me try to understand the mess I made ;)
Quote from: toxic on October 25, 2020, 12:18:18 AM
This turns into new questions for me now that I'm considering having different subnets... Is there a way for me to have several subnets on the same bridge ? Or can I combine things like my LAN10 that is "only" subnet 10.0.10.1/24 and have some other LAN interface that is on 10.0.0.1/16 to which I connect hosts that are on 10.0.0.0/24 for some of them and others on 10.0.10.0/24 or even another on 10.0.123.0/24 ?
Again, thanks for your kind help and patience in helping me try to understand the mess I made ;)
Never include subnets in a wider subnet. Every interface has its own subnet, don't mix it. Traffic passing from one subnet to another will run through your OPNsense.