Hi guys,
due to playing with my new hardware and the logs, I found something suspicious and I cannot explain where it comes from.
Since 30. November ~6PM my OPNsense queries DNS servers, which are not DNS servers. I have the new modem setup since 29. November.
For example the FW is contacting 162.159.1.248 on port 53 originated from the WAN interface
Interface Setup:
Wan: PPPoe => Draytek Vigor 166
LAN => Server with VMs
WIFI => Connected to a WIFI Router with several Networks
DMZ => one connected server
What I did so far to track the problem down
1. Run Wireshark on the server. Not the server and not the VMs are issueing the DNS request to the mentioned IP. To test if everything is really logged, I did a icmp check and used dig with the IP address. This I could see in the dump and on the FW itself and as well on the log host
2. create a dump on the firewall itself on all interfaces. Once with promicious mode enabled and once disabled. I can see within the dump of the firewall only that the external IP is connecting to the IP on Port 53. There is no NAT no nothing. I could only see the initiated connection in the logs from my server, when I manually initiated a ping or the icmp.
3. Tests from the Wireless Network connection to the FW. Also here I could not see anything, beside I manually did a ping or dig command on the given ip. Surely then it ended up in the logs
I do not understand where the DNS qeuries are coming from. There are evene more random IPs which are contacted via DNS queries on port 53. I checked the FW settings, there is nothing configured what could cause this.
Could it be that it's the Modem which builds the internet connection by the firewall over PPPoe? If so, what would that mean? Malicious firmware? Actually as I am still testing which firmware fits best, I use Vigor166_v4.2.5_MDM3. Any ideas how I can reverse the firmware to see what's inside?
fastboot
*Bump*
Anyone?
Is there any cli tool (like lsof on linux system) to see which process is connecting to an IP?
you don't say where is OPN in the chain. Please add that and confirm what is the Draytek Vigor doing, if anything other than dialing the PPoE creds.
On a "normal" setup where OPN is the firewall and router between your WAN and LANs, the firewall live view you screenshoted is the one where you trace the incoming call to that DNS server. In short, it will give you the ip of the client initiating the connection: into the LAN is attached to, gets processed by rules i.e. allowed or denied, then out of WAN. Then response back is processed in reverse.
How is DNS configured on your system? Under General? Unbound? How should your modem request DNS from your OPNsense? What are these porkbun domains resolved? Anybody from your LAN? Any client with strange DNS settings? Which rules on LAN?
So many questions..
First off, the Draytek cannot do any DNS lookups unless you have configured it to do so. It is configured as a bridge, probably you did not even assign a virtual IP on your WAN port in order to be abled to access its web interface and even if you did, you most probably did not allow traffic from its IP to the internet.
Porkbun is a hoster who also sells domains, so that the DNS requests you see are most probably recursive queries for domains hosted there directed at their nameservers (https://kb.porkbun.com/article/63-how-to-switch-to-porkbuns-nameservers). I would argue that some program on your PC or IoT device tries to phone home to somewhere and in the process, it resolves domains.
This could be malware activity, but is does not have to be. I would look at the content of the queries and go from there.
Quote from: cookiemonster on December 08, 2023, 03:52:02 PM
you don't say where is OPN in the chain. Please add that and confirm what is the Draytek Vigor doing, if anything other than dialing the PPoE creds.
On a "normal" setup where OPN is the firewall and router between your WAN and LANs, the firewall live view you screenshoted is the one where you trace the incoming call to that DNS server. In short, it will give you the ip of the client initiating the connection: into the LAN is attached to, gets processed by rules i.e. allowed or denied, then out of WAN. Then response back is processed in reverse.
Hi cookiemonster,
I thought this is obvious. But sure, let me explain.
DSL TAE -> Vigor 166 -> FW
As you can see in my previous post, the connection is initiated by the Firewall over the WAN interface, which is directly connected to the modem. VLAN for the PPPoE also configured on the FW and not on the modem.
The rest I am pretty much aware of. How the logging works etc... the logs will be transfered into my SIEM. This is how I realized this suspicious behavior. I did not have that with the old Router Setup some days ago.
Thanks for the help :)
Quote from: chemlud on December 08, 2023, 04:05:31 PM
How is DNS configured on your system? Under General? Unbound? How should your modem request DNS from your OPNsense? What are these porkbun domains resolved? Anybody from your LAN? Any client with strange DNS settings? Which rules on LAN?
So many questions..
Hi.
System - Settings - General (6 IPv4 servers, 2 IPv6 DNS Servers) Unbound not configured.
I do not know where this porkbun comes from. Not from me. I have a very detailed overview about any ip communication within my setup. So I really don't know... unfortunately :(
The Server on the LAN interface, is 100% not creating this requests. dumped the traffic, analyzed it... nothing. And If it would be the origin, I would see that on the FW itself, in the logs and as well in the dumps i created. Same for the Wifi and DMZ Networks.
Like said. The connection to this porkbun stuff is initiated over the PPPoE interface(WAN), nowhere else.
Quote from: fastboot on December 08, 2023, 05:56:36 PM
Like said. The connection to this porkbun stuff is initiated over the PPPoE interface(WAN), nowhere else.
You may be right on the DNS part, but I bet not so much about who causes this. It ssem obvious to me that some internal client wants to contact some domain that is hosted by Porkbun (maybe HTTP traffic) and your OpnSense DNS queries that name recursively - thus you see these DNS queries as coming from your WAN.
Quote from: meyergru on December 08, 2023, 05:10:06 PM
First off, the Draytek cannot do any DNS lookups unless you have configured it to do so. It is configured as a bridge, probably you did not even assign a virtual IP on your WAN port in order to be abled to access its web interface and even if you did, you most probably did not allow traffic from its IP to the internet.
Porkbun is a hoster who also sells domains, so that the DNS requests you see are most probably recursive queries for domains hosted there directed at their nameservers (https://kb.porkbun.com/article/63-how-to-switch-to-porkbuns-nameservers). I would argue that some program on your PC or IoT device tries to phone home to somewhere and in the process, it resolves domains.
This could be malware activity, but is does not have to be. I would look at the content of the queries and go from there.
Good points. PPPoE runs on L2. Indeed I did not configure an additional vlan for the mgmt of the vigor. Actually I deny and priv ranges on the WAN interface. Standard DNS in the config is 8.8.8.8. I did not change that. So the modem should be only used what I intended it do. Maybe this is hardcoded into the Modem? Then it will be encapsulated, goes over the FW and back to the internet? I don't know. Tried already to reverse the firmware.
Very interesting is, that the DNS Server 162.159.1.248 does not reply on any random requests (for instance dig @162.159.1.248 google.com) but it does for maceio.ns.porkbun.com and the others you can see in the screenshot
But your theory, that it's another device somewhere in my network I do not follow. As I would have seen that within the dumps I created on the NICs. It's simple L3 and I should see that communication. It also does not explain why I can see this only on the FW itself and only on the PPPoE interface.
It is clear that those nameservers do not answer random requests. They will answer only for domains they are authoritative for.
About my theory: Nope. See my last post. Look at the DNS query content and you will probably see what is being asked for. Maybe you even use a (transparent) proxy on your OpnSense. Either way, you would not see contacts to those nameservers from your clients.
Quote from: meyergru on December 08, 2023, 06:07:53 PM
Nope. See my last post.
Yup. As I disabled any other interface on the FW already. The logs of the connections will be still generated. And like that, its just L3 traffic. Running wireshark, tshark and tcpdump should reveal that. But it does not. I even intercept the ssl traffic.
I am really no UNIX expert. Is there anything what I could do in the shell to see which process (PID) is creating this requests? With a Linux System it would be easy, but here I am a little bit lost. Ideas? :)
All I am saying is that it is virtually impossible that this traffic comes from the Draytek.
You say that you intercept the SSL traffic. Isn't that what I wrote? If you use a transparent proxy, the DNS queries for the domains originate at your OpnSense, so you will not see them on other interfaces.
Why don't you inspect the DNS queries directed at the porkbun NS servers? Only this can shed light on what is being asked for and give a hint as to what is causing this. Just dump the traffic and use Wireshark to inspect it (or use tcpdump directly).
It could be traffic form the OpnSense itself, e.g. if you use plugins like Zenarmor. Or the Meraki switch.
Quote from: meyergru on December 08, 2023, 06:18:29 PM
All I am saying is that it is viertually impossible that this traffic comes from the Draytek.
You say that you intercept the SSL traffic. Isn't that what I wrote? If you use a transparent proxy, the DNS queries for the domains originate at your OpnSense, so you will not see them on other interfaces.
Why don't you inspect the DNS queries directed at the porkbun NS servers? Only this can shed light on what is being asked for and give a hint as to what is causing this.
uhm? First you say it cannot come from the Modem, now it could? Wasn't it you? Did you check the screenshot? This comes directly from Wireshark. No encryption. Cleartext. You can even see what the DNS is queried for. But no, intercepting ssl traffic has nothing to do with that. It was just meant that I take this serious. Not because I need that, it's just I like things like that.
Actually I have no network tap here to investigate further.
So surely It would be easier with board tools within the FW. Ideas?
I don't quite get the Vigor function but doesn't seem to matter.
This is how I do it: When you do your packet capture on OPN, select all interfaces in use, and set promiscuous mode. A file for each will be created. Then I use the WAN file first for the client IP. Then I go to the file of the interface with that network segment.
I read your first post again and I see you've done this BUT if the connection is visible on the WAN, it MUST have originated on the LAN or the system itself if it is a)hosting some application code that could be the culprit or b) it was compromised somehow.
In ALL cases, the packets captured will tell the origin, if you can't see it in your LAN captures, there's something not set correctly. VLANs by any chance ?
The Draytek Vigor is just a DSL NT bridge to convert (V)DSL to RJ45.
@fastboot: No, it cannot come from the Draytek, lookup what "virtually impossible" means.
I fail to see where you have shown the query content. All I see in the screenshots are queries directed at porkbun nameservers, but not what is actually being asked for. And you can use tcpdump on the OpnSense itself to capture that.
@fastboot
1. can you confirm if the V166 has been configured in Modem/Bridge Mode or in Router Mode?
2. the V166 (like all the Draytek modem/routers) have Google DNS IP addresses hardcoded (I'm afraid), but in bridge mode it doesn't make any difference as the appliance is a 'dumb' modem, so they are not being used.
Quote from: cookiemonster on December 08, 2023, 06:31:19 PM
I don't quite get the Vigor function but doesn't seem to matter.
This is how I do it: When you do your packet capture on OPN, select all interfaces in use, and set promiscuous mode. A file for each will be created. Then I use the WAN file first for the client IP. Then I go to the file of the interface with that network segment.
I read your first post again and I see you've done this BUT if the connection is visible on the WAN, it MUST have originated on the LAN or the system itself if it is a)hosting some application code that could be the culprit or b) it was compromised somehow.
In ALL cases, the packets captured will tell the origin, if you can't see it in your LAN captures, there's something not set correctly. VLANs by any chance ?
Hi Cookiemonster,
that was literally what I was saying. There must be an origin. But with the packet captures within my whole setup, I could not see anything to that destination ip, but the PPPoE interface. So my assumption is, that it comes from the modem itself.
"In ALL cases, the packets captured will tell the origin, if you can't see it in your LAN captures, there's something not set correctly. "
Indeed. That's why I've opened this Thread, as I cannot explain where it comes from. Because the origin within the dumps, and just on the firewall, is the firewall itself on interface PPPoE (WAN). Nowhere else.
Quote from: hushcoden on December 08, 2023, 07:29:25 PM
@fastboot
1. can you confirm if the V166 has been configured in Modem/Bridge Mode or in Router Mode?
2. the V166 (like all the Draytek modem/routers) have Google DNS IP addresses hardcoded (I'm afraid), but in bridge mode it doesn't make any difference as the appliance is a 'dumb' modem, so they are not being used.
1. can you confirm if the V166 has been configured in Modem/Bridge Mode or in Router Mode?
=> Yes, I do confirm
2. the V166 (like all the Draytek modem/routers) have Google DNS IP addresses hardcoded (I'm afraid), but in bridge mode it doesn't make any difference as the appliance is a 'dumb' modem, so they are not being used.
=> That's correct. But so far no queries to 8.8.8.8
But I did some changes here to test even more.
What I did
1. Remove all DNS Servers (IPv4+6) from System - Settings - General
2. Enable unbound DNS, configure several DNS Servers for IPv4+6 under DNS over TLS and hardened the settings.
Result so far: The DNS requests are silent. The IP I mentioned in the previous posts is not called again.
Also interesting is, that I even found way more DNS queries from the FW itself to random DNS Servers, which are nowhere configured.
So that the queries stopped now due to the changes and that I found even more DNS queries in the logs to random servers, raises way more questions to myself.
I will check to get a network tap next week, to put it between FW and Modem. Furthermore I've asked someone of our RedTeam to help me to reverse the firmware of the Draytek. When I started to use the Draytek, it was delivered with an older firmware. Due to a open CVE I updated and then the mess started. But let's see.
Still... If anyone could tell me which tool I can use on the shell to see which process calls an IP, I would be very gratefull. (like lsof on linux?) netstat does not work that well for this.
Have you made any progress in your analysis?
I stumbled across this topic as I have a similar setup and behaviour.
I have a DrayTek Vigor 165 with firmware 4.1.1_STD set up in bridge mode.
In my FW logs I see many requests on the WAN interface to (unknown to me) IP's on port 53 (attached screenshot).
I wonder a) where these requests are coming from and b) what strange DNS (?) servers these are.