Getting a wonderful DDoS on one circuit daily.. was able to get a tcpdump and see that it's a DNS amplification to a PPTP port..
Guess this block of IPs is a problem.. anyway..
DDoS starts when the errors start..
/usr/bin/netstat -i -b -n -I igc1 1
input igc2 output
packets errs idrops bytes packets errs bytes colls
18832 0 0 23961931 19142 0 2056571 0
20732 0 0 26692436 21197 0 2246518 0
18420 0 0 23084666 18980 0 1939937 0
17791 0 0 22109171 18277 0 2206791 0
15123 0 0 18495639 15479 0 1704420 0
20523 0 0 25776829 20787 0 3209901 0
9622 0 0 10348462 9872 0 1743120 0
12389 0 0 14020727 12975 0 2070747 0
10903 0 0 12418180 11251 0 1902214 0
16921 0 0 21012711 17189 0 2412258 0
17918 0 0 22356183 18418 0 2024011 0
20811 0 0 26971250 21166 0 2231476 0
21643 0 0 27830204 21867 0 2555701 0
12362 0 0 14710851 12806 0 1664004 0
6961 0 0 6636610 7218 0 1299675 0
12274 0 0 14071152 12703 0 1757263 0
30882 1 0 39894920 8935 0 3341910 0
46458 0 0 62128264 4893 0 3045759 0
46478 1 0 62121562 4900 0 2750044 0
46438 1 0 62133530 5133 0 2723948 0
46599 1 0 62123711 5206 0 2504516 0
input igc2 output
packets errs idrops bytes packets errs bytes colls
46493 0 0 62124955 5224 0 2187363 0
46617 1 0 62129957 5418 0 2452033 0
46596 1 0 62132498 5122 0 2913734 0
46650 1 0 62113622 5109 0 2384515 0
46714 1 0 62127545 5357 0 2296607 0
46584 3 0 62126785 5081 0 2390958 0
46612 0 0 62118032 5144 0 2270248 0
46739 1 0 62130613 5681 0 3326115 0
46514 0 0 62125384 5159 0 2275629 0
46653 0 0 62130561 5433 0 2302723 0
46600 0 0 62121725 5158 0 2322515 0
46792 1 0 62120351 5220 0 2617294 0
46811 0 0 62124396 5280 0 2484596 0
46597 0 0 62126038 5131 0 2381244 0
46725 1 0 62127029 5429 0 2406458 0
46595 0 0 62125301 5437 0 2592203 0
46661 3 0 62125865 5419 0 2400710 0
46650 1 0 62124639 5248 0 2519844 0
46610 0 0 62122192 5254 0 2346030 0
46676 0 0 62130274 5459 0 2687552 0
46898 0 0 62119543 5806 0 3204420 0
.. etc
Approximately 3 minutes worth..
Quote
08:36:35.176685 IP 42.62.176.70 > 111.222.333.444: ip-proto-17
08:36:35.176688 IP 42.62.176.70.53 > 111.222.333.444.1723: 1| 33/0/0 RRSIG, RRSIG, RRSIG, RRSIG, RRSIG, TXT "google-site-verification=yuAuTV0V218aUY-z4yyaeBY0B-icA3PcEFNCd72ZKk4", TXT "apple-domain-verification=ivyxTJSvycL1rKet", TXT "v=spf1 a mx include:spfa.renault.com include:spfb.renault.com include:spfc.renault.com include:spfd.renault.com exists:%{i}.spf.hc1506-8.eu.iphmx.com -all", TXT "3nnqAmrH2geG0012FzpfPzCbY+qeghGXlr0K+LYPlNZ04rbgRysxD+XwBO/kYhyrhm+O6pU0naULPJY0gHPjRQ==", TXT "zoho-verification=zb90149015.zmverify.zoho.com", TXT "mongodb-site-verification=hWhMU7S6paGXSMiTRzdhFYFc0NckzLdF", RRSIG[|domain]
08:36:35.176689 IP 36.91.138.130.53 > 111.222.333.444.1723: 1 6/13/1 RRSIG, RRSIG, RRSIG, RRSIG, RRSIG, RRSIG (1472)
08:36:35.176707 IP 111.222.333.444 > 36.91.138.130: ICMP 111.222.333.444 udp port 1723 unreachable, length 576
08:36:35.176715 IP 111.70.2.171.53 > 111.222.333.444.1723: 1 13/2/0 RRSIG, MX mx1.hc1506-8.eu.iphmx.com. 10, MX smtp2.renault.fr. 30, MX smtp.renault.fr. 20, MX mx2.hc1506-8.eu.iphmx.com. 10, RRSIG, RRSIG, SOA, RRSIG, RRSIG, NS anna.renault.fr., RRSIG, NS xenia.renault.fr. (1304)
08:36:35.176726 IP 111.222.333.444 > 111.70.2.171: ICMP 111.222.333.444 udp port 1723 unreachable, length 576
08:36:35.176749 IP 180.190.200.192.53 > 111.222.333.444.1723: 1| 32/0/0 DNSKEY, RRSIG, RRSIG, RRSIG, TXT "mongodb-site-verification=hWhMU7S6paGXSMiTRzdhFYFc0NckzLdF", TXT "3nnqAmrH2geG0012FzpfPzCbY+qeghGXlr0K+LYPlNZ04rbgRysxD+XwBO/kYhyrhm+O6pU0naULPJY0gHPjRQ==", TXT "2ml7l54tncj0sfz85z19bhy6kmbvhf40", TXT "onetrust-domain-verification=fc8a2586b8b247a28c56053c67dcd418", RRSIG, RRSIG, RRSIG[|domain]
08:36:35.176804 IP 178.205.90.201 > 111.222.333.444: ICMP 178.205.90.201 udp port 53 unreachable, length 65
08:36:35.176810 IP 189.3.74.18.53 > 111.222.333.444.1723: 1| 32/0/0 TXT "mt-24773710", TXT "docusign=c3a18a16-788c-484b-968b-6b4982433a67", TXT "amazonses:uINC55vCnY508CUO8Je4gL6XWtPX3btBCtcQjz2Vwjs=", TXT "3nnqAmrH2geG0012FzpfPzCbY+qeghGXlr0K+LYPlNZ04rbgRysxD+XwBO/kYhyrhm+O6pU0naULPJY0gHPjRQ==", TXT "facebook-domain-verification=8s50q3dhwvfs01uvnrwm8h29rpcntw", TXT "4l1SWsiprbXNsfRUEAfWklXtaSbfXsRaotj7HOf01kNe5wyIUw6dDiBNfAUjk8M/Dj9Gc8PzowuISHPOgAW83w==", TXT "tmes=281fb1a4ecc0f16f779e7a637e2df968", TXT "zoho-verification=zb90149015.zmverify.zoho.com", TXT "apple-domain-verification=ivyxTJSvycL1rKet", TXT "autodesk-domain-verification=4zOZypex_sR1HLFsXs7E", TXT "onetrust-domain-verification=811456c061094fd787edfbea1f50e0c2", TXT "google-site-verification=yuAuTV0V218aUY-z4yyaeBY0B-icA3PcEFNCd72ZKk4", TXT "apple-domain-verification=71mEATCbpJsvgxSj", RRSIG, RRSIG, RRSIG[|domain]
08:36:35.176814 IP 201.184.117.60.53 > 111.222.333.444.1723: 1| 41/0/0 SOA, RRSIG, RRSIG, RRSIG, RRSIG, RRSIG, RRSIG, RRSIG, RRSIG, RRSIG[|domain]
08:36:35.176818 IP 201.184.117.60 > 111.222.333.444: ip-proto-17
08:36:35.176819 IP 197.91.174.102.53 > 111.222.333.444.1723: 1| 37/0/0 RRSIG, RRSIG, MX smtp2.renault.fr. 30, MX smtp.renault.fr. 20, SOA, DS, DNSKEY, RRSIG, RRSIG, A 35.71.164.53, A 52.223.12.199, RRSIG, RRSIG, RRSIG[|domain]
What are the interface errors?
Why are there interface errors?
Quote
dmesg | grep igc2
igc2: <Intel(R) Ethernet Controller I225-V> mem 0x7fa00000-0x7fafffff,0x7fc00000-0x7fc03fff at device 0.0 on pci3
igc2: Using 1024 TX descriptors and 1024 RX descriptors
igc2: Using 4 RX queues 4 TX queues
igc2: Using MSI-X interrupts with 5 vectors
igc2: Ethernet address: 64:62:66:22:01:b1
igc2: netmap queues/slots: TX 4/1024, RX 4/1024
in the "tcpdump -i igc2 -n" output (as far as I can tell..) I was able to capture everything..
Thanks in advance..
Not sure what are you planning. Other that "do not expose public DNS on your WAN" you cannot prevent DDoS at your end. Move to your ISP.
Thank you for the response..
I am not running/allow dns..
they are sending the responses back to me.. hence the DDoS..
I am the 111.222.333.444 address.. and I'm not running a pptp server either..
I'm assuming you also have no idea what the hardware errors would be..
Which HW errors? So you are getting single errors (error is e.g. a malformed packet or out of buffer space) under DDoS? Sounds perfectly expected to me.
Once again, only your ISP (and their peer) can do something with the DDoS. Perimeter firewall is not the place to deal with those.
Looks like the sources are indeed open resolvers:
dig +short @36.91.138.130 www.google.com
64.233.170.147
64.233.170.104
64.233.170.99
64.233.170.103
64.233.170.106
64.233.170.105
dig +short @201.184.117.60 www.google.com
142.250.78.164
dig +short @111.70.2.171 www.google.com
142.251.42.228
It does, however, show that you're also sending ICMP port unreachable back:
08:36:35.176707 IP 111.222.333.444 > 36.91.138.130: ICMP 111.222.333.444 udp port 1723 unreachable, length 576You haven't created any 'reject' rather than 'drop' (block) rules to try and block this by any chance? As a reject rule for UDP would send a destination unreachable ICMP response.
Or have 1723 port forwarded? Or a DMZ?
Aside from the above, I would have hoped/assumed, that a stateful firewall - albeit UDP - would at least realise there wasn't a recent egress connection to that destination/port and so the response is invalid and should be dropped without an ICMP response. Although I'd have to test myself.
If it is valid behaviour, although I'm not convinced, then you could try to limit outbound ICMP responses to the destination ASNs, geo-ip countries, etc without disabling ICMP as a whole. Or, drop all ICMP destination unreachable replies.
If you're on a dynamic IP, or recently acquired the IP/block on your end, it's possible that you aren't/weren't the actual target - but this is just 'carry over'.
In terms of blocking the DNS replies, that are presumably replies to spoofed requests from your source IP:
- If the WAN IP is dynamic, change the IP
- If the WAN IP is static, ask for a new IP/subnet
- If it's always to UDP 1723 (which isn't the PPTP port, TCP 1723 is), ask the ISP if they can filter this port upstream
- If it's not using up much bandwidth in the grand scheme of things (i.e total bandwidth), try to work out why it's sending ICMP replies (to stop them) then move on with life...
From your netstat, every 1 second, 23961931 bytes is 20MB - then at peak 62122192 bytes, 60MB? Although I assume this is ALL traffic, assuming the circuit is otherwise loaded with traffic, not just DNS replies?
Could possibly be something like this:
https://blog.cloudflare.com/sad-dns-explained/
Quote"An attacker wants to know whether the target has an open port, so it sends a spoofed UDP message from the authoritative server to that port. If the port is open, no ICMP reply is sent and the rate counter remains unchanged. If the port is inaccessible, then an ICMP reply is sent (back to the authoritative server, not to the attacker) and the rate is increased by one. Although the attacker doesn't see the ICMP response, it has influenced the counter. The counter itself isn't known outside the server, but whether it has hit the rate limit or not can be measured by any outside observer by sending a UDP packet and waiting for a reply. If an ICMP "port unreachable" reply comes back, the rate limit hasn't been reached. No reply means the rate limit has been met. This leaks one bit of information about the counter to the outside observer, which in the end is enough to reveal the supposedly secret information (whether the spoofed request got through or not)."
@imx
Thank you for the response..
No incoming services.. never was; and yes this is a new block to me.. but obviously used. So no 1723 incoming..
Provider showing 1.5Gb to 2Gb coming to them (at peak) but I'm a 500 symmetrical, not sure how they are decided which part of the 1.5 to send me..
As this has been going on a week plus, I've tried various things, dropping, rejecting.. What I changed yesterday actually seemed to change today's event..
I was dropping src 53 tcp/udp, dst 1723 tcp and all that did was seem to get the traffic 'resent' also setting reject (rst) instead of drop also kept them coming..
But the big change was not allowing icmp to the range except from the provider and a few other hosts..
Possibly a ping sweep to see if things are still alive.. not sure
Maybe tomorrow will be different.. but today we didn't get anything.
Currently rejecting icmp and rejecting those services.. and waiting for tomorrow.
But yes those 60MB was all DNS when it would happen no outbound was possible and the interface would just start registering 'errors' which aren't logged within opnsense anywhere that I could find.. so going to netstat was the best way that I could see the 'system' errors and rule them out as 'application' errors..
dmesg would show:
[zone: pf frag entries] PF frag entries limit reached
pfctl -sa
LIMITS:
states hard limit 1621000
src-nodes hard limit 1621000
frags hard limit 204800
table-entries hard limit 1000000
pfctl -sa | grep ESTAB | wc -l
20285
so currently some 20k states on this device.
No current way to edit tcp timeouts in pf on opnsense.. but for the incoming lan side where the proxy is I've reduced the tcp timeout.. never could find what the frags where/are..
again, thanks for the response.
Quote from: bcookatpcsd on December 20, 2023, 07:37:30 PM
and yes this is a new block to me.. but obviously used. So no 1723 incoming..
If it were me, I'd: "YO, ISP! GO FIX!" :)
Fragments: Firewall -> Settings -> Advanced -> Firewall Maximum Fragments
.. although none of this really fixes the problem, you might just make it tolerable.
As it's a new block, I'd be firing this data at your ISP, telling them they need to either filter/scrub, or provide a new block as this one should be rested for a bit until the attacks subside
Hopeless apparently. Said that there's nothing to fix at perimeter FW about 5 times already. And there we are with absurd suggestions such as blocking ICMP.
Sigh.
Quote from: doktornotor on December 20, 2023, 07:52:15 PM
Hopeless apparently. Said that there's nothing to fix at perimeter FW about 5 times already. And there we are with absurd suggestions such as blocking ICMP.
Sigh.
absurd (adjective) impossible to take seriously
Quote from: doktornotor
Which HW errors? So you are getting single errors (error is e.g. a malformed packet or out of buffer space) under DDoS? Sounds perfectly expected to me.
Does it matter if it's maformed *OR* out of buffer space? Are those the same? Would they appear the same? Where would that be?
Please generate some traffic, send them at whatever speed you would like, across the Interweb, not just local.. pass them through a bunch of devices and different carrier equipment.. and then malform 'one' (or any number) you would like, so that they reach their destination.. not all off them, just a few.
Have the last mile carrier limit the traffic sent out of the (unknown number 200k/pps) they are going to send you (known 50k/pps). Send them to a 2.5Gb capable nic with whatever the buffer space is (another unknown number), and then turn off or on hardware CRC, and turn off or on hardware TSO, and then then off or on hardware LRO.
Don't worry about putting any of that into an equation to try and solve for all the unknown variables, don't even factor in any OS settings.. or any of the equipment along the route, or any people and their education or network knowledge, or anyone and their ability to write drivers, etc, any of that.. Don't take any of that into account..
Another variable is time.. Companies pay people to provision links, people want to sell you DDoS services, everyone wants to sh*t on open source solutions and just assume you missed a check box that says, ".. really I'm not lazy."
For the computer science and math teachers .. is your answer reasonable?
"Sounds perfectly expected to me."
I hope someday I can solve all all of the above without having to show work or ask any questions.
Asking questions, shows your interested; my 0.02..
Unfortunately the CCNP questions I had were nothing like that; but that was a while ago.. haven't seen any in the 401 refresh.. maybe when I get to the 410.. I don't know anyone who has their CCDE.. maybe that's where you get those questions.. and learn how to answer them; still not sure how FreeBSD and OpnSense would factor in though, let alone local hardware decisions.. never mind any of that..
But apparently there still is hope, or is it "Hopeless apparently"
The absurd action of rejecting icmp sweeps.. and causing a change in my local situation; inconceivable
That word.. I do not think it means what you think it means..
https://imgur.com/a/EVXP9zK
Quote from: bcookatpcsd on December 21, 2023, 05:03:42 PM
Does it matter if it's maformed *OR* out of buffer space? Are those the same? Would they appear the same? Where would that be?
No, it does not, they are not, and yes - they would still appear in the same "errors" column you posted. And you would still not solve anything whatsoever on your end. Blocking ICMP or not.