DDoS on External.. errors on line?

[bcookatpcsd]

Getting a wonderful DDoS on one circuit daily.. was able to get a tcpdump and see that it's a DNS amplification to a PPTP port..

Guess this block of IPs is a problem.. anyway..


DDoS starts when the errors start..

Code Select Expand

/usr/bin/netstat -i -b -n -I igc1 1

           input           igc2           output
   packets  errs idrops      bytes    packets  errs      bytes colls
     18832     0     0   23961931      19142     0    2056571     0
     20732     0     0   26692436      21197     0    2246518     0
     18420     0     0   23084666      18980     0    1939937     0
     17791     0     0   22109171      18277     0    2206791     0
     15123     0     0   18495639      15479     0    1704420     0
     20523     0     0   25776829      20787     0    3209901     0
      9622     0     0   10348462       9872     0    1743120     0
     12389     0     0   14020727      12975     0    2070747     0
     10903     0     0   12418180      11251     0    1902214     0
     16921     0     0   21012711      17189     0    2412258     0
     17918     0     0   22356183      18418     0    2024011     0
     20811     0     0   26971250      21166     0    2231476     0
     21643     0     0   27830204      21867     0    2555701     0
     12362     0     0   14710851      12806     0    1664004     0
      6961     0     0    6636610       7218     0    1299675     0
     12274     0     0   14071152      12703     0    1757263     0
     30882     1     0   39894920       8935     0    3341910     0
     46458     0     0   62128264       4893     0    3045759     0
     46478     1     0   62121562       4900     0    2750044     0
     46438     1     0   62133530       5133     0    2723948     0
     46599     1     0   62123711       5206     0    2504516     0
            input           igc2           output
   packets  errs idrops      bytes    packets  errs      bytes colls
     46493     0     0   62124955       5224     0    2187363     0
     46617     1     0   62129957       5418     0    2452033     0
     46596     1     0   62132498       5122     0    2913734     0
     46650     1     0   62113622       5109     0    2384515     0
     46714     1     0   62127545       5357     0    2296607     0
     46584     3     0   62126785       5081     0    2390958     0
     46612     0     0   62118032       5144     0    2270248     0
     46739     1     0   62130613       5681     0    3326115     0
     46514     0     0   62125384       5159     0    2275629     0
     46653     0     0   62130561       5433     0    2302723     0
     46600     0     0   62121725       5158     0    2322515     0
     46792     1     0   62120351       5220     0    2617294     0
     46811     0     0   62124396       5280     0    2484596     0
     46597     0     0   62126038       5131     0    2381244     0
     46725     1     0   62127029       5429     0    2406458     0
     46595     0     0   62125301       5437     0    2592203     0
     46661     3     0   62125865       5419     0    2400710     0
     46650     1     0   62124639       5248     0    2519844     0
     46610     0     0   62122192       5254     0    2346030     0
     46676     0     0   62130274       5459     0    2687552     0
     46898     0     0   62119543       5806     0    3204420     0
.. etc


Approximately 3 minutes worth..

08:36:35.176685 IP 42.62.176.70 > 111.222.333.444: ip-proto-17
08:36:35.176688 IP 42.62.176.70.53 > 111.222.333.444.1723: 1| 33/0/0 RRSIG, RRSIG, RRSIG, RRSIG, RRSIG, TXT "google-site-verification=yuAuTV0V218aUY-z4yyaeBY0B-icA3PcEFNCd72ZKk4", TXT "apple-domain-verification=ivyxTJSvycL1rKet", TXT "v=spf1 a mx include:spfa.renault.com include:spfb.renault.com include:spfc.renault.com include:spfd.renault.com exists:%{i}.spf.hc1506-8.eu.iphmx.com -all", TXT "3nnqAmrH2geG0012FzpfPzCbY+qeghGXlr0K+LYPlNZ04rbgRysxD+XwBO/kYhyrhm+O6pU0naULPJY0gHPjRQ==", TXT "zoho-verification=zb90149015.zmverify.zoho.com", TXT "mongodb-site-verification=hWhMU7S6paGXSMiTRzdhFYFc0NckzLdF", RRSIG[|domain]
08:36:35.176689 IP 36.91.138.130.53 > 111.222.333.444.1723: 1 6/13/1 RRSIG, RRSIG, RRSIG, RRSIG, RRSIG, RRSIG (1472)
08:36:35.176707 IP 111.222.333.444 > 36.91.138.130: ICMP 111.222.333.444 udp port 1723 unreachable, length 576
08:36:35.176715 IP 111.70.2.171.53 > 111.222.333.444.1723: 1 13/2/0 RRSIG, MX mx1.hc1506-8.eu.iphmx.com. 10, MX smtp2.renault.fr. 30, MX smtp.renault.fr. 20, MX mx2.hc1506-8.eu.iphmx.com. 10, RRSIG, RRSIG, SOA, RRSIG, RRSIG, NS anna.renault.fr., RRSIG, NS xenia.renault.fr. (1304)
08:36:35.176726 IP 111.222.333.444 > 111.70.2.171: ICMP 111.222.333.444 udp port 1723 unreachable, length 576
08:36:35.176749 IP 180.190.200.192.53 > 111.222.333.444.1723: 1| 32/0/0 DNSKEY, RRSIG, RRSIG, RRSIG, TXT "mongodb-site-verification=hWhMU7S6paGXSMiTRzdhFYFc0NckzLdF", TXT "3nnqAmrH2geG0012FzpfPzCbY+qeghGXlr0K+LYPlNZ04rbgRysxD+XwBO/kYhyrhm+O6pU0naULPJY0gHPjRQ==", TXT "2ml7l54tncj0sfz85z19bhy6kmbvhf40", TXT "onetrust-domain-verification=fc8a2586b8b247a28c56053c67dcd418", RRSIG, RRSIG, RRSIG[|domain]
08:36:35.176804 IP 178.205.90.201 > 111.222.333.444: ICMP 178.205.90.201 udp port 53 unreachable, length 65
08:36:35.176810 IP 189.3.74.18.53 > 111.222.333.444.1723: 1| 32/0/0 TXT "mt-24773710", TXT "docusign=c3a18a16-788c-484b-968b-6b4982433a67", TXT "amazonses:uINC55vCnY508CUO8Je4gL6XWtPX3btBCtcQjz2Vwjs=", TXT "3nnqAmrH2geG0012FzpfPzCbY+qeghGXlr0K+LYPlNZ04rbgRysxD+XwBO/kYhyrhm+O6pU0naULPJY0gHPjRQ==", TXT "facebook-domain-verification=8s50q3dhwvfs01uvnrwm8h29rpcntw", TXT "4l1SWsiprbXNsfRUEAfWklXtaSbfXsRaotj7HOf01kNe5wyIUw6dDiBNfAUjk8M/Dj9Gc8PzowuISHPOgAW83w==", TXT "tmes=281fb1a4ecc0f16f779e7a637e2df968", TXT "zoho-verification=zb90149015.zmverify.zoho.com", TXT "apple-domain-verification=ivyxTJSvycL1rKet", TXT "autodesk-domain-verification=4zOZypex_sR1HLFsXs7E", TXT "onetrust-domain-verification=811456c061094fd787edfbea1f50e0c2", TXT "google-site-verification=yuAuTV0V218aUY-z4yyaeBY0B-icA3PcEFNCd72ZKk4", TXT "apple-domain-verification=71mEATCbpJsvgxSj", RRSIG, RRSIG, RRSIG[|domain]
08:36:35.176814 IP 201.184.117.60.53 > 111.222.333.444.1723: 1| 41/0/0 SOA, RRSIG, RRSIG, RRSIG, RRSIG, RRSIG, RRSIG, RRSIG, RRSIG, RRSIG[|domain]
08:36:35.176818 IP 201.184.117.60 > 111.222.333.444: ip-proto-17
08:36:35.176819 IP 197.91.174.102.53 > 111.222.333.444.1723: 1| 37/0/0 RRSIG, RRSIG, MX smtp2.renault.fr. 30, MX smtp.renault.fr. 20, SOA, DS, DNSKEY, RRSIG, RRSIG, A 35.71.164.53, A 52.223.12.199, RRSIG, RRSIG, RRSIG[|domain]

What are the interface errors?

Why are there interface errors?

dmesg | grep igc2
igc2: <Intel(R) Ethernet Controller I225-V> mem 0x7fa00000-0x7fafffff,0x7fc00000-0x7fc03fff at device 0.0 on pci3
igc2: Using 1024 TX descriptors and 1024 RX descriptors
igc2: Using 4 RX queues 4 TX queues
igc2: Using MSI-X interrupts with 5 vectors
igc2: Ethernet address: 64:62:66:22:01:b1
igc2: netmap queues/slots: TX 4/1024, RX 4/1024

in the "tcpdump -i igc2 -n" output (as far as I can tell..) I was able to capture everything..

Thanks in advance..

[doktornotor]

Not sure what are you planning. Other that "do not expose public DNS on your WAN" you cannot prevent DDoS at your end. Move to your ISP.

[bcookatpcsd]

Thank you for the response..

I am not running/allow dns..

they are sending the responses back to me.. hence the DDoS..

I am the 111.222.333.444 address.. and I'm not running a pptp server either..

I'm assuming you also have no idea what the hardware errors would be..

[doktornotor]

Which HW errors? So you are getting single errors (error is e.g. a malformed packet or out of buffer space) under DDoS? Sounds perfectly expected to me.

Once again, only your ISP (and their peer) can do something with the DDoS. Perimeter firewall is not the place to deal with those.

[iMx]

Looks like the sources are indeed open resolvers:

Code Select Expand

dig +short @36.91.138.130 www.google.com
64.233.170.147
64.233.170.104
64.233.170.99
64.233.170.103
64.233.170.106
64.233.170.105

dig +short @201.184.117.60 www.google.com
142.250.78.164

dig +short @111.70.2.171 www.google.com
142.251.42.228


It does, however, show that you're also sending ICMP port unreachable back:

Code Select Expand

08:36:35.176707 IP 111.222.333.444 > 36.91.138.130: ICMP 111.222.333.444 udp port 1723 unreachable, length 576

You haven't created any 'reject' rather than 'drop' (block) rules to try and block this by any chance? As a reject rule for UDP would send a destination unreachable ICMP response.

Or have 1723 port forwarded? Or a DMZ?

Aside from the above, I would have hoped/assumed, that a stateful firewall - albeit UDP - would at least realise there wasn't a recent egress connection to that destination/port and so the response is invalid and should be dropped without an ICMP response. Although I'd have to test myself.

If it is valid behaviour, although I'm not convinced, then you could try to limit outbound ICMP responses to the destination ASNs, geo-ip countries, etc without disabling ICMP as a whole. Or, drop all ICMP destination unreachable replies.

If you're on a dynamic IP, or recently acquired the IP/block on your end, it's possible that you aren't/weren't the actual target - but this is just 'carry over'.

In terms of blocking the DNS replies, that are presumably replies to spoofed requests from your source IP:

- If the WAN IP is dynamic, change the IP
- If the WAN IP is static, ask for a new IP/subnet
- If it's always to UDP 1723 (which isn't the PPTP port, TCP 1723 is), ask the ISP if they can filter this port upstream
- If it's not using up much bandwidth in the grand scheme of things (i.e total bandwidth), try to work out why it's sending ICMP replies (to stop them) then move on with life...

From your netstat, every 1 second, 23961931 bytes is 20MB - then at peak 62122192 bytes, 60MB?  Although I assume this is ALL traffic, assuming the circuit is otherwise loaded with traffic, not just DNS replies?

Could possibly be something like this:

https://blog.cloudflare.com/sad-dns-explained/

"An attacker wants to know whether the target has an open port, so it sends a spoofed UDP message from the authoritative server to that port. If the port is open, no ICMP reply is sent and the rate counter remains unchanged. If the port is inaccessible, then an ICMP reply is sent (back to the authoritative server, not to the attacker) and the rate is increased by one. Although the attacker doesn't see the ICMP response, it has influenced the counter. The counter itself isn't known outside the server, but whether it has hit the rate limit or not can be measured by any outside observer by sending a UDP packet and waiting for a reply. If an ICMP "port unreachable" reply comes back, the rate limit hasn't been reached. No reply means the rate limit has been met. This leaks one bit of information about the counter to the outside observer, which in the end is enough to reveal the supposedly secret information (whether the spoofed request got through or not)."

[bcookatpcsd]

@imx

Thank you for the response..

No incoming services.. never was; and yes this is a new block to me.. but obviously used.  So no 1723 incoming..

Provider showing 1.5Gb to 2Gb coming to them (at peak) but I'm a 500 symmetrical, not sure how they are decided which part of the 1.5 to send me..

As this has been going on a week plus, I've tried various things, dropping, rejecting.. What I changed yesterday actually seemed to change today's event..

I was dropping src 53 tcp/udp, dst 1723 tcp and all that did was seem to get the traffic 'resent' also setting reject (rst) instead of drop also kept them coming..

But the big change was not allowing icmp to the range except from the provider and a few other hosts..

Possibly a ping sweep to see if things are still alive.. not sure

Maybe tomorrow will be different.. but today we didn't get anything.

Currently rejecting icmp and rejecting those services.. and waiting for tomorrow.

But yes those 60MB was all DNS when it would happen no outbound was possible and the interface would just start registering 'errors' which aren't logged within opnsense anywhere that I could find.. so going to netstat was the best way that I could see the 'system' errors and rule them out as 'application' errors..

dmesg would show:

[zone: pf frag entries] PF frag entries limit reached

pfctl -sa

LIMITS:
states        hard limit  1621000
src-nodes     hard limit  1621000
frags         hard limit   204800
table-entries hard limit  1000000


  pfctl -sa | grep ESTAB | wc -l
   20285

so currently some 20k states on this device.

No current way to edit tcp timeouts in pf on opnsense.. but for the incoming lan side where the proxy is I've reduced the tcp timeout.. never could find what the frags where/are..

again, thanks for the response.

[iMx]

and yes this is a new block to me.. but obviously used.  So no 1723 incoming..

If it were me, I'd: "YO, ISP! GO FIX!" :)

Fragments: Firewall -> Settings -> Advanced ->  Firewall Maximum Fragments

.. although none of this really fixes the problem, you might just make it tolerable. 

As it's a new block, I'd be firing this data at your ISP, telling them they need to either filter/scrub, or provide a new block as this one should be rested for a bit until the attacks subside

[doktornotor]

Hopeless apparently. Said that there's nothing to fix at perimeter FW about 5 times already. And there we are with absurd suggestions such as blocking ICMP.

Sigh.

[bcookatpcsd]

Hopeless apparently. Said that there's nothing to fix at perimeter FW about 5 times already. And there we are with absurd suggestions such as blocking ICMP.

Sigh.

absurd (adjective) impossible to take seriously

Which HW errors? So you are getting single errors (error is e.g. a malformed packet or out of buffer space) under DDoS? Sounds perfectly expected to me.

Does it matter if it's maformed *OR* out of buffer space? Are those the same? Would they appear the same? Where would that be?

Please generate some traffic, send them at whatever speed you would like, across the Interweb, not just local.. pass them through a bunch of devices and different carrier equipment.. and then malform 'one' (or any number) you would like, so that they reach their destination.. not all off them, just a few.

Have the last mile carrier limit the traffic sent out of the (unknown number 200k/pps) they are going to send you (known 50k/pps).  Send them to a 2.5Gb capable nic with whatever the buffer space is (another unknown number), and then turn off or on hardware CRC, and turn off or on hardware TSO, and then then off or on hardware LRO.

Don't worry about putting any of that into an equation to try and solve for all the unknown variables, don't even factor in any OS settings.. or any of the equipment along the route, or any people and their education or network knowledge, or anyone and their ability to write drivers, etc, any of that.. Don't take any of that into account..

Another variable is time.. Companies pay people to provision links, people want to sell you DDoS services, everyone wants to sh*t on open source solutions and just assume you missed a check box that says, ".. really I'm not lazy."


For the computer science and math teachers .. is your answer reasonable?

"Sounds perfectly expected to me."

I hope someday I can solve all all of the above without having to show work or ask any questions.

Asking questions, shows your interested; my 0.02..

Unfortunately the CCNP questions I had were nothing like that; but that was a while ago.. haven't seen any in the 401 refresh.. maybe when I get to the 410.. I don't know anyone who has their CCDE.. maybe that's where you get those questions.. and learn how to answer them; still not sure how FreeBSD and OpnSense would factor in though, let alone local hardware decisions.. never mind any of that..

But apparently there still is hope, or is it "Hopeless apparently"

The absurd action of rejecting icmp sweeps.. and causing a change in my local situation; inconceivable

That word.. I do not think it means what you think it means..

https://imgur.com/a/EVXP9zK

[doktornotor]

Does it matter if it's maformed *OR* out of buffer space? Are those the same? Would they appear the same? Where would that be?

No, it does not, they are not, and yes - they would still appear in the same "errors" column you posted. And you would still not solve anything whatsoever on your end. Blocking ICMP or not.