OPNsense Forum

Archive => 22.7 Legacy Series => Topic started by: ctr on December 11, 2022, 04:58:21 pm

Title: retransmission storm after upgrading to 22.7.9_3
Post by: ctr on December 11, 2022, 04:58:21 pm

After upgrading to 22.7.9_3 I observe massive storms of retransmissions for IPv6 traffic like this:

Code: [Select]

16:44:53.107317 IP6 2a01:dead::beef.22483 > 2600:1901:1:c36::.443: Flags [.], seq 0:1432, ack 1, win 517, options [nop,nop,TS val 1734819756 ecr 4230049189], length 1432
16:44:53.107320 IP6 2a01:dead::beef.22483 > 2600:1901:1:c36::.443: Flags [.], seq 0:1432, ack 1, win 517, options [nop,nop,TS val 1734819756 ecr 4230049189], length 1432
16:44:53.107322 IP6 2a01:dead::beef.22483 > 2600:1901:1:c36::.443: Flags [.], seq 0:1432, ack 1, win 517, options [nop,nop,TS val 1734819756 ecr 4230049189], length 1432
16:44:53.107325 IP6 2a01:dead::beef.22483 > 2600:1901:1:c36::.443: Flags [.], seq 0:1432, ack 1, win 517, options [nop,nop,TS val 1734819756 ecr 4230049189], length 1432
16:44:53.107327 IP6 2a01:dead::beef.22483 > 2600:1901:1:c36::.443: Flags [.], seq 0:1432, ack 1, win 517, options [nop,nop,TS val 1734819756 ecr 4230049189], length 1432
16:44:53.107330 IP6 2a01:dead::beef.22483 > 2600:1901:1:c36::.443: Flags [.], seq 0:1432, ack 1, win 517, options [nop,nop,TS val 1734819756 ecr 4230049189], length 1432
16:44:53.107333 IP6 2a01:dead::beef.22483 > 2600:1901:1:c36::.443: Flags [.], seq 0:1432, ack 1, win 517, options [nop,nop,TS val 1734819756 ecr 4230049189], length 1432
16:44:53.107335 IP6 2a01:dead::beef.22483 > 2600:1901:1:c36::.443: Flags [.], seq 0:1432, ack 1, win 517, options [nop,nop,TS val 1734819756 ecr 4230049189], length 1432
16:44:53.107338 IP6 2a01:dead::beef.22483 > 2600:1901:1:c36::.443: Flags [.], seq 0:1432, ack 1, win 517, options [nop,nop,TS val 1734819756 ecr 4230049189], length 1432
16:44:53.107340 IP6 2a01:dead::beef.22483 > 2600:1901:1:c36::.443: Flags [.], seq 0:1432, ack 1, win 517, options [nop,nop,TS val 1734819756 ecr 4230049189], length 1432
16:44:53.107343 IP6 2a01:dead::beef.22483 > 2600:1901:1:c36::.443: Flags [.], seq 0:1432, ack 1, win 517, options [nop,nop,TS val 1734819756 ecr 4230049189], length 1432
16:44:53.107346 IP6 2a01:dead::beef.22483 > 2600:1901:1:c36::.443: Flags [.], seq 0:1432, ack 1, win 517, options [nop,nop,TS val 1734819756 ecr 4230049189], length 1432
16:44:53.107348 IP6 2a01:dead::beef.22483 > 2600:1901:1:c36::.443: Flags [.], seq 0:1432, ack 1, win 517, options [nop,nop,TS val 1734819756 ecr 4230049189], length 1432
16:44:53.107351 IP6 2a01:dead::beef.22483 > 2600:1901:1:c36::.443: Flags [.], seq 0:1432, ack 1, win 517, options [nop,nop,TS val 1734819756 ecr 4230049189], length 1432
16:44:53.107353 IP6 2a01:dead::beef.22483 > 2600:1901:1:c36::.443: Flags [.], seq 0:1432, ack 1, win 517, options [nop,nop,TS val 1734819756 ecr 4230049189], length 1432
16:44:53.107356 IP6 2a01:dead::beef.22483 > 2600:1901:1:c36::.443: Flags [.], seq 0:1432, ack 1, win 517, options [nop,nop,TS val 1734819756 ecr 4230049189], length 1432
16:44:53.107358 IP6 2a01:dead::beef.22483 > 2600:1901:1:c36::.443: Flags [.], seq 0:1432, ack 1, win 517, options [nop,nop,TS val 1734819756 ecr 4230049189], length 1432
16:44:53.107361 IP6 2a01:dead::beef.22483 > 2600:1901:1:c36::.443: Flags [.], seq 0:1432, ack 1, win 517, options [nop,nop,TS val 1734819756 ecr 4230049189], length 1432
16:44:53.107364 IP6 2a01:dead::beef.22483 > 2600:1901:1:c36::.443: Flags [.], seq 0:1432, ack 1, win 517, options [nop,nop,TS val 1734819756 ecr 4230049189], length 1432
16:44:53.107366 IP6 2a01:dead::beef.22483 > 2600:1901:1:c36::.443: Flags [.], seq 0:1432, ack 1, win 517, options [nop,nop,TS val 1734819756 ecr 4230049189], length 1432
16:44:53.107369 IP6 2a01:dead::beef.22483 > 2600:1901:1:c36::.443: Flags [.], seq 0:1432, ack 1, win 517, options [nop,nop,TS val 1734819756 ecr 4230049189], length 1432
16:44:53.107372 IP6 2a01:dead::beef.22483 > 2600:1901:1:c36::.443: Flags [.], seq 0:1432, ack 1, win 517, options [nop,nop,TS val 1734819756 ecr 4230049189], length 1432
16:44:53.107374 IP6 2a01:dead::beef.22483 > 2600:1901:1:c36::.443: Flags [.], seq 0:1432, ack 1, win 517, options [nop,nop,TS val 1734819756 ecr 4230049189], length 1432

This is going on hundreds of times per second, effectively creating a DoS for other traffic (can't tell if due to bandwidth or firewall resource exhaustion). The traffic in questions seems to be generated by Squid, I can't observe the retransmission on an internal leg of the firewall, but the destinations belong to legit communication to several destinations (Apple, Microsoft) and by several distinct client types (Linux workstation, iOS smartphones).

The state tables also look weird for those, with ESTABLISHED:FIN_WAIT_2 states for the destinations in question.

This started after upgrading to 22.7.9_3, coming from 22.7.8

Disablinh Suricata, resetting states, reboot and even switching the underlying VM host (this is OPNsense running as KVM VM) did not help.

Title: Re: retransmission storm after upgrading to 22.7.9_3
Post by: Fright on December 18, 2022, 06:17:35 pm

out of curiosity, what does "sysctl kern.timecounter.hardware" return?