Lost connectivity to some internet service (SOLVED)

[Sangito]

I start by having an issue with Facebook as I said here:
https://forum.opnsense.org/index.php?topic=26505.msg133204#msg133204
Now it expands to some streaming service and some web API.
I check the firewall rule to see if this traffic got some hit, but only packet with. A tcpflag got reject and I have been explaining that a normal behaviour on state full firewall.
It, also occurs that is having some internet connectivity issue.
So I am looking for some hint to start to debug this behaviour.
Soo far if I can  make it work again, I will have to switch to my ISP router 
Is anybody having an idea to where i should start?

Thanks

[Sangito]

I made two captures of a request to Facebook.
It looks like the ack from my locale network doesn’t seem to reach the Facebook destination.
If a stateless firewall drop ack TCP flag how am I supposed to acknowledge is made between my computer and the web site?

[Janssuhhh]

Same problem here, but with different sites..
Facebook and Google work.

But for example the opnsense pages won't load.

What I've also noticed is doing Speedtest (through their app) will give a Ping and great upload, however the download test fails entirely.

[Sangito]

Here a request from a much simpler web site working well.
You can see the em0 interface been more verbose...
Could it be linked to the NAT interface ?

[Sangito]

Same problem here, but with different sites..
Facebook and Google work.

But for example the opnsense pages won't load.

What I've also noticed is doing Speedtest (through their app) will give a Ping and great upload, however the download test fails entirely.

How can you do the speedtest with their app?
Which app?
Can you provide a failed request like mine ?
What does the live firewall said about you failed request ?

[Sangito]

By inspecting the capture of the working request, I found that between the WAN interface and the LAN there is an additional packet the "TCP WINDOWS UPDATE"
It's the packet missing in the failed request.
I think this is the beginning of a debug trail.
Is someone knowing what is the role of the router in dealing "TCP windows update"  packet?

[Sangito]

OK, I'll do a new test.
I did a backup of my current setting.
I I made a factory reset of my router.
I redo the minimum configuration to make my router work (vlan and dhcp) and bam Facebook work again.

Soo, I save this "clean" setup as a backup and try to find out which part of the configuration of my router failed.
I first load "system" backup, everything works.
I second load the "OPNsense additional" and lost connectivity to Facebook.
I reload the clean backup, Facebook works again.
I load the "OPNsense additional" and Facebook work.
I load the system back up and lose Facebook.

Right now I'm looking for the backuped config to find which setting could cause errors.
Most of the config are certificates of my router and user and password.
There is also the plugging configuration i think there is the most suspect config, but stop all my service have not solved my problem.
Anybody have any idea ?

[cookiemonster]

Only that you're on the right track. Plugins, configuration settings, etc. I know it is not very helpful to say "it would be a widely reported problem" but it seems the clean install resolves, so just one thing to remember, that the LAN client first makes a dns query, when it is given, it will move to create an outbound connection out of WAN before it comes back in. That's the flow I would trace.

[fbantgat7]

I've experienced the same problem on an apu box with the upgrade from 22.1.2_2 to 22.1.4_1.  Some websites e.g. duckduckgo timeout, while others e.g. google work fine.  Interestingly, I can't access opnsense.org and an attempt to run an update status check also hangs.  I can ping the various inaccessible websites, DNS resolves fine, but http connectivity fails.  I've only checked with IPv4 websites so far.  In this state the router is obviously unusable.

The network is pretty much in continuous use, so I can't afford prolonged downtime to troubleshoot it in detail.  I've attempted 2 upgrades, but each time I restored the full previous working configuration, rather than restoring only parts of it to check conclusively which of the router settings may cause the problem.

The configuration is rather basic, except for some firewall rules blocking some social media and bufferbloat shaping with fq-codel.  Previous updates/upgrades since the 22.1 release did not have any problems.

Please post back if you have any workarounds, I'd be interested to try them out on my setup.  Otherwise I may wait for a future update/upgrade in case this issue is resolved.

[zeon]

Hello everyone,

I was just troubleshooting same behavior (so far Facebook is not working)
So far I see that packets coming to Facebook (31.13.93.54 and 157.240.241.17) are being blocked by the default rule (in my case it's rule 12)

Code: [Select]

rule 12/0(match): block in on vtnet1: 192.168.1.113.50103 > 31.13.93.54.5222: Flags [F.], seq 0, ack 1, win 65535, length 0
rule 12/0(match): block in on vtnet1: 192.168.1.113.49194 > 157.240.241.17.443: Flags [F.], seq 0, ack 1, win 65535, length 0
rule 12/0(match): block in on vtnet1: 192.168.1.113.50103 > 31.13.93.54.5222: Flags [F.], seq 0, ack 1, win 65535, length 0
rule 12/0(match): block in on vtnet1: 192.168.1.113.49194 > 157.240.241.17.443: Flags [F.], seq 0, ack 1, win 65535, length 0
rule 12/0(match): block in on vtnet1: 192.168.1.113.50103 > 31.13.93.54.5222: Flags [F.], seq 0, ack 1, win 65535, length 0
Actual rule 12 looks like this:

Code: [Select]

@12 block drop in log inet all label "02f4bab031b57d1e30553ce08e0ec131"
I don't understand why packets coming to some specific IPs are blocked by this default rule.
PS. If I just wipe out all the rules and create some manually (like nat and default allow rule) everything is fine.

I'm more than happy if somebody shed some light on this issue.
Thank you.

[yorch]

Same boat here, upgraded couple of days ago and I've lost connectivity to a few sites, in my case, some S3 URLs, and Debian and Ubuntu apt update servers.

@zeon how did you get those logs with the matching rule? I've been looking for that kind of info without success.

Thanks

[zeon]

@yorch

You can ssh into the box and run this

Code: [Select]

tcpdump -tenpi pflog0
You can limit the information it shows by applying some filters
This one is going to display information only related to vtnet1 interface and traffic that being blocked

Code: [Select]

tcpdump -tenpi pflog0 ifname vtnet1 and action block

[yorch]

that's good info, reading about PF also, thanks a lot!

[Sangito]

OK, I figured out what was my problem.
It looks like I activate the anti-DDOS and since it plays with the TCP SYN it could have played with the TCP windows.
That explains why some request where block.
I disable the feature and bam everything works back again
Thanks for the support guy.

[fbantgat7]

My Anti-DDOS setting is left to 'never (default)' and the problem of inaccessible domains remains.  So, there appear to be different causes to this problem.

I tried again with 22.1.5, after I reset the router to default and then configured manually WAN PPPoE with no additional firewall rules. At least with this update IPv6 dpinger works, but from a packet capture I can see there is a DNS issue.  The router's IPv4 address is shown as 'AAA.BBB.CC.DDD' and its IPv6 as '2001:999:8888::777:66', while I was trying to load 'https://opnsense.org' in a browser:

Code: [Select]

19:20:12.658812 AF IPv4 (2), length 68: AAA.BBB.CC.DDD.11310 > 52.88.110.101.443: Flags [.], ack 966689392, win 502, options [nop,nop,TS val 3265146862 ecr 2315127521,nop,nop,sack 1 {2897:3507}], length 0
19:20:12.819443 AF IPv4 (2), length 56: 52.88.110.101.443 > AAA.BBB.CC.DDD.11310: Flags [.], ack 1, win 110, options [nop,nop,TS val 2315147951 ecr 3265126597], length 0

19:20:13.732073 AF IPv6 (28), length 76: 2001:999:8888::777:66.14453 > 2001:500:3682::11.53: Flags [F.], seq 1446466141, ack 2254340269, win 518, options [nop,nop,sack 1 {1441:1720}], length 0
19:20:13.732545 AF IPv6 (28), length 83: 2001:999:8888::777:66.10254 > 2001:dcd:3::15.53: 12811% [1au] DNSKEY? us. (31)
19:20:13.748872 AF IPv6 (28), length 64: 2001:500:3682::11.53 > 2001:999:8888::777:66.14453: Flags [F.], seq 1720, ack 1, win 85, length 0
19:20:13.749034 AF IPv6 (28), length 76: 2001:999:8888::777:66.14453 > 2001:500:3682::11.53: Flags [.], ack 1, win 518, options [nop,nop,sack 1 {1441:1720}], length 0
19:20:13.750023 AF IPv6 (28), length 83: 2001:dcd:3::15.53 > 2001:999:8888::777:66.10254: 12811*-| 0/0/1 (31)
19:20:13.750541 AF IPv6 (28), length 84: 2001:999:8888::777:66.7531 > 2001:dcd:3::15.53: Flags [S], seq 288186767, win 65228, options [mss 1440,nop,wscale 7,sackOK,TS val 1823543909 ecr 0], length 0
19:20:13.767445 AF IPv6 (28), length 84: 2001:dcd:3::15.53 > 2001:999:8888::777:66.7531: Flags [S.], seq 1408109706, ack 288186768, win 28560, options [mss 1440,sackOK,TS val 171129600 ecr 1823543909,nop,wscale 9], length 0
19:20:13.767687 AF IPv6 (28), length 76: 2001:999:8888::777:66.7531 > 2001:dcd:3::15.53: Flags [.], ack 1, win 518, options [nop,nop,TS val 1823543927 ecr 171129600], length 0
19:20:13.767781 AF IPv6 (28), length 109: 2001:999:8888::777:66.7531 > 2001:dcd:3::15.53: Flags [P.], seq 1:34, ack 1, win 518, options [nop,nop,TS val 1823543927 ecr 171129600], length 33 8873% [1au] DNSKEY? us. (31)
19:20:13.785428 AF IPv6 (28), length 76: 2001:dcd:3::15.53 > 2001:999:8888::777:66.7531: Flags [.], ack 34, win 56, options [nop,nop,TS val 171129618 ecr 1823543927], length 0
19:20:13.786781 AF IPv6 (28), length 367: 2001:dcd:3::15.53 > 2001:999:8888::777:66.7531: Flags [P.], seq 1429:1720, ack 34, win 56, options [nop,nop,TS val 171129618 ecr 1823543927], length 291 [prefix length(10997) != length(289)] (invalid)
19:20:13.786929 AF IPv6 (28), length 88: 2001:999:8888::777:66.7531 > 2001:dcd:3::15.53: Flags [.], ack 1, win 518, options [nop,nop,TS val 1823543946 ecr 171129618,nop,nop,sack 1 {1429:1720}], length 0


19:20:14.238025 AF IPv6 (28), length 76: 2001:999:8888::777:66.25645 > 2001:502:ad09::29.53: Flags [F.], seq 708765903, ack 112987093, win 518, options [nop,nop,sack 1 {1441:1720}], length 0
19:20:14.238441 AF IPv6 (28), length 83: 2001:999:8888::777:66.18606 > 2001:dcd:1::15.53: 20508% [1au] DNSKEY? us. (31)
I reinstalled 22.1.2 which does not have this problem.