Strange network behaviour after spinning up new docker containers - MTU problem?

[chili-koch]

Hi together,
I’m struggeling a bit with my docker setup running in a Hetzner Cloud environment behind an Opnsense firewall:

Code: [Select]

Home network
192.168.178.0/24
|
Fritzbox Router
192.168.178.1
|
Wireguard VPN
10.10.10.0/28
|
OpnSense firewall
10.100.0.2
|
Docker host
10.100.0.3
|
Docker ipvlan (l3)
10.100.1.0/24
|
Docker Containers
10.100.1.2 - 10
The goal is to run applications like pihole, node-red, etc. within this environment and access them through the mentioned Wireguard VPN tunnel from my home network. Other applications like vaultwarden should be published directly through the Opnsense firewall.

Generally I can reach all my containers from the home network as well as clients within the home network from my containers.
Strange things begin to happen if I spin up a new docker container.
After a new container is deployed some of the other containers stop responding to http-requests (pihole web-interface for instance), but all containers are still reachable by ping. The pihole container seems to be able to respond to DNS requests as well.

I can see the http-request from the client in the affected docker container logs as well, but the client does not get any response.
After 10-15 minutes the problem is healing itself and all containers are reachable by http again!
Containers are randomly affected by this problem after deploying a new container.

My guess is a problem with MTU size, because smaller packages like DNS and ICMP are working fine.

ip link output from the docker host (ens10 is parent interface for ipvlan L3 mentioned above):

Code: [Select]

2: ens10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 86:00:00:6b:fd:0d brd ff:ff:ff:ff:ff:ff
    altname enp0s10
tcpdump output from the docker host (http is not working until 11:03:50.041785):

Code: [Select]

root@docker1:~# tcpdump -i ens10 host 10.100.1.2 and port 80
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens10, link-type EN10MB (Ethernet), snapshot length 262144 bytes
11:02:11.117125 IP 192.168.178.74.56240 > 10.100.1.2.http: Flags [S], seq 1293350010, win 64240, options [mss 1460,sackOK,TS val 1216921712 ecr 0,nop,wscale 7], length 0
11:02:11.117403 IP 10.100.1.2.http > 192.168.178.74.56240: Flags [S.], seq 1277008952, ack 1293350011, win 64308, options [mss 1410,sackOK,TS val 9145979 ecr 1216921712,nop,wscale 7], length 0
11:02:11.128123 IP 192.168.178.74.56240 > 10.100.1.2.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 1216921722 ecr 9145979], length 0
11:02:11.128635 IP 192.168.178.74.56240 > 10.100.1.2.http: Flags [P.], seq 1:339, ack 1, win 502, options [nop,nop,TS val 1216921723 ecr 9145979], length 338: HTTP: GET / HTTP/1.1
11:02:11.128696 IP 10.100.1.2.http > 192.168.178.74.56240: Flags [.], ack 339, win 501, options [nop,nop,TS val 9145990 ecr 1216921723], length 0
11:02:21.232300 IP 192.168.178.74.56240 > 10.100.1.2.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 1216931827 ecr 9145990], length 0
11:02:21.232458 IP 10.100.1.2.http > 192.168.178.74.56240: Flags [R], seq 1277008953, win 0, length 0
11:02:21.244553 IP 192.168.178.74.59402 > 10.100.1.2.http: Flags [S], seq 430052011, win 64240, options [mss 1460,sackOK,TS val 1216931839 ecr 0,nop,wscale 7], length 0
11:02:21.244669 IP 10.100.1.2.http > 192.168.178.74.59402: Flags [S.], seq 4106532121, ack 430052012, win 64308, options [mss 1410,sackOK,TS val 9156106 ecr 1216931839,nop,wscale 7], length 0
11:02:21.255185 IP 192.168.178.74.59402 > 10.100.1.2.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 1216931850 ecr 9156106], length 0
11:02:21.255888 IP 192.168.178.74.59402 > 10.100.1.2.http: Flags [P.], seq 1:339, ack 1, win 502, options [nop,nop,TS val 1216931850 ecr 9156106], length 338: HTTP: GET / HTTP/1.1
11:02:21.255958 IP 10.100.1.2.http > 192.168.178.74.59402: Flags [.], ack 339, win 501, options [nop,nop,TS val 9156118 ecr 1216931850], length 0
11:02:31.472087 IP 192.168.178.74.59402 > 10.100.1.2.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 1216942067 ecr 9156118], length 0
11:02:31.472178 IP 10.100.1.2.http > 192.168.178.74.59402: Flags [R], seq 4106532122, win 0, length 0
11:02:31.483113 IP 192.168.178.74.33770 > 10.100.1.2.http: Flags [S], seq 1392164708, win 64240, options [mss 1460,sackOK,TS val 1216942078 ecr 0,nop,wscale 7], length 0
11:02:31.483211 IP 10.100.1.2.http > 192.168.178.74.33770: Flags [S.], seq 3363726988, ack 1392164709, win 64308, options [mss 1410,sackOK,TS val 9166345 ecr 1216942078,nop,wscale 7], length 0
11:02:31.493069 IP 192.168.178.74.33770 > 10.100.1.2.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 1216942088 ecr 9166345], length 0
11:02:31.494101 IP 192.168.178.74.33770 > 10.100.1.2.http: Flags [P.], seq 1:339, ack 1, win 502, options [nop,nop,TS val 1216942088 ecr 9166345], length 338: HTTP: GET / HTTP/1.1
11:02:31.494164 IP 10.100.1.2.http > 192.168.178.74.33770: Flags [.], ack 339, win 501, options [nop,nop,TS val 9166356 ecr 1216942088], length 0
11:02:41.712069 IP 192.168.178.74.33770 > 10.100.1.2.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 1216952307 ecr 9166356], length 0
11:02:41.712204 IP 10.100.1.2.http > 192.168.178.74.33770: Flags [R], seq 3363726989, win 0, length 0
11:02:41.722208 IP 192.168.178.74.43020 > 10.100.1.2.http: Flags [S], seq 941533197, win 64240, options [mss 1460,sackOK,TS val 1216952318 ecr 0,nop,wscale 7], length 0
11:02:41.722338 IP 10.100.1.2.http > 192.168.178.74.43020: Flags [S.], seq 1827097930, ack 941533198, win 64308, options [mss 1410,sackOK,TS val 9176584 ecr 1216952318,nop,wscale 7], length 0
11:02:41.732613 IP 192.168.178.74.43020 > 10.100.1.2.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 1216952327 ecr 9176584], length 0
11:02:41.733044 IP 192.168.178.74.43020 > 10.100.1.2.http: Flags [P.], seq 1:339, ack 1, win 502, options [nop,nop,TS val 1216952328 ecr 9176584], length 338: HTTP: GET / HTTP/1.1
11:02:41.733121 IP 10.100.1.2.http > 192.168.178.74.43020: Flags [.], ack 339, win 501, options [nop,nop,TS val 9176595 ecr 1216952328], length 0
11:02:51.952165 IP 192.168.178.74.43020 > 10.100.1.2.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 1216962547 ecr 9176595], length 0
11:02:51.952342 IP 10.100.1.2.http > 192.168.178.74.43020: Flags [R], seq 1827097931, win 0, length 0
11:02:51.963350 IP 192.168.178.74.45550 > 10.100.1.2.http: Flags [S], seq 3054109757, win 64240, options [mss 1460,sackOK,TS val 1216962558 ecr 0,nop,wscale 7], length 0
11:02:51.963418 IP 10.100.1.2.http > 192.168.178.74.45550: Flags [S.], seq 3120337985, ack 3054109758, win 64308, options [mss 1410,sackOK,TS val 9186825 ecr 1216962558,nop,wscale 7], length 0
11:02:51.974066 IP 192.168.178.74.45550 > 10.100.1.2.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 1216962568 ecr 9186825], length 0
11:02:51.974971 IP 192.168.178.74.45550 > 10.100.1.2.http: Flags [P.], seq 1:339, ack 1, win 502, options [nop,nop,TS val 1216962568 ecr 9186825], length 338: HTTP: GET / HTTP/1.1
11:02:51.975130 IP 10.100.1.2.http > 192.168.178.74.45550: Flags [.], ack 339, win 501, options [nop,nop,TS val 9186837 ecr 1216962568], length 0
11:03:02.192142 IP 192.168.178.74.45550 > 10.100.1.2.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 1216972787 ecr 9186837], length 0
11:03:02.192303 IP 10.100.1.2.http > 192.168.178.74.45550: Flags [R], seq 3120337986, win 0, length 0
11:03:02.203517 IP 192.168.178.74.59172 > 10.100.1.2.http: Flags [S], seq 308538419, win 64240, options [mss 1460,sackOK,TS val 1216972798 ecr 0,nop,wscale 7], length 0
11:03:02.203634 IP 10.100.1.2.http > 192.168.178.74.59172: Flags [S.], seq 3907359373, ack 308538420, win 64308, options [mss 1410,sackOK,TS val 9197065 ecr 1216972798,nop,wscale 7], length 0
11:03:02.213037 IP 192.168.178.74.59172 > 10.100.1.2.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 1216972809 ecr 9197065], length 0
11:03:02.214499 IP 192.168.178.74.59172 > 10.100.1.2.http: Flags [P.], seq 1:339, ack 1, win 502, options [nop,nop,TS val 1216972809 ecr 9197065], length 338: HTTP: GET / HTTP/1.1
11:03:02.214608 IP 10.100.1.2.http > 192.168.178.74.59172: Flags [.], ack 339, win 501, options [nop,nop,TS val 9197076 ecr 1216972809], length 0
11:03:12.432273 IP 192.168.178.74.59172 > 10.100.1.2.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 1216983027 ecr 9197076], length 0
11:03:12.432439 IP 10.100.1.2.http > 192.168.178.74.59172: Flags [R], seq 3907359374, win 0, length 0
11:03:12.442704 IP 192.168.178.74.59148 > 10.100.1.2.http: Flags [S], seq 2014914663, win 64240, options [mss 1460,sackOK,TS val 1216983038 ecr 0,nop,wscale 7], length 0
11:03:12.442870 IP 10.100.1.2.http > 192.168.178.74.59148: Flags [S.], seq 3772085051, ack 2014914664, win 64308, options [mss 1410,sackOK,TS val 9207305 ecr 1216983038,nop,wscale 7], length 0
11:03:12.453218 IP 192.168.178.74.59148 > 10.100.1.2.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 1216983048 ecr 9207305], length 0
11:03:12.454266 IP 192.168.178.74.59148 > 10.100.1.2.http: Flags [P.], seq 1:339, ack 1, win 502, options [nop,nop,TS val 1216983048 ecr 9207305], length 338: HTTP: GET / HTTP/1.1
11:03:12.454398 IP 10.100.1.2.http > 192.168.178.74.59148: Flags [.], ack 339, win 501, options [nop,nop,TS val 9207316 ecr 1216983048], length 0
11:03:22.672539 IP 192.168.178.74.59148 > 10.100.1.2.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 1216993267 ecr 9207316], length 0
11:03:22.672649 IP 10.100.1.2.http > 192.168.178.74.59148: Flags [R], seq 3772085052, win 0, length 0
11:03:22.683831 IP 192.168.178.74.39042 > 10.100.1.2.http: Flags [S], seq 1180204649, win 64240, options [mss 1460,sackOK,TS val 1216993278 ecr 0,nop,wscale 7], length 0
11:03:22.683939 IP 10.100.1.2.http > 192.168.178.74.39042: Flags [S.], seq 236851059, ack 1180204650, win 64308, options [mss 1410,sackOK,TS val 9217546 ecr 1216993278,nop,wscale 7], length 0
11:03:22.694688 IP 192.168.178.74.39042 > 10.100.1.2.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 1216993289 ecr 9217546], length 0
11:03:22.695312 IP 192.168.178.74.39042 > 10.100.1.2.http: Flags [P.], seq 1:339, ack 1, win 502, options [nop,nop,TS val 1216993289 ecr 9217546], length 338: HTTP: GET / HTTP/1.1
11:03:22.695414 IP 10.100.1.2.http > 192.168.178.74.39042: Flags [.], ack 339, win 501, options [nop,nop,TS val 9217557 ecr 1216993289], length 0
11:03:32.911718 IP 192.168.178.74.39042 > 10.100.1.2.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 1217003507 ecr 9217557], length 0
11:03:32.911824 IP 10.100.1.2.http > 192.168.178.74.39042: Flags [R], seq 236851060, win 0, length 0
11:03:32.922831 IP 192.168.178.74.46838 > 10.100.1.2.http: Flags [S], seq 3155583952, win 64240, options [mss 1460,sackOK,TS val 1217003517 ecr 0,nop,wscale 7], length 0
11:03:32.922944 IP 10.100.1.2.http > 192.168.178.74.46838: Flags [S.], seq 2723829351, ack 3155583953, win 64308, options [mss 1410,sackOK,TS val 9227785 ecr 1217003517,nop,wscale 7], length 0
11:03:32.933347 IP 192.168.178.74.46838 > 10.100.1.2.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 1217003528 ecr 9227785], length 0
11:03:32.934199 IP 192.168.178.74.46838 > 10.100.1.2.http: Flags [P.], seq 1:339, ack 1, win 502, options [nop,nop,TS val 1217003528 ecr 9227785], length 338: HTTP: GET / HTTP/1.1
11:03:32.934293 IP 10.100.1.2.http > 192.168.178.74.46838: Flags [.], ack 339, win 501, options [nop,nop,TS val 9227796 ecr 1217003528], length 0
11:03:43.152202 IP 192.168.178.74.46838 > 10.100.1.2.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 1217013747 ecr 9227796], length 0
11:03:43.152372 IP 10.100.1.2.http > 192.168.178.74.46838: Flags [R], seq 2723829352, win 0, length 0
11:03:43.164038 IP 192.168.178.74.53566 > 10.100.1.2.http: Flags [S], seq 1409533456, win 64240, options [mss 1460,sackOK,TS val 1217013758 ecr 0,nop,wscale 7], length 0
11:03:43.164213 IP 10.100.1.2.http > 192.168.178.74.53566: Flags [S.], seq 2153763687, ack 1409533457, win 64308, options [mss 1410,sackOK,TS val 9238026 ecr 1217013758,nop,wscale 7], length 0
11:03:43.173784 IP 192.168.178.74.53566 > 10.100.1.2.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 1217013769 ecr 9238026], length 0
11:03:43.175519 IP 192.168.178.74.53566 > 10.100.1.2.http: Flags [P.], seq 1:339, ack 1, win 502, options [nop,nop,TS val 1217013770 ecr 9238026], length 338: HTTP: GET / HTTP/1.1
11:03:43.175609 IP 10.100.1.2.http > 192.168.178.74.53566: Flags [.], ack 339, win 501, options [nop,nop,TS val 9238037 ecr 1217013770], length 0
11:03:50.041785 IP 10.100.1.2.http > 192.168.178.74.53566: Flags [P.], seq 1:1399, ack 339, win 501, options [nop,nop,TS val 9244904 ecr 1217013770], length 1398: HTTP: HTTP/1.1 200 OK
11:03:50.042821 IP 10.100.1.2.http > 192.168.178.74.53566: Flags [P.], seq 1:1399, ack 339, win 501, options [nop,nop,TS val 9244904 ecr 1217013770], length 1398: HTTP: HTTP/1.1 200 OK
11:03:50.052563 IP 192.168.178.74.53566 > 10.100.1.2.http: Flags [.], ack 1399, win 500, options [nop,nop,TS val 1217020648 ecr 9244904], length 0
11:03:50.052641 IP 10.100.1.2.http > 192.168.178.74.53566: Flags [P.], seq 1399:2716, ack 339, win 501, options [nop,nop,TS val 9244914 ecr 1217020648], length 1317: HTTP
11:03:50.062586 IP 192.168.178.74.53566 > 10.100.1.2.http: Flags [.], ack 2716, win 501, options [nop,nop,TS val 1217020658 ecr 9244914], length 0
11:03:50.101971 IP 192.168.178.74.53566 > 10.100.1.2.http: Flags [P.], seq 339:628, ack 2716, win 501, options [nop,nop,TS val 1217020696 ecr 9244914], length 289: HTTP: GET /theme_head.1df11f603fda400762b7.js HTTP/1.1
11:03:50.102054 IP 192.168.178.74.38020 > 10.100.1.2.http: Flags [S], seq 1613604223, win 64240, options [mss 1460,sackOK,TS val 1217020696 ecr 0,nop,wscale 7], length 0
11:03:50.102123 IP 10.100.1.2.http > 192.168.178.74.38020: Flags [S.], seq 1100926057, ack 1613604224, win 64296, options [mss 1380,sackOK,TS val 9244964 ecr 1217020696,nop,wscale 7], length 0
11:03:50.102292 IP 192.168.178.74.38024 > 10.100.1.2.http: Flags [S], seq 1323210162, win 64240, options [mss 1460,sackOK,TS val 1217020696 ecr 0,nop,wscale 7], length 0
11:03:50.102332 IP 10.100.1.2.http > 192.168.178.74.38024: Flags [S.], seq 510922136, ack 1323210163, win 64296, options [mss 1380,sackOK,TS val 9244964 ecr 1217020696,nop,wscale 7], length 0
11:03:50.102742 IP 192.168.178.74.38040 > 10.100.1.2.http: Flags [S], seq 1751853741, win 64240, options [mss 1460,sackOK,TS val 1217020697 ecr 0,nop,wscale 7], length 0
11:03:50.102787 IP 10.100.1.2.http > 192.168.178.74.38040: Flags [S.], seq 477354078, ack 1751853742, win 64296, options [mss 1380,sackOK,TS val 9244965 ecr 1217020697,nop,wscale 7], length 0
11:03:50.103021 IP 192.168.178.74.38056 > 10.100.1.2.http: Flags [S], seq 4127846050, win 64240, options [mss 1460,sackOK,TS val 1217020699 ecr 0,nop,wscale 7], length 0
11:03:50.103050 IP 10.100.1.2.http > 192.168.178.74.38056: Flags [S.], seq 3255105752, ack 4127846051, win 64296, options [mss 1380,sackOK,TS val 9244965 ecr 1217020699,nop,wscale 7], length 0
11:03:50.104054 IP 10.100.1.2.http > 192.168.178.74.53566: Flags [P.], seq 2716:4194, ack 628, win 501, options [nop,nop,TS val 9244966 ecr 1217020696], length 1478: HTTP: HTTP/1.1 200 OK
11:03:50.104533 IP 10.100.1.2.http > 192.168.178.74.53566: Flags [P.], seq 4194:4665, ack 628, win 501, options [nop,nop,TS val 9244966 ecr 1217020696], length 471: HTTP
11:03:50.114324 IP 192.168.178.74.38024 > 10.100.1.2.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 1217020707 ecr 9244964], length 0
11:03:50.114324 IP 192.168.178.74.38020 > 10.100.1.2.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 1217020707 ecr 9244964], length 0
11:03:50.114708 IP 192.168.178.74.38020 > 10.100.1.2.http: Flags [P.], seq 1:304, ack 1, win 502, options [nop,nop,TS val 1217020707 ecr 9244964], length 303: HTTP: GET /app/main.a5d3a3a9125fcf07f286.css HTTP/1.1
11:03:50.114760 IP 10.100.1.2.http > 192.168.178.74.38020: Flags [.], ack 304, win 502, options [nop,nop,TS val 9244977 ecr 1217020707], length 0
11:03:50.114938 IP 192.168.178.74.38024 > 10.100.1.2.http: Flags [P.], seq 1:293, ack 1, win 502, options [nop,nop,TS val 1217020707 ecr 9244964], length 292: HTTP: GET /app/polyfills.4275a7f2798b637a6613.js HTTP/1.1
11:03:50.114958 IP 10.100.1.2.http > 192.168.178.74.38024: Flags [.], ack 293, win 502, options [nop,nop,TS val 9244977 ecr 1217020707], length 0
11:03:50.115107 IP 192.168.178.74.38040 > 10.100.1.2.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 1217020708 ecr 9244965], length 0
11:03:50.115270 IP 10.100.1.2.http > 192.168.178.74.38020: Flags [P.], seq 1:1490, ack 304, win 502, options [nop,nop,TS val 9244977 ecr 1217020707], length 1489: HTTP: HTTP/1.1 200 OK
11:03:50.115279 IP 192.168.178.74.38040 > 10.100.1.2.http: Flags [P.], seq 1:290, ack 1, win 502, options [nop,nop,TS val 1217020708 ecr 9244965], length 289: HTTP: GET /app/vendor.351bf938ddfd5e202e99.js HTTP/1.1
11:03:50.115292 IP 10.100.1.2.http > 192.168.178.74.38040: Flags [.], ack 290, win 502, options [nop,nop,TS val 9244977 ecr 1217020708], length 0
11:03:50.115408 IP 192.168.178.74.38056 > 10.100.1.2.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 1217020708 ecr 9244965], length 0
11:03:50.115679 IP 10.100.1.2.http > 192.168.178.74.38024: Flags [P.], seq 1:1482, ack 293, win 502, options [nop,nop,TS val 9244977 ecr 1217020707], length 1481: HTTP: HTTP/1.1 200 OK
11:03:50.115737 IP 192.168.178.74.38056 > 10.100.1.2.http: Flags [P.], seq 1:288, ack 1, win 502, options [nop,nop,TS val 1217020708 ecr 9244965], length 287: HTTP: GET /app/main.b193f247301cc37bf60b.js HTTP/1.1
11:03:50.115762 IP 10.100.1.2.http > 192.168.178.74.38056: Flags [.], ack 288, win 502, options [nop,nop,TS val 9244978 ecr 1217020708], length 0
11:03:50.115857 IP 192.168.178.74.53566 > 10.100.1.2.http: Flags [.], ack 4084, win 501, options [nop,nop,TS val 1217020709 ecr 9244966], length 0
11:03:50.115938 IP 192.168.178.74.53566 > 10.100.1.2.http: Flags [.], ack 4194, win 501, options [nop,nop,TS val 1217020709 ecr 9244966], length 0
11:03:50.115938 IP 192.168.178.74.53566 > 10.100.1.2.http: Flags [.], ack 4665, win 501, options [nop,nop,TS val 1217020710 ecr 9244966], length 0
11:03:50.116172 IP 10.100.1.2.http > 192.168.178.74.38040: Flags [P.], seq 1:1483, ack 290, win 502, options [nop,nop,TS val 9244978 ecr 1217020708], length 1482: HTTP: HTTP/1.1 200 OK
11:03:50.116266 IP 10.100.1.2.http > 192.168.178.74.38020: Flags [P.], seq 1490:4226, ack 304, win 502, options [nop,nop,TS val 9244978 ecr 1217020707], length 2736: HTTP
11:03:50.116277 IP 10.100.1.2.http > 192.168.178.74.38020: Flags [P.], seq 4226:5586, ack 304, win 502, options [nop,nop,TS val 9244978 ecr 1217020707], length 1360: HTTP
11:03:50.116326 IP 10.100.1.2.http > 192.168.178.74.38020: Flags [P.], seq 5586:8322, ack 304, win 502, options [nop,nop,TS val 9244978 ecr 1217020707], length 2736: HTTP
11:03:50.116334 IP 10.100.1.2.http > 192.168.178.74.38020: Flags [P.], seq 8322:9682, ack 304, win 502, options [nop,nop,TS val 9244978 ecr 1217020707], length 1360: HTTP
11:03:50.116413 IP 10.100.1.2.http > 192.168.178.74.38024: Flags [P.], seq 1482:4218, ack 293, win 502, options [nop,nop,TS val 9244978 ecr 1217020707], length 2736: HTTP
11:03:50.116422 IP 10.100.1.2.http > 192.168.178.74.38024: Flags [P.], seq 4218:5578, ack 293, win 502, options [nop,nop,TS val 9244978 ecr 1217020707], length 1360: HTTP
11:03:50.116496 IP 10.100.1.2.http > 192.168.178.74.38024: Flags [P.], seq 5578:8314, ack 293, win 502, options [nop,nop,TS val 9244978 ecr 1217020707], length 2736: HTTP
11:03:50.116507 IP 10.100.1.2.http > 192.168.178.74.38024: Flags [P.], seq 8314:9674, ack 293, win 502, options [nop,nop,TS val 9244978 ecr 1217020707], length 1360: HTTP
11:03:50.116557 IP 10.100.1.2.http > 192.168.178.74.38020: Flags [P.], seq 9682:12418, ack 304, win 502, options [nop,nop,TS val 9244978 ecr 1217020707], length 2736: HTTP

I'm not 100% sure if it is a Docker or Opnsense related issue.
Any ideas how I can troubleshoot this issue?
Thanks and BR
Michael

[elcoto]

(I don't really have experience with these tools you are using but I will just put my 2 cents since nobody has responded)

If everytime you spin up a new container this begins to present itself and then goes away if it is MTU related then does this mean that the container host corrects the MTU after a bit of time and the resets the MTU? And you have ruled out it's not some weird routing thing that is breaking when you spin up a container momentarily,?(Again, I don't fully understand what it happening and how the routing to the containers is done) Can you see anything funny with the docker containers interface tcpdump?

I don't know if mtu's are adaptive and can change by default.

ifconfig ens10 | grep -i mtu
ifconfig ipvlan(??) | grep -i mtu

and check if this changes when a container is created. And then perhaps as a 'workaround' lower these values?

But again I am just wildly guessing.