BAD STATE

[simonmcn]

I'm at a loss trying to understand my issues and could do with some help please.

I am running opnsense on Proxmox.  I'm passing through a trunk with vlans 10,20,30,40,50,60.  I also have another ethernet that I use for a PPPOE connection.

My internet works, seems reliable and has no issue from my main vlans of 30 (servers) and 50 (wifi).

I created, for each vlan a firewall rule allowing access to everything.  I just wanted it to work, then the plan was to set up the rules, and disable the global rule.

But I'm getting hung ssh, nfs and cifs connections.  Everything to the internet is fine, but inter vlan just seems to work intermittently and badly.

Any advice, ideas on how to diagnose please ?

[bartjsmit]

Working ping but hanging bulkier traffic can be caused by MTU mismatches. Perhaps some inconsistent jumbo frames?

[Seimus]

Maybe I dont understand correctly but

I am running opnsense on Proxmox.  I'm passing through a trunk with vlans 10,20,30,40,50,60.  I also have another ethernet that I use for a PPPOE connection.

My internet works, seems reliable and has no issue from my main vlans of 30 (servers) and 50 (wifi).

But then you show rules for VLAN50 as well live log.

So what exactly has the problem here? The VLAN50 or the other VLANs 10,20,40,60?
Can you ping from a device in these VLANs/Subnets to the GWs?
Do you have unique subnets for these VLANs on OPNsense?
Do you have proper MASK configured on devices in these VLANs?

Regards,
S.

[simonmcn]

Working ping but hanging bulkier traffic can be caused by MTU mismatches. Perhaps some inconsistent jumbo frames?

All my MTU should be 1500 or less.  It's certainly less for the wireguard and PPPOE interfaces.   The rest is configured as 1500

[simonmcn]

Maybe I dont understand correctly but

I am running opnsense on Proxmox.  I'm passing through a trunk with vlans 10,20,30,40,50,60.  I also have another ethernet that I use for a PPPOE connection.

My internet works, seems reliable and has no issue from my main vlans of 30 (servers) and 50 (wifi).

But then you show rules for VLAN50 as well live log.

So what exactly has the problem here? The VLAN50 or the other VLANs 10,20,40,60?
Can you ping from a device in these VLANs/Subnets to the GWs?
Do you have unique subnets for these VLANs on OPNsense?
Do you have proper MASK configured on devices in these VLANs?

Regards,
S.

I'm having issues with intervlan traffic.  Going from 50 to 30.  All traffic 30-30 which doesn't touch opnsense is fine.  and surprisingly web traffic from 50-30 seems to work fine, but it just may be that the packet size is small, or the session state different to cifs/nfs/ssh

All my masks are /24, each vlan is on a different subnet 10.150.10.0/24 for vlan 10, 10.150.30.0/24 for vlan 30 etc

All devices can ping both the gateways and the upstream servers.

All the devices can see each other, it seems that the session state is being reset by the pf I think perhaps.

[simonmcn]

I've just tested it.  if I turn the packet filter off.  All my problems go away.

[simonmcn]

more information:
root@OPNsense:~ # pfctl -si
Status: Enabled for 0 days 02:58:23             Debug: Loud

Interface Stats for vtnet0_vlan30     IPv4             IPv6
  Bytes In                        78388748                0
  Bytes Out                     1494252352                0
  Packets In
    Passed                          828852                0
    Blocked                            162                0
  Packets Out
    Passed                         1374002                0
    Blocked                              0                0

State Table                          Total             Rate
  current entries                     1768               
  searches                        45657736         4265.9/s
  inserts                           653014           61.0/s
  removals                          651256           60.8/s
Counters
  match                             728116           68.0/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                                  0            0.0/s
  normalize                              4            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                             0            0.0/s
  ip-option                              0            0.0/s
  proto-cksum                            0            0.0/s
  state-mismatch                     37898            3.5/s
  state-insert                          10            0.0/s
  state-limit                            0            0.0/s
  src-limit                              0            0.0/s
  synproxy                               0            0.0/s
  map-failed                             0            0.0/s

[Seimus]

You have there > 

state-mismatch                     37898            3.5/s

If a FW sees out of Order for TCP he will block it, TCP based traffic can pass thru a FW only after a Handshake is established.

S > D: TCP S
D > S: TCP SA
S > D: TCP A

Check the Live log. Create a filter with a specific source and destination from which you will test from to. Then if you see a session appear that is blocked, click the magnify glass and check the TCP Flags.

if there is really a TCP out of order it means you traffic is leaking somewhere or there is asymmetrical routing.

Regards,
S.