Messages

Hi everybody,

I have an important TCP/IP connection that runs through my opnsense 21.1 installation.
It is an IoT use case (Smartmeter Gateway sending telemetry data back to a Smartmeter Gateway operator).

The connection is from a fixed IP address in my network to a fixed IP address and Port (443) in the operator's datacenter.

What I would like to achieve in terms of monitoring:
- check whether there is exactly one established connection state between those IP addresses with Target Port 443
- check how much data per timespan has been send through the connection since it has been established
- see the point in time or duration since the connection has been established

I can find the desired information using pftop or the 'Inspect' button in the firewall rules menu. Ideally I could get this information out using the Telegraf plugin, but unfortunately is this kind of data not covered by the 'pf' input plugin of Telegraf. Any ideas how I can implement a connection state monitoring?

Thanks,
Martin

VMXNET3, Broadcom 57810 NICs, no LACP, Standard vSwitch
My VM has 1.5 GB of memory now. Before I had 1.0 GB and ran out of memory occasianally which also created very sluggish routing behaviour.

What fixed my problems was:
- Enabling all hardware accelerations/offloads under Interfaces / Settings
- Moving to faster hardware with 10 GbE NICs
- Updating to VMware ESXi 7.0 U1

I can't not 100% tell what really fixed the problems, but they are gone

I have ported the whole thing to ESXi 7.0 U1 on a latest generation i3 Processor. The system is now up and running with 20.7.4 since it has been released. I have assigned more RAM to the VM (1.5 GB instead of 1 GB).
All offloading capabilites have enabled with a Broadcom 57810 10 Gig NIC behind the vSwitches. Runs great so far.

Difference is that I use ESXi 7.0, all HW offloading is enabled and OPNsense is VLAN aware;

This made me think. I enabled all offloading capabilities including VLAN filtering and now the system is up and running for nearly 10 days. Thanks for putting me into the right direction.

Soon I will move the opnsense VM to a new ESXi 7.0 U1 box with a more powerful CPU, a 10 Gbit NIC and more RAM. Lets keep fingers crossed that the stability will stay.

Can you downgrade to version 10??

No, I can't. How can that help? 2.1.9 runs happily with VM version 14.

What version of VM are you running??

VM-Version 14 on ESXi 6.7

After 3 days of uptime the system again stopped forwarding packets.

Obviously it ran out of memory (see screenshot). I restarted all services through SSH, didn't not help.
Rebooted and latencies on the WAN were super high again and the system was extremely sluggish.

I downgraded to 2.1.9 again and all is fine again. There seems to be a problem with 20.7 in vSphere VM and it does not seem related to the vmxnet driver. Any further ideas?

I have now re-upgraded to 20.7.3 and giving it a try with vSphere/vmxnet drivers.

This is the ifconfig output:

Code Select Expand

vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=800028<VLAN_MTU,JUMBO_MTU>
        ether 00:0c:29:2d:79:14
        inet 192.168.179.1 netmask 0xffffff00 broadcast 192.168.179.255
        inet6 fe80::20c:29ff:fe2d:7914%vmx0 prefixlen 64 scopeid 0x1
        inet6 2003:dd:2f1b:f804:20c:29ff:fe2d:7914 prefixlen 64
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>


I never went to the commandline with 20.7 to check the ifconfig output. I will wait until opnsense 20.7 is based at least on FreeBSD 12.1-RELEASE-p8 and then re-try (and then also check / play with ifconfig). Thanks for the hints regarding the vmxnet driver.

I have found these links based on your comment:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236999
https://www.freebsd.org/security/advisories/FreeBSD-EN-20:16.vmx.asc

Fixed in 12.1-RELEASE-p8. But I'm not sure if this really addresses my problem because it happens only when TSO is enabled (which is disabled in the opnsense GUI). Is this what you meant?

Code Select Expand

vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:0c:29:2c:ec:cd
        hwaddr 00:0c:29:2c:ec:cd
        inet 192.168.179.1 netmask 0xffffff00 broadcast 192.168.179.255
        inet6 fe80::20c:29ff:fe2c:eccd%vmx0 prefixlen 64 scopeid 0x1
        inet6 2003:dd:2f26:6004:20c:29ff:fe2c:eccd prefixlen 64
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active

Edit: This is the ifconfig output from opnsense 2.1.9_1

Yes, I have disabled all Hardware Offloading and VLAN Hardware filtering options in Interfaces -> Settings.

Update: with 20.7.2 I retried the version - now with 'VLAN hardware filtering' turned off. Unfortunately the system freezes again within 48h of uptime. I'm back on 20.1 again which is stable on my vSphere host.

When I downgraded from 20.7.1 to 20.1.9_1 my system locked up after 24 hours or so. That was strange because 20.1 was stable and had months of uptime before. I tried to investigate further and found the setting 'VLAN Hardware Filtering' which was turned on by default starting with 20.7 (according to docs). When I took my latest config back from 20.7 to 20.1 I kept it turned on - and the system freezed.

I switched this setting to disabled and my 20.1 instance is running happily again for about 3 days. I will monitor uptime and if it stays stable I will again upgrade to 20.7 and disable VLAN Hardware filtering which seems to be a bad idea in conjunction with VMXNET3 network interfaces on VMware ESXi.

EDIT: After 7 days of uptime everything is still working smooth. Will re-upgrade to 20.7 soon.