[SOLVED] Torrent Issues (Possibly Connections) - Various Releases from 21.7.4

[zandrr]

I have an issue with bad torrent performance (potentially related to concurrent connections) that I have today narrowed as being introduced in 21.7.4 ... apologies in advance as a wall of text ensues.

Been sitting on this issue of mine since I first observed it when 21.7.5 was released, many months ago.
Since then, I have periodically failed at attempts to upgrade software from 27.7.1 to a later release. Eventually rolling back and ignoring it for a long period.

Specs:
Proxmox 6.5 to 7.2 . 1000/500M fibre internet.
OPNsense VM: virtio nics, openvswitch bridges, host cpu, 4G RAM, zfs/ufs storage as scsi.

Events:
- 21.7.5 was attempted initially at time of release, but rolled back due to torrent performance issue.
- 22.1 attempted at release also experienced the same problem, resulting in rollback.
- Tried 22.1.6 this weekend, and same issue apparent.
- Tried to settle on 21.7.8, but same issue apparent.
- Determined issue was at or before 21.7.5, when the issue first presented...

Went through versions incrementally, from 21.7.1 until performance obviously deteriorated, which was in 21.7.4.

Between 21.7.3 and 21.7.4 something broke that drastically affects torrent performance.
Any other issue that might be present aren't as obvious as the performance degradation on torrents.
Speedtests appear fine between known good and bad versions, so theorised likely related to volume of connections.

Torrent seeding:
- Reference constant in 21.7.3; ~230 peers -- ~10000-20000 states
- Reference constant in 22.1.6; ~10 peers -- ~4800-6000 states

The presentation of issue is immediately apparent if I boot a 'good' or 'bad' release.
- If I boot a good version, the load ramps quickly. Most obvious is HDD noise lol.
- If I boot a bad version, there is no significant load and connections remain low.

Reading the 21.7.4 notes, nothing seems particularly obvious to me. RSS might be the most relevant, though the notes suggest the capability is introduced but defaults to disabled. I would expect something in driver, kernel or system to be relevant.

Various configurations have been attempted across 3 PVE hosts with different PHY adapters (mostly Intel); I1219-LM, X520-DA2, I225-LM. A Chelsio T520-CR has also been used.

I have not tested on a baremetal system, nor extensively with non-virtio adapters (though have tried e1000 on an earlier occasion without a positive outcome).
Have not tried the native Linux Bridging on testing systems, but have run it historically on non-afllicted versions with no distinct difference in performance.
Tried the tunable for RSS enablement in 21.7.4 (per the release notes) with no apparent success.

I have two VM's (21.7.3_3 and 22.1.6) on my PVE host that I have been booting between to compare outcomes.
Happy to perform any useful tests where time permits.

The reason I open with an apology is due to the length of time I have sat on a problem, and any potential shortfalls in providing enough useful information.
Hoping there is an issue that may either be obvious, or just overlooked while I was digging for information.

Thank you for your patience.

Also, much gratitude to OPNsense contributors. Have been using OPNsense since 19.1 and definitely want to continue doing so.

[bunchofreeds]

Not sure what might be going on here, but...

You could run wireshark on your torrent VM. Might be worth seeing what's going on at that level.
Also what is your connection type with your ISP (PPPoE or DHCP etc.) PPPoE is a struggle with OPNsense at high speeds.
Are you able to swap in a physical router (Not OPNsense) and test?

I'd also be looking at your ISP and if they are filtering torrenting, as it sounds like your aren't tunnelling this traffic. ISP filtering can cause odd behaviour for this rather than on/off.

[zandrr]

Thanks for the response. I will be progressively (but slowly) working on isolating the cause, if some obvious reason isn't provided by someone with insight.
Trying the path of least resistance while working through possibilities.

ISP is DHCP (port auth) and no protocol/port filtering. I work in my ISP's core network, so can say that with confidence.

The behaviour appears to be correlated with the software aforementioned, so related to some change present in 21.7.4 onward.
Could policy, protocol timers or even driver interactions... a few variables to rule out

A temporary physical router would be feasible, but I already have a baseline working setup in 21.7.1/21.7.3 - do you have another reason to suggest this?
Personally would probably first trend toward a baremetal system running OPN, to attempt to rule out hypervisor or drivers

Appreciate the feedback

[opnfwb]

By chance have you tried a config without the virtio NICs for the OPNsense VM? Virtio continues to be a struggle on FreeBSD. If you can use E1000 or equivalent, I would try that and see if the issue resolves on the newer versions of OPNsense (22.1.x)

[zandrr]

Will be finishing the attempt on E1000 adapters tonight. I started setting it up late last night.
Edit: Sorry, I actually did test it. Went to continue and realised I had already drawn a conclusion. Long couple of days at work...

Tried a physical machine before that as well, with latest 22... and same issue.
It could be the config restoration, so will attempt baremetal again with a known good version (eg 21.7.3) and then, depending on that outcome, focus on reconfiguring from scratch. I want to avoid that if less certain about it making a difference.

Can you elaborate on the virtio struggles you mention; do you have a source I can look in to?
Been running this VM setup for a few years now. Granted, possibly been lucky enough to avoid obvious problems until now.
Had 1Gb fibre for longer than I have been running OPNsense

[opnfwb]

In my specific case it's been bad throughput or inconsistent throughput using virtio NICs on both pfSense and OPNsense.

This is always when virtio NICs are involved (or vmxnet3 in the case of VMware, although those are better in my use cases). For HyperV Gen2 VMs, I've had to disable RSC to get consistent throughput. Testing on VirtualBox with virtio NICs still yields bad throughput however, I only test there, I'd never actually run a firewall there in a production state.

Another thing that helped get my VM throughput up was setting the tunable

Code: [Select]

hw.ibrs_disable to 1, thus disabling it. By default OPNsense enables this and it is quite costly in my VM use cases. pfSense and FreeBSD ship with this disabled by default, and may result in their perceived "faster" performance. This would be worth trying as long as you understand the potential security implications in doing so.
More info here: https://docs.opnsense.org/troubleshooting/hardening.html

[zandrr]

Hm think I've found my issue. Looks like something with my Unbound overrides, which impacts my alias parsing, which in turn impacts some firewall rules.

- If I use the upgrade feature (eg 21.7.3 -> 21.7.4) my aliases that are dependent on Unbound overrides don't appear to resolve.
- If I import my config on a later release, my Unbound overrides aren't imported, which makes my host aliases useless and breaks firewall rules.

A lot of my firewall rules use aliases that resolve DNS for hosts. I had a justification when I set them up that way eons ago (a lot of VM lab work). Can definitely change my approach now.

Makes sense now I step back, and somewhat embarrassing. No resolution means my FW rules aren't matched, which explains the lack of connections and torrent throughput...
I blame not tackling the issue 6+ months ago when I first encountered it which clouded a methodical approach to troubleshooting.

Will re-evaluate my config since a lot of stuff is stale. Then will move on to a fresh 22.1 install.
Sorry to drag you through my journey. I am somewhat happy the issue doesn't appear to be something worse.