Messages

hmm as the last ( ??? ) resort I've fired os-sensei package reinstall and now it show session details. Not sure what went wrong

I was pushed by Zenarmor gui to upgrade to 1.11. After successful upgrade when I select anything from pie chart graph in the Blocks tab and select Session details it shows no sessions.
In fact I've already posted same question to sunnyval bug report:

Connections,Threats,partly Web - select Pie chart piece - session detail - show list of sessions
Blocks - select Pie chart piece - session detail - shows empty list of sessions (Loaded records: 0 / 0)
DNS - select bar graph - no sessions in the list either
TLS - shows (green) allowed and (red) blocked sessions

I've done configuration-reset reporting and erase reporting data and deleted that log file and everything in the /usr/local/sensei/log/archive as well (I was personally stunned by the full hdd). After opnsense restart I've got enough free space to run elasticsearch and zenarmor again - but it does not show the above mentioned sessions

What I've done so far:
-several times tryed to restart reporting
-reseted reports and let it delete all the data
-removed elasticsearch pkg and let it reinstall

I can see Alerts - Blocked Conversations Heatmap and Alerts - Blocked Local Hosts and Reasons noprob but I can see no session details

I'm still using OPNsense 21.7.8-amd64 (performance issues with 22.x discused elsewhere here)

I believe I've found something. I turned on the "Log packets matched from the default pass rules put in the ruleset"  on the both installations - so I hope I have the same settings.

on the 21.7 working fast I have this in live fw log

Code Select Expand


vmx_2 Mar 2 11:38:39 10.0.0.20:52258 81.0.212.201:443 tcp PROXY redirect
vmx_2 Mar 2 11:38:39 10.0.0.20:52257 88.86.101.2:443 tcp PROXY redirect
vmx_2 Mar 2 11:38:39 10.0.0.20:52256 81.0.212.201:443 tcp PROXY redirect
vmx_2 Mar 2 11:38:39 10.0.0.20:52255 52.210.251.145:443 tcp PROXY redirect
vmx_2 Mar 2 11:38:39 10.0.0.20:52254 52.210.251.145:443 tcp PROXY redirect
vmx_2 Mar 2 11:38:39 10.0.0.20:52253 77.78.109.86:443 tcp PROXY redirect
vmx_2 Mar 2 11:38:39 10.0.0.20:52252 77.78.109.86:443 tcp PROXY redirect
vmx_2 Mar 2 11:38:35 10.0.0.20:52251 142.251.36.67:443 tcp PROXY redirect
vmx_2 Mar 2 11:38:26 10.0.0.20:52250 52.210.251.145:443 tcp PROXY redirect


but on the OPNsense 22.1.2

Code Select Expand


vmx_2 2022-03-02T11:31:51 10.0.0.20:52049 81.0.212.201:443 tcp let out anything from firewall host itself
vmx_2 2022-03-02T11:31:51 10.0.0.20:52049 81.0.212.201:443 tcp PROXY redirect
vmx_2 2022-03-02T11:31:51 10.0.0.20:52048 88.86.101.2:443 tcp let out anything from firewall host itself
vmx_2 2022-03-02T11:31:51 10.0.0.20:52048 88.86.101.2:443 tcp PROXY redirect
vmx_2 2022-03-02T11:31:51 10.0.0.20:52047 18.203.13.148:443 tcp let out anything from firewall host itself
vmx_2 2022-03-02T11:31:51 10.0.0.20:52047 18.203.13.148:443 tcp PROXY redirect
vmx_2 2022-03-02T11:31:51 10.0.0.20:52046 18.203.13.148:443 tcp let out anything from firewall host itself
vmx_2 2022-03-02T11:31:51 10.0.0.20:52046 18.203.13.148:443 tcp PROXY redirect
vmx_2 2022-03-02T11:31:51 10.0.0.20:52045 77.78.109.86:443 tcp let out anything from firewall host itself
vmx_2 2022-03-02T11:31:51 10.0.0.20:52045 77.78.109.86:443 tcp PROXY redirect
vmx_2 2022-03-02T11:31:51 10.0.0.20:52044 77.78.109.86:443 tcp let out anything from firewall host itself
vmx_2 2022-03-02T11:31:51 10.0.0.20:52044 77.78.109.86:443 tcp PROXY redirect


PBR rule has quick ticked on and sloppy states set.

I think firewall sends data to the proxy as well to (default) routing - for example
one logged packet to proxy: seq   875304436
next line: seq   875304436 (let out anything from firewall host itself)

I don't see any let out anything from firewall host itself rules in the live log view on the 21.7.x installation.

Am I missing something on the 22.x? As I wrote - I have to installations as VMs - I have recently upgraded both, but after this difficulties, I revert my main rig to 21.7 and used the other one as a guinea pig for 22.1

I've wiresharked both installations from my test client and I see many TCP out-of-orders and TCP duplicates (mainly ACK, but also some PSHs). I tried to analyze it by L2 headers - to see what host sends the data - it seems req goes to set default GW mac and data returns from the proxy. Without the PBR rules, req goes to default gw mac and returns the same way. But with the 22.1 as the dGW I see many TCP retransmits and out-of-order ACKs etc.

it's unchecked in the both versions - needles to say proxy is not running inside opnsense - it's solitary instalation in the vm. opnsense has only one WAN

If I disable PBR rule data goes to opnsense dGW and it's fast up and down. If I set proxy in my browser - data goes fast as well. But when I use the PBR rule - data should (should - but see the test below) go to the proxy and then out via it's dGW - download speed is ok, but upload is only 10Mbps.
And all it started with 22.1 upgrade. I let my other opnsense instance upgraded to 22.1 and reinstalled my main rig to 21.7.8 and my speeds are ok.

In fact just now I did this test - client with dGW as (I'm using iptraf-ng on the proxy):
opnsense 21.7.8 - I can confirm hundreds of Mbps (ie. 914/920Mbps) up and down going through the proxy iface (10k pps both sides)

opnsense 22.1_1 - I can read hundreds Mbps out from proxy (10k pps both sides) and then 40Mbps multistream upload (rychlost.cz) on the client, but only <500kbps (about 50pps) on the proxy in (ie. output of client).

So with opnsense 22.1 - data seems to go up elsewhere - but why and how?

no - proxy has it's own dGW - it's not sending data back to opnsense to route it out to the internet. What's more weird is that with 21.7.8 it works fast - I had no reason to question my PBR rule so far

1) data without policy based route  -  full speed (outgoing from router)
2) data send directly to proxy via browser/system setting - full speed
3) data targeted to proxy GW - download full, upload 10Mbps

OPNsense 22.1.1_1 - not solved

I have recently switched from 21.7.8 to 22.1 just to found out upload via our PBRed squid proxy is poor (just 10Mbps per single server). We use proxy for http/https inspection and slight http cacheing, terminating ads and malware connections - so realy no real MITM for users.

Not changed my configuration - so my rule only changes gateway on the matching packets. This worked fine for some time - as my download and upload speed was about right for ISP speeds (1Gbps).  But when upgraded to 22.1 it went down and some web services are malfunctioning (ie. uploading PDF forms to the goverment servers etc. timeouts etc.)

My opnsense runs on ESXi 6.7U2 host with intel I210 and I350 with igbn driver. opnsense has vnics as vmxnet3. LANs are on a single vswitch with esxi tagged vlans, wan is on a separate vswitch. All MTUs set to 1500. All offloading in opnsense disabled. Zenarmor for all the LAN ifaces and surricata for WAN. I have one interface with VLANs in opnsense - parent interface assigned and enabled. Only tunables were queue settings for vmx, but it have been dropped recently. I know (from this phorum as well) it has been replaced by dev.vmx.0.iflib.... keys anyway.

I have tested all connections with iperf3 - client to fw, client to proxy, fw to proxy with similar results ~900Mbps TCP. While testing UDP to proxy - everything went ok, but testing against opnsense printouts

Code Select Expand


iperf3: OUT OF ORDER - incoming packet = 54887 and received packet = 0 AND SP = 277090
iperf3: OUT OF ORDER - incoming packet = 221983 and received packet = 0 AND SP = 277226
iperf3: OUT OF ORDER - incoming packet = 178360 and received packet = 0 AND SP = 277228
iperf3: OUT OF ORDER - incoming packet = 229939 and received packet = 0 AND SP = 281851
iperf3: OUT OF ORDER - incoming packet = 266111 and received packet = 0 AND SP = 282071
iperf3: OUT OF ORDER - incoming packet = 222533 and received packet = 0 AND SP = 282784
iperf3: OUT OF ORDER - incoming packet = 208849 and received packet = 0 AND SP = 282826
iperf3: OUT OF ORDER - incoming packet = 258064 and received packet = 0 AND SP = 283012
iperf3: OUT OF ORDER - incoming packet = 258554 and received packet = 0 AND SP = 283081


When I set up the client to directly use the proxy - everything works fine (930/930Mbps in inet tests). But if I put it via opnsense (ie. transparent mode) I get 930Mbps/8Mbps (8Mbps single server, about 30Mbps multiple servers).

So I went haywire and looked everywhere - but as I have test installation avaible on the another server (actually upgraded to 22.1 as well) I was able to check the behaviour is the same from the other one.

Later today I've grabbed 21.7 ISO and reinstalled it over my test instalation cloned VM, imported config and tested it. Now my incoming and outgoing rates on proxy can easily reach 900Mbps.

So I think I sorted out my infrastructure, cables, VLANs, proxy instalation and it's settings (deb10) and firewall rules. But I understand opnsense underwent mayor OS version change - but the same configuration is not working well anymore.

What are the settings I should look at? Do I miss something? Not realy used to FBSD systems - but I did not find anything useful -  actualy not here...

I'm going to let my testing installation VM clone run 21.7 (I must remember to take snapshots/backups before opnsense upgrade - this is not for the first time I got screwed a little bit) and run my main rig with 22.1 - in case somebody helps me rule it out...

THX

Thanks Franco - this is going to the another direction, unfortunately - I was celebrating too early. I will summarize it in my own guestion-thread

Sorry - apparently red herrings here. In fact - I had very slow (0.2Mbps) upload to web on my PBR squid. Not behaving like this it while not throught the PBRed proxy. So I looked everywhere. One of the last resort was set-up a priority level in the PBR rule up to level 3. Now I have ~930Mbps download and 800-900Mbps upload. Surricata on the wan only and zenarmor on the local ifaces.
In the process I've enabled RSS, net.isr.bindthreads etc. (which I already have had from 21.7).
Nothing seem to work - as we are crucialy dependent on the web - it's a real pain in the a..

Same issue with our esxi 6.7 opnsense box (upload is at tens of Mbps not Gbps like our ISP speed - download is not affected). Is there any solution to stay with 22.1? Is it the same problem like in hyper-v? we are using vmxnet nics

Same for me - did fs check & repair + disabled IDS and IPS, running as VM on vmw 6.7U3 (vmxnet3 nics). Disabling these services did the trick for now