Messages

[Monviech]

To Monviech

Maybe the os-caddy plugin fits your needs.

Did I undrestand correct, that it require from me to move all the configuration from nginx to the new http-server? It sounds quite complex task, because I use nginx not only on OPNsense side but in my local environment too.
I think it worth considering, but also requires to weigh everything -- it may take time and effort for migration.
Anyway, thanks for suggestion.

[Fright]

To Fright

I still don't understand why you think using an XFF header in the '<arbitrary_external_server_address>, $proxy_add_x_forwarded_for' format satisfies the standard way if it assumes the '<client>, <proxy1>, <proxy2>' format.
if there was no NAT in front of the nginx, the external address of the plugin would also not be included in this header

Look at this from the point of usage. If this is the only behavior that is intended, then why give the option of configuration at all for these headers? Right? They can be nailed to one option then.
"You can buy a car of any color as long as it is black." (С)  ;)

Anyway -- the flexibility is ensured by configurability.

[Fright]

to Fright
I didn't agree with some parts of your conclusions:

- this is an incorrect use of de-facto standard headers
These headers are intended to show the client where the original server/proxy resides in a heterogeneous systems. I don't have anything super special. Just a NUT with port forwarding.

-you could use your own headers for this setup
No, I can't. You are talking about some abstract suggestion, that 3rd party product somehow should handle non-standard headers. While we already have the de-facto standard headers for this purpose.

-you always have the option to use a completely hand-written Location (with the desired headers) and use it withe GUI-configured server via server _post-hook
Does it survive the update/upgrade of OPNsense? I mean -- is this a standard way for OPNsense?

-a request for such changes has little chance of being merged imho
What are the changes your are talking? Maybe we look at this problem from different points?
My suggestion is just give a possibility to the user switch on/off the logic of OPNsense, where it adds these headers. Just on/off -- no more.

[Software/Hardware specs:]

I expected much more speed.
I've conducted the test with iperf3 and wired connection.
In short: OPNsense gives about 100/90 Mbits/s, but it is too low numbers in comparison with plain linux machine and openvpn package (~650Mbit/s)

Software/Hardware specs:

Versions     OPNsense 24.1.3_1-amd64
FreeBSD  13.2-RELEASE-p10
OpenSSL  3.0.13
CPU type    Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz (with AES-NI)
1 GB port (yes, I tested it without vpn)



OpenSSL test:

Code Select Expand


root@OPNsense:~ # /usr/local/bin/openssl speed -evp aes-256-gcm
Doing AES-256-GCM for 3s on 16 size blocks: 30204438 AES-256-GCM's in 3.00s
Doing AES-256-GCM for 3s on 64 size blocks: 23236206 AES-256-GCM's in 2.99s
Doing AES-256-GCM for 3s on 256 size blocks: 15034680 AES-256-GCM's in 2.98s
Doing AES-256-GCM for 3s on 1024 size blocks: 7182913 AES-256-GCM's in 2.99s
Doing AES-256-GCM for 3s on 8192 size blocks: 1207259 AES-256-GCM's in 2.99s
Doing AES-256-GCM for 3s on 16384 size blocks: 607857 AES-256-GCM's in 2.98s
version: 3.0.13
built on: Mon Feb  5 20:57:43 2024 UTC
options: bn(64,64)
compiler: cc -fPIC -pthread -Wa,--noexecstack -Qunused-arguments -O2 -pipe  -fstack-protector-strong -fno-strict-aliasing -DL_ENDIAN -DOPENSSL_PIC -D_THREAD_SAFE -D_REENTRANT -DOPENSSL_BUILDING_OPENSSL -DNDEBUG
CPUINFO: OPENSSL_ia32cap=0xfffab2234f8bffff:0x4009c47ab
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
AES-256-GCM     161090.34k   497000.00k  1289676.42k  2458169.12k  3305229.28k  3345849.14k


iperf sends packets with maximum allowed size, so consider the speed of AES-256-GCM for 1024 bytes -- it is 2,458,169.12k (!) -- quite enough.


iperf3 test results:

Code Select Expand


$ iperf3 -c 192.168.8.1
Connecting to host 192.168.8.1, port 5201
[  5] local 192.168.8.5 port 39024 connected to 192.168.8.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  9.99 MBytes  83.8 Mbits/sec    4    181 KBytes       
[  5]   1.00-2.00   sec  11.0 MBytes  92.5 Mbits/sec    2    163 KBytes       
[  5]   2.00-3.00   sec  10.7 MBytes  89.4 Mbits/sec   23   76.2 KBytes       
[  5]   3.00-4.00   sec  9.74 MBytes  81.7 Mbits/sec    1    113 KBytes       
[  5]   4.00-5.00   sec  10.8 MBytes  90.4 Mbits/sec   45    124 KBytes       
[  5]   5.00-6.00   sec  11.3 MBytes  95.1 Mbits/sec   21    149 KBytes       
[  5]   6.00-7.00   sec  12.2 MBytes   102 Mbits/sec    0    200 KBytes       
[  5]   7.00-8.00   sec  10.6 MBytes  88.9 Mbits/sec   55   93.3 KBytes       
[  5]   8.00-9.00   sec  10.4 MBytes  87.4 Mbits/sec   61    113 KBytes       
[  5]   9.00-10.00  sec  10.5 MBytes  87.9 Mbits/sec   12    112 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   107 MBytes  89.9 Mbits/sec  224             sender
[  5]   0.00-10.01  sec   106 MBytes  89.3 Mbits/sec                  receiver


Code Select Expand


$ iperf3 --reverse -c 192.168.8.1
Connecting to host 192.168.8.1, port 5201
Reverse mode, remote host 192.168.8.1 is sending
[  5] local 192.168.8.5 port 42492 connected to 192.168.8.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  11.8 MBytes  99.3 Mbits/sec                 
[  5]   1.00-2.00   sec  10.8 MBytes  90.5 Mbits/sec                 
[  5]   2.00-3.00   sec  12.2 MBytes   102 Mbits/sec                 
[  5]   3.00-4.00   sec  11.3 MBytes  94.8 Mbits/sec                 
[  5]   4.00-5.00   sec  12.7 MBytes   106 Mbits/sec                 
[  5]   5.00-6.00   sec  12.9 MBytes   108 Mbits/sec                 
[  5]   6.00-7.00   sec  12.3 MBytes   103 Mbits/sec                 
[  5]   7.00-8.00   sec  13.2 MBytes   111 Mbits/sec                 
[  5]   8.00-9.00   sec  10.4 MBytes  87.2 Mbits/sec                 
[  5]   9.00-10.00  sec  11.3 MBytes  94.6 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   120 MBytes   100 Mbits/sec  106             sender
[  5]   0.00-10.00  sec   119 MBytes  99.7 Mbits/sec                  receiver


[tunable]

I conducted several tests after I've changed values in tunable.
Increased available window size (sendbuf/recvbuf), increased start size for sendspace/recvspace, disable net.inet.tcp.tso, disable security flags vm.pmap.pti and hw.ibrs_disable, enabled and tuned the RSS.

I've reached the maximum speed for real network cards at 1Gbit/s (they are passed through as real PCIe devices).

But the speed between Host and OPNsense VM is still low.
Inner (loopback) speed test shows about 18Gbit/s (almost what I expected).
Host <-> OPNsense test shows 1.5-1.7Gbit/s (but should also be about 17-18Gbit/s).


The question is to the developers of OPNsense. Is it a limit of FreeBSD overt virtio driver, or there are other options I can change in order to get closer to the expected speed ?
(i've already used all the hints from your documentation from https://docs.opnsense.org/troubleshooting/performance.html)

and one more observation:
In all tests the transfer speed from any outer machine (real or virtual) to the OPNsense is higher than the speed for reverse direction test, when the data is transferred to any machine from the OPNsense.

[mrElement]

mrElement, thanks for the reply.
I described everything correctly. As it is. Two types of NICs. Some of them are passed through, and other ones is just a bridged virtual interfaces.

But it doesn't matter. Even with one virtual interface the speed is quite low.

upd.
I know that it is the most frequent problem.
The simple advices (disable hardware offloading) helps only in quite low borders -- not the 250Mbit/s, but 1.5Gbit/s. It is not a game changer, maybe only in combination with something else.
I also increased max buffer sizes for kernel and for tcp specifically. Up to 8MiB.
But the difference is 15x !!  Instead of about 20Gbit/s i receives only 1.3Gbit.

[Environment]

Environment

Code Select Expand


Host:
Proxmox Virtual Environment 8.1.4

VM:
OPNsense 24.1.3_1-amd64
FreeBSD 13.2-RELEASE-p10
OpenSSL 3.0.13



For OPNsense VM several network cards are passed through as real PCIe devices. They are combined in bridge with a virtual interface (VirtIO paravirtualized) from the host machine.
Also I disabled hardware offloading in `Interfaces -> Settings`:

• Hardware CRC    Disable hardware checksum offload
• Hardware TSO    Disable hardware TCP segmentation offload
• Hardware LRO    Disable hardware large receive offload
• VLAN Hardware Filtering    Disable

- localhost to localhost gives ~40Gbit/s
- Linux VM to Host shows about 20Gbit/s speed in iperf3 simple test with default parameters.
- While the OPNsense VM (to the same host) shows only about 1.3-1.5Gbit/s.
- Doing the test with loopback interface inside OPNsense gives the number about 16-17Gbit/s.

Сould you tell me which way to look?

Cannot disable setting of special Forward headers for Nginx.

I'm talking about these ones:

Code Select Expand


proxy_set_header X-Real-IP $remote_addr;                                   
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;               
proxy_set_header X-Forwarded-Proto $scheme;                                 
proxy_set_header X-Forwarded-Port $server_port;                             
proxy_set_header X-Forwarded-Host $host;



My goal is to set my own values, but i cannot override them, because any `proxy_set_header` direcitve which is set in `GUID_post/*.conf` just adds new header, not override it.

I already created a similar topic for the old OPNsense branch, but the discussion doesn't come up to some solution or workaround. That is why, I named the new topic with more general subject name.

I think it shouldn't be a too complex logic to create switch e.g. for the server/location which may disable automatic adding of that headers.

Original post:

https://forum.opnsense.org/index.php?topic=37464.0

It would be nice to have some  working approach that can survive system reboots and configuration updates via GUI.
Maybe some of OPNsense developers have an idea?

It will work only if you have the full control over the server code in order to change it behavior so that it starts working with the new header.
But all ready products use the standard headers.

I don't know the actual architecture of opnsense, but I don't understand what is the core problem to add the possibility for configuration X-Forwarded headers. Let it be some checkbox -- use default behavior / use custom behavior (over pre/post files)

I think it would be better to allow to set/override the values for X-Forwarded- header over GUI at least in config.
For now it doesn't work.

[/usr/local/etc/nginx/nginx.conf]

> I think this is an incorrect use of the XFF header (intended to convey the client address)

I try to transfer not the client address, but the correct server address to the underlying backend.
I also tried headers-more module.
I is enabled in Nginx -> General Settings -> Global HTTP settings.
Then i found UUID in /usr/local/etc/nginx/nginx.conf for my server.

Part of config

Code Select Expand


server {                                                                       
  listen 127.0.0.1:1080;                                                     
  server_name  my.server.address;
 
  include feecb877-0831-4efa-8005-e26b0dde555f_pre/*.conf;

location  / {                                                                   
     BasicRule wl:19;                                                           
     DeniedUrl "/waf_denied.html";                                               
     autoindex off;                                                             
     http2_push_preload on;                                                     
     proxy_set_header Host $host;                                               
     proxy_http_version 1.1;                                                     
     proxy_set_header Upgrade $http_upgrade;                                     
     proxy_set_header Connection $connection_upgrade;                           
     proxy_buffer_size 8k;                                                       
     proxy_buffers 8 8k;                                                         
     proxy_busy_buffers_size 32k;                                               
     proxy_set_header X-TLS-Cipher $ssl_cipher;                                 
     proxy_set_header X-TLS-Protocol $ssl_protocol;                             
     proxy_set_header X-TLS-SNI-Host $ssl_server_name;                           
     # proxy headers for backend server                                         
     proxy_set_header X-Real-IP $remote_addr;                                   
     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;               
     proxy_set_header X-Forwarded-Proto $scheme;                                 
     proxy_set_header X-Forwarded-Port $server_port;                             
     proxy_set_header X-Forwarded-Host $host;                                   
     proxy_set_header X-TLS-Client-Intercepted $tls_intercepted;                 
     proxy_read_timeout 10s;                                                     
     proxy_send_timeout 10s;                                                     
     proxy_ignore_client_abort on;                                               
     proxy_request_buffering on;                                                 
     proxy_max_temp_file_size 1024m;                                             
     proxy_buffering on;                                                         
     proxy_pass http://upstream5fa240aef8aa4a9f895a4959a40bbbbb;                 
     proxy_hide_header X-Powered-By;                                             
     include 9370ea92-2ab9-45b4-b244-ceb86baa2887_post/*.conf;                   
   }     
  include feecb877-0831-4efa-8005-e26b0dde555f_post/*.conf;
}


I created files:

Code Select Expand


/usr/local/etc/nginx/feecb877-0831-4efa-8005-e26b0dde555f_pre/myheaders.conf
/usr/local/etc/nginx/9370ea92-2ab9-45b4-b244-ceb86baa2887_post/myheaders.conf
/usr/local/etc/nginx/feecb877-0831-4efa-8005-e26b0dde555f_post/myheaders.conf


The content is quite simple -- one line:

Code Select Expand


more_set_input_headers 'X-Forwarded-Port: 80';


Then I restarted nginx and tried simple request via curl. Aaaand it doesn't work. I still continue to receive X-Forwarded-Port: 1080.

e.g. nginx listens on 127.0.0.1:1443, but the real IP and PORT the request is come to is 10.x.x.x:433 (over the NAT).
And for such a case i set OUTER ip/port -- X-Forwarded-Port : 443, and for ip something like
X-Forwarded-For 10.x.x.x, $proxy_add_x_forwarded_for;

The use case is quite simple. Just try to make NAT forwarding to the nginx. And you receive wrong Port or/and IP.
This is just my case. My backend server use that headers for dynamic modifications of sources' URI's.
It would be nice to have a possibility (at least using a pre/post configs) to handle the situation when you need to change the hardcoded values.

Cannot disable setting of special Forward headers for Nginx -> HTTP server ->  Real IP Source.
Even if to choose None, they are still present.

I'm talking about these ones:

Code Select Expand


proxy_set_header X-Real-IP $remote_addr;                                   
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;               
proxy_set_header X-Forwarded-Proto $scheme;                                 
proxy_set_header X-Forwarded-Port $server_port;                             
proxy_set_header X-Forwarded-Host $host;   


My goal is to set my own values, but i cannot override them, because any `proxy_set_header` direcitve which is set in `GUID_post/*.conf` just adds new header, not override it.
This inability brakes my proxy logic. Maybe I miss something, or it is just isn't possible, but (I repeat myself) parameter 'None' for 'Real IP Source' does'n disable 'X-Real-IP' and 'X-Forwarded-' headers.

Is there a possibility to set own headers (or values) for the ones mentioned before?