nginx plugin

[ccesario]

Thanks @Fabian, I will test it.

Currently we facing a problem with HTTP servers.
When we add a second HTTP server the service does no start.

This is the code generated.

Code Select Expand

load_module /usr/local/libexec/nginx/ngx_stream_module.so;
load_module /usr/local/libexec/nginx/ngx_http_naxsi_module.so;
load_module /usr/local/libexec/nginx/ngx_mail_module.so;
load_module /usr/local/libexec/nginx/ngx_http_brotli_filter_module.so;
load_module /usr/local/libexec/nginx/ngx_http_brotli_static_module.so;

user www staff;
worker_processes  1;

error_log  /var/log/nginx/error.log;

events {
    worker_connections  1024;
}

http {
include       mime.types;

MainRule wl:19;


log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';
log_format  anonymized  ':: - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';

#tcp_nopush     on;

# 200M should be big enough for file servers etc.
client_max_body_size 200M;
brotli_static on;
brotli on;
gzip_static on;
gzip on;
server_tokens off;
sendfile Off;
default_type  application/octet-stream;
keepalive_timeout 60;

# TODO add when core is ready for allowing nginx to serve the web interface
# include nginx_web.conf;

# UPSTREAM SERVERS
upstream upstream16d9678a48cf438b8f71617150c53c4c {
server 10.15.0.9:80 weight=1 max_conns=500 max_fails=10 fail_timeout=10;

}

server {
    listen  [::]:80 ipv6only=off;
    # proxy headers for backend server
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    server_name  server1.com;
    charset utf-8;
    access_log  /var/log/nginx/server1.com.access.log main;
    error_log  /var/log/nginx/server1.com.error.log;
    #include tls.conf;
    error_page 404 /opnsense_error_404.html;
    error_page 500 501 502 503 504 /opnsense_server_error.html;
    # location to ban the host permanently
    set $naxsi_extensive_log 0;
    location @permanentban {
        access_log /var/log/nginx/permanentban.access.log main;
        internal;
        add_header Content-Type text/plain;
        add_header Charset utf-8;
        return 403 "You got banned permanently from this server.";
    }
    error_page 418 = @permanentban;
    location /opnsense_server_error.html {
        internal;
        root /usr/local/etc/nginx/views;
    }
    location /opnsense_error_404.html {
        internal;
        root /usr/local/etc/nginx/views;
    }
    location /waf_denied.html {
        root /usr/local/etc/nginx/views;
        access_log /var/log/nginx/waf_denied.access.log main;
    }
    location ^~ /.well-known/acme-challenge/ {
        default_type "text/plain";
        root /var/etc/acme-client/challenges;
    }
    # block based on User Agents - stuff I have found over the years in my server log
    if ($http_user_agent ~* Python-urllib|Nmap|python-requests|libwww-perl|MJ12bot|Jorgee|fasthttp|libwww|Telesphoreo|A6-Indexer|ltx71|okhttp|ZmEu) {
      return 418;
    }
    if ($http_user_agent ~ "Indy\sLibrary|Morfeus Fucking Scanner")
    {
      return 418;
    }

    location = /opnsense-report-csp-violation {
      include       fastcgi_params;
      fastcgi_param QUERY_STRING $query_string;
      fastcgi_param SCRIPT_FILENAME /usr/local/opnsense/scripts/nginx/csp_report.php;
      fastcgi_param TLS-Cipher $ssl_cipher;
      fastcgi_param TLS-Protocol $ssl_protocol;
      fastcgi_param TLS-SNI-Host $ssl_server_name;
      fastcgi_param SERVER-UUID "63cc87ec-228d-4bbd-a695-37118e761e8a";
      fastcgi_intercept_errors on;
      fastcgi_pass  unix:/var/run/php-webgui.socket;
    }
    location /opnsense-auth-request {
      internal;
      fastcgi_pass  unix:/var/run/php-webgui.socket;
      fastcgi_index index.php;
      fastcgi_param TLS-Cipher $ssl_cipher;
      fastcgi_param TLS-Protocol $ssl_protocol;
      fastcgi_param TLS-SNI-Host $ssl_server_name;
      fastcgi_param Original-URI $request_uri;
      fastcgi_param Original-HOST $host;
      fastcgi_param SERVER-UUID "63cc87ec-228d-4bbd-a695-37118e761e8a";
      fastcgi_param SCRIPT_FILENAME  /usr/local/opnsense/scripts/nginx/ngx_auth.php;
      fastcgi_intercept_errors on;
      include        fastcgi_params;
    }


}

server {
    listen  [::]:80 ipv6only=off;
    # proxy headers for backend server
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    server_name  server2.com;
    charset utf-8;
    access_log  /var/log/nginx/server2.com.access.log main;
    error_log  /var/log/nginx/server2.com.error.log;
    #include tls.conf;
    error_page 404 /opnsense_error_404.html;
    error_page 500 501 502 503 504 /opnsense_server_error.html;
    # location to ban the host permanently
    set $naxsi_extensive_log 0;
    location @permanentban {
        access_log /var/log/nginx/permanentban.access.log main;
        internal;
        add_header Content-Type text/plain;
        add_header Charset utf-8;
        return 403 "You got banned permanently from this server.";
    }
    error_page 418 = @permanentban;
    location /opnsense_server_error.html {
        internal;
        root /usr/local/etc/nginx/views;
    }
    location /opnsense_error_404.html {
        internal;
        root /usr/local/etc/nginx/views;
    }
    location /waf_denied.html {
        root /usr/local/etc/nginx/views;
        access_log /var/log/nginx/waf_denied.access.log main;
    }
    location ^~ /.well-known/acme-challenge/ {
        default_type "text/plain";
        root /var/etc/acme-client/challenges;
    }
    # block based on User Agents - stuff I have found over the years in my server log
    if ($http_user_agent ~* Python-urllib|Nmap|python-requests|libwww-perl|MJ12bot|Jorgee|fasthttp|libwww|Telesphoreo|A6-Indexer|ltx71|okhttp|ZmEu) {
      return 418;
    }
    if ($http_user_agent ~ "Indy\sLibrary|Morfeus Fucking Scanner")
    {
      return 418;
    }

    location = /opnsense-report-csp-violation {
      include       fastcgi_params;
      fastcgi_param QUERY_STRING $query_string;
      fastcgi_param SCRIPT_FILENAME /usr/local/opnsense/scripts/nginx/csp_report.php;
      fastcgi_param TLS-Cipher $ssl_cipher;
      fastcgi_param TLS-Protocol $ssl_protocol;
      fastcgi_param TLS-SNI-Host $ssl_server_name;
      fastcgi_param SERVER-UUID "1eb8f7b2-f81d-4f31-bbeb-173c9678bfa7";
      fastcgi_intercept_errors on;
      fastcgi_pass  unix:/var/run/php-webgui.socket;
    }
    location /opnsense-auth-request {
      internal;
      fastcgi_pass  unix:/var/run/php-webgui.socket;
      fastcgi_index index.php;
      fastcgi_param TLS-Cipher $ssl_cipher;
      fastcgi_param TLS-Protocol $ssl_protocol;
      fastcgi_param TLS-SNI-Host $ssl_server_name;
      fastcgi_param Original-URI $request_uri;
      fastcgi_param Original-HOST $host;
      fastcgi_param SERVER-UUID "1eb8f7b2-f81d-4f31-bbeb-173c9678bfa7";
      fastcgi_param SCRIPT_FILENAME  /usr/local/opnsense/scripts/nginx/ngx_auth.php;
      fastcgi_intercept_errors on;
      include        fastcgi_params;
    }


}
# mail {
# }


The error generated is

2018/09/24 18:23:03 [emerg] 65933#100194: duplicate listen options for [::]:80 in /usr/local/etc/nginx/nginx.conf:149
2018/09/24 18:23:03 [emerg] 66135#100194: duplicate listen options for [::]:80 in /usr/local/etc/nginx/nginx.conf:149


My comments
- is it needed duplicate the locations related to opnsense Gui  ?
- I just fix the service replacing
       this line
                listen  [::]:80 ipv6only=off;   
       by this
                listen  80 ipv6only=off;


Could you help us with it?

Regards
CArlos



This reference help me with it https://serverfault.com/questions/638367/do-you-need-separate-ipv4-and-ipv6-listen-directives-in-nginx

[ccesario]

@Fabian, I solve it.. and I create a proposed patch in https://github.com/opnsense/plugins/pull/868

With this all server  directive it works

Best regards

Carlos

[ccesario]

Hi @Fabian...

Currently we are trying proxy_pass to internal server with diferent path

like

Code Select Expand

location /name/ {
    proxy_pass http://127.0.0.1/remote/;
}

as describe in documentation.
http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass

But it seems the Opnsense code does not implement it or I did not found it.

Is it possible you instruct us to do it!?

Best regards,
Carlos

[ccesario]

@Fabian, related to my last post, I have created a small patch to implement it.

If it is usable and accord the code, I can sent a pull request.

Best regards

Carlos

Code Select Expand

--- opnsense/service/templates/OPNsense/Nginx/location.conf.ori 2018-09-25 12:19:57.540066000 -0300
+++ opnsense/service/templates/OPNsense/Nginx/location.conf     2018-09-25 12:20:46.343048000 -0300
@@ -87,7 +87,7 @@
{% if location.upstream is defined and (location.php_enable is not defined or location.php_enable != '1') %}
{% set upstream = helpers.getUUID(location.upstream) %}
     proxy_set_header Host $host;
-    proxy_pass http{% if upstream.tls_enable == '1' %}s{% endif %}://upstream{{ location.upstream.replace('-','') }};
+    proxy_pass http{% if upstream.tls_enable == '1' %}s{% endif %}://upstream{{ location.upstream.replace('-','') }}{% if location.new_urlpattern != '' %}{{ location.new_urlpattern }};{% endif %}
{%   if upstream.tls_enable == '1' %}
{%     if upstream.tls_client_certificate is defined and upstream.tls_client_certificate != '' %}
     proxy_ssl_certificate_key /usr/local/etc/nginx/key/{{ upstream.tls_client_certificate }}.key;
--- opnsense/mvc/app/models/OPNsense/Nginx/Nginx.xml.ori        2018-09-25 12:22:43.430414000 -0300
+++ opnsense/mvc/app/models/OPNsense/Nginx/Nginx.xml    2018-09-25 13:09:52.750377000 -0300
@@ -205,6 +205,9 @@
         <Required>N</Required>
         <multiple>N</multiple>
       </upstream>
+      <new_urlpattern type="TextField">
+        <Required>N</Required>
+      </new_urlpattern>
       <root type="TextField">
         <Required>N</Required>
       </root>
--- opnsense/mvc/app/controllers/OPNsense/Nginx/forms/location.xml.ori  2018-09-25 13:13:04.693659000 -0300
+++ opnsense/mvc/app/controllers/OPNsense/Nginx/forms/location.xml      2018-09-25 13:16:54.587260000 -0300
@@ -62,6 +62,12 @@
     <help>Select an upstream to proxy to or connect via FastCGI if chosen.</help>
   </field>
   <field>
+    <id>location.new_urlpattern</id>
+    <label>New location path</label>
+    <type>text</type>
+    <help>Select a new path for upstream to proxy.</help>
+  </field>
+  <field>
     <id>location.limit_request_connections</id>
     <label>Limit Requests</label>
     <type>select_multiple</type>

[fabian]

it may work with a rewrite because this does append. You may also create a pull request if you really need it this way.

[ccesario]

Hi @Fabian,

Could you please validate the new path_prefix patch  according your suggestions on github?!

Code Select Expand

diff --git a/www/nginx/src/opnsense/mvc/app/controllers/OPNsense/Nginx/forms/location.xml b/www/nginx/src/opnsense/mvc/app/controllers/OPNsense/Nginx/forms/location.xml
index 3cbfc2b..358875a 100644
--- a/www/nginx/src/opnsense/mvc/app/controllers/OPNsense/Nginx/forms/location.xml
+++ b/www/nginx/src/opnsense/mvc/app/controllers/OPNsense/Nginx/forms/location.xml
@@ -62,6 +62,12 @@
     <help>Select an upstream to proxy to or connect via FastCGI if chosen.</help>
   </field>
   <field>
+    <id>location.path_prefix</id>
+    <label>Path prefix</label>
+    <type>text</type>
+    <help>Define an optional path prefix for this location.</help>
+  </field>
+  <field>
     <id>location.limit_request_connections</id>
     <label>Limit Requests</label>
     <type>select_multiple</type>
diff --git a/www/nginx/src/opnsense/mvc/app/models/OPNsense/Nginx/Nginx.xml b/www/nginx/src/opnsense/mvc/app/models/OPNsense/Nginx/Nginx.xml
index 91f1669..4a56c7d 100644
--- a/www/nginx/src/opnsense/mvc/app/models/OPNsense/Nginx/Nginx.xml
+++ b/www/nginx/src/opnsense/mvc/app/models/OPNsense/Nginx/Nginx.xml
@@ -205,6 +205,10 @@
         <Required>N</Required>
         <multiple>N</multiple>
       </upstream>
+      <path_prefix type="TextField">
+        <Required>N</Required>
+        <mask>/^[^" \t]+$/i</mask>
+      </path_prefix>
       <root type="TextField">
         <Required>N</Required>
       </root>
diff --git a/www/nginx/src/opnsense/service/templates/OPNsense/Nginx/location.conf b/www/nginx/src/opnsense/service/templates/OPNsense/Nginx/location.conf
index 6fb180f..70fbc2f 100644
--- a/www/nginx/src/opnsense/service/templates/OPNsense/Nginx/location.conf
+++ b/www/nginx/src/opnsense/service/templates/OPNsense/Nginx/location.conf
@@ -87,7 +87,11 @@ location {{ location.matchtype }} {{ location.urlpattern }} {
{% if location.upstream is defined and (location.php_enable is not defined or location.php_enable != '1') %}
{% set upstream = helpers.getUUID(location.upstream) %}
     proxy_set_header Host $host;
+{% if location.path_prefix is defined and location.path_prefix != '' %}
+    proxy_pass http{% if upstream.tls_enable == '1' %}s{% endif %}://upstream{{ location.upstream.replace('-','') }}{{ location.path_prefix }};
+{% else %}
     proxy_pass http{% if upstream.tls_enable == '1' %}s{% endif %}://upstream{{ location.upstream.replace('-','') }};
+{% endif %}
{%   if upstream.tls_enable == '1' %}
{%     if upstream.tls_client_certificate is defined and upstream.tls_client_certificate != '' %}
     proxy_ssl_certificate_key /usr/local/etc/nginx/key/{{ upstream.tls_client_certificate }}.key;


Best regards
Carlos

[fabian]

Except the label in the form it looks good (I would accept it). Every word should with an capital letter to be consistent with the rest of the UI. If you want to, you may mark it as advanced (form) to hide it in the normal view as it is not an everyday field. The i at the end is not required but also does not hurt because it means that the regular expression is not case sensitive (for example matching "Test" with /[a-z]+/i and /[a-z]+/ would deliver different results because the first one includes the "T", the second not)

[ccesario]

Hi @Fabian, thank you by your comments.!!!

Let me know, the OPNsense 18.7.4 did not include the lasted nginx commit codes ?

Regards
Carlos

[fabian]

Does not look like it happend. FYI: If you are on the main page of the repository, you can choose also a tag in the dropdown which is the release. Then you can look into the plugin directory which will show the latest commit message.

[fabian]

I forgot:  I've merged it to master so the changes should be in devel.

[ccesario]

I forgot:  I've merged it to master so the changes should be in devel.

Ohhh .... We are waiting this eheheh

[franco]

just want to drop by and say: thanks for your work, guys! 8)

[franco]

@Carlos: regarding your latest patches question? Do you use the development version or the release version? The release version wasn't updated.

[ccesario]

Hello @Franco,
Well, my patches are sent using Master branch, but I don know where @Fabian make the commits :)

Regards
Carlos

[fabian]

I usually use a feature branch in the plugins repository. There is a select box to switch between branches. The advantage when I develop this way is that you need at least one parameter less in opnsense-patch and franco can commit on this branch as well (did not ever happen yet but maybe it will some day in the future).