after update Nginx wont start error (OPNsense 23.1.10-amd64)

[RamSense]

after the hotfix, I have narrowed it down. When you disable the naxsi rules:

rules SQL Injections 1000-1099 and File Uploads 1500-1600

Nginx starts and works. Hope someone else can see what the "problem" is with naxsi / these rules with the latest nginx

[andyw]

Since updateing to 23.1.10-1 my Nginx wont start at all. I have tried rolling back Nginx and applying the patch however this does not appear to help. Has anyone got any suggestions how to resolve this issue?

[franco]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=271064
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=271963

naxsi is currently in limbo in FreeBSD ports due to upstream being changed and known bugs in PCRE2 support...


Cheers,
Franco

[RamSense]

Thanks Franco for the info and links. That explains it what is buggin here, and looking forward to the next update ;-)

@andyw: you can look at: SERVICES: NGINX: LOG FILE
And see the errors there for trying to solve it. Are you using Naxsi rules?

[andyw]

@RamSense I am using Naxsi rules. I have attached the debug log as per your request. Hope this helps.

[RamSense]

Ok, update nginx to latest version.
Disable naxsi rules SQL Injections 1000-1099 and File Uploads 1500-1600
Save and reload config and start nginx again.
Hope that works for you also.

[franco]

There is a test version now based on the new naxsi upstream version 1.4:

# opnsense-revert -z nginx

I appreciate all comments on this.


Cheers,
Franco

[RamSense]

Ok, first impression:
did the [# opnsense-revert -z nginx]

Nginx still runs after adding the naxsi rules back. Did a nginx hard stop en start, still works. That's the good part

looking at the log, I still see the error appearing, but not fatal anymore(?) since nginx keeps running...:
[emerg] 98860#122256: Naxsi-Config : Incorrect line MainRule id:1500 (/usr/obj/usr/ports/www/nginx/work/naxsi-29793dc/naxsi_src/naxsi_skeleton.c/973)... in /usr/local/etc/nginx/nginx.conf:50

I also noticed this in the log, what I do not remember being there before:
nginx: [warn] could not build optimal variables_hash, you should increase either variables_hash_max_size: 1024 or variables_hash_bucket_size: 64; ignoring variables_hash_bucket_size

[franco]

You can compare with other nginx like this:

good 1.22: # opnsense-revert -r 23.1.9 nginx
bad 1.24: # opnsense-revert nginx
better? 1.24: # opnsense-revert -z nginx


Cheers,
Franco

[RamSense]

With the naxsi rules enabled and started with reverting in same order as stated below:

1.22: # opnsense-revert -r 23.1.9 nginx. ->nginx fails to start:

ginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
nginx: [emerg] module "ngx_http_vhost_traffic_status_module" is already loaded in /usr/local/etc/nginx/nginx.conf:7

1.24: # opnsense-revert nginx -> nginx fails to start:

nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
nginx: [emerg] Naxsi-Config : Incorrect line MainRule id:1500 (/usr/obj/usr/ports/www/nginx/work/naxsi-29793dc/naxsi_src/naxsi_skeleton.c/973)... in /usr/local/etc/nginx/nginx.conf:50

1.24: # opnsense-revert -z nginx -> nginx running with only this error:

nginx: [warn] could not build optimal variables_hash, you should increase either variables_hash_max_size: 1024 or variables_hash_bucket_size: 64; ignoring variables_hash_bucket_size

[andyw]

For me nginx still will not start after trying all options. I have not disabled any naxsi rules as per @RamSense suggestion as couldn't find an optimal way to disable the rules. I could delete them and possibly comment them out via shell however was waiting to see if others are experiencing the same issues before diving in.
On a side note I have also noticed the gateway widget not showing as online since the update and this only comes back online when doing a reboot of the isp modem. Not sure if this is related or not but only started since the update.

the logs show:
2023-06-24T19:54:09   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-24T19:54:09   Emergency   nginx    nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-24T19:54:09   Debug   nginx    NGINX setup routine started.
2023-06-24T19:53:14   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-24T19:53:14   Emergency   nginx    nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-24T19:53:14   Debug   nginx    NGINX setup routine started.
2023-06-24T19:51:58   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-24T19:51:58   Emergency   nginx    nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-24T19:51:58   Debug   nginx    NGINX setup routine started.
2023-06-24T19:51:50   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-24T19:51:50   Emergency   nginx    nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-24T19:51:50   Debug   nginx    NGINX setup routine started.
2023-06-24T19:51:41   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-24T19:51:41   Emergency   nginx    nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-24T19:51:41   Debug   nginx    NGINX setup routine started.
2023-06-24T19:51:37   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-24T19:51:37   Emergency   nginx    nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-24T19:51:36   Debug   nginx    NGINX setup routine started.
2023-06-23T09:23:31   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-23T09:23:31   Emergency   nginx    nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-23T09:23:31   Debug   nginx    NGINX setup routine started.
2023-06-22T20:10:41   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T20:10:41   Emergency   nginx    nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-22T20:10:41   Debug   nginx    NGINX setup routine started.
2023-06-22T20:10:16   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T20:10:16   Emergency   nginx    nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-22T20:10:16   Debug   nginx    NGINX setup routine started.
2023-06-22T20:10:07   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T20:10:07   Emergency   nginx    nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-22T20:10:07   Debug   nginx    NGINX setup routine started.
2023-06-22T20:06:57   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T20:06:57   Emergency   nginx    nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-22T20:06:57   Debug   nginx    NGINX setup routine started.
2023-06-22T20:06:47   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T20:06:47   Emergency   nginx    nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-22T20:06:47   Debug   nginx    NGINX setup routine started.
2023-06-22T20:04:15   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T20:04:15   Emergency   nginx    nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-22T20:04:15   Debug   nginx    NGINX setup routine started.
2023-06-22T19:58:13   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:58:13   Emergency   nginx    nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-22T19:58:13   Debug   nginx    NGINX setup routine started.
2023-06-22T19:56:05   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:56:05   Emergency   nginx    nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-22T19:56:05   Debug   nginx    NGINX setup routine started.
2023-06-22T19:53:04   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:53:04   Emergency   nginx    nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-22T19:53:04   Debug   nginx    NGINX setup routine started.
2023-06-22T19:52:25   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:52:25   Emergency   nginx    nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-22T19:52:25   Debug   nginx    NGINX setup routine started.
2023-06-22T19:52:10   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:52:10   Emergency   nginx    nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-22T19:52:10   Debug   nginx    NGINX setup routine started.
2023-06-22T19:51:42   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:51:42   Emergency   nginx    nginx: [emerg] module "ngx_http_vhost_traffic_status_module" is already loaded in /usr/local/etc/nginx/nginx.conf:7
2023-06-22T19:51:42   Debug   nginx    NGINX setup routine started.
2023-06-22T19:46:07   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:46:07   Emergency   nginx    nginx: [emerg] module "ngx_http_vhost_traffic_status_module" is already loaded in /usr/local/etc/nginx/nginx.conf:7
2023-06-22T19:46:06   Debug   nginx    NGINX setup routine started.
2023-06-22T19:44:58   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:44:58   Emergency   nginx    nginx: [emerg] module "ngx_http_vhost_traffic_status_module" is already loaded in /usr/local/etc/nginx/nginx.conf:7
2023-06-22T19:44:58   Debug   nginx    NGINX setup routine started.
2023-06-22T19:43:42   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:43:42   Emergency   nginx    nginx: [emerg] module "ngx_http_vhost_traffic_status_module" is already loaded in /usr/local/etc/nginx/nginx.conf:7
2023-06-22T19:43:42   Debug   nginx    NGINX setup routine started.
2023-06-22T19:42:51   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:42:51   Emergency   nginx    nginx: [emerg] module "ngx_http_vhost_traffic_status_module" is already loaded in /usr/local/etc/nginx/nginx.conf:7
2023-06-22T19:42:51   Debug   nginx    NGINX setup routine started.
2023-06-22T19:42:47   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:42:47   Emergency   nginx    nginx: [emerg] module "ngx_http_vhost_traffic_status_module" is already loaded in /usr/local/etc/nginx/nginx.conf:7
2023-06-22T19:42:47   Debug   nginx    NGINX setup routine started.
2023-06-22T19:31:38   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:31:38   Emergency   nginx    nginx: [emerg] Naxsi-Config : Incorrect line MainRule id:1500 (/usr/obj/usr/ports/www/nginx/work/naxsi-29793dc/naxsi_src/naxsi_skeleton.c/973)... in /usr/local/etc/nginx/nginx.conf:50
2023-06-22T19:31:38   Debug   nginx    NGINX setup routine started.
2023-06-22T19:31:27   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:31:27   Emergency   nginx    nginx: [emerg] Naxsi-Config : Incorrect line MainRule id:1500 (/usr/obj/usr/ports/www/nginx/work/naxsi-29793dc/naxsi_src/naxsi_skeleton.c/973)... in /usr/local/etc/nginx/nginx.conf:50
2023-06-22T19:31:27   Debug   nginx    NGINX setup routine started.
2023-06-22T19:31:22   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:31:22   Emergency   nginx    nginx: [emerg] Naxsi-Config : Incorrect line MainRule id:1500 (/usr/obj/usr/ports/www/nginx/work/naxsi-29793dc/naxsi_src/naxsi_skeleton.c/973)... in /usr/local/etc/nginx/nginx.conf:50
2023-06-22T19:31:22   Debug   nginx    NGINX setup routine started.
2023-06-22T19:31:07   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:31:07   Emergency   nginx    nginx: [emerg] Naxsi-Config : Incorrect line MainRule id:1500 (/usr/obj/usr/ports/www/nginx/work/naxsi-29793dc/naxsi_src/naxsi_skeleton.c/973)... in /usr/local/etc/nginx/nginx.conf:50
2023-06-22T19:31:07   Debug   nginx    NGINX setup routine started.
2023-06-22T19:26:46   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:26:46   Emergency   nginx    nginx: [emerg] Naxsi-Config : Incorrect line MainRule id:1500 (/usr/obj/usr/ports/www/nginx/work/naxsi-29793dc/naxsi_src/naxsi_skeleton.c/973)... in /usr/local/etc/nginx/nginx.conf:50
2023-06-22T19:26:46   Debug   nginx    NGINX setup routine started.
2023-06-22T19:25:10   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:25:10   Emergency   nginx    nginx: [emerg] Naxsi-Config : Incorrect line MainRule id:1500 (/usr/obj/usr/ports/www/nginx/work/naxsi-29793dc/naxsi_src/naxsi_skeleton.c/973)... in /usr/local/etc/nginx/nginx.conf:50
2023-06-22T19:25:10   Debug   nginx    NGINX setup routine started.
2023-06-22T19:23:45   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:23:45   Emergency   nginx    nginx: [emerg] Naxsi-Config : Incorrect line MainRule id:1500 (/usr/obj/usr/ports/www/nginx/work/naxsi-29793dc/naxsi_src/naxsi_skeleton.c/973)... in /usr/local/etc/nginx/nginx.conf:50
2023-06-22T19:23:44   Debug   nginx    NGINX setup routine started.
2023-06-22T19:22:38   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:22:38   Emergency   nginx    nginx: [emerg] Naxsi-Config : Incorrect line MainRule id:1500 (/usr/obj/usr/ports/www/nginx/work/naxsi-29793dc/naxsi_src/naxsi_skeleton.c/973)... in /usr/local/etc/nginx/nginx.conf:50
2023-06-22T19:22:38   Debug   nginx    NGINX setup routine started.
2023-06-22T19:21:31   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:21:31   Emergency   nginx    nginx: [emerg] Naxsi-Config : Incorrect line MainRule id:1500 (/usr/obj/usr/ports/www/nginx/work/naxsi-29793dc/naxsi_src/naxsi_skeleton.c/973)... in /usr/local/etc/nginx/nginx.conf:50
2023-06-22T19:21:31   Debug   nginx    NGINX setup routine started.
2023-06-02T00:00:00   Emergency   nginx    2023/06/02 00:00:00 [emerg] 32693#100322: open() "/var/log/nginx/permanentban.access.log" failed (13: Permission denied)
2023-06-02T00:00:00   Emergency   nginx    2023/06/02 00:00:00 [emerg] 32693#100322: open() "/var/log/nginx/tls_handshake.log" failed (13: Permission denied)
2023-05-26T22:05:18   Debug   nginx    NGINX setup routine completed.
2023-05-26T22:05:17   Debug   nginx    NGINX setup routine started.
2023-04-27T09:22:04   Debug   nginx    NGINX setup routine completed.
2023-04-27T09:22:04   Debug   nginx    NGINX setup routine started.
2023-04-26T00:00:00   Emergency   nginx    2023/04/26 00:00:00 [emerg] 47781#100532: open() "/var/log/nginx/permanentban.access.log" failed (13: Permission denied)
2023-04-26T00:00:00   Emergency   nginx    2023/04/26 00:00:00 [emerg] 47781#100532: open() "/var/log/nginx/tls_handshake.log" failed (13: Permission denied)
2023-04-24T18:56:51   Debug   nginx    NGINX setup routine completed.
2023-04-24T18:56:51   Debug   nginx    NGINX setup routine started.
2023-04-24T18:37:34   Debug   nginx    NGINX setup routine completed.
2023-04-24T18:37:34   Debug   nginx    NGINX setup routine started.
2023-04-24T18:35:17   Debug   nginx    NGINX setup routine completed.
2023-04-24T18:35:17   Debug   nginx    NGINX setup routine started.
2023-04-21T17:00:59   Debug   nginx    NGINX setup routine completed.
2023-04-21T17:00:59   Debug   nginx    NGINX setup routine started.

[franco]

1.22: # opnsense-revert -r 23.1.9 nginx. ->nginx fails to start:

ginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
nginx: [emerg] module "ngx_http_vhost_traffic_status_module" is already loaded in /usr/local/etc/nginx/nginx.conf:7

This is funny and the reason why the hotfix for missing load_module was necessary on 22.1.10...

1.24: # opnsense-revert nginx -> nginx fails to start:

nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
nginx: [emerg] Naxsi-Config : Incorrect line MainRule id:1500 (/usr/obj/usr/ports/www/nginx/work/naxsi-29793dc/naxsi_src/naxsi_skeleton.c/973)... in /usr/local/etc/nginx/nginx.conf:50

Ok as expected.

1.24: # opnsense-revert -z nginx -> nginx running with only this error:

nginx: [warn] could not build optimal variables_hash, you should increase either variables_hash_max_size: 1024 or variables_hash_bucket_size: 64; ignoring variables_hash_bucket_size

[/quote]

Looks good then. I'll try to move the naxsi change to FreeBSD port.


Cheers,
Franco

[franco]

2023-06-24T19:54:09   Emergency   nginx    nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-24T19:54:09   Emergency   nginx    nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9

You have a partial opnsense-patch state. Just install the latest hotfix and/or run

# opnsense-revert os-nginx


Cheers,
Franco

[andyw]

OPNsense 23.1.11 resolves the issue and Nginx is back up and running.
Thank you