After updating to OPNsense 23.1.10-amd64 and a forced reboot, nginx wont start.
First it was because of Naxsi rules (1500, 1000 etc) after disabling them, it still wont start:
nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
nginx: [emerg] unknown directive "vhost_traffic_status_zone" in /usr/local/etc/nginx/opnsense_http_vhost_plugins/vts.conf:1
But.I do not know what is meant by this. Before OPNsense 23.1.10-amd64 it was working and no changes in config other dan now disabling some Naxsi rules....
Others having issues with nginx after updating also?
Probably nginx update broke this, not the plugin. Not sure what to do. Missing build option in port or did they remove the feature between 1.22 and 1.24?
https://github.com/opnsense/ports/commit/005bb3d28162
Cheers,
Franco
VTS option is/was enabled and the shared object is there:
# pkg info -l nginx | grep vhost_
/usr/local/libexec/nginx/ngx_http_vhost_traffic_status_module.so
So they removed the config option?
BTW, for now:
# opnsense-revert -r 23.1.9 nginx
Cheers,
Franco
This perhaps? https://github.com/opnsense/plugins/commit/d26aaa7352f9
# opnsense-patch -c plugins d26aaa7352f9
Cheers,
Franco
Thanks Franco for the fast replies.
The revert back worked as a charm, so that I can confirm.
back to the update and tried the patch. Another error now (besides the Naxsi ones)
nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
Sorry, typo! https://github.com/opnsense/plugins/commit/682780e6
# opnsense-patch -c plugins 682780e6
Cheers,
Franco
Thnx, tried it right away, same error?
nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
It needs a proper reload (config is probably still not rendered according to last commit).
Tried reloading config and than a forced reboot for the ultimate reload, but still same error after applying last patch:
nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
ok found the problem, besides your fix
it has to do with the Naxsi rules. When I disable: rules 1000-1099, 1400-1500; 1500-1600
Quotenginx: [emerg] Naxsi-Config : Incorrect line MainRule id:1000 (/usr/obj/usr/ports/www/nginx/work/naxsi-29793dc/naxsi_src/naxsi_skeleton.c/973)... in /usr/local/etc/nginx/nginx.conf:74
nginx: [emerg] Naxsi-Config : Incorrect line MainRule id:1500 (/usr/obj/usr/ports/www/nginx/work/naxsi-29793dc/naxsi_src/naxsi_skeleton.c/973)... in /usr/local/etc/nginx/nginx.conf:50
and some and hit reload and start nginx works, so there has been some change with Naxsi in Nginx what does not start or the Naxsi rules are no longer compatible to this nginx?
Ok but the other patch is required for the error to go away, yes?
Cheers,
Franco
Yes that is correct.
I did another opnsense-revert -r 23.1.9 nginx, installed the normal update, removed the naxsi as stated above, and restarted nginx did work.
Is it that the patches I did are still there? because I know I needed that one for the first error to being fixed:
nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
the change is in "os-nginx" not "nginx" so as long as you don't revert "os-nginx" you are fine. I'm going to hotfix in a bit.
Cheers,
Franco
ok and thnx for the fast following up
after the hotfix, I have narrowed it down. When you disable the naxsi rules:
rules SQL Injections 1000-1099 and File Uploads 1500-1600
Nginx starts and works. Hope someone else can see what the "problem" is with naxsi / these rules with the latest nginx
Since updateing to 23.1.10-1 my Nginx wont start at all. I have tried rolling back Nginx and applying the patch however this does not appear to help. Has anyone got any suggestions how to resolve this issue?
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=271064
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=271963
naxsi is currently in limbo in FreeBSD ports due to upstream being changed and known bugs in PCRE2 support...
Cheers,
Franco
Thanks Franco for the info and links. That explains it what is buggin here, and looking forward to the next update ;-)
@andyw: you can look at: SERVICES: NGINX: LOG FILE
And see the errors there for trying to solve it. Are you using Naxsi rules?
@RamSense I am using Naxsi rules. I have attached the debug log as per your request. Hope this helps.
Ok, update nginx to latest version.
Disable naxsi rules SQL Injections 1000-1099 and File Uploads 1500-1600
Save and reload config and start nginx again.
Hope that works for you also.
There is a test version now based on the new naxsi upstream version 1.4:
# opnsense-revert -z nginx
I appreciate all comments on this.
Cheers,
Franco
Ok, first impression:
did the [# opnsense-revert -z nginx]
Nginx still runs after adding the naxsi rules back. Did a nginx hard stop en start, still works. That's the good part
looking at the log, I still see the error appearing, but not fatal anymore(?) since nginx keeps running...:
[emerg] 98860#122256: Naxsi-Config : Incorrect line MainRule id:1500 (/usr/obj/usr/ports/www/nginx/work/naxsi-29793dc/naxsi_src/naxsi_skeleton.c/973)... in /usr/local/etc/nginx/nginx.conf:50
I also noticed this in the log, what I do not remember being there before:
nginx: [warn] could not build optimal variables_hash, you should increase either variables_hash_max_size: 1024 or variables_hash_bucket_size: 64; ignoring variables_hash_bucket_size
You can compare with other nginx like this:
good 1.22: # opnsense-revert -r 23.1.9 nginx
bad 1.24: # opnsense-revert nginx
better? 1.24: # opnsense-revert -z nginx
Cheers,
Franco
With the naxsi rules enabled and started with reverting in same order as stated below:
1.22: # opnsense-revert -r 23.1.9 nginx. ->nginx fails to start:
Quoteginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
nginx: [emerg] module "ngx_http_vhost_traffic_status_module" is already loaded in /usr/local/etc/nginx/nginx.conf:7
1.24: # opnsense-revert nginx -> nginx fails to start:
Quotenginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
nginx: [emerg] Naxsi-Config : Incorrect line MainRule id:1500 (/usr/obj/usr/ports/www/nginx/work/naxsi-29793dc/naxsi_src/naxsi_skeleton.c/973)... in /usr/local/etc/nginx/nginx.conf:50
1.24: # opnsense-revert -z nginx -> nginx running with only this error:
Quotenginx: [warn] could not build optimal variables_hash, you should increase either variables_hash_max_size: 1024 or variables_hash_bucket_size: 64; ignoring variables_hash_bucket_size
For me nginx still will not start after trying all options. I have not disabled any naxsi rules as per @RamSense suggestion as couldn't find an optimal way to disable the rules. I could delete them and possibly comment them out via shell however was waiting to see if others are experiencing the same issues before diving in.
On a side note I have also noticed the gateway widget not showing as online since the update and this only comes back online when doing a reboot of the isp modem. Not sure if this is related or not but only started since the update.
the logs show:
2023-06-24T19:54:09 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-24T19:54:09 Emergency nginx nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-24T19:54:09 Debug nginx NGINX setup routine started.
2023-06-24T19:53:14 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-24T19:53:14 Emergency nginx nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-24T19:53:14 Debug nginx NGINX setup routine started.
2023-06-24T19:51:58 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-24T19:51:58 Emergency nginx nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-24T19:51:58 Debug nginx NGINX setup routine started.
2023-06-24T19:51:50 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-24T19:51:50 Emergency nginx nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-24T19:51:50 Debug nginx NGINX setup routine started.
2023-06-24T19:51:41 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-24T19:51:41 Emergency nginx nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-24T19:51:41 Debug nginx NGINX setup routine started.
2023-06-24T19:51:37 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-24T19:51:37 Emergency nginx nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-24T19:51:36 Debug nginx NGINX setup routine started.
2023-06-23T09:23:31 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-23T09:23:31 Emergency nginx nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-23T09:23:31 Debug nginx NGINX setup routine started.
2023-06-22T20:10:41 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T20:10:41 Emergency nginx nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-22T20:10:41 Debug nginx NGINX setup routine started.
2023-06-22T20:10:16 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T20:10:16 Emergency nginx nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-22T20:10:16 Debug nginx NGINX setup routine started.
2023-06-22T20:10:07 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T20:10:07 Emergency nginx nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-22T20:10:07 Debug nginx NGINX setup routine started.
2023-06-22T20:06:57 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T20:06:57 Emergency nginx nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-22T20:06:57 Debug nginx NGINX setup routine started.
2023-06-22T20:06:47 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T20:06:47 Emergency nginx nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-22T20:06:47 Debug nginx NGINX setup routine started.
2023-06-22T20:04:15 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T20:04:15 Emergency nginx nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-22T20:04:15 Debug nginx NGINX setup routine started.
2023-06-22T19:58:13 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:58:13 Emergency nginx nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-22T19:58:13 Debug nginx NGINX setup routine started.
2023-06-22T19:56:05 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:56:05 Emergency nginx nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-22T19:56:05 Debug nginx NGINX setup routine started.
2023-06-22T19:53:04 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:53:04 Emergency nginx nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-22T19:53:04 Debug nginx NGINX setup routine started.
2023-06-22T19:52:25 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:52:25 Emergency nginx nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-22T19:52:25 Debug nginx NGINX setup routine started.
2023-06-22T19:52:10 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:52:10 Emergency nginx nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
2023-06-22T19:52:10 Debug nginx NGINX setup routine started.
2023-06-22T19:51:42 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:51:42 Emergency nginx nginx: [emerg] module "ngx_http_vhost_traffic_status_module" is already loaded in /usr/local/etc/nginx/nginx.conf:7
2023-06-22T19:51:42 Debug nginx NGINX setup routine started.
2023-06-22T19:46:07 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:46:07 Emergency nginx nginx: [emerg] module "ngx_http_vhost_traffic_status_module" is already loaded in /usr/local/etc/nginx/nginx.conf:7
2023-06-22T19:46:06 Debug nginx NGINX setup routine started.
2023-06-22T19:44:58 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:44:58 Emergency nginx nginx: [emerg] module "ngx_http_vhost_traffic_status_module" is already loaded in /usr/local/etc/nginx/nginx.conf:7
2023-06-22T19:44:58 Debug nginx NGINX setup routine started.
2023-06-22T19:43:42 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:43:42 Emergency nginx nginx: [emerg] module "ngx_http_vhost_traffic_status_module" is already loaded in /usr/local/etc/nginx/nginx.conf:7
2023-06-22T19:43:42 Debug nginx NGINX setup routine started.
2023-06-22T19:42:51 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:42:51 Emergency nginx nginx: [emerg] module "ngx_http_vhost_traffic_status_module" is already loaded in /usr/local/etc/nginx/nginx.conf:7
2023-06-22T19:42:51 Debug nginx NGINX setup routine started.
2023-06-22T19:42:47 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:42:47 Emergency nginx nginx: [emerg] module "ngx_http_vhost_traffic_status_module" is already loaded in /usr/local/etc/nginx/nginx.conf:7
2023-06-22T19:42:47 Debug nginx NGINX setup routine started.
2023-06-22T19:31:38 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:31:38 Emergency nginx nginx: [emerg] Naxsi-Config : Incorrect line MainRule id:1500 (/usr/obj/usr/ports/www/nginx/work/naxsi-29793dc/naxsi_src/naxsi_skeleton.c/973)... in /usr/local/etc/nginx/nginx.conf:50
2023-06-22T19:31:38 Debug nginx NGINX setup routine started.
2023-06-22T19:31:27 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:31:27 Emergency nginx nginx: [emerg] Naxsi-Config : Incorrect line MainRule id:1500 (/usr/obj/usr/ports/www/nginx/work/naxsi-29793dc/naxsi_src/naxsi_skeleton.c/973)... in /usr/local/etc/nginx/nginx.conf:50
2023-06-22T19:31:27 Debug nginx NGINX setup routine started.
2023-06-22T19:31:22 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:31:22 Emergency nginx nginx: [emerg] Naxsi-Config : Incorrect line MainRule id:1500 (/usr/obj/usr/ports/www/nginx/work/naxsi-29793dc/naxsi_src/naxsi_skeleton.c/973)... in /usr/local/etc/nginx/nginx.conf:50
2023-06-22T19:31:22 Debug nginx NGINX setup routine started.
2023-06-22T19:31:07 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:31:07 Emergency nginx nginx: [emerg] Naxsi-Config : Incorrect line MainRule id:1500 (/usr/obj/usr/ports/www/nginx/work/naxsi-29793dc/naxsi_src/naxsi_skeleton.c/973)... in /usr/local/etc/nginx/nginx.conf:50
2023-06-22T19:31:07 Debug nginx NGINX setup routine started.
2023-06-22T19:26:46 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:26:46 Emergency nginx nginx: [emerg] Naxsi-Config : Incorrect line MainRule id:1500 (/usr/obj/usr/ports/www/nginx/work/naxsi-29793dc/naxsi_src/naxsi_skeleton.c/973)... in /usr/local/etc/nginx/nginx.conf:50
2023-06-22T19:26:46 Debug nginx NGINX setup routine started.
2023-06-22T19:25:10 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:25:10 Emergency nginx nginx: [emerg] Naxsi-Config : Incorrect line MainRule id:1500 (/usr/obj/usr/ports/www/nginx/work/naxsi-29793dc/naxsi_src/naxsi_skeleton.c/973)... in /usr/local/etc/nginx/nginx.conf:50
2023-06-22T19:25:10 Debug nginx NGINX setup routine started.
2023-06-22T19:23:45 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:23:45 Emergency nginx nginx: [emerg] Naxsi-Config : Incorrect line MainRule id:1500 (/usr/obj/usr/ports/www/nginx/work/naxsi-29793dc/naxsi_src/naxsi_skeleton.c/973)... in /usr/local/etc/nginx/nginx.conf:50
2023-06-22T19:23:44 Debug nginx NGINX setup routine started.
2023-06-22T19:22:38 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:22:38 Emergency nginx nginx: [emerg] Naxsi-Config : Incorrect line MainRule id:1500 (/usr/obj/usr/ports/www/nginx/work/naxsi-29793dc/naxsi_src/naxsi_skeleton.c/973)... in /usr/local/etc/nginx/nginx.conf:50
2023-06-22T19:22:38 Debug nginx NGINX setup routine started.
2023-06-22T19:21:31 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-22T19:21:31 Emergency nginx nginx: [emerg] Naxsi-Config : Incorrect line MainRule id:1500 (/usr/obj/usr/ports/www/nginx/work/naxsi-29793dc/naxsi_src/naxsi_skeleton.c/973)... in /usr/local/etc/nginx/nginx.conf:50
2023-06-22T19:21:31 Debug nginx NGINX setup routine started.
2023-06-02T00:00:00 Emergency nginx 2023/06/02 00:00:00 [emerg] 32693#100322: open() "/var/log/nginx/permanentban.access.log" failed (13: Permission denied)
2023-06-02T00:00:00 Emergency nginx 2023/06/02 00:00:00 [emerg] 32693#100322: open() "/var/log/nginx/tls_handshake.log" failed (13: Permission denied)
2023-05-26T22:05:18 Debug nginx NGINX setup routine completed.
2023-05-26T22:05:17 Debug nginx NGINX setup routine started.
2023-04-27T09:22:04 Debug nginx NGINX setup routine completed.
2023-04-27T09:22:04 Debug nginx NGINX setup routine started.
2023-04-26T00:00:00 Emergency nginx 2023/04/26 00:00:00 [emerg] 47781#100532: open() "/var/log/nginx/permanentban.access.log" failed (13: Permission denied)
2023-04-26T00:00:00 Emergency nginx 2023/04/26 00:00:00 [emerg] 47781#100532: open() "/var/log/nginx/tls_handshake.log" failed (13: Permission denied)
2023-04-24T18:56:51 Debug nginx NGINX setup routine completed.
2023-04-24T18:56:51 Debug nginx NGINX setup routine started.
2023-04-24T18:37:34 Debug nginx NGINX setup routine completed.
2023-04-24T18:37:34 Debug nginx NGINX setup routine started.
2023-04-24T18:35:17 Debug nginx NGINX setup routine completed.
2023-04-24T18:35:17 Debug nginx NGINX setup routine started.
2023-04-21T17:00:59 Debug nginx NGINX setup routine completed.
2023-04-21T17:00:59 Debug nginx NGINX setup routine started.
Quote from: RamSense on June 24, 2023, 12:18:54 PM
1.22: # opnsense-revert -r 23.1.9 nginx. ->nginx fails to start:
Quoteginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
nginx: [emerg] module "ngx_http_vhost_traffic_status_module" is already loaded in /usr/local/etc/nginx/nginx.conf:7
This is funny and the reason why the hotfix for missing load_module was necessary on 22.1.10...
Quote from: RamSense on June 24, 2023, 12:18:54 PM
1.24: # opnsense-revert nginx -> nginx fails to start:
Quotenginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
nginx: [emerg] Naxsi-Config : Incorrect line MainRule id:1500 (/usr/obj/usr/ports/www/nginx/work/naxsi-29793dc/naxsi_src/naxsi_skeleton.c/973)... in /usr/local/etc/nginx/nginx.conf:50
Ok as expected.
Quote from: RamSense on June 24, 2023, 12:18:54 PM
1.24: # opnsense-revert -z nginx -> nginx running with only this error:
Quotenginx: [warn] could not build optimal variables_hash, you should increase either variables_hash_max_size: 1024 or variables_hash_bucket_size: 64; ignoring variables_hash_bucket_size
[/quote]
Looks good then. I'll try to move the naxsi change to FreeBSD port.
Cheers,
Franco
Quote from: andyw on June 24, 2023, 09:12:12 PM
2023-06-24T19:54:09 Emergency nginx nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
2023-06-24T19:54:09 Emergency nginx nginx: [emerg] invalid number of arguments in "load_module" directive in /usr/local/etc/nginx/nginx.conf:9
You have a partial opnsense-patch state. Just install the latest hotfix and/or run
# opnsense-revert os-nginx
Cheers,
Franco
OPNsense 23.1.11 resolves the issue and Nginx is back up and running.
Thank you