(Solved) Firewall not rebooting

[emiletenhagen]

After upgrading to 19.7.2 with zabbix agent installed the system will not reboot untill the zabbix_agentd pid is killed.

[Droppie391]

same here, delete pid and reboot works. can this have something to do with the fact that the /var/run/zabbix directory (and pid) have user-group set to zabbix:zabbix?

[emiletenhagen]

I did try the following to find a solution, but unfortunately it did not work.

What I found was that the command from the system does not stop the processes, but only terminates them.

   77820   root   45   0   1034m   3132k   wait   1   0:00   0.00%   /bin/sh -c /usr/local/etc/rc.d/zabbix_agentd stop; /usr/bin/killall zabbix_agentd; exit 0
   12057   root   52   0   1034m   3408k   wait   0   0:00   0.00%   /bin/sh /usr/local/etc/rc.d/zabbix_agentd stop
   34983   zabbix   20   0   15m   6m   nanslp   1   0:00   0.00%   zabbix_agentd: collector #1 [terminated] (zabbix_agentd)
   86178   zabbix   20   0   15m   6m   wait   0   0:00   0.00%   /usr/local/sbin/zabbix_agentd -c /usr/local/etc/zabbix_agentd.conf
   12963   zabbix   20   0   15m   6m   nanslp   1   0:00   0.00%   zabbix_agentd: active checks #1 [terminated] (zabbix_agentd)
   11863   zabbix   20   0   15m   6m   nanslp   1   0:00   0.00%   zabbix_agentd: listener #3 [terminated] (zabbix_agentd)
   75731   zabbix   20   0   15m   6m   nanslp   0   0:00   0.00%   zabbix_agentd: listener #2 [terminated] (zabbix_agentd)
   35366   zabbix   20   0   15m   6m   nanslp   1   0:00   0.00%   zabbix_agentd: listener #1 [terminated] (zabbix_agentd)

In the zabbix_agentd.log you see a SIGTERM for the process and the sub-processes are terminated, as shown.
When the sub-processes (status nanslp) are terminated with kill -9 the main process is stopped.

I tried it with a newer version of the zabbix agent but this does not work either.

Now each time when there is a configuration change I have to kill the procceses manually before I can apply.

[franco]

Sounds like this should be reported to zabbix authors.


Cheers,
Franco

[emiletenhagen]

I'm not sure, depends on who compiled the zabbix_agentd. If opnsense just copies the available lts version to their repo your correct.

I did some extra testing by downloading the agent version from the opnsense repo from 19.1 :https://pkg.opnsense.org/FreeBSD:11:amd64/19.1/latest/All/ , installed this one and it works.

This is version 4.0.9.

I will report this to Zabbix.

[franco]

Interesting, Zabbix is 4.0.11 currently. Might be work looking for regressions in 4.0.10 or 4.0.11 changelog indeed.


Cheers,
Franco

[emiletenhagen]

The problem was already known by Zabbix.

https://support.zabbix.com/browse/ZBX-16460

Rgds.

[franco]

Interesting, thanks for the link! Looks like it only affects version .11 and no fix yet.


Cheers,
Franco

[the-mk]

experiencing similar issue: I had to manually reboot OPNsense after upgrading from 19.7.2 to 19.7.3 (before it started upgrading it informed me that it will reboot the firewall afterwards, but it never happened).
at the moment they say that it is fixed in a pre-release...
I am curious when it finds its way into OPNsense ;)

[the-mk]

did another test:
it seems that both zabbix-agent and zabbix-proxy prevent OPNsense from rebooting.
both report to be on version 4.0.11
so if you'd like to reboot your OPNsense you should make sure that all zabbix_agent and zabbix_proxy processes are "away" (kill -9 $PID --> all that you can find with "ps ax | grep zabbix")

[emiletenhagen]

Updated to Opnsense 19.7.4 which updates zabbixagent to 4.0.12 and the problem is solved.

Thanks for the update!!

[franco]

I thought this was supposedly fixed in upstream 4.0.13. Better double-check this.


Cheers,
Franco

[emiletenhagen]

Sorry to say, but the system is rebooting now.

Rgds

[franco]

Lucky reboot. Nothing to be sorry about. ;)

[the-mk]

before having a lucky reboot of OPNsense you need to kill the zabbix_* processes on the console or via SSH - they remain after the upgrade from 19.7.3 to 19.7.4.
But after that the services are starting and stopping very smooth :)