zabbix agent not working

[xofer]

I feel stupid, but I cannot get zabbix agent to function on a particular opnsense machine. It works fine on several of them, but not at all in one.

Opnsense 22.1.2_1
os-zabbix-agent 1.11

I have also tried to install other zabbix version (os-zabbix54-agent), same behaviour
Zabbix agent conf is pretty basic:


Zabbix agent starts up with no errors, seems to listen to port 10050:

Code Select Expand

# sockstat | grep 10050
zabbix   zabbix_age 39652 4  tcp4   127.0.0.1:10050       *:*
zabbix   zabbix_age 39642 4  tcp4   127.0.0.1:10050       *:*
zabbix   zabbix_age 39623 4  tcp4   127.0.0.1:10050       *:*
zabbix   zabbix_age 39546 4  tcp4   127.0.0.1:10050       *:*
zabbix   zabbix_age 39464 4  tcp4   127.0.0.1:10050       *:*

However, when i run (it should immediately return zabbix agent version), it just lags until a timeout:

Code Select Expand

root@mono:~ # zabbix_get -s 127.0.0.1 -p 10050 -k agent.version
zabbix_get [93235]: Timeout while executing operation


it just hangs. Tcpdump on lo0 shows only packets going to port 10050, nothing returns:

Code Select Expand

17:08:08.858071 IP 127.0.0.1.27937 > 127.0.0.1.10050: Flags [S], seq 2348730218, win 65228, options [mss 16344,nop,wscale 7,sackOK,TS val 1000361919 ecr 0], length 0
17:08:09.868097 IP 127.0.0.1.27937 > 127.0.0.1.10050: Flags [S], seq 2348730218, win 65228, options [mss 16344,nop,wscale 7,sackOK,TS val 1000362929 ecr 0], length 0
17:08:12.095885 IP 127.0.0.1.27937 > 127.0.0.1.10050: Flags [S], seq 2348730218, win 65228, options [mss 16344,nop,wscale 7,sackOK,TS val 1000365157 ecr 0], length 0

Zabbix agent log with extended debugging seems normal, seems to loop on its own stuff, nothing "extra" gets added when i run zabbix_get:

Code Select Expand

99440:20220303:172553.024 In update_cpustats()
99440:20220303:172553.024 End of update_cpustats()
99440:20220303:172553.024 zbx_setproctitle() title:'collector [idle 1 sec]'
99440:20220303:172554.074 zbx_setproctitle() title:'collector [processing data]'
99440:20220303:172554.074 In update_cpustats()
99440:20220303:172554.074 End of update_cpustats()
99440:20220303:172554.074 zbx_setproctitle() title:'collector [idle 1 sec]'
99440:20220303:172555.085 zbx_setproctitle() title:'collector [processing data]'
99440:20220303:172555.085 In update_cpustats()
99440:20220303:172555.085 End of update_cpustats()
99440:20220303:172555.085 zbx_setproctitle() title:'collector [idle 1 sec]'
99440:20220303:172556.149 zbx_setproctitle() title:'collector [processing data]'
99440:20220303:172556.149 In update_cpustats()
99440:20220303:172556.149 End of update_cpustats()
99440:20220303:172556.150 zbx_setproctitle() title:'collector [idle 1 sec]'
99440:20220303:172557.159 zbx_setproctitle() title:'collector [processing data]'
99440:20220303:172557.159 In update_cpustats()
99440:20220303:172557.159 End of update_cpustats()
99440:20220303:172557.159 zbx_setproctitle() title:'collector [idle 1 sec]'


I am totally at a loss what to try next.

On another host, same settings, same software versions, all works well:

Code Select Expand

# zabbix_get -s 127.0.0.1 -p 10050 -k agent.version
5.0.19

[xofer]

Ok, it seems i found the reason for zabbix agent not responding, but not the underlying cause yet.

It seems to be blocked by firewall:

Code Select Expand

<134>1 2022-03-03T18:15:09+02:00 ##### filterlog 88758 - [meta sequenceId="14242"] 10,,,02f4bab031b57d1e30553ce08e0ec131,lo0,match,block,in,4,0x0,,64,0,0,DF,6,tcp,60,127.0.0.1,127.0.0.1,65087,10050,0,S,2442566319,,65228,,mss;nop;wscale;sackOK;TS

This machine does not have Firewall > Rules > Loopback in the GUI which i have on other hosts where it contains automatically generated rules to allow anything from lo0.

I do not think that i have done any allow rules on lo0 (127.0.0.1) on any opnsense machines, but on this one, the section is missing, automatically generated rules are missing and i cannot choose Loopback interface when creating a "Floating" rule.

Also, verified with pfctl that on this host lo0 pass rules are missing.

I managed to create a floating rule to allow traffic from 127.0.0.1 to 127.0.0.1 ip on any interface and this seems to alleviate the original problem. Zabbix agent now responds.

However, the updated issue is now different - how come the Loopback section is missing and how do i get it back?