Messages

Hi,

after the recent updates to 24.7.8 and Zenarmor 1.18.2 the cloud Node status shows "Up" again after a
PPPoE Interface reset. It may have been fixed also already in one tor two earlier Update Iterations
(quite sure it also worked with Zenarmor 1.18.1, I did not test every single Version).
Great - and thanks for your support

Greetings,
Stephan

I don't think it is just a cosmetic issue. Zenarmor looses connectivity to it's cloud nodes, so it is unable
to query them for reputation information. Restarting the engine re-establishes the connection, that's why I still use my workaround decribed earlier in this thread, by adding a zenarmor engine restart command to the end of
/usr/local/etc/rc.newwanip.
I have to re-edit the file after every update. While I can live with it, I still think it should be fixed,
I think it cannot be so difficult to re-establish those connections from scratch when they are lost,
instead of sitting there with known-to-be-offline connections and waiting forever. I'd expect few lines of code
doing that.

Regards,
Stephan

The workaround works fine since some days now. So it is clear, that under normal circumstances,
Zenarmor maintains a good Cloud Node Status. It just does not recover from a PPPoE Interface
reset. ( it may do so if the Carrier justs disconnects the interface to enforce IP adress renewal,
I didn't test that scenario, but it may be different from performing an interface reset from a cron job
to force the IP address renewal to a specific time.)

Upgrading to OPNsense 24.1.9_3-amd64 didn't improve this. I had to re-add the zenarmorctl engine restart into
rc.newwanip.

best regards,
Stephan

looks like zenarmor either misses that the public IP adress changes, or it ignores it and does not open
a new connection to the CTI servers.

root@opnsense:/usr/local/etc # netstat -n | grep 5355
udp4       0      0 185.171.XXX.YYY.13285   35.198.172.108.5355   
udp4       0      0 185.171.XXX.YYY.44679   34.65.117.157.5355     
udp4       0      0 185.171.XXX.YYY.25812   34.65.117.157.5355     
udp4       0      0 185.171.XXX.YYY.56199   35.198.172.108.5355   

after PPPoE interface reset still shows the udp connections from the old IP address instead of the new IP address on the PPPoE interface.

In the logs the Cloud Reputation Servers just go from healthy to unhealty.

My workaround now is to restart zenarmor after a new IP is aquired by adding this to
/usr/local/etc/rc.newwanip

/* restart zenarmor engine to re-establish connection to CTI servers */
log_msg("Restarting zenarmor engine");
mwexecf('/usr/local/sbin/zenarmorctl engine restart');

seems to work ok for now.

... repeated the test. Node Status goes Down right after PPPoE Interface reset via cron job and stays down,
now for > 2 hours.

Hi,

I have been some days off in holoday, so I didn't follow this activity, sorry.

I now have 1.17.4 installed, same behaviour. But I have been able to nail it down.

Because my ISP forces a PPPoE Reset every 24 hours to renew the IP Address assigned to my Firewall,
I am running a cron job which performs a regular interface Reset of the PPPoE Interface in the very early
morning at 4.00 to always have the IP Renewal at night.

It seems that this is the point in time where the Node status goes down and does not recover (what I would be expecting). I verified this by having another cron job resetting the interface at "wake hours" ;-), right before
execution I have restarted the Zenarmor to have the Nodes Status 100%, right after execution It was Down at 0%
and didn't recover in the following 10 Minutes.


Thanks a lot,
Stephan

yes, node status sill "DOWN" 0% after some time. Still not shure if it is really an issue or if the system is using
the cache. Only me and my wife behind this zenarmor, so not too much different websites visited.

lewald,

can you please describe in a liitle more detail? I do not have / can not find those files / directories.
cd ~ui/zenarmor/#/0/settings gives me "no such user"
Thanks a lot,
Regards,
Stephan

Recent Updates (OPNsense 24.1.7_4-amd64 and Zenarmor 1.17.3 - May 20, 2024 4:08 PM)
brought back the CTI Server list in the Zenarmor Settings -> Cloud Threat Intelligence.

Cloud Nodes Status still goes down to "DOWN 0%"

I do not protect WAN Interfaces with Zenarmor, only internal LAN and Guest Networks

I can ping the CTI Servers.
The test with nc -u seems to be a bit pointless,
I also get the succeeded reply, but when I try random ports
or IP adresses I get the same succeeded message.
I always have to use CRTL-C to get out of nc then - no further activity
occours after the succeeded message.

What would be the correct way to test UDP Connectivity to those servers?

running

root@opnsense:~ # nc -u -v 104.198.6.78 5355
Connection to 104.198.6.78 5355 port [udp/*] succeeded!

in one window and

root@opnsense:~ # tcpdump -n -i pppoe1 host 104.198.6.78
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on pppoe1, link-type NULL (BSD loopback), capture size 262144 bytes
11:20:20.368732 IP 113.30.181.18.2034 > 104.198.6.78.5355: UDP, length 1
11:20:20.368768 IP 113.30.181.18.2034 > 104.198.6.78.5355: UDP, length 1
11:20:20.368801 IP 113.30.181.18.2034 > 104.198.6.78.5355: UDP, length 1
11:20:20.368818 IP 113.30.181.18.2034 > 104.198.6.78.5355: UDP, length 1

in another window tells me nothing comes back.

Adding the Source of the CTI Servers and the Source Port 5535 into an ACL didn't change
anything, so either really nothing is coming back, or the servers only respond to
specific UDP requests which authenticate as legitime queries, which is what I think and hope ...

Hi,

I notice in the Zenarmor Dashboard that the Cloud Nodes Status drops over time to 0%

This has been the case since 1.17.1 (maybe earlier, I am a new user to Opnsense and Zenarmor),
in 1.17.2 the Global CTI Server disappeared ( as described in the release notes) , showing only Europe and Europe2 for me.

In addition, in 1.17.2, in the Zenarmor -> Settings -> Cloud Thread Intelligence -> Cloud Reputation Servers
I see no longer any Servers. In 1.17.1 I had a bunch of Servers shown there (US, Australia, etc).
The "Re-check Reputation Servers" Button is still there.

In the Log under System -> Log Files -> Backend I have Log entries like the following coming once per
Minute (so likely triggered by the "sensai periodicals" cron Job), which could be related:


[d65d2101-36a1-46cf-861d-7cfaaa43934b] Script action failed with Command '/usr/local/opnsense/scripts/OPNsense/Zenarmor/nodes_status.py --mode 'read'' returned non-zero exit status 1. at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/actions/script_output.py", line 44, in execute subprocess.check_call(script_command, env=self.config_environment, shell=True, File "/usr/local/lib/python3.11/subprocess.py", line 413, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '/usr/local/opnsense/scripts/OPNsense/Zenarmor/nodes_status.py --mode 'read'' returned non-zero exit status 1.


Also, I have Logs once per Minute pointing to
/usr/local/opnsense/scripts/OPNsense/Zenarmor/userenrich.py

which may or may not be related.

Does Cloud Nodes Status = Down or Cloud Nodes Status < 100% imply loss of security,
because Reputation Data cannot be retreived, or does it cause Network Delays because
Queries to the Reputation Servers are stalling?

I am not 100% sure but it could be the case that the CTI Servers disappeared after the thast
Upgrade to Opnsense itself, which came 2 days or so after Zenarmor 1.17.2. Opnsense ist at 24.1.7.

best regards,
Stephan

Zenarmor 1.17.2 ist available for installation. In the release notes they mention a fixed bug:

"The problem that was causing the program to crash when the firewall could not resolve the hostname of the global cyber threat intelligence server has been fixed, improving the stability and dependability."

https://www.zenarmor.com/docs/support/release-notes

I just installed it. The Global CTI Server is gone (also described in the release notes).

I am optimistic that the issues should be resolved by this update.

Hi turbo_lag,
that's great to hear that I could point you towards a solution!

Your finding again points to some issue around resolving cti.zenarmor.net.
In between, while poking around, I also noticed that the Cloud Node Status is going up and down,
sometimes 100%, sometimes less. May be related, if name resolution fails, it cannot connect.


Right now at time of writing, for me both european Servers are at 0%, Global CTI is at 100%. I am in Germany.

IPv6 may be one aspect of that, in my network I have IPv6 disabled on all Opnsense interfaces since my
Internet provider doesn't give me an IPv& address yet. I can run queries foer AAAA revords over IPv4,
but the result will be useless as IPv6 is disabled.

I hope that this will also point the developpers into the right direction.

best regards
Stephan

Hi,

i have enabled unbound DNS for local name resolution, using the same internal DNS servers as forwarders
for all domains, and Zenarmor is running stable now!
The eastpect process no longer gets terminated by Signal 3, and I have re-enabled the native Netmap driver
without any issues yet.

For me that is another indication that the root cause is DNS related, which I consider a bug because
no packet crossing a monitored / protected interface should kick Zenarmor off the rails.
If it is really the cti.zenarmor.com query, I cannot tell, but it looks different in the logs than all the others,
and there is a time relationship between that query and eastpect re-initialisation.

Also, in the unbound reporting, I now see queries to

cti.zenarmor.com.  and
cti.zenarmor.com.<my local domain>.

which may be another bug, without any noticable side effects, it's just an DNS query that has no answer,
but it should'n not occour in the first place.

I am lucky for now because I have found a running solution ;-) (Though I do not understand 100% why it makes a difference weather the DNS is sent directly by the FreeBSD TCP Stack or via local unbound. Maybe, due to caching, it is send over the interface less often, or at other time points (not when Zenarmor itself waits for the reply).

regards,
Stephan