1.17 update caused eastpect crash

[nicktayl88]

Since updating to Zenarmor 1.17 I am getting the following message in logs

<6>pid 3076 (eastpect), jid 0, uid 0: exited on signal 3

When the error happens I lose all network access to the device for 2-3 minutes, this error is happening roughly once every 5-10 minutes.

Running on an Intel N5105 / I226-V device. Works fine if I disable Zenarmor, and worked fine before upgrading.

Any ideas on how I can fix this one?

[sy]

Hi,

Please share a report by selecting all checkboxes via Have Feedback option. The team checks the logs and configuration to find out the root cause.

[almodovaris]

I suggest switching between native driver and emulated driver.

[just4fun]

Hi,

I have the same behaviour, and I also find "exited on signal 3" in the logs.
My hardware is Intel N100 with 4 x 226-V, 16GB Ram, Samsung NVME.

I have enabled Zenarmor on two internal interfaces (LAN and GUEST), only the worker process
of the LAN interface gets terminated, not that from the the GUEST interface. As there is user traffic from the LAN and no User Traffic on
the GUEST I assume that it is some Packet coming from the LAN causing this, but I have no clue.
LAN Ethernet Link seems stable, also no indication of Link Flapping on the Switch the LAN port is connected to.
For me it happens at random times, every some hours.
When using the native netmap driver, the event is drastic, loosing Network and connectivity to the Opnsense device
itself, it looks like re-initialising is difficult.
When using the emulated netmap driver, the worker process is just restarted, and that's it, so it is barely noticable,
but still happens.

Also, in the log of the main process I find an indication that the worker process does not reply to keepalive
for 21 seconds and thus will be restarted. I guess that where the SIGQUIT Signal (3) comes from.

Inbound packets go first to Zenarmor and then to the firewall rules, if I understood correctly, so no way
to filter out trouble-causing packets before they reach Zenarmor.

Sounds more like a Zenarmor issue to me than a Opensense issue.

Regards,
Stephan

[just4fun]

Hi, I have been digging around in the logfiles after setting loglevel to Debug4.
In my setup, there is no DNS Service ( no unbound etc) running on the opnsense system, but I have
two DNS Servers (dnsmasq) running in the LAN, listening on Port 53, nothing special.
Both of them use
- DNSCrypt to query opendns servers for most queries
- plain DNS (udp/53) to my Carrier's DNS Servers ( to resolve hostnames for streaming services to get close-by CDN distribution points for Audio and Video streams)

So some DNS queries may go from the Opnsense host to the internal DNS Server and then again through the
opnsense host to external DNS.


What I found is, that at that time point, when the worker stops responding to heartbeats (you can see that in
the main log, it records the missed heardbeats each second until it reaches 21) I have a sequence like the following in the worker log: (10.XXX.YYY.254 is opnsense, 10.XXX.YYY.250 is internal DNS)

2024-05-02T18:24:34.608386 DBG1 [Connection::setDgaQueryState] [UDP][CLI][opnsense][10.XXX.YYY.254:38940]<->[UDP][SRV][hh_sgsvr_1][10.XXX.YYY.250:53][74bde58f-4c78-4f21-ac9d-4b9ad97e04f4]DGA query state changed to 'not_suspicious'
2024-05-02T18:24:34.608826 DBG2 [peeker_peek_packets] dns extracted name: cti.zenarmor.net
2024-05-02T18:24:34.608840 DBG4  WebApplicationsClassifier: trying hostname: cti.zenarmor.net
2024-05-02T18:24:34.608848 DBG4  WebApplicationsClassifier: trying hostname: zenarmor.net
2024-05-02T18:24:34.608863 DBG2 [PolicyManager::matchPolicyPacketInfo] dir: EGRESS, port: LAN, if: nm0::igc1, maclocal: 80ee73e6f0da, macremote: a8b8e001cfde, iplocal: 10.XXX.YYY.250, ipremote: 10.XXX.YYY.254, dnslocal: hh_sgsvr_1, dnsremote: opnsense, username: n/a, usergroup: n/a
2024-05-02T18:24:34.608872 DBG4 [UserDevice::setIPv4] Updated last_seen 80ee73e6f0da, Now: 1714667074:
2024-05-02T18:24:34.608879 DBG2 [PolicyManager::matchPolicyPacketInfo] [10.XXX.YYY.250:53<->10.XXX.YYY.254:38940] matched policy: System Default Policy
2024-05-02T18:24:34.608890 DBG4 WebCategoryManager: expired cache for domain: cti.zenarmor.net
2024-05-02T18:24:34.608907 DBG3 [10.XXX.YYY.250:53 <-> 10.XXX.YYY.254:38940] checkExceptions1 (System Base Policy)
2024-05-02T18:24:34.608917 DBG3 [10.XXX.YYY.250:53 <-> 10.XXX.YYY.254:38940] checkExceptions1.5 (cti.zenarmor.net)
2024-05-02T18:24:34.608925 DBG3 [Policy::checkExceptions] hostname: 'cti.zenarmor.net'
2024-05-02T18:24:34.608930 DBG3 [Policy::checkExceptions] hostname: 'cti.zenarmor.net'
2024-05-02T18:24:34.608935 DBG3 [Policy::checkExceptions] hostname: 'cti.zenarmor.net'
2024-05-02T18:24:34.608940 DBG3 [Policy::checkExceptions] hostname: 'cti.zenarmor.net'
2024-05-02T18:24:34.608944 DBG3 [Policy::checkExceptions] hostname: 'cti.zenarmor.net'
2024-05-02T18:24:34.608949 DBG3 [Policy::checkExceptions] hostname: 'cti.zenarmor.net'
2024-05-02T18:24:34.608957 DBG2 [10.XXX.YYY.250:53 <-> 10.XXX.YYY.254:38940] checkExceptions: DNS-pi is not whitelisted
2024-05-02T18:24:34.608966 DBG3 [10.XXX.YYY.250:53 <-> 10.XXX.YYY.254:38940] checkExceptions1 (System Default Policy)
2024-05-02T18:24:34.608975 DBG3 [10.XXX.YYY.250:53 <-> 10.XXX.YYY.254:38940] checkExceptions1.5 (cti.zenarmor.net)
2024-05-02T18:24:34.608988 DBG1 DNSWatcher: duplicate DNS entry: cti.zenarmor.net
2024-05-02T18:24:34.608994 DBG1 DNSWatcher: exceeded max pending resps size dropping cti.zenarmor.net[Policy::checkExceptions] [74bde58f-4c78-4f21-ac9d-4b9ad97e04f4] hostname: 'opnsense'
2024-05-02T18:24:34.609014 DBG3 [Policy::checkExceptions] [74bde58f-4c78-4f21-ac9d-4b9ad97e04f4] hostname: 'opnsense'
2024-05-02T18:24:34.609042 DBG4 WebCategoryManager: expired cache for domain: cti.zenarmor.net


These pattern around cti.zenarmor.net seem to occour only exactly at that times when the worker process stops responding to the heartbeats, so one could think it struggles on its own DNS query.
Grepping through the log for "DNSWatcher: exceeded max pending resps size dropping" only finds them at that time point in question.

hope this helps,
regards, Stephan

[just4fun]

Hi,

i have enabled unbound DNS for local name resolution, using the same internal DNS servers as forwarders
for all domains, and Zenarmor is running stable now!
The eastpect process no longer gets terminated by Signal 3, and I have re-enabled the native Netmap driver
without any issues yet.

For me that is another indication that the root cause is DNS related, which I consider a bug because
no packet crossing a monitored / protected interface should kick Zenarmor off the rails.
If it is really the cti.zenarmor.com query, I cannot tell, but it looks different in the logs than all the others,
and there is a time relationship between that query and eastpect re-initialisation.

Also, in the unbound reporting, I now see queries to

cti.zenarmor.com.  and
cti.zenarmor.com.<my local domain>.

which may be another bug, without any noticable side effects, it's just an DNS query that has no answer,
but it should'n not occour in the first place.

I am lucky for now because I have found a running solution ;-) (Though I do not understand 100% why it makes a difference weather the DNS is sent directly by the FreeBSD TCP Stack or via local unbound. Maybe, due to caching, it is send over the interface less often, or at other time points (not when Zenarmor itself waits for the reply).

regards,
Stephan

[turbo_lag]

Hey just4fun,

Wanted to thank you for pointing me in the right direction here.  I also experienced repeated crashing after updating to 1.17. 

I have an internal Unbound server and when looking at the queries from the firewall, I saw that all *.zenarmor.com domains asked for A and AAAA records except for cti.zenarmor.com.  This only asked for AAAA records.

Since I did not have IPv6 enabled at the time, it was failing.  After enabling IPv6 on my WAN interface to allow the firewall to respond via v6, ZenArmor has been up and running without a crash for a few hours.  I am also running the native Netmap driver.

If you dig/nslookup cti.zenarmor.com, there are absolutely A records, so maybe this was an oversight in the 1.17 code to not ask for them.  I'm curious if this is the same for you.

turbo_lag

[just4fun]

Hi turbo_lag,
that's great to hear that I could point you towards a solution!

Your finding again points to some issue around resolving cti.zenarmor.net.
In between, while poking around, I also noticed that the Cloud Node Status is going up and down,
sometimes 100%, sometimes less. May be related, if name resolution fails, it cannot connect.


Right now at time of writing, for me both european Servers are at 0%, Global CTI is at 100%. I am in Germany.

IPv6 may be one aspect of that, in my network I have IPv6 disabled on all Opnsense interfaces since my
Internet provider doesn't give me an IPv& address yet. I can run queries foer AAAA revords over IPv4,
but the result will be useless as IPv6 is disabled.

I hope that this will also point the developpers into the right direction.

best regards
Stephan

[turbo_lag]

Well, it seems this may have been only a partial solution.  After monitoring it yesterday, I am back to getting crashes, but only once every two hours or so.

I'm now trying the emulated driver and, if that fails, will spin up Unbound on the firewall for local resolution.

[Greg_E]

  me waiting for the semester to end so I can build and install my production firewall, and hoping this gets fixed before I get there.

[IHK]

Hi,

To detect Crash and Engine problems faster, you can activate "Core File" as follows.
You can change the log level to Zenarmor - Settings - Logging - Level (Debug 4).

https://www.zenarmor.com/docs/opnsense/configuring/configuring-zenarmor-privacy-settings-on-opnsense-firewall#enable-engine-core-file-generation

Please share a report by selecting all checkboxes via Have Feedback option. The team checks the logs and configuration to find out the root cause.

[sy]

Hi All,

Can you try to edit config file then restart the engine service by following the instructions

edit /usr/local/zenarmor/etc/eastpect.cfg
change the line
[CTI]
enabled    = false

Then restart engine service via Dashboard or
service eastpect restart
 

[Fórest]

Hi,

having the same problem since 09.05.2024 with version 1.17.1. Zenarmor crashes after 5-10s again and again. It dosen't matter if i use the native or emulated driver.

While debugging i narrowed it down, that the dns record "cti.zenarmor.com" cloud not be resolved. I'm using AdGuard Home that runs on port 53 on my OPNsense as my main DNS Server. In the log i cloud see, that this domain is not being resolved. When I disabled the Cloud Threat Intelligence it was running with an emulated driver.

Right now I uninstalled Zenarmor, so that my firewall is stable again.

For the last year i never had an problem with my Zenarmor installation using the native driver

[turbo_lag]

Editing eastpect.cfg to disable CTI seems to have resolved the issue for me.

[just4fun]

Zenarmor 1.17.2 ist available for installation. In the release notes they mention a fixed bug:

"The problem that was causing the program to crash when the firewall could not resolve the hostname of the global cyber threat intelligence server has been fixed, improving the stability and dependability."

https://www.zenarmor.com/docs/support/release-notes

I just installed it. The Global CTI Server is gone (also described in the release notes).

I am optimistic that the issues should be resolved by this update.