Zenarmor External Elastic Database (Permission and TLS Issue)

[cyyyyyy]

New to OPNSense and trying out Zenarmor. Been trying the whole day and trying to set up external elastic-search.
What I want to achieve:
- Self-Signed cert to work
- Create user with appropriate permission for zenarmor to use:


Steps I have taken:
- Copy the generated ca cert and key and import into OPNSense "SYSTEM: TRUST: CERTIFICATES"
- Setup Zenarmour and Proceed as Usual...

What Happened:
- After copying the Cert into OPNSense, at least I am getting "Reporting Database" is running
- But when I go into Reporting & Data > Reporting Database Settings > "Perform Index Check" I get "Remote database connection failed" However this does not show up when I reinstall zenamour and add it at the installation page.
- Indicies still get created though
- So I am not very sure whether it is a cert issue or a permission issue
- Creating a non self-signed, no user account elastic does not have this issue.


Permission Given for the Zenamor Elastic User:
Cluster Privileges: manage_index_templates, manage_ilm, monitor
Index Privileges: All where given [all, write, delete, manage, manage_ilm, create_index, auto_configure]

   
• alert\*
• conn\*
• dns\*
• http*
• sip*
• tls*

Have also try to "curl" my IP from OPNsense shell to "10.0.0.27:9200" I will get:

Code Select Expand


curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html


curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Even though CA cert were installed already, any idea what could have went wrong?

Docker Compose

Code Select Expand




version: "3.8"
volumes:
  certs:
  es01-data:
  kibana01-data:


networks:
  elk-network:


services:
  # Only for initial setup
  setup:
    container_name: setup
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [ ! -f config/certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f config/certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: elk01-es01\n"\
          "    dns:\n"\
          "      - elk01-es01\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "      - 10.0.0.27\n"\
          "  - name: elk01-kibana01\n"\
          "    dns:\n"\
          "      - elk01-kibana01\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "      - 10.0.0.27\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://elk01-es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://elk01-es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/elk01-es01/elk01-es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120
    networks:
      - elk-network
    env_file:
      - stack.env


  elk01-es01:
    depends_on:
      setup:
        condition: service_healthy
    container_name: elk01-es01
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    labels:
      co.elastic.logs/module: elasticsearch
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - es01-data:/usr/share/elasticsearch/data
    ports:
      - ${ES_PORT}:9200
    environment:
      - node.name=elk01-es01
      - cluster.name=${CLUSTER_NAME}
      - discovery.type=single-node
      - network.host=0.0.0.0
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/elk01-es01/elk01-es01.key
      - xpack.security.http.ssl.certificate=certs/elk01-es01/elk01-es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/elk01-es01/elk01-es01.key
      - xpack.security.transport.ssl.certificate=certs/elk01-es01/elk01-es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120
    networks:
      - elk-network
    env_file:
      - stack.env


  elk01-kibana01:
    depends_on:
      elk01-es01:
        condition: service_healthy
    container_name: elk01-kibana
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    labels:
      co.elastic.logs/module: kibana
    volumes:
      - certs:/usr/share/kibana/config/certs
      - kibana01-data:/usr/share/kibana/data
    ports:
      - ${KIBANA_PORT}:5601
    environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=https://elk01-es01:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
      - XPACK_SECURITY_ENCRYPTIONKEY=${ENCRYPTION_KEY}
      - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=${ENCRYPTION_KEY}
      - XPACK_REPORTING_ENCRYPTIONKEY=${ENCRYPTION_KEY}
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120
    networks:
      - elk-network
    env_file:
      - stack.env


[sy]

Hi,

Normally Zenarmor handles self-signed cetificate. Please share a report by using Have Feedback option in UI by selecting Zenarmor logs and configuration checkboxes.

[cyyyyyy]

Hi have confirm it is neither a certificate or permission issue.


1. I realized indices are getting created
2. I think there is a bug, the password is not getting passed correctly when performing the "Perform Index Check"

Code Select Expand


[::INFO::] 2024-03-05 03:46:44 <FILE: ConfigurationController.php ConfigurationController:dbPerformIndexCheckAction>
[REQUEST] GET /api/zenarmor/configuration/db-perform-index-check?
[RESPONSE_BODY]
{"error":true,"message":"Remote database connection failed.","indexes":[]}


Have also submitted the feedback

[furfix]

Same issue here. Everything is working fine, except for the Index Check. The connection fails.

[sy]

Hi There,

Thanks for reporting the issue. We are going to investigate it and publish a patch with the upcoming maintenance release.

[Styx13]

Hello

I have a similar issue here.

Running OPNsense 24.7.1

Zenarmor version:
Engine 1.17.6  (Aug 5, 2024 12:52 PM)
Database 1.17.24080514 (Aug 5, 2024 12:52 PM)

Elasticsearch 8.11.3

The index check reports connection fails as described by furfix with the following message:
"Error (200)
Remote database connection failed."

But in addition to that, on the Zenarmor Dashboard, the reporting database status is Stopped:
"Reporting Database
Status: Stopped
Type: Remote Elasticsearch
Version: 8.11.3"

Also in the dashboard, I never see any data, other than the traffic graph (no top threat, no top host, no top apps, ..)
Under report: there is no data to display
Under live sessions: empty

But under Settings > Reporting & Data, if I click "Perform Health Check" I guess a success message:
"Success
Health check performed successfully."

If I connect to Elasticsearch and look for indices, I can find the indices created by zenarmor.
If I query those indices for all records, I can see over 2000 records in some of them and growing.

So somehow it is failing to access the data that it is writing.

I checked my user's role and it has permissions "all" for cluster and "all" for "zenarmor_*" indices.

Am I missing something?

[Styx13]

Ok i got it to work by giving the elasticsearch user for zenarmor the "superuser" role.
And now every is green and working.

So why isn't it sufficient to give it "all" for cluster and "all" for "zenarmor_*" indices.?