Messages

Hi Ad,

no objections to anything you said, and I also appreciate the prompt and helpful support I got from you during this story.

Keeping deliveries within EU is a workaround that was discussed on the way, but that is not exactly the problem here.
The outcome was more related to some aspects of overall communication experience: some key questions never answered (or explicitly denied answer which would be fair), time frame stretched indefinitely. This went way beyond my boundary of acceptable roughly two months ago. Consequently, I simply cannot rely on the time estimations of your colleague with whom I was mostly communicating, and the likelihood of staying without appliance for too long is considerable. Besides, that already took inappropriate toll of effort.

Still, please note that I am very angry and frustrated now, so that the above is a biased impression.
I really hope I made it clear that it is an unfortunate personal experience which should not necessarily be considered as basis for any conclusions about the company.

Nevertheless, I would like to evaluate alternative paths. Just looking over alternative devices, unfortunately, does not render an answer easily. This is why I reckoned it would not harm asking, in case someone might have been looking in similar directions.

Hi,

I have been very happy with the performance of D840, and the design is lovely. Unfortunately, the communication with the manufacturer was one of the most dreadful I had ever. Everyone was nice, but I wasted more than three months on waiting and repeating the same questions over and over.
This is very much related to an edge case triggered by the warranty issue I have, but nonetheless, the warranty is effectively useless (at least in my country), which suggests to find some alternative hardware.

But the original need still stands. I would like:

• A more or less blackbox solution, so that OPNSense Business can be installed with modest efforts.
• Capabilities and performance on the level of D840 (Netboard A20, 4 cores, 8Gb): throughput with IDPS and at least one SFP+.
• Customer support that answers emails, not only for enterprise accounts.
• Not loud.
• Preferably, rack mountable.

I know there is a wide range of generic servers, but it is very difficult to compare on paper, and any information about noise is very difficult to find, especially for rack mountable devices: most people don't care how loud anything in the rack is.

Still, any relevant personal experiences will be greatly appreciated.

Now that the "custom options" are gone for Unbound DNS since OPNsense 21.7, how do I configure Unbound DNS with DNSCRYPT-PROXY ?

It appears that the only straight way is Enable Forwarding Mode with DNSCrypt-Proxy being listed in system DNS.
Ugly and will also create madness with multiple WANs.

The only problem with that is the port. You cannot specify a port in the system settings and you cannot have 2 services listening on the same port (53).

So am I right to assume that the custom options have only disappeared from the configuration GUI but are still taken into account when entered in unbound.conf?

Virtual IPs?

Now that the "custom options" are gone for Unbound DNS since OPNsense 21.7, how do I configure Unbound DNS with DNSCRYPT-PROXY ?

It appears that the only straight way is Enable Forwarding Mode with DNSCrypt-Proxy being listed in system DNS.
Ugly and will also create madness with multiple WANs.

@ingvarr,

Thanks for taking the time to post this. I think there's a bunch of good hints for us to use here. We'll have a thorough look and get back to you.

You are very welcome. And I am happy to elaborate upon any items and share my k8s manifests if relevant (running Elastic in single node microk8s).

BTW, I did forget one thing — recreating the indices if they died (e.g. Elastic was reset) — I did not find a better way than to reinstall Sensei and restore configuration (it might be possible to manually create stuff or to try and reuse something like that retire script, but I am too lazy), but then I saw it in the recent release notes (the version is not yet propagated to the repository).
Very nice.

[Permissions]

I have tried to reach with part of this to Sunnyvalley support, but I reckon that it gets broader and might benefit from a discussion.
Sensei does, indeed, look like a promising initiative. And the idea of offloading the DB out of the firewall is great — I'd love to see it more in OPNSense (partly there are pieces like remote logging here and there, but sparse).
Unfortunately, as of now, the support is very incomplete.
I am going to try and collect (perhaps, to be updated) the list of my own concerns. Any comments and opinions will be appreciated — both from SV and the community.
A disclaimer: Sunnyvalley has no obligations to me, for I did not buy any service (only considering), and the discrepancies listed hereby are not implied by any promises made by the company that I am aware of, thus this is just a conversation.

Permissions

Sensei requires a lot to run. That would make sense for an enterprise that can afford a dedicated ELK installation for the firewall, but for a humble home setup this is burdensome.
The documentation mentions indices (<HOSTUUID>_)?(conn|http|tls|alert|sip|dns)_<date>. Well, this is incorrect. First of all, it is dash, not underscore before date. Second, there are also aliases $1_all and $1_write, which makes sense, but would be nice to say a word in the manual.
Now, this is the role definition that seems to have worked for me:

    offensive_role_name:
      cluster: [ "monitor", "read_ilm", "manage_own_api_key" ]
      indices:
        - names:
      #      - "/(conn|http|tls|alert|dns|sip)(_all|_read|_write)(-[23][0-9][01][0-9][0-9][0-9])?/"
            - "/(conn|http|tls|alert|dns|sip).*/"
          privileges: [ "all" ]

Please note the commended more precise definition: I believe it should work, but I just got tired of guesswork.
Interestingly, when I alter the DB URL in the configuration, I get an error — and I do, even if I give sensei user super permissions. Still, the indices are getting created and rolled over, all right.
But of course, it would be better to use a dedicated service account.

Suggested actions:

• Update the manual to reflect actual naming and to mention aliases.
• Provide a role definition example with least privilege necessary.
• Consider switching to service accounts with tokens instead of username and password.

Index naming

By the same reason of reasonably expected coexistence with other datasets, it is a pain to have a diverse set of names pertaining to a single service without a common prefix.

Possible solutions:

• Provide a configuration option for both free and business versions to prepend something to index and alias names is necessary for the sake of prettiness and tidiness,
• Or just prefix sensei-<version>_

Index properties, replicas number above all

I have noticed that newly created indices have zero replicas.
Moreover, scripts/datastore/retire_elasticsearch.py says in elasticsearch_rollover(index_name):
       set_numberof_replicas = '{"index": {"number_of_replicas": 0}}'
Why? This hurts. I can't even touch the ELK without missing primary shards...

Possible solutions:

• Make replicas number configurable,
• Or make replicas number unset (so that the cluster settings apply),
• Or make index template configurable and/or allow to enable relying on the pre-created index template

Apologies again, it is not happening by the manual: it wants permissions for "conn_all", not "conn_<date>" as described.

Apparently, index rights were insufficient. It wanted to know something about the cluster.
This is what allowed me to move to the point of creating indices:

roles.yml: |-
    stupid_sensei:
      cluster: [ "monitor" ]
      indices:
        - names:
            - "/(conn|http|tls|alert|dns|sip)_[23][0-9][01][0-9][0-9][0-9]/"
          privileges: [ "all" ]

Would be good to know what can vast "monitor" be reduced to.

Missed it initially:

Free and Home tiers will have indexes with [indextype]_[date] format

Well, that is very unfortunate. Would be much, much more convenient to be able to set a common prefix. Same applies to paid subscription. Otherwise it is implied that I either run a separate moose for Sensei, or trust it utterly...

No, I am not a premium user. I want to try it first. But I am hesitant to let ELK run on the appliance.

However, it is not at all obvious how to set up appropriate permissions for Sensei. The manual https://www.sunnyvalley.io/post/using-remote-elasticsearch-for-sensei-reporting/) is vague: it does say that HOTUUID information is available for premium, but before that it also lists the names of the indices.

Whatever is correct, is not that important. I would like to evaluate Sensei, but the documentation seems to assume that the user will give very high privileges in ElasticSearch, which is just a bad idea.
Unfortunately, I am not that familiar with Elastic (trying to avoid the beast usually) — so it is a bit challenging.

Any hint on the right path here will be greatly appreciated.

Hi,

I'd like to use ES database for other things in addition to Sensei. Which means that everyone shall only have access to own indices. Unfortunately, it is not possible to set proper permissions for Sensei user without knowing the host id (node-uuid is not set in the beginning). Is there a way to retrieve or set to a fixed value somehow?

Th.

Hi.
I assume that I am missing some of the basics, but what about passing part of the traffic through VPN?
I have some wireguard interfaces that grab traffic for some of the nodes, and they have their own gateways.
Now, physically that all goes to the same WAN, upon firewall rules with gateway specified.
So, I have:
* WAN1 with shaper rules, gets some traffic.
* WAN_covert_hole87 on Wireguard (physically same WAN1 link), gets some traffic.

Does WAN_covert_hole87 need a separate pair of rules, or shaper applies to anything that goes to the physical interface, no matter virtual gateway ceremonies?

[1. The cache sometimes causes trouble]

I have set the Unbound + DNSCrypt-proxy bundle as described, for it makes sense.
Unfortunately, some of the requests seems to fail from my computer.
Further investigation revealed the following:

1. The cache sometimes causes trouble
When DNSCrypt-proxy restarts for too long, Unbound panics and says that all is lost, and then my macOS caches that there is no IP for the name, and I get a blank page.
This behaviour can be eliminated with forward-first: yes, as someone mentioned, but this poses obvious disadvantages.
Another possibility is forward-no-cache: yes, but that might be even worse.
Now, a better option is to enable Unbound DNS :: Advanced :: Serve expired responses, because the most affected records are short-TTL ones.

2. DNSSEC hardening causes trouble
When Unbound DNS :: Advanced :: Harden DNSSEC data is enabled (a naive thing to do, it was), this often happens:

Code Select Expand

2021-06-20T03:33:04 unbound[76608] [76608:0] debug: validator[module 0] operate: extstate:module_wait_module event:module_event_moddone
2021-06-20T03:33:04 unbound[76608] [76608:0] info: validator operate: query wiki.mageia.org. A IN
2021-06-20T03:33:04 unbound[76608] [76608:3] debug: verify: signature mismatch

Disabling the option solves the problem.

Now, what I don't understand at all is that when I change forwarders from DNSCrypt to 9.9.9.9@53, it seems fine (with other settings unchanged):

Code Select Expand


2021-06-20T15:57:58 unbound[13980] [13980:1] debug: validator[module 0] operate: extstate:module_wait_subquery event:module_event_pass
2021-06-20T15:57:58 unbound[13980] [13980:1] info: validator operate: query wiki.mageia.org. A IN
2021-06-20T15:57:58 unbound[13980] [13980:1] info: Verified that unsigned response is INSECURE

Does it mean that DNSCrypt somehow tries to sign that's unsigned, causing frustration in Unbound?..

Some awkward behaviour is also visible around them new TYPE65 requests, but I did not reveal anything specific.

3. Even when all works, Unbound seems to treat forwarders inefficiently

Code Select Expand


2021-06-20T15:57:57 unbound[13980] [13980:1] info: resolving wiki.mageia.org. A IN
2021-06-20T15:57:57 unbound[13980] [13980:1] info: processQueryTargets: wiki.mageia.org. A IN
2021-06-20T15:57:57 unbound[13980] [13980:1] info: sending query: wiki.mageia.org. A IN
2021-06-20T15:57:57 unbound[13980] [13980:1] debug: sending to target: <.> 9.9.9.9#53
2021-06-20T15:57:57 unbound[13980] [13980:1] debug: cache memory msg=138269 rrset=146053 infra=10617 val=136881
2021-06-20T15:57:57 unbound[13980] [13980:1] debug: iterator[module 1] operate: extstate:module_wait_reply event:module_event_reply
2021-06-20T15:57:57 unbound[13980] [13980:1] info: iterator operate: query wiki.mageia.org. A IN
2021-06-20T15:57:57 unbound[13980] [13980:1] info: sanitize: removing extraneous answer RRset: sucuk.mageia.org. A IN
2021-06-20T15:57:57 unbound[13980] [13980:1] info: response for wiki.mageia.org. A IN
2021-06-20T15:57:57 unbound[13980] [13980:1] info: reply from <.> 9.9.9.9#53
2021-06-20T15:57:57 unbound[13980] [13980:1] info: query response was CNAME
2021-06-20T15:57:57 unbound[13980] [13980:1] info: resolving wiki.mageia.org. A IN
2021-06-20T15:57:57 unbound[13980] [13980:1] info: processQueryTargets: wiki.mageia.org. A IN
2021-06-20T15:57:57 unbound[13980] [13980:1] info: sending query: sucuk.mageia.org. A IN


So, what I read from this, when Unbound gets CNAME instead of A, it does query for that A, but first complains that it is from another domain. I am mildly concerned that this is not right.

Recommendations for the tutorial

• Add a suggestion to enable Unbound DNS :: Advanced :: Serve expired responses.
• Add a suggestion to disable Unbound DNS :: Advanced :: Harden DNSSEC data.
• Propose a consensus for DNSBL: it is possible to use them in either DNSCrypt-Proxy or Unbound, but some advice might be helpful for users. My personal opinion is that Unbound is a better place (as of 21.4), but that might be debatable.
• Investigate the extraneous RRSet magic.

Thank you for the attention.