Hash-based URL filtering with "own" blacklist in OPNsense – is it feasible?

Started by friede, Today at 12:04:43 PM

Previous topic - Next topic
Hello everyone,

I'm currently working as an IT administrator and studying part-time in a Bachelor's program. As part of my academic research, I'm evaluating the feasibility of integrating a legally maintained domain blocking list from a German federal authority into an open-source firewall — ideally OPNsense.

This so-called "BPjM module" is used to block access to internet content considered harmful to minors in Germany. The list is not public and is only provided to manufacturers under a formal agreement. However, I'm in contact with the responsible authority and may obtain access for research purposes.

The list consists of three components:
- MD5 hash of the domain (with optional "www." stripped)
- MD5 hash of the URL path
- Path depth (as an integer)

Each list entry is line-matched across three files.

The idea is to integrate this into OPNsense via a transparent proxy with HTTPS filtering (SSL Bump), using a custom helper script to match requested URLs against the hash list and block access if a match is found — all without any client-side configuration.

My questions:
- Has anyone used a hashed domain/path list like this with Squid in OPNsense?
- Is `external_acl_type` + a custom script a viable approach? (suggested by ChatGPT...)
- Are there better-suited open-source firewall systems for this use case?

I'm still assessing whether this could be the foundation for a solid thesis project. Any experience, advice, or recommendations would be greatly appreciated.

Thanks in advance!
-Friede