I have been experimenting with the Unbound Block List (DEFAULT and USER DEFINED) functionality.
It appears that the DEFAULT list removes duplicate domain and domain pattern entries.
Example: If I select all OPNsense DEFAULT options, the logs indicate it loads over 6M individual entries and coalesces it down to @ 3.5M unique entries. Unfortunately, this final list does not appear to be written to disk for final inspection.
It is not clear if the same thing happens with the USER DEFINED lists and it appears it does NOT remove duplicates between the DEFAULT and USER DEFINED lists.
Example: If I add some of the same lists that are in the DEFAULT lists into the USER DEFINED LISTS, the final aggregate number increases (should NOT increase because they are identical duplicate lists, URLs are identical).
A simple strcmp() of the list URLs would eliminate such duplicates along with a possible warning log for the user to clean up his duplicate main lists.
Also, it would be nice if both DEFAULT and USER LISTS would have granular duplicate removals at the domain and domain pattern levels (e.g. sort -u).
It appears that the DEFAULT list removes duplicate domain and domain pattern entries.
Example: If I select all OPNsense DEFAULT options, the logs indicate it loads over 6M individual entries and coalesces it down to @ 3.5M unique entries. Unfortunately, this final list does not appear to be written to disk for final inspection.
It is not clear if the same thing happens with the USER DEFINED lists and it appears it does NOT remove duplicates between the DEFAULT and USER DEFINED lists.
Example: If I add some of the same lists that are in the DEFAULT lists into the USER DEFINED LISTS, the final aggregate number increases (should NOT increase because they are identical duplicate lists, URLs are identical).
A simple strcmp() of the list URLs would eliminate such duplicates along with a possible warning log for the user to clean up his duplicate main lists.
Also, it would be nice if both DEFAULT and USER LISTS would have granular duplicate removals at the domain and domain pattern levels (e.g. sort -u).