Create Alias from JSON format

[Benqer]

Hi,

I'm using spamhaus drop and dropv6 list.

https://www.spamhaus.org/blocklists/do-not-route-or-peer/

Currently the files are available in txt format, but in the future they wanted to change to json format.

"For long-term users of the DROP files in text format, we recommend you update your configuration with the above JSON files as soon as your cycles allow. If you require continued long-term use of a text file, the jq command can always be used to convert the JSON."

Is there an easy way to create the alias with the json files?

thanks!

[Legally a Shrimp]

No idea if there's an "opnSense way" of doing this, but here's how it could be done:


0) check System → Settings → Administration → Secure Shell → Secure Shell Server: [✓] Enable Secure Shell
0.1) click Save
0.2) SSH into opnSense
0.3) press 8, Enter

1) run ee /usr/local/www/scrape.php
1.1) copy and paste

Code: [Select]

<?php// this script comes "as is"// use it at your own risk$cacheDirectory = "/var/cache/scraper/";$cacheMaxAge = 60; // in seconds// no changes should be necessary below thiserror_reporting(0);header("Content-Type: text/plain");if (!is_dir($cacheDirectory)) {  if (!mkdir($cacheDirectory, 777, true)) {    echo "# could not create cache directory";    exit;  }}if (!@filter_var($_GET["url"], FILTER_VALIDATE_URL)) {  echo "# invalid url";  exit;}$currentTime = time();$currentDate = date(DATE_RFC2822);$cachePath6 = $cacheDirectory . "/" . md5($_GET["url"]);$cachePath4 = $cachePath6 . ".IPv4.txt";$cachePath6 = $cachePath6 . ".IPv6.txt";$cacheOldAge6 = false;if (!@$_GET["v"] || $_GET["v"] == 6) {  $cacheOldAge6 = $currentTime - filectime($cachePath6) > $cacheMaxAge;}$cacheOldAge4 = false;if (!@$_GET["v"] || $_GET["v"] == 4) {  $cacheOldAge4 = $currentTime - filectime($cachePath4) > $cacheMaxAge;}if ($cacheOldAge6 || $cacheOldAge4) {  $curlHandle = curl_init();  curl_setopt($curlHandle, CURLOPT_URL, urldecode($_GET["url"]));  curl_setopt($curlHandle, CURLOPT_RETURNTRANSFER, true);  curl_setopt($curlHandle, CURLOPT_HEADER, false);  curl_setopt($curlHandle, CURLOPT_TIMEOUT, 10);  curl_setopt($curlHandle, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/115.0");  $response = curl_exec($curlHandle);  $httpCode = curl_getinfo($curlHandle, CURLINFO_HTTP_CODE);  curl_close($curlHandle);  if ($response !== false && $httpCode == 200) {    $response = trim(preg_replace("/(?:[\#;]|\/{2}).*/", "", $response));    if (preg_match_all("/((?:[\da-f]{0,4}:){2,7}(?:(?:(?:(?:25[0-5]|2[0-4]\d|1?\d\d?)\.){3}(?:25[0-5]|2[0-4]\d|1?\d\d?))|[\da-f]{0,4}|:))(?:\/(12[0-8]|1[01][0-9]|[1-9]?[0-9]))?/", $response, $matches)) {      file_put_contents($cachePath6, "# {$currentDate} - IPv6\n" . implode("\n", array_map(function ($a, $b) {        if (filter_var($a, FILTER_VALIDATE_IP, FILTER_FLAG_IPV6)) {          return "{$a}/" . ($b ? $b : "128");        }      }, $matches[1], $matches[2])));    } else {      @unlink($cachePath6);    }    if (preg_match_all("/((?:(?:25[0-5]|2[0-4]\d|1?\d\d?)\.){3}(?:25[0-5]|2[0-4]\d|1?\d\d?))(?:\/(3[0-2]|[12]?[0-9]))?/", $response, $matches)) {      file_put_contents($cachePath4, "# {$currentDate} - IPv4\n" . implode("\n", array_map(function ($a, $b) {        if (filter_var($a, FILTER_VALIDATE_IP, FILTER_FLAG_IPV4)) {          return "{$a}/" . ($b ? $b : "32");        }      }, $matches[1], $matches[2])));    } else {      @unlink($cachePath4);    }  }}if (!@$_GET["v"] || $_GET["v"] == 6) {  echo @file_get_contents($cachePath6);}if (!@$_GET["v"] || $_GET["v"] == 4) {  echo @file_get_contents($cachePath4);}

1.2) press Escape, Enter, Enter

2) click Firewall → Aliases → [+] Add
2.0) (enabled should be checked by default)
2.1) copy and paste Name: IPv4_Spamhaus_DROP
2.2) select Type: URL Table (IPs)
2.3) set Refresh Frequency: Hours: 6
2.4) copy and paste Content: http://localhost/scrape.php?v=4&url=https://www.spamhaus.org/drop/drop_v4.json
2.5) copy and paste Description: Spamhaus DROP (IPv4)
2.6) click Save

3) click Firewall → Aliases → [+] Add
3.0) (enabled should be checked by default)
3.1) copy and paste Name: IPv6_Spamhaus_DROP
3.2) select Type: URL Table (IPs)
3.3) set Refresh Frequency: Hours: 6
3.4) copy and paste Content: http://localhost/scrape.php?v=6&url=https://www.spamhaus.org/drop/drop_v6.json
3.5) copy and paste Description: Spamhaus DROP (IPv6)
3.6) click Save

4) click Firewall → Aliases → [+] Add
4.0) (enabled should be checked by default)
4.1) copy and paste Name: IP_Spamhaus_DROP
4.2) select Type: Network(s)
4.3) copy and paste Content: IPv6_Spamhaus_DROP,IPv4_Spamhaus_DROP
4.4) copy and paste Description: Spamhaus DROP
4.5) click Save
4.6) click Apply

5) click Firewall → Rules → Floating → [+] Add
5.0) (disabled should be unchecked and quick should be checked by default)
5.1) select Action: Reject
5.2) select TCP/IP Version: IPv4+IPv6
5.3) select Destination: IP_Spamhaus_DROP
5.4) copy and paste Description: reject traffic to networks in Spamhaus DROP lists
5.5) click Save
5.5) optionally move the rule to where it makes sense
5.6) click Apply changes



Edit: This evening I had some more spare time, so I rewrote the script to no longer actually parse the JSON data but use regular expressions instead, which makes it a little more versatile.

[rolsch]

Thanks "Legally a Shrimp"!

I ask this in the DEV-Section again:
https://forum.opnsense.org/index.php?topic=41210.0

[flaax]

Thanks Legally a Shrimp for the great work with the php script!

For the instruction, point 5, firewall rule:
Please consider to choose block as action.
Since reject will send an anwser and you usually don't want this.


Also, if direction in
Source: <your alias / spamhaus_drop>
Destination is your OPNsense (or any)