OPNsense Forum
Archive => 18.7 Legacy Series => Topic started by: datiscum on October 30, 2018, 09:31:47 pm
-
First simple test:
I have assembled the URL to test the Firewall API as follows.
https://192.168.11.1/api/firewall/aliasutil/aliases
{"message":"controller OPNsense\\Firewall\\Api\\AliasutilController not found","status":400}
In the path: /usr/local/opnsense/mvc/app/controllers/OPNsense/Firewall/Api
I copied the following file
cp AliasUtilController.php AliasutilController.php
Please pay attention to upper/lower case here
After that it worked and I got back a list of all defined aliases.
Did I do something wrong?
or does it have to be corrected ?
Which call can be used to reload an "alias" ?
-
This should do it. :)
https://192.168.11.1/api/firewall/alias_util/aliases
Cheers,
Franco
-
PS: a POST to api/firewall/alias/reconfigure will reload the aliases for the firewall to use.
-
Thanks, the URL works now.
But by reloading an alias, I meant something else.
I would like to trigger the reloading of a "URL Table (IPs)" before the time has elapsed.
e.g. /api/firewall/alias/reconfigure/MyAlias
I couldn't find an API call that seemed to do that.
-
Hey Franco - I tried just now (running latest) and I had the exact same experience datiscum had...
Before I copied / soft linked the file, I got an error. After I copied it worked.
Datiscum: Where is this documented? I've been meaning to get a start on it for something similar - pushing IP's to block into a table through the API. Basically moving the results of utilities like log monitoring to block on the edge of the network.
Thanks!!!
m
-
Hi m,
Did you use the correct URL? ".../alias_util/..."
New API firewall bits in docs are not up yet, but should arrive soon... https://docs.opnsense.org/development/api.html?highlight=api#core-api
Cheers,
Franco
-
Ugh - I see the difference now on third look. Thanks :-)
-
Hi m,
I looked at the code on my OPNSense, with "ssh" access to the console.
Looks like it's not on the agenda right now, as I need it.
So far I simply added a function to the file "/usr/local/opnsense/mvc/app/controllers/OPNsense/Core/Api/FirmwareController.php".
public function aliasreloadAction()
{
$response = array();
$stat = stat('/var/db/aliastables/HillscheidIPv4BlockList.txt');
if (!$stat)
{
$response['command'] = 'stat';
$response['status'] = 'failed';
}
{
$ctime = $stat['ctime'] - 93600;
$response['command'] = 'touch';
if (!touch('/var/db/aliastables/HillscheidIPv4BlockList.md5.txt', $ctime, $ctime))
{
$response['status'] = 'failed';
}
else
{
system('/usr/local/etc/rc.update_urltables_datiscum ', $ret);
$response['status'] = 'ok';
}
}
return $response;
}
About the API URL "https://IP:Port/api/core/firmware/aliasreload"
he reloads my alias. This has been working very well for a few years now.
But with every update the API file has to be updated.
-
It's easier to discuss actual problem descriptions. It looks like you say URL tables are not applied because their TTL hasn't expired yet and you look for a force flush variant?
Files provided by OPNsense will be overwritten for safety reasons, but it's easily possible to build your on API endpoints. That's how plugins are made.
Cheers,
Franco
-
Yes, that's right!
A TTL could always be deposited for an alias.
But to make it work in real time you have to be able to force a reload.
I have my own Syslog which gets the data from several other SyslogServers. These are stored in a DB and can lock out an IP address on the OPNSense within seconds. If the syslog server triggers an alias reload for this.
Wenn die neue API bereit ist, sollte diese Funktion kein Problem darstellen ! oder ?
Thanks
-
Die neue API ist ja schon da, fehlt also nur das Force-Reload :)
But back to English... It's a special use case that should already be covered by the current API but seems not to be. I'll talk to Ad about this next week.
Cheers,
Franco
-
@datiscum: Thanks for the detail - sounds like you are doing exactly what I want to do. I needed something "smarter" and more general than individual sysems like fil2ban and other log watchers which ban locally without context. I wanted to collect that data, filtering for false positives (for example a bad smtp password from a site with lots of recent GOOD logins shouldn't be banned, but logged / warned.
I was looking for a way to insert / delete table entries as I thoguht this might be more efficent than a table alias reload but I imagine if that's possible you will be looking at that too.
@franco: When you hear back about the apparent omission, can you please post back here? It sounds like if that function was present, @datiscum wouldn't have to patch each time.
Do either of you have any thoughts about the most efficent way to accomplish this sort of thing (i.e. editing of tables?) I wouldn't want too frequent reloads to place an undo load on the router.
If you look at http://www.openbsd.org/faq/pf/tables.html there is an example:
#pfctl -t spammers -T add 203.0.113.0/24
#pfctl -t spammers -T delete 203.0.113.0/24
#pfctl -t spammers -T show
Is there a preferred way to invoke that sort of thing in OPNsense via API? Or is ssh the only way?
We could periodically update the table file (in case of reloads or fail over) on active / backup carp units, and use the add/delete as a way to modify the live rule set instead of constantly reloading.
My assumption is that reloading the table frequently would have much higher load than just editing the live table?
Is there an API method to do this (or generic API for a command?) or would it be better to ssh to the router, run teh associated pfctl options, write the file, and use the API just for periodic reloads?
Thanks again guys :-)
;D
-
The nginx does this in its permanent ban feature ("Banned") from the logs. You may need a plugin to do this currently.
-
@franco
Did you talk about it -- and what came of it ?
Will the function be available soon?
Thanks,
Sascha
-
Hi,
I'm trying to add an IP to an alias list through API. So I POST to http://192.168.0.1/api/firewall/alias_util/add/<MY_ALIAS_NAME
with these datas:
address:192.168.0.200
But all I get is an error:
{
"errorMessage": "[OPNsense\\Firewall\\Alias:aliases.alias.bdb02887-c1c8-4c28-a337-dd833a55e704.content] Entry \"192.168.0.200/32\" is not a valid hostname or IP address.\n",
"errorTitle": "An API exception occured"
}
Did I do something wrong? Or is the API not ready yet?
-
@camdarley
It seems like you found a minor regression, the same validation error also triggers in the UI when using host type networks.
https://github.com/opnsense/core/commit/c7d8f5c1017b9d0ee0522e1089bf12c7120ae284 (https://github.com/opnsense/core/commit/c7d8f5c1017b9d0ee0522e1089bf12c7120ae284) should fix the issue.
Best regards,
Ad
-
Thanks AdSchellevis.
I've been able to make it work by setting the alias type as Network instead of Host.
But then I noticed another issue: if the alias is empty, it's not possible to add an address.
This is the error I get:
{ errorMessage:
'[OPNsense\\Firewall\\Alias:aliases.alias.bdb02887-c1c8-4c28-a337-dd833a55e704.content] Entry "" is not a valid hostname or IP address.\n',
errorTitle: 'An API exception occured'
}
When there is at least one address in the alias, it work.
-
Hi camdarley,
It looks like the address tag is omitted from your request, you can easily check functionality using the gui and inspect the request made by your browser ?(which should be the same for the api call).
Best regards,
Ad
-
@AdSchellevis
I see that the address is missing, yet it was set in my request.
My point is: for the exact same post request, it works when there is already an address in the alias list, but it doesn't when the list is empty.
The UI use a different command: http://192.168.0.1/api/firewall/alias/setItem/<alias_uuid>, and define the whole alias list in the same request (with {"alias:{..., "content":"<address>"}}).
Best
-
/ui/firewall/alias_util/ (https://forum.opnsense.org/ui/firewall/alias_util/) see top center.
-
Sorry I didn't know there was another alias UI.
But I get the exact same error in this GUI.
Also, if I revert the alias type from Network to Host, I got two errors (here with {address:"192.168.0.50}):
{
"errorMessage": "[OPNsense\\Firewall\\Alias:aliases.alias.2684d3af-6af4-4248-934f-6035a668489a.content] Entry \"\" is not a valid hostname or IP address.\n[OPNsense\\Firewall\\Alias:aliases.alias.2684d3af-6af4-4248-934f-6035a668489a.content] Entry \"192.168.0.50/32\" is not a valid hostname or IP address.\n",
"errorTitle": "An API exception occured"
}
Also, if the address was missing in my request, the response should be {status: failed}
-
I noticed this same behavior too. I even verified that it happens even if the alias is referenced in a firewall rule or not referenced in a firewall rule as I thought that could be it, but it happens either way. Basically if the host type alias is empty then there is an error on API posting an address. I tried a few variations such as 1.0.32.1/32 (returned an error about invalid address like the alias was empty) and 1.0.32.1\32 (returned "not an address").
This is the error if the alias is empty:
{"errorMessage":"[OPNsense\\Firewall\\Alias:aliases.alias.c87dab5e-d37b-4bb0-9f01-ec950f0891b7.content] Entry \"\" is not a valid hostname or IP address.\n","errorTitle":"An API exception occured"}
I tried curl and PostMan, same in both. Adding one address in manually through the UI lets the API work immediately so its not that big a deal (in my opinion).
This is on 18.7.9 and I didnt do the patch referenced a few posts up.
-
It's actually pretty simple, if it works in the browser, the same call will also work using the api.
A simple example using python, looks like this:
import requests
key='my_key'
secret='my_secret'
endpoint = "https://[host]/api/firewall/alias_util/add/bogons"
r = requests.post(url=endpoint, auth=(key, secret), json={'address': '0.0.0.0/8'})
print (r.status_code, r.text)
Which will add 0.0.0.0/8 to the alias named bogons.
-
It's actually pretty simple, if it works in the browser, the same call will also work using the api.
A simple example using python, looks like this:
import requests
key='my_key'
secret='my_secret'
endpoint = "https://[host]/api/firewall/alias_util/add/bogons"
r = requests.post(url=endpoint, auth=(key, secret), json={'address': '0.0.0.0/8'})
print (r.status_code, r.text)
Which will add 0.0.0.0/8 to the alias named bogons.
Yes, but the current UI doesn't use the same endpoint. The default UI, /ui/firewall/alias use a different endpoint.
And I can confirm your code doesn't works if the the alias is empty when you try to add a new address.
I tried curl and PostMan, same in both. Adding one address in manually through the UI lets the API work immediately so its not that big a deal (in my opinion).
You're right, but you need to use the correct UI: /ui/firewall/alias and not /api/firewall/alias_util
-
... I wrote the code in OPNsense, there is a chance I know what it's for ;)
Both endpoints serve a different purpose, the util endpoint is effectuated immediately, the other one only changes the configuration content and needs a different payload (just look in the user interface how it should look, like suggested earlier).
-
Just wondering if you have this sorted out. I too have found these issues.
Also if I create an alias with api/firewall/alias/addItem it works fine. BUT, the flush command does NOT work in API or in the Browser GUI. It gets the {"status":"done"} result from API, but in the GUI when you edit the alias, the IP address is still there. If you go to Diag, pfTables, the alias entries do NOT appear. After a reboot (just to make sure), the Aliases reappear in pfTables and are still in the alias itself. (p.s. I just updated to 19.1.b_492-amd64, as the features mention "firewall alias API conversion" more complete than 18.7) so I think I can say the Flush function doesnt seem to work.
My use case is to (via API)
1. Flush the Alias, and
2. re populate with another (updated) list.
Can/Is there a way to addItem address/s in an array, or alias_util/add use an address array. I tried a few permeations but get a "cant convert array to string" (or something to that effect) error.
-
Just realised, looking around, my Use Case is perfectly covered by Url Tables (IP).
-
... I wrote the code in OPNsense, there is a chance I know what it's for ;)
Both endpoints serve a different purpose, the util endpoint is effectuated immediately, the other one only changes the configuration content and needs a different payload (just look in the user interface how it should look, like suggested earlier).
I think I understand now how it works.
Don't get me wrong: I'm very grateful for all the work you and OPNSense team have done so far and all I want is to help as I can.
But I can't understand why you won't accept there is a BUG here: there is an endpoint which doesn't work in specific conditions... period. This is the basic definition of a BUG. And the fact that the error message is wrong too doesn't help either.
So I'll summarize the bug (maybe you misunderstood or I wasn't clear enough):
- I have an empty alias (an alias with no items)
- I try to add a new item to the alias by using alias_util endpoint (from both API and UI)
- I send a request with the address item (API), or I fill the form field with a valid IP address (UI).
- From UI and API, I got an error saying: Entry "" is not a valid hostname or IP address (note that the address disappeared during process).
- If I do the same with an alias containing at least 1 item, the exact same steps works.
The only reason I will agree that this is not a bug would be if the alias_util endpoint is going to disappear in the next releases and is only there for testing purposes.
P.S: Happy New Year!
-
@camdarley Happy new year to you too!
This topic bounced between different endpoints with different use-cases, it's not that we don't want to admit a bug when it's there... but clear bug reports, preferably using the steps in the user interface (in this case /ui/firewall/alias_util/) help us to replicate the issue.
My only point was (and is), if it's not happing in the UI, it's likely not related to OPNsense.
Eventually I was able to replicate your issue using an empty alias, resulting in the following patch.
https://github.com/opnsense/core/commit/b69210f118b1c8b2900709190c3f4d688f0e7b60
We normally try to make sure that endpoints are used in the user interface somewhere, so we don't have to worry about all the different languages and tools available to test them.
Best regards,
Ad