nginx 1.7: banning, even if Learning Mode on.

[mrpsycho]

latest opnsense with latest nginx module.

i turned  Learning Mode for every location ON, and still see errors: Unsucessful status code You got banned permanently from this server.

how to turn this blocking rule? or to setup for bypass all traffic?

[mrpsycho]

also, i see 192.168.2.225 in banned list....
then im trying to look for log:

root@OPNsense:~ # cat /var/log/nginx/* | grep 2.225
root@OPNsense:~ #


and there is nothing!

how i can prevent it from being banned?

[mrpsycho]

after several tests i found problem - this is "User Agent"

and this is actually sucks, that it blocks on nginx by incorrect user agent.

because this is first what spammers change.... but it blocks relevant traffic from scripts and other dev stuff.

[fabian]

and this is actually sucks, that it blocks on nginx by incorrect user agent.

https://tools.ietf.org/html/rfc2616#section-14.43
The user agent may contain some information to identify your application (in reality it almost always does).

because this is first what spammers change.... but it blocks relevant traffic from scripts and other dev stuff.

No, believe me, most of the scanners in my log don't. They don't even understand the HTTPS redirect.In my opinion, every developer working with HTTP should know how to set a proper user agent. If not, the feature can still be disabled by a checkbox.
For learning mode: This controls the WAF (naxsi), not the bot defence.

[mrpsycho]

Thank you, @fabian.

it make sense... but where to find this checkbox?

on server config i cant see it...


and yes, in mobile development customer asks to set special User Agent... like: okhttp/2.1.2.2

[fabian]

you need to toggle the advanced settings to make it visible

[mrpsycho]

great! thanks a lot!

[TeKK]

I see that the included protection for bots blocks the following User-Agents:

Code Select Expand

Python-urllib|Nmap|python-requests|libwww-perl|MJ12bot|Jorgee|fasthttp|libwww|Telesphoreo|A6-Indexer|ltx71|okhttp|ZmEu|sqlmap|LMAO/2.0|ltx71|zgrab|Ronin/2.0|Hakai/2.0

Is it possible to edit the list of default bad bots from GUI or would I have to edit from CLI every time the nginx config is changed? I have an application that uses the User-Agent "OKhttp" and would like to just allow this user-agent rather than disabling the entire protection from GUI

[fabian]

no, it is hardcoded in the config. You can use the plugin interface to create  a similar config and disable the internal check.


https://github.com/opnsense/plugins/blob/master/www/nginx/src/opnsense/service/templates/OPNsense/Nginx/http.conf#L241

The UUID is the id you get in the config.xml and in the api for the HTTP server.

[TeKK]

no, it is hardcoded in the config. You can use the plugin interface to create  a similar config and disable the internal check.


https://github.com/opnsense/plugins/blob/master/www/nginx/src/opnsense/service/templates/OPNsense/Nginx/http.conf#L241

The UUID is the id you get in the config.xml and in the api for the HTTP server.

Thank you. This worked perfectly. Took me a minute to realize i had to create the UUID_pre/ folder.

Regards

[fabian]

no problem - it is not well documented since this is an internal plugin hook for plugins, that want to use the nginx plugin for their purposes. There is also a UUID_post which is after all internal rules and there is a special directory for custom vhosts (also for streams).