OPNsense Forum

English Forums => General Discussion => Topic started by: ObecalpEffect on March 21, 2016, 05:16:42 am

Title: [SOLVED] Limit to Number of Networks in an Alias?
Post by: ObecalpEffect on March 21, 2016, 05:16:42 am
Is there a limit to the number of networks that can be in a single ALIAS?  I have a very long list of CIDR networks in a single ALIAS that I'm trying to add to but after clicking + and entering a new one, followed by "save", they don't appear on the page when it reloads.
Title: Re: Limit to Number of Networks in an Alias?
Post by: franco on March 21, 2016, 10:07:29 am
Several limits depending on various subsystems especially PHP and main memory itself, maybe even upload limits or script timeouts. None of those are there for limiting, but they will put an end to large data sets in weird forms.

How much entries are we talking about? From where do you add them, the import page?
Title: Re: Limit to Number of Networks in an Alias?
Post by: ObecalpEffect on March 21, 2016, 03:15:48 pm
I didn't import them, which might have been my problem.  I created a new ALIAS and then pasted all 2176 CIDR address into a single field on that page.  It *seemed* to work and take them all but it turned out to be missing about 40 entries.

I've created a new ALIAS and used the "import" feature this time and it successfully took and lists all 2176 entries.

Am I reaching the upper limits of the factors that might limit the number of entries in a single ALIAS?

Thank you for the help.
Title: Re: Limit to Number of Networks in an Alias?
Post by: franco on March 21, 2016, 03:50:07 pm
Hrm, okay. When using the import, each Network has its own field, but pasting all into one on the edit page seems to work as comma-separated lists. That's probably an artifact and the truncation is more or less expected given that it should only handle one network per line. I have some larger works for aliases, I'll add a note in this ticket:

https://github.com/opnsense/core/issues/443

Ok to mark this as [SOLVED]? :)
Title: Re: [SOLVED] Limit to Number of Networks in an Alias?
Post by: franco on March 21, 2016, 03:50:37 pm
Oh, thank you, disregard that last question. :D
Title: Re: Limit to Number of Networks in an Alias?
Post by: ObecalpEffect on March 21, 2016, 04:40:47 pm
The list of CIDR networks that I successfully pasted into the import ALIAS page wasn't comma separated.  Each CIDR was on its own line, so it was a plain text file of 2165 lines.  So far so good.

Thanks again  :)
Title: Re: Limit to Number of Networks in an Alias?
Post by: ObecalpEffect on March 21, 2016, 08:16:59 pm
I think I spoke too soon.  I needed to add another couple CIDR networks to the new ALIAS that I created and after adding them and hitting save, the list didn't appear to have all the previously added CIDRs.  Maybe it's just a web server/php issue with showing them?

Where are the ALIAS config files stored?  Could I add new CIDRs to the text file and then restart a service from the shell console?
Title: Re: Limit to Number of Networks in an Alias?
Post by: franco on March 22, 2016, 07:35:10 am
You can review the differential config changes via System: Configuration: History to confirm what you suspect.

I think you are right, the import page will take all of those since it's a flat file, the edit page will render all values into a form, and on submit it will forget the ones that hit the POST limit...

This file likely has the limit values you seek:

https://github.com/opnsense/core/blob/master/src/etc/rc.php_ini_setup#L147-L158

Editing it at /usr/local/etc/rc.php_ini_setup and simply running it to apply the config defaults. Then check back in the GUI.

This may also be the culprit, limiting to 1000. Can you confirm the dropping of so many aliases in your history (over 1000)?

http://php.net/manual/de/info.configuration.php#ini.max-input-vars
Title: Re: Limit to Number of Networks in an Alias?
Post by: ObecalpEffect on March 22, 2016, 02:33:06 pm
I'm not sure if this is what you wanted me to test/verify so sorry in advance.

I edited /usr/local/etc/rc.php_ini_setup and not knowing what to do, I just doubled everything in the section you highlighted to look like this:
suhosin.get.max_array_depth = 10000
suhosin.get.max_array_index_length = 512
suhosin.get.max_vars = 10000
suhosin.get.max_value_length = 1000000
suhosin.post.max_array_depth = 10000
suhosin.post.max_array_index_length = 512
suhosin.post.max_vars = 10000
suhosin.post.max_value_length = 18000000
suhosin.request.max_array_depth = 10000
suhosin.request.max_array_index_length = 512
suhosin.request.max_vars = 10000
suhosin.request.max_value_length = 18000000

Then I ran that file:
/usr/local/etc/./rc.php_ini_setup

Then I opened the ALIAS with the very long list of CIDR networks.  It initially shows me the full list and I hit + to add a new one and then hit save but it's missing on the page after hitting save.  In fact the list is much shorter after hitting save than it is when I initially edit it.

I'm not sure which file has the "max_input_vars" variable as I couldn't find it in /usr/local/etc/rc.php_ini_setup.

Vielen Dank.
Title: Re: Limit to Number of Networks in an Alias?
Post by: franco on March 22, 2016, 11:21:42 pm
Sorry for not being clear on this. For standard PHP variables, anywhere in this random area is fine:

https://github.com/opnsense/core/blob/master/src/etc/rc.php_ini_setup#L103-L110

Just add a line with

max_input_vars = 10000

I think the default is 1000 so that might be your trouble. Did you check with System: Configuration: History to see how many aliases get dropped? If you have +2000 after import and 1000 caps you must lose about half of your aliases with this limit in place. That should be pretty long config.xml difference.

Thanks for looking into this. :)
Title: Re: Limit to Number of Networks in an Alias?
Post by: ObecalpEffect on March 23, 2016, 03:27:58 pm
I added "max_input_vars = 10000" to /usr/local/etc/rc.php_ini_setup and then ran it with /usr/local/etc/./rc.php_ini_setup.

I then imported a new ALIAS with 2213 CIDR lines in it.  That worked and when I click on edit, it shows all 2213 lines.

I then tried to add one more CIDR  to it.  I clicked "+" and then entered it and hit save.  The resulting page listed only 497 CIDR entries, but if I then return to Aliases/All, and edit it, it still lists all the original 2213 lines, however my new CIDR addition is not there.

Thanks.
Title: Re: Limit to Number of Networks in an Alias?
Post by: franco on March 23, 2016, 04:21:41 pm
Ok thanks for looking into this, I'll have to generate a bit of test data and see what's going on there.
Title: Re: Limit to Number of Networks in an Alias?
Post by: ObecalpEffect on March 23, 2016, 04:47:13 pm
I'm not sure if it's helpful or not, but I've attached the long list (2213 lines) of CIDR networks in case it helps with testing.

I also tried with "max_input_vars = 20000" but it didn't help and I got the same results.

Thanks again,
ObecalpEffect.
Title: Re: Limit to Number of Networks in an Alias?
Post by: franco on March 23, 2016, 05:22:26 pm
Thanks, I think I got it... https://github.com/opnsense/core/commit/877b317f45f095

Patching your system:

# cd /usr/local/etc
# fetch https://raw.githubusercontent.com/opnsense/core/877b317f45f095/src/etc/rc.php_ini_setup
# chmod 755 rc.php_ini_setup
# cd /usr/local/www
# fetch https://raw.githubusercontent.com/opnsense/core/877b317f45f095/src/www/firewall_aliases_edit.php
# /usr/local/etc/rc.restart_webgui

After that it should work. :)

Ironic side note: Chrome struggles with rendering 2k entries as well.
Title: Re: Limit to Number of Networks in an Alias?
Post by: franco on March 23, 2016, 05:57:29 pm
Small chmod amendment just in case.... use with care
Title: [SOLVED] Re: Limit to Number of Networks in an Alias?
Post by: ObecalpEffect on March 23, 2016, 07:33:39 pm
Thanks franco.  Replacing those two files and restarting the webgui did the trick.   Now when I hit save after adding a new entry to an ALIAS, it takes me back to the full list of ALIASes.  I then checked the ALIAS I just added an entry to and it was still there.  I'll mark this as solved!

On a side note, and relating to another thread I posted today about not being able to fetch updates, I wasn't able to fetch those two files from my OPNsense box, they timed out.  I had to use wget from a linux box that is behind the OPNsense firewall.  I'm not sure why I would be able to download it from a box inside my network, but not from the OPNsense box itself.  Whatever it it, I'm guessing it is also why I can't fetch updates.  I can ping raw.githubusercontent.com from the OPNsense box.   :-\

Thanks again for all the help.
Title: Re: [SOLVED] Limit to Number of Networks in an Alias?
Post by: franco on March 23, 2016, 07:39:37 pm
Thank you for the persistence. Let's take the DNS problem to the other thread and solve it there. 8)