[SOLVED] Limit to Number of Networks in an Alias?

[TheLatestWire]

Is there a limit to the number of networks that can be in a single ALIAS?  I have a very long list of CIDR networks in a single ALIAS that I'm trying to add to but after clicking + and entering a new one, followed by "save", they don't appear on the page when it reloads.

[franco]

Several limits depending on various subsystems especially PHP and main memory itself, maybe even upload limits or script timeouts. None of those are there for limiting, but they will put an end to large data sets in weird forms.

How much entries are we talking about? From where do you add them, the import page?

[TheLatestWire]

I didn't import them, which might have been my problem.  I created a new ALIAS and then pasted all 2176 CIDR address into a single field on that page.  It *seemed* to work and take them all but it turned out to be missing about 40 entries.

I've created a new ALIAS and used the "import" feature this time and it successfully took and lists all 2176 entries.

Am I reaching the upper limits of the factors that might limit the number of entries in a single ALIAS?

Thank you for the help.

[franco]

Hrm, okay. When using the import, each Network has its own field, but pasting all into one on the edit page seems to work as comma-separated lists. That's probably an artifact and the truncation is more or less expected given that it should only handle one network per line. I have some larger works for aliases, I'll add a note in this ticket:

https://github.com/opnsense/core/issues/443

Ok to mark this as [SOLVED]?

[franco]

Oh, thank you, disregard that last question.

[TheLatestWire]

The list of CIDR networks that I successfully pasted into the import ALIAS page wasn't comma separated.  Each CIDR was on its own line, so it was a plain text file of 2165 lines.  So far so good.

Thanks again 

[TheLatestWire]

I think I spoke too soon.  I needed to add another couple CIDR networks to the new ALIAS that I created and after adding them and hitting save, the list didn't appear to have all the previously added CIDRs.  Maybe it's just a web server/php issue with showing them?

Where are the ALIAS config files stored?  Could I add new CIDRs to the text file and then restart a service from the shell console?

[franco]

You can review the differential config changes via System: Configuration: History to confirm what you suspect.

I think you are right, the import page will take all of those since it's a flat file, the edit page will render all values into a form, and on submit it will forget the ones that hit the POST limit...

This file likely has the limit values you seek:

https://github.com/opnsense/core/blob/master/src/etc/rc.php_ini_setup#L147-L158

Editing it at /usr/local/etc/rc.php_ini_setup and simply running it to apply the config defaults. Then check back in the GUI.

This may also be the culprit, limiting to 1000. Can you confirm the dropping of so many aliases in your history (over 1000)?

http://php.net/manual/de/info.configuration.php#ini.max-input-vars

[TheLatestWire]

I'm not sure if this is what you wanted me to test/verify so sorry in advance.

I edited /usr/local/etc/rc.php_ini_setup and not knowing what to do, I just doubled everything in the section you highlighted to look like this:
suhosin.get.max_array_depth = 10000
suhosin.get.max_array_index_length = 512
suhosin.get.max_vars = 10000
suhosin.get.max_value_length = 1000000
suhosin.post.max_array_depth = 10000
suhosin.post.max_array_index_length = 512
suhosin.post.max_vars = 10000
suhosin.post.max_value_length = 18000000
suhosin.request.max_array_depth = 10000
suhosin.request.max_array_index_length = 512
suhosin.request.max_vars = 10000
suhosin.request.max_value_length = 18000000

Then I ran that file:
/usr/local/etc/./rc.php_ini_setup

Then I opened the ALIAS with the very long list of CIDR networks.  It initially shows me the full list and I hit + to add a new one and then hit save but it's missing on the page after hitting save.  In fact the list is much shorter after hitting save than it is when I initially edit it.

I'm not sure which file has the "max_input_vars" variable as I couldn't find it in /usr/local/etc/rc.php_ini_setup.

Vielen Dank.

[franco]

Sorry for not being clear on this. For standard PHP variables, anywhere in this random area is fine:

https://github.com/opnsense/core/blob/master/src/etc/rc.php_ini_setup#L103-L110

Just add a line with

max_input_vars = 10000

I think the default is 1000 so that might be your trouble. Did you check with System: Configuration: History to see how many aliases get dropped? If you have +2000 after import and 1000 caps you must lose about half of your aliases with this limit in place. That should be pretty long config.xml difference.

Thanks for looking into this.

[TheLatestWire]

I added "max_input_vars = 10000" to /usr/local/etc/rc.php_ini_setup and then ran it with /usr/local/etc/./rc.php_ini_setup.

I then imported a new ALIAS with 2213 CIDR lines in it.  That worked and when I click on edit, it shows all 2213 lines.

I then tried to add one more CIDR  to it.  I clicked "+" and then entered it and hit save.  The resulting page listed only 497 CIDR entries, but if I then return to Aliases/All, and edit it, it still lists all the original 2213 lines, however my new CIDR addition is not there.

Thanks.

[franco]

Ok thanks for looking into this, I'll have to generate a bit of test data and see what's going on there.

[TheLatestWire]

I'm not sure if it's helpful or not, but I've attached the long list (2213 lines) of CIDR networks in case it helps with testing.

I also tried with "max_input_vars = 20000" but it didn't help and I got the same results.

Thanks again,
ObecalpEffect.

[franco]

Thanks, I think I got it... https://github.com/opnsense/core/commit/877b317f45f095

Patching your system:

# cd /usr/local/etc
# fetch https://raw.githubusercontent.com/opnsense/core/877b317f45f095/src/etc/rc.php_ini_setup
# chmod 755 rc.php_ini_setup
# cd /usr/local/www
# fetch https://raw.githubusercontent.com/opnsense/core/877b317f45f095/src/www/firewall_aliases_edit.php
# /usr/local/etc/rc.restart_webgui

After that it should work.

Ironic side note: Chrome struggles with rendering 2k entries as well.

[franco]

Small chmod amendment just in case.... use with care