Performance problem

[maekar]

Hi,

I work in a school, and we have ~700 iPad and ~300 computers in the network. The Firewall specs are:

Intel Xeon E5-2620 v2
16 GB RAM
Dual Nic Intel Gigabit IT-350-T2
Samsung SSD 750 EVO 250GB

The Internet line is a 300/300MB FTTH and the net schema is:

ISP router ------ OPNSense Firewall ------- LAN Network

Since a while, I've having a lot of performance troubles in specific moments of the day. 95% of time everything works great but sometimes is like the network is overloaded and the web navigation become really slow. For example, everday at 9:15 (when the classes start and everybody begin to work) it happens. Sometimes for a few minutes, sometimes for an entire hour, and is impossible to work in that condition.

As I said, I've been months with this troubles and I tested some things I thought were the cause of the problem. I finally discard this (maybe I'm wrong):

1) Is not a proxy performance problem. I have it in transparanent mode with NO SSL. If I disable it, the problem continues.
2) Is not a DNS problem. I tried with DNS Resolver, DNS Forwarder, with default config, the problem continues.
3) Is not an ISP problem. When the network is "overloaded", if I conect my computer directly to the ISP router, the navigation speed is great while in the LAN network is really crappy.
4) Is not coming from a particular area of the school. I try to unplug every single cable in the main switch and anything change. (If I unplug all of them at the same time, the navigation in my computer is great).
5) Is not a ISP throughput problem. The FTTH line is 300/300. When the network is "overloaded", the traffic graph usually shows no more 50-100MB. In other moments, when the network works fine, I can reach 300MB without any issues.

Last thing I did is replace in the firewall the HDD by an SSD, and install the last version of OPNSense from scratch, and nothing change.

I'm very lost and I don't know how to debug the problem. I don't know what to look and how to do it to find the cause of the problem. How can I discard that is a Firewall related problem?

Thanks in advance.

P.S: Sorry for my english, I need the help of Google Translator many times.

[bartjsmit]

You need to monitor the firewall during peak hours to look for performance bottlenecks. The three most likely areas are network I/O, memory and CPU (roughly in that order). Use top, iostat and vmstat to compare baseline performance to peak performance and you should see which component is a likely culprit. You can also run traffic captures from an internal client to see which part of the connection has latency (DNS lookups, authentication, presentation, etc.)

Bart...