Topics

1

18.1 Legacy Series / Timeout Server stroke with HAPROXY

« on: May 02, 2018, 12:55:04 pm »

Hi all,

I am running HAPROXY 2.6 on OPNsense 18.1.6-amd64. I have several web servers on an internal network that are reached through HAPROXY. This used to work fine, but since several weeks I get time outs on .js and or .css files. The files on which the time outs occur keep changing. Also, I don't know exactly when the problems started because it is not happing all the time.

Now when I go to a website hosted on my web servers sometimes it takes up to 30-35 seconds for the site to load. With the dev tools or websites likes pingdom.com you can see a connection error on these .js and or .css files (again, it keeps changing).

In the HAPROXY log file you can see the files marked with sH. According to the documentation:

sH     The "timeout server" stroke before the server could return its
          response headers. This is the most common anomaly, indicating too
          long transactions, probably caused by server or database saturation.
          The immediate workaround consists in increasing the "timeout server"
          setting, but it is important to keep in mind that the user experience
          will suffer from these long response times. The only long term
          solution is to fix the application.

When I access the web sites from the internal network I can't reproduce the problem. Also from the OPNsense console I can successfully curl the failed files over and over again without the problem occurring, yet when I try coming from the internet from any kind of machine (windows, Linux, iOS - Firefox, Chrome, Safari, Edge/IE) the problem occurs 1 out of 3 tries easily.

I don't fully understand how to read the documentation above. Saturation in the database seems unlikely since the files don't need database access and aren't called from a database entry. The webserver seems fine and the problem doesn't occur accessing the sites internally. The long term solution is to fix the application? What application?

Also, sometimes when this problem occurs, yet again, not al of the time, the haproxy service on the OPNsense machine takes 100% CPU for the same amount of time it takes for the site to load.

These are the only CPU spikes on a machine that has nothing to do all day. The sites have a very low visitor rate.

Hardware:
lscpu output:
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  2
Core(s) per socket:  4
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               94
Model name:          Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
Stepping:            3
CPU MHz:             3033.631
CPU max MHz:         4000,0000
CPU min MHz:         800,0000
BogoMIPS:            6816.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            8192K
NUMA node0 CPU(s):   0-7
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves ibpb ibrs stibp dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp

RAM: 32GB
Hard Drive: 2 x 4TB 6Gb/s 7200RPM


OPNsense:
Virtual
1 CPU (Skylake)
2 GiB RAM
CPU utilization: stable between 10-15%
Memory usage: stable at 17%
Disk usage: stable at 7%
Plugings: haproxy, acme-client (letsencrypt)
Number of websites configured in HAPROXY: 12

Web servers (3):
Virtual
OpenSUSE Tumbleweed
Apache 2.4
1 CPU
1 - 4 GiB Ram
CPU utilization: lower than 1%

Database server (1):
Virtual
OpenSUSE Tumbleweed
MariaDB
1 CPU
2 GiB RAM
CPU utilization: lower than 1%

Websites software:
Wordpress
iTop
Moodle
ownCloud (with (usually) 2 clients that poll the server every few seconds over HTTPS)
Plain HTML

The problem occurs on all websites except the plain HTML which has no js or css files.

So far I was just using this for personal servers, but I am planning on renting server space once I have everything fully automated on the back-end. And then this problem came along.

I have no idea where to look now for a solution. I could add another CPU to OPNsense but nothing really indicates this is the problem.

When I shutdown all servers but one webserver, the database server and OPNsense the problem remains. So definitely no overcommitting.

I'd like to understand the problem before I add more hardware servers.

Any ideas anyone?

2

17.7 Legacy Series / HAProxy redirect root to www subdomain

« on: November 27, 2017, 01:26:05 pm »

I have several webservers behind HAProxy. I use the LetsEncrypt pluging that also redirects poort 80 to 443. I tried to redirect a domain.com to www.domain.com but I can't seem to get it working. Perhaps I am even thinking in the wrong direction.

What would be a decent way to configure this? I had my front-end also listen to domain.com as well as to www.domain.com and then an acl and action that put www. in front of the domain. But so far no go.

Also, would you need an ACL and action per domain or it is possible to do it all with one?

3

17.7 Legacy Series / HAProxy: Client Certificates

« on: November 21, 2017, 09:57:38 pm »

I configured 3 apache servers with several virtual hosts. HAProxy makes it all possible, with SSL offloading.

Now I want a couple of management sites to be protected with a client certificate. How do I this? I have no idea where to start. I found some tuts for HAProxy, but what I read there doesn't match the HAProxy plugin in OPNsense.

Can anyone help?

4

17.7 Legacy Series / [SOLVED] Webserver behind HAProxy not fully functional

« on: November 20, 2017, 02:55:30 pm »

Hi,

I have set up the HAProxy plug-in with SSL offloading for several Apache2 web-servers in the back-end. All Apache servers host several virtual hosts.

Everything seems to work just fine, except for some minor, yet important stuff.

First I installed Moodle on one of the back-end servers, and after reading the Moodle forum and finding a settings that makes Moodle reverse proxy aware everything worked just fine.

Then I tried to install iTop, and this setup shows two problems when I try to install it from the other side of the reverse proxy:

1. Buttons on the sites do not respond when clicked. The setup shows advanced info when the button is clicked. This doesn't work unless you are on a workstation that is in the back-end.
2. Creating the config file. This entire step is skipped and it just gives you an error when it tries to load the now configured site. The error is that it can't find the config file, since it just doesn't create it al all. Again, when I install it from a workstation on the back-end, everything works.

Perhaps something with static/dynamic pages? So far I can't find something in the logs that look suspicious.

In the HAProxy I enabled the X-forwared-For header option.

Is there anything else I can do? I tried finding Apache settings but it seems that a back-end server doesn't need any extra configuration except for logging the client IP instead of the Proxy IP.

I am running OPNsense 1.7.7 with HAProxy 1.17

5

17.7 Legacy Series / FW rule not working with Single host or network

« on: September 18, 2017, 03:02:09 pm »

Hi,

I am fairly new to OPNsense, perhaps I missed other settings.

With HAProxy I made a Frontend that listens to a certain domain. I also made a Back-end with, atm, one web server in it. The idea is to add more web servers when needed.

Then I made a firewall rule in the External_Network. The rule simply says to accept port 80 traffic from any external source going to port 80 on any internal source.

Now when I punch in the domain in my web browser it shows me the website on the internal web server.

Now I want to tighten the rule, so I change it from any external source to a internal single host or network and I enter the IP address of the web server.

Somehow, the rule doesn't work any more now.

I am not sure what extra information to include in this post. What am I missing here?