Messages

1

19.1 Legacy Series / Re: Network performance and latency issues with haproxy

« on: April 30, 2019, 10:47:30 pm »

Here it is. redacted_http_test is the no-ssl backend I've been testing with.

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    # NOTE: Could be a security issue, but required for some feature.
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket level admin
    nbproc                      1
    nbthread                    1
    tune.ssl.default-dh-param   1024
    spread-checks               0
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0

defaults
    log     global
    option redispatch -1
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    # WARNING: pass through options below this line
    mode http
    option httplog

# autogenerated entries for ACLs

# autogenerated entries for config in backends/frontends

# autogenerated entries for stats


# Frontend: redacted_ssl ()
frontend redacted_ssl
    bind *:443 name *:443 ssl  crt-list /tmp/haproxy/ssl/5cc4fcfb4d50d1.96664841.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: host_redacted
    acl acl_5cc4fe5883d959.03367346 hdr_end(host) -i redacted

    # ACTION: map_redacted
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/5cc509feaa78f3.47022982.txt)] if acl_5cc4fe5883d959.03367346

# Frontend: letsencrypt ()
frontend letsencrypt
    bind *:80 name *:80
    mode http
    option http-keep-alive
    default_backend acme_challenge_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: redacted_http_test ()
frontend redacted_http_test
    bind *:8080 name *:8080
    mode http
    option http-keep-alive
    default_backend server1_apache
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: host_redacted
    acl acl_5cc4fe5883d959.03367346 hdr_end(host) -i redacted

    # ACTION: map_redacted
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/5cc509feaa78f3.47022982.txt)] if acl_5cc4fe5883d959.03367346

# Backend: acme_challenge_backend (Added by Let's Encrypt plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse never
    server acme_challenge_host 127.0.0.1:43580

# Backend: server1_apache ()
backend server1_apache
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse never
    server server1_apache backend.lan:80

# Backend: backend_grafana ()
backend backend_grafana
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse never
    server backend_grafana grafana.lan:80

2

19.1 Legacy Series / Network performance and latency issues with haproxy

« on: April 28, 2019, 05:44:21 pm »

First, I'm quite impressed by the HAProxy and Let's Encrypt plugins, and how they work together.

I'm running opnsense on an APU2 board, and I'm noticing:

• Slow transfer rates with high CPU usage
• Latency spikes for unrelated network traffic when HAProxy is under load

For example, the backend web server (a basic Apache setup) can saturate the 1Gb network connection with no issues. If I do a straight HTTP proxy through HAProxy in opnsense (no SSL offloading), performance caps out at about 200Mbit. I expected closer to 300 or 400 Mbits. During a transfer, the HAProxy process is using close to 100% of a CPU core (the board has 4 cores) with 50% system CPU usage. If I ping a host during the speed test from opnsense, latency goes up significantly for all connections going through the firewall.

So far, I noticed that at http://www.haproxy.org it's mentioned that pf causes quite a performance hit compared to Linux, but it's not clear how recent that note is.

Any ideas on how to solve the latency issues? I can live with slower transfer performance, but lag spikes will be a deal breaker for me.

3

18.7 Legacy Series / Stale WAN IP used for SIP NAT mapping (Dynamic state reset)

« on: November 23, 2018, 07:27:30 pm »

OPNSense is connected via DSL / pppoe for it's WAN connection. I have an OBiHAI SIP bridge for VoIP access. If the WAN IP changes, the old NAT mapping are still used, causing packets to be sent with the wrong source IP address. This breaks WAN connectivity until the states are killed.

• In the firewall states dump, I filter on port 5060 to see the inbound and outbound mappings.
• Note your current WAN IP, and click "reload" at the WAN interface in the overview to force a new connection.
• After the IP has renewed, reload the states dump and note the outbound IP address is the old IP address and not the new one.
• Killing the states restores WAN connectivity to the SIP bridge.

I've verified the wrong source IP is being sent from a packet capture of the pppoe interface. What's surprising to me is that nothing else other than this one mapping appears to be affected by this.

In the firewall advanced options, I found "Dynamic state reset" which was not enabled. Turning that on fixed the stale mappings. Is there any reason why that option shouldn't be on by default?

This could be related to switching ISPs from one that used DHCP to DSL and PPPoE. Is this a setting normally set during the setup wizard, which would be missed if you manually changed WAN settings after the initial install?

4

17.7 Legacy Series / Re: Cannot ping ISPs DNS servers since upgrade from 17.1.11 to 17.7

« on: August 08, 2017, 04:28:26 pm »

The patch didn't do anything, regardless of the override setting. I did have that enabled.

However, this pointed out a better workaround for now - if I statically set DNS, the bad routes aren't added.

5

17.7 Legacy Series / Re: Cannot ping ISPs DNS servers since upgrade from 17.1.11 to 17.7

« on: August 07, 2017, 05:55:38 pm »

I've got exactly the same problem since upgrading. Deleting the two routes for each DNS server fixes the problem, and renewing the WAN IP brings them back.

6

17.1 Legacy Series / Re: How is netflow data supposed to be rotated?

« on: June 27, 2017, 02:32:56 am »

One more bit - I just noticed that the UI isn't showing any data for the past 2 weeks. However, disk use has been steadily increasing in that time.

7

17.1 Legacy Series / How is netflow data supposed to be rotated?

« on: June 27, 2017, 02:26:35 am »

I re-enabled netflow about a month ago, and ever since it's been slowly growing in disk use. This is on a home connection, so it's not /that/ much traffic to log. flowd and flowd_aggregate are both running:

root@blackbox:/var # service flowd status
flowd is running as pid 36517 36532.
root@blackbox:/var # service flowd_aggregate status
flowd_aggregate is running as pid 42215.
root@blackbox:/var # du -sh log/flowd* netflow
3.6G   log/flowd.log
 11M   log/flowd.log.000001
 11M   log/flowd.log.000002
 11M   log/flowd.log.000003
 11M   log/flowd.log.000004
 12M   log/flowd.log.000005
 11M   log/flowd.log.000006
 11M   log/flowd.log.000007
 11M   log/flowd.log.000008
 11M   log/flowd.log.000009
 11M   log/flowd.log.000010
5.3G   netflow

I'm going to have to reset the data manually as I'm almost out of disk space. My understanding is simply running flowd_aggregate should be enough here. Is there anything else to check?

8

17.1 Legacy Series / Re: collectd network plugin is missing encryption support

« on: June 09, 2017, 11:10:38 pm »

It did! Thanks.

9

17.1 Legacy Series / Re: collectd network plugin is missing encryption support

« on: May 24, 2017, 03:15:30 pm »

Of course, the above ldd paste was on the colletctd binary, not network.so

libgcrypt is in the network plugin, using the default build options:

Code: [Select]

root@blackbox:~ # ldd /usr/local/lib/collectd/network.so
/usr/local/lib/collectd/network.so:
libgcrypt.so.20 => /usr/local/lib/libgcrypt.so.20 (0x2e31a0b000)
libgpg-error.so.0 => /usr/local/lib/libgpg-error.so.0 (0x2e31d1d000)
libc.so.7 => /lib/libc.so.7 (0x2e3102c000)


10

17.1 Legacy Series / Re: Are backups of "volatile" files required on reboots for non-nano installs?

« on: May 24, 2017, 03:09:45 pm »

Well, at one point, the disk (16GB) actually filled up completely with netflow data and I had to manually clear it out. I haven't replicated that. /var doesn't look to be RAM mounted, which is why I don't see why there needs to be a backup and restore process at all.

11

17.1 Legacy Series / Re: collectd network plugin is missing encryption support

« on: May 22, 2017, 02:24:38 pm »

Here's ldd after install:

Code: [Select]

root@blackbox:/usr/ports/net-mgmt/collectd5 # ldd /usr/local/sbin/collectd������ /usr/local/sbin/collectd:
        libm.so.5 => /lib/libm.so.5 (0x3442ba34000)
        libthr.so.3 => /lib/libthr.so.3 (0x3442bc5f000)
        libstatgrab.so.10 => /usr/local/lib/libstatgrab.so.10 (0x3442be87000)
        libdevstat.so.7 => /lib/libdevstat.so.7 (0x3442c099000)
        libltdl.so.7 => /usr/local/lib/libltdl.so.7 (0x3442c29f000)
        libc.so.7 => /lib/libc.so.7 (0x3442af14000)
        libkvm.so.7 => /lib/libkvm.so.7 (0x3442c4a8000)
        libelf.so.2 => /lib/libelf.so.2 (0x3442c6b7000)
You're right, there is no gcrypt support (nor shown in pkg info as well). However, encryption support is working fine. Perhaps there's multiple libraries it can use to support it, but I'm not finding anything obvious. I'd probably lean towards enabling gcrypt just to match what previous builds did.

I have a cron job for a nightly firmware upgrade. It looks like collectd has been replaced by the binary version on me from the repos. I thought I locked the package after install, but perhaps I didn't. Is there anything else to do when installing ports to keep them in place?

12

17.1 Legacy Series / Are backups of "volatile" files required on reboots for non-nano installs?

« on: May 21, 2017, 02:35:43 am »

I have OPNsense installed on a 16GB SSD. I noticed that reboots were taking a long time. It turns out the culprit was the netflow database; gzip'ing was taking 5-10 minutes, blocking a reboot. This commit illustrates the code fairly well. Any reason I can't just comment them out?

13

17.1 Legacy Series / Re: collectd network plugin is missing encryption support

« on: May 18, 2017, 10:45:50 pm »

Turns out that running the build from 16.7 didn't throw any errors, but did send wrong data in the networking plugin. Following https://forum.opnsense.org/index.php?topic=2004 I installed collectd from ports and crypt support is working fine.

14

17.1 Legacy Series / Re: dhclient exits on WAN IP renewal

« on: May 11, 2017, 08:15:34 pm »

It looks like this is still an issue on 17.1.6. Is there an upstream issue I can track?

15

17.1 Legacy Series / [SOLVED] collectd network plugin is missing encryption support

« on: May 10, 2017, 07:40:51 pm »

collectd has been broken for the past week, as the network plugin is no longer linked to libgcrypt and libgpg-error.

Code: [Select]

May 10 13:22:32 blackbox collectd[32863]: network plugin: Option `SecurityLevel' is not allowed here.
May 10 13:22:32 blackbox collectd[32863]: network plugin: Option `Username' is not allowed here.
May 10 13:22:32 blackbox collectd[32863]: network plugin: Option `Password' is not allowed here.
I manually installed collectd 5.7.0 from 16.7 as I couldn't find prior builds for 17.1. That version has:

Code: [Select]

root@blackbox:/tmp # ldd /usr/local/lib/collectd/network.so
/usr/local/lib/collectd/network.so:
libgcrypt.so.20 => /usr/local/lib/libgcrypt.so.20 (0x3482980b000)
libgpg-error.so.0 => /usr/local/lib/libgpg-error.so.0 (0x34829b1d000)
libc.so.7 => /lib/libc.so.7 (0x34828dfa000)
And in the latest 5.7.1 release:

Code: [Select]

root@blackbox:/usr/local/etc # ldd /usr/local/lib/collectd/network.so
/usr/local/lib/collectd/network.so:
libc.so.7 => /lib/libc.so.7 (0x205546c4000)
For now, I've locked the older package, though if there's prior builds actually against 17.1 I can install that would be better. Or, is there a way to simply blacklist the specific release of collectd, so when a new (presumably fixed) version is out it will be upgraded to automatically?