Topics

1

22.7 Production Series / [Solved] Is RSS available on 22.7 series?

« on: September 21, 2022, 06:18:15 pm »

I did some RSS testing when I was on 21.7.5 and there was no discernible impact so I left the tunables in when I upgraded to 22.7. I later saw some of the tunables labelled as unsupported so I removed from the UI. All good.

I am tracing a marked reduction in bitrate from WAN to LAN which made me look at tunables again and the status of RSS as part of the elimination process. Cables and similar are part of the process at the moment of course, and in the meantime, this question.

BEFORE:
I had OPN 21.7.5, OpenSSL, Hardware is APU4.
Upgraded BIOS beforehand, coreboot v 4.14.0.6 .
Network interfaces on this system are igb.

penguin@OPNsense:~ % sudo sysctl -a | grep -i 'isr.bindthreads\|isr.maxthreads\|inet.rss.enabled\|inet.rss.bits'
net.inet.rss.enabled: 1
net.inet.rss.bits: 2
net.isr.bindthreads: 1
net.isr.maxthreads: 4
penguin@OPNsense:~ % sudo netstat -Q
Configuration:
Setting                        Current        Limit
Thread count                         4            4
Default queue limit                256        10240
Dispatch policy                 direct          n/a
Threads bound to CPUs          enabled          n/a

Protocols:
Name   Proto QLimit Policy Dispatch Flags
ip         1   1000    cpu   hybrid   C--
igmp       2    256 source  default   ---
rtsock     3    256 source  default   ---
arp        4    256 source  default   ---
ether      5    256    cpu   direct   C--
ip6        6    256    cpu   hybrid   C--
ip_direct     9    256    cpu   hybrid   C--
ip6_direct    10    256    cpu   hybrid   C--


NOW:
Now on OPN 22.7.4, OpenSSL, same hardware.

root@OPNsense:/home/penguin # sysctl -a | grep -i 'isr.bindthreads\|isr.maxthreads\|inet.rss.enabled\|inet.rss.bits'
net.inet.rss.enabled: 0
net.inet.rss.bits: 2
net.isr.bindthreads: 0
net.isr.maxthreads: 1

root@OPNsense:/home/penguin # netstat -Q
Configuration:
Setting                        Current        Limit
Thread count                         1            1
Default queue limit                256        10240
Dispatch policy                 direct          n/a
Threads bound to CPUs         disabled          n/a

Protocols:
Name   Proto QLimit Policy Dispatch Flags
ip         1   1000   flow  default   ---
igmp       2    256 source  default   ---
rtsock     3    256 source  default   ---
arp        4    256 source  default   ---
ether      5    256 source   direct   ---
ip6        6   1000   flow  default   ---

So it looks like during the upgrade, RSS was disabled. Question 1: Is that expected?
Since I realised that today, I've re-added the tunables that are accepted "net.inet.rss.bits", "net.isr.maxthreads" and "net.inet.rss.enabled" and rebooted . I also had to but netstat -Q shows that threads aren't bound to CPU enabled.
Configuration:
Setting                        Current        Limit
Thread count                         4            4
Default queue limit                256        10240
Dispatch policy                 direct          n/a
Threads bound to CPUs         disabled          n/a

Protocols:
Name   Proto QLimit Policy Dispatch Flags
ip         1   1000    cpu   hybrid   C--
igmp       2    256 source  default   ---
rtsock     3    256 source  default   ---
arp        4    256 source  default   ---
ether      5    256    cpu   direct   C--
ip6        6   1000    cpu   hybrid   C--
ip_direct     9    256    cpu   hybrid   C--
ip6_direct    10    256    cpu   hybrid   C--

Question 2: Should I expect the setting to change ie. is wrong at present without threads bound to CPU?
Question 3: How do I make it so?

Thanks for the assistance.

2

Web Proxy Filtering and Caching / nginx process not stopping

« on: September 08, 2022, 02:48:05 pm »

Hello.
I'm on OPN ver OPNsense 22.7.4-amd64 with os-nginx (installed)   1.29_1
I had nginx setup and working but later I didn't needed runing so I stopped it until I need it again later. Mostly but not only using the UI.
Looking for something else, I noticed an nginx process seemed to be starting. I went to check and indeed the UI shows nginx disabled "Enable nginx" (unticked) and the top-right icon appears showing running. If I click on the square stop button, it flashes but stays running. Logs:

Code: [Select]

me@OPNsense:~ % sudo tail /var/log/nginx/error.log
2022/09/08 13:29:00 [notice] 18393#120692: signal process started
2022/09/08 13:30:00 [notice] 47431#100680: signal process started
2022/09/08 13:31:00 [notice] 74833#100689: signal process started
2022/09/08 13:32:00 [notice] 6906#100622: signal process startedSignal every minute to start.

Code: [Select]

me@OPNsense:~ % sudo nginx -T
Password:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Unticking enable, apply; followed by unticking and apply seems to stop it but needs doing again after a reboot.
Any ideas where I need to look for what might be restarting it?

3

Web Proxy Filtering and Caching / Where to place nginx customisations

« on: July 28, 2022, 06:05:37 pm »

Hi.
I'm trying to replicate a working setup from a standalone nginx installation into the OPN nginx plugin.
So far I can see traffic hitting nginx but I can't get the service to work. It's most likely that I need to add customisation to the configuration at least whilst diagnosing.
Setup is:
OPNsense 22.1.10-amd64
FreeBSD 13.0-STABLE
OpenSSL 1.1.1q 5 Jul 2022
os-nginx (installed)   1.28_1

My config:
cat /usr/local/etc/nginx/nginx.conf

Code: [Select]

load_module /usr/local/libexec/nginx/ngx_stream_module.so;
load_module /usr/local/libexec/nginx/ngx_http_naxsi_module.so;
load_module /usr/local/libexec/nginx/ngx_mail_module.so;
load_module /usr/local/libexec/nginx/ngx_http_brotli_filter_module.so;
load_module /usr/local/libexec/nginx/ngx_http_brotli_static_module.so;
load_module /usr/local/libexec/nginx/ngx_http_js_module.so;

user www staff;
worker_processes 1;

error_log  /var/log/nginx/error.log;

events {
    worker_connections  1024;
}

http {
include       mime.types;



log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';
log_format  main_ext  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" '
                      '"$host" sn="$server_name" '
                      'rt=$request_time '
                      'ua="$upstream_addr" us="$upstream_status" '
                      'ut="$upstream_response_time" ul="$upstream_response_length" '
                      'cs=$upstream_cache_status';
log_format  handshake   '"$http_user_agent" "$ssl_ciphers" "$ssl_curves"';
log_format  anonymized  ':: - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';

#tcp_nopush     on;
# https intercept detection
js_import /usr/local/opnsense/scripts/nginx/ngx_functions.js;
js_set $tls_intercepted ngx_functions.check_intercept;

# 200M should be big enough for file servers etc.
client_max_body_size 200M;
brotli_static on;
brotli on;
gzip_static on;
gzip on;
server_tokens off;
sendfile Off;
default_type  application/octet-stream;
keepalive_timeout 60;

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

# TODO add when core is ready for allowing nginx to serve the web interface
# include nginx_web.conf;




# UPSTREAM SERVERS


include opnsense_http_vhost_plugins/*.conf;

}
stream {
    # LOG FORMATS
    log_format main '$remote_addr [$time_local] '
                     '$protocol $status $bytes_sent $bytes_received '
                     '$session_time';
    log_format anonymized ':: [$time_local] '
                     '$protocol $status $bytes_sent $bytes_received '
                     '$session_time';

    # UPSTREAM SERVERS
    upstream upstream15ad4a8a20f14c19a47af261eaa5249e {
        server 192.168.5.1:53 weight=5;
    }
    upstream upstream7f19e5deb72b4c318458205c45343b52 {
        server 192.168.5.1:8053 weight=5;
    }

    # upstream maps


    include opnsense_stream_vhost_plugins/*.conf;

    # servers
    server {

        listen 127.0.0.1:8054 proxy_protocol;

        access_log  /var/log/nginx/stream_7e5528ea-bfe6-46f5-be19-3e967da85398.access.log main;
        error_log  /var/log/nginx/stream_7e5528ea-bfe6-46f5-be19-3e967da85398.error.log info;


        include 7e5528ea-bfe6-46f5-be19-3e967da85398_pre/*.conf;

        proxy_ssl off;
        proxy_pass upstream15ad4a8a20f14c19a47af261eaa5249e;
        proxy_protocol on;

        include 7e5528ea-bfe6-46f5-be19-3e967da85398_post/*.conf;

    }
}
# mail {
# }
I want to include for instance a njs module in my upstream "upstream15ad4a8a20f14c19a47af261eaa5249e" for example.
Similar a logging customisation for my stream block.

Where do I need to place them please?

p.s. the README in the opnsense_stream_vhost_plugins directory has this
"Place server {} in this directory - they are not affected by the enable / disable flag"
but I fail to understand if that's my hint.

4

General Discussion / Is njs support included in the nginx plugin?

« on: July 25, 2022, 12:27:19 am »

Hello. I did a forum search but I didn't find an answer.
Is njs supported in the nginx plugin?
I ask because I have a little server in my LAN running nginx on ubuntu and I needed njs . It was installed and all was good.
I have since had to shut the host down and migrated my VMs. Instead of setting a new VM I'd like to see if I can run nginx directly on OPN but it won't work for my use without njs. If I need to build nginx to include the njs support, I wouldn't do it on OPN.
Thanks for the advice.

5

Tutorials and FAQs / OPN as a PXE boot server

« on: October 02, 2021, 05:02:55 pm »

This mini tutorial is intended to setup a PXE boot server in OPNSense.
The clever parts I have used with permission are documented by Kraileth in his blog:
Links:
https://eerielinux.wordpress.com/2021/01/24/multi-os-pxe-booting-from-freebsd-12-introduction-pt-1/
https://eerielinux.wordpress.com/2021/01/30/multi-os-pxe-booting-from-freebsd-12-required-services-pt-2/
https://eerielinux.wordpress.com/2021/02/05/multi-os-pxe-booting-from-freebsd-12-pxe-menu-and-bsd-pt-3/
https://eerielinux.wordpress.com/2021/02/20/multi-os-pxe-booting-from-freebsd-12-linux-illumos-and-more-pt-4-2/
All I’ve done is adapted it to OPNSense and I made a couple of choices for my needs and infrastructure availability.

What we’ll have at the end of this tutorial is OPNSense acting as a PXE boot server that can be used for clients to network boot on a LAN interface. As an working example we’ll be serving a FreeBSD 12 boot option. I am using OPNsense 21.7.2_1-amd64. An important note is this is for BIOS boot, not UEFI.

The only additional dependency required is a plugin “os-tftp” found in the “default” mirror of OPNSense i.e. OPNSense package repository.

As a baseline I describe my setup. I don’t have VLANs and my flat network is in the 192.168.5.0/24 range. OPN has a lan IP of 192.168.5.1. I have a FreeBSD-based NAS with a reserved IP of 192.168.5.5 that has an available webserver running on port 8081. I will use this storage+internal webserver to serve the large isos. It is perfectly possible and simpler to host them in OPN but I have limited storage on my firewall appliance and I don’t want to run a webserver on it either.
I use DHCPv4 and not v6.
I use Unbound for name resolution.

Prerequisites:
A shell account that can elevate permissions with sudo on OPN.
A shell account that can elevate permissions with sudo on a local webserver.
Internet connection to download some necessary files. I use a separate computer and then transfer to the firewall, to avoid costly mistakes. Some will prefer to do it all on the firewall itself.
Some familiarity with basic shell commands and utilities like wget, nano, etc.

Now we start.
1. Install the tftp plugin in OPNSense
Go to System > Firmware > Plugins. Search for os-tftp. Click on the + sign at the end of the row to install it.
You will get a message “The root folder for transfering files is /usr/local/tftp.”
Once the WebUI refreshes you’ll have an entry in Services > TFTP > General where the service can be enabled or disabled.

On a shell connect to OPNSense and verify the path exists. For me it didn’t so I created it

Code: [Select]

$ sudo mkdir -p /usr/local/tftp
Then back on the WebUI I enabled the tftp service and entered my OPN LAN address 192.168.5.1 and save. Now we have a tftp server listening. To check:

Code: [Select]

$ sudo sockstat -4l
Password:
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS     
root          in.tftpd            48436 4       udp4   192.168.5.1:69                    *:*
I did root around the system to figure out how the plugin sets things up but I couldn’t find much. I was fumbling in the dark with this. I see it does not use inetd but I can’t yet find it’s config. I found an entry in etc/rc.d/ftpd but not much else. Nevertheless it is running and working fine for our purposes, and starts after reboot.
If someone finds where the control and config files are, let me know.

1.1 Create the pxe boot files being given by ftpd

Code: [Select]

sudo mkdir -p /usr/local/tftp/pxelinux.cfg
sudo nano /usr/local/tftp/pxelinux.cfg/default
Insert the following and save:

DEFAULT vesamenu.c32
PROMPT 0
MENU TITLE PXE Boot Menu (Main)

LABEL bsd-oses
   MENU LABEL BSD Operating Systems
   KERNEL vesamenu.c32
   APPEND pxelinux.cfg/bsd


Now for a test, from a linux or FreeBSD machine on the LAN do:

Code: [Select]

$ cd /tmp
$ tftp 192.168.5.1
tftp> get pxelinux.cfg/default
Received 190 bytes in 0.1 seconds
tftp> quit
$ rm default
That confirms our tftp daemon on OPN is serving the files correctly. I was expecting to need to open the port 69 but it seems the default pass rule from the LAN takes care of it.
You will need to adjust the firewall rules if you don’t have that permissive rule in place.

Now we need to create the referenced file “bsd”:

Code: [Select]

$ sudo nano /usr/local/tftp/pxelinux.cfg/bsdinsert this:

MENU TITLE PXE Boot Menu (BSD)

LABEL main-menu
   MENU LABEL Main Menu
   KERNEL vesamenu.c32
   APPEND pxelinux.cfg/default

This step 1.1 has created a default file and a bsd file, both inside a subdirectory under our tftp root.
Now we’re ready to get those files they reference.

2. Get the PXElinux files.
Ordinarily I do this on a separate machine and then transfer the files but on this ocassion I’ll do it directly on OPN.

Code: [Select]

$ cd /tmp
$ sudo pkg fetch -y syslinux
$ sudo mkdir -p /tmp/syslinux
$ sudo tar -C /tmp/syslinux -xvf /var/cache/pkg/syslinux-6.03.pkg
Now we copy the required files:

Code: [Select]

$ sudo cp /tmp/syslinux/usr/local/share/syslinux/bios/core/lpxelinux.0 /usr/local/tftp/pxelinux.0
$ sudo cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/elflink/ldlinux/ldlinux.c32 /usr/local/tftp/
$ sudo cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/menu/vesamenu.c32 /usr/local/tftp/
$ sudo cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/lib/libcom32.c32 /usr/local/tftp/
$ sudo cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/libutil/libutil.c32 /usr/local/tftp/
$ sudo cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/modules/pxechn.c32 /usr/local/tftp/
$ sudo cp /tmp/syslinux/usr/local/share/syslinux/bios/memdisk/memdisk /usr/local/tftp/
$ sudo rm -r /tmp/syslinux
At this point we have on our tftp server the files required for menus and the files required to get the clients to boot to a prompt for an installation. However we don’t have an installation media to offer yet.
For our example we are now going to get a FreeBSD 12.2 distribution iso.
Kraileth makes a great explanation of why for FreeBSD we need an mfsBSD https://mfsbsd.vx.sk/ image that can network boot.

3. Getting the image media available on our webserver.
TFTP is not meant to be used to transfer large files like distro isos. It is too slow, transferring files by limited block sizes, and the clients need to acknowledge each block, taking a long time and prone to timeouts. Fortunately we can use a local webserver. I have a XigmaNAS server that simply requires me to tick a box to enable the webserver service, choose a port and a root directory.



I chose to use http on port 8081 and the NAS has an ip of 192.168.5.5 so that’s the IP the webserver is bound to, on port 8081. We’ll need this information in a little bit.
The webroot is “/mnt/Deimos/www”  .
3.1 Now I need to connect to a shell on the NAS:

Code: [Select]

$ ssh <myuser>@nasip
Then create a subdirectory under the root:

Code: [Select]

$  sudo mkdir -p /mnt/Deimos/www/pxe
I think I had to change ownership before proceeding with subdirectories. This part is from memory, it might be needed as a later step, i.e. after the whole path was created.

Code: [Select]

$ sudo /usr/sbin/chown -R www:www /mnt/Deimos/www/pxe
Then finally creating them with a sane structure:

Code: [Select]

$ sudo mkdir -p /mnt/Deimos/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/
$ sudo fetch https://mfsbsd.vx.sk/files/iso/12/amd64/mfsbsd-12.2-RELEASE-amd64.iso -o /mnt/Deimos/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/mfsbsd.iso
$ sudo gzip -9 /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/mfsbsd.iso
3.2 Now we need to edit the pxelinux.cfg/bsd file and append:

LABEL fbsd-pxe-install
   MENU LABEL Install FreeBSD 12.2 (PXE)
   MENU DEFAULT
   KERNEL memdisk
   INITRD http://192.168.5.5:8081/pxe/bsd/fbsd/amd64/12.2-RELEASE/mfsbsd.iso
   APPEND iso raw

4. Final changes in OPNSense.
This was trial and error & some additional reading online. I can’t explain how or why too well but it works for me.
I don’t use Ipv6, only v4.
I go to Services > DHCPv4 > LAN
4.1 In “TFTP server” field I entered the LAN ip of OPN: 192.168.5.1
In the “Set Bootfile” field I entered: pxelinux.0


To troubleshoot permutations of these fields I used the nmap macro that @Kraileth suggests on one is his posts to check DHCP responses. I used to get a bootfile and server offered:
From a linux machine on my lan:

Code: [Select]

$ sudo nmap --script broadcast-dhcp-discover
[sudo] password for cookiemonster:

Starting Nmap 7.60 ( https://nmap.org ) at 2021-10-01 23:13 BST
Pre-scan script results:
| broadcast-dhcp-discover:
|   Response 1 of 1:
|     IP Offered: 192.168.5.189
|     DHCP Message Type: DHCPOFFER
|     Server Identifier: 192.168.5.1
|     IP Address Lease Time: 5m00s
|     Subnet Mask: 255.255.255.0
|     Router: 192.168.5.1
|     Domain Name Server: 192.168.5.1
|     Domain Name: moomooland
|     Bootfile Name: pxelinux.0
|_    TFTP Server Name: 192.168.5.1
WARNING: No targets were specified, so 0 hosts scanned.
Nmap done: 0 IP addresses (0 hosts up) scanned in 1.69 seconds
5. Testing with an actual client.
For this I used VirtualBox. I created a new Virtual machine.
Type: BSD
Version: FreeBSD (64 bit)
Memory: 1 GB; 18 GB vdi storage
Boot Order: Network, Hard Disk, Optical
Network: Bridged Adapter , Name: the name of the adapter on the host that is plugged into the LAN    that will connect to OPN.
With these settings I am saying, try to boot from the network and that network is the one where OPN is the DHCP server, giving the ip address and the network boot options.



Then I can start the Virtual Machine and verify:





At some point I had to do packet captures to verify the client to figure out that there was a tftp transfer despite the client complaining that thee was no file received.
The problem was that I had entered the “default” or “bsd” file. I didn’t realise it had to be the pxelinux.0 one that was needed in OPN.

6

General Discussion / OPN as a PXE boot server

« on: September 17, 2021, 12:17:37 am »

Hi.
I'm trying to setup OPN to serve PXE booting options to install OSs from the network.
Same question as https://forum.opnsense.org/index.php?topic=23128.0 but it's an old post.
Once working I shall create the relevant documentation.

I've installed the os-tftp plugin and created the directory /usr/local/tftp and put in there my pxe files extracted from the syslinux-6.03.pkg i.e. pxelinux.0  and .c32 files

I've created my /usr/local/tftp/pxelinux.cfg/default file with my entries there pointing to a webserver I have in my LAN and started testing.

My tests are failing with:
"PXE-T01:File not found"

What I've found is that the plugin starts the tftp server and I can get the files with a tftp client and a "get" verb.
Also that it doesn't seem to use inetd as the service control mechanism and rc.d instead. All good.
I've also as a test included the OPN LAN address in the Services > DHCP4 > TFTP and the absolute path & NBP file.

Is the plugin still in development and not finished? I'd like some pointers to troubleshoot.

7

21.7 Legacy Series / Data usage counter

« on: August 14, 2021, 10:50:53 pm »

I did search but I didn't find result, with my apologies if I've searched incorrectly.
I'd like to have a data consumption counter with a configurable starting day of the month. It seems if netflow is enabled, the data is available. I am aware the accounting could differ from the ISP.
I'd like to know how much data I've consumed in a month, every month.
Is this information available, if not can I submit a feature request?

8

21.7 Legacy Series / igb on APU4 with OPNS 21.7

« on: July 30, 2021, 05:34:37 pm »

I am looking forward to performance improvements on these. One can hope.
With the release notes stating the release includes Intel contributions to the drivers, I'd like to know if there are any "old" tunables that aren't needed anymore.

Has anybody upgraded an APU4D4?, any observations?

With only 4 GB of RAM, do we need to use zfs tunables to prevent use of ARC?
What igb tunables are you using?

9

Intrusion Detection and Prevention / [Solved] Understanding GeoIP rules and their logging

« on: July 06, 2021, 04:56:09 pm »

Hi, I'm struggling to confirm if I've setup GeoIP correctly and the logging (or lack thereof) is the expected behaviour.
I have an SSH server in the LAN that I wish to protect with GeoIP blocking. It listens on default port 22.
The setup in OPNS is as per the manual signup to maxmind and setup an Alias called "GeoProtect_Allow" and select only UK from all countries.
NAT rules has the port forward:
LAN   TCP           *   *           LAN address   22, 80, 443   *         *   Anti-Lockout Rule   
LAN   TCP/UDP   *   *           ! LAN net           53 (DNS)          192.168.5.154   53 (DNS)   
WAN   TCP           *   *           WAN address   22 (SSH)          192.168.5.2   22 (SSH)   

My WAN firewall rules have the GeoIP just after the default bogons,dhcp and these manual ones:
log,first match      IPv4 UDP   *                              *      WAN addres       1193         *      *   vpn_in      
log,first match      IPv4 TCP   ! GeoProtect_Allow    *   WAN address   22 (SSH)   *   *   GeoIP_SSH_Allow      
log,first match      IPv4 TCP   *                           *   192.168.5.2   22 (SSH)   *   *

So if I understand correctly the second WAN rule says block the incoming connections to port 22 unless they are in coming from UK. It is set to block. The last one is set to pass.

Troubleshooting so far:
- I've changed from an alias to include all countries I want to block to just the one I want to allow and adjusting the WAN rule to pass. I settled with the current other way around as I think it more logical to be more efficient.
- I've checked there are tables with ip populated in /var/db/aliastables/GeoProtect_Allow.self.txt
- I've checked the rules appear in /tmp/rules.debug
- Followed /var/log/filter/filter{date}.log

My questions:
1. I see in the firewall live logs no record of blocks. Is this expected? The WAN rule is set to log.
2. I see in the firewall live logs records of passes and returns and the respective hits on my SSH server with the auth failures from break-in attempts. The source IPs are from countries expected to be blocked by the rule according to whois checks. How can I verify another way that it is indeed blocking? Other logs?

Thanks in advance.

10

21.1 Legacy Series / [Solved] Making the firewall use another resolver

« on: June 25, 2021, 07:07:32 pm »

The problem: there is no name resolution on the firewall itself. Updates fail, etc. Drill from the console fail i.e "OPNsense:~ % drill example.com
Error: error sending query: Could not send or receive, because of network error"

The setup:
Version: OPNsense 21.1.6-amd64

System > General:
DNS Server is empty and all other three options are not ticked.
As expected /etc/resolv.conf contains "nameserver 127.0.0.1"

I run the "stubby" dns resolver on 127.0.0.1 on port 8053.
Doing a drill command domain search from the console giving it the port 8053 works fine. Stubby works.

Unbound options:

Code: [Select]

server:
do-not-query-localhost: no
forward-zone:
name: "."
forward-addr: 127.0.0.1@8053Unbound is set to LAN and WAN.

Services > DHCPv4 LAN has in the DNS Servers "192.168.5.154" which is the ip of my pi-hole, set to send the client queries upstream back to OPN's static lan ip of 192.168.5.1 that by the unbound forwarding, pushes the query from port 53 to 8053 and everything is good.

My DHCP clients asking DHCP options get an ip and told to use pi-hole for dns queries. There are a couple of nat & firewall rules to force any client trying to bypass it. So far so good. All works as expected.
This included also the firewall itself when I finished setting this up.

I noticed yesterday, after some reboots have happened that the name resolution wasn't working on the firewall.

My troubleshooting hasn't showed me a lot, probably I'm not using the logging correctly.
For instance if I manually add the port in /etc/resolv.conf then the problem goes away as expected from the setup.
Ticking/unticking "Do not use the local DNS service as a nameserver for this system" adds/removes the loopback but doesn't resolve the problem and I wasn't expecting it to.

So the question is if the "forward-addr" forwarding should be expected on the firewall, i.e. it should work as a valid configuration OR it should not be expected by me to work and therefore, What should I do to make the persistent sending of dns queries to 127.0.0.1:8053 on the firewall system (not dhcp clients, that works).
I'll appreciate some guidance.

11

Hardware and Performance / apu4d4 low throughput

« on: June 10, 2021, 06:47:52 pm »

Hello, I'm new to OPSense and I have tried to look on the manual, forum posts and online sources but I will need to ask for some assistance please.
I have this board to setup my router and firewall. I have a 550/75 mbps fiber to the home line.

The problem I have is that I seem to be unable to get more than around 300 mbps throughput.

Setup:
Default firewall rules and two of my own to redirect DNS to an internal client.
No Suricata.
Netflow running.
One OpenVPN server running. No active clients connected.
OPNsense 21.1.6 installed.
No VLANs, only a single flat LAN network.
The LAN interface goes via Cat7 cable to a Gigabit managed switch. No VLANs on the switch.
I have opendns stubby running as recursive resolver.
Unbound running, with opendns stubby as upstream resolver.
My memory usage hovers on 14%. My cpu usage hovers between 3 % and 64 %. It's a bit spikey but seems normally well, not maxed.

Tunables used:
I've collected from various threads what seems valid tunables on these nics i211AT and I have created a loader.conf.local with the following contents:

Code: [Select]

cat /boot/loader.conf.local
amdtemp_load="YES"
ahci_load="YES"
aesni_load="YES"
if_igb_load="YES"
flowd_enable="YES"
flowd_aggregate_enable="YES"
legal.intel_igb.license_ack="1"
legal.intel_ipw.license_ack="1"
legal.intel_iwi.license_ack="1"
# this is the magic. If you don't set this, queues won't be utilized properly
# allow multiple processes for receive/transmit processing
#hw.igb.rx_process_limit="-1"
h#w.igb.tx_process_limit="-1"

net.pf.states_hashsize="2097152"

hw.igb.num_queues="0"

hw.igb.enable_aim="1"

hw.igb.enable_msix="1"
hw.pci.enable_msix="1"
hw.igb.rx_process_limit="-1"
hw.igb.tx_process_limit="-1"

vm.pmap.pti="0"
hw.ibrs_disable="0"

hint.p4tcc.0.disabled="1"
hint.acpi_throttle.0.disabled="1"
hint.acpi_perf.0.disabled="1"
dev.igb.0.eee_control="0"
dev.igb.0.fc="0"

hint.p4tcc.1.disabled="1"
hint.acpi_throttle.1.disabled="1"
hint.acpi_perf.1.disabled="1"
dev.igb.1.eee_control="0"
dev.igb.1.fc="0"

hint.p4tcc.2.disabled="1"
hint.acpi_throttle.2.disabled="1"
hint.acpi_perf.2.disabled="1"
dev.igb.2.eee_control="0"
dev.igb.2.fc="0"

hint.p4tcc.3.disabled="1"
hint.acpi_throttle.3.disabled="1"
hint.acpi_perf.3.disabled="1"
dev.igb.3.eee_control="0"
dev.igb.3.fc="0"

Testing:
To start a baseline, two clients in the LAN iperf from each other and get just over 1 Gbps transfer on two streams (-P 2 option).
When I iperf from any of these clients to public iperf test servers I get only about 350 mbps, for example:

Code: [Select]

~$ iperf3 -p 5200 -f m -V -c speedtest.wtnet.de -P 2 -t 10 -R
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.03  sec   222 MBytes   186 Mbits/sec  379             sender
[  5]   0.00-10.00  sec   219 MBytes   184 Mbits/sec                  receiver
[  7]   0.00-10.03  sec   136 MBytes   114 Mbits/sec   38             sender
[  7]   0.00-10.00  sec   133 MBytes   112 Mbits/sec                  receiver
[SUM]   0.00-10.03  sec   358 MBytes   299 Mbits/sec  417             sender
[SUM]   0.00-10.00  sec   352 MBytes   296 Mbits/sec                  receiver
snd_tcp_congestion cubic
rcv_tcp_congestion cubic
Naturally I have used a handful of public test servers. Results vary but 350 mbps is the max I've achieved. This is after adding the tunables. Without them there were a little lower.

Questions:
- Is my testing flawed, should I do something different?
- Are other tunable suggestions I should try?
And finally, the big question:
- Is it recomended that I install an older version of OPNSense?
I've seen the thread related to 21.1 moving to a different subsystem that is not yet as performant, on account of of freebsd 12.1-RELEASE having that.
I've also noticed Deciso is sticking with OPNsense 20.7 Release for their devices being sold. Maybe it is the right version for production use.

Thanks for reading and I look forward to some advice.

12

21.1 Legacy Series / Unbound, stubby and pi-hole

« on: May 24, 2021, 10:38:45 pm »

Hello.
I have just built my first proper router and firewall with opnsense, replacing a router running DD-WRT.
This is an introduction and request for friendly assistance.
I am familiar with linux and freebsd but not with the subsystems OPNSense uses. I am on a learning curve.

My goal is to have functionally the setup I had before, sending LAN clients for their DNS lookups to a pi-hole, which in turn sends the allowed queries onward to the router, which in turn uses getdns stubby to go to public TLS enabled resolvers for DNSoverTLS.

I am introducing the parts one by one and testing instead of all at once. Currently I want to get stubby and unbound to work, leaving pi-hole out. The pi-hole ip is 192.168.5.154

My problem appears as soon as I change unbound for the setup.

I have read and followed the various posts on this subject on this forum but either I am missing a step or OPSense has changed enough since they were written that I need to start from scratch.

I am using OPNSense version OPNsense 21.1.5-amd64. The host is a PC Engines APU4D2.
The steps so far taken:
1. Install getdns stubby by pkg install on OPNSense. The version is getdns-1.5.2_4. The router ip is static 192.168.5.160
It is configured to run on 127.0.0.1@8053 with only two upstream resolvers until the setup is working. The config is this:

Code: [Select]

root@OPNsense:/usr/local/etc/stubby # cat /usr/local/etc/stubby/stubby.yml

resolution_type: GETDNS_RESOLUTION_STUB

dns_transport_list:
  - GETDNS_TRANSPORT_TLS

tls_authentication: GETDNS_AUTHENTICATION_REQUIRED

tls_query_padding_blocksize: 128

edns_client_subnet_private : 1

round_robin_upstreams: 0

idle_timeout: 10000

tls_ca_path: "/etc/ssl/"

tls_cipher_list: "EECDH+AESGCM:EECDH+CHACHA20"

tls_ciphersuites: "TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256"

tls_min_version: GETDNS_TLS1_2

tls_max_version: GETDNS_TLS1_3

listen_addresses:
  - 127.0.0.1@8053
#  - 0::1@8053

dnssec: GETDNS_EXTENSION_TRUE

appdata_dir: "/var/cache/stubby"

tls_ca_file: "/usr/local/share/certs/ca-root-nss.crt"

upstream_recursive_servers:

  - address_data: 9.9.9.9
    tls_auth_name: "dns.quad9.net"
    tls_port: 853

  - address_data: 1.1.1.1
    tls_auth_name: "cloudflare-dns.com"
    tls_port: 853


2. Pi-hole is not running a DHCP server. It is set to use a single upstream DNS server with the router ip of 192.168.5.160 . It is listening only on its eth0 interface. There are no other interfaces configured. It is a amd64 virtual machine on an ESXi host. It's IP address is 192.168.5.154

3, On OPNSense:
3..1 On " System: Settings: General" > Networking > DNS servers: I have set 127.0.0.1. "DNS server options" I have none selected, none ticked.
3.2 On " Services: Unbound DNS: General" :
- Enable Unbound is ticked.
- Network interfaces I have both LAN and WAN selected. I don't have an option of ALL.
- DHCP registration is ticked
- DHCP static mappings is ticked

In this setup my LAN clients are getting their DNS server allocation from DHCP pointing them to 192.168.5.160. The OPNSense router is resolving DNS for them as a "normal" setup.

As soon as I enable these additional options in Unbound, my LAN clients fail to get their DNS queries resolved:
server:

Code: [Select]

do-not-query-localhost: no
forward-zone:
name: "."
forward-addr: 192.168.5.160@8053I have tried setting these too but they seemed to cause Unbound to fail to start.
#forward-tls-upstream: yes
#logservfail: yes

Unfortunately with no logging available on stubby I'm really stumped on where/how to see where the failure occurs.

Can I ask for some pointers to start my investigation please.