Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Topics - cookiemonster

Pages: [1] 2

24.7 Production Series / Should the Services widget match service names to plugin names

« on: October 09, 2024, 11:36:05 am »

Hello.
Version 24.7.5_3
It might have been asked already when transitioning to OPN 24 series and new dashboard, if so apologies.
I find the Services widget when adding services to show, do not seem to match the services on the left-hand side that are used when configuring them.
Examples:
- Widget: Reverse Proxy and Web Server
I have os-nginx installed and enabled. I imagine this is what the widget is refering to.

- Widget: ddclient
I have both Dynamic DNS and Dynamic DNS (Legacy) installed. Only the new one enabled, result of preparation for deprecation.
Neither match the entry in the widget listing.

This is only cosmetic lack of consistency and the new dashboard is actually nice.
So is a question really, are we expecing them to match or is this how is designed to be?

Virtual private networks / [SOLVED] Unable to set allowed IPs to '0.0.0.0' for wireguard client

« on: June 26, 2024, 12:04:01 am »

I wonder if someone can give me a hand with this.

I am on OPNsense 23.7.12_5-amd64
Errors take the form of:

Code: [Select]

/usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '192.168.5.0/24' -interface 'wg1'' returned exit code '1', the output was ''

Code: [Select]

/usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '0.0.0.0' -interface 'wg1'' returned exit code '1', the output was ''

I have tried stopping the wireguard service first before modifying it but it is the same behaviour.

Those are the errors when changing or adding those networks to an already configured peer. Currently this peer has allowed ips of: 10.0.0.4/32 and works fine but I want to allow it to also reach my LAN which is 192.168.5.0/24. Also there are VLANs on it.
Clearly I have a setting somewhere that creates a bigger problem because when I change the allowed ips to 0.0.0.0/0, all my LAN clients start failing DNS queries. I'm nowhere near getting a handle on the problem.

23.7 Legacy Series / /usr/local/lib/python3.9 space usage

« on: May 14, 2024, 11:05:42 am »

OPNsense 23.7.12_5-amd64
FreeBSD 13.2-RELEASE-p7
OpenSSL 1.1.1w

Hello. I seem to recall this or previous version refactored Unbound to use a devfs device but don't remeeber details. Anyway, that might be nothing to do with today's question.
I've noticed that regularly, as in a number of times a day, the filesystem usage appears to balloon before reducing to more "normal" levels by itself.
When "abnormal":

Code: [Select]

$ df -h
Filesystem                  Size    Used   Avail Capacity  Mounted on
zroot/ROOT/default           16G     10G    6.0G    64%    /
devfs                       1.0K    1.0K      0B   100%    /dev
/dev/ada0p1                 260M    1.8M    258M     1%    /boot/efi
zroot/var/mail              6.0G    112K    6.0G     0%    /var/mail
zroot/var/audit             6.0G     96K    6.0G     0%    /var/audit
zroot/usr/src               6.0G     96K    6.0G     0%    /usr/src
zroot/tmp                   6.0G    3.8M    6.0G     0%    /tmp
zroot                       6.0G     96K    6.0G     0%    /zroot
zroot/var/log               6.4G    433M    6.0G     7%    /var/log
zroot/var/crash             6.0G     96K    6.0G     0%    /var/crash
zroot/var/tmp                12G    5.9G    6.0G    50%    /var/tmp
zroot/usr/ports             6.0G     96K    6.0G     0%    /usr/ports
zroot/usr/home              6.0G     96K    6.0G     0%    /usr/home
devfs                       1.0K    1.0K      0B   100%    /var/dhcpd/dev
/dev/md43                    48M    8.0K     44M     0%    /usr/local/zenarmor/output/active/temp
tmpfs                       100M    576K     99M     1%    /usr/local/zenarmor/run/tracefs
devfs                       1.0K    1.0K      0B   100%    /var/unbound/dev
/usr/local/lib/python3.9     16G     10G    6.0G    64%    /var/unbound/usr/local/lib/python3.9

That seems like Unbound is using 10 GB of storage and a 64%.

A few minutes later, more "normal":

Code: [Select]

$ df -h
Filesystem                  Size    Used   Avail Capacity  Mounted on
zroot/ROOT/default           22G    8.5G     14G    38%    /
devfs                       1.0K    1.0K      0B   100%    /dev
/dev/ada0p1                 260M    1.8M    258M     1%    /boot/efi
zroot/var/mail               14G    112K     14G     0%    /var/mail
zroot/var/audit              14G     96K     14G     0%    /var/audit
zroot/usr/src                14G     96K     14G     0%    /usr/src
zroot/tmp                    14G    3.8M     14G     0%    /tmp
zroot                        14G     96K     14G     0%    /zroot
zroot/var/log                14G    433M     14G     3%    /var/log
zroot/var/crash              14G     96K     14G     0%    /var/crash
zroot/var/tmp                14G    132K     14G     0%    /var/tmp
zroot/usr/ports              14G     96K     14G     0%    /usr/ports
zroot/usr/home               14G     96K     14G     0%    /usr/home
devfs                       1.0K    1.0K      0B   100%    /var/dhcpd/dev
/dev/md43                    48M    8.0K     44M     0%    /usr/local/zenarmor/output/active/temp
tmpfs                       100M    576K     99M     1%    /usr/local/zenarmor/run/tracefs
devfs                       1.0K    1.0K      0B   100%    /var/unbound/dev
/usr/local/lib/python3.9     22G    8.5G     14G    38%    /var/unbound/usr/local/lib/python3.9

Now using 8.5 Gb for a 38%.

In System: Settings: Logging; I have nothing entered and nothing ticked, so all defaults.
In Unbound: Advanced I have:
Log Queries = enabled
Log Replies = enabled
Tag Queries and Replies = enabled
Log local actions = disabled
Log SERVFAIL = enabled
Log Level Verbosity = Level 1 (default)
Log validation level = Level 0 (default)

As to why it makes a difference, that is the system starts swapping (which I can deal with by increasing the storage) but the real important part, it might be a sign of a problem I'm unaware of.
Can someone tell me what causes this pseudo filesystem to grow in normal OPN operation?

General Discussion / VLANs and mixed tagged and untagged traffic

« on: October 20, 2023, 01:45:13 pm »

I've been battling with my VLAN setup to make it compliant with best freebsd advice and not mix tagged and untagged packets but maybe after all the permutations of settings in the managed switch, the concepts that I thought were clear to me are beginning to blur. I presently don't have a problem to solve in my setup.

I have a LAN interface on OPN. The physical cabling of it goes to a port on the managed mikrotik switch.
The switch has a few ports assigned to VLAN 200. The rest have been on default VLAN 1. I recently changed the default to 100, just to avoid VLAN 1, no other reason other than avoid vendor quirks on default 1.
Clients on the access ports are all VLAN-unaware and there are only three ports being tagged with VLAN 200, and the rest are presently left untagged on ingress.
On OPN I have a VLAN tagged 200 with parent igc1 [LAN].
That means OPN will have by this what I would call "normal" interface definition scheme, a mixture a both tagged and untagged packets flowing through the wire.

How are we then to not mix the packets?

General Discussion / Rules association

« on: September 07, 2023, 05:45:53 pm »

Hello.
I really need assistance to spot my error in setting firewall rules.
It is a recent problem, my setup has been working fine for months. The changes that have occurred recently are beyond my understanding of OPN.

Problem:
My LAN clients have been losing DNS resolution regularly.
When it happens, all digs to their configured server 192.168.5.1:53 time out, they have been working for hours.

Setup:
OPNsense 23.1.11_1-amd64
Unbound listens on all interfaces on port 5353.
AdguardHome listens on all interfaces on port 53 of OPN.
Only IPV4 in this installation.
System > Settings > General | DNS servers is empty.
Allow DNS server list to be overridden by DHCP/PPP on WAN is not ticked.
Do not use the local DNS service as a nameserver for this system is ticked.
DHCPv4 has empty fields in the DNS servers field for each interface. Unbound sends DHCPv4 responses pointing the clients to 192.168.5.1 as DNS server, clearly to TCP/UDP port 53.

Code: [Select]

Outbound rules (Hybrid mode):

Interface 	Source 	Source Port 	Destination 	Destination Port 	NAT Address 	NAT Port 	Static Port 	Description
LAN 	LAN net 	tcp/udp/ * 	192.168.5.1/32 	tcp/udp/ 53 	Interface address 	* 	NO 	Prevents hardcoded DNS clients from giving unexpected source error after DNS redirected to Adguard.

Has a match local tag "forward"

Code: [Select]

OPT1 	OPT1 net 	tcp/udp/ * 	192.168.6.1/32 	tcp/udp/ 53 	Interface address 	* 	NO 	Prevents hardcoded DNS clients from giving unexpected source error after DNS redirected to Adguard.

Has a match local tag "forward"

Code: [Select]

NAT:PORT FORWARD:
					Source 				Destination 		NAT
 Interface 	Proto 	Address 	Ports 	Address 	Ports 	IP 	Ports 	Description
<-> 	LAN 	TCP/UDP 	* 	* 	! LAN net 	53 (DNS) 	192.168.5.1 	53 (DNS) 	LAN-Redirect DNS requests to internal DNS resolver Adguard

This rule has a Set local tag "forward-AdG-LAN". Match local tag is empty. NAT reflection is "Use system default". Filter rule association is "Rule LAN-Redirect DNS requests to internal DNS resolver Adguard" which is the rule I modified today to have that label so I could identify it in this association.
Before today the local tag as "forward" and the associated rule was "Redirect DNS requests to internal DNS resolver Adguard"

Code: [Select]

|>      OPT1 	TCP/UDP 	* 	* 	! OPT1 net 	53 (DNS) 	192.168.6.1 	53 (DNS) 	OPT1-Redirect DNS requests to internal DNS resolver Adguard

This rule has a Set local tag "forward-AdG-OPT1". Match local tag is empty. NAT reflection is "Use system default".

This second rule had to be recreated after removing and re-adding the interface OPT1. I guess that this is the reason it isn't shown as a linked rule i.e. Filter rule association is "none".

Code: [Select]

Firewall: Rules: LAN
Protocol 		Source 	Port 	Destination 	Port 	Gateway 	Schedule 		Description
IPv4 TCP/UDP 	* 		* 		192.168.5.1 	53 (DNS) 	* 		* 				LAN-Redirect DNS requests to internal DNS resolver Adguard

This rule I can't edit. I guess it was automatically created by the NAT reflection originally.

Diagnostic
- Suspecting post OPN upgrade from 22.7 to 22.3 I removed and re-added the interface in assigments page.
- Elimination process. I could see a lot of stalled Zenarmor socket-type connections. Changed Zenarmor from native to emulated, then stopped completely. Reboots have happened since. Presently not runnning.
- Elimination process. No errors seen in Unbound logs, dmesg, system logs, no logging available in AdGuard for the service but logs from clients stop.
- When it happens, I can ssh to OPN. dig commands to local host port 53 time out. dig to 5353 succeed.

I suspect I have setup rule reflections incorrectly. Why it is a sudden problem I suspect it was latent. The OPT1 interface is only started being used very recently.

How do I need to interpret the match rule functionality? Does it mean if I have two NAT Port Forward rules that have the same set local tag will confuse the NAT outbound rules if they have the same match local tag and even on different interfaces? In which case, we need set and match to be on the same interfaces only (normally)?
Shouldn't this scenario still not be a problem after all because even if the match sends the packets to the interface that wasn't where it came from, in this case the service is listening in all interfaces?
For now I have prepended or appended an interface name on each rule I can to try to match "set" to the correct "match" but when I re-enabled the OPT1 port forward rule, the problem re-occurred. I will try doing it and resetting states just in case.
From docs: "NAT rules are always processed before filter rules! So for example, if you define a NAT : port forwarding rules without a associated rule, i.e. Filter rule association set to Pass, this has the consequence, that no other rules will apply!"
So is this my problem, that I need to associate with a rule, but I can't tell which rule is the one to use for OPT1 when the list of rules I can identify are the one for Rule LAN-Redirect... , Rule Amazonia... and a bunch of others that just have "Rule".

Please advise.

22.7 Legacy Series / [Solved] Is RSS available on 22.7 series?

« on: September 21, 2022, 06:18:15 pm »

I did some RSS testing when I was on 21.7.5 and there was no discernible impact so I left the tunables in when I upgraded to 22.7. I later saw some of the tunables labelled as unsupported so I removed from the UI. All good.

I am tracing a marked reduction in bitrate from WAN to LAN which made me look at tunables again and the status of RSS as part of the elimination process. Cables and similar are part of the process at the moment of course, and in the meantime, this question.

BEFORE:
I had OPN 21.7.5, OpenSSL, Hardware is APU4.
Upgraded BIOS beforehand, coreboot v 4.14.0.6 .
Network interfaces on this system are igb.

penguin@OPNsense:~ % sudo sysctl -a | grep -i 'isr.bindthreads\|isr.maxthreads\|inet.rss.enabled\|inet.rss.bits'
net.inet.rss.enabled: 1
net.inet.rss.bits: 2
net.isr.bindthreads: 1
net.isr.maxthreads: 4
penguin@OPNsense:~ % sudo netstat -Q
Configuration:
Setting Current Limit
Thread count 4 4
Default queue limit 256 10240
Dispatch policy direct n/a
Threads bound to CPUs enabled n/a

Protocols:
Name Proto QLimit Policy Dispatch Flags
ip 1 1000 cpu hybrid C--
igmp 2 256 source default ---
rtsock 3 256 source default ---
arp 4 256 source default ---
ether 5 256 cpu direct C--
ip6 6 256 cpu hybrid C--
ip_direct 9 256 cpu hybrid C--
ip6_direct 10 256 cpu hybrid C--

NOW:
Now on OPN 22.7.4, OpenSSL, same hardware.

root@OPNsense:/home/penguin # sysctl -a | grep -i 'isr.bindthreads\|isr.maxthreads\|inet.rss.enabled\|inet.rss.bits'
net.inet.rss.enabled: 0
net.inet.rss.bits: 2
net.isr.bindthreads: 0
net.isr.maxthreads: 1

root@OPNsense:/home/penguin # netstat -Q
Configuration:
Setting Current Limit
Thread count 1 1
Default queue limit 256 10240
Dispatch policy direct n/a
Threads bound to CPUs disabled n/a

Protocols:
Name Proto QLimit Policy Dispatch Flags
ip 1 1000 flow default ---
igmp 2 256 source default ---
rtsock 3 256 source default ---
arp 4 256 source default ---
ether 5 256 source direct ---
ip6 6 1000 flow default ---

So it looks like during the upgrade, RSS was disabled. Question 1: Is that expected?
Since I realised that today, I've re-added the tunables that are accepted "net.inet.rss.bits", "net.isr.maxthreads" and "net.inet.rss.enabled" and rebooted . I also had to but netstat -Q shows that threads aren't bound to CPU enabled.
Configuration:
Setting Current Limit
Thread count 4 4
Default queue limit 256 10240
Dispatch policy direct n/a
Threads bound to CPUs disabled n/a

Protocols:
Name Proto QLimit Policy Dispatch Flags
ip 1 1000 cpu hybrid C--
igmp 2 256 source default ---
rtsock 3 256 source default ---
arp 4 256 source default ---
ether 5 256 cpu direct C--
ip6 6 1000 cpu hybrid C--
ip_direct 9 256 cpu hybrid C--
ip6_direct 10 256 cpu hybrid C--

Question 2: Should I expect the setting to change ie. is wrong at present without threads bound to CPU?
Question 3: How do I make it so?

Thanks for the assistance.

Web Proxy Filtering and Caching / nginx process not stopping

« on: September 08, 2022, 02:48:05 pm »

Hello.
I'm on OPN ver OPNsense 22.7.4-amd64 with os-nginx (installed) 1.29_1
I had nginx setup and working but later I didn't needed runing so I stopped it until I need it again later. Mostly but not only using the UI.
Looking for something else, I noticed an nginx process seemed to be starting. I went to check and indeed the UI shows nginx disabled "Enable nginx" (unticked) and the top-right icon appears showing running. If I click on the square stop button, it flashes but stays running. Logs:

Code: [Select]

me@OPNsense:~ % sudo tail /var/log/nginx/error.log
2022/09/08 13:29:00 [notice] 18393#120692: signal process started
2022/09/08 13:30:00 [notice] 47431#100680: signal process started
2022/09/08 13:31:00 [notice] 74833#100689: signal process started
2022/09/08 13:32:00 [notice] 6906#100622: signal process started

Signal every minute to start.

Code: [Select]

me@OPNsense:~ % sudo nginx -T
Password:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful

Unticking enable, apply; followed by unticking and apply seems to stop it but needs doing again after a reboot.
Any ideas where I need to look for what might be restarting it?

Web Proxy Filtering and Caching / Where to place nginx customisations

« on: July 28, 2022, 06:05:37 pm »

Hi.
I'm trying to replicate a working setup from a standalone nginx installation into the OPN nginx plugin.
So far I can see traffic hitting nginx but I can't get the service to work. It's most likely that I need to add customisation to the configuration at least whilst diagnosing.
Setup is:
OPNsense 22.1.10-amd64
FreeBSD 13.0-STABLE
OpenSSL 1.1.1q 5 Jul 2022
os-nginx (installed) 1.28_1

My config:
cat /usr/local/etc/nginx/nginx.conf

Code: [Select]

load_module /usr/local/libexec/nginx/ngx_stream_module.so;
load_module /usr/local/libexec/nginx/ngx_http_naxsi_module.so;
load_module /usr/local/libexec/nginx/ngx_mail_module.so;
load_module /usr/local/libexec/nginx/ngx_http_brotli_filter_module.so;
load_module /usr/local/libexec/nginx/ngx_http_brotli_static_module.so;
load_module /usr/local/libexec/nginx/ngx_http_js_module.so;

user www staff;
worker_processes 1;

error_log  /var/log/nginx/error.log;

events {
    worker_connections  1024;
}

http {
include       mime.types;



log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';
log_format  main_ext  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" '
                      '"$host" sn="$server_name" '
                      'rt=$request_time '
                      'ua="$upstream_addr" us="$upstream_status" '
                      'ut="$upstream_response_time" ul="$upstream_response_length" '
                      'cs=$upstream_cache_status';
log_format  handshake   '"$http_user_agent" "$ssl_ciphers" "$ssl_curves"';
log_format  anonymized  ':: - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';

#tcp_nopush     on;
# https intercept detection
js_import /usr/local/opnsense/scripts/nginx/ngx_functions.js;
js_set $tls_intercepted ngx_functions.check_intercept;

# 200M should be big enough for file servers etc.
client_max_body_size 200M;
brotli_static on;
brotli on;
gzip_static on;
gzip on;
server_tokens off;
sendfile Off;
default_type  application/octet-stream;
keepalive_timeout 60;

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

# TODO add when core is ready for allowing nginx to serve the web interface
# include nginx_web.conf;




# UPSTREAM SERVERS


include opnsense_http_vhost_plugins/*.conf;

}
stream {
    # LOG FORMATS
    log_format main '$remote_addr [$time_local] '
                     '$protocol $status $bytes_sent $bytes_received '
                     '$session_time';
    log_format anonymized ':: [$time_local] '
                     '$protocol $status $bytes_sent $bytes_received '
                     '$session_time';

    # UPSTREAM SERVERS
    upstream upstream15ad4a8a20f14c19a47af261eaa5249e {
        server 192.168.5.1:53 weight=5;
    }
    upstream upstream7f19e5deb72b4c318458205c45343b52 {
        server 192.168.5.1:8053 weight=5;
    }

    # upstream maps


    include opnsense_stream_vhost_plugins/*.conf;

    # servers
    server {

        listen 127.0.0.1:8054 proxy_protocol;

        access_log  /var/log/nginx/stream_7e5528ea-bfe6-46f5-be19-3e967da85398.access.log main;
        error_log  /var/log/nginx/stream_7e5528ea-bfe6-46f5-be19-3e967da85398.error.log info;


        include 7e5528ea-bfe6-46f5-be19-3e967da85398_pre/*.conf;

        proxy_ssl off;
        proxy_pass upstream15ad4a8a20f14c19a47af261eaa5249e;
        proxy_protocol on;

        include 7e5528ea-bfe6-46f5-be19-3e967da85398_post/*.conf;

    }
}
# mail {
# }

I want to include for instance a njs module in my upstream "upstream15ad4a8a20f14c19a47af261eaa5249e" for example.
Similar a logging customisation for my stream block.

Where do I need to place them please?

p.s. the README in the opnsense_stream_vhost_plugins directory has this
"Place server {} in this directory - they are not affected by the enable / disable flag"
but I fail to understand if that's my hint.

General Discussion / Is njs support included in the nginx plugin?

« on: July 25, 2022, 12:27:19 am »

Hello. I did a forum search but I didn't find an answer.
Is njs supported in the nginx plugin?
I ask because I have a little server in my LAN running nginx on ubuntu and I needed njs . It was installed and all was good.
I have since had to shut the host down and migrated my VMs. Instead of setting a new VM I'd like to see if I can run nginx directly on OPN but it won't work for my use without njs. If I need to build nginx to include the njs support, I wouldn't do it on OPN.
Thanks for the advice.

Tutorials and FAQs / OPN as a PXE boot server

« on: October 02, 2021, 05:02:55 pm »

This mini tutorial is intended to setup a PXE boot server in OPNSense.
The clever parts I have used with permission are documented by Kraileth in his blog:
Links:
https://eerielinux.wordpress.com/2021/01/24/multi-os-pxe-booting-from-freebsd-12-introduction-pt-1/
https://eerielinux.wordpress.com/2021/01/30/multi-os-pxe-booting-from-freebsd-12-required-services-pt-2/
https://eerielinux.wordpress.com/2021/02/05/multi-os-pxe-booting-from-freebsd-12-pxe-menu-and-bsd-pt-3/
https://eerielinux.wordpress.com/2021/02/20/multi-os-pxe-booting-from-freebsd-12-linux-illumos-and-more-pt-4-2/
All I’ve done is adapted it to OPNSense and I made a couple of choices for my needs and infrastructure availability.

What we’ll have at the end of this tutorial is OPNSense acting as a PXE boot server that can be used for clients to network boot on a LAN interface. As an working example we’ll be serving a FreeBSD 12 boot option. I am using OPNsense 21.7.2_1-amd64. An important note is this is for BIOS boot, not UEFI.

The only additional dependency required is a plugin “os-tftp” found in the “default” mirror of OPNSense i.e. OPNSense package repository.

As a baseline I describe my setup. I don’t have VLANs and my flat network is in the 192.168.5.0/24 range. OPN has a lan IP of 192.168.5.1. I have a FreeBSD-based NAS with a reserved IP of 192.168.5.5 that has an available webserver running on port 8081. I will use this storage+internal webserver to serve the large isos. It is perfectly possible and simpler to host them in OPN but I have limited storage on my firewall appliance and I don’t want to run a webserver on it either.
I use DHCPv4 and not v6.
I use Unbound for name resolution.

Prerequisites:
A shell account that can elevate permissions with sudo on OPN.
A shell account that can elevate permissions with sudo on a local webserver.
Internet connection to download some necessary files. I use a separate computer and then transfer to the firewall, to avoid costly mistakes. Some will prefer to do it all on the firewall itself.
Some familiarity with basic shell commands and utilities like wget, nano, etc.

Now we start.
1. Install the tftp plugin in OPNSense
Go to System > Firmware > Plugins. Search for os-tftp. Click on the + sign at the end of the row to install it.
You will get a message “The root folder for transfering files is /usr/local/tftp.”
Once the WebUI refreshes you’ll have an entry in Services > TFTP > General where the service can be enabled or disabled.

On a shell connect to OPNSense and verify the path exists. For me it didn’t so I created it

Code: [Select]

$ sudo mkdir -p /usr/local/tftp
Then back on the WebUI I enabled the tftp service and entered my OPN LAN address 192.168.5.1 and save. Now we have a tftp server listening. To check:

Code: [Select]

$ sudo sockstat -4l
Password:
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS      
root     	    in.tftpd            48436 4       udp4   192.168.5.1:69                    *:*

I did root around the system to figure out how the plugin sets things up but I couldn’t find much. I was fumbling in the dark with this. I see it does not use inetd but I can’t yet find it’s config. I found an entry in etc/rc.d/ftpd but not much else. Nevertheless it is running and working fine for our purposes, and starts after reboot.
If someone finds where the control and config files are, let me know.

1.1 Create the pxe boot files being given by ftpd

Code: [Select]

sudo mkdir -p /usr/local/tftp/pxelinux.cfg
sudo nano /usr/local/tftp/pxelinux.cfg/default

Insert the following and save:

DEFAULT vesamenu.c32
PROMPT 0
MENU TITLE PXE Boot Menu (Main)

LABEL bsd-oses
   MENU LABEL BSD Operating Systems
   KERNEL vesamenu.c32
   APPEND pxelinux.cfg/bsd

Now for a test, from a linux or FreeBSD machine on the LAN do:

Code: [Select]

$ cd /tmp
$ tftp 192.168.5.1
tftp> get pxelinux.cfg/default
Received 190 bytes in 0.1 seconds
tftp> quit
$ rm default

That confirms our tftp daemon on OPN is serving the files correctly. I was expecting to need to open the port 69 but it seems the default pass rule from the LAN takes care of it.
You will need to adjust the firewall rules if you don’t have that permissive rule in place.

Now we need to create the referenced file “bsd”:

Code: [Select]

$ sudo nano /usr/local/tftp/pxelinux.cfg/bsdinsert this:

MENU TITLE PXE Boot Menu (BSD)

LABEL main-menu
   MENU LABEL Main Menu
   KERNEL vesamenu.c32
   APPEND pxelinux.cfg/default

This step 1.1 has created a default file and a bsd file, both inside a subdirectory under our tftp root.
Now we’re ready to get those files they reference.

2. Get the PXElinux files.
Ordinarily I do this on a separate machine and then transfer the files but on this ocassion I’ll do it directly on OPN.

Code: [Select]

$ cd /tmp
$ sudo pkg fetch -y syslinux
$ sudo mkdir -p /tmp/syslinux
$ sudo tar -C /tmp/syslinux -xvf /var/cache/pkg/syslinux-6.03.pkg

Now we copy the required files:

Code: [Select]

$ sudo cp /tmp/syslinux/usr/local/share/syslinux/bios/core/lpxelinux.0 /usr/local/tftp/pxelinux.0
$ sudo cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/elflink/ldlinux/ldlinux.c32 /usr/local/tftp/
$ sudo cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/menu/vesamenu.c32 /usr/local/tftp/
$ sudo cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/lib/libcom32.c32 /usr/local/tftp/
$ sudo cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/libutil/libutil.c32 /usr/local/tftp/
$ sudo cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/modules/pxechn.c32 /usr/local/tftp/
$ sudo cp /tmp/syslinux/usr/local/share/syslinux/bios/memdisk/memdisk /usr/local/tftp/
$ sudo rm -r /tmp/syslinux

At this point we have on our tftp server the files required for menus and the files required to get the clients to boot to a prompt for an installation. However we don’t have an installation media to offer yet.
For our example we are now going to get a FreeBSD 12.2 distribution iso.
Kraileth makes a great explanation of why for FreeBSD we need an mfsBSD https://mfsbsd.vx.sk/ image that can network boot.

3. Getting the image media available on our webserver.
TFTP is not meant to be used to transfer large files like distro isos. It is too slow, transferring files by limited block sizes, and the clients need to acknowledge each block, taking a long time and prone to timeouts. Fortunately we can use a local webserver. I have a XigmaNAS server that simply requires me to tick a box to enable the webserver service, choose a port and a root directory.

I chose to use http on port 8081 and the NAS has an ip of 192.168.5.5 so that’s the IP the webserver is bound to, on port 8081. We’ll need this information in a little bit.
The webroot is “/mnt/Deimos/www” .
3.1 Now I need to connect to a shell on the NAS:

Code: [Select]

$ ssh <myuser>@nasip
Then create a subdirectory under the root:

Code: [Select]

$ sudo mkdir -p /mnt/Deimos/www/pxe
I think I had to change ownership before proceeding with subdirectories. This part is from memory, it might be needed as a later step, i.e. after the whole path was created.

Code: [Select]

$ sudo /usr/sbin/chown -R www:www /mnt/Deimos/www/pxe
Then finally creating them with a sane structure:

Code: [Select]

$ sudo mkdir -p /mnt/Deimos/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/
$ sudo fetch https://mfsbsd.vx.sk/files/iso/12/amd64/mfsbsd-12.2-RELEASE-amd64.iso -o /mnt/Deimos/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/mfsbsd.iso
$ sudo gzip -9 /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/mfsbsd.iso

3.2 Now we need to edit the pxelinux.cfg/bsd file and append:

LABEL fbsd-pxe-install
   MENU LABEL Install FreeBSD 12.2 (PXE)
   MENU DEFAULT
   KERNEL memdisk
   INITRD http://192.168.5.5:8081/pxe/bsd/fbsd/amd64/12.2-RELEASE/mfsbsd.iso
   APPEND iso raw

4. Final changes in OPNSense.
This was trial and error & some additional reading online. I can’t explain how or why too well but it works for me.
I don’t use Ipv6, only v4.
I go to Services > DHCPv4 > LAN
4.1 In “TFTP server” field I entered the LAN ip of OPN: 192.168.5.1
In the “Set Bootfile” field I entered: pxelinux.0

To troubleshoot permutations of these fields I used the nmap macro that @Kraileth suggests on one is his posts to check DHCP responses. I used to get a bootfile and server offered:
From a linux machine on my lan:

Code: [Select]

$ sudo nmap --script broadcast-dhcp-discover
[sudo] password for cookiemonster: 

Starting Nmap 7.60 ( https://nmap.org ) at 2021-10-01 23:13 BST
Pre-scan script results:
| broadcast-dhcp-discover: 
|   Response 1 of 1: 
|     IP Offered: 192.168.5.189
|     DHCP Message Type: DHCPOFFER
|     Server Identifier: 192.168.5.1
|     IP Address Lease Time: 5m00s
|     Subnet Mask: 255.255.255.0
|     Router: 192.168.5.1
|     Domain Name Server: 192.168.5.1
|     Domain Name: moomooland
|     Bootfile Name: pxelinux.0
|_    TFTP Server Name: 192.168.5.1
WARNING: No targets were specified, so 0 hosts scanned.
Nmap done: 0 IP addresses (0 hosts up) scanned in 1.69 seconds

5. Testing with an actual client.
For this I used VirtualBox. I created a new Virtual machine.
Type: BSD
Version: FreeBSD (64 bit)
Memory: 1 GB; 18 GB vdi storage
Boot Order: Network, Hard Disk, Optical
Network: Bridged Adapter , Name: the name of the adapter on the host that is plugged into the LAN that will connect to OPN.
With these settings I am saying, try to boot from the network and that network is the one where OPN is the DHCP server, giving the ip address and the network boot options.

Then I can start the Virtual Machine and verify:

At some point I had to do packet captures to verify the client to figure out that there was a tftp transfer despite the client complaining that thee was no file received.
The problem was that I had entered the “default” or “bsd” file. I didn’t realise it had to be the pxelinux.0 one that was needed in OPN.

General Discussion / OPN as a PXE boot server

« on: September 17, 2021, 12:17:37 am »

Hi.
I'm trying to setup OPN to serve PXE booting options to install OSs from the network.
Same question as https://forum.opnsense.org/index.php?topic=23128.0 but it's an old post.
Once working I shall create the relevant documentation.

I've installed the os-tftp plugin and created the directory /usr/local/tftp and put in there my pxe files extracted from the syslinux-6.03.pkg i.e. pxelinux.0 and .c32 files

I've created my /usr/local/tftp/pxelinux.cfg/default file with my entries there pointing to a webserver I have in my LAN and started testing.

My tests are failing with:
"PXE-T01:File not found"

What I've found is that the plugin starts the tftp server and I can get the files with a tftp client and a "get" verb.
Also that it doesn't seem to use inetd as the service control mechanism and rc.d instead. All good.
I've also as a test included the OPN LAN address in the Services > DHCP4 > TFTP and the absolute path & NBP file.

Is the plugin still in development and not finished? I'd like some pointers to troubleshoot.

21.7 Legacy Series / Data usage counter

« on: August 14, 2021, 10:50:53 pm »

I did search but I didn't find result, with my apologies if I've searched incorrectly.
I'd like to have a data consumption counter with a configurable starting day of the month. It seems if netflow is enabled, the data is available. I am aware the accounting could differ from the ISP.
I'd like to know how much data I've consumed in a month, every month.
Is this information available, if not can I submit a feature request?

21.7 Legacy Series / igb on APU4 with OPNS 21.7

« on: July 30, 2021, 05:34:37 pm »

I am looking forward to performance improvements on these. One can hope.
With the release notes stating the release includes Intel contributions to the drivers, I'd like to know if there are any "old" tunables that aren't needed anymore.

Has anybody upgraded an APU4D4?, any observations?

With only 4 GB of RAM, do we need to use zfs tunables to prevent use of ARC?
What igb tunables are you using?

Intrusion Detection and Prevention / [Solved] Understanding GeoIP rules and their logging

« on: July 06, 2021, 04:56:09 pm »

Hi, I'm struggling to confirm if I've setup GeoIP correctly and the logging (or lack thereof) is the expected behaviour.
I have an SSH server in the LAN that I wish to protect with GeoIP blocking. It listens on default port 22.
The setup in OPNS is as per the manual signup to maxmind and setup an Alias called "GeoProtect_Allow" and select only UK from all countries.
NAT rules has the port forward:
LAN   TCP     *   *     LAN address   22, 80, 443   *     *   Anti-Lockout Rule
LAN   TCP/UDP   *   *     ! LAN net     53 (DNS)     192.168.5.154   53 (DNS)
WAN   TCP     *   *     WAN address   22 (SSH)     192.168.5.2   22 (SSH)

My WAN firewall rules have the GeoIP just after the default bogons,dhcp and these manual ones:
log,first match IPv4 UDP   * * WAN addres 1193 * *   vpn_in
log,first match IPv4 TCP   ! GeoProtect_Allow    *   WAN address   22 (SSH)   *   *   GeoIP_SSH_Allow
log,first match IPv4 TCP   *     *   192.168.5.2   22 (SSH)   *   *

So if I understand correctly the second WAN rule says block the incoming connections to port 22 unless they are in coming from UK. It is set to block. The last one is set to pass.

Troubleshooting so far:
- I've changed from an alias to include all countries I want to block to just the one I want to allow and adjusting the WAN rule to pass. I settled with the current other way around as I think it more logical to be more efficient.
- I've checked there are tables with ip populated in /var/db/aliastables/GeoProtect_Allow.self.txt
- I've checked the rules appear in /tmp/rules.debug
- Followed /var/log/filter/filter{date}.log

My questions:
1. I see in the firewall live logs no record of blocks. Is this expected? The WAN rule is set to log.
2. I see in the firewall live logs records of passes and returns and the respective hits on my SSH server with the auth failures from break-in attempts. The source IPs are from countries expected to be blocked by the rule according to whois checks. How can I verify another way that it is indeed blocking? Other logs?

Thanks in advance.

21.1 Legacy Series / [Solved] Making the firewall use another resolver

« on: June 25, 2021, 07:07:32 pm »

The problem: there is no name resolution on the firewall itself. Updates fail, etc. Drill from the console fail i.e "OPNsense:~ % drill example.com
Error: error sending query: Could not send or receive, because of network error"

The setup:
Version: OPNsense 21.1.6-amd64

System > General:
DNS Server is empty and all other three options are not ticked.
As expected /etc/resolv.conf contains "nameserver 127.0.0.1"

I run the "stubby" dns resolver on 127.0.0.1 on port 8053.
Doing a drill command domain search from the console giving it the port 8053 works fine. Stubby works.

Unbound options:

Code: [Select]

server:
do-not-query-localhost: no
forward-zone:
name: "."
forward-addr: 127.0.0.1@8053

Unbound is set to LAN and WAN.

Services > DHCPv4 LAN has in the DNS Servers "192.168.5.154" which is the ip of my pi-hole, set to send the client queries upstream back to OPN's static lan ip of 192.168.5.1 that by the unbound forwarding, pushes the query from port 53 to 8053 and everything is good.

My DHCP clients asking DHCP options get an ip and told to use pi-hole for dns queries. There are a couple of nat & firewall rules to force any client trying to bypass it. So far so good. All works as expected.
This included also the firewall itself when I finished setting this up.

I noticed yesterday, after some reboots have happened that the name resolution wasn't working on the firewall.

My troubleshooting hasn't showed me a lot, probably I'm not using the logging correctly.
For instance if I manually add the port in /etc/resolv.conf then the problem goes away as expected from the setup.
Ticking/unticking "Do not use the local DNS service as a nameserver for this system" adds/removes the loopback but doesn't resolve the problem and I wasn't expecting it to.

So the question is if the "forward-addr" forwarding should be expected on the firewall, i.e. it should work as a valid configuration OR it should not be expected by me to work and therefore, What should I do to make the persistent sending of dns queries to 127.0.0.1:8053 on the firewall system (not dhcp clients, that works).
I'll appreciate some guidance.

Pages: [1] 2