Topics

1

24.7 Production Series / KEA DHCPv4 Option 6 (DNS) multiple values + Option 43

« on: November 27, 2024, 07:48:37 pm »

Hello,
Running ISC DHCPv4 on OPNsense 24.10.1-amd64 (business edition). KEA DHCPv4 server seems stable enough to considering moving over. For ISC, Opnsense GUI provided only 2 values for DNS servers per subnet. However, one could use Additional Options-->6 followed by hex string to add >2 DNS alternatives. I have 4 configured (adguard+pihole running on VMs + on Pi2 - for when server reboots are required).

Code: [Select]

c0:a8:64:24:c0:a8:64:22:c0:a8:64:23:c0:a8:64:24
I'd like to maintain this setup in KEA. Their https://downloads.isc.org/isc/kea/2.6.1/doc/html/arm/dhcp4-srv.html indicates that multiple values are possible. But, because I haven't migrated to KEA yet, I can't tell whether such multiple (more than 2) values in DNS options will be supported. Any guidance would be much appreciated.

Furthemore, in order to provide Ruckus/Cisco Wi-APs with controller information, i use Option 43 like so

Code: [Select]

type=string "hex 060c3139322e3136382e302e3431"
But, in the GUI, i'm unable to find a way to provide such additional custom options which survive reboot?

2

General Discussion / looking for ipv6 filterlog samples for {TCP, UDP, ICMP} please

« on: August 01, 2024, 06:23:17 am »

Hello all,
I'm sure many of you have implemented bsmithio's project https://github.com/bsmithio/OPNsense-Dashboard/tree/master for obtaining and rendering OPNSense telemetry data via TIG stack (Telegraf, InfluxDBv2, Grafana).

A recent change in OPNSense's filterlog data caused the filterlog processing in graylog to break. I forked (https://github.com/morikplay/OPNsense-Dashboard) the aforementioned project and fixed it for IPv4 packets (+ few enhancements). Not having enabled IPv6 in my home network, I am unable to complete the changes for others to potentially benefit from. I did look at this link https://github.com/opnsense/ports/blob/master/opnsense/filterlog/files/description.txt Franco sent in the one of the forum threads. But, I'd appreciate a few (5-10) sample filterlog traces for ipv6 packets with UDP, TCP and ICMP each, please. This will help me verify implementation and give back to the open-source community.

3

24.1 Legacy Series / Post 24.4.1 (business) upgrade FW initiated traffic is blocking

« on: June 28, 2024, 03:11:21 pm »

(updated w/ logs - initial post was done via cellphone)
Hello experts,
When on 23.x business edition, life was great. 24. X Upgrade was to make it better. To a large degree it is. But, I have a strange new problem which I’m unable to solve. Two plugins: crowdsec (8080 port) and Telegraf (port 8086 for influx) stopped working. Logs indicate a connection timeout for both services. The destination endpoints (on opt6) are fine, and reachable to:from elsewhere both inside and outside the network; just not for when originating from firewall for non-ICMP traffic. No rule changes at my end. Results in a timeout.

Code: [Select]

traceroute to 192.168.100.21 (192.168.100.21), 64 hops max, 40 byte packets
 1  crowdsec-lapi (192.168.100.21)  0.656 ms  0.416 ms  0.330 ms

Live log doesn’t show packet blocks. It does show “let packets from firewall itself in the out direction but nothing in the reverse direction (which should be allowed by default given the stateful nature of flows).

Code: [Select]

curl -vi --connect-timeout 10 http://crowdsec-lapi.esco.ghaar:8080
* Host crowdsec-lapi.esco.ghaar:8080 was resolved.
* IPv6: (none)
* IPv4: 192.168.100.21
*   Trying 192.168.100.21:8080...
* ipv4 connect timeout after 9999ms, move on!
* Failed to connect to crowdsec-lapi.esco.ghaar port 8080 after 10006 ms: Timeout was reached
* Closing connection
curl: (28) Failed to connect

interface capture shows:

Code: [Select]

Servers
vlan0.100 2024-06-28
07:37:50.442037 f4:90:ea:00:9f:72 00:50:56:82:d8:b4 ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.100.1.31315 > 192.168.100.21.8080: Flags [S], cksum 0x8070 (correct), seq 445912424, win 65535, options [mss 8960,nop,wscale 12,sackOK,TS val 1292126707 ecr 0], length 0
Servers
vlan0.100 2024-06-28
07:37:50.442400 00:50:56:82:d8:b4 f4:90:ea:00:9f:72 ethertype IPv4 (0x0800), length 74: (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.100.21.8080 > 192.168.100.1.31315: Flags [S.], cksum 0xe967 (correct), seq 3873949677, ack 445912425, win 43440, options [mss 1460,sackOK,TS val 3838080763 ecr 1292126707,nop,wscale 9], length 0
Servers
vlan0.100 2024-06-28
07:37:51.442697 f4:90:ea:00:9f:72 00:50:56:82:d8:b4 ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.100.1.31315 > 192.168.100.21.8080: Flags [S], cksum 0x7c87 (correct), seq 445912424, win 65535, options [mss 8960,nop,wscale 12,sackOK,TS val 1292127708 ecr 0], length 0
Servers
vlan0.100 2024-06-28
07:37:51.443231 00:50:56:82:d8:b4 f4:90:ea:00:9f:72 ethertype IPv4 (0x0800), length 74: (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.100.21.8080 > 192.168.100.1.31315: Flags [S.], cksum 0xe57e (correct), seq 3873949677, ack 445912425, win 43440, options [mss 1460,sackOK,TS val 3838081764 ecr 1292126707,nop,wscale 9], length 0
Servers
vlan0.100 2024-06-28
07:37:52.462713 00:50:56:82:d8:b4 f4:90:ea:00:9f:72 ethertype IPv4 (0x0800), length 74: (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.100.21.8080 > 192.168.100.1.31315: Flags [S.], cksum 0xe182 (correct), seq 3873949677, ack 445912425, win 43440, options [mss 1460,sackOK,TS val 3838082784 ecr 1292126707,nop,wscale 9], length 0
Servers
vlan0.100 2024-06-28
07:37:53.642675 f4:90:ea:00:9f:72 00:50:56:82:d8:b4 ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.100.1.31315 > 192.168.100.21.8080: Flags [S], cksum 0x73ef (correct), seq 445912424, win 65535, options [mss 8960,nop,wscale 12,sackOK,TS val 1292129908 ecr 0], length 0
Servers
vlan0.100 2024-06-28
07:37:53.643161 00:50:56:82:d8:b4 f4:90:ea:00:9f:72 ethertype IPv4 (0x0800), length 74: (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.100.21.8080 > 192.168.100.1.31315: Flags [S.], cksum 0xdce6 (correct), seq 3873949677, ack 445912425, win 43440, options [mss 1460,sackOK,TS val 3838083964 ecr 1292126707,nop,wscale 9], length 0
Servers
vlan0.100 2024-06-28
07:37:55.662758 00:50:56:82:d8:b4 f4:90:ea:00:9f:72 ethertype IPv4 (0x0800), length 74: (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.100.21.8080 > 192.168.100.1.31315: Flags [S.], cksum 0xd502 (correct), seq 3873949677, ack 445912425, win 43440, options [mss 1460,sackOK,TS val 3838085984 ecr 1292126707,nop,wscale 9], length 0
Servers
vlan0.100 2024-06-28
07:37:57.842474 f4:90:ea:00:9f:72 00:50:56:82:d8:b4 ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.100.1.31315 > 192.168.100.21.8080: Flags [S], cksum 0x6387 (correct), seq 445912424, win 65535, options [mss 8960,nop,wscale 12,sackOK,TS val 1292134108 ecr 0], length 0
Servers
vlan0.100 2024-06-28
07:37:57.842885 00:50:56:82:d8:b4 f4:90:ea:00:9f:72 ethertype IPv4 (0x0800), length 74: (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.100.21.8080 > 192.168.100.1.31315: Flags [S.], cksum 0xcc7e (correct), seq 3873949677, ack 445912425, win 43440, options [mss 1460,sackOK,TS val 3838088164 ecr 1292126707,nop,wscale 9], length 0
Servers
vlan0.100 2024-06-28
07:38:01.966765 00:50:56:82:d8:b4 f4:90:ea:00:9f:72 ethertype IPv4 (0x0800), length 74: (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.100.21.8080 > 192.168.100.1.31315: Flags [S.], cksum 0xbc62 (correct), seq 3873949677, ack 445912425, win 43440, options [mss 1460,sackOK,TS val 3838092288 ecr 1292126707,nop,wscale 9], length 0


Repeating of seq#s indicates (to me) that .100.1 (opnsense) is:

• establishing socket open to .100.21:8080 (server in question)
• server responds with SYN ACK
• but opnsense doesn't respond with an ACK

iii would mean opnsense is eating it up? But, why?

I’ve tried enabling various combination of explicit rules to allow “opt 6 address” —> “server net + ports” combination to no avail. On disabling entire firewall, the first issuance of curl command succeeds. In that I get a 401 unauthorized. But immediately following it, subsequent connection attempts end up in a black hole.

How might I go about troubleshooting this behavior?

Edit#1: What is strange(r) indeed is that this behavior is occuring on every subnet as long as a) traffic originates from opnsense and b) initial few attempts of connection establishment succeed, but then subsequent attempts time out.

Code: [Select]

#nc -4znvw 10 192.168.0.58 443
Connection to 192.168.0.58 443 port [tcp/*] succeeded!
#nc -4znvw 10 192.168.0.58 443
nc: connect to 192.168.0.58 port 443 (tcp) failed: Operation timed out
# nc -4znvw 10 192.168.0.58 443
nc: connect to 192.168.0.58 port 443 (tcp) failed: Operation timed out


4

General Discussion / crowdsec + external postgresql

« on: June 19, 2024, 07:45:33 pm »

In need of the experts' advise once again.

What: os-crowdsec installed and works like a charm w/ local SQLite db. When switching it out to an (external) postgresql on the local network, all hell breaks loose.

System Info:

• Opnsense 24.4_8-amd64
• FreeBSD 13.2-RELEASE-p11
• os-sec 1.07

postgresql config in crowdsec config.yaml

Code: [Select]

db_config:
  type: pgx
  user: <user>
  password: <pwd>
  host: <host_ip>
  port: <host_port>
  db_name: crowdsec
  sslmode: prefer
  max_open_conns: 100
  decision_bulk_size: 2000
  flush:
    max_items: 10000
    max_age: 90d

Issue crowdsec service does not start after the change to config.yaml. It can't seem to connect to postgresql database. Database is verified to be up, and credentials work when using psql locally on db server and also remotely via another ubuntu machine.

Code: [Select]

[fbfdf7e6-bc7e-4543-b7bc-d7fadff59603] Script action stderr returned "b'{"level":"error","msg":"error while performing request: dial tcp <ip>:8080: i/o timeout; 4 retries left","time":"2024-06-19T01:39:21-07:00"}\n{"level":"error","msg":"error while performing request: dial tcp <ip>:8080: i/o timeout; 3 retries l'"


5

General Discussion / Unbound + dnsmasq

« on: November 28, 2023, 08:49:41 pm »

Hello experts,
I have the following setup:
Internet<--DoT-->Unbound(also maintains DHCP mappings)<--(normal_DNS)-->Pihole<--(normal_DNS)-->clients
 clients are configured w/ Pihole addresses. Pihole is configured with Unbound as upstream DNS. Unbound is configured with DNS over TLS for WAN resolution.

Recently, I purchased a roborock S8 vacuum cleaner. Created a firewall rule to allow VLAN_x traffic to certain FQDN (

Code: [Select]

mqtt-us.roborock.com)over 8883 port. It worked great for a day. App stopped working the next day. A quick dig revealed the issue. Destination IP addresses had changed. So, I manually updated the address in firewall rule. Great. Day#2 same issue. Same solution. Day#3 same issue. So on and so forth. A more elegant solution was required.
 

Code: [Select]

$ dig mqtt-us.roborock.com

; <<>> DiG 9.10.6 <<>> mqtt-us.roborock.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46538
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;mqtt-us.roborock.com. IN A

;; ANSWER SECTION:
mqtt-us.roborock.com. 583 IN CNAME mqtt-slb-1st-1913472363.us-east-1.elb.amazonaws.com.
mqtt-slb-1st-1913472363.us-east-1.elb.amazonaws.com. 60 IN A 44.209.56.31
mqtt-slb-1st-1913472363.us-east-1.elb.amazonaws.com. 60 IN A 52.7.27.196
mqtt-slb-1st-1913472363.us-east-1.elb.amazonaws.com. 60 IN A 54.235.188.250

;; Query time: 355 msec
;; SERVER: a.b.c.d#53
;; WHEN: Tue Nov 28 11:11:30 PST 2023
;; MSG SIZE  rcvd: 159

Their servers are hosted in AWS fronted via application loadbalancers. Therefore, A/AAAA addresses keep changing (not just rotating).

The answer to this problem seems to lie in https://forum.opnsense.org/index.php?topic=27650.0 thread. Meaning, use dnsmasq to resolve a wildcard / specific_domain, store result in an alias which is used in Opnsense firewall rule. Details of which points to https://github.com/opnsense/core/issues/4145. Great! I'd like to try it. But, my issue: how to enable dnsmasq with unbound just for those domains?

It took a long time to make end-to-end DNS flows in the home setup functional. How does one go about enabling dnsmasq to work together w/ unbound with minimal changes? Because unbound is running on 53, at minimum, i'll assume i can't run dnsmasq on the same port? If case be, how to configure dnsmasq and only have it respond to wildcard queries, from unbound, related to

Code: [Select]

mqtt*.roborock.com? google gods aren't showing mercy.

Help please.

6

General Discussion / LAGGy thoughts

« on: January 05, 2023, 03:32:19 am »

In trying to configure new hardware (capable of 2x25G), I’m running into issues wrt importing configuration from old hardware. Specifically interested in link aggregation (IEEE 802.3ad, 802.1ax). Link aggregation is strictly an OSI Layer-2 concept. I have LACP appropriately configured on the switch side. But, in order to employ LAGG, opnsense seems to require enabling the LAGG interface; doing so requires it be given an IP address. So, this is a bit puzzling to me. Any particular reason for why opnsense designers and/or FreeBSD folks chose such an approach?

 I have over 30 VLANs, 300+ devices, and a rather large number of firewall rules currently serving non-LAGG’ed interfaces. Trying to find the right way to design the opnsense rule system. A penny for your thoughts?

vlan3: IP range IPR1 (Main LAN)
Vlan2: IPR2 (opt2)
vlan100: IPR3 (opt3)
etc
IP Ranges are non-overlapping.
FW Rules are based on above lans; not directly on interfaces - to allow future portability.
Switch has default gw set on IP1 in IPR1.

Now to create a lag, opnsense adds the following to config.xml

Code: [Select]

<lagg>igb0, igb2, igb3 … </lagg>
To enable LAGG, an IP address range seems mandatory.

Code: [Select]

<lan><if>lagg0</if><enable>1</enable></lagg>
 I’d rather not change my existing VLAN setup else my fw rules will be wonky.

Would there be a way to directly achieve this via config.xml? Conceptually, if assign a new IP range to lagg0, will the range have to cover IP ranges of all VLANs? If so, then I won’t be able to have granular per VLAN rules (which are per IP range based off vlan ids. If I select a new IP range then how will it carry traffic belonging to IP addresses not part of its range on trunk interface towards switch?

7

General Discussion / General rule of thumb when creating rules

« on: December 01, 2022, 03:16:00 am »

Hello experts,

After google searching here + reddit, I wanted to distill best practice when creating new firewall rules for new interfaces. Of course, this doesn't fit every use-case however i wanted to understand if the general direction seems right. A penny for your thoughts?

Background:

• VLANs implemented @ switch-level
• All inter-VLAN traffic from switch must be routed to OpnSense (for visibility and rule enforcement)
• Provisioning of rules is not done at "interface" level.
  
  • Instead it is done at VLAN-level.
  • VLANs are then assigned to interfaces. In my case, 3 ports on firewall are LACP'ed together.
  • This LAGG'ed interface is not "enabled" in configuration i.e. only L2 processing occurs.
  • Each VLAN is then assigned to this LAGG'ed interface.
  • Each VLAN has its firewall rules.
  This makes for an easier migration to a better HW platform.

Thumb Rules:

• If inter-VLAN routing across multiple VLANs is required, best to create floating rules e.g. if VLAN1,2,3 need access to each other, at specific ports, but not to VLAN 4,5,6 then group VLANs, create floating rules @ group level along with source/dest addr/port filters
• if required, create RFC1908 inversion rule to allow external access (E.g. port 22, 80, 443 etc) to outside
• if required, handling of NTP and DNS ports. Make this rule "not immediate". I've learnt that IoT devices are particularly nasty in ignoring NTP and DNS settings set via signalling or external means. For this, NAT re-direction is also required in addition
• Each interface/VLAN then has its own set of rules specific to that interface/VLAN

I read somewhere that the first rule should always be to allow traffic into that interface's address. But, I'm unable to ascertain whether that is a good idea.

Attached is a sample config for my IoT VLAN.

8

General Discussion / sshlockout setting

« on: December 01, 2022, 12:06:03 am »

Hello experts,

Issue: After 3 consecutive failed attempts at ssh'ing as root, from a LAN machine (say ip1), I was no longer able to ssh.
Fix: ssh as root from a different machine (ip2), issued

Code: [Select]

pfctl -T flush -t sshlockout to flush the entry, and life was good.

Questions:

• When viewing Firewall->Diagnostics->Sessions->'select rule' sshlockout had two entries for the rule. Why? One for LAN and another for WAN interface?
• Neither of entries showed the culprit ip address (ip1). Both entries were empty.
• For my future reference, how does one view entries in sshlockout table of pfctl?
• For my future reference, which configuration parameter does one tweak to adjust sshlockout? e.g. increase or decrease # of consecutive attempts? or total # of attempts in X mins etc?

Your time and responses are much appreciated.
[/list]

9

Hardware and Performance / igb vs igc

« on: November 16, 2022, 01:05:40 pm »

(still a noob on OpnSense & FreeBSD)

In trying to play around with system tunables, I had a question.
OS: FreeBSD 13.0
OPNSense: 22.7.7_1-amd64
HW: Protecli VP2410
Processor: Intel(R) Celeron(R) J4125 CPU @ 2.00GHz

It has 4 1G ports by Intel NIC.

Code: [Select]

dmesg | grep -t Intel
shows

Code: [Select]

igb0: <Intel(R) I211 (Copper)> port 0x1000-0x101f mem 0x91100000-0x9111ffff,0x91120000-0x91123fff at device 0.0 on pci1
igb1: <Intel(R) I211 (Copper)> port 0x2000-0x201f mem 0x91200000-0x9121ffff,0x91220000-0x91223fff at device 0.0 on pci2
igb2: <Intel(R) I211 (Copper)> port 0x3000-0x301f mem 0x91300000-0x9131ffff,0x91320000-0x91323fff at device 0.0 on pci3
igb3: <Intel(R) I211 (Copper)> port 0x4000-0x401f mem 0x91400000-0x9141ffff,0x91420000-0x91423fff at device 0.0 on pci6

Interface names are igbX where X = 0,1,2,3

But when I try to tune a corresponding value e.g. hw.igb.rx_process_limits it shows as unsupported.

So upon checking

Code: [Select]

sysctl hw following is seen

Code: [Select]

hw.igc.max_interrupt_rate: 8000
hw.igc.eee_setting: 1
hw.igc.rx_process_limit: 100
hw.igc.sbp: 1
hw.igc.smart_pwr_down: 0
hw.igc.rx_abs_int_delay: 66
hw.igc.tx_abs_int_delay: 66
hw.igc.rx_int_delay: 0
hw.igc.tx_int_delay: 66
hw.igc.disable_crc_stripping: 0

A bit more reading indicates igb as being different to igc. If HW is Intel i211, then per latest release notes( https://downloadmirror.intel.com/757201/Release_Notes_27.7.pdf ) mention driver as being igb (page 11).

But then shouldn't systcl have hw.igb as parameters one could tune?

10

General Discussion / inodes /boot/efi full

« on: October 23, 2022, 08:15:47 am »

Noob here (so pardon the glaringly obvious snafus)

Recent purchaser of protectli VPN2410 running OPNSense 22.7.6. Only add-on package, other than defaults, is `os-netstat`. In sifting through the system logs  (via netstat GUI), a critical error flashed.

Code: [Select]

:~ # df -ih

Filesystem            Size    Used   Avail Capacity iused ifree %iused  Mounted on

zroot/ROOT/default    442G    946M    441G     0%     59k  925M    0%   /

devfs                 1.0K    1.0K      0B   100%       0     0  100%   /dev

/dev/ada0p1           260M    1.8M    258M     1%     512     0  100%   /boot/efi

zroot/tmp             441G    684K    441G     0%      35  925M    0%   /tmp

zroot/var/log         441G     40M    441G     0%     107  925M    0%   /var/log

zroot/var/audit       441G     96K    441G     0%       9  925M    0%   /var/audit

zroot                 441G     96K    441G     0%       7  925M    0%   /zroot

zroot/usr/src         441G     96K    441G     0%       7  925M    0%   /usr/src

zroot/var/crash       441G     96K    441G     0%       7  925M    0%   /var/crash

zroot/usr/ports       441G     96K    441G     0%       7  925M    0%   /usr/ports

zroot/usr/home        441G     96K    441G     0%       7  925M    0%   /usr/home

zroot/var/mail        441G    128K    441G     0%      11  925M    0%   /var/mail

zroot/var/tmp         441G     96K    441G     0%       8  925M    0%   /var/tmp

devfs                 1.0K    1.0K      0B   100%       0     0  100%   /var/dhcpd/dev

devfs                 1.0K    1.0K      0B   100%       0     0  100%   /var/unbound/dev

Previous searches on similar issue revealed a few hints (in context of opnsense nano installation). What would be the resize inodes on /boot/efi partition?