"Recent posts
#1
26.7, 26.10 Series / Re: 26.7.7: Lark HTTPS connect...
Last post by Brilliant Technologies Lt - Today at 09:18:09 AMthans！
#2
26.7, 26.10 Series / Re: 26.7.7: Lark HTTPS connect...
Last post by Brilliant Technologies Lt - Today at 09:17:23 AMthans！
#3
26.7, 26.10 Series / Re: 26.7.7: Lark HTTPS connect...
Last post by Brilliant Technologies Lt - Today at 09:16:53 AMthans！
#4
26.7, 26.10 Series / Re: 26.7.7: Lark HTTPS connect...
Last post by Brilliant Technologies Lt - Today at 09:16:36 AMthans！
#5
26.7, 26.10 Series / Re: 26.7.7: Lark HTTPS connect...
Last post by Brilliant Technologies Lt - Today at 09:10:23 AMUpdate: the blocking mechanism was traced to CrowdSec's blocklist enforcement. Sharing the findings and the targeted workaround for anyone encountering similar symptoms.
Thank you, pfry, for explaining the state/TCP-flags checks, and alveston for the advice to compare builds and configurations before rolling back. Both were useful suggestions.
What we found:
1. oa.larksuite.com resolved to 95.100.110.20 and 95.100.110.24. Client TCP/443 connections to both timed out before TLS.
2. Interfaces > Diagnostics > Port Probe, run from OPNsense itself toward 95.100.110.20:443 with its WAN source address, returned 'Permission denied'.
3. Firewall > Diagnostics > Aliases > Find references showed that BOTH destination addresses belonged to crowdsec_blocklists. The ordinary search box only searches the currently selected table; Find references was the useful check here.
4. Inspecting the automatically generated firewall rules showed CrowdSec IPv4/IPv6 inbound and outbound block rules with quick matching. The IPv4 outbound rule blocked destinations in crowdsec_blocklists. The normal LAN allow rule therefore did not provide an exception to this separate automatic block.
The action taken was to add only these two addresses to a local CrowdSec allowlist, keeping CrowdSec protection for other addresses. On this installation the following commands were supported and executed successfully in the OPNsense root shell:
The add command reported 'added 2 values to allowlist lark_cdn' and '2 decisions deleted by allowlists'. Subsequent targeted deletion commands returned zero because the decisions had already been removed:
Allow time for the remediation component/bouncer to synchronize, then check connectivity and the application. There is no need to clear the entire blocklist or disable all firewall filtering. If the allowlist already exists, skip its creation. Consult the CrowdSec documentation for your installed version:
https://docs.crowdsec.net/docs/local_api/centralized_allowlists/
Important qualifications:
- This identifies active CrowdSec bans and matching enforcement rules; it does not establish why those IPs were banned. The original decision origin/scenario was not recorded before removal. I cannot say whether these came from community intelligence or a local detection.
- There is no evidence here establishing an OPNsense 26.7.7 regression. The upgrade initially appeared related because of the timing.
- Suricata was in IDS mode; its alert-only operation should not have been conflated with CrowdSec's separate blocking component.
- Correction to my initial report: the client's address was initially misidentified, so the earlier state-violation log attributed to that client is not reliable evidence for this connection. The direct blocklist and automatic-rule checks above are the stronger evidence.
- These are the CDN addresses observed at the time, not a universal or permanent Lark IP list. An IP allowlist trusts the whole IP, potentially including other tenants of a shared CDN. Use narrowly scoped exceptions, review them, and check current resolved addresses rather than blindly copying these into unrelated installations.
The application also briefly recovered and failed again before this investigation; the reason for that intermittent behavior was not independently established. The decisive finding was the two CrowdSec decisions, rather than a speculative state reset.
Thanks again to everyone who took the time to help. I hope these checks save others time when a default LAN allow rule is present but connections are still blocked by an automatically managed security rule.
Thank you, pfry, for explaining the state/TCP-flags checks, and alveston for the advice to compare builds and configurations before rolling back. Both were useful suggestions.
What we found:
1. oa.larksuite.com resolved to 95.100.110.20 and 95.100.110.24. Client TCP/443 connections to both timed out before TLS.
2. Interfaces > Diagnostics > Port Probe, run from OPNsense itself toward 95.100.110.20:443 with its WAN source address, returned 'Permission denied'.
3. Firewall > Diagnostics > Aliases > Find references showed that BOTH destination addresses belonged to crowdsec_blocklists. The ordinary search box only searches the currently selected table; Find references was the useful check here.
4. Inspecting the automatically generated firewall rules showed CrowdSec IPv4/IPv6 inbound and outbound block rules with quick matching. The IPv4 outbound rule blocked destinations in crowdsec_blocklists. The normal LAN allow rule therefore did not provide an exception to this separate automatic block.
The action taken was to add only these two addresses to a local CrowdSec allowlist, keeping CrowdSec protection for other addresses. On this installation the following commands were supported and executed successfully in the OPNsense root shell:
Code Select
cscli allowlists create lark_cdn -d "Lark CDN exception"
cscli allowlists add lark_cdn 95.100.110.20 95.100.110.24
cscli allowlists inspect lark_cdn
The add command reported 'added 2 values to allowlist lark_cdn' and '2 decisions deleted by allowlists'. Subsequent targeted deletion commands returned zero because the decisions had already been removed:
Code Select
cscli decisions delete --ip 95.100.110.20
cscli decisions delete --ip 95.100.110.24
Allow time for the remediation component/bouncer to synchronize, then check connectivity and the application. There is no need to clear the entire blocklist or disable all firewall filtering. If the allowlist already exists, skip its creation. Consult the CrowdSec documentation for your installed version:
https://docs.crowdsec.net/docs/local_api/centralized_allowlists/
Important qualifications:
- This identifies active CrowdSec bans and matching enforcement rules; it does not establish why those IPs were banned. The original decision origin/scenario was not recorded before removal. I cannot say whether these came from community intelligence or a local detection.
- There is no evidence here establishing an OPNsense 26.7.7 regression. The upgrade initially appeared related because of the timing.
- Suricata was in IDS mode; its alert-only operation should not have been conflated with CrowdSec's separate blocking component.
- Correction to my initial report: the client's address was initially misidentified, so the earlier state-violation log attributed to that client is not reliable evidence for this connection. The direct blocklist and automatic-rule checks above are the stronger evidence.
- These are the CDN addresses observed at the time, not a universal or permanent Lark IP list. An IP allowlist trusts the whole IP, potentially including other tenants of a shared CDN. Use narrowly scoped exceptions, review them, and check current resolved addresses rather than blindly copying these into unrelated installations.
The application also briefly recovered and failed again before this investigation; the reason for that intermittent behavior was not independently established. The decisive finding was the two CrowdSec decisions, rather than a speculative state reset.
Thanks again to everyone who took the time to help. I hope these checks save others time when a default LAN allow rule is present but connections are still blocked by an automatically managed security rule.
#6
26.7, 26.10 Series / Re: [Solved by reboot] 26.7 Up...
Last post by franco - Today at 08:53:23 AMBack then I think what fetch needed was a limit to the retries so escape the vicious cycle, but such an option doesn't exist. I trade a ticket for investigating if that would make sense: then we could re-introduce -a but limit it to e.g. 2-3 retries max.
Cheers,
Franco
Cheers,
Franco
#7
German - Deutsch / Re: WireGuard-Tunnel ohne Inte...
Last post by Billie34 - Today at 04:17:12 AMBesonders interessant ist, dass der WireGuard-Handshake funktioniert und der Internetzugang unmittelbar nach einem manuellen Firewall-Reload wiederhergestellt wird.
#8
General Discussion / Newbie Tries AGH + Unbound
Last post by goofyman23 - Today at 01:50:31 AMNot too long ago, I got OPNSense setup and was able to replace my aging ASUS router. I've been happy overall, but overwhelmed at times. I've got AdGuard Home running for filtering/blocklists but recently enabled Unbound for another layer of privacy.
I knew going in that DNS lookups would be slow, but after ~48 hours, I think the performance is slower than what it should be. This is where I'm looking for input and advice. Perhaps I'm impatient and need to wait it out more to see if it normalizes.
AGH is reporting average processing times of ~200-300ms and ~500-700ms for upstream processing.
Unbound Statistics I think look good considering the amount of cold caches (see attached). But, at the same time I see recursive lookups of 2000-4000ms in Reports > Unbound.
AGH forwards everything to Unbound for DNS and Unbound is the only upstream server.
About the only things I changed within Unbound are:
Enabled Prefetch DNS Key Support, Hide Identity/Version, and enabled DNSSEC Support
DHCP is done via DNSMasq and not AGH.
I knew going in that DNS lookups would be slow, but after ~48 hours, I think the performance is slower than what it should be. This is where I'm looking for input and advice. Perhaps I'm impatient and need to wait it out more to see if it normalizes.
AGH is reporting average processing times of ~200-300ms and ~500-700ms for upstream processing.
Unbound Statistics I think look good considering the amount of cold caches (see attached). But, at the same time I see recursive lookups of 2000-4000ms in Reports > Unbound.
AGH forwards everything to Unbound for DNS and Unbound is the only upstream server.
About the only things I changed within Unbound are:
Enabled Prefetch DNS Key Support, Hide Identity/Version, and enabled DNSSEC Support
DHCP is done via DNSMasq and not AGH.
#9
Tutorials and FAQs / Re: [HowTo] OPNsense Boot Load...
Last post by nero355 - Today at 12:32:53 AMQuote from: Lucid1010 on November 03, 2026, 08:04:47 PMHowever, what is the reason OPNsense itself doesn't update the bootloader when performing a firmware upgrade?Because FreeBSD doesn't so OPNsense doesn't too :)
IIRC the reason was something like "Too many variables involved that increase the chance for a unootable system afterwards if things go wrong" or so...
I will look it up and add the explanation in a future update! ;)
#10
26.7, 26.10 Series / Re: [Solved by reboot] 26.7 Up...
Last post by danmac - Today at 12:04:40 AMQuote from: franco on November 03, 2026, 09:58:10 AMSo -a has a history for us: https://github.com/opnsense/update/issues/90Blimey lol :/ thanks for the link, I also skimmed 7825
So the danger is that fetch -a can retry forever getting nowhere. Also I see some confusion regarding the -T flag, as we discussed earlier I thought it was connect timeout only ... (?)
I can understand the difficulty of devising a solution that makes everybody happy, with their own little networks broken in their own little ways ;)
If it's on the roadmap to switch to curl anyway, it occurs to me I'm in danger of wasting your time with this ... although as previously noted, curl failed in just the same way, at least for me. curl does seem to have more levers to influence the retry logic though, eg. --retry, --retry-max-time, --connect-timeout, --max-time, etc
Quote from: franco on November 03, 2026, 09:58:10 AMI also don't quite get what's happening here:
Downloading kernel-26.7.8-amd64.txz with remote size 33253768:
kernel-26.7.8-amd64.txz 88% of 31 MB 691 kBps 05s
fetch: kernel-26.7.8-amd64.txz appears to be truncated: 29328007/33253768 bytes
kernel-26.7.8-amd64.txz 31 MB 634 kBps 06s
Verifying signature with trusted certificate pkg.opnsense.org.20260608... done
It's truncated and complained loudly, then soft-resumed at the end of the transfer or started anew? And then it downloads all and verifies ok? Why doesn't it work the first time? This isn't a client-side riddle IMO.
Yeah it just resumes the remaining portion of the transfer after it hits the unceremonious yet highly predictable SSL EOF - I end up with everything fully downloaded and verified.
I still can't help but suspect these woes are (P)MTU / MSS related. Firmware updates seem to be a particular pain-point with 5G, and the guy in issue 90 had some wacky VPN + proxy setup. (on that note I wonder if the fetch -T timeout is satisfied by the connection to the proxy rather than actually contacting the mirror) May also explain why some say changing mirrors helps. I never had this problem on the exact same firewall via VDSL (1500 MTU, ethernet @ 1508 MTU to a Draytek modem in bridge mode, OPNsense handling PPPoE) it's only an issue now I'm on 5G. (1380 MTU)
I had the AI waifu check my MTU configs (at least in theory) last night, bit of a long story, funny to see its "thinking" loops similar to my wetware when I'm handling MTU calculations haha :/ "hmmm ... wait ... let me recompute cleanly ..." xD
Maybe in amongst these ramblings (sorry) will be a clue or two for someone down the line. I'm still new to 5G and it was installed in haste/anger on an 8+ year old somewhat complicated install so I should probably return with a fresh install and wireshark, et al. I'll even stop ignoring the new firewall rules page ;)
o7 thx, appreciate your time Franco :)
