26.7.7: Lark HTTPS connections time out after upgrade; hotspot works

[Brilliant Technologies Lt]

Hello,

After upgrading OPNsense to 26.7.7, several Lark (larksuite.com) features stopped loading, including Reports and Approvals. They worked before the upgrade. The same Mac and account work on a mobile hotspot but fail through the usual OPNsense network. The root cause has not been established.

Symptoms:
- https://oa.larksuite.com/report/record/entry stays blank and eventually reports a page error. Other Lark features are also affected.
- DNS lookup returns an Akamai CNAME chain and 95.100.110.24 / 95.100.110.20.
- curl -vI --connect-timeout 10 --max-time 20 https://oa.larksuite.com/ resolves both IPv4 addresses, attempts TCP/443 on each, then times out with error 28 before any TLS handshake. No IPv6 address was returned in this curl test.
- No VPN/proxy is enabled on the client. nslookup shows a loopback DNS resolver (127.0.2.2); its providing software has not been identified.

Checks attempted:
- Default IPv4/IPv6 allow rules from LAN network to any are present. Exact LAN subnet membership and routing/NAT topology still need verification.
- Added a temporary TCP/443 pass rule for the affected client to both destination IPs: no improvement.
- Temporarily disabling Suricata was attempted without improvement; independent confirmation that the service fully stopped is not available.
- Rebooted OPNsense: no improvement.
- A packet capture attempt yielded no usable capture, so WAN transmission and return traffic are not yet verified.

Logs:
Live View showed TCP/443 from the affected client to 95.100.110.24 blocked under 'Default deny / state violation rule'. TCP flags and exact correlation with a fresh connection attempt were not captured. Kernel logs also show 'pf: BAD state' and 'pf: loose state match' involving other clients; these are not proven related to this client's Lark failure.

Are there known 26.7.7 PF state, interface, NAT, or forwarding changes matching these symptoms? What minimal diagnostics can distinguish local PF/NAT/routing trouble from upstream connectivity trouble? If this matches a known regression, is there an officially recommended fix or targeted rollback?

The previous working version, full current build suffix, hardware/NIC details, and full topology have not yet been collected and can be supplied if required.

Thank you.

[pfry]

[...]Live View showed TCP/443 from the affected client to 95.100.110.24 blocked under 'Default deny / state violation rule'. TCP flags and exact correlation with a fresh connection attempt were not captured. Kernel logs also show 'pf: BAD state' and 'pf: loose state match' involving other clients; these are not proven related to this client's Lark failure.[...]

Looks like a failed session setup, i.e. no state = timeout. I'd start with the info (specifically TCP flags) from that state violation - it's easiest to view from the "i" in the Live View; it's also in the session log. A loose match may be normal (it's the default); state mismatches can stem from path inconsistencies (bypassing the firewall). Given the loose state default, bypassing the expected interface is generally less of an issue.

[alveston]

It may also help to compare the complete OPNsense build and configuration with the previous working version before attempting a rollback.

[Brilliant Technologies Lt]

Update: the blocking mechanism was traced to CrowdSec's blocklist enforcement. Sharing the findings and the targeted workaround for anyone encountering similar symptoms.

Thank you, pfry, for explaining the state/TCP-flags checks, and alveston for the advice to compare builds and configurations before rolling back. Both were useful suggestions.

What we found:
1. oa.larksuite.com resolved to 95.100.110.20 and 95.100.110.24. Client TCP/443 connections to both timed out before TLS.
2. Interfaces > Diagnostics > Port Probe, run from OPNsense itself toward 95.100.110.20:443 with its WAN source address, returned 'Permission denied'.
3. Firewall > Diagnostics > Aliases > Find references showed that BOTH destination addresses belonged to crowdsec_blocklists. The ordinary search box only searches the currently selected table; Find references was the useful check here.
4. Inspecting the automatically generated firewall rules showed CrowdSec IPv4/IPv6 inbound and outbound block rules with quick matching. The IPv4 outbound rule blocked destinations in crowdsec_blocklists. The normal LAN allow rule therefore did not provide an exception to this separate automatic block.

The action taken was to add only these two addresses to a local CrowdSec allowlist, keeping CrowdSec protection for other addresses. On this installation the following commands were supported and executed successfully in the OPNsense root shell:

Code Select Expand

cscli allowlists create lark_cdn -d "Lark CDN exception"
cscli allowlists add lark_cdn 95.100.110.20 95.100.110.24
cscli allowlists inspect lark_cdn

The add command reported 'added 2 values to allowlist lark_cdn' and '2 decisions deleted by allowlists'. Subsequent targeted deletion commands returned zero because the decisions had already been removed:

Code Select Expand

cscli decisions delete --ip 95.100.110.20
cscli decisions delete --ip 95.100.110.24

Allow time for the remediation component/bouncer to synchronize, then check connectivity and the application. There is no need to clear the entire blocklist or disable all firewall filtering. If the allowlist already exists, skip its creation. Consult the CrowdSec documentation for your installed version:
https://docs.crowdsec.net/docs/local_api/centralized_allowlists/

Important qualifications:
- This identifies active CrowdSec bans and matching enforcement rules; it does not establish why those IPs were banned. The original decision origin/scenario was not recorded before removal. I cannot say whether these came from community intelligence or a local detection.
- There is no evidence here establishing an OPNsense 26.7.7 regression. The upgrade initially appeared related because of the timing.
- Suricata was in IDS mode; its alert-only operation should not have been conflated with CrowdSec's separate blocking component.
- Correction to my initial report: the client's address was initially misidentified, so the earlier state-violation log attributed to that client is not reliable evidence for this connection. The direct blocklist and automatic-rule checks above are the stronger evidence.
- These are the CDN addresses observed at the time, not a universal or permanent Lark IP list. An IP allowlist trusts the whole IP, potentially including other tenants of a shared CDN. Use narrowly scoped exceptions, review them, and check current resolved addresses rather than blindly copying these into unrelated installations.

The application also briefly recovered and failed again before this investigation; the reason for that intermittent behavior was not independently established. The decisive finding was the two CrowdSec decisions, rather than a speculative state reset.

Thanks again to everyone who took the time to help. I hope these checks save others time when a default LAN allow rule is present but connections are still blocked by an automatically managed security rule.

[Brilliant Technologies Lt]

thans！

[Brilliant Technologies Lt]

thans！

[Brilliant Technologies Lt]

thans！

[Brilliant Technologies Lt]

thans！