Messages

Hi @fright

DNS is set correctly and propagated.

Hi,

Same issue here on:

OPNsense 21.10.1-amd64
FreeBSD 12.1-RELEASE-p21-HBSD
OpenSSL 1.1.1l 24 Aug 2021
ACME Client plugin:    3.4

During cert renewal the timeout causes all connections to be dropped. So there seem to be two issues. Cert not properly renewing and connections being dropped during the process.

ACME log:
2021-12-19T11:11:29   acme.sh[44099]   Can not init api for: https://acme-v02.api.letsencrypt.org/directory.
2021-12-19T11:11:28   acme.sh[58460]   Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 6
2021-12-19T11:11:10   acme.sh[68125]   Sleep 10 and retry.
2021-12-19T11:11:10   acme.sh[36103]   Can not init api for: https://acme-v02.api.letsencrypt.org/directory.
2021-12-19T11:11:09   acme.sh[2756]   Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 6
2021-12-19T11:10:51   acme.sh[76135]   Sleep 10 and retry.

System Log:
2021-12-19T03:07:21   opnsense-business[73486]   AcmeClient: validation for certificate failed: REDACTED
2021-12-19T03:07:21   opnsense-business[73486]   AcmeClient: domain validation failed (http01)
2021-12-19T03:00:00   opnsense-business[73486]   AcmeClient: using challenge type: REDACTED
2021-12-19T03:00:00   opnsense-business[73486]   AcmeClient: using IPv4 address: REDACTED
2021-12-19T03:00:00   opnsense-business[73486]   AcmeClient: using IPv4 address: REDACTED
2021-12-19T03:00:00   opnsense-business[73486]   AcmeClient: account is registered: REDACTED
2021-12-19T03:00:00   opnsense-business[73486]   AcmeClient: using CA: letsencrypt
2021-12-19T03:00:00   opnsense-business[73486]   AcmeClient: renew certificate: REDACTED
2021-12-19T03:00:00   opnsense-business[73486]   AcmeClient: certificate must be issued/renewed: REDACTED

This seems to be the reason it drops the connection every night (it tries to renew the cert). It happens both when trying to manually renew or via cron. 

After a bit more searching I found this Reddit post https://www.reddit.com/r/PFSENSE/comments/gxzs42/review_of_netgear_lb2120_4g_lte_and_pfsense/

It states "Netgear created a non-standard TCP/IP implementation. For example, mobile ISP sends DHCP IP of 110.65.12.76, gateway of 110.65.12.1, subnet mask of /24. Netgear then modifies the subnet mask to /32 !!"

I think this is the problem! I changed my subnet with supersede subnet-mask option in the Option modifiers field of the advanced setting of the interface DHCP settings. Let's wait 24 hours and see :)

Don't know if it helps you but my PPPoE connection only started working when I applied mss clamping.

So setting my MTU on the physical interface to 1508, so the PPPoE tunnel gets an MTU of 1500 (according to RFC 4638) and applying MSS clamping 1448 made everything work great. Dropped CPU load as well.

Hi,

I recently bought a Netgear LB2120 in order to have a simple way to have a backup internet connection. I got everything working. The Netgear is in bridgemode and the WAN4G interface I created on the OPNsense is getting a public IP, is able to ping and I've been able to add it to a gatewaygroup. However once a day the gateway fails. Turning on and of the interface fixes the problem.

First i thought this had to do withd DHCP lease time so I've been tinkering with that by adding dhcp-lease-time 14400 to DHCP client configuration -> Advanced -> Option Modifiers at the bottom of the interface settings. Tinkering around with this did not help. I also unchecked Disable State Killing on Gateway Failure in the advanced firewall settings.

Then I stumbled onto this (I'm using a T-mobile SIM) https://community.t-mobile.nl/4g-voor-thuis-568/faq-4g-voor-thuis-technische-vragen-269795.

They state on the forum that every 24 hours T-mobile gives a physical/datalayer disconnect. "The WAN DHCP client should be able to detect such a disconnect and request a new IP immediately." This seems consistent with the problems I'm experiencing. Can anyone give me some pointers on how to make OPNsense request a new IP? Or switch the interface off an on again in case of failure?

Cheers! 

Thanks for posting the solution after you found it! This (hopefully) saved my but! :)

Wanted to thank you! Solved this solved an issue on my 1Gb fiber link , which experienced packet loss when under heavy load. :)

The update to LibreSSL 3.3.5 fixed the problem for me. Both emailing with Monit and Nextcloud backups (both with lets encrypt certs) now get accepted again.

Does anyone have similar problems? Or maybe even someone who has fixed them by now?

Since the router in my case (initiating email or connection to Nextcloud) is the client this should have nothing to do with the certificate, but with the CA's marked as trusted by the OS right? So shouldn't this be an update of the OS (BSD)? Maybe it's handling the cross signed versions of the CA wrong or something?

[cross-signed]

I've followed these steps and my webgui cert works like a charm, I'm also able to check for updates.

let me guess..
added cross-signed ISRG Root X1 cert to Trusted?)

Kind of. I did delete the old DST Root CA X3 (called R3 (Let's Encrypt) from Trusted and recreated all the certificates.
This then seems to have added the ISRG Root X1 (called R3 (ACME Client). I removed this and recreated all of them again. This solved the issues.
I can update from mirrors with LE certs, DoT is working again with uncensoreddns.org and the openssl s_client also verifies the LE certs again.

Thanks
KH

But I'm having other problems that still persist.

I'm on:
OPNsense 21.7.3_3-amd64
FreeBSD 12.1-RELEASE-p20-HBSD
LibreSSL 3.3.4

Monit can't send any more emails:

2021-10-01T18:11:49   monit[38088]   Aborting event   
2021-10-01T18:11:49   monit[38088]   SMTP: Error sending data to the mailserver -- No error: 0   
2021-10-01T18:11:49   monit[38088]   SSL: write error -- error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed   
2021-10-01T18:11:49   monit[38088]   Mail: SSL server certificate verification error: certificate has expired

And Nextcloud backups also stopped working.

2021-10-02T10:34:39   php-cgi[17483]   {"url":"https:\/\/DOMAINREPLACEDFORPRIVACY\/ocs\/v1.php\/cloud\/user","content_type":null,"http_code":0,"header_size":0,"request_size":0,"filetime":-1,"ssl_verify_result":10,"redirect_count":0,"total_time":0.081055,"namelookup_time":0.061367,"connect_time":0.066082,"pretransfer_time":0,"size_upload":0,"size_download":0,"speed_download":0,"speed_upload":0,"download_content_length":-1,"upload_content_length":-1,"starttransfer_time":0,"redirect_time":0,"redirect_url":"","primary_ip":"IPRELACEDFORPRIVACY","certinfo":[],"primary_port":443,"local_ip":"45.83.233.8","local_port":60733,"http_version":0,"protocol":2,"ssl_verifyresult":0,"scheme":"HTTPS","appconnect_time_us":0,"connect_time_us":66082,"namelookup_time_us":61367,"pretransfer_time_us":0,"redirect_time_us":0,"starttransfer_time_us":0,"total_time_us":81055}   
2021-10-02T10:34:39   php-cgi[17483]   Cannot get real username


The Nextcloud server has a let's encrypt certificate but it's not signed with the old CA.


Any suggestions?

RADVD seems to try and address the loopback interface, but the loopbackinterface is called lo0 not lo. I can't seem to find where in the config of radvd this is configured. The search continues

Hi Patrick,

IPv6 works on all internal interfaces, so I don't think using that option will help. I increased radvd debuglevel to 5  and this is what it reports:


2021-09-30T20:20:28   radvd[40069]   polling for 10 second(s), next iface is lo   
2021-09-30T20:20:28   radvd[40069]   lo next scheduled RA in 10 second(s)   
2021-09-30T20:20:28   radvd[40069]   send_ra_forall failed on interface lo   
2021-09-30T20:20:28   radvd[40069]   not sending RA for lo, interface is not ready   
2021-09-30T20:20:28   radvd[40069]   lo not found: Device not configured   
2021-09-30T20:20:28   radvd[40069]   timer_handler called for lo   
2021-09-30T20:20:18   radvd[40069]   polling for 10 second(s), next iface is lo   
2021-09-30T20:20:18   radvd[40069]   lo next scheduled RA in 10 second(s)   
2021-09-30T20:20:18   radvd[40069]   send_ra_forall failed on interface lo   
2021-09-30T20:20:18   radvd[40069]   not sending RA for lo, interface is not ready   
2021-09-30T20:20:18   radvd[40069]   lo not found: Device not configured   
2021-09-30T20:20:18   radvd[40069]   timer_handler called for lo   
2021-09-30T20:20:08   radvd[40069]   polling for 10 second(s), next iface is lo   
2021-09-30T20:20:08   radvd[40069]   lo next scheduled RA in 10 second(s)   
2021-09-30T20:20:08   radvd[40069]   send_ra_forall failed on interface lo   
2021-09-30T20:20:08   radvd[40069]   not sending RA for lo, interface is not ready   
2021-09-30T20:20:08   radvd[40069]   lo not found: Device not configured
2021-09-30T20:20:08   radvd[40069]   timer_handler called for lo   
2021-09-30T20:19:58   radvd[40069]   polling for 10 second(s), next iface is lo   
2021-09-30T20:19:58   radvd[40069]   lo next scheduled RA in 10 second(s)   
2021-09-30T20:19:58   radvd[40069]   send_ra_forall failed on interface lo   
2021-09-30T20:19:58   radvd[40069]   not sending RA for lo, interface is not ready   
2021-09-30T20:19:58   radvd[40069]   lo not found: Device not configured   
2021-09-30T20:19:58   radvd[40069]   timer_handler called for lo   
2021-09-30T20:19:58   radvd[40069]   polling for 0 second(s), next iface is lo   
2021-09-30T20:19:58   radvd[40069]   interface lo does not exist or is not set up properly, ignoring the interface   
2021-09-30T20:19:58   radvd[40069]   lo not found: Device not configured   
2021-09-30T20:19:58   radvd[40069]   validated pid file, /var/run/radvd.pid: 40069


Any suggestions?

If I understand correctly all the rules mentioned in the RFC https://datatracker.ietf.org/doc/html/rfc4890 are taken care of automagically by OPNsense. I found this post on the Netgate forum discussing this https://forum.netgate.com/topic/138243/2-4-4-icmpv6-firewall-rules. They mention everything should work out of the box (except for Echo requests).

The above seem to correspond with my experience. Since everything seems to be working correctly, except RADVD telling me that the network is down. How does it check this?

Since I receive a /48 via DHCPv6 through the IPv4 PPPoE tunnel, my WAN interface and  IPv6 default gateway both are a link local address and do not have their own IPv6 address. Could this be the problem? Because of this IPv6 gateway monitoring also can't be enabled as far as I understand (and would be not very useful anyway).

Hi Patrick,

I have not disabled them. These are the rules that are automagically generated on the same internal example interface:


These are the floating rules generated:


As mentioned IPv6 seems to be working correctly (at least according to all the ipv6 tests on the web I've tried). The only problem is RADVD stating the network is down while it clearly is not (and me being a noob when it comes to IPv6). ;)