Weird search engine behavior after upgrade to 25.7.6

[Jyling]

Late last night, when I just finished setting everything up on the router, google.com began to behave erratically: throwing security errors for a few minutes, then working fine, and then again, but Bing and duck duck worked.
This morning, both google and bing are throwing security and DNS error. Duck is still working.
As I was typing this, Bing stopped throwing DNS errors and began to work. When it was not, this was its info:

;; AUTHORITY SECTION:
bing.com.        300    IN    SOA    a1-169.akam.net. hostmaster.bing.com. 2023042200 3600 600 604800 300

Now I got this instead:

;; ANSWER SECTION:
bing.com.        3266    IN    A    150.171.27.10
bing.com.        3266    IN    A    150.171.28.10

Google is still throwing cert errors:

Did Not Connect: Potential Security Issue
Error code: SSL_ERROR_BAD_CERT_DOMAIN

Checked their cert, looks legit:

Subject Name
Common Name dns.google
Issuer Name
Country US
Organization Google Trust Services
Common Name WR2
Miscellaneous
Serial Number 00:F8:D2:A5:84:7A:02:30:82:09:EA:36:F3:65:9C:34:CE
Signature Algorithm SHA-256 with RSA Encryption
Version 3
Fingerprints
SHA-256 0C:8F:8E:73:EF:9B:12:C7:CA:48:29:CA:F7:59:D0:18:85:9C:E4:4D:26:AE:D3:51:3D:AF:2C:A5:A1:75:4A:9F
SHA-1 CE:DA:C1:03:FC:8C:63:AB:6F:CC:78:F9:B1:DF:81:06:7A:7A:42:0C
Subject Key ID
Key ID 7F:10:2B:94:96:4B:89:F2:EF:59:55:F3:EA:DA:49:DB:77:32:4C:43
Authority Key ID
Key ID DE:1B:1E:ED:79:15:D4:3E:37:24:C3:21:BB:EC:34:39:6D:42:B2:30

Is it my SNAFU or are they and/or Akamai really having issues?
Checked my time. It's synched to the router, and there it's the correct local time.

UPDATE: 30 seconds later, Bing is throwing DNS errors again, but the dig for it remains the same:

;; ANSWER SECTION:
bing.com.        2997    IN    A    150.171.27.10
bing.com.        2997    IN    A    150.171.28.10

UPDATE 2: A couple minutes later, google began to work. For how long??? Their cert looks different now:

Public Key Info
Algorithm Elliptic Curve
Key Size 256
Public Value
04:35:32:C8:31:9D:E4:74:E2:C7:CC:16:22:4E:E3:76:BA:FE:C2:8A:FA:BC:77:57:6B:05:5F:9D:17:A4:2B:7A:58:01:A9:FC:98:60:00:98:17:46:A2:14:0D:F4:5C:18:1A:BD:B6:CE:F1:D8:B6:5A:E3:FD:14:5E:65:F7:DB:73:5E
Miscellaneous
Serial Number 29:0A:39:E4:5E:99:D7:6F:0A:3D:47:74:14:0A:3A:FF
Signature Algorithm SHA-256 with RSA Encryption
Version 3
Fingerprints
SHA-256 B7:D9:C5:14:9E:24:2A:E0:DE:23:41:E5:B0:F0:10:5E:EB:C3:66:B3:4F:D9:14:ED:52:6C:5D:D0:7D:FC:9F:A1
SHA-1 21:9C:5C:29:E1:79:B5:ED:56:03:5A:DB:A7:59:E9:CD:8C:F7:7B:84

[Jyling]

Since the yesterday's morning, google.com and bing.com were intermittently on and off available or not, but in the afternoon they mostly worked.
This morning, again google is failing:

Error code: SSL_ERROR_BAD_CERT_DOMAIN

There is no cert at all, plain HTTP. There is nothing in the news on Bing. Am I the only one experiencing this? If I am, then this is pointing at the router, is it not? But how could it be possible that only the 2x major search engines and everything that is served from google cloud are affected?

[meyergru]

What did you finish setting up in your router? Some kind of transparent proxy? Crowdsec, suricata, zenarmor or any other kind of blocking or traffic inspection?

If the certificate is not being trusted, it can be either of: you are being transferred to the wrong site because your DNS is malfunctioning or the certificate is manipulated by some kind of man-in-the-middle, like a transparent proxy and you have not installed their CA certificate in your browser.

You could see that by inspecting the certificate and see if it matches the domain you wanted to call. The error message suggests that it is a different domain, so the interesting question is: which is it and why did your DNS request return its IP instead of the correct one?

[Jyling]

None of that.

UPDATE: The plot thickens. I can open Google from one VM but not from LAN PCs or other VMs. I took an old web browser on the VM where it does not work and tried to open google:

The domain list in the cert does not cover their WWW:

www.google.com uses an invalid security certificate.

The certificate is only valid for the following names:
  dns.google, dns.google.com, *.dns.google.com, 8888.google, dns64.dns.google, 8.8.8.8, 8.8.4.4, 2001:4860:4860::8888, 2001:4860:4860::8844, 2001:4860:4860::6464, 2001:4860:4860::64 

(Error code: SSL_ERROR_BAD_CERT_DOMAIN)

But on the VM where it works, the cert is completely different and does cover the WWW:


Common Name www.google.com

How is it possible that different devices on the same LAN intermittently get different certs from google and google only?
All of the LAN devices use open sense as their DNS. All of the VMs have bridged adapters, so they are in the same subnet and for all purposes can be considered LAN devices as well. At my wits end.

[Jyling]

I inserted several images into the https://forum.opnsense.org/index.php?msg=251710 but none of them are showing. The [img] tags are still in the body, when I edit the message. Could the admins look into that because the feeling of wasting my breath is very demeaning.

[Jyling]

The weirdest thing, IMO, is that it is only the browser (any i.e. Chrome, Firefox, Palemoon) that seems to have the problems. Open SSL reports the same cert everywhere.

[pfry]

None of that. [...]

The weirdest thing, IMO, is that it is only the browser (any i.e. Chrome, Firefox, Palemoon) that seems to have the problems. [...]

Browser proxy? Although modern browsers seem to only use the system proxy, at least on Windows. (I thought to check, but none of my Linux or FreeBSD installs has a GUI at the moment.)

[Jyling]

No proxies here. None whatsoever.

[Jyling]

The provider's DNS server was FUBAR. Removed it, and their 2ndary is working fine. So much gray hair, for naught. This is what happens when cheap businesses hire the cheapest labor.

[Patrick M. Hausen]

Don't use your provider's recursive DNS, then. Just run Unbound instead.

[Jyling]

I was too angry and too quick to blame inept sysadmins at my provider. They are still that, but the root cause does not seem to be entirely their fault.
Several hours later, the same problem caught up with me, from that other server, and I realized that the problem is likely Google's own making.
This is what my provider's server returns for google.com, and this does not work:

;; ANSWER SECTION:
google.com.      300   IN   A   192.178.192.101
google.com.      300   IN   A   192.178.192.113
google.com.      300   IN   A   192.178.192.139
google.com.      300   IN   A   192.178.192.100
google.com.      300   IN   A   192.178.192.102
google.com.      300   IN   A   192.178.192.138

And this is what 8.8.8.8 returns for the same query, and this works:

;; ANSWER SECTION:
www.google.com.      252   IN   A   142.250.137.103
www.google.com.      252   IN   A   142.250.137.99
www.google.com.      252   IN   A   142.250.137.106
www.google.com.      252   IN   A   142.250.137.147
www.google.com.      252   IN   A   142.250.137.105
www.google.com.      252   IN   A   142.250.137.104

Google either is not honestly informing ISP's name servers of their own IP addresses, or they do but those addresses are of non-working hosts.
This is either intentional - so as to herd and goad all internet users to the Google's own name server, or this is a SNAFU waiting to be solved.

Since this started, in my particular case, on Nov 1, and today is Nov 4 and this is still not resolved by Google, I call BS. Somebody should've noticed and fixed this SNAFU. Its continuation sounds fishy.

Don't

I've no budget for that. I do what my superiors tell me.
Oh, and by the way, as I posted in https://forum.opnsense.org/index.php?topic=49584.0 Unbound is refusing to work in 25.7.6. I've not been able to resolve this yet.