How to Fix ERR_CERT_AUTHORITY_INVALID on GrapheneOS Device

[isaacthekind]

I recently switched my phone to use the secure operating system GrapheneOS ( https://grapheneos.org/ ). However, now when I try to browse with my phone on my local network, I get ERR_CERT_AUTHORITY_INVALID (pictured). I have tried:

- restarting DHCP
- deleting the lease
- rebooting the phone
- clearing the browser cache

If someone could give me some tips to debug this, that would be much appreciated.

[schnipp]

Your phone either does not know the CA which has signed the x.509 certificate of "www.startpage.com" or somebody is intercepting the https connection (opnsense with TLS interception enabled or somebody doing MitM).

[isaacthekind]

I see. Well I have not configured SSL interception so I assume it is the phone not knowing the CA. I included a picture of what I see when I do "view certificate" on the phone. Any idea how I can resolve this?

[meyergru]

Obviously www.startpage.com loops back to your OpnSense. That is why your phone tells you that the certificate does not even match the called URL, let alone if it accepts the issuing CA.

It seems you have a misconfigured DNS.

It appears that the URL your phone calls is https://connectivitycheck.grapheneos.network/generate_204 for whatever reason and that points back to your OpnSense... if you call that URL, there is some hints about what is happening. However, apart from a DNS misconfiguration (or IPv6 misconfigured), this seems to be on GrapheneOS and not on OpnSense.

[isaacthekind]

Sorry if my question is frustrating. I don't doubt that this is obvious to you as an experienced user, but I am just learning, I only recently became interested in security, and just learned about DNS for first time maybe 2 weeks ago in networking class, so I'm a total noob (though I do have programming background). I'm trying my best to understand, really not trying to waste anyone's time.

My DNS servers are set to 8.8.8.8 and 8.8.4.4, which are Google's servers. I'm not sure what would be misconfigured about that. Could you give me a sense of what you are referring to when you say DNS is misconfigured?

EDIT: I see you edited your comment after I posted to provide some more info. I will show this to the GrapheneOS folks.

[isaacthekind]

They seem to be insisting it is not GrapheneOS. I really am at a loss here.

[meyergru]

What do you get when you try to resolve any of the DNS entries? Can you ping those names?

Is startpage.com the only site that does not work? What do your other network clients do? There are specific settings for GrapheneOS you can try that are mentioned under https://grapheneos.org/faq#default-connections, for example disable those connectivity checks.

Have you tried if IPv4 and IPv6 is working for you? See https://ipv6-test.com/ and / or https://test-ipv6.com/

There are many ways you can have your DNS misconfigured. For example, if you use a local resolver that uses Google DNS externally, but you chose to use "network" as your local domain - in that case, all requests for connectivitycheck.grapheneos.network could go wrong...

[isaacthekind]

Replying to all those in order here:

When I ping them I see what I would expect, packets are making the round trip (ping.png).

When I resolve them I see what i would expect, "dns.google" (resolve.png).

No external pages work, all give the same error except pages internal to my network, like the OPNsense GUI or my Nextcloud instance (accessed with local IPs).

All clients fail: email, browser, Signal, app store.

I have, at the request of the GrapheneOS people tried disabling connectivity checks, changing the check provider, turning off per-session MAC address generation. None of these had any effect, at which point they concluded that it is not GrapheneOS (I'm not sure if they are correct about this, this just seemed to be what they were saying.).

IPV4 is working, IPV6 is not set up.

Regarding your last paragraph about choosing "network" as the local domain, maybe this is the issue, but I am not sure i understand. Right now domain is "localdomain" (domain.png), maybe you or someone else will see something that looks off about my setup.

[isaacthekind]

I don't think it included my DNS configuration photo, so I will just post that here.

[meyergru]

What I meant was: what do other devices that do not use GrapheneOS do? If you device/OS gets it wrong, it is self-explanatory that all apps fail on that device.

Up until now I do not understand if anything on your network works in that it has:

a. DNS resolution
b. ping connectivity
c. can browse at all
d. can browse with HTTPS

You will have to break the problem down. Currently, it sure looks like you have some kind of redirection to your OpnSense, possible some kind of portwarding gone wrong, if it only affects HTTPS?

[isaacthekind]

Oh, I see. All other devices can do a, b, c, and d. I'll include a sketch of my network topology (topology.png).

My server hosts a Nextcloud instance, and I had to enable port forwarding for that (port-forward.png), some rules as well because I proxy through Cloudflare to obscure my IP (rules.png) and some settings to avoid DNS rebind warnings (settings.png).

[isaacthekind]

Sorry, I forgot my laptop in that image, I've linked a corrected topology image.

[meyergru]

I guess it is the combination of port-forwarding and proxying through cloudflare combined with reflection enabled. Somehow your web traffic gets redirected to OpnSense itself, which is definitely what you see. I would try to disable that one by one and go from there.

On the other hand, the smartphone is behind an OpenWRT AP, which can also "bend" things around. Does your laptop have the same problems?

So, it can be one of three things:

1. Some kind of weird behaviour that only GrapheneOS triggers
2. Something wrong with your AP, maybe it is configured as a router instead of a bridge?
3. Some misconfiguration of port-forwarding and / or proxying through Cloudflare

I would try to disable the Cloudflare proxy first.

For a newbie, you are stacking up quite a bit of complicated things...

[isaacthekind]

I'm glad we have clarity on what the issue is, even though the cause is still mysterious.

Nope, the laptop works as expected, so I do not think it is OpenWRT.

As for the 3 potential causes you mentioned:

1. It could certainly be something weird about GrapheneOS, though I have no idea what that would be or how to find out. I have tried messing with just about every setting in GrapheneOS at this point.

2. I don't think OpenWRT has a binary router/bridge switch like more simple firmware does, if there is a specific router type behaviour that may be the problem, I could check for it.

3. I have a hard time seeing how it could have to do with the port forwarding or Cloudflare proxying, because those are solely for my server. And you can see in the topology image that the server is not between the phone and the gateway. It could have to do with reflection, I do not actually understand reflection, I just knew that enabling it resolved the DNS rebind trouble I was having, and had planned to circle back to it later to build a proper understanding.

As for stacking up complicated tech. I agree, but I do have some background (4th year CS, not general computer noob, just networking noob). And I have a hard time not being a perfectionist, I would rather just start with the best tools and deal with the learning curve. I did this with Haskell for ex, and that was painful but was probably the best decision I made in my programming history, now I use it for everything and love it. Generally speaking everything is going fine with these networking tools, I just have hit a real challenge with this GrapheneOS/OPNsense interaction stuff.

[meyergru]

Another thing: you have your OpnSense UI running on port 443, the same port that you port-forward to your internal server. I tend to use another port for this, as this may have adverse side-effects with NAT and service binding.

I usually use HAproxy's SNI frontend on port 443 and use that to proxy my internal servers by name with ACME certificates.