This isn't the first time I've seen this and haven't figured out the cause yet. Short version is that every few updates to OPNsense, AGH crashes badly and can't be restarted without wiping it out and starting fresh. Trying to track down the cause and resolve it once and for all. I've seen a few people post with similar issues, but they mostly just get told off (ie, "Doesn't happen to me, so it's your problem" type of replies). I acknowledge this isn't happening to all firewalls, but I have two of them doing this...and oddly, they don't always break on the same updates.
Ok, here's the pile of info on what I'm seeing and tested, using today's crash as an example. Previous run-ins with this issue were identical:
- AGH and OPNsense work great for months at a time. Not a single crash, failure, or error.
- AGH is the sole DNS on the firewall. I do not use Unbound.
- When this occurs, it's usually after an update to OPNsense or the AGH container. Either can trigger it, but the results are the same.
- About 2hrs after the update to 24.7.10_1, I began getting alerts from my Uptime Kuma system that AGH wasn't responding. Except...it was when I checked. Appears almost as if the AGH plugin crashed and restarted.
- About 4hrs after the update to 24.7.10_1, I began getting reports that users had sites not loading. If they refresh, it works fine. Again, sounds like AGH crashing and restarting, but my pings to it never show a drop.
- At 5hrs, I got a bunch of panic-calls. The internet was 100% down. In fact, OPNsense had crashed. No ping, but all the lights are still on. Had to hard power-off the system. Unfortunately, this wipes my memory-logs, so I have no idea why OPNsense died. Aside from patch-reboots, it's been running for about 1yr without being fully powered off, so this is new.
- It came back up and AGH won't start. I gave it a bit and it never started. Since there's no AGH logs, I have no idea why. Manual start runs for about 3 seconds, then stops again. These are identical symptoms to the last time this happened about 4mo ago.
- As I was advised last time, I uninstalled AGH and reinstalled, then migrated back the yaml file. Same results. It sometimes fails to start, sometimes just crashes.
- Again, as advised last time, I reinstalled AGH and started with a clean yaml. It starts fine and has no issues or indications of a problem. Last time I had to manually rebuild everything for no good reason. Looks like I need to again.
- Plenty of others have had issues with updates breaking AGH, so there's definitely something going on. However, I don't know what the trigger is or what's actually breaking and would appreciate some advice, guidance, or whatnot. Starting clean with AGH every 2-3 OPNsense updates isn't viable.
- In the interim, Unbound works fine. I'd really prefer to use AGH, but if it's going to eat its own face every few updates, that may not be an option.
Any ideas or things I may not have tried?
Quote from: DenverTech on December 04, 2024, 05:01:42 AM
When this occurs, it's usually after an update to OPNsense or the AGH container. Either can trigger it, but the results are the same.
What container? How did you install AGH on OPNsense?
I used the community plugin from maxit/mimugmail, so it's installed via the plugin. Nothing too fancy or special there.
Out of curiosity, is there any way to log errors with plugins (such as from console)? If I could get a log of what is exploding in AGH, that would make this a lot easier.
there seem to be a couple of hotfixes today for 24.7.10.x. I suggest you retest after system is stable again.
Quote from: DenverTech on December 04, 2024, 07:33:57 AM
Out of curiosity, is there any way to log errors with plugins (such as from console)? If I could get a log of what is exploding in AGH, that would make this a lot easier.
Nothing to find in the /usr/local/AdGuardHome/data directory?
Also, log on to OPNsense via SSH, use drill to specifically throw a request at the AdGuard Home port, observe what happens.
Quote from: Patrick M. Hausen on December 04, 2024, 01:43:49 PM
Quote from: DenverTech on December 04, 2024, 07:33:57 AM
Out of curiosity, is there any way to log errors with plugins (such as from console)? If I could get a log of what is exploding in AGH, that would make this a lot easier.
Nothing to find in the /usr/local/AdGuardHome/data directory?
Also, log on to OPNsense via SSH, use drill to specifically throw a request at the AdGuard Home port, observe what happens.
I checked the AGH folder, including data, but all it has is the query log. I ran queries and only the query log changes. Nothing appears to be an "informational" log.
Quote from: cookiemonster on December 04, 2024, 01:33:50 PM
there seem to be a couple of hotfixes today for 24.7.10.x. I suggest you retest after system is stable again.
Good point. I've been watching those. They don't really describe the issue I'm having, but yeah, good call.
For AdGH logging specifically there's another way. I can't really remember how long ago I did this modification but a few AdGH versions ago they introduced some additional logging capabilities by config. I update AdGH from time to time using the UI. The installation is from mimugmail's repo.
Please if you do it, check you have a version that has this capability and take notice of the schema changes. Make a backup of the config file: /usr/local/AdGuardHome/AdGuardHome.yaml before any changes.
I am on Version: v0.107.54 of AdGH.
On my config I have added/changed the following settings in the log section:
log:
enabled: true
file: /var/log/AdGuardHome/AdGuardHome.log
max_backups: 30
max_size: 10
max_age: 31
compress: false
local_time: false
verbose: false
This is very recently, before I had:
log:
enabled: true
file: /var/log/AdGuardHome/AdGuardHome.log
max_backups: 3
max_size: 100
max_age: 3
compress: false
local_time: false
verbose: false
but I realised I wanted to rotate earlier and not have 100 MB files. I have just recently made the change so will need to keep an eye on successful rotation.
So, take backup, make your desired changes, restart AdGH from : System: Diagnostics: Services. Read the new log file to see it starting. Keep an eye to make sure the log files are growing and rotating as expected.
More info on settings https://github.com/AdguardTeam/AdGuardHome/wiki/Configuration
Quote from: cookiemonster on December 04, 2024, 11:42:56 PM
For AdGH logging specifically there's another way. I can't really remember how long ago I did this modification but a few AdGH versions ago they introduced some additional logging capabilities by config. I update AdGH from time to time using the UI. The installation is from mimugmail's repo.
Please if you do it, check you have a version that has this capability and take notice of the schema changes. Make a backup of the config file: /usr/local/AdGuardHome/AdGuardHome.yaml before any changes.
I am on Version: v0.107.54 of AdGH.
On my config I have added/changed the following settings in the log section:
log:
enabled: true
file: /var/log/AdGuardHome/AdGuardHome.log
max_backups: 30
max_size: 10
max_age: 31
compress: false
local_time: false
verbose: false
This is very recently, before I had:
log:
enabled: true
file: /var/log/AdGuardHome/AdGuardHome.log
max_backups: 3
max_size: 100
max_age: 3
compress: false
local_time: false
verbose: false
but I realised I wanted to rotate earlier and not have 100 MB files. I have just recently made the change so will need to keep an eye on successful rotation.
So, take backup, make your desired changes, restart AdGH from : System: Diagnostics: Services. Read the new log file to see it starting. Keep an eye to make sure the log files are growing and rotating as expected.
More info on settings https://github.com/AdguardTeam/AdGuardHome/wiki/Configuration
Awesome, thank you! That should help a ton with troubleshooting. I've got it logging now and will monitor.
Nice.