Messages

It looks like you're using a HTTP-01 challenge type in ACME. I recommend to use DNS-01, it is much more reliable.

I'm now seeing a duplicate certificate for one domain in the HAProxy Public Service Certificates, even though there is only 1 certificate for that domain in the ACME plugin list. 

os-haproxy displays all certificates from System->Trust->Certificates. You need to check this page to get more details about the duplicate certificate.

Besides that os-acme-client will also log a message if a certificate is imported into System->Trust->Certificates, so you should be able to trace this.

A hotfix is available:
https://github.com/opnsense/plugins/issues/3779#issuecomment-1917956814

Plugin maintainer here. I don't plan another overhaul of the HAProxy plugin GUI. ;D So if anyone wants to start building a documentation, yes, please go ahead. Or if you think that some information could be added to one of the "introduction" pages, please submit your suggestion on GitHub...

I may review documentation (GitHub pull requests), but I can't put any effort into writing documentation in the foreseeable future. That being said, there's some good documentation out there...

Ciao
- Frank

@fraenki, my router is headless.  Is there any other way of finding the failed on startup message logs?

AFAICT, no. If you have a serial console, you would still be able to see all boot messages...

Is there another logfile from the startup process?

AFAIK, the console messages from the "service" command are not recorded anywhere.

You may have to watch the console while during the boot process. Maybe make a video in order to not miss the HAProxy error message.


Ciao
- Frank

A related message in the log looks like this:

Code Select Expand


[1ff53fdb-8812-4a5b-bd04-04cddac2fa89] Script action failed with Command
  'configctl template reload OPNsense/HAProxy 2 > /dev/null;
  /usr/local/opnsense/scripts/OPNsense/HAProxy/syncCerts.py sync --output json '
returned non-zero exit status 1.


This message is unrelated to HAProxy startup problems. It is the "cert_sync_bulk" action, which is either triggered by a cron job or the related GUI button.

Having that said, we need to get more information to find out why HAProxy is not starting. I'd suggest to run the following command on the OPNsense shell as user root:

Code Select Expand


service haproxy start


The output should contain hints about the startup issue.


Ciao
- Frank

os-haproxy version 3.12 will include GUI options to enable the Prometheus exporter:
https://github.com/opnsense/plugins/issues/2764#issuecomment-1287528782

Thanks for providing a test build!

Unfortunately I will not be able to test it in the upcoming weeks. Maybe someone else could test it?
Be sure to change the tunable net.inet.tcp.sack.enable back to "1" in order to see if the new build solves this issue.

A fix has finally arrived in FreeBSD:
https://cgit.freebsd.org/src/commit/?id=5ae83e0d871bc7cbe4dcc9a33d37eb689e631efe

Maybe @franco could incooparate it into the next (minor) release of OPNsense  :)

The crashes are gone after enabling the workaround that was suggested in the FreeBSD bug ticket:
add tunable net.inet.tcp.sack.enable with value "0" to disable SACK support.


Regards
- Frank

I'm also experiencing crashes and my trace looks pretty similar (but it's not the same):

I'be upgraded to 22.7.b. Unfortunately, in my case the firewall still crashes with pretty much the same panic:

Code Select Expand


Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 02
fault virtual address = 0x18
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80d3d9ed
stack pointer         = 0x28:0xfffffe013339d500
frame pointer         = 0x28:0xfffffe013339d570
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 0 (if_io_tqg_2)
trap number = 12
panic: page fault
cpuid = 2
time = 1655658689

KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe013339d2c0
vpanic() at vpanic+0x17f/frame 0xfffffe013339d310
panic() at panic+0x43/frame 0xfffffe013339d370
trap_fatal() at trap_fatal+0x385/frame 0xfffffe013339d3d0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe013339d430
calltrap() at calltrap+0x8/frame 0xfffffe013339d430
--- trap 0xc, rip = 0xffffffff80d3d9ed, rsp = 0xfffffe013339d500, rbp = 0xfffffe013339d570 ---
m_copydata() at m_copydata+0x4d/frame 0xfffffe013339d570
tcp_output() at tcp_output+0x1339/frame 0xfffffe013339d750
tcp_do_segment() at tcp_do_segment+0x2cfd/frame 0xfffffe013339d830
tcp_input_with_port() at tcp_input_with_port+0xafb/frame 0xfffffe013339d990
tcp_input() at tcp_input+0xb/frame 0xfffffe013339d9a0
ip_input() at ip_input+0x15f/frame 0xfffffe013339da30
netisr_dispatch_src() at netisr_dispatch_src+0xb9/frame 0xfffffe013339da80
ether_demux() at ether_demux+0x138/frame 0xfffffe013339dab0
ether_nh_input() at ether_nh_input+0x355/frame 0xfffffe013339db10
netisr_dispatch_src() at netisr_dispatch_src+0xb9/frame 0xfffffe013339db60
ether_input() at ether_input+0x69/frame 0xfffffe013339dbc0
ether_demux() at ether_demux+0x121/frame 0xfffffe013339dbf0
ether_nh_input() at ether_nh_input+0x355/frame 0xfffffe013339dc50
netisr_dispatch_src() at netisr_dispatch_src+0xb9/frame 0xfffffe013339dca0
ether_input() at ether_input+0x69/frame 0xfffffe013339dd00
iflib_rxeof() at iflib_rxeof+0xc27/frame 0xfffffe013339de00
_task_fn_rx() at _task_fn_rx+0x72/frame 0xfffffe013339de40
gtaskqueue_run_locked() at gtaskqueue_run_locked+0x15d/frame 0xfffffe013339dec0
gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xc2/frame 0xfffffe013339def0
fork_exit() at fork_exit+0x7e/frame 0xfffffe013339df30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe013339df30
--- trap 0, rip = 0xffffffff80c3137f, rsp = 0, rbp = 0x6 ---
mi_startup() at mi_startup+0xdf/frame 0x6


IIRC there was a way to only install the kernel of the 22.7 pre-release, not the whole thing... but can't remember the details.

I'm also experiencing crashes and my trace looks pretty similar (but it's not the same):

Code Select Expand


KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe01334652c0
vpanic() at vpanic+0x17f/frame 0xfffffe0133465310
panic() at panic+0x43/frame 0xfffffe0133465370
trap_fatal() at trap_fatal+0x385/frame 0xfffffe01334653d0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe0133465430
calltrap() at calltrap+0x8/frame 0xfffffe0133465430
--- trap 0xc, rip = 0xffffffff80d37acd, rsp = 0xfffffe0133465500, rbp = 0xfffffe0133465570 ---
m_copydata() at m_copydata+0x4d/frame 0xfffffe0133465570
tcp_output() at tcp_output+0x1339/frame 0xfffffe0133465750
tcp_do_segment() at tcp_do_segment+0x2cd5/frame 0xfffffe0133465830
tcp_input_with_port() at tcp_input_with_port+0xafb/frame 0xfffffe0133465990
tcp_input() at tcp_input+0xb/frame 0xfffffe01334659a0
ip_input() at ip_input+0x15f/frame 0xfffffe0133465a30
netisr_dispatch_src() at netisr_dispatch_src+0xb9/frame 0xfffffe0133465a80
ether_demux() at ether_demux+0x138/frame 0xfffffe0133465ab0
ether_nh_input() at ether_nh_input+0x355/frame 0xfffffe0133465b10
netisr_dispatch_src() at netisr_dispatch_src+0xb9/frame 0xfffffe0133465b60
ether_input() at ether_input+0x69/frame 0xfffffe0133465bc0
ether_demux() at ether_demux+0x121/frame 0xfffffe0133465bf0
ether_nh_input() at ether_nh_input+0x355/frame 0xfffffe0133465c50
netisr_dispatch_src() at netisr_dispatch_src+0xb9/frame 0xfffffe0133465ca0
ether_input() at ether_input+0x69/frame 0xfffffe0133465d00
iflib_rxeof() at iflib_rxeof+0xc27/frame 0xfffffe0133465e00
_task_fn_rx() at _task_fn_rx+0x72/frame 0xfffffe0133465e40
gtaskqueue_run_locked() at gtaskqueue_run_locked+0x15d/frame 0xfffffe0133465ec0
gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xc2/frame 0xfffffe0133465ef0
fork_exit() at fork_exit+0x7e/frame 0xfffffe0133465f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0133465f30
--- trap 0, rip = 0xffffffff80c2b91f, rsp = 0, rbp = 0x6 ---
mi_startup() at mi_startup+0xdf/frame 0x6


@magnust, is your issue fixed in the 22.7 snapshot?

Should I try to use Tunables?
It´s better in updates I think and restore config..

Definitely; do not modify files locally. Setting Tunables from the GUI is the better approach.