How do I get a kernel with the needful to run dtrace?

[Wolfspyre]

I posted on the reddit forum as well:

https://www.reddit.com/r/opnsense/comments/15bdy5c/trying_to_figure_what_i_brokedetuned_wrt_my_fw/

My topology is admittedly overly complicated....
I'm working to refine it; but that's a different can of rhinos.
I've somehow managed to horribly de-tune my firewall pair, and I'm attempting to diagnose what's the most busted.... this is MOSTLY a learning exercise; as while I've been a sysadmin/nerd for decades, I've not ventured too far down the path of real profiling...
I'm seeing godawful performance from haproxy... so I figured that would be a good 'performance tuning' path to go down...
I'm trying to kick off dtrace, to generate flamegraphs to see if I can identify what's going on; however when I run what I believe to be a simple dtrace from the the fbsd dtrace oneliners page: https://wiki.freebsd.org/DTrace/One-Liners 

Which is referenced on the opnsense dtrace howto page:
https://docs.opnsense.org/development/how-tos/dtrace.html

...

I get a somewhat unhelpful error:


# dtrace -n 'profile-99 /arg0/ { @\[stack()\] = count(); }'


dtrace: invalid probe specifier profile-99 /arg0/ { @\[stack()\] = count(); }: "/usr/lib/dtrace/ip.d", line 1: operator -> cannot be applied to a forward declaration: no struct ip definition is available


While the opnsense dtrace howto page does EXPLICITLY say:

Keep in mind that user space can also be analyzed using DTrace, but this is beyond the scope of this document.

There's not much else, guidance wise, on how to connect the dots....

Do I really need to recompile the kernel to  do this?

Am I jus TOTALLY DOIN IT WRONG? :)

Any helpful advise or pointers would be greatly appreciated....
Is there an obvious reason that a kernel which has the needful prebaked isn't easily available/usable by default?

would it be better from an issue reproducibility perspective to have debug-decorated-kernel available to boot from for enhanced diagnostics? or is that bad logic?

[Wolfspyre]

I gotta admit, I'm a little surprised that nobody's got a pointer to some docs.

is there a reasonable howto on getting a generic opnsense install to the point that one can start the dtrace howto linked?

it's been >20y since I last compiled freebsd userland, and to say I'm rusty would be a kindness....

I don't mind getting my hands dirty, but it feels like I'm missing an obvious step or two.

[Wolfspyre]

seems this thread https://forum.opnsense.org/index.php?topic=35185.0  has enough info for me to break new and exciting things!! thanks much!