Messages

I had quite a journey when I wanted to debug why my OPNSense appliance suddenly rebooted several times. So in the crash log I saw this:

Code Select Expand


Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 03
fault virtual address = 0x54
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80f5e626
stack pointer         = 0x28:0xfffffe00004c2140
frame pointer         = 0x28:0xfffffe00004c2190
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 12 (swi4: clock (0))
trap number = 12
panic: page fault
cpuid = 3
time = 1603188380
__HardenedBSD_version = 1200059 __FreeBSD_version = 1201000
version = FreeBSD 12.1-RELEASE-p10-HBSD #0  517e44a00df(stable/20.7)-dirty: Mon Sep 21 16:21:17 CEST 2020
    root@sensey64:/usr/obj/usr/src/amd64.amd64/sys/SMP
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00004c1df0
vpanic() at vpanic+0x1a2/frame 0xfffffe00004c1e40
panic() at panic+0x43/frame 0xfffffe00004c1ea0
trap_fatal() at trap_fatal+0x39c/frame 0xfffffe00004c1f00
trap_pfault() at trap_pfault+0x49/frame 0xfffffe00004c1f60
trap() at trap+0x29f/frame 0xfffffe00004c2070
calltrap() at calltrap+0x8/frame 0xfffffe00004c2070
--- trap 0xc, rip = 0xffffffff80f5e626, rsp = 0xfffffe00004c2140, rbp = 0xfffffe00004c2190 ---
in6_setscope() at in6_setscope+0xa6/frame 0xfffffe00004c2190
ip6_forward() at ip6_forward+0x359/frame 0xfffffe00004c22e0
pf_test6() at pf_test6+0x1c82/frame 0xfffffe00004c2470
pf_check6_out() at pf_check6_out+0x3f/frame 0xfffffe00004c24a0
pfil_run_hooks() at pfil_run_hooks+0x87/frame 0xfffffe00004c2530
ip6_output() at ip6_output+0x1a06/frame 0xfffffe00004c27c0
icmp6_reflect() at icmp6_reflect+0x2f0/frame 0xfffffe00004c2870
icmp6_error() at icmp6_error+0x4aa/frame 0xfffffe00004c28c0
nd6_llinfo_timer() at nd6_llinfo_timer+0x340/frame 0xfffffe00004c2940
softclock_call_cc() at softclock_call_cc+0x143/frame 0xfffffe00004c29f0
softclock() at softclock+0x79/frame 0xfffffe00004c2a10
ithread_loop() at ithread_loop+0x1d4/frame 0xfffffe00004c2a70
fork_exit() at fork_exit+0x83/frame 0xfffffe00004c2ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004c2ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
KDB: enter: panic
panic.txt0600001213743533234  7140 ustarrootwheelpage faultversion.txt06000022713743533234  7623 ustarrootwheelFreeBSD 12.1-RELEASE-p10-HBSD #0  517e44a00df(stable/20.7)-dirty: Mon Sep 21 16:21:17 CEST 2020
    root@sensey64:/usr/obj/usr/src/amd64.amd64/sys/SMP

OPNsense (c) 2014-2020 Deciso B.V.


So the kernel panic was related to IPv6. In my case I didn't use IPv6 due to another bug in 20.7 but I still receive those packets on my VDSL WAN interface. The reason for that is, that I have a static IPv6 network assigned to me, a /64 for the WAN side and a /54 for my own usage. So my provider did forward those packets still to my VDSL and through my PPPoE connection since it's static, my own and the provider doesn't know that I disabled IPv6 DHCP on my PPPoE connection.

So the fix was to enable the IPv6 on WAN side (the /64) but I could still keep it disabled on LAN side (the /56) to prevent another bug.

But nevertheless I would argue that this should be fixed in the FreeBSD Kernel so it doesn't trigger a panic, just ignoring/dropping those packets.

So my question would be, is this something that needs to be adressed by the OPNSense folks or rather on FreeBSD side or even HardenedBSD side?
So I would like to have some guidance where to report which details.
While it might be a corner case (static IPv6 but disabled on PPPoE) it was really hard to figure it out.

I'm glad I'm not the only one with IPv6 issues :)

I don't have DHCPv6 running on OPNSense, since I have a static IPv6 net I just use SLAAC. So what got worse from 20.1 to 20.7 is the fact, that the IPv6 setting for the LAN interface stops working as well, while prior only clients lost it some time.

A workaround was, that when ever I had a PPPoE disconnect I did run `/var/etc/rtsold_pppoe0_script.sh` but that doesn't fix it now. If I go into the LAN interface section and don't change anything just press "apply" for the configuration again it will work. So there is something that's not triggering anymore when the WAN side receives updates.

I looked into the changelog but didn't find anything that leads to the obvious issue.

Habt ihr mal in /var/unbound/access_lists.conf geschaut ob dort euer Netzwerk zu sehen ist? Ich hatte auch Probleme und mit viel debuggen festgestellt, dass dort mein IPv6 Netzwerk fehlt obwohl ich sogar dank Business Anschluss statisches IPv6 (und IPv4) habe. Als ich dort mein /56 eingetragen habe und unbound restarted hab lief es auch. Nun muss ich es nur noch persistent hinkriegen.

Geht mit Services: Unbound DNS: Access Lists immerhin auch per UI!

I just stumbled upon the same issue, I was wondering why DNS was so slow and with wireshark I saw refused in the reply. Setting my local IPv6 network (which is static btw) in /var/unbound/access_lists.conf fixed it.

At the moment your best option is configuring the DNS servers manually. Most ISPs don't change them very often. You mentioned Deutsche Telekom. Their current DNS resolver addresses have been in use since at least 2014.

I looked into that and they change from time to time when I look into what I receive via PPPoE, they even document this fact:

https://www.telekom.de/hilfe/festnetz-internet-tv/e-mail/e-mail-server-e-mail-protokolle-und-e-mail-einrichtung/wichtige-server-der-telekom?samChecked=true (see DNS section, german language though)

Since you always get the same prefix from your ISP, you don't really need tracking and can just configure the LAN interface(s) statically. That should work more reliably.

Didn't thought of that, I w ill try this out.

Thanks :)

Nevertheless it would be nice to know more about the internals, but I guess I have to dig into the code a bit more.

Do you have the LAN interface on a static IPv6 address with its own /64 internal subnet?

No, I did set the current options:

• IPv6 Interface: VDSL (my primary WAN uplink)
• IPv6 Prefix ID: 0x24
• Manual Configuration: Allow manual adjustments of DHCPv6 and RA (checked)

And I have range defined at the DHCPv6 section.
[/list]

Do you run RADVD to send out router advertisements on your LAN subnet?

Services: RA for the LAn is set to:

• RA: Stateless
• Priority: Normal
• Advertise Default Gateway

I updated to 19.7 and so far everything quite nice, thanks. But this issue is still the same for me, so quoting myself from  (https://forum.opnsense.org/index.php?topic=13375.0):

Hi,

I have a VDSL Telekom uplink with a static IPv4 and IPv6 where the latter offers me a /56 IPv6 network. This works very well if I start the OPNSense or after I configure the LAN interface. So the LAN interface takes care of requesting the IPv6 network information and forwards this to the client.
Sometimes when the VDSL uplink is gone (power outage, other issues) and reconnects the VDSL uplink interface receives the IPv4 and also the IPv6 for the WAN side but it doesn't update the LAN interface which has lost the IPv6 information in the meantime.

My workaround for now is to go to the LAN interface in the WebUI, don't change anything, just click save and apply changes (although none have happened) and thus the interface is reconfigured again and IPv6 works again.

So there seems to be an issue with the LAN interface noticing the WAN interface gone and back online. Can you give me any hints how I could further debug this, increase log messages or which parts are involved?
If anyone know a solution even better :)
But I guess I need to dig into this deeper to provide you with necessary details to properly fix this issue.
So it would be helpful for me to know which parts/scripts/services are involved so I can try if I can fix it myself.

Thanks

So any hints how to solve/fix/patch it myself are welcome :)

I updated to 19.7 and so far everything quite nice, thanks. But this issue is still the same for me, so quoting myself from (https://forum.opnsense.org/index.php?topic=13374.0):

Hi,

I have two uplinks, one via PPPoE (VDSL) and one via DHCP (Cable). The issue is that I just want to use the DNS servers provided by the VDSL uplink since those are the only ones which have the correct DNS entries for VoIP. The option Allow DNS server list to be overridden by DHCP/PPP on WAN is global and thus I end up with a mixed resolv.conf. This results in VoIP issues.
I also don't want to hardcode the DNS servers for VDSL as they could change.
Is there a way to handle this, at least somewhere if not in the Web UI?
Or any other solution to solve this.

Thanks

So any hints how to solve/fix/patch it myself are welcome :)

It's not a company connect contract. So the DNS servers might change from time to time and I receive them via PPPoE. But that would be another workaround, yes.

I think it's still a valid setup to have the option to define the override DNS not global but on the interfaces.

[tel.t-online.de]

Yes I am Telekom Business, but I need correct replies for tel.t-online.de which change from time to time. If you use any other servers you end up with 217.0.128.133 which is not correct for all usecases.
I got this information from a Telekom technician who helped me debug some outage with my telephone. And this was the reason.

Well it doesn't happen all the time, maybe once a month and I can fix it. But I host some services behind the OPNSense and the DNS has the AAAA records as well so it's an issue for the time between issue occurs and I fix it manually.
Also the static IPs I have I pay for so would like to have a working solution and keen to work on that myself :)

Hi,

I have a VDSL Telekom uplink with a static IPv4 and IPv6 where the latter offers me a /56 IPv6 network. This works very well if I start the OPNSense or after I configure the LAN interface. So the LAN interface takes care of requesting the IPv6 network information and forwards this to the client.
Sometimes when the VDSL uplink is gone (power outage, other issues) and reconnects the VDSL uplink interface receives the IPv4 and also the IPv6 for the WAN side but it doesn't update the LAN interface which has lost the IPv6 information in the meantime.

My workaround for now is to go to the LAN interface in the WebUI, don't change anything, just click save and apply changes (although none have happened) and thus the interface is reconfigured again and IPv6 works again.

So there seems to be an issue with the LAN interface noticing the WAN interface gone and back online. Can you give me any hints how I could further debug this, increase log messages or which parts are involved?
If anyone know a solution even better :)
But I guess I need to dig into this deeper to provide you with necessary details to properly fix this issue.
So it would be helpful for me to know which parts/scripts/services are involved so I can try if I can fix it myself.

Thanks

[Allow DNS server list to be overridden by DHCP/PPP on WAN]

Hi,

I have two uplinks, one via PPPoE (VDSL) and one via DHCP (Cable). The issue is that I just want to use the DNS servers provided by the VDSL uplink since those are the only ones which have the correct DNS entries for VoIP. The option Allow DNS server list to be overridden by DHCP/PPP on WAN is global and thus I end up with a mixed resolv.conf. This results in VoIP issues.
I also don't want to hardcode the DNS servers for VDSL as they could change.
Is there a way to handle this, at least somewhere if not in the Web UI?
Or any other solution to solve this.

Thanks

[conservative]

Habe nun mal die Firewall auf conservative gestellt und das Port Forwarding wieder raus. Tut aktuell, mal sehen ob doch mal ein Telefonat nicht reinkommt.

@ojessie: kannst mal nen Screenshot von deiner Konfiguration auf Seiten der FB posten oder auch mal mit tcpdump mitsniffen auf beiden Interfaces, der SIP traffic ist da gut zu erkennen.