Messages

This is gold! Thank you so much for putting it together. Exactly what I was looking for.

I realize this is a bit anecdotal but thought I would share:

I used to have Let's Encrypt working for internal domains using the TXT challenge method with Google Cloud Platform when I had Go Daddy as my registrar. Then I switched over to Google Domains (the registrar, not the same as Google Cloud DNS) and somewhere in the transition ACME stopped working.

I'm in the process of troubleshooting and it may as well be something I've neglected, but it makes me suspicious to see someone else with the same setup (Google as registrar and DNS provider) having the same "Can't find a matching managed zone!"

Same behavior here. I'm running 19.7.6 on a SuperMicro SYS-5019A-FTN4.

Apparently there were driver issues on FreeBSD 10 a couple of years back, according to these:

https://www.servethehome.com/day-0-with-intel-atom-c3000-getting-nics-working/
https://forum.netgate.com/topic/120704/atom-c3758-supermicro-a2sdi-8c-hln4f-pfsense

But from what I understand the drivers are available in FreeBSD 11 (which is the case with OPNsense 19.x) so I suspect this may not be related.

I'd be glad to help with forensics but am not a FreeBSD person and would need some pointers on how to research this.

Hear hear! Chiming in as a beats plugin would be amazingly useful.

Just metricbeats alone would help tons, although packet and files would be great as well.

For those who are interested in following the standard syslog -> logstash, the github referencing the post above is active and surprising up-to-date on the latest versions of java / elk and only a tad behind opnsense (18.1).

https://github.com/a3ilson/pfelk

That won't work for two reasons: ::8040 is outside of your /60. And you should only enter the range itself.
You can use ::8 - ::c (two /62s).

Thank you, Maurice! This comment alone is worth *gold* to me. I was misunderstanding how the range worked. My two misconceptions here:
1. I was entering the entire block, including the last address which is not actually part of the range.
2. The range is defined by the first address of the last selectable block (in this case ::c). It took me a while to figure out why ::c and not ::f, given ::8 - ::c = 5x /64 networks and not 8x /64.

Just for my edification, if I were to choose a prefix delegation size of 63, would that make the range ::8 - ::e?

Oh, and by the way: You can use Request only an IPv6 prefix on the ISP-facing OPNsense's WAN, but not on your lab OPNsense! The way OPNsense does Prefix Delegation requires the DHCPv6 client to also request an address. Otherwise the required route won't be created. I consider this to be a bug, but it has been discussed a long time ago and it seems they won't fix it.

I ran into that post of yours, which was also very helpful. Thanks again. All very helpful!

TL;DR: don't use interface tracking with prefix delegation.

When using interface tracking, I'm not able to get anything other than a /64 assigned, and that is reflected in the web UI for DHCPv6. It states that the subnet is a /64 but also states that the available prefix delegation size is /61, which on paper seems reasonable, however I have not found a way forward using this configuration.

What I have found, however, is that if I statically assign my internet facing OPNsense's LAN interface to 2001:db8:a:8038::1/61, then DHCPv6 automatically starts delegating prefixes. My downstream OPNsense (LAB) can now obtain an address for it's LAN and request a PD of /62 or less, as intended.

It seems that in order to delegate a prefix, the interface on which DHCPv6 is listening must be assigned an address coming from a part of the block that it is providing the addresses from. In other words, if a /60 is granted to a host, then its interface needs to be assigned to a /61 and DHCPv can only delegate a /62?

I don't understand if the above is a bug/limitation of OPNsense, or if it's working as intended and I'm missing a key part of how to effectively design / partition subnets. I assume the latter but wouldn't be surprised if it were the former. Or both?

[status code for PD-0: no prefixes]

And here's the corresponding output of dhcp6c -f on the lab OPNsense:

Code Select Expand

Nov/06/2019 23:52:45: Sending Solicit
Nov/06/2019 23:52:45: a new XID (26162c) is generated
Nov/06/2019 23:52:45: set client ID (len 14)
Nov/06/2019 23:52:45: set elapsed time (len 2)
Nov/06/2019 23:52:45: set option request (len 4)
Nov/06/2019 23:52:45: set IA_PD prefix
Nov/06/2019 23:52:45: set IA_PD
Nov/06/2019 23:52:45: send solicit to ff02::1:2%em1
Nov/06/2019 23:52:45: reset a timer on em1, state=SOLICIT, timeo=0, retrans=1083
Nov/06/2019 23:52:45: receive advertise from fe80::ae1f:6bff:feb1:f2d4%em1 on em1
Nov/06/2019 23:52:45: get DHCP option IA_PD, len 59
Nov/06/2019 23:52:45:   IA_PD: ID=0, T1=0, T2=0
Nov/06/2019 23:52:45: get DHCP option status code, len 43
Nov/06/2019 23:52:45:   status code: no prefixes
Nov/06/2019 23:52:45: get DHCP option client ID, len 14
Nov/06/2019 23:52:45:   DUID: 00:01:01:01:25:55:2b:c5:08:00:27:4c:4d:5d
Nov/06/2019 23:52:45: get DHCP option server ID, len 14
Nov/06/2019 23:52:45:   DUID: 00:01:01:01:25:01:09:d5:ac:1f:6b:b1:f2:d4
Nov/06/2019 23:52:45: server ID: 00:01:01:01:25:01:09:d5:ac:1f:6b:b1:f2:d4, pref=-1
Nov/06/2019 23:52:45: reset timer for em1 to 0.999509
Nov/06/2019 23:52:46: picked a server (ID: 00:01:01:01:25:01:09:d5:ac:1f:6b:b1:f2:d4)
Nov/06/2019 23:52:46: Sending Request
Nov/06/2019 23:52:46: a new XID (37e72e) is generated
Nov/06/2019 23:52:46: set client ID (len 14)
Nov/06/2019 23:52:46: set server ID (len 14)
Nov/06/2019 23:52:46: set elapsed time (len 2)
Nov/06/2019 23:52:46: set option request (len 4)
Nov/06/2019 23:52:46: set status code
Nov/06/2019 23:52:46: set IA_PD
Nov/06/2019 23:52:46: send request to ff02::1:2%em1
Nov/06/2019 23:52:46: reset a timer on em1, state=REQUEST, timeo=0, retrans=906
Nov/06/2019 23:52:46: receive reply from fe80::ae1f:6bff:feb1:f2d4%em1 on em1
Nov/06/2019 23:52:46: get DHCP option IA_PD, len 59
Nov/06/2019 23:52:46:   IA_PD: ID=0, T1=0, T2=0
Nov/06/2019 23:52:46: get DHCP option status code, len 43
Nov/06/2019 23:52:46:   status code: no prefixes
Nov/06/2019 23:52:46: get DHCP option client ID, len 14
Nov/06/2019 23:52:46:   DUID: 00:01:01:01:25:55:2b:c5:08:00:27:4c:4d:5d
Nov/06/2019 23:52:46: get DHCP option server ID, len 14
Nov/06/2019 23:52:46:   DUID: 00:01:01:01:25:01:09:d5:ac:1f:6b:b1:f2:d4
Nov/06/2019 23:52:46: Received REPLY for REQUEST
Nov/06/2019 23:52:46: make an IA: PD-0
Nov/06/2019 23:52:46: status code for PD-0: no prefixes
Nov/06/2019 23:52:46: IA PD-0 is invalidated
Nov/06/2019 23:52:46: remove an IA: PD-0
Nov/06/2019 23:52:46: reset a timer on em1, state=INIT, timeo=0, retrans=924
Nov/06/2019 23:52:46: executes /var/etc/dhcp6c_wan_script.sh
OK
Nov/06/2019 23:52:47: script "/var/etc/dhcp6c_wan_script.sh" terminated
Nov/06/2019 23:52:47: removing an event on em1, state=REQUEST
Nov/06/2019 23:52:47: removing server (ID: 00:01:01:01:25:01:09:d5:ac:1f:6b:b1:f2:d4)
Nov/06/2019 23:52:47: got an expected reply, sleeping.

I see nothing of interest here other than status code for PD-0: no prefixes... but then again, I'm not sure what I should be looking for.

Update to the above:

I was at least able to confirm that PD seems to be working as I had originally imagined.

AFAIK, there doesn't seem to be an easy way of directly identifying what prefix has been delegated to you (not logged anywhere I could find) but I was able to get more output from dhcp6c by SSHing as root, killing the dhcp6c process and then running it manually with -f (foreground process) which printed the following useful details to STDOUT.

Code Select Expand

Nov/06/2019 22:46:52: Sending Solicit
Nov/06/2019 22:46:52: a new XID (503f61) is generated
Nov/06/2019 22:46:52: set client ID (len 14)
Nov/06/2019 22:46:52: set elapsed time (len 2)
Nov/06/2019 22:46:52: set option request (len 4)
Nov/06/2019 22:46:52: set IA_PD prefix
Nov/06/2019 22:46:52: set IA_PD
Nov/06/2019 22:46:52: send solicit to ff02::1:2%ngeth0
Nov/06/2019 22:46:52: reset a timer on ngeth0, state=SOLICIT, timeo=0, retrans=1091
Nov/06/2019 22:46:52: receive advertise from fe80::a2f3:e4ff:fe5a:1630%ngeth0 on ngeth0
Nov/06/2019 22:46:52: get DHCP option server ID, len 10
Nov/06/2019 22:46:52:   DUID: 00:03:01:01:a0:f3:e4:5a:16:30
Nov/06/2019 22:46:52: get DHCP option client ID, len 14
Nov/06/2019 22:46:52:   DUID: 00:01:01:01:25:53:4a:ce:f0:18:98:78:ba:c5
Nov/06/2019 22:46:52: get DHCP option IA_PD, len 41
Nov/06/2019 22:46:52:   IA_PD: ID=0, T1=1800, T2=2880
Nov/06/2019 22:46:52: get DHCP option IA_PD prefix, len 25
Nov/06/2019 22:46:52:   IA_PD prefix: 2001:db8:a:8030::/60 pltime=3600 vltime=1928440319504
Nov/06/2019 22:46:52: server ID: 00:03:01:01:a0:f3:e4:5a:16:30, pref=-1
Nov/06/2019 22:46:52: reset timer for ngeth0 to 0.996775
Nov/06/2019 22:46:53: picked a server (ID: 00:03:01:01:a0:f3:e4:5a:16:30)
Nov/06/2019 22:46:53: Sending Request
Nov/06/2019 22:46:53: a new XID (d3d2a5) is generated
Nov/06/2019 22:46:53: set client ID (len 14)
Nov/06/2019 22:46:53: set server ID (len 10)
Nov/06/2019 22:46:53: set elapsed time (len 2)
Nov/06/2019 22:46:53: set option request (len 4)
Nov/06/2019 22:46:53: set IA_PD prefix
Nov/06/2019 22:46:53: set IA_PD
Nov/06/2019 22:46:53: send request to ff02::1:2%ngeth0
Nov/06/2019 22:46:53: reset a timer on ngeth0, state=REQUEST, timeo=0, retrans=909
Nov/06/2019 22:46:53: receive reply from fe80::a2f3:e4ff:fe5a:1630%ngeth0 on ngeth0
Nov/06/2019 22:46:53: get DHCP option server ID, len 10
Nov/06/2019 22:46:53:   DUID: 00:03:01:01:a0:f3:e4:5a:16:30
Nov/06/2019 22:46:53: get DHCP option client ID, len 14
Nov/06/2019 22:46:53:   DUID: 00:01:01:01:25:53:4a:ce:f0:18:98:78:ba:c5
Nov/06/2019 22:46:53: get DHCP option IA_PD, len 41
Nov/06/2019 22:46:53:   IA_PD: ID=0, T1=1800, T2=2880
Nov/06/2019 22:46:53: get DHCP option IA_PD prefix, len 25
Nov/06/2019 22:46:53:   IA_PD prefix: 2001:db8:a:8030::/60 pltime=3600 vltime=1928440319504
Nov/06/2019 22:46:53: Received REPLY for REQUEST
Nov/06/2019 22:46:53: make an IA: PD-0
Nov/06/2019 22:46:53: create a prefix 2001:db8:a:8030::/60 pltime=3600, vltime=3600
Nov/06/2019 22:46:53: add an address 2001:db8:a:8032:ae1f:6bff:feb1:f2d4/64 on ix0
Nov/06/2019 22:46:53: add an address 2001:db8:a:8031:ae1f:6bff:feb1:f2d6/64 on ix2
Nov/06/2019 22:46:53: executes /var/etc/dhcp6c_wan_script.sh
OK
Nov/06/2019 22:47:06: script "/var/etc/dhcp6c_wan_script.sh" terminated
Nov/06/2019 22:47:06: removing an event on ngeth0, state=REQUEST
Nov/06/2019 22:47:06: removing server (ID: 00:03:01:01:a0:f3:e4:5a:16:30)
Nov/06/2019 22:47:06: got an expected reply, sleeping.

And that seems to confirm that 2001:db8:a:8030::/60 has indeed been delegated to my internet facing OPNsense.

I will now attempt to reproduce the conditions that cause the "no IPv6 pools on this shared network" while running dhcpc in the foreground on the lab OPNsense and see if that produces any additional useful output.

I'll state upfront that IPv6 is not my strong suit. I'm attempting to setup a lab so I can educate myself. That said, I'm a bit stuck and could use some advice. I've searched this forum and online for days and cannot figure out what's going on.

I have two OPNsense hosts. One is on a physical host with 3 ports: WAN, WIFI and LAN. The other is virtual with two ports WAN is bridged to the physical LAN network, and it has it's own LAN on a private network. 

On the internet facing host, I am able to successfully request a /60 prefix from my ISP, of which I'm reserving the first half /61 for internal addresses. I am requesting a prefix only and letting the WAN be assigned a random external IP address, which is not in the range of the /60 being delegated to me.

My home network (WIFI) is tracking the WAN (0x1) and has a /64. It doesn't have DHCP enabled, RA is set to "unmanaged" and hosts are using SLAAC. It's prefix is 2001:db8:a:8031::/64. Everything looks fine.

My physical lan (LAN) is also tracking the WAN (0x2) and has another /64. It does have DHCP enabled and RA is set to "assisted". It's prefix is 2001:db8:a:8032::/64. I can reserve leases for specific hosts as expected and all good was well.

From the above, I'm assuming that I'm being delegated the prefix 2001:db8:a:8030::/60.

Here's where things start to go sideways.

The OPNsense DHCPv6 on my LAN indicates an available prefix delegation size of 61. That makes sense to me, given I should only be able to delegate a subset of the block delegate and the next smallest prefix size is 61.

Given I've already used two /64s in the first /61 for my WIFI and LAN networks, I assume this is referring to the latter /61 (from ::8038 to ::8040), however in the Prefix Delegation Size dropdown the available options are 48, 52, 56, 60, 62, 63, and 64. Notice there's no 61. Odd?

Given there's no 61, I've selected 62 and entered the tail-end of my range, let's say ::803c - ::8040. I believe this should give me 4 /64s but I cannot acquire prefixes from it. 

Running:
$ sudo tcpdump -i ix0 -vv ip6 and dst port 547 or dst port 546

Results:

Code Select Expand

16:12:25.942585 IP6 (hlim 1, next-header UDP (17) payload length: 84) fe80::a00:27ff:fe3e:af77.dhcpv6-client > ff02::1:2.dhcpv6-server: [udp sum ok] dhcp6 request (xid=b33316 (client-ID hwaddr/time type 1 time 626338757 0800274c4d5d) (server-ID hwaddr/time type 1 time 620825045 ac1f6bb1f2d4) (elapsed-time 0) (option-request DNS-server DNS-search-list) (IA_PD IAID:0 T1:0 T2:0 (status-code NoPrefixAvail)))

16:12:25.942785 IP6 (hlim 64, next-header UDP (17) payload length: 111) fe80::ae1f:6bff:feb1:f2d4.dhcpv6-server > fe80::a00:27ff:fe3e:af77.dhcpv6-client: [udp sum ok] dhcp6 reply (xid=b33316 (IA_PD IAID:0 T1:0 T2:0 (status-code NoPrefixAvail)) (client-ID hwaddr/time type 1 time 626338757 0800274c4d5d) (server-ID hwaddr/time type 1 time 620825045 ac1f6bb1f2d4))

16:12:26.627149 IP6 (hlim 1, next-header UDP (17) payload length: 89) fe80::a00:27ff:fe3e:af77.dhcpv6-client > ff02::1:2.dhcpv6-server: [udp sum ok] dhcp6 solicit (xid=33e9bb (client-ID hwaddr/time type 1 time 626338757 0800274c4d5d) (elapsed-time 0) (option-request DNS-server DNS-search-list) (IA_PD IAID:0 T1:0 T2:0 (IA_PD-prefix ::/64 pltime:4294967295 vltime:4294967295)))

16:12:26.627393 IP6 (hlim 64, next-header UDP (17) payload length: 111) fe80::ae1f:6bff:feb1:f2d4.dhcpv6-server > fe80::a00:27ff:fe3e:af77.dhcpv6-client: [udp sum ok] dhcp6 advertise (xid=33e9bb (IA_PD IAID:0 T1:0 T2:0 (status-code NoPrefixAvail)) (client-ID hwaddr/time type 1 time 626338757 0800274c4d5d) (server-ID hwaddr/time type 1 time 620825045 ac1f6bb1f2d4))

Logs under /var/log/dhcpd.log seem to confirm as much:

Code Select Expand

Nov  6 16:08:50 core dhcpd: Solicit message from fe80::a00:27ff:fe3e:af77 port 546, transaction ID 0x8D5CE000
Nov  6 16:08:50 core dhcpd: Unable to pick client prefix: no IPv6 pools on this shared network
Nov  6 16:08:50 core dhcpd: Sending Advertise to fe80::a00:27ff:fe3e:af77 port 546
Nov  6 16:08:51 core dhcpd: Request message from fe80::a00:27ff:fe3e:af77 port 546, transaction ID 0x60E31500
Nov  6 16:08:51 core dhcpd: Unable to pick client prefix: no IPv6 pools on this shared network
Nov  6 16:08:51 core dhcpd: Sending Reply to fe80::a00:27ff:fe3e:af77 port 546

On the LAB OPNsense side:

OPNsense is on a host on the physical LAN running virtualbox. The VM has one bridged interface (WAN) and another private network (LAB). It can acquire an IPv6 address from the DHCPv6 server for the WAN address and I can reserve the lease based on it's DUID, however, but it cannot acquire a prefix. Running tcpdump on it's WAN interface yields the same results:

Code Select Expand

16:21:03.282083 IP6 (hlim 1, next-header UDP (17) payload length: 89) fe80::a00:27ff:fe3e:af77.dhcpv6-client > ff02::1:2.dhcpv6-server: [udp sum ok] dhcp6 solicit (xid=aa1cfd (client-ID hwaddr/time type 1 time 626338757 0800274c4d5d) (elapsed-time 0) (option-request DNS-server DNS-search-list) (IA_PD IAID:0 T1:0 T2:0 (IA_PD-prefix ::/64 pltime:4294967295 vltime:4294967295)))

16:21:03.282581 IP6 (hlim 64, next-header UDP (17) payload length: 111) fe80::ae1f:6bff:feb1:f2d4.dhcpv6-server > fe80::a00:27ff:fe3e:af77.dhcpv6-client: [udp sum ok] dhcp6 advertise (xid=aa1cfd (IA_PD IAID:0 T1:0 T2:0 (status-code NoPrefixAvail)) (client-ID hwaddr/time type 1 time 626338757 0800274c4d5d) (server-ID hwaddr/time type 1 time 620825045 ac1f6bb1f2d4))

16:21:04.289572 IP6 (hlim 1, next-header UDP (17) payload length: 84) fe80::a00:27ff:fe3e:af77.dhcpv6-client > ff02::1:2.dhcpv6-server: [udp sum ok] dhcp6 request (xid=624ddf (client-ID hwaddr/time type 1 time 626338757 0800274c4d5d) (server-ID hwaddr/time type 1 time 620825045 ac1f6bb1f2d4) (elapsed-time 0) (option-request DNS-server DNS-search-list) (IA_PD IAID:0 T1:0 T2:0 (status-code NoPrefixAvail)))

16:21:04.290115 IP6 (hlim 64, next-header UDP (17) payload length: 111) fe80::ae1f:6bff:feb1:f2d4.dhcpv6-server > fe80::a00:27ff:fe3e:af77.dhcpv6-client: [udp sum ok] dhcp6 reply (xid=624ddf (IA_PD IAID:0 T1:0 T2:0 (status-code NoPrefixAvail)) (client-ID hwaddr/time type 1 time 626338757 0800274c4d5d) (server-ID hwaddr/time type 1 time 620825045 ac1f6bb1f2d4))

I've also tried manually running dhclient -6 -P from a different VM and get similar results (status-code NoPrefixAvail).

I've tried tweaking a number of different options (soliciting prefix only, adjusting the prefix size and range accordingly, different RA modes) to no avail.

Any words of wisdom of how to tackle this? What should I look at next?