Messages

Should OPNSense be assigning a public IPv6 IP to my workstation in this context? My understanding of IPv6 is limited, so I apologize if this is a dumb question.

That depends on if you have configured router advertisements, but if you only really get a single /64, then you might need to enable DHCPv6.

It seems that for whatever reason, DHCPv6 dislikes my DNS address despite it working previously?

The DNS address in question is an ULA,fdbc:baa4:65a9::4

Is there a place where that conf file would be persisteted so I can look at it? I cannot find the file in /etc/dhcpdv6.conf?

Running OPNsense 24.7.3_1

Code Select Expand


/services_dhcpv6.php: The command '/usr/local/sbin/dhcpd -6 -user dhcpd -group dhcpd -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid ix0_vlan10 ix0_vlan105 bridge0 ix0_vlan20 ix0_vlan110' returned exit code '1', the output was 'Internet Systems Consortium DHCP Server 4.4.3-P1 Copyright 2004-2022 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ /etc/dhcpdv6.conf line 20: semicolon expected. option custom-lan-0 fdbc: ^ Configuration file errors encountered -- exiting If you think you have received this message due to a bug rather than a configuration issue please read the section on submitting bugs on either our web page at www.isc.org or in the README file before submitting a bug. These pages explain the proper process and the information we find helpful for debugging. exiting.'


Are you sure your internet provider is not dropping outbound port 53?

Thanks, that was it.

However after disabling it, I had to manually stop syslog-ng and only after I was able to unmount the tmpfs.

Hey,

Random question, but does your containers be also unable to access internet via the gw? If yes, and firewall log shows state violations, that's probably related to your issue.

It seems that my logs are filling up the tempfs quite often, and to my understanding, it should be saved to my zfs pool instead of an tmpfs?

Code Select Expand


ilesystem                                     Size    Used   Avail Capacity  Mounted on
zroot/ROOT/default                             194G     19G    175G    10%    /
devfs                                          1.0K    1.0K      0B   100%    /dev
/dev/gpt/efiboot0                              260M    1.8M    258M     1%    /boot/efi
zroot                                          175G     96K    175G     0%    /zroot
zroot/tmp                                      175G    2.1M    175G     0%    /tmp
zroot/var/mail                                 175G    136K    175G     0%    /var/mail
zroot/var/crash                                175G    104K    175G     0%    /var/crash
zroot/usr/ports                                177G    1.6G    175G     1%    /usr/ports
zroot/usr/home                                 175G     96K    175G     0%    /usr/home
zroot/jails                                    175G    120K    175G     0%    /jails
zroot/var/tmp                                  175G    200K    175G     0%    /var/tmp
zroot/var/log                                  176G    1.1G    175G     1%    /var/log
zroot/usr/src                                  176G    559M    175G     0%    /usr/src
zroot/var/audit                                175G     96K    175G     0%    /var/audit
zroot/vms                                      183G    7.7G    175G     4%    /zroot/vms
zroot/jails/containers                         175G    104K    175G     0%    /jails/containers
zroot/jails/templates                          175G    104K    175G     0%    /jails/templates
zroot/jails/media                              175G    191M    175G     0%    /jails/media
zroot/vms/podman                               175G    132K    175G     0%    /zroot/vms/podman
zroot/jails/templates/13.2-RELEASE             176G    550M    175G     0%    /jails/templates/13.2-RELEASE
zroot/jails/containers/dnsdist                 176G    978M    175G     1%    /jails/containers/dnsdist
zroot/jails/containers/bind                    176G    638M    175G     0%    /jails/containers/bind
zroot/jails/containers/unifi                   177G    2.0G    175G     1%    /jails/containers/unifi
zroot/jails/templates/13.2-RELEASE-skeleton    175G    104K    175G     0%    /jails/templates/13.2-RELEASE-skeleton
tmpfs                                          3.2G     76M    3.1G     2%    /var/log
devfs                                          1.0K    1.0K      0B   100%    /var/dhcpd/dev
devfs                                          1.0K    1.0K      0B   100%    /var/captiveportal/zone0/dev
zroot/vms/FedoraIoT                            175G    120K    175G     0%    /zroot/vms/FedoraIoT


Something to note on the opensense-revert command is that if somehow you got unreacheable v6 default as a result of not having working WAN IPv6 address due to the dhcp6c issue, you may be unable to also revert.

The Github issue I made is this one for this: https://github.com/opnsense/core/issues/7595

I had to do the following:

• Disable IPv6 on WAN
• Run the revert
• Reboot
• See that it did not fix it so another revert, this time to 24.1.0
• Reboot
• Re-enable IPv6
• Use the pkg install that's in this thread.
• Now I have working IPv6 WAN dhcp6 again

Code Select Expand


opnsense-revert -r 24.1.8 dhcp6c


Code Select Expand


pkg add -f https://pkg.opnsense.org/FreeBSD:13:amd64/snapshots/misc/dhcp6c-20240607_1.pkg


It seems that I had configured Cilium in a way where it was not communicating with the nodes over the L3, but instead went thought the gateway/firewall.

Now it works better as it's routing inside the L3 instead, aka had to enable the autoDirectNodeRoutes Helm value as otherwise it will not add the PodCIDR routes to the kernel.

https://docs.cilium.io/en/stable/network/concepts/routing/#native-routing:

In native routing mode, Cilium will delegate all packets which are not addressed to another local endpoint to the routing subsystem of the Linux kernel. This means that the packet will be routed as if a local process would have emitted the packet. As a result, the network connecting the cluster nodes must be capable of routing PodCIDRs.

In order to run the native routing mode, the network connecting the hosts on which Cilium is running on must be capable of forwarding IP traffic using addresses given to pods or other workloads.

I'm really at a loss as to how I should configure my Firewall to properly not have connection hiccups. The question now is that how would I properly address this problem?

I'm using Cilium as my CNI, with native routing, which means that all pod traffic show up. My current problem is that I'm facing interminiant connection timeouts when connecting to an VIP LB address that is announced to my firewall via BGP from Cilium.

I have disabled state tracking on the rules allowing POD_CIDRs to traverse the firewall, which is more than likely part of the problem.


Kubernetes nodes are on 10.0.105.0/24, while one is also at 192.168.2.129 (single-node cluster, having this issue).
BGP LB prefix for the first cluster is 192.168.10.0/24. Firewall is at 192.168.2.1 (10.0.105.1).
When I do:

Code Select Expand

mtr -T -P 443 192.168.10.3

I can definitely see that there's packets getting dropped, and my Cilium's hubble shows response packets getting dropped due to TTL exceeded.

Code Select Expand


May  8 16:49:26.517: 192.168.2.129:33011 (world-ipv4) <> networking/nginx-internal-controller-6cc54b48b7-7z2js:443 (ID:106123) TTL exceeded DROPPED (TCP Flags: SYN)
May  8 16:49:26.520: 192.168.2.129:33015 (world-ipv4) <> networking/nginx-internal-controller-6cc54b48b7-xjsxs:443 (ID:106123) TTL exceeded DROPPED (TCP Flags: SYN)


It seems that I had to disable logging to disk, and re-enable it for it to populate logging configs using newer template after upgrading to 24.1.

Actually it seems like pretty much nothing is logging anything in /var/log. syslog is apperantly running, but it's not even saying anything anywhere?

It seems that for whatever reason, when I look at my firewall live view, it still lists previous things (from 1st of this month) and the HTTP request it's using also returns old data. This seems like a bug.

The path `/api/diagnostics/firewall/log/?digest=d21b05e4ea74d1b86ba00545605ef869&limit=1000` seems to return always the same data thus the page doesn't show anything new either?

The raw firewall logs show up-to-date information. How should I go about getting it to update? I used the auto-reflesh, but it doesn't currently do anything?

Hey,

Thanks for that. I got confused as the interface page wouldn't let me do it so I was able to do it via Virtual IPs and set the type to Alias IP.

Let me try to be more precise.

I'm trying to have IPv6 TRACK from WAN on my LAN interfaces, whilst having a separate ULA prefix on LAN for BGP communication (Kubernetes) due to to how the IPv6 delegation on WAN tends to change, with it my other interfaces too as they track it so it works like it's supposed to.

The ULA (Unicast Local Address) has nothing to do with my upstream ISP's DHCPv6 and thus prefix delegation.

Example:

I have an host in LAN, that has 2 IPv6 address from two separate subnets: 2001:14ba::* and fd9d:7a72:44eb:d::/64 before moving to opensense. I want to keep that. The first subnet is from a DHCPv6-PD though the track interface.

I DO NOT want DHCPv6 on LAN, as that's handled via SLAAC / Router Advertisements.


The question thus is: Can I currently not have a track IPv6 in use for a LAN interface AND manual static IPv6 prefix on the same interface? If it's possible, what's the correct way to do so?

I don't know how it wasn't clear on the first post, prefix/subnet to me are the same thing or at least they are usually used interchangeably.

It seems that there's no way to currently add more prefixes to LAN other than static or track?
I think it should be allowed to also allow for ULA prefix to be set for interfaces, as if you're using a DHCPv6 delegated address for BGP neighbors, that's going to be very painful to have to change when your prefix happens to change instead of just doing BGP over the ULA?

The IPv6 standard allows for multiple prefixes, so why not opnsense?