Best way to route a public /64 static ipv6 subnet from WAN to LAN clients

[RunicTowel409]

Hi there,

I got a switch for my rack from my datacenter provider which is connected to an upstream router from the datacenter out of my control. The switch has a public and static /64 IPv6 subnet assigned to it. So I can give clients connected to it any address from it, specify the datacenter router as gateway and that works.

Now I'm struggling a bit because I wanted to only connect one OPNsene to that switch and have my clients on another switch of mine behind OPNsense. And naturally the clients should get public IPv6 addresses from the subnet assigned to my switch and I don't want to NAT and all that because it's IPv6 after all 😅

This is my very first real world encounter with IPv6 so bear with me, but somehow this isn't as easy as I thought because (I think) OPNsense isn't forwarding any NDP from LAN to WAN. And from my very first readings so far (I think) I cannot use track interface due to the static setup (DHCPv6 is not existent). But I am really not sure.

How should a situation like that be set up ideally? Can I advertise the OPNsense somehow as router for the /64 or smaller subnets from it to the upstream router of the datacenter? I tried a bit with RA 'Router Only' on the WAN interface but that didn't seem to work but I'm also not sure if I did it right... well anyway, I'd be glad if someone could help me get some light into that IPv6 world. Thanks a lot!

[Maurice]

Without DHCPv6 Prefix Delegation, the upstream router would have to statically route the /64 to OPNsense. If the datacenter provider won't allow you to configure that (and instead relies on NDP), there isn't a lot OPNsense can do. NDP is only meant to be used between hosts on the same link. ND proxies can work around this, but unfortunately OPNsense doesn't have one (yet).

OPNsense can advertise itself as a router for (a subnet of) the /64 on the WAN interface, yes (RA "Router Only" mode and "Advertise Routes" for the desired subnet). But the upstream router of the datacenter will most likely ignore these Router Advertisements.

So your best option is to ask your datacenter provider for a routed prefix.

Cheers
Maurice

[RunicTowel409]

Hi Maurice,

thanks for your answer, I appreciated it very much.

The routed prefix (only /56 then) would be possible but really expansive. Especially if I'd want it for more than one rack. (I'm talking about over 100$/month😅 the DC is a small local one and has other advantages for me but their IPv6 pricing is hideous - though IPv6 services from them are quite new so I guess it will get a lot cheaper somewhere in future.)

Anyway, I seem to have found a (to me) sound looking, though not perfect, solution. I do claim the IPs for my clients as virtual IPs on the WAN without service binding and then NPTv6 them to private fc00:: address. That requires more manual configuration than I hoped for but for my somewhat 20 clients its okay for getting somewhat that resembles a real dual stack... would you see pitfalls with that approach?

Talking about NDP proxy, wouldn't it be easier to simply allow OPNsense to register to whole subnets (of any size) for NDP only similar to virtual single IPs? So I mean as far as I've seen NS on the link now and I understand it there is no initial IP claiming. I see the router's NS for all requests to anything in the /64. So if the OPNsense would simply always answer to send that package to it's link it should be nearly exactly the same as if a it was routed to it (besides a by 1 messed up hop limit). From here on every thing could work like normal without the messy NPTv6.
That could be a nice poor man's routing if I don't miss somehting but idk how much people have similar situations like this one

[Maurice]

> I'm talking about over 100$/month

Crazy.

> would you see pitfalls with that approach?

Other than having to configure each address individually, that's indeed doable. You should use addresses from fd00::/8 for your hosts though.
The downside is that source address selection prefers IPv4 addresses over ULAs when connecting to public IP addresses, so IPv6 won't get used a lot for outbound connections. Not an issue for inbound though.

> wouldn't it be easier to simply allow OPNsense to register to whole subnets (of any size) for NDP only similar to virtual single IPs?

OPNsense does have such a feature for IPv4 (Virtual IP Mode "Proxy ARP"), but unfortunately not (yet) for IPv6. Your use case is not unusual though and such a feature has been requested before, e. g.:

https://github.com/opnsense/plugins/issues/3609
https://github.com/opnsense/core/issues/7079

[RunicTowel409]

So, after configuring the third client, I got annoyed. Since I just needed a very simple NDP 'answer all' solution (to mimic routing) without a fancy proxy between interfaces, I wrote a simple script for it. In case someone else finds themselves in a similar situation or has a deeper understanding of OPNsense and wants to create a more integrated solution, here it is for reference.

Disclaimer: I do hate python but it ships with OPNsense and to do something quick (and somewhat dirty) it is a good tool. And the actual package filtering should get done by the kernel and not python here so the solution should be fine even if heavier loads of traffic hit the interface. And I mimicked the log line of binat to see what’s going on in the FW log though the entry is a little messy. Lastly, I use monit to keep a close eye on it. But obviously I only tested against the upstream router(s) of my DC so who knows if/how it would work in another environment.

Code: [Select]

#!/bin/sh

# Define your interface
INTERFACE="em0"

# Define the prefix you want to claim (this example is for 2aba:baba:abab:abab:ab::/72. Note the split
# in [48:4], [52:4], [56:1]. I haven't checked if a simpler syntax like [48:9] would work, but splitting
# into 4, 2, and 1 byte groups should be safer to be accepted)
BPF_FILTER="icmp6 and ip6[40] == 135 and ip6[48:4] == 0x2abababa and ip6[52:4] == 0xabababab and ip6[56:1] == 0xab"

####################################
# No more config needed below here #
####################################

# Variables
SCRIPT_NAME="ndp-prefix-bouncer.py"
SERVICE_NAME="ndp_prefix_bouncer"
SERVICE_FILE="/usr/local/etc/rc.d/$SERVICE_NAME"
SCRIPT_PATH="/usr/local/bin/$SCRIPT_NAME"
MONIT_CONFIG_PATH="/usr/local/etc/monit.opnsense.d/ndp_prefix_bouncer.conf"
PYTHON_PATH=$(which python3)

# Check if Python is installed correctly
if [ ! -x "$PYTHON_PATH" ]; then
    echo "Python 3 is not installed correctly or not found in the expected path."
    exit 1
fi

# Step 1: Install Python dependencies
echo "Installing Python dependencies..."
$PYTHON_PATH -m ensurepip --upgrade
$PYTHON_PATH -m pip install scapy netifaces

# Step 2: Create the Python script
echo "Creating the Python script at $SCRIPT_PATH..."
cat <<EOF > $SCRIPT_PATH
#!$PYTHON_PATH
from scapy.all import sniff, sendp, Ether, IPv6, ICMPv6ND_NS, ICMPv6ND_NA, ICMPv6NDOptDstLLAddr
import netifaces
import logging
import logging.handlers

INTERFACE = "$INTERFACE"
BPF_FILTER = "$BPF_FILTER"

def get_mac_address(interface):
    return netifaces.ifaddresses(interface)[netifaces.AF_LINK][0]['addr']

MAC_ADDRESS = get_mac_address(INTERFACE)

logger = logging.getLogger('NdpPrefixBouncer')
logger.setLevel(logging.INFO)
filter_log_handler = logging.handlers.SysLogHandler(address='/var/run/log', facility='local0')
formatter = logging.Formatter('filterlog: %(message)s')
filter_log_handler.setFormatter(formatter)
logger.addHandler(filter_log_handler)

def create_ndp_response(original_packet):
    target_ip = original_packet[ICMPv6ND_NS].tgt

    response = Ether(dst=original_packet[Ether].src, src=MAC_ADDRESS) / \\
                IPv6(src=target_ip, dst=original_packet[IPv6].src) / \\
                ICMPv6ND_NA(tgt=target_ip, R=1, S=1, O=1) / \\
                ICMPv6NDOptDstLLAddr(lladdr=MAC_ADDRESS)
   
    return response

def packet_callback(pkt):
    if ICMPv6ND_NS in pkt:
        target_ip = pkt[ICMPv6ND_NS].tgt
        src_ip = pkt[IPv6].src
        log_entry = f"0,,,0,{INTERFACE},match,Received_NS_for_destination_IP_and_sending_NDP_response_to_claim_it,in,6,0x00,0x00000,58,ipv6-icmp,58,,{src_ip},{target_ip},"
        logger.info(log_entry)
        response_packet = create_ndp_response(pkt)
        sendp(response_packet, iface=INTERFACE, verbose=False) 

sniff(iface=INTERFACE, prn=packet_callback, filter=BPF_FILTER, store=0)
EOF

chmod +x $SCRIPT_PATH

# Step 3: Create the service script to start at boot and monitore
echo "Creating service script at $SERVICE_FILE..."
cat <<EOF > $SERVICE_FILE
#!/bin/sh
#
# PROVIDE: ndp_prefix_bouncer
# REQUIRE: NETWORKING
# KEYWORD: shutdown
#
. /etc/rc.subr

name="ndp_prefix_bouncer"
rcvar="\${name}_enable"

# Specify the command and interpreter
command="/usr/local/bin/ndp-prefix-bouncer.py"
command_interpreter="/usr/local/bin/python3"
pidfile="/var/run/\${name}.pid"

# Set the start command to run the script in the background
start_cmd="ndp_prefix_bouncer_start"
stop_cmd="ndp_prefix_bouncer_stop"

# Function to start the service as a daemon
ndp_prefix_bouncer_start() {
    echo "Starting \${name}..."
    nohup \${command_interpreter} \${command} > /dev/null 2>&1 &
    echo \$! > \${pidfile}
}

# Function to stop the service
ndp_prefix_bouncer_stop() {
    if [ -f \${pidfile} ]; then
        kill -15 \$(cat \${pidfile})
        rm -f \${pidfile}
    else
        echo "\${name} is not running."
    fi
}

load_rc_config \$name
: \${ndp_prefix_bouncer_enable:="YES"}

run_rc_command "\$1"
EOF

chmod +x $SERVICE_FILE

if ! grep -q "^ndp_prefix_bouncer_enable=" /etc/rc.conf.local; then
    echo 'ndp_prefix_bouncer_enable="YES"' >> /etc/rc.conf.local
fi

# Step 4: Configure Monit
echo "Creating Monit configuration at $MONIT_CONFIG_PATH..."
cat <<EOF > $MONIT_CONFIG_PATH
check process ndp_prefix_bouncer with pidfile /var/run/ndp_prefix_bouncer.pid
    start program = "/usr/local/etc/rc.d/ndp_prefix_bouncer start"
    stop program = "/usr/local/etc/rc.d/ndp_prefix_bouncer stop"
    if does not exist then restart
    if 3 restarts within 5 cycles then alert
EOF

echo "Reloading Monit configuration..."
service monit reload

# Step 5: Go!
echo "Starting NDP Prefix Bouncer service..."
service ndp_prefix_bouncer start

echo "Installation complete. NDP Prefix Bouncer is running and being monitored by Monit."


[Maurice]

Wow, crafting your own Ethernet frames, brutal!
But seems pretty sophisticated and looks like it does the job. Definitely a creative approach.

[RunicTowel409]

The poor man's world is a creative one 😅 though I feel a bit sorry for the upstream router. The poor guy seems to not be allowed to drop neighbor cache entries without ensuring they are gone by a NS before. But all NS will be answered by the script for all time to come keeping every IP every pinged in its cache... Oh well, but I guess/hope for it at some point it would do a cache flush eventually.

Anyway, besides that so far it works like I wanted. I still will watch it some more before I would consider it for anything productive.