Cannot ping/traceroute/fetch/curl - But nslookup works and get an IP by WAN DHCP

[luckylinux]

I'm a bit at a loss here after 2 Days of trying to get OPNSense to work in a VM on top of Proxmox VE hosted at Hetzner (Dedicated Server).

Setup is Bridged (using Linux Bridge). IP Routing (on Proxmox VE) is DISABLED via /etc/sysctl.d/*.conf.

1st Issue: Linux Traceroute (UDP) behaves VERY differently than Windows Traceroute (ICMP). I didn't know that

2nd Issue. Linux Traceroute (UDP) uses port 53 by Default (DFNS), which I blocked outbound in my network at Home  .

Solution:

Code: [Select]

traceroute -U -p 33434 xx.xxx.xxx.xxx
UDP seems to be very finicky though, especially at the last hop (Hetzner Gateway -> my Server) ...

At least with OPNSense, that is.

From my Network at Home I can do the following
Traceroute Home Network -> Proxmox Host:

• PASS UDP port 33434-33474: traceroute -U -p 33434 xxx.xxx.xxx.proxmox
• PASS TCP port 33434-33474: traceroute -T -p 33434 xxx.xxx.xxx.proxmox
• PASS TCP port 80: traceroute -T xxx.xxx.xxx.proxmox
• PASS ICMP: traceroute -I xxx.xxx.xxx.proxmox
• PASS PING: ping xxx.xxx.xxx.proxmox

Traceroute Home Network -> OPNSense VM:

• FAILS UDP port 33434-33474 (last hop fails): traceroute -U -p 33434 xx.xxx.xxx.opnsense
• PASS TCP port 33434-33474: traceroute -T -p 33434 xxx.xxx.xxx.opnsense
• PASS TCP port 80: traceroute -T xxx.xxx.xxx.opnsense
• PASS ICMP: traceroute -I xxx.xxx.xxx.opnsense
• PASS PING: ping xxx.xxx.xxx.opnsense

The more "interesting" stuff happens OUTBOUND from the Server(s) though

Traceroute Proxmox Host -> 142.250.186.46 (one of Google "A" Records):

• PASS UDP port 33434-33474: traceroute -U -p 33434 142.250.186.46
• FAILS TCP port 33434-33474: traceroute -T -p 33434 142.250.186.46
• PASS TCP port 80: traceroute -T 142.250.186.46
• PASS ICMP: traceroute -I 142.250.186.46
• PASS PING: ping 142.250.186.46

Traceroute OPNSense VM -> 142.250.186.46 (one of Google "A" Records):

• FAILS UDP port 33434-33474: traceroute -P UDP -p 33434 142.250.186.46
• FAILS TCP port 33434-33474: traceroute -P TCP -p 33434 142.250.186.46
• FAILS TCP port 80: traceroute -P TCP 142.250.186.46
• FAILS ICMP: traceroute -P ICMP 142.250.186.46
• FAILS PING: ping 142.250.186.46

And worst still, it's all intermittent.
Heck I cannot even ping my Hetzner Gateway from the OPNSense VM.
I can get a WAN IP via DHCP from Hetzner DHCP Server without Issues.
I can get DNS queries working across without Issues (and get a reply from Hetzner DNS Server).
I can connect via OpenVPN from Home to OPNSense VM without Issues

But for some weird reason, I cannot do ANYTHING from the OPNSense Firewall itself.
Not curl, not fetch, not updates, nothing. Even if I try to disable IPv6, etc. Nothing helps.

I am really lost as to why it's going across in one direction but not the other. Firewall rules should be fine, checked both on the Proxmox VE Firewall Rules **FOR** the OPNSense VM as well as the rules in OPNSense itself.

Heck Outbound rules are even more loose than Inbound Rules (on both Firewalls), so why it's not working it's beyond me  .

Maybe something to do with the Hardening of the /etc/sysctl.d/*.conf I did ?

Kernel Hardening

Code: [Select]

# eBPF exposes quite large attack surface and must therefore be restricted
# These sysctls restrict eBPF to the CAP_BPF capability (CAP_SYS_ADMIN on kernel versions prior to 5.8) and enable JIT hardening techniques, such as constant blinding.
kernel.unprivileged_bpf_disabled=1
net.core.bpf_jit_harden=2

# Restricts loading TTY line disciplines to the CAP_SYS_MODULE capability to prevent unprivileged attackers from loading vulnerable line disciplines with the TIOCSETD ioctl,
dev.tty.ldisc_autoload=0

#  The userfaultfd() syscall is often abused to exploit use-after-free flaws. Due to this, this sysctl is used to restrict this syscall to the CAP_SYS_PTRACE capability.
vm.unprivileged_userfaultfd=0

# This only permits symlinks to be followed when outside of a world-writable sticky directory,
# when the owner of the symlink and follower match or when the directory owner matches the symlink's owner.
# This also prevents hardlinks from being created by users that do not have read/write access to the source file. Both of these prevent many common TOCTOU races.
fs.protected_symlinks=1
fs.protected_hardlinks=1

# These prevent creating files in potentially attacker-controlled environments, such as world-writable directories, to make data spoofing attacks more difficult.
fs.protected_fifos=2
fs.protected_regular=2

# Process that run with elevated privileges may still dump their memory even after these settings.
# To prevent them from doing so, set the following via sysctl:
fs.suid_dumpable=0

# Similar to core dumps, swapping or paging copies parts of memory to disk, which can contain sensitive information.
# The kernel should be configured to only swap if absolutely necessary
vm.swappiness=1

# dmesg is the kernel log. It exposes a large amount of useful kernel debugging information, but this can often leak sensitive information, such as kernel pointers.
# Changing the above sysctl restricts the kernel log to the CAP_SYSLOG capability.
kernel.dmesg_restrict=1

# Despite the value of dmesg_restrict, the kernel log will still be displayed in the console during boot.
# Malware that is able to record the screen during boot may be able to abuse this to gain higher privileges. This option prevents those information leaks.
# This must be used in combination with certain boot parameters described below to be fully effective.
kernel.printk=3 3 3 3

# A kernel pointer points to a specific location in kernel memory.
# These can be very useful in exploiting the kernel, but kernel pointers are not hidden by default — it is easy to uncover them by, for example, reading the contents of /proc/kallsyms.
# This setting aims to mitigate kernel pointer leaks. Alternatively, you can set kernel.kptr_restrict=1 to only hide kernel pointers from processes without the CAP_SYSLOG capability.
kernel.kptr_restrict=2


# Performance events add considerable kernel attack surface and have caused abundant vulnerabilities
# This sysctl restricts all usage of performance events to the CAP_PERFMON capability (CAP_SYS_ADMIN on kernel versions prior to 5.8).
# Be aware that this sysctl also requires a kernel patch that is only available on certain distributions.
# Otherwise, this setting is equivalent to kernel.perf_event_paranoid=2, which only restricts a subset of this functionality.
# kernel.perf_event_paranoid=3


# The SysRq key exposes a lot of potentially dangerous debugging functionality to unprivileged users.
# Contrary to common assumptions, SysRq is not only an issue for physical attacks, as it can also be triggered remotely.
# The value of this sysctl makes it so that a user can only use the secure attention key, which will be necessary for accessing root securely. Alternatively, you can simply set the value to 0 to disable SysRq completely.
kernel.sysrq=0

# ptrace is a system call that allows a program to alter and inspect another running process, which allows attackers to trivially modify the memory of other running programs.
# This restricts usage of ptrace to only processes with the CAP_SYS_PTRACE capability.
# Alternatively, set the sysctl to 3 to disable ptrace entirely.
kernel.yama.ptrace_scope=2

# Forbid creating Kernel Core Dumps
kernel.core_pattern=|/bin/false

# Controls whether core dumps will append the PID to the core filename
# Useful for debugging multi-threaded applications
kernel.core_uses_pid = 1

# Ensure that we still allow sufficient logging
kernel.printk_ratelimit=1
kernel.printk_ratelimit_burst=50

Network Hardening:

Code: [Select]

# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# WE MUST ALSO SPECIFY EXPLICITELY eth0 HERE (maybe also vmbr0)
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

# Disable IPv4 multicast igmp report on local link plug
net.ipv4.igmp_link_local_mcast_reports=0

# Ignore IPv4 ICMP echo packets sent to a broadcast address
net.ipv4.icmp_echo_ignore_broadcasts=1

# Ignore IPv6 ICMP echo packets sent to anycast & multicast address
# This works very similarly to net.ipv4.icmp_echo_ignore_broadcasts
net.ipv6.icmp.echo_ignore_anycast=1
net.ipv6.icmp.echo_ignore_multicast=1

# This setting avoids filling up log files with unnecessary error messages coming from invalid responses to broadcast frames.
net.ipv4.icmp_ignore_bogus_error_responses=1

# Don't ignore directed pings
# This will also make PING the Proxmox VE Host Impossible !
net.ipv4.icmp_echo_ignore_all=0
net.ipv6.icmp.echo_ignore_all=0

# Disables the acceptance of ICMP redirect messages.
# Make sure no one can modify the routing Table
# Ignore IPv4 ICMP redirect messages
net.ipv4.conf.default.accept_redirects=0
net.ipv4.conf.all.accept_redirects=0
net.ipv4.conf.eth0.accept_redirects=0
#net.ipv4.conf.vmbr0.accept_redirects=0

# Ignore IPv6 ICMP redirect messages
net.ipv6.conf.default.accept_redirects=0
net.ipv6.conf.all.accept_redirects=0
net.ipv6.conf.eth0.accept_redirects=0
#net.ipv6.conf.vmbr0.accept_redirects=0

# Accepting 'secure' ICMP redirects (from those gateways listed as default gateways) has few legitimate uses.
# It should be disabled unless it is absolutely required.
net.ipv4.conf.default.secure_redirects=0
net.ipv4.conf.all.secure_redirects=0
net.ipv4.conf.eth0.secure_redirects=0
#net.ipv4.conf.vmbr0.secure_redirects=0

# A node should not send out IPv4 ICMP redirects, unless it acts as a router.
net.ipv4.conf.default.send_redirects=0
net.ipv4.conf.all.send_redirects=0
net.ipv4.conf.eth0.send_redirects=0
#net.ipv4.conf.vmbr0.send_redirects=0

# Do NOT act as a Router
# IP forwarding should only be enabled on systems acting as routers.

# Disable IPv4 Packet Forwarding
net.ipv4.ip_forward=0

net.ipv4.conf.default.forwarding=0
net.ipv4.conf.all.forwarding=0
net.ipv4.conf.eth0.forwarding=0
#net.ipv4.conf.vmbr0.forwarding=0

# Disable IPv6 Packet Forwarding
net.ipv6.conf.default.forwarding=0
net.ipv6.conf.all.forwarding=0
net.ipv6.conf.eth0.forwarding=0
#net.ipv6.conf.vmbr0.forwarding=0

# No source routed packets here
# This setting disables the acceptance of packets with the SSR option set in the IPv4 packet header.
# Packets that use Source Routing will be rejected.
# This prevents IP packet redirection, that is redirection to a host behind a firewall, that is not directly reachable otherwise.
net.ipv4.conf.default.accept_source_route=0
net.ipv4.conf.all.accept_source_route=0
net.ipv4.conf.eth0.accept_source_route=0
#net.ipv4.conf.vmbr0.accept_source_route=0

# Do not accept packets with SRR option
net.ipv6.conf.default.accept_source_route=0
net.ipv6.conf.all.accept_source_route=0
net.ipv6.conf.eth0.accept_source_route=0
#net.ipv6.conf.vmbr0.accept_source_route=0

# Disable IPv6 Router Advertisements, since they can result in a man-in-the-middle attack
net.ipv6.conf.default.accept_ra=0
net.ipv6.conf.all.accept_ra=0
net.ipv6.conf.eth0.accept_ra=0
#net.ipv6.conf.vmbr0.accept_ra=0

# Turn on and log spoofed, source routed, and redirect packets
net.ipv4.conf.default.log_martians=1
net.ipv4.conf.all.log_martians=1
net.ipv4.conf.eth0.log_martians=1
#net.ipv4.conf.vmbr0.log_martians=1

# Turn on syncookies for SYN flood attack protection
# This enables TCP SYN Cookie Protection for IPv4 and IPv6.
net.ipv4.tcp_syncookies=1

# Turn on reverse path filter in Strict Mode
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1
net.ipv4.conf.eth0.rp_filter=1
#net.ipv4.conf.vmbr0.rp_filter=1

# Disable forwarding of all multicast packets on all interfaces
net.ipv4.conf.default.mc_forwarding=0
net.ipv4.conf.all.mc_forwarding=0
net.ipv4.conf.eth0.mc_forwarding=0
#net.ipv4.conf.vmbr0.mc_forwarding=0

net.ipv6.conf.default.mc_forwarding=0
net.ipv6.conf.all.mc_forwarding=0
net.ipv6.conf.eth0.mc_forwarding=0
#net.ipv6.conf.vmbr0.mc_forwarding=0

# Drop RST packets for sockets in the time-wait state
net.ipv4.tcp_rfc1337=1

# Disable TCP SACK
net.ipv4.tcp_sack=0
net.ipv4.tcp_dsack=0
net.ipv4.tcp_fack=0

# TCP timestamps also leak the system time and should therefore be disabled
net.ipv4.tcp_timestamps=0

# Enable IPv6 Privacy Extensions
net.ipv6.conf.default.use_tempaddr=2
net.ipv6.conf.all.use_tempaddr=2
#net.ipv6.conf.eth0.use_tempaddr=2
#net.ipv6.conf.vmbr0.use_tempaddr=2

# The TCP SYN backlog defines the number of SYN packets that are queued for further processing.
# Once the queue limit is exceeded, all new incoming SYN-packets are dropped and new TCP connections will not be possible (or the SYN cookie protection kicks in)
# Increasing this value improves the protection against TCP SYN flood attacks.
net.ipv4.tcp_max_syn_backlog=4096

#########################################################
############ NEWLY INTRODUCED ON 2024-05-17 #############
#########################################################

# Number of global unicast IPv6 addresses can be assigned to each interface
net.ipv6.conf.default.max_addresses=1
net.ipv6.conf.all.max_addresses=1
net.ipv6.conf.eth0.max_addresses=1
#net.ipv6.conf.vmbr0.max_addresses=1

# Number of neighbor solicitations to send out per address
net.ipv6.conf.default.dad_transmits=0
net.ipv6.conf.all.dad_transmits=0
net.ipv6.conf.eth0.dad_transmits=0
#net.ipv6.conf.vmbr0.dad_transmits=0

# Router advertisements can cause the system to assign a global unicast address to an interface
net.ipv6.conf.default.autoconf=0
net.ipv6.conf.all.autoconf=0
net.ipv6.conf.eth0.autoconf=0
#net.ipv6.conf.vmbr0.autoconf=0

# Setting controls whether the system will accept Hop Limit settings from a router advertisement
net.ipv6.conf.default.accept_ra_defrtr=0
net.ipv6.conf.all.accept_ra_defrtr=0
net.ipv6.conf.eth0.accept_ra_defrtr=0
#net.ipv6.conf.vmbr0.accept_ra_defrtr=0

# Learn prefix information in router advertisement
net.ipv6.conf.default.accept_ra_pinfo=0
net.ipv6.conf.all.accept_ra_pinfo=0
net.ipv6.conf.eth0.accept_ra_pinfo=0
#net.ipv6.conf.vmbr0.accept_ra_pinfo=0

# Do not accept Router Preference from RA
net.ipv6.conf.default.accept_ra_rtr_pref=0
net.ipv6.conf.all.accept_ra_rtr_pref=0
net.ipv6.conf.eth0.accept_ra_rtr_pref=0
#net.ipv6.conf.vmbr0.accept_ra_rtr_pref=0

# Number of Router Solicitations to send until assuming no routers are present
net.ipv6.conf.default.router_solicitations=0
net.ipv6.conf.all.router_solicitations=0
net.ipv6.conf.eth0.router_solicitations=0
#net.ipv6.conf.vmbr0.router_solicitations=0

# Newly Introduced on 2024-05-20
# Source: https://forum.level1techs.com/t/solved-linux-vm-bridged-network-arp-ok-no-ip-traffic/191172/14
net.bridge.bridge-nf-call-iptables=0
net.bridge.bridge-nf-call-ip6tables=0
net.bridge.bridge-nf-call-nf_call_arptables=0

# eBPF exposes quite large attack surface and must therefore be restricted
# These sysctls restrict eBPF to the CAP_BPF capability (CAP_SYS_ADMIN on kernel versions prior to 5.8) and enable JIT hardening techniques, such as constant blinding.
#net.core.bpf_jit_harden=2

# Logging
net.core.message_cost=5
net.core.message_burst=50

The Hardening is mostly due to several "MAC Abuse Emails" that Hetzner sent me, after apparently my Server was replying to / forwarding Broadcast traffic.

I also have /usr/local/bin/disable-network-flooding

Code: [Select]

#!/bin/bash

# Optional Argument for Verbose Output
verbose=${1-""}


# Disable Unicast flooding
# This **might** also disable OPNSense Communication
# Try to disable it for now
mapfile -t list < <( find /sys -iname *unicast_flood* )

for item in "${list[@]}"
do
    # Echo
    if [[ -n "${verbose}" ]]
    then
       echo "Executing: echo 0 > ${item}"
    fi

    # Disable unicast flooding on all interfaces
    echo 0 > $item
done

I also turned on the "Promiscous Mode" on the OPNSense WAN Interface (and forced the MAC Address to be the same as the VM MAC Address, just to make sure that I don't get another MAC Abuse Email just for that  ).

It didn't help unfortunately.

I have several Outbound Rules for the WAN Interface and everything in the Logs appear green.
Proxmox VE Firewall didn't seem to make a Difference whether ON or OFF (I turned it back ON as I don't want yet another Abuse Email  ).

In dmesg I can see some "martians" were logged, however I am not sure how significant that is:

Code: [Select]

[  104.721147] IPv4: martian source 172.16.1.1 from 172.16.1.1, on dev vmbr2001
[  104.721153] ll header: 00000000: ff ff ff ff ff ff bc 24 XX XX XX XX XX XX
[  192.017638] IPv4: martian source xx.xxx.xxx.proxmox from xx.xxx.xxx.opnsense, on dev fwbr100i0
[  192.017643] ll header: 00000000: ff ff ff ff ff ff XX XX XX XX XX XX XX XX
[ 1714.411527] IPv4: martian source 172.16.1.1 from 172.16.1.1, on dev vmbr2001
[ 1714.411533] ll header: 00000000: ff ff ff ff ff ff bc 24 XX XX XX XX XX XX

EDIT 1: Spoke too soon, also DNS Seems to go down after a while / reboot etc  .

[luckylinux]

I also see these in the OPNsense Logs.

Not sure what is going on  .

Code: [Select]

2024-05-21T07:57:00 Notice kernel pf: State failure on: 1 | 5
2024-05-21T07:57:00 Notice kernel pf: BAD state: TCP out wire: 104.16.199.238:443 XX.XXX.XXX.XXX:56002 stack: - [lo=143274092 high=143339628 win=65228 modulator=0 wscale=7] [lo=581943148 high=582008428 win=65160 modulator=0 wscale=13] 4:2 SA seq=1080993867 (1080993867) ack=143274092 len=0 ackskew=0 pkts=6:3 dir=in,rev
2024-05-21T07:56:59 Notice kernel pf: State failure on: 1 | 5
2024-05-21T07:56:59 Notice kernel pf: BAD state: TCP out wire: 104.16.199.238:443 XX.XXX.XXX.XXX:56002 stack: - [lo=143274092 high=143339628 win=65228 modulator=0 wscale=7] [lo=581943148 high=582008428 win=65160 modulator=0 wscale=13] 4:2 SA seq=1080993867 (1080993867) ack=143274092 len=0 ackskew=0 pkts=6:3 dir=in,rev
2024-05-21T07:56:48 Notice kernel pf: State failure on: 1 | 5
2024-05-21T07:56:48 Notice kernel pf: BAD state: TCP out wire: 146.75.122.49:443 XX.XXX.XXX.XXX:25769 stack: - [lo=4086063453 high=4086128989 win=65228 modulator=0 wscale=7] [lo=2954974221 high=2955039501 win=65535 modulator=0 wscale=9] 4:2 SA seq=3200134137 (3200134137) ack=4086063453 len=0 ackskew=0 pkts=5:2 dir=in,rev
2024-05-21T07:56:47 Notice kernel pf: State failure on: 1 | 5
2024-05-21T07:56:47 Notice kernel pf: BAD state: TCP out wire: 146.75.122.49:443 XX.XXX.XXX.XXX:25769 stack: - [lo=4086063453 high=4086128989 win=65228 modulator=0 wscale=7] [lo=2954974221 high=2955039501 win=65535 modulator=0 wscale=9] 4:2 SA seq=3200134137 (3200134137) ack=4086063453 len=0 ackskew=0 pkts=5:2 dir=in,rev
2024-05-21T07:56:44 Notice kernel pf: State failure on: 1 | 5
2024-05-21T07:56:44 Notice kernel pf: BAD state: TCP out wire: 104.16.199.238:443 XX.XXX.XXX.XXX:56002 stack: - [lo=143274092 high=143339628 win=65228 modulator=0 wscale=7] [lo=581943148 high=582008428 win=65160 modulator=0 wscale=13] 4:2 SA seq=826542212 (826542212) ack=143274092 len=0 ackskew=0 pkts=5:3 dir=in,rev
2024-05-21T07:56:43 Notice kernel pf: State failure on: 1 | 5
2024-05-21T07:56:43 Notice kernel pf: BAD state: TCP out wire: 104.16.199.238:443 XX.XXX.XXX.XXX:56002 stack: - [lo=143274092 high=143339628 win=65228 modulator=0 wscale=7] [lo=581943148 high=582008428 win=65160 modulator=0 wscale=13] 4:2 SA seq=826542212 (826542212) ack=143274092 len=0 ackskew=0 pkts=5:3 dir=in,rev

[luckylinux]

Nothing I did seemed to help.

Finally I CHECKED "Disable reply-to" (on WAN Rules) In Firewall -> Settings -> Advanced and it works as expected (finally !)  .