1
General Discussion / Cannot ping/traceroute/fetch/curl - But nslookup works and get an IP by WAN DHCP
« Last post by luckylinux on Today at 08:22:39 pm »I'm a bit at a loss here after 2 Days of trying to get OPNSense to work in a VM on top of Proxmox VE hosted at Hetzner (Dedicated Server).
1st Issue: Linux Traceroute (UDP) behaves VERY differently than Windows Traceroute (ICMP). I didn't know that
2nd Issue. Linux Traceroute (UDP) uses port 53 by Default (DFNS), which I blocked outbound in my network at Home .
Solution:
UDP seems to be very finicky though, especially at the last hop (Hetzner Gateway -> my Server) ...
At least with OPNSense, that is.
From my Network at Home I can do the following
Traceroute Home Network -> Proxmox Host:
Traceroute Home Network -> OPNSense VM:
The more "interesting" stuff happens OUTBOUND from the Server(s) though
Traceroute Proxmox Host -> 142.250.186.46 (one of Google "A" Records):
Traceroute OPNSense VM -> 142.250.186.46 (one of Google "A" Records):
And worst still, it's all intermittent.
Heck I cannot even ping my Hetzner Gateway from the OPNSense VM.
I can get a WAN IP via DHCP from Hetzner DHCP Server without Issues.
I can get DNS queries working across without Issues (and get a reply from Hetzner DNS Server).
I can connect via OpenVPN from Home to OPNSense VM without Issues
But for some weird reason, I cannot do ANYTHING from the OPNSense Firewall itself.
Not curl, not fetch, not updates, nothing. Even if I try to disable IPv6, etc. Nothing helps.
I am really lost as to why it's going across in one direction but not the other. Firewall rules should be fine, checked both on the Proxmox VE Firewall Rules **FOR** the OPNSense VM as well as the rules in OPNSense itself.
Heck Outbound rules are even more loose than Inbound Rules (on both Firewalls), so why it's not working it's beyond me .
Maybe something to do with the Hardening of the /etc/sysctl.d/*.conf I did ?
Kernel Hardening
Network Hardening:
The Hardening is mostly due to several "MAC Abuse Emails" that Hetzner sent me, after apparently my Server was replying to / forwarding Broadcast traffic.
I also have /usr/local/bin/disable-network-flooding
1st Issue: Linux Traceroute (UDP) behaves VERY differently than Windows Traceroute (ICMP). I didn't know that
2nd Issue. Linux Traceroute (UDP) uses port 53 by Default (DFNS), which I blocked outbound in my network at Home .
Solution:
Code: [Select]
traceroute -U -p 33434 xx.xxx.xxx.xxx
UDP seems to be very finicky though, especially at the last hop (Hetzner Gateway -> my Server) ...
At least with OPNSense, that is.
From my Network at Home I can do the following
Traceroute Home Network -> Proxmox Host:
- PASS UDP port 33434-33474: traceroute -U -p 33434 xxx.xxx.xxx.proxmox
- PASS TCP port 33434-33474: traceroute -T -p 33434 xxx.xxx.xxx.proxmox
- PASS TCP port 80: traceroute -T xxx.xxx.xxx.proxmox
- PASS ICMP: traceroute -I xxx.xxx.xxx.proxmox
- PASS PING: ping xxx.xxx.xxx.proxmox
Traceroute Home Network -> OPNSense VM:
- FAILS UDP port 33434-33474 (last hop fails): traceroute -U -p 33434 xx.xxx.xxx.opnsense
- PASS TCP port 33434-33474: traceroute -T -p 33434 xxx.xxx.xxx.opnsense
- PASS TCP port 80: traceroute -T xxx.xxx.xxx.opnsense
- PASS ICMP: traceroute -I xxx.xxx.xxx.opnsense
- PASS PING: ping xxx.xxx.xxx.opnsense
The more "interesting" stuff happens OUTBOUND from the Server(s) though
Traceroute Proxmox Host -> 142.250.186.46 (one of Google "A" Records):
- PASS UDP port 33434-33474: traceroute -U -p 33434 142.250.186.46
- FAILS TCP port 33434-33474: traceroute -T -p 33434 142.250.186.46
- PASS TCP port 80: traceroute -T 142.250.186.46
- PASS ICMP: traceroute -I 142.250.186.46
- PASS PING: ping 142.250.186.46
Traceroute OPNSense VM -> 142.250.186.46 (one of Google "A" Records):
- FAILS UDP port 33434-33474: traceroute -P UDP -p 33434 142.250.186.46
- FAILS TCP port 33434-33474: traceroute -P TCP -p 33434 142.250.186.46
- FAILS TCP port 80: traceroute -P TCP 142.250.186.46
- FAILS ICMP: traceroute -P ICMP 142.250.186.46
- FAILS PING: ping 142.250.186.46
And worst still, it's all intermittent.
Heck I cannot even ping my Hetzner Gateway from the OPNSense VM.
I can get a WAN IP via DHCP from Hetzner DHCP Server without Issues.
I can get DNS queries working across without Issues (and get a reply from Hetzner DNS Server).
I can connect via OpenVPN from Home to OPNSense VM without Issues
But for some weird reason, I cannot do ANYTHING from the OPNSense Firewall itself.
Not curl, not fetch, not updates, nothing. Even if I try to disable IPv6, etc. Nothing helps.
I am really lost as to why it's going across in one direction but not the other. Firewall rules should be fine, checked both on the Proxmox VE Firewall Rules **FOR** the OPNSense VM as well as the rules in OPNSense itself.
Heck Outbound rules are even more loose than Inbound Rules (on both Firewalls), so why it's not working it's beyond me .
Maybe something to do with the Hardening of the /etc/sysctl.d/*.conf I did ?
Kernel Hardening
Code: [Select]
# eBPF exposes quite large attack surface and must therefore be restricted
# These sysctls restrict eBPF to the CAP_BPF capability (CAP_SYS_ADMIN on kernel versions prior to 5.8) and enable JIT hardening techniques, such as constant blinding.
kernel.unprivileged_bpf_disabled=1
net.core.bpf_jit_harden=2
# Restricts loading TTY line disciplines to the CAP_SYS_MODULE capability to prevent unprivileged attackers from loading vulnerable line disciplines with the TIOCSETD ioctl,
dev.tty.ldisc_autoload=0
# The userfaultfd() syscall is often abused to exploit use-after-free flaws. Due to this, this sysctl is used to restrict this syscall to the CAP_SYS_PTRACE capability.
vm.unprivileged_userfaultfd=0
# This only permits symlinks to be followed when outside of a world-writable sticky directory,
# when the owner of the symlink and follower match or when the directory owner matches the symlink's owner.
# This also prevents hardlinks from being created by users that do not have read/write access to the source file. Both of these prevent many common TOCTOU races.
fs.protected_symlinks=1
fs.protected_hardlinks=1
# These prevent creating files in potentially attacker-controlled environments, such as world-writable directories, to make data spoofing attacks more difficult.
fs.protected_fifos=2
fs.protected_regular=2
# Process that run with elevated privileges may still dump their memory even after these settings.
# To prevent them from doing so, set the following via sysctl:
fs.suid_dumpable=0
# Similar to core dumps, swapping or paging copies parts of memory to disk, which can contain sensitive information.
# The kernel should be configured to only swap if absolutely necessary
vm.swappiness=1
# dmesg is the kernel log. It exposes a large amount of useful kernel debugging information, but this can often leak sensitive information, such as kernel pointers.
# Changing the above sysctl restricts the kernel log to the CAP_SYSLOG capability.
kernel.dmesg_restrict=1
# Despite the value of dmesg_restrict, the kernel log will still be displayed in the console during boot.
# Malware that is able to record the screen during boot may be able to abuse this to gain higher privileges. This option prevents those information leaks.
# This must be used in combination with certain boot parameters described below to be fully effective.
kernel.printk=3 3 3 3
# A kernel pointer points to a specific location in kernel memory.
# These can be very useful in exploiting the kernel, but kernel pointers are not hidden by default — it is easy to uncover them by, for example, reading the contents of /proc/kallsyms.
# This setting aims to mitigate kernel pointer leaks. Alternatively, you can set kernel.kptr_restrict=1 to only hide kernel pointers from processes without the CAP_SYSLOG capability.
kernel.kptr_restrict=2
# Performance events add considerable kernel attack surface and have caused abundant vulnerabilities
# This sysctl restricts all usage of performance events to the CAP_PERFMON capability (CAP_SYS_ADMIN on kernel versions prior to 5.8).
# Be aware that this sysctl also requires a kernel patch that is only available on certain distributions.
# Otherwise, this setting is equivalent to kernel.perf_event_paranoid=2, which only restricts a subset of this functionality.
# kernel.perf_event_paranoid=3
# The SysRq key exposes a lot of potentially dangerous debugging functionality to unprivileged users.
# Contrary to common assumptions, SysRq is not only an issue for physical attacks, as it can also be triggered remotely.
# The value of this sysctl makes it so that a user can only use the secure attention key, which will be necessary for accessing root securely. Alternatively, you can simply set the value to 0 to disable SysRq completely.
kernel.sysrq=0
# ptrace is a system call that allows a program to alter and inspect another running process, which allows attackers to trivially modify the memory of other running programs.
# This restricts usage of ptrace to only processes with the CAP_SYS_PTRACE capability.
# Alternatively, set the sysctl to 3 to disable ptrace entirely.
kernel.yama.ptrace_scope=2
# Forbid creating Kernel Core Dumps
kernel.core_pattern=|/bin/false
# Controls whether core dumps will append the PID to the core filename
# Useful for debugging multi-threaded applications
kernel.core_uses_pid = 1
# Ensure that we still allow sufficient logging
kernel.printk_ratelimit=1
kernel.printk_ratelimit_burst=50
Network Hardening:
Code: [Select]
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# WE MUST ALSO SPECIFY EXPLICITELY eth0 HERE (maybe also vmbr0)
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# Disable IPv4 multicast igmp report on local link plug
net.ipv4.igmp_link_local_mcast_reports=0
# Ignore IPv4 ICMP echo packets sent to a broadcast address
net.ipv4.icmp_echo_ignore_broadcasts=1
# Ignore IPv6 ICMP echo packets sent to anycast & multicast address
# This works very similarly to net.ipv4.icmp_echo_ignore_broadcasts
net.ipv6.icmp.echo_ignore_anycast=1
net.ipv6.icmp.echo_ignore_multicast=1
# This setting avoids filling up log files with unnecessary error messages coming from invalid responses to broadcast frames.
net.ipv4.icmp_ignore_bogus_error_responses=1
# Don't ignore directed pings
# This will also make PING the Proxmox VE Host Impossible !
net.ipv4.icmp_echo_ignore_all=0
net.ipv6.icmp.echo_ignore_all=0
# Disables the acceptance of ICMP redirect messages.
# Make sure no one can modify the routing Table
# Ignore IPv4 ICMP redirect messages
net.ipv4.conf.default.accept_redirects=0
net.ipv4.conf.all.accept_redirects=0
net.ipv4.conf.eth0.accept_redirects=0
#net.ipv4.conf.vmbr0.accept_redirects=0
# Ignore IPv6 ICMP redirect messages
net.ipv6.conf.default.accept_redirects=0
net.ipv6.conf.all.accept_redirects=0
net.ipv6.conf.eth0.accept_redirects=0
#net.ipv6.conf.vmbr0.accept_redirects=0
# Accepting 'secure' ICMP redirects (from those gateways listed as default gateways) has few legitimate uses.
# It should be disabled unless it is absolutely required.
net.ipv4.conf.default.secure_redirects=0
net.ipv4.conf.all.secure_redirects=0
net.ipv4.conf.eth0.secure_redirects=0
#net.ipv4.conf.vmbr0.secure_redirects=0
# A node should not send out IPv4 ICMP redirects, unless it acts as a router.
net.ipv4.conf.default.send_redirects=0
net.ipv4.conf.all.send_redirects=0
net.ipv4.conf.eth0.send_redirects=0
#net.ipv4.conf.vmbr0.send_redirects=0
# Do NOT act as a Router
# IP forwarding should only be enabled on systems acting as routers.
# Disable IPv4 Packet Forwarding
net.ipv4.ip_forward=0
net.ipv4.conf.default.forwarding=0
net.ipv4.conf.all.forwarding=0
net.ipv4.conf.eth0.forwarding=0
#net.ipv4.conf.vmbr0.forwarding=0
# Disable IPv6 Packet Forwarding
net.ipv6.conf.default.forwarding=0
net.ipv6.conf.all.forwarding=0
net.ipv6.conf.eth0.forwarding=0
#net.ipv6.conf.vmbr0.forwarding=0
# No source routed packets here
# This setting disables the acceptance of packets with the SSR option set in the IPv4 packet header.
# Packets that use Source Routing will be rejected.
# This prevents IP packet redirection, that is redirection to a host behind a firewall, that is not directly reachable otherwise.
net.ipv4.conf.default.accept_source_route=0
net.ipv4.conf.all.accept_source_route=0
net.ipv4.conf.eth0.accept_source_route=0
#net.ipv4.conf.vmbr0.accept_source_route=0
# Do not accept packets with SRR option
net.ipv6.conf.default.accept_source_route=0
net.ipv6.conf.all.accept_source_route=0
net.ipv6.conf.eth0.accept_source_route=0
#net.ipv6.conf.vmbr0.accept_source_route=0
# Disable IPv6 Router Advertisements, since they can result in a man-in-the-middle attack
net.ipv6.conf.default.accept_ra=0
net.ipv6.conf.all.accept_ra=0
net.ipv6.conf.eth0.accept_ra=0
#net.ipv6.conf.vmbr0.accept_ra=0
# Turn on and log spoofed, source routed, and redirect packets
net.ipv4.conf.default.log_martians=1
net.ipv4.conf.all.log_martians=1
net.ipv4.conf.eth0.log_martians=1
#net.ipv4.conf.vmbr0.log_martians=1
# Turn on syncookies for SYN flood attack protection
# This enables TCP SYN Cookie Protection for IPv4 and IPv6.
net.ipv4.tcp_syncookies=1
# Turn on reverse path filter in Strict Mode
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1
net.ipv4.conf.eth0.rp_filter=1
#net.ipv4.conf.vmbr0.rp_filter=1
# Disable forwarding of all multicast packets on all interfaces
net.ipv4.conf.default.mc_forwarding=0
net.ipv4.conf.all.mc_forwarding=0
net.ipv4.conf.eth0.mc_forwarding=0
#net.ipv4.conf.vmbr0.mc_forwarding=0
net.ipv6.conf.default.mc_forwarding=0
net.ipv6.conf.all.mc_forwarding=0
net.ipv6.conf.eth0.mc_forwarding=0
#net.ipv6.conf.vmbr0.mc_forwarding=0
# Drop RST packets for sockets in the time-wait state
net.ipv4.tcp_rfc1337=1
# Disable TCP SACK
net.ipv4.tcp_sack=0
net.ipv4.tcp_dsack=0
net.ipv4.tcp_fack=0
# TCP timestamps also leak the system time and should therefore be disabled
net.ipv4.tcp_timestamps=0
# Enable IPv6 Privacy Extensions
net.ipv6.conf.default.use_tempaddr=2
net.ipv6.conf.all.use_tempaddr=2
#net.ipv6.conf.eth0.use_tempaddr=2
#net.ipv6.conf.vmbr0.use_tempaddr=2
# The TCP SYN backlog defines the number of SYN packets that are queued for further processing.
# Once the queue limit is exceeded, all new incoming SYN-packets are dropped and new TCP connections will not be possible (or the SYN cookie protection kicks in)
# Increasing this value improves the protection against TCP SYN flood attacks.
net.ipv4.tcp_max_syn_backlog=4096
#########################################################
############ NEWLY INTRODUCED ON 2024-05-17 #############
#########################################################
# Number of global unicast IPv6 addresses can be assigned to each interface
net.ipv6.conf.default.max_addresses=1
net.ipv6.conf.all.max_addresses=1
net.ipv6.conf.eth0.max_addresses=1
#net.ipv6.conf.vmbr0.max_addresses=1
# Number of neighbor solicitations to send out per address
net.ipv6.conf.default.dad_transmits=0
net.ipv6.conf.all.dad_transmits=0
net.ipv6.conf.eth0.dad_transmits=0
#net.ipv6.conf.vmbr0.dad_transmits=0
# Router advertisements can cause the system to assign a global unicast address to an interface
net.ipv6.conf.default.autoconf=0
net.ipv6.conf.all.autoconf=0
net.ipv6.conf.eth0.autoconf=0
#net.ipv6.conf.vmbr0.autoconf=0
# Setting controls whether the system will accept Hop Limit settings from a router advertisement
net.ipv6.conf.default.accept_ra_defrtr=0
net.ipv6.conf.all.accept_ra_defrtr=0
net.ipv6.conf.eth0.accept_ra_defrtr=0
#net.ipv6.conf.vmbr0.accept_ra_defrtr=0
# Learn prefix information in router advertisement
net.ipv6.conf.default.accept_ra_pinfo=0
net.ipv6.conf.all.accept_ra_pinfo=0
net.ipv6.conf.eth0.accept_ra_pinfo=0
#net.ipv6.conf.vmbr0.accept_ra_pinfo=0
# Do not accept Router Preference from RA
net.ipv6.conf.default.accept_ra_rtr_pref=0
net.ipv6.conf.all.accept_ra_rtr_pref=0
net.ipv6.conf.eth0.accept_ra_rtr_pref=0
#net.ipv6.conf.vmbr0.accept_ra_rtr_pref=0
# Number of Router Solicitations to send until assuming no routers are present
net.ipv6.conf.default.router_solicitations=0
net.ipv6.conf.all.router_solicitations=0
net.ipv6.conf.eth0.router_solicitations=0
#net.ipv6.conf.vmbr0.router_solicitations=0
# Newly Introduced on 2024-05-20
# Source: https://forum.level1techs.com/t/solved-linux-vm-bridged-network-arp-ok-no-ip-traffic/191172/14
net.bridge.bridge-nf-call-iptables=0
net.bridge.bridge-nf-call-ip6tables=0
net.bridge.bridge-nf-call-nf_call_arptables=0
# eBPF exposes quite large attack surface and must therefore be restricted
# These sysctls restrict eBPF to the CAP_BPF capability (CAP_SYS_ADMIN on kernel versions prior to 5.8) and enable JIT hardening techniques, such as constant blinding.
#net.core.bpf_jit_harden=2
# Logging
net.core.message_cost=5
net.core.message_burst=50
The Hardening is mostly due to several "MAC Abuse Emails" that Hetzner sent me, after apparently my Server was replying to / forwarding Broadcast traffic.
I also have /usr/local/bin/disable-network-flooding
Code: [Select]
#!/bin/bash
# Optional Argument for Verbose Output
verbose=${1-""}
# Disable Unicast flooding
# This **might** also disable OPNSense Communication
# Try to disable it for now
mapfile -t list < <( find /sys -iname *unicast_flood* )
for item in "${list[@]}"
do
# Echo
if [[ -n "${verbose}" ]]
then
echo "Executing: echo 0 > ${item}"
fi
# Disable unicast flooding on all interfaces
echo 0 > $item
done