Hi,
I am trying to familiarize with OPNsense by deploying it on Hetzner.
I have:
- router: a server (10.0.0.10) with OPNsense installed on it, I can access the WebUI/ssh from the WAN interface.
- node: another server (10.0.0.20, no public IP) with Debian 12:
$ cat /etc/resolv.conf
nameserver 10.0.0.10
$ ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000
link/ether 86:00:00:dd:15:64 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.20/32 brd 10.0.0.20 scope global dynamic enp7s0
valid_lft 86200sec preferred_lft 86200sec
inet6 fe80::8400:ff:fedd:1564/64 scope link
valid_lft forever preferred_lft forever
$ ip route
default via 10.0.0.1 dev enp7s0
10.0.0.0/24 via 10.0.0.1 dev enp7s0
10.0.0.1 dev enp7s0 scope link
169.254.169.254 via 10.0.0.1 dev enp7s0
I can ssh to the node via the router:
ssh -J root@$IP_PUBLIC 10.0.0.20
I can ping the router from the node:
$ ssh -J root@$IP_PUBLIC admin@10.0.0.20 ping 10.0.0.10
PING 10.0.0.10 (10.0.0.10) 56(84) bytes of data.
64 bytes from 10.0.0.10: icmp_seq=1 ttl=63 time=0.955 ms
I can resolve dns:
$ ssh -J root@$IP_PUBLIC admin@10.0.0.20 dig example.com
; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50112
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;example.com. IN A
;; ANSWER SECTION:
example.com. 300 IN A 96.7.128.198
example.com. 300 IN A 23.192.228.80
example.com. 300 IN A 23.192.228.84
example.com. 300 IN A 23.215.0.136
example.com. 300 IN A 23.215.0.138
example.com. 300 IN A 96.7.128.175
;; Query time: 268 msec
;; SERVER: 10.0.0.10#53(10.0.0.10) (UDP)
;; WHEN: Wed Jan 22 08:23:35 UTC 2025
;; MSG SIZE rcvd: 136
But the node does not have access to the internet.
For example if I try to ping 1.1.1.1 from the node, via tcpdump on the router, I can see the request but no response:
root@OPNsense:~ # tcpdump -i vtnet1 not port 22
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vtnet1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
08:25:56.061477 IP 10.0.0.20 > one.one.one.one: ICMP echo request, id 904, seq 1, length 64
08:25:57.064052 IP 10.0.0.20 > one.one.one.one: ICMP echo request, id 904, seq 2, length 64
From the WebUI > Reporting > Traffic > Top talkers, I can see the one.one.one.one.(1.1.1.1) request.
Could you suggest where I can go to get more insight on why a request is blocked/...?
Thanks
Giving both the routeable and non-public IPs as being in the same subnet is a little confusing, but I guess that 10.0.0.0/24 is your "LAN" between both machines with your server having 10.0.0.10 and OpnSense having 10.0.0.1 (not .20), because that is the default router in your server network configuration (unless that is the problem already).
Did you enable outbound NAT for that subnet?
Thanks for the reply.
To clarify:
* LAN network CIDR is 10.0.0.0/24.
* OPNsense IP is 10.0.0.10
* 10.0.0.1 is Hetzner gateway required to route traffic in the private network
I have set out a hetzner network route: destination 0.0.0.0/0, gateway 10.0.0.10.
Similar to https://community.hetzner.com/tutorials/how-to-route-cloudserver-over-private-network-using-pfsense-and-hcnetworks
Regarding outbound NAT, I tried:
* default automatic
* hybrid with an additional LAN, any source, any destination, translate to WAN interface
no difference
Quoting your first post (this is on your server!):
$ ip route
default via 10.0.0.1 dev enp7s0
10.0.0.0/24 via 10.0.0.1 dev enp7s0
10.0.0.1 dev enp7s0 scope link
169.254.169.254 via 10.0.0.1 dev enp7s0
How do you expect OpnSense to do anything if packets do not even touch it?
Quote from: meyergru on January 22, 2025, 10:14:42 AMQuoting your first post (this is on your server!):
$ ip route
default via 10.0.0.1 dev enp7s0
10.0.0.0/24 via 10.0.0.1 dev enp7s0
10.0.0.1 dev enp7s0 scope link
169.254.169.254 via 10.0.0.1 dev enp7s0
How do you expect OpnSense to do anything if packets do not even touch it?
From my understanding, the node has be to set to the hetzner gateway (10.0.0.1), otherwise the private traffic cannot be routed inside the virtual private network.
From my ping/dns/tcpdump experiment, packets are actually arriving at OPNsense but for some reason OPNsense is not "responding" the outbound request from the node.
It might be the case that you have to set 10.0.0.1 as router in order to reach your OpnSense, but of you specify that as "default" route, any packet that is not destined for 10.0.0.0/24 will go to it - and not through OpnSense.
I am sorry, I am trying to understand.
But if the default route would be a problem, I should not see the ping request arriving to OPNsense via tcpdump or the connection via the WebUI.
Am I correct?
No, not generally. You can communicate between OpnSense and your server over the 10.0.0.0/24 network, thus the webgui is reachable. The problem arises when you try to communicate to something different than 10.0.0.0/24.
IDK why you see the ping for 1.1.1.1 on your OpnSense, though, maybe promicuous mode. I would try to remove the default route and keep the 10.0.0.0/24 route on the server.
P.S.: You know you can setup this all on one server, like described here (https://forum.opnsense.org/index.php?msg=220167)?
Although if you want to only familiarize yourself with OpnSense and using Hetzner is not the final goal, that will be no help.
Probably, though, it would be easier to use a local Proxmox instance to simulate all this instead of relying on Hetzner infrastructure.
In that case, the main post from that thread (https://forum.opnsense.org/index.php?msg=220166) might be helpful.
Ok, are you suggesting to set default route for the node to 10.0.0.10 (OPNSense private ip)?
If I try, get:
$ sudo ip route add default via 10.0.0.10
Error: Nexthop has invalid gateway.
From your first post:
2: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000
link/ether 86:00:00:dd:15:64 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.20/32 brd 10.0.0.20 scope global dynamic enp7s0
/32? I assume that's why it objects to the .10 gateway. Shouldn't that be a /24?
But using the firewall as your gateway will (or at least should) not solve your issue. The server is essentially in parallel to the firewall, so using it as a gateway will result in asymmetric paths to/from Hetzner (or anything else run through the gateway). Not recommended in general, and definitely not with a (potentially) stateful filter (firewall). Consider your goals; if you want to bounce them off the folks here, post a basic description and drawing of your plan.
Sure.
The aim is to deploy OPNsense on hetzner and route all traffic (in particular public) through it.
Each node will just have a private interface (no public ip).
In other words:
- node > OPNsense > internet
At the moment I am configuring the node via cloud-init:
#cloud-config
users:
- name: admin
shell: /bin/bash
sudo: ALL=(ALL) NOPASSWD:ALL
ssh_authorized_keys:
- <key>
write_files:
- path: /etc/resolv.conf
content: |
nameserver 10.0.0.10
- path: /etc/network/interfaces
content: |
auto enp7s0
iface enp7s0 inet static
address 10.0.0.20
netmask 255.255.255.0
network 10.0.0.0
gateway 10.0.0.10
dns-nameservers 10.0.0.10
- path: /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg
content: |
network:
config: disabled
runcmd:
- ip route add default via 10.0.0.1
Quote from: power46 on January 23, 2025, 08:24:12 AM[...]
- node > OPNsense > internet
OK. In that scenario I would expect the devices to be physically (or virtually) connected following that model, such that you have (separate) "LAN" and "WAN" interfaces on the OPNsense device with unique IP subnet assignments. So if your Hetzner link requires the 10.0.0.0/24 subnet you'd assign (an address from) that on the WAN side and something else (from, say 172.16.0.0/24 for illustration purposes, just to be clearly distinct) on the LAN side. Your "nodes" would reside on the LAN side and have an address from the LAN subnet, with no connection to your Hetzner link other than through the OPNsense device. Something like:
[node (172.16.0.10/24)] - [(172.16.0.1/24) OPNsense (10.0.0.10/24)] - [(10.0.0.1/24) Hetzner]
Of course you have unlimited options (I'd hate to try to draw my setup, and it's pretty simple), but that's the basic expectation for your scenario. A baseline from which you can describe your desired topology.
QuoteAt the moment I am configuring the node via cloud-init:
I'm allergic to "clouds" and "virtual" things, so I'm no help there.