Outbound access on Hetzner

[power46]

Hi,

I am trying to familiarize with OPNsense by deploying it on Hetzner.

I have:

- router: a server (10.0.0.10) with OPNsense installed on it, I can access the WebUI/ssh from the WAN interface.
- node: another server (10.0.0.20, no public IP) with Debian 12:

Code Select Expand

$ cat /etc/resolv.conf
nameserver 10.0.0.10

$ ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000
    link/ether 86:00:00:dd:15:64 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.20/32 brd 10.0.0.20 scope global dynamic enp7s0
       valid_lft 86200sec preferred_lft 86200sec
    inet6 fe80::8400:ff:fedd:1564/64 scope link
       valid_lft forever preferred_lft forever

$ ip route
default via 10.0.0.1 dev enp7s0
10.0.0.0/24 via 10.0.0.1 dev enp7s0
10.0.0.1 dev enp7s0 scope link
169.254.169.254 via 10.0.0.1 dev enp7s0

I can ssh to the node via the router:

Code Select Expand

ssh -J root@$IP_PUBLIC 10.0.0.20

I can ping the router from the node:

Code Select Expand

$ ssh -J root@$IP_PUBLIC admin@10.0.0.20 ping 10.0.0.10
PING 10.0.0.10 (10.0.0.10) 56(84) bytes of data.
64 bytes from 10.0.0.10: icmp_seq=1 ttl=63 time=0.955 ms

I can resolve dns:

Code Select Expand

$ ssh -J root@$IP_PUBLIC admin@10.0.0.20 dig example.com

; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50112
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;example.com. IN A

;; ANSWER SECTION:
example.com. 300 IN A 96.7.128.198
example.com. 300 IN A 23.192.228.80
example.com. 300 IN A 23.192.228.84
example.com. 300 IN A 23.215.0.136
example.com. 300 IN A 23.215.0.138
example.com. 300 IN A 96.7.128.175

;; Query time: 268 msec
;; SERVER: 10.0.0.10#53(10.0.0.10) (UDP)
;; WHEN: Wed Jan 22 08:23:35 UTC 2025
;; MSG SIZE  rcvd: 136


But the node does not have access to the internet.

For example if I try to ping 1.1.1.1 from the node, via tcpdump on the router, I can see the request but no response:

Code Select Expand

root@OPNsense:~ # tcpdump -i vtnet1 not port 22
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vtnet1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
08:25:56.061477 IP 10.0.0.20 > one.one.one.one: ICMP echo request, id 904, seq 1, length 64
08:25:57.064052 IP 10.0.0.20 > one.one.one.one: ICMP echo request, id 904, seq 2, length 64

From the WebUI > Reporting > Traffic > Top talkers, I can see the one.one.one.one.(1.1.1.1) request.

Could you suggest where I can go to get more insight on why a request is blocked/...?

Thanks

[meyergru]

Giving both the routeable and non-public IPs as being in the same subnet is a little confusing, but I guess that 10.0.0.0/24 is your "LAN" between both machines with your server having 10.0.0.10 and OpnSense having 10.0.0.1 (not .20), because that is the default router in your server network configuration (unless that is the problem already).

Did you enable outbound NAT for that subnet?

[power46]

Thanks for the reply.

To clarify:

* LAN network CIDR is 10.0.0.0/24.
* OPNsense IP is 10.0.0.10
* 10.0.0.1 is Hetzner gateway required to route traffic in the private network

I have set out a hetzner network route: destination 0.0.0.0/0, gateway 10.0.0.10.

Similar to https://community.hetzner.com/tutorials/how-to-route-cloudserver-over-private-network-using-pfsense-and-hcnetworks

Regarding outbound NAT, I tried:

* default automatic
* hybrid with an additional LAN, any source, any destination, translate to WAN interface

no difference 

[meyergru]

Quoting your first post (this is on your server!):

Code Select Expand

$ ip route
default via 10.0.0.1 dev enp7s0
10.0.0.0/24 via 10.0.0.1 dev enp7s0
10.0.0.1 dev enp7s0 scope link
169.254.169.254 via 10.0.0.1 dev enp7s0

How do you expect OpnSense to do anything if packets do not even touch it?

[power46]

Quoting your first post (this is on your server!):

Code Select Expand

$ ip route
default via 10.0.0.1 dev enp7s0
10.0.0.0/24 via 10.0.0.1 dev enp7s0
10.0.0.1 dev enp7s0 scope link
169.254.169.254 via 10.0.0.1 dev enp7s0

How do you expect OpnSense to do anything if packets do not even touch it?

From my understanding, the node has be to set to the hetzner gateway (10.0.0.1), otherwise the private traffic cannot be routed inside the virtual private network.

From my ping/dns/tcpdump experiment, packets are actually arriving at OPNsense but for some reason OPNsense is not "responding" the outbound request from the node.

[meyergru]

It might be the case that you have to set 10.0.0.1 as router in order to reach your OpnSense, but of you specify that as "default" route, any packet that is not destined for 10.0.0.0/24 will go to it - and not through OpnSense.

[power46]

I am sorry, I am trying to understand.

But if the default route would be a problem, I should not see the ping request arriving to OPNsense via tcpdump or the connection via the WebUI.

Am I correct?

[meyergru]

No, not generally. You can communicate between OpnSense and your server over the 10.0.0.0/24 network, thus the webgui is reachable. The problem arises when you try to communicate to something different than 10.0.0.0/24.

IDK why you see the ping for 1.1.1.1 on your OpnSense, though, maybe promicuous mode. I would try to remove the default route and keep the 10.0.0.0/24 route on the server.

P.S.: You know you can setup this all on one server, like described here?
Although if you want to only familiarize yourself with OpnSense and using Hetzner is not the final goal, that will be no help.
Probably, though, it would be easier to use a local Proxmox instance to simulate all this instead of relying on Hetzner infrastructure.
In that case, the main post from that thread might be helpful.

[power46]

Ok, are you suggesting to set default route for the node to 10.0.0.10 (OPNSense private ip)?

If I try, get:

Code Select Expand

$ sudo ip route add default via 10.0.0.10
Error: Nexthop has invalid gateway.


[pfry]

From your first post:

Code Select Expand

2: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000
    link/ether 86:00:00:dd:15:64 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.20/32 brd 10.0.0.20 scope global dynamic enp7s0

/32? I assume that's why it objects to the .10 gateway. Shouldn't that be a /24?

But using the firewall as your gateway will (or at least should) not solve your issue. The server is essentially in parallel to the firewall, so using it as a gateway will result in asymmetric paths to/from Hetzner (or anything else run through the gateway). Not recommended in general, and definitely not with a (potentially) stateful filter (firewall). Consider your goals; if you want to bounce them off the folks here, post a basic description and drawing of your plan.

[power46]

Sure.

The aim is to deploy OPNsense on hetzner and route all traffic (in particular public) through it.

Each node will just have a private interface (no public ip).

In other words:

- node > OPNsense > internet

At the moment I am configuring the node via cloud-init:

Code Select Expand

#cloud-config
users:
  - name: admin
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - <key>
write_files:
  - path: /etc/resolv.conf
    content: |
      nameserver 10.0.0.10
  - path: /etc/network/interfaces
    content: |
      auto enp7s0
      iface enp7s0 inet static
        address 10.0.0.20
        netmask 255.255.255.0
        network 10.0.0.0
        gateway 10.0.0.10
        dns-nameservers 10.0.0.10
  - path: /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg
    content: |
      network:
        config: disabled
runcmd:
  - ip route add default via 10.0.0.1


[pfry]

[...]
- node > OPNsense > internet

OK. In that scenario I would expect the devices to be physically (or virtually) connected following that model, such that you have (separate) "LAN" and "WAN" interfaces on the OPNsense device with unique IP subnet assignments. So if your Hetzner link requires the 10.0.0.0/24 subnet you'd assign (an address from) that on the WAN side and something else (from, say 172.16.0.0/24 for illustration purposes, just to be clearly distinct) on the LAN side. Your "nodes" would reside on the LAN side and have an address from the LAN subnet, with no connection to your Hetzner link other than through the OPNsense device. Something like:

[node (172.16.0.10/24)] - [(172.16.0.1/24) OPNsense (10.0.0.10/24)] - [(10.0.0.1/24) Hetzner]

Of course you have unlimited options (I'd hate to try to draw my setup, and it's pretty simple), but that's the basic expectation for your scenario. A baseline from which you can describe your desired topology.

At the moment I am configuring the node via cloud-init:

I'm allergic to "clouds" and "virtual" things, so I'm no help there.