Mixing public and private IP forwarding

[nicksc]

Hi all,

I have a somewhat strange setup that I would like to validate. After a few hours of trying to get it to work I started wondering if it was possible at all.

My ISP issued me a subnet on my request to use for my homelab to give VM's external IP's without NAT.

- My firewall (DEC3862) is hooked directly to the ISP.
- Proxy ARP has been setup and appears to work
- Outbound NAT is excluded for the public IP range
- When I put the VM with the public IP behind the LAN interface. I get the following result, which I understand since the IP is not in the subnet of the interface.

Code: [Select]

┌────────────┐         ┌─────────────┐           ┌────────────┐
│     VM     ├────────►│    Router   ├──────────►│  Internet  │
│  1.2.3.4   │         │  10.234.0.1 │           │   8.8.8.8  │
│            │     x───┼             ◄───────────┤            │
└────────────┘         └─────────────┘           └────────────┘

When I create a VLAN without IP and point it directly at the external gateway the VM keeps ARPing for that gateway which I understand.

After a lengthy search I understand that bridge mode/transparant mode or 1-to-1 NAT should work. The first one is a bit much since I also have a household that requires regular LAN services. The 1-to-1 NAT or port-forwarding would solve the problem, but makes learning virtualised routers/firewalls a lot harder since I'd have to remap everything.

So my question, given that the traffic is almost going full circle. Is there any way to convince the router to accept the ARP entry for my VM for an IP that is not in it's subnet and route in that direction without a full WAN/LAN bridge. Or is there perhaps any other way to do this? Thanks!

[dseven]

To try to make suggestions, I'd first need to know; how is the subnet supplied to you? Did they give you an IP address for your router, and subnet that is routed to that address, or did they give you a subnet, and require that your router use that subnet directly?

[nicksc]

Thanks for your reply. I suspect the latter, that they provided me a subnet. When I ping one of the IP's in the subnet from a VPS I can see it arriving on my WAN interface which is Proxy ARPing the entire range. Since it is an residential connection this is not within their normal flow, so I have little instructions to go on nor do I expect much support from their side.

But I can confirm that traffic able to go out from the subnet and is also being routed back properly. Just getting it back to the sender is challenging.

[dseven]

If that's the case, I think 1:1 BINAT would be best for your case - so your VMs would have private addresses on some internal (RFC1918) subnet, and be mapped to public addresses from your ISP-provided internet-routable subnet, and everything that doesn't been to be internet-reachable would be NATed on the firewall's primary WAN address.

[nicksc]

Thanks for the suggestion, that does seem to work  Unfortunately some apps PPTP/Wireguard/IPSec do benefit from having the external IP assigned to the interface. I use this to remain connected, but also to P2P to other networks. Do you perchance see any solution which allows me to use external IP's directly on VM's?

[Patrick M. Hausen]

Assign the first address of that subnet to an interface of OPNsense, e.g. OPT1. Connect VMs to that interface, give VMs the remaining IP addresses with the OPNsense address as the default gateway. Disable NAT.

While RFC 1918 did introduce a concept of "public" and "private" IP addresses, nowhere does it say, that public ones must be placed on a "WAN" interface. And proxy ARP is evil.

Just configure your interfaces and let routing do its magic.

[nicksc]

Assign the first address of that subnet to an interface of OPNsense, e.g. OPT1. Connect VMs to that interface, give VMs the remaining IP addresses with the OPNsense address as the default gateway. Disable NAT.

While RFC 1918 did introduce a concept of "public" and "private" IP addresses, nowhere does it say, that public ones must be placed on a "WAN" interface. And proxy ARP is evil.

Just configure your interfaces and let routing do its magic.

Heey, thanks for your reply! If possible I could use another small suggestion    I tried this setup this morning and it works, but only with ProxyARP. I have a /24 currently. Preferably I would like to dodge NAT.
- .1 is routed to the ISP's gateway
- .2 is my IP Alias on the interface
- .3 is my VM

I am getting the feeling that there is no static route to my .2, given that I can see .1 desperately ARPing for .3 when I have no proxy ARP. I am also

Now it is my intention to run this in HA mode, is that possible with Proxy ARP our should I add 253 addresses as IP alias individually?

Thanks again for thinking with me on this 

[dseven]

Leaving aside the complicated stuff for a minute ...

How is your WAN interface configured to get service from your ISP? DHCP? PPPoE? Static? Something else?

Are the IP address of your WAN interface and your gateway within the subnet that they've assigned to you, or have they assigned you a completely separate subnet and routed it to your WAN address?

This is a pfSense docuement, but it describes the scenarios. To attempt to make suggestions, we need to determine which scenario we're dealing with...

[nicksc]

WAN is indeed in a different /29 subnet which is mostly already used with static IP's. I suspect it is as described in the PFsense documentation as transport or interconnect network. My secondary network with public IP's is a /24 from an entirely different range.

[dseven]

So why are you doing proxy ARP? Why don't you just configure your LAN interface (either the default one, or an additional one that you create) with the first address from the /24 subnet, and give your homelab hosts addresses within that subnet (either via DHCP or static configuration)? What problem are you trying to solve with proxy ARP?

[nicksc]

Hey, it appears that the first IP is in use by the ISP, so I took the second as my gateway with an IP Alias. If I do not proxy ARP then the ISP is unaware of how to route my VM since I can see it searching for the VM's IP via ARP. Proxy ARP fixes this for me, but I feel that proxy ARP is not a proper solution.

Any suggestions on how to 'dodge' proxy ARP are welcome.

[dseven]

You need to get the definitive answer on this, as you don't appear to know. Ask your ISP if that subnet is routed to your existing WAN IP address, or if you are required to connect your router directly to that subnet and use an upstream gateway on it.

[nicksc]

I got a reply from the ISP. Indeed the range is not statically routed. the nearest switch has a .1 configured as gateway. So if I ARP that IP I will receive the traffic.

I suspect that for a HA setup I have to create many IP Aliases, or does ProxyARP also work?

[Patrick M. Hausen]

Did you ask them if they could route that larger network to one specific IP of that /29 uplink/transfer network?

[nicksc]

I have confirmation that static routing unfortunately is not supported, which might explain the issues since that is what I was trying. Is there any solution in this situation in which I can realise a highly available setup perchance?