Wrong Source IP Used on Outbound Traffic

[tuaris]

Not sure the proper way to describe this (I'm not 100% familiar with the terminology). 

Subnett: 63.X.X.0/28
WAN IP: 63.X.X.3

I have (and always had) 2 virtual IP's setup.
Virtual IP 1: 63.X.X.2
Virtual IP 1: 63.X.X.4

No 1:1 NAT Setup.  Virtual IP's used for some Port forward rules.

Normally as expected any host within the LAN would connect to (for example) a webserver and use the WAN IP as it's "Source Address"

After upgrading to 18.1, LAN hosts now randomly switch the source IP between the WAN, and any one of the Virtual IP:

Example:

Code: [Select]

root@saturn:~ # fetch -qo - http://showthisip.com/\?simple
63.x.x.3
root@saturn:~ # fetch -qo - http://showthisip.com/\?simple
63.x.x.4
root@saturn:~ # fetch -qo - http://showthisip.com/\?simple
63.x.x.4
root@saturn:~ # fetch -qo - http://showthisip.com/\?simple
63.x.x.4
root@saturn:~ # fetch -qo - http://showthisip.com/\?simple
63.x.x.3
root@saturn:~ # fetch -qo - http://showthisip.com/\?simple
63.x.x.3
root@saturn:~ # fetch -qo - http://showthisip.com/\?simple
63.x.x.2
root@saturn:~ # fetch -qo - http://showthisip.com/\?simple
63.x.x.4
root@saturn:~ # fetch -qo - http://showthisip.com/\?simple
63.x.x.4
root@saturn:~ # fetch -qo - http://showthisip.com/\?simple
63.x.x.4
root@saturn:~ # fetch -qo - http://showthisip.com/\?simple
63.x.x.3
root@saturn:~ # fetch -qo - http://showthisip.com/\?simple
63.x.x.4
root@saturn:~ # fetch -qo - http://showthisip.com/\?simple
63.x.x.2


[marjohn56]

I also have multiple WAN IP's. I use 1:1 NAT, firewall rules for the respective ports and it works flawlessly.

I did notice a similar thing that you mention whilst 'messing' around and setting aliases on the WAN port using ifconfig. Once I removed those and went back to the 1:1 NAT, no issues.

[michaelsage]

I had this issue to. I fixed it by changing:

Translation / target in the outbound NAT from interface address to WAN address.

I hadn't experienced this issue before the 18.1 upgrade.

[tuaris]

I have "Automatic outbound NAT rule generation" selected. 

Updated to 18.1.1 didn't solve the problem.

[franco]

Hi there,

This is the new behaviour to use round-robin on all available IP addresses... it has VIPs assigned on the interface so they want to be used.

You can set outbound mode to hybrid or manual to fix the NAT behaviour to a single IP explicitly or set sticky mode for your connections so the IP assignments don't flip per connection of a user.

We will make an additional note in the original 18.1 update change long for all new upgraders from 17.7 to see.


Cheers,
Franco

[tuaris]

This is the new behaviour to use round-robin on all available IP addresses... it has VIPs assigned on the interface so they want to be used.

I don't think it's a good idea to enable this by default.  It breaks on sites that lock your session to a specific IP address for security purposes.

[franco]

How are you using the virtual IPs on the WAN?


Cheers,
Franco

[tuaris]

How are you using the virtual IPs on the WAN?

I use the Virtual IPs as destination's for "NAT: Port Forward".   This is what used to be called "Server NAT" in m0n0wall: https://doc.m0n0.ch/handbook/nat-server.html

[franco]

Okay, I see. I suppose that is a form of hybrid 1:1 with traditional port forward?

Well, I double-checked and all we do in 181 is simplify NAT by passing to pf an interface and it automatically does this round-robin outbound NAT. It's not perfectly clear which IP is the primary one (even though the VIPs could be considered auxiliary to change that behaviour manual NAT is required anyway) so to avoid connections shifting from one IP to the next we've added "Sticky outbound NAT" to Firewall: Settings: Advanced which will help your case.

Maybe that already helps?

It could likely be switched on by default in subsequent image releases. We've done the same for other "sticky" settings in the past, too.


Cheers,
Franco

[tuaris]

I enabled "Sticky oubound NAT" under " Firewall: Settings: Advanced".  Tried resetting states and tried rebooting.  The source address used does appear to remain stable much longer (but not permanently).  It's slightly better.   

However, in my testing it's still using the wrong one.  It uses the Virtual IP's but never the actual IP assigned to the WAN interface as per "Interfaces: [WAN]:  IPv4 address".

The below should actually be 63.x.x.3, which is what assigned to the WAN interface on the firewall.

Code: [Select]

root@saturn:~ # fetch -qo - http://showthisip.com/\?simple
63.x.x.4
root@saturn:~ # fetch -qo - http://showthisip.com/\?simple
63.x.x.4
root@saturn:~ # fetch -qo - http://showthisip.com/\?simple
63.x.x.4
root@saturn:~ # fetch -qo - http://showthisip.com/\?simple
63.x.x.4
root@saturn:~ # fetch -qo - http://showthisip.com/\?simple
63.x.x.4
root@saturn:~ # fetch -qo - http://showthisip.com/\?simple
63.x.x.4
root@saturn:~ # fetch -qo - http://showthisip.com/\?simple
63.x.x.4
root@saturn:~ # fetch -qo - http://showthisip.com/\?simple
63.x.x.4
root@saturn:~ # fetch -qo - http://showthisip.com/\?simple
63.x.x.4
root@saturn:~ # fetch -qo - http://showthisip.com/\?simple
63.x.x.4
root@saturn:~ # fetch -qo - http://showthisip.com/\?simple
63.x.x.4


[franco]

It's not supposed to fix it, but at least pin client connections to a particular IP for consistency.

Please use a manual or hybrid rule to NAT on "WAN address" if you want the old behaviour.


Cheers,
Franco

[truster]

Hi Franco, i'm new to opnsense but used othersense™ for years...

i don't think, that this one is a good idea. It will brakes everything... terrible with Use of ipsec because the SA is not matching with other IPs than configured.

I think the best idea is a boolean switch to enable the old behavior without have to create custom outbound masquerade rules.

alternativly we could make use of a switch while creating an ip alias...eg:"do not use this alias for outbound nat."

i do not see any benefit for outbound nat-rr on a single wan....

we use ip alias for special service eg:
on ip 1.2.3.1 = everything else
ip 1.2.3.2 owa.domain.tld with HAPROXYA
ip 1.2.3.3 openvpn.domain.tld
ip 1.2.3.4 vta.domain.tld (telephone appliance)
io 1.2.3.5 remote.domain.tld (web RDP Broker / Gateway)


cheers
dave

edit:
we also had troubles with outgoing smtp traffic. it comes with different IPs so we get blocked very fast.

[franco]

Hi Dave,

I feel that VIPs on WAN are a luxury item. I'm not sure how to "undo" what we did without adding complexity back that we were happy to get rid of while being able to ditch all the fragmented firewall rule and NAT generation code.

Then again Ad wrote this so I'm not sure what is involved in providing such a toggle. Not against tickets asking for it, but someone will have to do that work eventually so it's not coming back fast:

https://github.com/opnsense/core/issues


Thanks,
Franco