Multi WAN Routing from Postfix on OPNSense Instance

[dawc21]

Hello Smart People!

I have been banging my head against the wall for 3 days with this one and could really use a fresh look at this.  The solution is probably real simple but I am just not able to see it.  I know just enough about firewalls / networking to be dangerous... 

The issue:
I have 2 WAN Connections configured in a failover / Load balanced Configuration and for the life of me cannot seem to force all SMTP Traffic outbound through one of the WAN Connections.

Mail config:
Postfix running on OPNSense and as a relay along with RSPAMD
Mail Server Running on Internal LAN

For the sake of this discussion I'll get real scientific and call the connections WAN_1 and WAN_2 with the following characteristics:

WAN_1 - SLOW! and reliable 10 Mbit connection - Static IP - Reverse Look IP look ups possible for hostname - SPF Records - Able to Send / Receive email

WAN_2 - FASTER and Semi Reliable 150 Mbits Connection (Starlink Beta  ) - No Reverse Lookup possible - ISP Nat'd - Address Changes - Unable to receive email. Outbound email is typically rejected from foreign mail systems due to obvious Reasons .

I also have a Gateway Group aptly named WAN_GW_GROUP. WAN_1 is configured as Tier 1 , weighted 1 and WAN_2 Is Second tier weighted 5

I would like to Force all outbound Postfix (Running on OPNSense) SMTP Though WAN 1

Things I have tried with Zero Success:

A) I have tried binding postfix to the WAN_1 outbound IP address.  This works.  All email goes out from the correct IP address however all inbound email begins to fail due to the External IP address not being able to reach the LAN side of the firewall. (Makes sense) (Postfix is bound to External IP )

B) Create a Floating outbound rule bound to WAN_1 and WAN_2 forcing the gateway of WAN_1.  This almost works however with WAN 2 Weighted heavier almost all traffic attempts to go in this direction however WAN_2 is unable to reach the WAN_1 gateway (again makes sense )

C) Tried rules all over the place in multitudes of combinations and permeations that do not achieve what I am looking to accomplish.

Any insight anyone can offer will be greatly appreciated. 

[AbstractGeo]

So, I'm coming from a "previous project" background - apologies if I get any of this wrong!

Probably you want outgoing mail to *never* go out via WAN_2 - Starlink, right?

So, you'd create a firewall rule - on LAN (or your DMZ if it's a seperate network)
The rule will look like:

• Source IP: Mail Server
• Destination IP: All
• Destination Port: 25
• Transport/Protocol: TCP
• Gateway: WAN_1GW This is the magic bit!

(Why did I post if I'm not current w/OPNSense? Well, my OWN starlink setup just arrived, so it's time to get away from my nightmare bonded DSL - 12 megabits, WOW - and I'm going to want to use a real platform; I was curious how people's experience with Starlink & its various bits were!)

[Neo]

Couple of things here from setting up my own multi-wan configuration recently (and others with more OpnSense experience, please correct me if I've got something wrong)...

First, If your goal, aside from routing SMTP for Postfix, is to have all traffic use WAN2 and only use WAN1 if WAN2 is down then I think you may need to reverse your gateway setup:

WAN2 would have the higher weight of 5 and be Tier1
WAN1 would have the lower weight of 1 and be Tier5 (or even Tier2)

If your goal is to load balance such that the faster and slower link are aggregated with the fast link being used much more often then the slower link then you'd want both WAN1 and WAN2 gateways on the same Tier but with the greater weight given to the faster link... I would personally tend to recommend against that for a couple of reasons: (A) the ratio of 10 megabit to 150 megabit is just to wide a gap to effectively load balance, and (B) the "load balancing" in OpnSense is somewhat limited since, as I think I understand it, it is basically just doing a "weighted round-robin on new connections". Thus my opinion, for whatever that is worth, is that you'd be best setting your slower connection up to be fail-over only (Tier5), having your main Firewall rule push traffic through your routing group, and then writing rules that override that for traffic you want to use the slower link for on purpose...

Which brings me to your original question...

I'm not familiar with Postfix specifically or setting that up to run on the OpnSense itself. However, if you can configure it as bound to the LAN and assuming that configuration would cause OpnSense to treat it as traffic originated on the LAN (as opposed to seeing it as some alternate form of "internal" traffic) then you should be able to create a single rule, in Firewall / Rules / LAN (placed ABOVE the general rule routing traffic via the gateway group) that routes SMTP over WAN1...

You can see in the attached snapshot I've done something similar to cause all traffic headed to DYNU to utilize WAN2 (that, in my case, is the link that has the dynamic IP I'm trying to update)... Note that this rule must be placed ABOVE the rule with the gateway group so it acts as an "override"...

Hope that helps.

[mimugmail]

SMTP Client bind address on LAN, then do gateway rule via pf for outbound. For inbound a portforward in WAN1 to LAN address

[dawc21]

Thanks for taking the time to Reply!

Unfortunately the outbound sender is postfix on OPNsense.  If it were on the LAN this would be gravy.

Im sure its something really silly but I'm just missing  it.

So, I'm coming from a "previous project" background - apologies if I get any of this wrong!

Probably you want outgoing mail to *never* go out via WAN_2 - Starlink, right?

So, you'd create a firewall rule - on LAN (or your DMZ if it's a seperate network)
The rule will look like:

• Source IP: Mail Server
• Destination IP: All
• Destination Port: 25
• Transport/Protocol: TCP
• Gateway: WAN_1GW This is the magic bit!

(Why did I post if I'm not current w/OPNSense? Well, my OWN starlink setup just arrived, so it's time to get away from my nightmare bonded DSL - 12 megabits, WOW - and I'm going to want to use a real platform; I was curious how people's experience with Starlink & its various bits were!)

[mimugmail]

When you hit advanced you can set the outgoing client IP of postfix when sending. If its LAN, then your multiwan firewall rule will kick in

[dawc21]

Thank-you for the response.  I have bound postfix to the lan address however when email is leaving postfix it appears to be coming outbound from the Active Gateway and then is caught by the firewall rule.  It then sits queued as the preferred gateway is not reachable by postfix.

[

quote author=Neo link=topic=22085.msg104827#msg104827 date=1616020613]
Couple of things here from setting up my own multi-wan configuration recently (and others with more OpnSense experience, please correct me if I've got something wrong)...

First, If your goal, aside from routing SMTP for Postfix, is to have all traffic use WAN2 and only use WAN1 if WAN2 is down then I think you may need to reverse your gateway setup:

WAN2 would have the higher weight of 5 and be Tier1
WAN1 would have the lower weight of 1 and be Tier5 (or even Tier2)

If your goal is to load balance such that the faster and slower link are aggregated with the fast link being used much more often then the slower link then you'd want both WAN1 and WAN2 gateways on the same Tier but with the greater weight given to the faster link... I would personally tend to recommend against that for a couple of reasons: (A) the ratio of 10 megabit to 150 megabit is just to wide a gap to effectively load balance, and (B) the "load balancing" in OpnSense is somewhat limited since, as I think I understand it, it is basically just doing a "weighted round-robin on new connections". Thus my opinion, for whatever that is worth, is that you'd be best setting your slower connection up to be fail-over only (Tier5), having your main Firewall rule push traffic through your routing group, and then writing rules that override that for traffic you want to use the slower link for on purpose...

Which brings me to your original question...

I'm not familiar with Postfix specifically or setting that up to run on the OpnSense itself. However, if you can configure it as bound to the LAN and assuming that configuration would cause OpnSense to treat it as traffic originated on the LAN (as opposed to seeing it as some alternate form of "internal" traffic) then you should be able to create a single rule, in Firewall / Rules / LAN (placed ABOVE the general rule routing traffic via the gateway group) that routes SMTP over WAN1...

You can see in the attached snapshot I've done something similar to cause all traffic headed to DYNU to utilize WAN2 (that, in my case, is the link that has the dynamic IP I'm trying to update)... Note that this rule must be placed ABOVE the rule with the gateway group so it acts as an "override"...

Hope that helps.
[/quote]