Link-net with CARP and HA

[chuckygang]

Well new here.  Guess this is asked but cannot find it.

I have a /25 net that is delivered to me via a "LINK Net" on 2 connections.  so I have 2 machines setup like:

port1 to ISP with IP z.z.z.2 witth carp to z.z.z.1, port 2 to our WAN switch with ip x.x.x.2 and carp to x.x.x.1

(and a 2nd opnsense with port1 to ISP with IP z.z.z.z.3 with carp to z.z.z.1 and port 2 to our WAN switch with IP x.x.x.3 and carp to x.x.x.1

so this linknet is a small /29 just to handle the link..

thing is.. I seems not to be able to pass any traffic EXCEPT pings to my WAN net.
if I check firewall logs I see the default deny rule triggers on my traffic. but even if setting rules on all ports to allow all it is refused.

so what more do I need to do to allow traffic to pass though.. NOT NATed between those 2 ethernet ports?

Any good tutorial to handle this?


SO YES.  there are public IPs on both sides.
(if possible to have a 3rd network with NAT for managment it would be a plus)

[chuckygang]

Disabling firewall will make traffic go through. 

but this exposes the opnsense GUI to the world so that is not a real option

[Monviech (Cedrik)]

If you only have public IPs, did you set Firewall: NAT: Outbound to "Manual outbound NAT rule generation
(no automatic rules are being generated)"?

[chuckygang]

Yes.. it is set to manual.

machines in the "WAN" part can go out on the net perfectly.   and checking its IP it does get the iP it is  set up just to confirm that no NAT is being done.

but traffic from outside -> in does not work unless I disable the firewall completly..  EVEN if I have a "Allow all" rule set (do not worry.. just to test at the beginning)

[Monviech (Cedrik)]

Can you try to create an "Allow Any" rule in "Firewall: Rules: Floating"? If you don't select an interface it matches on all interfaces. It's great for troubleshooting.

[chuckygang]

Done that.  still not working.. and that really puzzles me.
as in the log I can see my traffic being caught by the default deny-rule.

I even tried to match the source/destination IP  mask   and everything just to see if I can trigger it. but no.

[Monviech (Cedrik)]

I have a suspicion, but to prove it can you create the Floating Any Allow rule. Set the protocol of the rule to "TCP" and at the bottom of the rule "Advanced features", enable "TCP flags - Any flags" and "State Type - Sloppy State".

Then try to connect to a TCP destination, like with ssh, http or https.


Suspicion: https://docs.netgate.com/pfsense/en/latest/troubleshooting/asymmetric-routing.html

[chuckygang]

Hmm yes THAT works..

[chuckygang]

Will read up on the link you sent and check more about this.  thanx for hinting me to a good direction.

[Monviech (Cedrik)]

Happens quite often somehow.

Always make sure you seperate layer 2 broadcast domains. (Which means, different switches or different VLANs)

Each IP Network should have its own VLAN or switch for routing to work right. If there are shortcuts on layer 2 with ARP protocol or on layer 3 with ICMP Redirects, you get asynchronous routing which makes TCP fail.

[chuckygang]

the link you linked to shows the automatic fix to "Bypass firewall rules for traffic on the same interface"  but this did not work. (most likly as it is on 2 different interfaces)

but now I have more to work on and will dig into this tonight when he phone does not ring

[chuckygang]

I did put a VM on the other net with windows server. and just to do some tests (this macine will be removed) with RDP up..

I can connecty to rdp from outside.  but it cuts connections a short time aprox every 2nd minute.

and now I cannot connect anything OUT from the VM except pings..
no logs in firewall logs. and even turning off firewall totally no traffic out.

it worked once.  but then magically dead. (except the magic. PING!)

[chuckygang]

Well that can be ignored. a reboot and all seems to be stable now.. 

so  thank you for pointing to the correct direction.  now some final tests and then time to go to productionmode