Why do I need to temporarily disable firewall to bring up peer's GRE interface?

[Monviech (Cedrik)]

Heres the example setup from the documentation:

https://docs.opnsense.org/manual/how-tos/dynamic_routing_ospf.html#ipsec-failover-with-policy-based-tunnels-gre-and-ospf

I wrote that, but Im not an expert on GRE, but I didnt have issues with it not establishing with the firewall enabled.

Maybe what you have is a new issue, or something that wasnt noticed yet.

Though I know others who e.g., get Hurricane Electric IPv6 nets over GRE and there arent too many reports that something is wrong with that either.

So the simplest assumption is that the firewall rules are wrong somehow.

[lmoore]

Since I'm running GRE over IPSec, do I need a GRE rule on WAN

No - I hadn't fully visualised your set-up when I posted that rule.

In review, you don't mention having OPNsense logging of Default block being enabled. This is where I would expect PF to log the blocked packets arriving from the peer at the other end.

As you don't have an interface for IPsec, testing with a rule which is applied to all interfaces other than your WAN may help;

e.g. setting Invert interface in the rule I provided earlier. You may want to enable logging initially to verify the rule is being used.

Code Select Expand

@329 pass log quick on ! re0 inet proto gre all no state label "9dc1eb15-4dd3-44b1-adbc-04385d9f9955"

[Patrick M. Hausen]

L2TP over IPsec is of course still widely used. And as I stated: the modern alternative - in case you need a dedicated interface for e.g. a routing protocol - is to use a virtual tunnel interface instead of a policy based VPN. OPNsense can do that perfectly well.

I guess your firewall rules are not sufficient for the GRE over IPsec to be permitted. But that is simply too much complexity for my tastes when IPsec with a dedicated interface exists and is well supported.

https://docs.opnsense.org/manual/vpnet.html#route-based-vti

[tbk49]

Since I'm running GRE over IPSec, do I need a GRE rule on WAN

In review, you don't mention having OPNsense logging of Default block being enabled. This is where I would expect PF to log the blocked packets arriving from the peer at the other end.

As you don't have an interface for IPsec, testing with a rule which is applied to all interfaces other than your WAN may help;

e.g. setting Invert interface in the rule I provided earlier. You may want to enable logging initially to verify the rule is being used.

Code Select Expand

@329 pass log quick on ! re0 inet proto gre all no state label "9dc1eb15-4dd3-44b1-adbc-04385d9f9955"

Under firewall | settings | advanced | logging, the option to log default deny/block rule is enabled.

My floating rule looks a little different to yours:

Code Select Expand

pass log quick on ! re0 inet proto gre from {any} to {any} no state label "<label>" # Test rule for GRE
pass log quick on ! re0 inet proto icmp from {any} to {any} keep state label "<label>" # Test rule for ICMP

Different after the protocol and before 'no state label'. I also added one for ICMP. What am I supposed to be seeing? All I see are hits against the 'IN' direction, nothing on 'OUT' except for OSPF to multicast on the GRE interface. The peer's GRE tunnel is still not coming up either.

Fyi, there is an interface for IPsec, it's called enc0 in ifconfig (from fw terminal) or just 'IPsec' in the GUI.

[lmoore]

My floating rule looks a little different to yours

You've extracted the information from /tmp/rules.debug, mine looks just likes yours in there.

The rule I provided was extracted from the ruleset using pfctl.

there is an interface for IPsec, it's called enc0

Sorry, I should have been clearer. I was referring to a VTI.

I was hoping logging rules would at least provide you with some clues, but it seems that it isn't.

My old OpenBSD firewall had some IPsec tunnels configured on it. The pf rules for IPsec were written along the lines of 'pass in on enc0 .... no state'.

There is one which is written differently;

Code Select Expand

pass in log on enc0 proto ipencap from bbb.bbb.bbb.0/24 to aaa.aaa.aaa.12 keep state (if-bound)

This one was set up 12 years ago and I don't recall why ipencap (protocol 4) was used, though there is another rule to the remote peer. I would have set up the remote peer and as it was a FortiGate, I most probably set it up with a VTI.

I also had a static route to bbb.bbb.bbb.0/24 via aaa.aaa.aaa.12.

I doubt any of this is of any help in your situation.

Have you considered the suggestions from @Patrick & @Cedrik?

[tbk49]

Heres the example setup from the documentation:

https://docs.opnsense.org/manual/how-tos/dynamic_routing_ospf.html#ipsec-failover-with-policy-based-tunnels-gre-and-ospf

I wrote that, but Im not an expert on GRE, but I didnt have issues with it not establishing with the firewall enabled.

Maybe what you have is a new issue, or something that wasnt noticed yet.

Though I know others who e.g., get Hurricane Electric IPv6 nets over GRE and there arent too many reports that something is wrong with that either.

So the simplest assumption is that the firewall rules are wrong somehow.

Yes, I don't think my firewall is operating correctly, but only because turning it off brings up the other side's GRE tunnel, but I feel like there could be other variables involved.

For example, let's ignore GRE, and just focus on IPsec. I have a number of peers and profiles for each peer. So I have multiple loopbacks for this purpose. If I set one of them to a /32 address and adjust the peer's loopback address the same, just like in your how-to, once the IPsec re-establishes, if I ping the opnsense IPsec loopback IP from the peer on the other /32, I cannot see the ICMP traffic on enc0 (tcpdump -nvv -i enc0) even though the peer reports receiving ICMP replies back! However, if I revert back to a /30 address for the peer and opnsense loopbacks, I can see the ICMP traffic on enc0!

If I have a floating firewall rule for ICMP and set "state type" = none, when the IP's are /32, I don't see anything in the live log, but I see plenty of IN and OUT when the IP's are /30.

Is that expected behaviour? Seems abnormal to me.

[tbk49]

I was hoping logging rules would at least provide you with some clues, but it seems that it isn't.

I suspect some traffic isn't being either picked up on enc0 or passed to pf and this is why I'm having difficulty pinning down the problem. See my reply to Cedrik.

Have you considered the suggestions from @Patrick & @Cedrik?

I actually used that how-to Cedrik wrote when I was setting this up. I don't know what to do. Maybe I'll look at the VTI, but the reason I didn't go with that option in the beginning was because of the caveat mentioned in the 'important' box in the doc (https://docs.opnsense.org/manual/vpnet.html#ipsec). I didn't want to go with VTI and later find this caveat to be an issue for my situation later down the line, when comparatively speaking, the GRE over IPsec seemed much simpler to do.

Thanks for the "no state" option. That's useful to see both in and out traffic in the live log. "keep state" doesn't show the return because it gets classed as related traffic in the state.

[Patrick M. Hausen]

The caveat is essentially: don't mix policy based and VTI based connections. Pick one of them for all peers.

[tbk49]

I guess your firewall rules are not sufficient for the GRE over IPsec to be permitted. But that is simply too much complexity for my tastes when IPsec with a dedicated interface exists and is well supported.

The firewall rules to allow GRE over IPsec are straightforward. Allow GRE protocol on IPsec interface, ICMP, and OSPF. Then make the GRE interfaces accept anything from anywhere just to test connectivity. Rules can be made more restrictive when the baseline connectivity is confirmed.

I'm looking at VTI. The guide says policies are attached to VTIs by reqid. So is this only an option for two opnsense firewalls? Or can the reqid be specified in the client's config somewhere irrespective of its vendor?

Also, I cannot interpret the topology in the doc (https://docs.opnsense.org/manual/how-tos/ipsec-s2s-conn-route.html#preparations). If both firewalls are separated by public internet, and have WAN IP's assigned dynamically, how does this change the required properties of the VTI config as shown?

[nero355]

For example, let's ignore GRE, and just focus on IPsec.

I have a number of peers and profiles for each peer. So I have multiple loopbacks for this purpose.
If I set one of them to a /32 address and adjust the peer's loopback address the same, just like in your how-to, once the IPsec re-establishes, if I ping the opnsense IPsec loopback IP from the peer on the other /32, I cannot see the ICMP traffic on enc0 (tcpdump -nvv -i enc0) even though the peer reports receiving ICMP replies back!
However, if I revert back to a /30 address for the peer and opnsense loopbacks, I can see the ICMP traffic on enc0!

If I have a floating firewall rule for ICMP and set "state type" = none, when the IP's are /32, I don't see anything in the live log, but I see plenty of IN and OUT when the IP's are /30.

Is that expected behaviour?
Seems abnormal to me.

Sounds logical to me : In order to connect two peers you need at least a /30 subnet :)