Hello, guys. I really hope that there are experts among you.
I have next setup:
First site:
OPNsense edge gate\fw (ISP public ip, for example 1.1.1.1) (DMZ ip: for example 10.1.1.1)
Cisco Router in DMZ with tunnels (GRE) interfaces: 10.1.1.2
Second site:
OPNsense edge gate\fw (ISP public ip, for example: 2.2.2.2)
I have working GRE tunnel by scheme:
S2 OPNSense -> MyOPNSense -> (NAT GRE) -> Cisco
by this scheme I have ~60-80mbit troughpout
Today, for testing I made GRE tunnel in local network (vm to cisco), and I get over 600mbit!
Maybe, in OPNsense have settings for GRE over NAT? Because it's very strange.
What can bottleneck?
Ex configs:
Cisco:
interface ga0/0 - ip address 10.1.1.2/24 (DMZ)
interface Tunnel2
(GRE) ip address 10.0.91.1/30
(GRE) tunnel source 10.1.1.2
(GRE) tunnel destination 2.2.2.2
S2 OPNsense:
em0 (WAN, public ISP, anyway...) - ip: 2.2.2.2
gre0:
source - 2.2.2.2
destination - 1.1.1.1
gre local 10.0.91.2/30
If need, I can provide more info
And sorry for my bad english
I'd say you need some MTU/MSS clamping. Firewall ‣ Settings ‣ Normalization. I'd start with something like 1360 for the Max MSS.
Good idea. But, it is globally, right? On GRE sites (cisco\opnsense mtu set to 1476)
From here, I have next question - where natting GRE, we decrement MTU?
No, it is not globally, it's per interface (group). See the hints here: https://docs.opnsense.org/manual/how-tos/wireguard-s2s.html
so, I'm trying start iperf with udp, (iperf3 -c server.behind.tunnel -u -b 120M) and get 120Mbit\s (of course with losses, its UDP). But I don't understand yet, where do I need to set MSS? In the firewall settings (normalization) or on the gates with GRE? or both (OPNsense with GRE, OPNsense with NAT GRE and Cisco with GRE)?
What's unclear about the normalization settings from the link I've posted? (It's about Wireguard but it's exactly the same place and same settings -- just applied to GRE interface(s).