Messages

Also very minor addition to this weird thread. Don't use .local as your domain suffix. It is reserved for mDNS (RFC 6762) and queries for it will never reach your DNS resolver. While at it also don't use .test, .example, .invalid and .localhost as they are also considered reserved per RFC 2606 and later RFC 6761.

I can imagine a lot of "weirdness" can be explained by this.

Hi,

I have two core routers in my network that advertise a default route via eBGP private ASN to two OPNSense firewalls in HA mode with FRR installed and configured for this topology.

The design is this way so we can use BGP+BFD for WAN redundancy where failover needs to be very fast.

The BGP sessions and BFD are up and stable, and I can see the learned routes in FRR with vtysh -c "show ip route"

However, the learned routes are never installed in the FreeBSD kernel routing table (netstat -rn).
The kernel still shows a static default route pointing at the interface gateway even when i untick upstream gateway in the gateway config, and removing it from the gateway config on the WAN interface. When removing the default route in the OPNSense CLI the routing table is the same but now without default route.

How can i configure OPNSense and FRR to allow for advertised routes learned from a BGP neigbour to be injected in the FreeBSD routing table?

Thanks and kind regards,

Misja

Hi,

Also interested in trying this out.

Misja

Proxmox as Hypervisor? Sometimes when I have issues with IPv6 I open the vtnet interface affected, change nothing then save and apply and it starts working again. My WAN interface is PCI passthrough and works flawless. I am considering to also apply PCI Passthrough on my LAN interfaces to avoid weirdness (stuff i can't explain nor shows up in a log) like this. But I never had weirdness with IPv4.

Might be something with Proxmox or the VirtIO driver.

Without the TCPdump or logs it is hard to tell what is going on. But considering that one interface is called K8S i think there might be a kubernetes cluster on that side. There is then another network involved in the kubernetes cluster. Depending on the firewall rules that might be blocked. Maybe on the K8S side try to use a rule with source any to any instead of source network to any.

Yeah something seems off with Either MTU or Packet Fragmentation. But it is hard to pinpoint.

@home I run OPNSense as VM inside Proxmox. I know not recommended and stuff. But I find it convenient.

Before I was connected via a Capble ISP and the cable modem in bridged mode. No PPPoE just Ethernet with MTU of 1500 and DHCP/SLAAC for IPv4/6. The Wan Facing interface was also virtual (VirtIO driver). No problems ever with Packet Fragments. I work for a Telco so I can monitor our SBC's and we accept fragmented packets. So packets larger than 1500 bytes are known to arrive and be processed by our SBC.

After that GlassFiber was connect at my place so I changed ISP to make use of that. Now it is PPPoE and my setting are the same as meyergru. When rebooting OPNSense or Proxmox I used to get your issues as well. Then I would reboot the ONT and everything started working again. I could see the packets leaving OPNSense but they never arrived at the SBC. Invisible to see where the packets where dropped. I thought maybe a Auto Neg issue between Proxmox and the ONT. Note this was also UDP packets larger than 1500 bytes.

After a while I was fed up with that. So I performed a PCI passthrough on proxmox letting OPNSense manage the Wan interface and the problem is gone. Everything works as expected again. It seems weird to me as the Operating System is responsible for MTU and Packet Fragmentation (L3) as the OS controls the IP stack. The network driver should be responsible for L2.

I am having a hard time wrapping my brain around this.

@fbantgat7

Is your OPNSense install virtualized and using a virtual NIC connected to an ONT?

Either way, should packet fragmentation break VoIP?

That depends on the Operator. Some allow for packet fragmentation on Access Realms others don't. Most Operators handle larger then 1500 bytes packets with the advice to use TCP instead of UDP. Offcourse they should then allow for TCP.

Oracle (Acme Packet) SBC's now ship with the advise to use TCP for SIP period as INVITES grow bigger and bigger these days.

I never had imagined that a SIP INVITE or REGISTER could be that big.

SIP REGISTER Messages usually don't get very big. That's why fbantgat7 phones where still registering causing me to suspect a MTU issue.

You probably would have noticed problems for some websites that have no PMTU discovery as well.

I think this affects UDP only. But I can't be sure.

At least I am quite sure that using "static port" will do no harm whatsoever. If you have an explanation or a solution for the OP on why his outgoing SIP calls do not work, go ahead.

Up to this point, I still think that the outbound NAT rules are wrong one way or another. Maybe the DNS entries do not map to the suspected IPs, maybe it is even worse and he is behind CGNAT or other type of double NAT.

I agree it won't harm. I just say normally in a B2BUA with symmetric signalling situation which is pretty common these days it shouldn't be needed. Removing the NAT config could also remove errors in his config.

I agree it is probably some misconfiguration and that could also be a misconfiguration of the phone. If he is behind cgnat or double Nat I wish him the best of luck. I hope he has IPv6 working in that case.

[Outbound calls:]

Have here several Gigasets, no special gym needed to make them work. Just some FW rules to allow for the IPs of the provider and done here...

Exactly and this is how it is supposed to work. Normally you should not have the need for static translations. I have a few OPNSense boxes installed at large sites without the need to do anything special to make phones work.

I hope I explain this right.


Outbound calls:

In a line like this in the Message body of an INVITE announces on what port it would like to receive RTP on (the firewall doesn't do anything with this):

Code Select Expand

m=audio 52154 RTP/AVP 8 0 9 101

Likewise in the subsequential 200 OK message the other party announces what port it wants to receive RTP on:

Code Select Expand

m=audio 58130 RTP/AVP 8 101

Also this is part of Codec Negotiation.

After the 200 OK and some other messages the phone will start sending RTP to the port announced in the 200 Ok  from the port itself announced in the First INVITE. Opnsense should create that PAT/NAT Translation and Firewall PINHOLE where the other party can return the RTP to because that stream is being sent and allowed. No proxy required on the Edge side of things.

RTP packet Source port 52154  Destination port 58130

Inbound Calls:

For INBOUND calls the server Sends the INVITE to the Address and (randomised by Edge Firewall/Router) Port that are in the SIP Register cache table maintained by the SBC of the operator. The principle is the same but this time the SIP Server announces it's port in the INVITE and the Client in the 200 OK. The client will again start the RTP stream and so on.

The proxy is in this case on the operators site. And I call this an SBC (Session Border Controller).

The only time I can imagine you would need a static NAT translation is when you don't use the SIP Register mechanism and the SIP Server use static registrations.

If you have access to captures you also might want to check if your INVITES don't exceed 1500 bytes. Fix when it does by disabling CODECS that are not used and diversion headers and stuff like that. Also check the INVITES for IP addresses other than what the phone itself has. Disable NAT Traversal stuff and STUN on the phones might help since they habitually change private IP addresses to public ones. You want your INVITES to contain the real IP addresses of the phones since the SBC of the operator will fix that. If the contents are confusing to the SBC of the operator, it might reply to wrong IP Addresses or ports. Resulting in you not getting replies to INVITES. That's why the majority of operators advice to disable SIP-Helpers Like SIP-ALG and others. OPNSense doesn't have SIP-Helpers by default so you should be good there.

If configured right the SIP Proxy (SIP-ALG Plugin os-siproxd) could also help you by the way. But I never tried anything with it. So I would use it as last resort.

Normally, it shouldn't be necessary to create manual NAT rules for SIP Phones. They should register with SIP, which also creates a dynamic NAT translation and Firewall Pinhole (if not restricted). In an INVITE, there would also be signalling information about what ports and IP's to be used for RTP (the actual Audio) and those should also be the real IP addresses. The best suggestion I have is to leave that process to be handled automatically.

I as a VoIP engineer find OPNSense and PFsense a most excellent Edge Firewalls straight out the box. 

Maybe this is too easy but did you Apply the changes after saving them?

Excellent thanks!

Hi There,

I seem unable to download the Packet Capture PCAP under interface>Diagnostics->Packet Capture -- Jobs. Clicking on the download button causes the pcap to load in the web browser, which is a bit inconvenient.

I tried Firefox and Edge both gave the same result. Seems like a bug to me.

Do others experience the same?

Or Zerotier. That's available as plugin.