Messages

Good-day Folks,

I have two networks that I want to keep isolated but allow a select group of PCs to be able to reach both LANs (for management).  The networks are on two separate physical interfaces on my OPNSense box.   Subnets are:

• LAN-A 10.0.10.0/24
• LAN-B 10.0.20.0/24

I have everything working thus far, but noticed that pings from LAN-B (which is the new one I created) are getting routed out the WAN interface (I confirmed with a traceroute from a host on that network).  However, from the OPNSense router itself, the traceroute and ping works as expected.

What am I missing here?

Yep, everything checks out - so it's really throwing me for a loop.  Over 25yrs of SysAdmin experience and I thought I'd seen it all, but nope.

I did a Wireshark capture (screenshot below) and it confirms two way traffic, so doesn't look like OPNsense is blocking anything.  Hmmm

[Good-day Folks,]

Good-day Folks,

I have a strange issue that I have been troubleshooting for a couple of days now, and I'm at my wits end.  Can't seem to figure it out, but all seems to be pointing to OPNsense as the possible culprit; in that, it may be silently filtering out traffic I need for a critical application on my network - QuickBooks.

My environment:

Code Select Expand

OPNsense 23.1_6-amd64
FreeBSD 13.1-RELEASE-p5
OpenSSL 1.1.1s 1 Nov 2022

I run a small network for my Church and use OPNsense as my edge router/firewall.  The problem I'm having is that certain functions inside of QuickBooks, which rely on the application being able to open a particular web application, are failing and I can't figure out why.  I've spent several hours, over multiple days, with the QuickBooks Support Team and they have not been able to resolve the problem but have repeatedly insisted that perhaps my local firewall may be blocking traffic to their cloud platform (which is hosted on Amazon AWS).  So, to test, I simply installed QuickBooks on my home PC and tried to replicate the problem.  Unfortunately for me, it worked there - confirming that their hunch was correct.  My edge firewall is, somehow, filtering out the traffic.

So to troubleshoot, did the following:

• I disabled the local Windows Firewall - no effect. 
• I created an ALLOW ALL firewall rule on the WAN interface - no effect.
• I disabled the packet filter on OPNsense - no effect.

I don't have any IPS/IDS services running on the OPNsense box, nor do I have the Sensei plugin running either.  Additionally, I ran the SysInternals Procmon utility on the PC having the issues, and I was able to capture the URLs QuickBooks is attempting to reach.  They are (1) ec2-35-161-218-244.us-west-2.compute.amazonaws.com and (2) ec2-44-239-233-20.us-west-2.compute.amazonaws.com; both of which I confirmed via the OPNsense live view firewall log, are reachable and not being filtered.

The only test I have not run yet is to physically bypass the OPNsense device and plug my core switch directly into the ISP router.  Before I do that, though, I thought I'd reach out to the community to see if anyone has run into a similar issue and could offer up some advice.  All help is greatly appreciated, please!

Screenshot showing Procmon Capture of Traffic from QuickBooks


Screenshot showing OPNsense Firewall Logs of the traffic being allowed

I recently had a major speed issue when using wireguard, turns out that I didn't have my hardware accel. features enabled in System->Settings->Misc.

Thanks to your pointer, I checked and discovered that I didn't have any Hardware Acceleration enabled.  I went ahead and enabled all the options available and rebooted the router.  That seems to have done the trick, and I'm not getting near wire speed (see screenshot below), and the Waveform Bufferbloat tester is now returning an "A" score when I enable the Traffic Shaper rules.  With the rules off, I get a "B" score.

I'm gonna mark this issue as resolved for now.  Thanks again for your help.

A quick chime-in -- I doubt buffer bloat is having that significant of an effect.  Keep digging. There must be a major bottleneck somewhere. 

Yeah, I have suspicion that this may not be my issue as well.....still troubleshooting.

You didn't inadvertently set up a 100Mbps interface speed by accident?  Running over WiFi?

Nope.  I checked and confirmed that the interface is set to "autoselect" and I can see that it has autonegotiated to a value of "1000baseT <full-duplex>"

I recently had a major speed issue when using wireguard, turns out that I didn't have my hardware accel. features enabled in System->Settings->Misc.

Thanks for the pointer, I will check my configuration and see.

A quick update -

I've been reading up on Bufferbloat, and suspecting that perhaps that may be at play here, I setup traffic shaping for both Upload and Download traffic using the FlowQueue-CoDel scheduler type.  However, I don't think the shaper is really working, because when I look under Firewall > Shaper > Status, I don't see any packets.  I'll admit, using the traffic shaper is new to me, so I may have missed something.

Anyway, here's what the Waveform Bufferbloat tester is showing for my current settings:

[Hello Folks,]

Hello Folks,

I trust you're all doing well.  So today I setup a second WAN link (1Gbps/1Gbps from Verizon) and added it to a WAN Group that I setup, following this tutorial - https://blog.actorsfit.com/a?ID=01100-f41de98e-89dc-4697-8a3c-c8b83251046b.  Everything seems to be working, but I'm noticing that my download speeds aren't getting above 100Mbps but can't seem to pinpoint where the drop is coming from.

As you can see from the two separate speed tests below, the upload speed is fine - which possibly rules out bad cable - so I'm suspecting that I may have missed a setting somewhere my OPNSense configuration.  Any pointers in the right direction is greatly appreciated, thank you all.

I'm running OPNsense 22.1.6-amd64

I noticed earlier today that I am unable to make changes to my OpenVPN Server configuration.  After making a change and clicking the Save button, all everything on the Web UI indicates that the action was saved.  But when I click the edit button to confirm that the change has taken effect, I see the old values.

Is anybody experiencing this problem as well?

I have been looking at the logs and whenever I do a ping I don't see a block.  I have a floating rule to allow ICMP on all interface and I see that rule get triggered, but then the ping doesn't go anywhere.

Strange thing is, as you can see in the screenshot below, I am able to ping out to the Internet from that very same device.  But for whatever reason I cannot ping it from the OPT1 interface and it cannot ping the OPT1 interface.....weird.

I just wanted to add a quick update that for the same systems that aren't pingable, I am able to create a port forward to port 80/tcp and reach their web interface without any issues.  Someone I think my firewall rules aren't allowing the ICMP traffic but I'm just not seeing where the problem is coming from.

[Good-day folks,]

Good-day folks,

So I have an OPNsense box with three LANs defined as follows:

• LAN - 10.0.10.1/24
• OPT1 - 10.0.11.1/24
• OPT2 - 10.0.12.1/24

The appropriate firewall rules are in place to ensure that devices behind those networks can route out to the Internet and all seems okay - so no issues there.

The problem I'm having is that, I am unable to ping a couple of devices on each of those networks (even from the interfaces directly, using the diagnostics tools in the Admin Interface).  These are Wireless Access Points that I'd like to add to my monitoring system and monitor their uptime.  At first I thought that perhaps it was the devices themselves that were rejecting the ping packets, however, I pulled each of them off, connected them to an unmanaged switch and viola, I could ping them.  So the issues appears to be on my OPNsense firewall.

I manage this box remotely using an OpenVPN tunnel, which is configured with the above local networks.  And with this, I am able to successfully ping the interface address of each network (as evidenced below).

Code Select Expand

C:\Users\kisme>ping 10.0.10.1 && ping 10.0.11.1 && ping 10.0.12.1

Pinging 10.0.10.1 with 32 bytes of data:
Reply from 10.0.10.1: bytes=32 time=15ms TTL=64
Reply from 10.0.10.1: bytes=32 time=15ms TTL=64
Reply from 10.0.10.1: bytes=32 time=15ms TTL=64
Reply from 10.0.10.1: bytes=32 time=16ms TTL=64

Ping statistics for 10.0.10.1:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 15ms, Maximum = 16ms, Average = 15ms

Pinging 10.0.11.1 with 32 bytes of data:
Reply from 10.0.11.1: bytes=32 time=15ms TTL=64
Reply from 10.0.11.1: bytes=32 time=17ms TTL=64
Reply from 10.0.11.1: bytes=32 time=15ms TTL=64
Reply from 10.0.11.1: bytes=32 time=16ms TTL=64

Ping statistics for 10.0.11.1:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 15ms, Maximum = 17ms, Average = 15ms

Pinging 10.0.12.1 with 32 bytes of data:
Reply from 10.0.12.1: bytes=32 time=15ms TTL=64
Reply from 10.0.12.1: bytes=32 time=15ms TTL=64
Reply from 10.0.12.1: bytes=32 time=15ms TTL=64
Reply from 10.0.12.1: bytes=32 time=15ms TTL=64

Ping statistics for 10.0.12.1:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 15ms, Maximum = 15ms, Average = 15ms

Unfortunately, a traceroute from one of the interfaces in question fails:

Code Select Expand

# /usr/sbin/traceroute -w 2 -I  -n  -m '18' -s '10.0.12.1'   '10.0.12.201'
traceroute to 10.0.12.201 (10.0.12.201) from 10.0.12.1, 18 hops max, 48 byte packets
1  * * *
2  * * *
3  * * *
4  * * *
5  * * *
6  * * *
7  * * *
8  * * *
9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *

What am I missing here?  Any help/guidance is appreciated, thanks.  Ready and willing to post whatever portions of my config are needed, just ask please.

The way to mitigate this is to trunk VLAN's through more than one physical link into a LAG for resilience.

Bart...

Aah, I never considered a LAGG.  Will look into it and see if I can leverage that.  Thanks for the suggestion.

Hi kagbasi-wgsdac,
I use VLANs with OPNSense. Virtual interfaces are linked to physical ones.
Cheers...

Darn, I was hoping you wouldn't have said that.  Unfortunately, this means that if the physical interface goes down, so does any VLAN attached to that interface.  Hmm, that poses a risk to us and I'll have to rethink and look for an alternative then.  I really wanted to push to use OPNSense for our core router on this project, but will be tough sell if VLANs are attached the physical NIC.

Quick question,

I'm part of a network design project for a Wireless Internet Service Provider and the topic has come up about how they're going to handle multiple customers.  A suggestion was floated about purchasing the OPNsense Quad Core Gen4 10GB 4 port SSD device (DEC4640) and creating VLANs.  However, another question was raised by the Network Engineer, who's coming from a Cisco environment, about how OPNSense implements VLANs.

Does it do the "Router on a Stick" approach - whereby the virtual interfaces are dependent on the availability of the physical NIC they are attached to? or

Does it do what Cisco does in their IOS and create real virtual interfaces that are detached from the underlying NIC?

Hope someone from the OPNSense team or Decisio can provide some feedback on this, thanks.

Mine did that as well after the upgrade, once I rebooted it was fine.

Thanks for that information.  I will find some time today and reboot and see if I get a similar outcome.  Thanks again.