Trying out OPNSENSE and must be missing something related to performance config

[brainchill]

Forgive my ignorance, I'm clearly new to opnsense but I figured that I would give it a try.

I used m0n0wall for years and was kind of turned off by the throw everything and the kitchen sink approach that PFsense took so I migrated to endian for a long time.

My issue is that I've just installed the amd64 version of opnsense on my box and I seem to be getting about 15% less throughput down than I did on the same box using endian.

This is the box that I'm running it on

https://www.amazon.com/gp/product/B01GIVQI3M/ref=oh_aui_search_detailpage?ie=UTF8&psc=1

When I boot it into endian or even just bare ubuntu with iptables nat and firewall running I consistently get
930-950Mbps down and nearly exactly the same up but if I boot into opnsense the best I ever get is 790 down but about 900 up.

I've repeated this test about a dozen times. I've tried adding more ram, taking away ram, etc and am just not getting anywhere. Is there something that I can check/do to get the performance to be closer to the same since we are talking about an apples to apples comparison hardware, cabling, etc wise and the only difference is the OS?

[opnfwb]

Based on the screenshots I'm assuming this is AT&T gigapower? I had gigapower for a while and used OPNsense with it and constantly had 941/941 speeds. However, my setup isn't exactly like yours but, I do run intel NICs.

I would suggest simple things first from a tuning standpoint to see if you can rule out issues with the way OPNsense is using the NICs on that Protectli box. Forum member dcol has an excellent Intel NIC tuning guide posted in the IDS section: https://forum.opnsense.org/index.php?topic=6590.0

The main things I would check would be to make sure your WAN and LAN NICs have different IRQs, make sure they're using MSI-X, and turn off flow control on both of those NICs (all of these items are covered in dcol's guide linked above). Beyond that, are you using the AT&T provided gateway in DMZ passthrough, or did you bypass the gateway completely? I personally bypass the AT&T gateway when I had gigapower and just plumbed OPNsense WAN port directly to the fiber PON. This made troubleshooting and speed tweaking a lot simpler because I wasn't relying on the AT&T equipment to keep up.

[opnfwb]

Just remembered another thing worth checking would be traffic shaping. Since you're using a new install, this probably isn't an issue. But a traffic shaper can influence those speed tests.

When I had AT&T gigapower and I enabled fq_codel without any tuning, it would actually look exactly like your OPNsense speed results. Downloads would be the 600s, uploads were 700s to 800s. If you have any traffic shaping enabled, turn that off first and see what happens.

[GPz1100]

I personally bypass the AT&T gateway when I had gigapower and just plumbed OPNsense WAN port directly to the fiber PON. This made troubleshooting and speed tweaking a lot simpler because I wasn't relying on the AT&T equipment to keep up.

Would you be so kind as to provide the script(s) you used to set this up.  I'm trying to do the same with a few scripts that are floating on various sites but have run into roadblocks.  Please detail your configuration too.  From what I understand the gateway has to remain for periodic authentication with the mothership, but all traffic is actually processed by the firewall (opnsense).

Thanks!

[opnfwb]

I did not use scripts to accomplish this. There are some scripts available on Linux based platforms, most of which are posted on DSL reports forums.

In my case, I used a "smart" switch with 3 ports in a VLAN segment. Plug in the ONT and the WAN port on the PACE gateway to the VLAN segment on the smart switch. Wait for the PACE gateway to negotiate with ATT and do all of the 802.1x authentication. When this is done, the PACE gateway will show two green lights and internet traffic will work.

At this point, unplug the WAN port of the PACE gateway from the switch, and plug in your OPNsense WAN port to the same VLAN segment. You need to ensure that OPNsense WAN port has the MAC of the PACE gatway spoofed to the OPNsense WAN. When you plug in the OPNsense WAN port, ATT will automatically issue you an IP address because the PACE gatway already authenticated the fiber connection for the ONT. As long as the port never goes down, the ONT will stay online indefinitely, and your IP will renew every 2 weeks automatically using OPNsense only. Just make sure to keep the "smart" switch online so that it doesn't drop the connection, if it drops, you'll have to do 802.1x port swap again.

You could script this within OPNsense, I am quite sure. However, I did not have that knowledge and the physical switch made it easier for me to work around. I used this config for 6 months straight without a single issue. The only downside to ATT was that they filter youtube traffic, even though its 1gbit fiber. So I'd go to watch a 1080p video, and it would buffer and stutter and drop down to 320p, pathetic for a 1gbit connection. I get better performance from the cable company on their DOCSIS network. I would not recommend ATT unless they are your only option. Good luck, and post back results of your setup.

[GPz1100]

Got it - i believe this is called the swap method.  I may just have to give that a shot.  I don't have a managed switch but am somewhat familiar with robocfg on the asus routers with merlin firmware. I've defined a few vlans already for internal lan use so this wouldn't be much different.

Which vlan vid did you configure your switch to?  And why the 3 ports when you're only using 2 of them?

My goal is a more automated solution where the gateway either remains plugged in or can be plugged in periodically to renew the authentication.  During this 6 months no issues renewing?  I recall reading in other threads that eapol renewal occurs every 14 days.

Absolutely no issues with youtube here on att gigabit.  I've streamed 720p/hd/4k/8k successfully.  Granted 8k stutters because of lack of gpu/cpu processing power, not lack of bandwidth.  If you have snort enabled for web traffic, that might be your issue.

[opnfwb]

The initial idea was that I could leave the PACE gateway plugged in, and just swap VLANs back and forth on the switch through the management interface without having to physically plug/unplug the PACE gateway. In my experience, this wasn't necessary, because I kept the PON, the switch, and my OPNsense box on a UPS. As long as the PON ethernet link never goes down, it doesn't need to reauth in my experience. Unless ATT changed something recently.

Regarding youtube throttling, I know it was ATT because there were two massive threads on the ATT forums at the time with people all over the US complaining about it. I could also instantly resolve the issue by connecting to a VPN, and the same video I was watching would instantly buffer and play fluidly without a problem. Disconnect the VPN, instant video buffering. Maybe ATT fixed their network now, but I've had similar issues with them in the past using their UVerse service and also throttling video services (youtube, vimeo, etc.). This is now pretty far off topic but, you get the idea. If it's working well then I hope that it stays stable for you.

Regarding the VLAN on the switch, it doesn't matter what VLAN # you segment the ports to. It just can't be VLAN0, so an unmanaged switch didn't work for me. I had to use one of those small 8 port "smart" switches and manually group 3 ports in a VLAN segment to get the PACE gateway to auth with the PON. After that, I could use the swap method and it worked fine. I never had to reconnect the PACE gateway unless I unplugged the ethernet port between the PON and the switch. As soon as the PON detects that the ethernet port is down, you will need to re-auth, or at least I had to do that 6 months ago when I had the service. Hopefully this helps, post back with your results, I'm curious to know if this method is still working.

[GPz1100]

Can you post a pic of how you have the vlans configured on the switch (vlan configuration screen)?

I have the newer gateway, the bgw210 which from everything i've read uses tagged vlan0 explicitly to communicate with the ONT.

[opnfwb]

I don't have screenshots of the setup and I'm not using the switch anymore so I can't post an exact config at this point. I'm going from memory but basically, what I recall is this.

I could use a "dumb" switch to always get the PACE gateway online. It looked like this:
ONT -----> DumbSwitchPort1
PACE ---> DumbSwitchPort2

With that config, the PACE gateway would always show green lights and the service was up. However, unplugging the PACE gateway and plugging in OPNsense with the spoofed MAC did not pull an IP address.

Using the same setup, but with a statically assigned VLAN, the OPNsense WAN port then pulled an IP. Something about the VLAN tagging between the gateway and the ONT was causing the dumb switch to not push all the traffic to the OPNsense box. Manually setting a static VLAN (such as VLAN10, or VLAN5, it doesn't matter) pushes all of the VLAN0 traffic between the switch ports defined with the static VLAN and allowed the OPNsense WAN port to receive the traffic and pull an IP. After that it was off to the races. It worked very well. Also worth noting, I only had Internet service, I did not have any TV or phone service. So this may be entirely different if you have other services riding inline with the internet service.

[GPz1100]

I'm going to give this a shot.. Got a netgear gs108ev3 on the way.  I am using the bgw210 gateway.  From what I've read, this (the bgw) was the gateway that some ngv598/599 customers received due to wifi issues or other reasons.  It goes to reason that att wouldn't change the authentication process just because of an upgraded gateway.

Wish me luck

[GPz1100]

I don't have screenshots of the setup and I'm not using the switch anymore so I can't post an exact config at this point. I'm going from memory but basically, what I recall is this.

I could use a "dumb" switch to always get the PACE gateway online. It looked like this:
ONT -----> DumbSwitchPort1
PACE ---> DumbSwitchPort2

With that config, the PACE gateway would always show green lights and the service was up. However, unplugging the PACE gateway and plugging in OPNsense with the spoofed MAC did not pull an IP address.

Using the same setup, but with a statically assigned VLAN, the OPNsense WAN port then pulled an IP. Something about the VLAN tagging between the gateway and the ONT was causing the dumb switch to not push all the traffic to the OPNsense box. Manually setting a static VLAN (such as VLAN10, or VLAN5, it doesn't matter) pushes all of the VLAN0 traffic between the switch ports defined with the static VLAN and allowed the OPNsense WAN port to receive the traffic and pull an IP. After that it was off to the races. It worked very well. Also worth noting, I only had Internet service, I did not have any TV or phone service. So this may be entirely different if you have other services riding inline with the internet service.

This is just WACK!!@#

Reread your post above several times. Your comment about the dumb (unmanaged) switch got the gears rolling.

I have a very plain basic 5 port gigabit dlink switch.  This is nothing special, about as basic as it gets.. Power brick outputs to a mini usb connector basic. https://www.amazon.com/D-Link-5-Port-Gigabit-Switch-DGS-1005G/dp/B003X7TRWE

BEFORE

Wan dhcp log going through the RGW passthrough (ONT<>RGW<>utm/pfsense)

Code: [Select]

2018:05:01-14:19:44 utm dhclient: DHCPREQUEST on eth1 to 192.168.1.254 port 67
2018:05:01-14:19:47 utm dhclient: Killed old client process
2018:05:01-14:19:48 utm dhclient: DHCPREQUEST on eth1 to 255.255.255.255 port 67
2018:05:01-14:19:49 utm dhclient: DHCPREQUEST on eth1 to 255.255.255.255 port 67
2018:05:01-14:19:49 utm dhclient: DHCPACK from 192.168.1.254
2018:05:01-14:19:49 utm dhclient: bound to 107.A.B.C -- renewal in 298 seconds.

107.A.B.C is my public ip.

For sh!ts and giggles I tried your suggestion above.  Port numbers really don't matter but consider the following.
Plugged in as follows

Port 1 = ONT
Port 2 = wan port of RGW

Broadband light flashed for a bit then went solid.  Good sign.

My utm box already had its mac spoofed to the RGW and DHCP enabled for wan IP.  Plugged it into port 3 and quickly unplugged RGW cable from port 2.  Results below:

AFTER

Code: [Select]

2018:05:01-14:47:52 utm dhclient: Killed old client process
2018:05:01-14:47:53 utm dhclient: DHCPREQUEST on eth1 to 255.255.255.255 port 67
2018:05:01-14:48:15 utm dhclient: DHCPREQUEST on eth1 to 255.255.255.255 port 67
2018:05:01-14:48:20 utm dhclient: DHCPREQUEST on eth1 to 255.255.255.255 port 67
2018:05:01-14:48:26 utm dhclient: DHCPDISCOVER on eth1 to 255.255.255.255 port 67 interval 5
2018:05:01-14:48:26 utm dhclient: DHCPREQUEST on eth1 to 255.255.255.255 port 67
2018:05:01-14:48:26 utm dhclient: [b]DHCPOFFER from 99.137.x.y[/b]
2018:05:01-14:48:31 utm dhclient: DHCPREQUEST on eth1 to 255.255.255.255 port 67
2018:05:01-14:48:37 utm dhclient: DHCPDISCOVER on eth1 to 255.255.255.255 port 67 interval 8
2018:05:01-14:48:37 utm dhclient: DHCPREQUEST on eth1 to 255.255.255.255 port 67
2018:05:01-14:48:37 utm dhclient: [b]DHCPOFFER from 99.137.x.y[/b]
2018:05:01-14:48:43 utm dhclient: DHCPREQUEST on eth1 to 255.255.255.255 port 67
2018:05:01-14:48:55 utm dhclient: DHCPDISCOVER on eth1 to 255.255.255.255 port 67 interval 6
2018:05:01-14:48:55 utm dhclient: DHCPREQUEST on eth1 to 255.255.255.255 port 67
2018:05:01-14:48:55 utm dhclient: [b]DHCPOFFER from 99.137.x.y[/b]
2018:05:01-14:48:58 utm dhclient: DHCPREQUEST on eth1 to 255.255.255.255 port 67
2018:05:01-14:48:58 utm dhclient: [b]DHCPACK from 99.137.x.y[/b]
2018:05:01-14:48:58 utm dhclient: [b]bound to 107.A.B.C -- renewal in 580413 seconds.[/b]

99.137.x.y Is the next hop after 192.168.1.254 in any traceroute (through the RGW).

The whole process took about a minute to re-establish connectivity.  UTM has an option to disable/re-enable the interface.  I think it's the equivalent of ifconfig {interface} down/up command. 



The renewal interval is an odd number above.  I've performed a forced wan ip renew several times.  Each time the renewal in.... number is different.

Code: [Select]

renewal in 572429 seconds.
renewal in 568221 seconds.
renewal in 457117 seconds.
renewal in 505034 seconds.
[code]

This comes out to:
[code]
seconds days
572429 6.62
568221 6.57
457117 5.29
505034 5.84

So I guess we'll find out what happens in about a week. 

Still, this is waaaaaaaaaaaayyyyyyyyy too easy!  At this point the rgw remains unplugged and powered down.  If this continues to work like this after week 3 I will truly be impressed.  It eliminates the need for any additional hardware, scripting knowledge and other associated headaches.

Speed tests are about what they were before. Not many (any?) fiber customers in my neighborhood so full gigabit speeds are attainable most times.

As expected traceroutes no longer have the 192.168.1.254 hop.  Also, my electric bill will be a tad cheaper.  Per the UPS, consumption is down about 8 watts or about $8 a year

THANK YOU AGAIN!!!!

[GPz1100]

Full bypass has been achieved.  See this thread over at dslr for config details for UTM.  Similar method works with other platforms too (there's instructions in a different thread on doing it with pfsense which should translate well to opnsense).

https://www.dslreports.com/forum/r31927134-ATT-Fiber-Sophos-UTM-instead-of-gateway