Messages

196

18.7 Legacy Series / Re: 16.1 and 16.7 update sets have been removed

« on: September 18, 2018, 07:54:59 pm »

I would prefer upload instructions, preferably sftp, as I don't know where to publish it without causing the mirrors to resync it which would work against the general idea of deprecating the use of older versions.


Cheers,
Franco

You should at least issue a warning (similar to a new release issued), when old versions will be deleted. Not in ad-hoc manner, like randomly after 18.7.3 is out. Would be preferred to push this info right into the face of users when they are e.g. at the download page. Currently neither the https://opnsense.org/download/ nor the on-device opnsense webgui / firmware update says anything about when the actual release should expire.

Take this as kind recommendation, to make the product more mature in terms of support predictability.

197

Hardware and Performance / Re: PC Engines APU2 1Gbit traffic not achievable

« on: September 13, 2018, 01:15:38 pm »

Me again.

I did some further testing. No PPPoE involved (dont have access to the internet line at the moment, only performed pure IP<-->IP in my lab, opnsense is only in the transit, not running iperf itself).

Found the option in the menu, where I can literally turn off the firewall (disable all packet filtering), which also disables NAT, and turns opnsense into a plain routing box.

Results (iperf -P 1 == single flow):
1)->firewall disabled, NAT disabled: can easily transmit 890-930 Mbit from WAN-->LAN, and vice versa, CPU load is approx 1x core 65% INT , another 1x core 10-30% in INT, the rest is idle. Throughput is stable, very minimal variation.
2)->firewall enabled, NAT disabled: this time its peak at 740-760 Mbit from WAN-->LAN, and vice versa, CPU load 1x 100% INT + 1x 20% INT, rest is idle. Occasionally, I get these strange drops to around 560 Mbit or to around 630 Mbit.
3)->firewall enabled, NAT enabled: LAN -->WAN: approx 650-720 Mbit, WAN-->LAN: around 460 Mbit constantly (100%+20% INT)

Results for 2) and 3) are not really consistent, and greatly vary between iperf sessions. So does the CPU load characteristics (sometimes less INT load results a higher throughput, other times double the INT load results much lower throughput).

Providing iperf -P 4 gives also very variable results:
- sometimes 1,2 or even 3 sessions are 0Kbit/sec, while the 4th session achieves the maximum throughput that was measure with single flow (-P 1)
- other times 1 flow has double throughput than the other 3 (unbalanced)

198

Hardware and Performance / Re: PC Engines APU2 1Gbit traffic not achievable

« on: September 08, 2018, 04:56:14 pm »

Well, I did test the IPFIRE as well on APU2 (I used latest ipfire-2.21-core123).

I could only achieve the same 250-290 Mbit/sec for the same torrent, as yesterday with the opnsense. Because I was suspicious, I also tried to connect my laptop directly to my ISP (I set up the pppoe profile directly on my PC), and tried it without any middle-router: speed was the same 250-280 Mbit/sec this time. So I think there is a problem with my bloody ISP today (yesterday I managed to get 600 Mbit so there must be something going on today). There is no point continuing this testing until I can figure out what the hell is happening.

If anyone can share with me the simplest PPPOE simulator config, based on Freebsd or Linux, I am going to try that on a powerful PC connecting to my APU, and completely rule out the uncertain ISP from this equation for these tests (I would run the IPERF on the PPPOE-simulator PC itself, being the WAN-endpoint for IPERF). Torrent would be difficult to simulate in such topology, so have to revert to iperf first.

199

Hardware and Performance / Re: PC Engines APU2 1Gbit traffic not achievable

« on: September 07, 2018, 06:53:12 pm »

If I remember correctly you said this on the FreeBSD Net List regarding OPN and IPFire. I'll check next week.

Yes, you are right! Some weeks ago, I did run the ipfire distrib on the APU. But that was only in an isolated LAN, without access to pppoe or to the internet. So I could run my iperf benchmarks without breaking the production internet

Today, unfortunately I wasted a lot of time to make the opensense work on my production pppoe internet connection. Basically the Default gateway was not activated properly after the pppoe session came up, so any internet traffic failed with TTL expired error.

My existing config in opnsense I was using static ip for the WAN (remember, I used an isolated LAN earlier for iperf testing). Today I changed the WAN config from static IP to pppoe. But some previous static Lan def gw config was stuck, and wasnt deleted properly (actually dmesg log complained about 2 gateways failed to remove). I logged into console, and tried couple of times  to reset the interface assignment and re-do the ip addressing, then logged into GUI and switched from WAN static IP to pppoe. The CLI console does not allow me to perform advanced config, like pppoe setup, so I had to perform that from GUI.
But it was still broken. I got ppoe session up, and I recieved public IP from my ISP, but the default gw was still the IP of my oldconfig LAN IP.

That is when I decided to login to consoleagain, select Option 4) factory reset, and re-did the Initial setup wizard from scratch on the GUI. I selected WAN type:pppoe, and this way I succeeded. But it wasted half of my day.

https://github.com/opnsense/core/issues/2186
I found this bugreport about pppoe default gateway not updating after the pppoe session activates, but it looked like that bug was fixed in 18.1.9 or so. Seems i was hitting something similar, dont really know.

So basically I did not have time to switch the operating system, boot ipfire, and repeat the same tests under linux OS. Planning to do it in the next coming days.

200

Hardware and Performance / Re: PC Engines APU2 1Gbit traffic not achievable

« on: September 07, 2018, 06:11:21 pm »

You mean IPFire on the same hardware?

No, not ipfire. Sorry if I was unclear

I installed a competely different equipment ( Asus AC66U B1 router) just for comparison to see if that router can reach the wirespeed gigabit.

On the APU I could not test ipfire today due to not enough time, but maybe in the coming days I will do another round of tests using the ipfire.

Need to find a timeslot when no users are using the internet

201

Hardware and Performance / Re: PC Engines APU2 1Gbit traffic not achievable

« on: September 07, 2018, 05:56:51 pm »

Yet another small addendum:

finally I managed to test throughput over pppoe, under real life conditions.

Results are quite weak:
approx. 250-270 Mbit/sec (WAN-->LAN traffic direction) was achieved with the APU2. Not iperf this time, but tested with some torrent (so nobody can tell that I was pushing for unrealistic expectations over 1 single flow).
Again, the router was only a transit device, the torrent client was running on a PC behind the APU. SSD wasnt the bottleneck during download.

As a comparison, using a different vendor router, I was able to achieve 580-600 Mbit/sec easily downloading the same test torrent. Didnt investigate if it could go higher or not with this different vendor router, but thats still more than double performance difference.

202

Hardware and Performance / Re: PC Engines APU2 1Gbit traffic not achievable

« on: September 07, 2018, 12:59:54 pm »

No, never!

The 2 iperf endpoints are running on a PC connected to LAN (igb1) and another PC connected to WAN (igb0), the APU is always just a transit device (packet forwarding / packet filtering  / NAT translation between igb1 and igb0 and vice versa), never terminating any iperf traffic directly on it.

203

Hardware and Performance / Re: PC Engines APU2 1Gbit traffic not achievable

« on: September 07, 2018, 12:44:24 pm »

https://calomel.org/freebsd_network_tuning.html

# Disable Hyper Threading (HT), also known as Intel's proprietary simultaneous
# multithreading (SMT) because implementations typically share TLBs and L1
# caches between threads which is a security concern. SMT is likely to slow
# down workloads not specifically optimized for SMT if you have a CPU with more
# than two(2) real CPU cores. Secondly, multi-queue network cards are as much
# as 20% slower when network queues are bound to real CPU cores and well as SMT
# virtual cores due to interrupt processing inefficiencies.
machdep.hyperthreading_allowed="0"  # (default 1, allow Hyper Threading (HT))

# Intel igb(4): The Intel i350-T2 dual port NIC supports up to eight(
# input/output queues per network port, the card has two(2) network ports.
#
# Multiple transmit and receive queues in network hardware allow network
# traffic streams to be distributed into queues. Queues can be mapped by the
# FreeBSD network card driver to specific processor cores leading to reduced
# CPU cache misses. Queues also distribute the workload over multiple CPU
# cores, process network traffic in parallel and prevent network traffic or
# interrupt processing from overwhelming a single CPU core.
#
# http://www.intel.com/content/dam/doc/white-paper/improving-network-performance-in-multi-core-systems-paper.pdf
#
# For a firewall under heavy CPU load we recommend setting the number of
# network queues equal to the total number of real CPU cores in the machine
# divided by the number of active network ports. For example, a firewall with
# four(4) real CPU cores and an i350-T2 dual port NIC should use two(2) queues
# per network port (hw.igb.num_queues=2). This equals a total of four(4)
# network queues over two(2) network ports which map to to four(4) real CPU
# cores. A FreeBSD server with four(4) real CPU cores and a single network port
# should use four(4) network queues (hw.igb.num_queues=4). Or, set
# hw.igb.num_queues to zero(0) to allow the FreeBSD driver to automatically set
# the number of network queues to the number of CPU cores. It is not recommend
# to allow more network queues than real CPU cores per network port.
#
# Query total interrupts per queue with "vmstat -i" and use "top -CHIPS" to
# watch CPU usage per igb0:que. Multiple network queues will trigger more total
# interrupts compared to a single network queue, but the processing of each of
# those queues will be spread over multiple CPU cores allowing the system to
# handle increased network traffic loads.
hw.igb.num_queues="2"  # (default 0 , queues equal the number of CPU real cores)

# Intel igb(4): FreeBSD puts an upper limit on the the number of received
# packets a network card can process to 100 packets per interrupt cycle. This
# limit is in place because of inefficiencies in IRQ sharing when the network
# card is using the same IRQ as another device. When the Intel network card is
# assigned a unique IRQ (dmesg) and MSI-X is enabled through the driver
# (hw.igb.enable_msix=1) then interrupt scheduling is significantly more
# efficient and the NIC can be allowed to process packets as fast as they are
# received. A value of "-1" means unlimited packet processing and sets the same
# value to dev.igb.0.rx_processing_limit and dev.igb.1.rx_processing_limit . A
# process limit of "-1" is around one(1%) percent faster than "100" on a
# saturated network connection.
hw.igb.rx_process_limit="-1"  # (default 100 packets to process concurrently)

I have also went through this. No measurable improvement in throughput.

machdep.hyperthreading_allowed="0"  # (default 1, allow Hyper Threading (HT)) --> NOT APPLICABLE to my case. This AMD CPU has 4 physical cores, and  sysctl hw.ncpu --> 4, so HT (even if supported, I am not sure) is not active currently.

hw.igb.num_queues="2"  # (default 0 , queues equal the number of CPU real cores)
--> I have 4 cores, 2 active NIC, each NIC supports up to 4 queues. I used by default
hw.igb.num_queues="0", but tried it with hw.igb.num_queues="2" as well.
No improvement in throughput (for single-flow).
But! It seems degraded the multi-flow performance heavily.

hw.igb.enable_msix=1 was like that since the beginning
hw.igb.rx_process_limit="-1"  --> was set, but no real improvement in throughput
dev.igb.0.rx_processing_limit and dev.igb.1.rx_processing_limit is both set to "-1" as per previous entry did

I am very sad that this wont be solveable under Opnsense without switching to competitors or switching the hardware itself.

Some small addendum:
recently I noticed (maybe when upgraded to 18.7.1_3, but TBH not sure), that sometimes (depends on the actual throughput / interrupt load shared among cores) the serial-console hangs during iperf. As soon as the iperf session is finished or I interrupt the session manually, serial-console becomes live again. Noticed during running "top" on console: I noticed refresh stopped / frozen during the iperf session, keyboard wasnt working either while the iperf traffic happened. As soon the iperf session finished, "top" continued to produce output / console responds to keystrokes.

Seems has to do something with the fact when throughput is alternating between those 2-3 discrete levels randomly among iperf sessions.

204

Hardware and Performance / Re: PC Engines APU2 1Gbit traffic not achievable

« on: September 04, 2018, 04:02:28 pm »

I ordered this via company, no tax, so only 160EUR

https://www.amazon.de/PC-Engines-APU-2C4-Netzteil-schwarzes/dp/B01GEIEI7M

I really meant to support this evaluation effort. So if there is still something needed, let us know!

205

Hardware and Performance / Re: PC Engines APU2 1Gbit traffic not achievable

« on: September 04, 2018, 02:33:13 pm »

local = German? I can ask my boss is the company is willing to test such a device ..

I am not from Germany, I live in eastern Europe, just converted my local currency to EUR for an approximate estimation. But your local PC shop may sell these devices even cheaper:

http://pcengines.ch/order.htm

206

Hardware and Performance / Re: PC Engines APU2 1Gbit traffic not achievable

« on: September 04, 2018, 01:18:04 pm »

Looks like I'm the only one willing to chip in?

@KantFreeze:
Lets be reasonable. Nobody will send equipment as a compliment to unknown people on the internet. At least that is my view.

@mimugmail: how about a donation towards you, so you can buy a brand new APU2 for yourself, and you could spend some valuable time to see its max. performance capabilities, and document your findings? No need to return the device at the end, you should keep it for future Opnsense release benchmarks / regression tests.

I bought my APU2 from a local reseller (motherboard + black case + external PSU + a 16Gb mSATA SSD), sum was approx. 200 EUR. If there are 10 real volunteers, I am willing to spend 20 EUR (non-refundable) "donation" on this project.

DM me for the details, if you are interested.

207

Hardware and Performance / Re: Question About Hardware Compatibility - UPDATE 8/14/2018

« on: August 16, 2018, 12:57:33 pm »

The positive suggestions by other(s) were based on personal sympathy of the vendor / board itself, not based on the real technical requirements by the OP.

You think you have present two facts to prove your point, but the first one is an opinion and the second one is a lie by your own admission that the technical requirements were not stated in the first place. You're not making sense and continue your narrative against the best practice of friendly and productive tone in this community.

You're wasting your time and the only thing it gets you is that I'm reminding you of upholding etiquette and suggest to step away from this thread now.


Cheers,
Franco

If my tone is not friendly (no swearing, not being impolite with anyone)  or my warning (it was a warning, nothing more!) is not productive, there are big issues on this forum. Anyway, you achieved your goal against real-life critic: I will not post into this thread anymore.

Btw. the fact you dodged my real question about performance of DECISO products (that have similar specs to PCengines APU2) is also an alarming sign to me.

Have a great day!

208

Hardware and Performance / Re: Question About Hardware Compatibility - UPDATE 8/14/2018

« on: August 16, 2018, 10:09:48 am »

@ricsip
You are right!
But TWB asked explicit for a sbc and for the money the APU2-Boards offers great performance and opportunities (i.e. m-pci slots). So i recommend this board. The iperf may be a problem for you. Others may habe no problem with this!

best regards
Dirk

"The iperf may be a problem for you"
Its not a problem about iperf. Iperf is simply the tool to confirm there is performance problems with the HW+SW combo under conditions that anybody can possibly face, if signing a contract for 1Gbit internet.

209

Hardware and Performance / Re: Question About Hardware Compatibility - UPDATE 8/14/2018

« on: August 16, 2018, 10:07:58 am »

Let me ask something. If someone recommends the APU2 to OP, what reason is behind that recommendation? The OP didnt clarify the bandwidth requirements. It does matter if the APU2 will be serving a 10 Mbit ADSL or a 1Gbit Fiber WAN with PPPoE. It does matter if there will be 2 lines in the firewall ruleset, or 1000. It does matter if IPS/IDS will be activated with many rules, or no IPS/IDS at all.

You are right. That being said, what's the reasoning behind recommending to avoid it without knowing the same? Do you plan to jump in every time someone asks a basic hardware question and recommend avoiding APU devices? Find the device that makes you happy and stick with it.

Beware PCengines APU2, if you want to use it on 1-Gigabit links, where it fails miserably to reach anything close to wire-speed, compared to some Linux-based firewall distrib.
--> Thats what I said. It clearly defines the bottleneck, that the APU2 cannot exceed under opnsense. But it can easily achieve under IPfire.

The positive suggestions by other(s) were based on personal sympathy of the vendor / board itself, not based on the real technical requirements by the OP.

Is that so hard to acknowledge, there is no reliable benchmark on the opnsense wiki, so that one can make sizing based on?
For example, if I were to buy, would you recommend your in-house Deciso DEC600
https://www.deciso.com/product-catalog/dec600/
for the same workload I try to achieve?
Or DEC620? Or DEC2600 ? I really don't know.

210

Hardware and Performance / Re: Question About Hardware Compatibility - UPDATE 8/14/2018

« on: August 15, 2018, 04:35:20 pm »

A yes or no would have been clearer. FWIW, you seem emotionally attached to the fact that this hardware doesn't live up to your expectations. Please don't let your sentiment lessen your appearance while helping others.

Cheers,
Franco

Yes, indeed, I am usually passionate about things I do.

Let me ask something. If someone recommends the APU2 to OP, what reason is behind that recommendation? The OP didnt clarify the bandwidth requirements. It does matter if the APU2 will be serving a 10 Mbit ADSL or a 1Gbit Fiber WAN with PPPoE. It does matter if there will be 2 lines in the firewall ruleset, or 1000. It does matter if IPS/IDS will be activated with many rules, or no IPS/IDS at all.

There are no publically available benchmarks, that could say, OPnsense 18.7 + APU2 is good up to X mbit traffic (with NAT / without NAT?) if adding the following layer, one after each other:
a) there are X inbound rules in pf
b) there are Y rules in Suricata
c) x sessions in openVPN using y crypto algorithm serving z mbit bandwidth
d) total aggregated CPU load is under 400%

So its neither sufficient to say in generic, that APU2 is what OP is looking for.