Messages

Huh! Yeah, I'd expect the em driver... but then igb and em seem oddly interchangeable.

ifconfig may also show the devices. My concern was this line in their lame excuse for a data sheet: "3 x 1G bypass bridge pair", which usually indicates the presence of a hardware bypass, which must be deactivated (via custom software interface) to use the interfaces... somewhat normally. So the drivers may load, the interfaces may be configurable, but they will be electrically isolated by the bypass. But that's a supposition - I didn't find better docs offhand.

Perlman's (initial) work pre-dated or was contemporaneous to the development of the OSI model. Heck, for all I know she was in on it, as she contributed to DECNET and ISO standards such as CLNS/CLNP and IS-IS. (I believe she was also one of the DIX folks who transitioned to the IEEE 802 group. Seifert always thought Ethernet would dominate networking; back then I don't think anybody would have picked IP to dominate.)

Anyway, the OSI model is just a model. I can make a device that forwards using any information within the packet... or without. Conventions aid communication, though... most of the time.

[...]I believe in OPNsense we can do 2+ ports for a single bridge.[...]

I have... 6 physical interfaces on my (current) firewall, with another 9 via a 10-port switch (acting as a port expander, where ports 1-9 are assigned unique port VLANs and 10 is a tagged uplink plugged into the firewall). So 14 available interfaces, each assigned to one of four (non-transparent) bridges. Works great. Most folks here think it would give their network the plague. (Who knows? They might be right.)

[...]It's pure hobby, but it does have to be functional. So please just tell me to drop the stupid project  if that you pros opinions[...]

Nah, try it out. You can have a good old time trying different stuff. Never can tell. Could fix your back in short order.

I know what she meant, but in technical terms she stated it incorrectly.[...]

Heh. Heck, grab her book - it's not bad. Or Rich Seifert's.

[...]I dont think we can bridge two different vlans, has to be the same vlan. This means all host IPs are local, the bridge needs to learn MACs and then proxy-arp what it knows about, so that L2 can function normally.

Not sure what you're describing there. FreeBSD bridging seems to be pretty straightforward. ARP is... well, bridged. No proxy, unless you configure one (and FreeBSD default proxy ARP is cheeesy).

To the original poster: Bridging works for me. My Internet link is bridged, and managing several internal networks on the firewall (I have a few more than two interfaces, and everything runs through the firewall) is quite easy with bridges. The only downside I've seen is FreeBSD's susceptibility to ARP proxies (my Internet ONT is one, so I have static ARP entries for everything on that bridge except for the ONT). (A "layer"-agnostic filter would be nice, but hey.) Not much more to it. It may work for you, or not.

Yep, looks like a bad stick. You could test it with memtest86 just to verify, but I doubt it would give a different result.

The only Nemix DIMMs I've seen used re-marked chips - not for me. YMMV. Heck, it might even have a warranty... but the manufacturer has an incentive to reject claims at the moment.

[...]I also see a SMART thermal message but that seems like more of a 'notice'.[...]

What temperature? The controller shouldn't exceed its limits, but higher temperatures are generally detrimental. Might be worth addressing while you're in it.

not a single error logged, see attached file[...]

Ha! Much cleaner than the ones I have handy to check (AMD CPUs, with most stuff at v2-4).

Well, good luck with it.

[...]some other idea? honestly unless somebody is coming with something i am starting to think like a bsd problem, because i can not really udnerstand what is going on...

I wouldn't trust stress-ng to uncover issues, but that's up to you. mprime doesn't stress non-hyperthreading Coffee Lakes much, for instance.

How about the PCI query?

It could be a software issue. My experience is limited to a couple FreeBSD and a couple OPNsense machines, all bare-metal. Recent/relevant experience, that is. Most of the pf messages could point to really wacky network issues, but (in particular) I wouldn't expect a duplicate flow ID to be a soft issue (within the scope of a standard OPNsense install). I just don't have enough experience poking into pf to say with certainty.

Yak! I haven't debugged pf (no need thus far), but I would not expect such output on a functioning system. Unless someone else has a better idea, I'd run a thorough set of tests on the computer. "pciconf -lcev" (correctable errors seem to be normal), memtest86+, mprime (I run mprime from a live Linux image, as the executable is handy), etc.

Oh, from the firewall itself, pf activity will (normally) be "let anything out..." - very simple state creation/matching. The CLI test likely doesn't spew sessions like a web app, but that's an assumption.

[...]the firewall will not determine which circuit the inbound packets come from[...]

For inbound traffic I would differentiate by interface and destination address; destination alone should be fine. I haven't set it up myself (I avoid hinky routing), but I wouldn't expect it to be a problem (could be a bad assumption). But what advantage would an additional L3 device have?

(Heh: My Internet link died right in the middle of posting this. Trying to sell me on a backup?)

[...]You need a layer 3 switch to run PBR to handle this.

? The firewall can handle this under most circumstances.

PPPoE is apparently single-thread on FreeBSD... but that should only halve your performance (or thereabouts) on that dual-core CPU. The CPU load figure may be misleading, but even proving that premise may not be enlightening. You could try OPNsense on a machine with greater single-thread performance, but a 3.4GHz Skabylake shouldn't be too bad. A Zen 5 (e.g. 9600X at 5.4GHz) might double it (on this workload). I haven't used PPPoE (knocking on wood...) myself (even my old DSLs used bridging or routing for static IPs, which I've always had; my only PPP link was ISDN, and 128k was pretty easy to achieve).

Edit: For the heck of it (quoting myself), you might try "netstat" - "-m", "-i", perhaps "-Q", "-T", "-x", "-s" options (most have to be issued separately), and see if anything looks bad. I'm not sure if these will provide useful data for a PPPoE device.

[...]honestly, my FW ruleset is quite small (<200 rules)[...]

On how many interfaces? I have four (bridges), and ~100 rules. But then I have static IPs, so I have 2-4 NAT rules.

The number is less important than the rules themselves, of course. You could post those, or at least the ones relevant to your speed test.

I dont know where or how to look at this point[...]

I'd watch the live log (rule logging must be enabled) and make sure the ruleset is working as expected. (I'm lazy, and also look at the "Firewall States" dashboard widget for a total, as well as the "Sessions" and "States" GUI diags.) With so many rules I would not expect a functional loop. Also, "netstat" - "-m", "-i", perhaps "-Q", "-T", "-x", "-s" options (most have to be issued separately), and see if anything looks bad.

[...]I dont think it has something to do with the NIC, remember that when i disable the FW the load of the system under the same test is < 18%....[...]

I could be barking up the wrong tree, but I wouldn't expect the pf codepath to stall so badly where the default does not.

[...]Could this test suggests that it has something to do with the NAT or perhaps with any WAN rule? (can we discard pf general performance issue?)

Probably. How about rule behavior,e.g. number of sessions spawned by the test? I can think of ways to overload pf (e.g. "pass" rules in a deliberately loopy network), but these would normally affect functionality as well as speed.

Huh. Stalling on interrupts (95%). Are you using Realtek Ethernets with the factory (not the plugin) driver? If so, try the plugin. If not, what Ethernet interfaces do you have?

An interrupt handler issue? I haven't seen one myself. Do you have any unusual sysctls (tuneables) configured?

Can you paste a "top" capture?