Messages

I think the issue here is probably smartctl not reporting the value title in the same way as the manufacturer. I was curious enough about this that I quickly pulled the drive and ran the manufacturer's diag tool on it. In my case, this is a Kingston SSD.

Smartctl reports the value as 'SSD_Life_left' whereas Kingston actually lists it as "SSD Wear Indicator" and shows the wear at 12% with a remaining estimated life of 88%.

The swapped ID titles in smartctl don't make this any easier to decipher however, it looks like the drive has a long life ahead of it (fingers crossed ;) ).

At 88% life remaining, I'm using roughly 5% of the SSD life every year. At this rate I'd have another 16 years remaining.

Actually you have 12% remaining lol.

:o Are you sure about that? I've watch it slowly tick down from the high 90s to where it's currently at now, in the high 80s after 2+ years.

I agree with the consensus that the SSD lifespan is not a concern for most firewall use cases. Here are the stats on my cheapo 120GB SATA SSD that has been running OPNsense non-stop for 2.3 years.

Code Select Expand

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x0032   000   100   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       20426
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       161
148 Unknown_Attribute       0x0000   100   100   000    Old_age   Offline      -       0
149 Unknown_Attribute       0x0000   100   100   000    Old_age   Offline      -       0
167 Write_Protect_Mode      0x0000   100   100   000    Old_age   Offline      -       0
168 SATA_Phy_Error_Count    0x0012   100   100   000    Old_age   Always       -       0
169 Bad_Block_Rate          0x0000   100   100   000    Old_age   Offline      -       5
170 Bad_Blk_Ct_Erl/Lat      0x0000   100   100   010    Old_age   Offline      -       0/13
172 Erase_Fail_Count        0x0032   100   100   000    Old_age   Always       -       0
173 MaxAvgErase_Ct          0x0000   100   100   000    Old_age   Offline      -       149 (Average 118)
181 Program_Fail_Count      0x0032   100   100   000    Old_age   Always       -       0
182 Erase_Fail_Count        0x0000   100   100   000    Old_age   Offline      -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
192 Unsafe_Shutdown_Count   0x0012   100   100   000    Old_age   Always       -       59
194 Temperature_Celsius     0x0022   073   069   000    Old_age   Always       -       27 (Min/Max 22/31)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
199 SATA_CRC_Error_Count    0x0032   100   100   000    Old_age   Always       -       0
218 CRC_Error_Count         0x0032   100   100   000    Old_age   Always       -       1
231 SSD_Life_Left           0x0000   012   012   000    Old_age   Offline      -       88
233 Flash_Writes_GiB        0x0032   100   100   000    Old_age   Always       -       7601
241 Lifetime_Writes_GiB     0x0032   100   100   000    Old_age   Always       -       12530
242 Lifetime_Reads_GiB      0x0032   100   100   000    Old_age   Always       -       122
244 Average_Erase_Count     0x0000   100   100   000    Old_age   Offline      -       118
245 Max_Erase_Count         0x0000   100   100   000    Old_age   Offline      -       149
246 Total_Erase_Count       0x0000   100   100   000    Old_age   Offline      -       1301864

At 88% life remaining, I'm using roughly 5% of the SSD life every year. At this rate I'd have another 16 years remaining. And this is on a very cheap Kingston 120GB SATA SSD. A higher capacity and higher end SSD would be able to balance writes more effectively and would likely have an even greater lifespan for this use case. Plus, the SSD is faster, silent, and uses less power than a traditional spinning disk.

To go from 17/20 to 19/20 on ipv6-test.com I had to do the following.

First make a rule on WAN to allow ICMPv6 Echo Requests. Screenshot provided.

Then I had to edit the windows firewall and remove the Local Subnet from the scope. By default when allowing ICMP on the windows firewall, it limits the scope to only computers on the local subnet. Removing this allows an external system to get a ping response in combination with the firewall rule that we added to WAN.

After those two changes I now score 19/20. The only thing missing for me on the ipv6-test website is the hostname.

With regard to plaintext resolver queries, I don't think that's a good or bad thing. Just something you'll need to decide. Plenty of people use plaintext queries with DNSSEC and it just works. I included that detail because you seem concerned with having to put too much technical effort or understanding in to any of this. And that (default) solution out of the box fits most of your requests perfectly a)easily implemented/default b)checkbox for additional security (DNSSEC) c)doesn't rely on the ISPs DNS.

If you do nothing else, at least read the snbforums thread comparing the two DNS providers and you'll get a much better understanding for how Quad9 and CloudFlare DNS operate as a business. For me personally, that thread made the decision of choosing a DoT provider much easier. Also the first link I posted (ctrl.blog) has the reasoning for why we'd want to use the CN plus the IP address and port for DoT.

Thank you for explaining all that.  I took a look at the Unbound configuration and saw what you are talking about, the only thing I don't really understand is that there is a field for "Verify CN" (the help tip is "Verify if CN in certificate matches this value") but I am not sure what you are supposed to put there.  If you look at the list of servers in the top post there is nothing indicated as a "CN"; I do see that most have "tls_pubkey_pinset" entries but I assume that's not the same thing.

With all due respect and I don't mean this negatively but this shows you haven't look in to how DoT works. You'll either need a pinset or ideally, avoid using a pinset and instead have the TLS connection match with the dns name issued to the certificate so that the resolver can verify that the queries are actually coming from an intended source (dns.quad9.net, one.one.one.one, etc. etc.). This is no doubt more complicated than "regular" DNS, but then again so is an Unbound resolver (which OPNsense ships with by default) compared to a dnsmasq based forwarder configuration in most consumer gear.

The other thing is there are no default or suggested entries; .....  Or failing that, a link to an article somewhere that contains that information.

Stubby ships with a default config. The DoT providers that you wish to use will need to be your own decision. It's quite normal to not ship a list of defaults because people may not want them or even worse, they could end up being monetized and not in a user's interest for privacy. Again the rule here is if you're going to enable DoT, you'll need to do a small amount of searching to understand it and make the choice which provider you think has your best interests in mind. I wouldn't expect OPNsense devs, or anyone else for that matter, to do that for me.

Again not trying to sound like a "let me google that for you" post but this is all covered extensively here in the forums over many years (DoT was first launched with an Unbound update back in April 2018).

Basic info here and some config examples (note, the config isn't needed because new OPNsense does that for you with the DoT GUI page): https://www.ctrl.blog/entry/unbound-tls-forwarding.html

A discussion about the two major DoT providers, CloudFlare and Quad9, and additional input from Quad9's management. CloudFlare also posts in that thread with their input.: https://www.snbforums.com/threads/cloud9-dns.56918/

Maybe DNS over TLS is too new, or there really are that few servers that support it, if so I apologize if I am asking for too much, but I am just trying to understand if this is something that is possible. As for the comment about the article being for those "that wish to try something a bit more performant than Unbound DoT", I guess I would say that while I suppose everyone would like maximum performance (however they define that), not everyone has the time, patience, or ability to learn how to tweak things to the very max (that's as true of computer networking as it is of mechanical things such as car engines).  I would just be happy if it works reliably and doesn't become a bottleneck to network traffic.

If you just want working and reliable DoT with minimal configuration, the built in DoT GUI page in OPNsense is the way to go. Choose a DoT provider, input their port, IP, DNS name and you're up and running.

You also need to be aware of the differences between a resolver and a forwarder. Anytime you go DoT, you're always putting all of your trust in the provider you choose to forward your queries to. You're essentially abdicating the resolver portion of Unbound and just having it forward all of its queries to a chosen provider. Yes, they will be encrypted so that your ISP can't see them, but the provider can still decrypt and see them. So you're exchanging anonymity at the DNS level for anonymity between your ISP and your DNS traffic.

OPNsense ships by default in resolver mode, which means all of your DNS queries are sent in plaintext so that your ISP (or anyone else) in the middle can potentially see them, but the queries are sent randomly to hundreds of various root servers and the local Unbound service within OPNsense resolves them and caches them. OPNsense also has two easy check boxes during the initial setup to enable DNSSEC in resolver mode, which adds an additional layer of security from the root resolvers (there's those nice check boxes again  ;) ) The likelihood that a single DNS provider would be able to get enough metadata on your network activity would be reduced, at the expense of everyone else being able to potentially see it on the wire. It's always a trade off. You need to educate yourself and decide. And yes, most people just give up and use consumer gear or just use the defaults at this stage in the game.

Your post seems a bit off topic given that with the 21.7 series, the developers actually added a separate DoT section in Unbound. Input the IP, DNS name, and port and that's it. It doesn't get much easier than that. Both OPNsense and pfSense do a good job at mainstreaming DoT and not having a huge technical barrier in place.

The more complex configuration you see in this thread would be for users that wish to try something a bit more performant than Unbound DoT. Thus we have to setup a separate resolver for DoT and forward internal Unbound queries to that resolver (Stubby/getdns).

The reason this is complicated is that there really isn't a single solution for all of the examples that you give. If you want IPS/IDS, there's no way to know for sure that your traffic will be the same as another use case. That's why a) its complicated to configure and b) you need to know your traffic and device use cases. These cases rarely fit in to the "I just want a checkbox and done" crowd.

Question regarding this configuration, will this survive OPNsense release upgrades and will the stubby package automatically be upgraded if a newer version is available?

FreeBSD 12 is still the limiting factor with VMX3 NICs. However, you can get better throughput by adding cores and making sure that the VM hardware given to the OPNsense router is on the VMware compatibility list.

With 2 cores, supported VM hardware, and openvmtools running on OPNsense I can see around 2gbit/sec throughput when testing internal transfers through the vSwitch. This is mainly limited by the CPU speed of the host, throwing more cores and/or faster CPU clock at the VM gets even higher throughput.

Since you have provided no details on the hosting environment, it's hard to say what else you should try. At the very least separate the iPerf client/server to different VMs, don't host one of the iPerf instances directly on OPNsense. Push/pull the traffic through OPNsense using a client/server VM setup sitting on each side of the LAN/WAN for the OPNsense VM.

If you have enough NICs available in the host and don't need vMotion for the OPNsense VM, try using NIC pass-through, that should yield much better results.

I'd take ZFS without ECC any day over UFS without ECC, all else being equal. At least with ZFS, you get some power failure tolerance that UFS doesn't provide.

I'm having trouble understanding your setup from the description.

But it seems like you're having two issues.

Issue 1) wireless clients on the AP are not getting good speeds. You'll need to check if they are negotiating on 2.4ghz or 5ghz band. OPNsense is unlikely to have any impact on these clients and their wifi connection to the netgear router. If you plug the netgear router directly in to the modem, do wifi clients have any change in speed?

Issue 2) you only get 800mbps when using OPNsense and a wired connection to your modem. If you plug directly in to the modem, you get closer to the advertised 980mbps speed. You can try the following system tunables and see if these help, you may need a reboot after you save these in the tunables before they take effect.
Disable Flow Control

Code Select Expand

dev.igb.0.fc 0
dev.igb.1.fc 0

Disable EEE (energy efficient ethernet)

Code Select Expand

dev.igb.0.eee_control 0
dev.igb.1.eee_control 0

Need more info here. What kind of CPU and how fast? What ethernet cards are in use? How fast in the connection being traffic shaped? If you turn off the traffic shaper and re-test bandwidth usage, does the CPU still show 100%?

I don't have these specific cards so I can't give any direct experience with them. I did some searching around though.

It looks like the pfSense DEVs added the bnxt driver starting with 2.5.

There's another thread here where an OPNsense user claimed they resolved it with this:

Code Select Expand

kldload if_bnxt

Link to that thread: https://forum.opnsense.org/index.php?topic=21447.0

I don't personally use the PPPoE function however, I have read that it is single threaded. Is one of the threads on the OPNsense VM being CPU bound during the speed tests?

Still, with that hardware and virtualization, 900mbps isn't too bad.

And you can not tell me, that there is no problem with the firewall, when the performance degrades 50 times by just moving a firewall rule from the floating rules to the interface rules.

There's obviously an issue that you have found. But we don't know what causes it and you seem unwilling to try a very simple method to rule out a potential variable (the firewall rule sorting) by just spinning up a client/server VM and pushing traffic that way.

Your lack of information about your environment also means most of us are shooting in the dark trying to help you. What is your ESXi version? Are you running openvmtools on all of the firewall appliances? Which NICs have you tried (vmx3, e1000)? What VM hardware version are you running for the OPNsense appliance?

You also haven't given us information on the networks. Are we talking about purely virtual routing where OPNsense is pushing traffic from all of your VLANs to various vSwitches or vDS managed port groups? Or is OPNsense pushing traffic out of the VM back on to a physical layer? That can be a huge variable too.

If you are pushing traffic back out to a physical layer, is a port over subscribed or used in another vSwitch that is causing the bandwidth variables?