Messages

I am seeing what appears to be 6 times the number of state lookups compared to my PPS.

If I look at I see an average of 600-1000 PPS over the last hour.

In `pfctl -s info` I see between 5000-7000 state lookups a second.

State Table                          Total             Rate
  current entries                      357
  searches                        21275522         5310.9/s
  inserts                            18675            4.7/s
  removals                           18318            4.6/s
Counters
  match                              20103            5.0/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                                  0            0.0/s
  normalize                              0            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                             0            0.0/s
  ip-option                              0            0.0/s
  proto-cksum                            0            0.0/s
  state-mismatch                         0            0.0/s
  state-insert                           0            0.0/s
  state-limit                            0            0.0/s
  src-limit                              0            0.0/s
  synproxy                               0            0.0/s
  map-failed                             0            0.0/s


I did tcpdump on my interfaces and I am not seeing any abnormal about of traffic (broadcasts, multicast, etc..).  This appears to mainly be my Nest camera's sending video to mother google.

I expected the PPS to be close to the state lookups numbers.  Am I missing something or is this normal?

Yes, I can not let it go...I have still been tinkering with this when time permits.

I found something interesting that maybe an issue with comcast.  Out-of-Order packets!

From netstat -s

43819 out-of-order packets (63444003 bytes)
18910 discarded due to memory problems

These counters only increment when I do a speedtest.net.

Setting this net.inet.tcp.reass.maxqueuelen="1000"
The default is 100

I was able to prevent out-of-oder packets from being dropped.  As I understand it this limit was put in due to a security vulnerability involving packet reassembly.  It appears Comcast has an issue with out-of-order packets so I had to give it a little more room to work.

Unfortunately it did not help my speed (likely hitting a different limit).

I suspect this would not have turned up in an iperf as packets are likely not coming in out-of-order.

Unfortunately no, I have Comcast and as I understand it, as long as my DUID does not change, I should keep that same prefix, but if I change the interface to static using the prefix::1, I will no longer be sending the DHCP request to get the prefix and when DHCP times out, it would be released and assigned to another customer.  To get s static prefix I would have to pay 3x more for business service.

I see a setting "Allow manual adjustment of DHCPv6 and Router Advertisements", I guess I will pay with that and see what not does :).

I recently got IPv6 working.  I am using Track Interface on my LAN interface as I do not have static IP's.  I am looking at setting up squid to do things like block youtube during homework time.  I also need to do things like disable internet access at night to prevent them from sneaking and playing games etc...

I can do these things with IPv4, but I want to play with IPv6 to learn more.  Like I want to figure out how to force my IoT devices to use my secondary ISP connection (Multi-WAN), but with IPv6 and the lack of NAT, its unclear how I would do this without preventing OPNsense from responding to router advertisements coming from these devices.  This is a topic for another day :).

I was hoping to be able to write Firewall rules based on hostname, but as the SLAAC hosts are not getting addresses via DHCP, they are not in the DNS zone file.  With devices getting multiple IPv6 addresses that are not static, its unclear to me how to write a firewall rule.

Any tricks for say using the MAC address and querying the NDP table?

[#!/bin/tcsh/home/compunction/speedtest-cli.py --server 11143 > test.1 &/home/compunction/speedtest-cli.py --server 1767 > test.2 &/home/compunction/speedtest-cli.py --server 10391 > test.3 &/home/compunction/speedtest-cli.py --server 8169 > test.4 &/home/compunction/speedtest-cli.py --server 13653 > test.5 &/home/compunction/speedtest-cli.py --server 13655 > test.6 &/home/compunction/speedtest-cli.py --server 10575 > test.7 &/home/compunction/speedtest-cli.py --server 3595 > test.8 &/home/compunction/speedtest-cli.py --server 5296 > test.9 &/home/compunction/speedtest-cli.py --server 8707 > test.10 &/home/compunction/speedtest-cli.py --server 12407 > test.11 &/home/compunction/speedtest-cli.py --server 15853 > test.12 &sleep 30/bin/cat /home/compunction/test.* | /usr/bin/grep Download | /usr/bin/awk '{print $2}' > test.output/bin/cat /home/compunction/test.outputecho "Total: "/bin/cat /home/compunction/test.output | /usr/bin/awk '{sum+=$1} END{print sum}'/bin/rm /home/compunction/test.*]

Interesting, so single TCP session vs multiple TCP sessions.

Did a quick and dirty test form my OPNsense box just to test the WAN.  Please do not make fun of my scripting, I could have done a for loop and done it 100 different ways, but I am not a dev :)

The output doing a speedtest to 12 different servers are the same time yielded 932.92Mbps.  I would say that is close enough and does prove that the bandwidth is there, just not for a single tcp session.

----------SCRIPT----------
#!/bin/tcsh
/home/compunction/speedtest-cli.py --server 11143 > test.1 &
/home/compunction/speedtest-cli.py --server 1767 > test.2 &
/home/compunction/speedtest-cli.py --server 10391 > test.3 &
/home/compunction/speedtest-cli.py --server 8169 > test.4 &
/home/compunction/speedtest-cli.py --server 13653 > test.5 &
/home/compunction/speedtest-cli.py --server 13655 > test.6 &
/home/compunction/speedtest-cli.py --server 10575 > test.7 &
/home/compunction/speedtest-cli.py --server 3595 > test.8 &
/home/compunction/speedtest-cli.py --server 5296 > test.9 &
/home/compunction/speedtest-cli.py --server 8707 > test.10 &
/home/compunction/speedtest-cli.py --server 12407 > test.11 &
/home/compunction/speedtest-cli.py --server 15853 > test.12 &
sleep 30
/bin/cat /home/compunction/test.* | /usr/bin/grep Download | /usr/bin/awk '{print $2}' > test.output
/bin/cat /home/compunction/test.output
echo "Total: "
/bin/cat /home/compunction/test.output | /usr/bin/awk '{sum+=$1} END{print sum}'
/bin/rm /home/compunction/test.*

----------OUTPUT----------
40.09
42.32
69.85
81.78
288.80
41.57
44.78
45.37
86.77
50.05
64.90
76.64
Total:
932.92

Intel(R) Core(TM) i7-8809G CPU @ 3.10GHz (3096.15-MHz K8-class CPU)
real memory  = 34359738368 (32768 MB)
igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k>
em0: <Intel(R) PRO/1000 Network Connection 7.6.1-k>

I have wondered myself if they could be throttling me, but when I test speed test on two different sites from two different machines, I see the speed drop as I would expect.  But general internet latency is likely a factor as well.  I need to grab my old hardware and setup a iperf box on the wan side.  I could likely put a 192.168.100.x IP on it has I have a route for that to get the router interface. (since I only have one WAN IP).

[net.inet.tcp.tso="0"dev.igb.0.fc="0"dev.em.0.fc="0"hw.igb.rx_process_limit="-1"hw.igb.tx_process_limit="-1"hw.em.rx_process_limit="-1"hw.em.tx_process_limit="-1"dev.igb.0.eee_disabled="1"machdep.hyperthreading_allowed="0"hw.igb.txd="2048"hw.igb.rxd="2048"hw.em.txd="2048"hw.em.rxd="2048"net.link.ifqmaxlen="4096"hw.igb.max_interrupt_rate="32000"hw.em.max_interrupt_rate="32000"net.inet.tcp.sendbuf_inc="32768"net.inet.tcp.recvbuf_inc="32768"net.inet.tcp.sendbuf_max="4194304"net.inet.tcp.recvbuf_max="4194304"]

@Ric878, you and I are in the same boat brother.  I have not found a solution yet, but been tinkering for a few weeks.

To rule out routing, IPS, etc...I have been testing with a python script so only the WAN interface is in scope.  If I plug my MacBook into the MB8600 I get between 930-950.  With my OPNsense box I see a max of 600-630.

This test shows only 428.87, but its the middle of the day ;-)
compunction@opnsense:~ % ./speedtest-cli.py
Retrieving speedtest.net configuration...
Testing from Comcast Cable (73.237.238.141)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Comcast (Atlanta, GA) [20.61 km]: 12.635 ms
Testing download speed................................................................................
Download: 428.87 Mbit/s
Testing upload speed................................................................................................
Upload: 41.34 Mbit/s
compunction@opnsense:~ %

A few things I have tried:
net.inet.tcp.tso="0"
dev.igb.0.fc="0"
dev.em.0.fc="0"
hw.igb.rx_process_limit="-1"
hw.igb.tx_process_limit="-1"
hw.em.rx_process_limit="-1"
hw.em.tx_process_limit="-1"
dev.igb.0.eee_disabled="1"
machdep.hyperthreading_allowed="0"
hw.igb.txd="2048"
hw.igb.rxd="2048"
hw.em.txd="2048"
hw.em.rxd="2048"
net.link.ifqmaxlen="4096"
hw.igb.max_interrupt_rate="32000"
hw.em.max_interrupt_rate="32000"
net.inet.tcp.sendbuf_inc="32768"
net.inet.tcp.recvbuf_inc="32768"
net.inet.tcp.sendbuf_max="4194304"
net.inet.tcp.recvbuf_max="4194304"

I have also tried swapping cat5e cables...no luck.

Still in the search for a resolution :).

Thanks everyone for the information.  I ended up building my own box with dual intel Nic's.  Now on to figuring out why I am only getting 500-600Mbps on this Nic, when I can get 950 with I connect my Mac directly to the cable modem.  But that is for another thread if I do not find an answer via google.

root@opnsense:/home/compunction # ./speedtest-cli.py --server 11143
Retrieving speedtest.net configuration...
Testing from Comcast Cable (<removed>)...
Retrieving speedtest.net server list...
Retrieving information for the selected server...
Hosted by NationalNet (Marietta, GA) [8.53 km]: 12.133 ms
Testing download speed................................................................................
Download: 505.33 Mbit/s
Testing upload speed................................................................................................
Upload: 42.06 Mbit/s
root@opnsense:/home/compunction # ./speedtest-cli.py --server 11143 --source 192.168.24.1
Retrieving speedtest.net configuration...
Testing from Comcast Cable (<removed>)...
Retrieving speedtest.net server list...
Retrieving information for the selected server...
Hosted by NationalNet (Marietta, GA) [8.53 km]: 12.936 ms
Testing download speed................................................................................
Download: 292.27 Mbit/s
Testing upload speed................................................................................................
Upload: 27.74 Mbit/s
root@opnsense:/home/compunction #

Yeah I popped the nic in my Mac and I can hit over 900Mbps, so it does appear to be a driver issue.

Qotom is new to me, how reliable are they?

Thanks for the info, it looks like that one uses a ASIX AX88772 chip, maybe I can find usb-c adapter with a chip supported by the same driver.

I had a 1Gbps Comcast link installed today and setup multi-wan with my existing 100Mbps AT&T link.  I got that all working and PBR on the firewall to split my traffic and failover.  But as I started testing speeds I noticed that my USB-C NIC maxes out at 60Mbps.  I assumed my AT&T connection was only capable of 60Mbps before, but now I am finding its a interface issue (likely driver).  As I am using an Intel NUC, I only have one physical interface can I am using that on the LAN side, so I am stuck with Thunderbolt or USB-C for my WAN interfaces.  They are coming up as ue0 and ue1.

ugen1.4: <Realtek USB 10100 LAN> at usbus1, cfg=1 md=HOST spd=SUPER (5.0Gbps) pwr=ON (64mA)
ugen1.5: <Realtek USB 101001000 LAN> at usbus1, cfg=1 md=HOST spd=SUPER (5.0Gbps) pwr=ON (64mA)

I tried to see if another driver supported these NIC's but that does not appear to be the case.

Does anyone have recommendations for different NIC's
OR
Know how to boost the performance on the ones I have?

Thanks
Mark

I am trying to setup SSL inspection for a few machines on my network.  I setup Squid in transparent mode with port forward.  I am able to get to https://www.yahoo.com without issue but if I try to get to https://opnsense.org I get the following error in the cache.log.

2017/09/06 14:13:08 kid1| Error negotiating SSL connection on FD 26: error:140A1175:SSL routines:ssl_bytes_to_cipher_list:inappropriate fallback (1/-1)

I am using an internal CA and I have imported the certificate into the browsers trusted root.

Any idea's?

Almost forgot, to get the delegated address on my LAN I had to enable "Directly send SOLICIT" on WAN.

I am playing with this myself, and I still trying to learn the new world of IPv6 at the same time.

Here is what I have so far:
IP Passthrough setup on my ATT NVG589 Gateway.

On OPNsense WAN
-IPv4 set to DHCP
-IPv6 set to DHCPv6

On OPNsense LAN
-IPv4 set static 192.168. address
-IPv6 set to Track Interface "WAN"

On WAN I get a routable IPv4 and IPv6 address.  On LAN I have my Static RFC1918 address and I get a routable IPv6 address from the delegated prefix listed on the router.

Now that I have this setup and the /64 appears to be properly delegated to my LAN, I need to figure out to my clients will get addresses.  not sure if they should be using SLAAC or if I should configured DHCPv6 on OPNsense.

Hello all, I am new to OPNsense and this is the first time I am plying with IPv6.  I want to write firewall rules based on hostname.  Therefore I want to make sure my hosts IPv6 address is in DNS.  I have IPv4 working as expected, but the IPv6 addresses are not showing up in DNS.  I do see dnsmasq listeing on my IPv6 space with this command line argument --listen-address=2602:306:<removed>, but no IPv6 addresses in /var/etc/dnsmasq-hosts.  I was looking for a guide on the wiki, but I do not see one :(.  What are I missing...

Thanks!
Mark