Messages

1

19.7 Legacy Series / Re: kernel panics through multiple releases

« on: February 28, 2020, 08:08:05 pm »

Just poking my head back in here. One day I'd like to move back to opnsense.

Been on pfsense for a few months now and no file corruption and no panics, so I'm pretty sure I'm not running into a hardware issue (especially after all the hardware testing). Probably not even really an opnsense issue, but a HardenedBSD issue. My guess is this - my hardware is older (Core2Duo era), Shawn probably has nothing like it that he tests on regularly and it's just a bug that's not been addressed that shows up on this particular hardware.

2

19.7 Legacy Series / Re: kernel panics through multiple releases

« on: December 23, 2019, 08:19:17 am »

I've done that a few times already.

3

19.7 Legacy Series / Re: kernel panics through multiple releases

« on: December 22, 2019, 07:59:05 pm »

Franco - understood, but the problem goes away with any other OS running on the hardware, so I'm just thinking it's a bug and I'm running on hardware that's not well-tested with HBSD. I'll certainly report back if in the future I see similar issues under pfsense...

I guess if I'm feeling adventurous I could try the "wrap it in a vmware bubble blanket", but that feels a little extreme.

4

19.7 Legacy Series / kernel panics through multiple releases

« on: December 21, 2019, 01:56:18 am »

Hi all - I was pretty happy with opnsense as a concept. Coming from pfsense, I was a bit saddened by how that project has changed over the years, especially the move to require AES-NI CPU support in the future (which they seem to have backed off from). So opnsense looked like a good option, and the fact that you've already started the process of "cleaning house" on old code was a big deal to me.

That said, last week I moved back to pfsense. It became necessary because no matter what I did (replacing hardware, turning off "big" features like IDS/IPS, clean reinstalls, etc.) I was just getting fairly regular kernel panics. The more I watched this, the more I realized that with UFS I was getting serious data corruption each time (as shown by the built-in 'health check') and for a time I thought perhaps that was the root of my problem - some prior release paniced once and then subsequent panics were the result of corruption in some kernel module or something. I eventually moved to ZFS using the nice bootstrapping tool provided and I saw a few panics, the last of which left the system unbootable (panic during mountroot).

A few threads where I brought up the panics, but didn't really find any resolution, mostly me talking to myself at some point:

https://forum.opnsense.org/index.php?topic=14323.0 (configd)
https://forum.opnsense.org/index.php?topic=12267.msg68445#msg68445 (zfs install)

So I yanked the drive, put in an old drive (one that also had opnsense on it that I'd swapped out to test if the corruption was a drive failure), and installed pfsense w/the zfs install option. A week later and it's still going (and thankfully aliases and dhcp static mappings are pretty easy to export/import across platforms) and it's still working without any panics. This is great, but I'm also on a platform that promises to obsolete my hardware with the next major release (which may not come given how much time their other linux-based project is getting).

So what's my point in posting?

Just calling attention to the issue, giving people with similar hardware a chance to find this via google, whatever. My gut feeling is that while HardenedBSD is great, it sees WAY less hardware than mainline FreeBSD and it's just not happy with my old Core2Duo (E7500, 2.93GHz) Dell. It reminds me of the early days of OpenBSD - secure, but as you add more protections, you end up with less stability because you're bailing out whenever you hit an unexpected condition. This is GOOD - it means your protections and correctness in following spec is working. It's bad if you have users that hit the bugs and don't have the manpower to follow up. Anyhow, I've done the "submit a bug" thing after each of these panics for the last year or so so there's a record for anyone wanting to look at it. And I have plenty of spare drives around and a copy of my last config so if anyone ever wants to troubleshoot with me, I have no problem flipping over to opnsense again for testing.

From my end though, I've hit a dead end - the built-in Dell diagnostics all pass, memtest86 passes, SMART passes on all drives I've tried (after a "long" self-test), pegging the cpu with benchmarkers doesn't trigger the bug, CPU fan is fine, so not sure what else I could do.

5

19.7 Legacy Series / Re: OpenVPN server listen on multiple UDP ports?

« on: November 29, 2019, 05:26:16 pm »

Reconfiguring the clients is fine, as it's just my phone and laptop.

I just want to make sure that the server side doesn't care about the redirect or that there's nothing in the handshake where the client and server try to enforce that the ports match (ie: client hitting port 443, server listening on 1194) - like some kind of primitive defense against a MiTM attack.

Just recently I was somewhere with guest wifi and I was not able to hit my home ovpn instance on the default 1194 port, but was able to hit another server on 443 UDP.

6

19.7 Legacy Series / OpenVPN server listen on multiple UDP ports?

« on: November 29, 2019, 12:20:25 am »

I know the server technically can't, but if I have it listening on 1194, and I'd like to add a handful of other ports that I suspect wouldn't be blocked, is there any issue with doing this using port forwards?

7

19.7 Legacy Series / Re: OpenVPN: remote routes work from shell, not from LAN

« on: November 26, 2019, 09:12:57 pm »

Finally!

On the server (a FreeBSD host), I had to do something to manipulate OpenVPN's internal routing.

In my "ccd" directory, I added a client-specific file for my opnsense box, with one line: "iroute 10.3.2.0 255.255.255.0", which if I understand this correctly, tells the openvpn process to shove packets it sees destined to that subnet to a specific connected client.

I then had to add a "route" statement to the server's openvpn.conf, "route 10.3.2.0 255.255.255.0". This adds a route (kernel) to the openvpn tun interface I think.

Neither really explains why my tcpdump wasn't showing the incoming traffic without that "iroute" statement, but I guess that remains a mystery.

Next up, what if I want to set some firewall rules on the opnsense openvpn client tunnel interface? This config works, but it also leaves my local LAN fully exposed to the remote network (I don't want that, I want outbound-only).

8

19.7 Legacy Series / Re: OpenVPN: remote routes work from shell, not from LAN

« on: November 26, 2019, 07:59:21 pm »

OK, so I added a rule with a bunch of aliases that represent the remote networks of interest. For gateway, left that at "default".

I still can't ping across the VPN, but the traffic is now visible on the firewall's tun interface:

Code: [Select]

root@SporkLab:~ # ifconfig ovpnc2
ovpnc2: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
inet6 fe80::2e0:4cff:fe70:1219%ovpnc2 prefixlen 64 scopeid 0x9
inet 10.99.0.6 --> 10.99.0.5 netmask 0xffffffff
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
groups: tun openvpn
Opened by PID 70079
root@SporkLab:~ #
root@SporkLab:~ # tcpdump -nv -i ovpnc2 dst 10.99.0.1
tcpdump: listening on ovpnc2, link-type NULL (BSD loopback), capture size 262144 bytes
13:55:18.885355 IP (tos 0x0, ttl 63, id 32780, offset 0, flags [none], proto ICMP (1), length 84)
    10.3.2.40 > 10.99.0.1: ICMP echo request, id 27137, seq 0, length 64
13:55:19.888343 IP (tos 0x0, ttl 63, id 55650, offset 0, flags [none], proto ICMP (1), length 84)
    10.3.2.40 > 10.99.0.1: ICMP echo request, id 27137, seq 1, length 64
13:55:20.889000 IP (tos 0x0, ttl 63, id 22218, offset 0, flags [none], proto ICMP (1), length 84)
    10.3.2.40 > 10.99.0.1: ICMP echo request, id 27137, seq 2, length 64
13:55:21.889970 IP (tos 0x0, ttl 63, id 44700, offset 0, flags [none], proto ICMP (1), length 84)
    10.3.2.40 > 10.99.0.1: ICMP echo request, id 27137, seq 3, length 64
But it doesn't show up at the other end:

Code: [Select]

[root@foo-b /home/spork]# tcpdump -vn -i tun0
tcpdump: listening on tun0, link-type NULL (BSD loopback), capture size 262144 bytes


^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
[root@foo-b /home/spork]#
This is where ovpn's opaqueness trips me up. Why is it just eating the packets and not forwarding them on? The 10.99.0.1 IP should be an easy one, that's part of the tunnel subnet.

Of note, pinging 10.99.0.1 from the firewall's shell still works.

9

19.7 Legacy Series / Re: OpenVPN: remote routes work from shell, not from LAN

« on: November 25, 2019, 09:42:30 pm »

What would a rule like that look like? In my dropdown for destination, I've got "default", each individual physical gateway, null routes and the failover group, but not openvpn.

10

19.7 Legacy Series / OpenVPN: remote routes work from shell, not from LAN

« on: November 25, 2019, 08:03:03 pm »

I gave up on this before, but thought I'd try again with a fresh config on both ends.

Verified the server side is OK by connecting with a desktop client and verifying my certs are OK, that I can ping the remote OVPN interface and some IPs behind the VPN. All is well.

Also, if I ssh into the opnsense box, no problem. I can ping what I expect to be able to ping.

From the LAN though, my traffic all goes out the main WAN connection. Verified this with tcpdump.

Some quick examples follow...

From the shell:

Code: [Select]

root@SporkLab:/home/sporkadmin # ping 10.99.0.1
PING 10.99.0.1 (10.99.0.1): 56 data bytes
64 bytes from 10.99.0.1: icmp_seq=0 ttl=64 time=5.702 ms
64 bytes from 10.99.0.1: icmp_seq=1 ttl=64 time=7.859 ms

root@SporkLab:/home/sporkadmin # ping 10.88.77.72
PING 10.88.77.72 (10.88.77.72): 56 data bytes
64 bytes from 10.88.77.72: icmp_seq=0 ttl=64 time=7.829 ms
64 bytes from 10.88.77.72: icmp_seq=1 ttl=64 time=6.264 ms
When pinging from a host on the LAN, a tcpdump on the tun interface shows nothing:

Code: [Select]

frankentosh:2015-Hackintosh-Drive spork$ ping 10.99.0.1
PING 10.99.0.1 (10.99.0.1): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2

root@SporkLab:/home/sporkadmin # tcpdump -vn -i ovpnc2 dst 10.99.0.1
tcpdump: listening on ovpnc2, link-type NULL (BSD loopback), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
But if I run tcpdump on the main WAN interface, I see what should be tunneled going right out the WAN interface:

Code: [Select]

root@SporkLab:/home/sporkadmin # tcpdump -vn -i re0 dst 10.99.0.1
tcpdump: listening on re0, link-type EN10MB (Ethernet), capture size 262144 bytes


14:01:34.515541 IP (tos 0x0, ttl 63, id 64433, offset 0, flags [none], proto ICMP (1), length 84)
    WAN IP > 10.99.0.1: ICMP echo request, id 5983, seq 0, length 64
14:01:35.516589 IP (tos 0x0, ttl 63, id 45486, offset 0, flags [none], proto ICMP (1), length 84)
    WAN IP > 10.99.0.1: ICMP echo request, id 5983, seq 1, length 64
14:01:36.516356 IP (tos 0x0, ttl 63, id 1206, offset 0, flags [none], proto ICMP (1), length 84)
    WAN IP > 10.99.0.1: ICMP echo request, id 5983, seq 2, length 64
I have no custom NAT rules for outbound.

I do have dual WAN setup as described in the docs.

Any idea what's happening?

11

19.1 Legacy Series / Re: 19.1.4 ISO - ZFS install option missing?

« on: November 25, 2019, 11:08:33 am »

So this is kind of crazy. Got a few panics, but no swap was enabled (forgot to change ada2 to ada0 in fstab).

'zpool scrub' shows me no errors, but the health check shows some bad checksums, and I found the dashboard was not loading because "interfaces.inc" just had some random garbage sprinkled in the file. And the dashboard choking on those errors happened while the system was up and running.

Health check has this, not sure if this is corruption or if these are leftovers from the initial FreeBSD install:

Code: [Select]

***GOT REQUEST TO AUDIT HEALTH***
>>> Check installed kernel version
Version 19.7.7 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 19.7.7 is correct.
>>> Check for missing or altered base files
Error 2 ocurred.
lib/libcrypto.so.8:
sha256digest (0xbfe218c482ce5e1bcce776e265ee46b61a2425c91523de5a6472e1d3a39775e9, 0x1a55439f97072bbc6de73d19bbaa6e71e289a91fe86d9971af0483b3795c20b4)
usr/lib/libarchive.so.6:
sha256digest (0x917956bf32b66a471e0b8d6d0a4ecd2c3deaafd037a6778fc4c55cf0590c334a, 0xb2767e3b22acc19e0f40a05e74a5892a86095e4bd59e156c01fb60b2d71675f0)
usr/bin/awk:
sha256digest (0xc08dd78ff24e3d9d8484f55aeedd4105c3de8cd65be29db6b02aef61363795a9, 0xe3d9ce8f7c16df56f399543462f8bc73facfd012ce6af77543f0088e416aa325)
usr/bin/openssl:
sha256digest (0x9ce124fcee5cff90a2869273e48e9910b24873d454eb2f8af4c12258254ed391, 0x81f4e33a0e22fa02a9d4ccf2308074c8d15d3b5c0f1a8926ccce7c5413086bf6)
usr/bin/nawk:
sha256digest (0xc08dd78ff24e3d9d8484f55aeedd4105c3de8cd65be29db6b02aef61363795a9, 0xe3d9ce8f7c16df56f399543462f8bc73facfd012ce6af77543f0088e416aa325)
>>> Check for and install missing package dependencies
/usr/lib/libarchive.so.6: Undefined symbol "@FBSD_1.0"
>>> Check for missing or altered package files
/usr/lib/libarchive.so.6: Undefined symbol "@FBSD_1.0"
***DONE***
Might have to spend a few months on that other firewall distribution (if they still let me run w/o AESNI, ha!), if for no other reason than to regain my sanity.

12

19.1 Legacy Series / Re: 19.1.4 ISO - ZFS install option missing?

« on: November 25, 2019, 02:14:47 am »

Thanks - just got around to this today, was really easy:

- backup config
- grab another drive (I suspect with all the file corruption I've been seeing around my kernel panics, the drive has to be bad even if SMART isn't showing anything), stick it in any PC laying around
- Install FreeBSD 11.2
- "pkg add" the nss cert stuff and git
- run the bootstrap script
- upload old config
- pop drive into firewall

And with zfs, I am hoping that even if the box panics for any reason I've got a better chance of not having corrupted files in the base.

Now I'm going to fiddle with zfs a bit and do the "poor man's RAID" and set "copies=2" on a few of the filesystems...

13

19.1 Legacy Series / Re: 19.1.4 ISO - ZFS install option missing?

« on: November 15, 2019, 03:36:01 am »

Just curious, is the opnsense-bootstrap method still supported?

And I assume future updates don't care about whether the box was originally an opnsense install or was migrated from a stock FreeBSD install?

14

19.7 Legacy Series / Re: Recommend me a VPN

« on: October 21, 2019, 10:24:39 pm »

I agree with mimugmail.
Why no IPsec.

Also since IPSEC relies on GRE being let through, NAT not breaking it, etc. I do prefer something that just uses one protocol over one port. Easier to diagnose basic connectivity.

An example - ongoing issue where one of these carriers is doing something, including a note that some content inspection gear is doing something dumb:

https://puck.nether.net/pipermail/outages/2019-October/012696.html

15

19.7 Legacy Series / Re: [SOLVED] configd not running, won't start

« on: October 21, 2019, 10:21:53 pm »

And another panic yesterday, but no mention of UFS.

Code: [Select]

Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer = 0x20:0xffffffff80c1166d
stack pointer         = 0x28:0xfffffe0119a9b750
frame pointer         = 0x28:0xfffffe0119a9b7c0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 52270 (ntpd)

Code: [Select]

db:0:kdb.enter.default>  bt
Tracing pid 52270 tid 100179 td 0xfffff8001df9c000
__mtx_lock_sleep() at __mtx_lock_sleep+0xcd/frame 0xfffffe0119a9b7c0
_cv_wait_sig() at _cv_wait_sig+0x1f8/frame 0xfffffe0119a9b810
seltdwait() at seltdwait+0xc3/frame 0xfffffe0119a9b850
kern_select() at kern_select+0x850/frame 0xfffffe0119a9ba40
sys_select() at sys_select+0x56/frame 0xfffffe0119a9ba80
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe0119a9bbb0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0119a9bbb0
--- syscall (93, FreeBSD ELF64, sys_select), rip = 0x2e98a427d0a, rsp = 0x7249552dfca8, rbp = 0x7249552dfce0 ---
Submitted via the built-in reporting tool.