Topics

1

19.7 Legacy Series / kernel panics through multiple releases

« on: December 21, 2019, 01:56:18 am »

Hi all - I was pretty happy with opnsense as a concept. Coming from pfsense, I was a bit saddened by how that project has changed over the years, especially the move to require AES-NI CPU support in the future (which they seem to have backed off from). So opnsense looked like a good option, and the fact that you've already started the process of "cleaning house" on old code was a big deal to me.

That said, last week I moved back to pfsense. It became necessary because no matter what I did (replacing hardware, turning off "big" features like IDS/IPS, clean reinstalls, etc.) I was just getting fairly regular kernel panics. The more I watched this, the more I realized that with UFS I was getting serious data corruption each time (as shown by the built-in 'health check') and for a time I thought perhaps that was the root of my problem - some prior release paniced once and then subsequent panics were the result of corruption in some kernel module or something. I eventually moved to ZFS using the nice bootstrapping tool provided and I saw a few panics, the last of which left the system unbootable (panic during mountroot).

A few threads where I brought up the panics, but didn't really find any resolution, mostly me talking to myself at some point:

https://forum.opnsense.org/index.php?topic=14323.0 (configd)
https://forum.opnsense.org/index.php?topic=12267.msg68445#msg68445 (zfs install)

So I yanked the drive, put in an old drive (one that also had opnsense on it that I'd swapped out to test if the corruption was a drive failure), and installed pfsense w/the zfs install option. A week later and it's still going (and thankfully aliases and dhcp static mappings are pretty easy to export/import across platforms) and it's still working without any panics. This is great, but I'm also on a platform that promises to obsolete my hardware with the next major release (which may not come given how much time their other linux-based project is getting).

So what's my point in posting?

Just calling attention to the issue, giving people with similar hardware a chance to find this via google, whatever. My gut feeling is that while HardenedBSD is great, it sees WAY less hardware than mainline FreeBSD and it's just not happy with my old Core2Duo (E7500, 2.93GHz) Dell. It reminds me of the early days of OpenBSD - secure, but as you add more protections, you end up with less stability because you're bailing out whenever you hit an unexpected condition. This is GOOD - it means your protections and correctness in following spec is working. It's bad if you have users that hit the bugs and don't have the manpower to follow up. Anyhow, I've done the "submit a bug" thing after each of these panics for the last year or so so there's a record for anyone wanting to look at it. And I have plenty of spare drives around and a copy of my last config so if anyone ever wants to troubleshoot with me, I have no problem flipping over to opnsense again for testing.

From my end though, I've hit a dead end - the built-in Dell diagnostics all pass, memtest86 passes, SMART passes on all drives I've tried (after a "long" self-test), pegging the cpu with benchmarkers doesn't trigger the bug, CPU fan is fine, so not sure what else I could do.

2

19.7 Legacy Series / OpenVPN server listen on multiple UDP ports?

« on: November 29, 2019, 12:20:25 am »

I know the server technically can't, but if I have it listening on 1194, and I'd like to add a handful of other ports that I suspect wouldn't be blocked, is there any issue with doing this using port forwards?

3

19.7 Legacy Series / OpenVPN: remote routes work from shell, not from LAN

« on: November 25, 2019, 08:03:03 pm »

I gave up on this before, but thought I'd try again with a fresh config on both ends.

Verified the server side is OK by connecting with a desktop client and verifying my certs are OK, that I can ping the remote OVPN interface and some IPs behind the VPN. All is well.

Also, if I ssh into the opnsense box, no problem. I can ping what I expect to be able to ping.

From the LAN though, my traffic all goes out the main WAN connection. Verified this with tcpdump.

Some quick examples follow...

From the shell:

Code: [Select]

root@SporkLab:/home/sporkadmin # ping 10.99.0.1
PING 10.99.0.1 (10.99.0.1): 56 data bytes
64 bytes from 10.99.0.1: icmp_seq=0 ttl=64 time=5.702 ms
64 bytes from 10.99.0.1: icmp_seq=1 ttl=64 time=7.859 ms

root@SporkLab:/home/sporkadmin # ping 10.88.77.72
PING 10.88.77.72 (10.88.77.72): 56 data bytes
64 bytes from 10.88.77.72: icmp_seq=0 ttl=64 time=7.829 ms
64 bytes from 10.88.77.72: icmp_seq=1 ttl=64 time=6.264 ms
When pinging from a host on the LAN, a tcpdump on the tun interface shows nothing:

Code: [Select]

frankentosh:2015-Hackintosh-Drive spork$ ping 10.99.0.1
PING 10.99.0.1 (10.99.0.1): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2

root@SporkLab:/home/sporkadmin # tcpdump -vn -i ovpnc2 dst 10.99.0.1
tcpdump: listening on ovpnc2, link-type NULL (BSD loopback), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
But if I run tcpdump on the main WAN interface, I see what should be tunneled going right out the WAN interface:

Code: [Select]

root@SporkLab:/home/sporkadmin # tcpdump -vn -i re0 dst 10.99.0.1
tcpdump: listening on re0, link-type EN10MB (Ethernet), capture size 262144 bytes


14:01:34.515541 IP (tos 0x0, ttl 63, id 64433, offset 0, flags [none], proto ICMP (1), length 84)
    WAN IP > 10.99.0.1: ICMP echo request, id 5983, seq 0, length 64
14:01:35.516589 IP (tos 0x0, ttl 63, id 45486, offset 0, flags [none], proto ICMP (1), length 84)
    WAN IP > 10.99.0.1: ICMP echo request, id 5983, seq 1, length 64
14:01:36.516356 IP (tos 0x0, ttl 63, id 1206, offset 0, flags [none], proto ICMP (1), length 84)
    WAN IP > 10.99.0.1: ICMP echo request, id 5983, seq 2, length 64
I have no custom NAT rules for outbound.

I do have dual WAN setup as described in the docs.

Any idea what's happening?

4

19.7 Legacy Series / Recommend me a VPN

« on: October 10, 2019, 01:02:57 am »

I'm kind of annoyed with OpenVPN as I could never get it to work in my particular scenario for site-to-site use. I find it's great for getting from a coffee shop to my home net though, so I'll leave that as-is.

But I have 3-4 other sites where I would like to have site-to-site setups between my home (simple network - two WANs, one just for backup, one LAN net, that's it) and some remote networks.

My requirements are:

- The other end only has proprietary stuff that only does IPSEC, so I have to tunnel back to a FreeBSD host at the other end rather than the router (I know this complicates things)
- I need to filter the traffic on my end - I should be able to reach out, none of the remote sites should reach in
- I do need to add additional routes, accessed via the remote sites
- The other end is FreeBSD in all cases, so whatever I run has to support FreeBSD

OpenVPN confuses me in these type of use cases as it has it's own internal/hidden routing table. If anyone thinks it could support the above, I'd give it a try, but I've had no luck with this on OPNSense (worked on pfsense, but not with any setup that let me filter traffic).

Or if you want to make a case for using the Cisco and SonicWall IPSEC VPNs at these sites instead, I'm all ears, but I fear interoperability headaches, and it seems like adding additional remote routes is a real pain.

Or pitch me on something I've not mentioned!

5

19.7 Legacy Series / [SOLVED] configd not running, won't start

« on: September 25, 2019, 11:23:38 pm »

Logged-in to the GUI today to see if the box is still panicing every night and most of the data in the dashboard was blank.

In the "services" pane, I saw that "configd" was not running.  On trying to start it, this is logged in the system logs:

Code: [Select]

opnsense: /status_services.php: The command '/usr/local/etc/rc.d/configd start' returned exit code '1',
the output was 'Starting configd. Traceback (most recent call last): File "/usr/local/opnsense/service/configd.py",
line 37, in <module> import logging File "/usr/local/lib/python3.7/logging/__init__.py", line 26, in <module> import sys,
os, time, io, traceback, warnings, weakref, collections.abc File "/usr/local/lib/python3.7/traceback.py", line 5,
in <module> import linecache File "/usr/local/lib/python3.7/linecache.py", line 8, in <module> import functools
ModuleNotFoundError: No module named 'functools' /usr/local/etc/rc.d/configd: WARNING: failed to start configd'
Any idea what that's about?

No access via ssh, I assume something is broken there as well.

6

19.1 Legacy Series / Internet outage, all hell breaks loose

« on: February 10, 2019, 03:08:20 am »

Just updated to 19.1.1 last night and it seemed to work well.

A few hours ago, I start getting txts that my fios line is down, and sure enough, there appears to be no internet access (fairly rare occurrence with FTTH outside of maintenance hours, TBH).  So my first thought is that like 18.7, opnsense had paniced or something, or I'd hit some new bug in 19.1.

Web interface worked, but only to a limited extent - dashboard showed some info, but actually toggling things (enable disable fios interface) or renewing the dhcp lease, no response.  I ssh'd in and ran 'dmesg' and it was just full of "[zone: pf states] PF states limit reached" messages. Digging a bit more with 'pfctl -ss', I saw that it was basically all outbound DNS requests, presumably from unbound.

I killed unbound, but couldn't remember how to manually kill states (and couldn't google it!). So then I just checked my fios interface and I think I confirmed an outage by noting that tcpdump was showing me absolutely nothing (or the interface was locked up?).

I attempted the "restart all services" in hopes of getting the full GUI back, and it was hanging on restarting cron, had to "kill -9" poor cron in the shell. Things were still odd. It appears php-fpm and configd/python were just dying:

Code: [Select]

pid 64855 (python2.7), uid 0: exited on signal 10 (core dumped)
pid 78374 (python2.7), uid 0: exited on signal 10 (core dumped)
[HBSD SEGVGUARD] [python2.7 (78374)] Suspension expired.
 -> pid: 78374 ppid: 78269 p_pax: 0xa50<SEGVGUARD,ASLR,NOSHLIBRANDOM,NODISALLOWMAP32BIT>
[HBSD SEGVGUARD] [/usr/local/bin/python2.7 (35385)] Suspension expired.
 -> pid: 35385 ppid: 34118 p_pax: 0xa50<SEGVGUARD,ASLR,NOSHLIBRANDOM,NODISALLOWMAP32BIT>
pid 43398 (sleep), uid 0: exited on signal 10
[HBSD SEGVGUARD] [/bin/sleep (81756)] Suspension expired.
 -> pid: 81756 ppid: 81407 p_pax: 0xa50<SEGVGUARD,ASLR,NOSHLIBRANDOM,NODISALLOWMAP32BIT>
[HBSD SEGVGUARD] [/usr/local/bin/php (46831)] Suspension expired.
 -> pid: 46831 ppid: 38217 p_pax: 0xa50<SEGVGUARD,ASLR,NOSHLIBRANDOM,NODISALLOWMAP32BIT>
pid 8347 (python2.7), uid 0: exited on signal 10 (core dumped)
ovpns1: link state changed to DOWN
pid 8749 (python2.7), uid 0: exited on signal 11 (core dumped)
[HBSD SEGVGUARD] [python2.7 (8749)] Suspension expired.
 -> pid: 8749 ppid: 38443 p_pax: 0xa50<SEGVGUARD,ASLR,NOSHLIBRANDOM,NODISALLOWMAP32BIT>
ovpns1: link state changed to UP
[HBSD SEGVGUARD] [/usr/local/bin/python2.7 (48064)] Suspension expired.
 -> pid: 48064 ppid: 47524 p_pax: 0xa50<SEGVGUARD,ASLR,NOSHLIBRANDOM,NODISALLOWMAP32BIT>
pid 23052 (awk), uid 0: exited on signal 10 (core dumped)
[HBSD SEGVGUARD] [/usr/bin/awk (74876)] Suspension expired.

No idea what that's about other than it must be some HardenedBSD feature that's giving that extra info. I have way more of that logged if it's of interest.

Finally gave up and rebooted and internet came back and no more unusual state table bloating and so far no dying php/python.

Not sure if this was an outage triggering all this chaos or if the chaos happened and rebooting was the thing that resolved the outage. The totally blank tcpdump gave me pause.  I got a new IP when rebooting, which is unusual for fios, but common after maintenance and outages sooo???

unbound filling the state table is kind of odd too - not sure why it would just keep firing off new queries when no answers are received.

Anyone want more info?

7

18.7 Legacy Series / OpenVPN client firewall rules

« on: October 11, 2018, 12:47:46 am »

I'm still migrating from that other *sense, so bear with me...

At home, I have my current firewall setup as an openvpn client to 3-4 sites.  Those sites are all just openvpn running on FreeBSD, not any sort of firewall distro. This works, but my current config is ugly, with a mess of NAT rules, some weird magic to make firewall rules work on the openvpn client interfaces, and other things that were really just arrived at by accident.

I was looking at the docs and it's not totally clear to me - how is firewalling on the openvpn client interfaces handled? Can I treat it like any other interface or is it "special" because it's openvpn (which admittedly complicates things - it basically has its own internal routing table).

If it matters, I also run openvpn as a server on this same firewall for remote access.

8

18.7 Legacy Series / Verizon fios and DHCPv6-PD?

« on: October 09, 2018, 02:29:48 am »

Anyone here on fios and in an area that's had v6 turned up?  Apparently it's "coming soon" to at least the northeast US.

My understanding is that the "PD" there is "prefix delegation".  I'm very confused by that part - if your ISP hands out a new prefix, what happens while all your hosts still have their leases active but the prefix routed to your connection has changed?

Any general issues to be aware of with DHCPv6-PD and opnsense?  Or IPv6 and opnsense in general (I'm coming from another firewall vendor that has 'sense' in the name)?

9

General Discussion / Switching from pfSense - features

« on: May 07, 2017, 03:52:01 am »

Hi all,

I'd really like to learn about the differences between the two firewall products, ideally on my home connection so I'm not mucking around with someone's connectivity who's actually paying for service.

Last time I looked at OPNsense was at least a year ago and there were a few missing features.  I think everything is there now, but can someone confirm?  I have two connections, 100/100 FiOS and 3.0/768kb/s DSL. These are my must-haves for home use:

- Dual WAN support (primary and backup)
- Traffic shaping that works for dual WANs where each WAN connection is a different speed
- Traffic shaping that just prioritizes a few things up/down - VoIP (based on my VoIP phone IPs), ssh, DNS, ICMP, OpenVPN, and a handful of other things
- OpenVPN (I use it as a server for when I'm working outside my home, and it's acting as a VPN client for 3 remote sites)
- Dynamic DNS updates
- DNSSEC-capable resolver (not forwarder)

If all of that's available in the current stable version, I'm good to go...

10

General Discussion / QoS/shaping Guide?

« on: February 12, 2016, 11:44:27 pm »

Based on some forum searching, there used to be a wiki entry on the shaping/QoS but that link seems dead.

Since this is completely different from pfsense and there's no wizard, any pointers on how this works and how to configure?

My usual rules are pretty simple:

-prioritize icmp, dns, ssh, tcp ACKs, openvpn, IRC, jabber
-whatever magic the pfsense wizard applies for voip (port 5060 obviously, not sure what it's doing for the RTP streams, just using DSCP tags or something?)
-everything else is "normal"
-some things (BT, Usenet) get lower priority

Is this possible yet or wait for future releases?  Didn't see anything on the roadmap page.

11

General Discussion / [SOLVED] Hardware Forum?

« on: February 12, 2016, 07:52:15 am »

Perhaps this forum will do for now, but it would be nice to have some user-reported hardware success/failure stories.

I've been looking for something small, dual LAN, and very cheap recently, and this looks awesome, but I'm not sure if it would be at all supported:

http://www.hystou.com/products/fanless-mini-pc/celeron-mini-pc/small-fanless-pc-mini-computer-barebone-system-with-intel-celeron-processor-n3150-208ghz-2hdmi-1276.html

$110 as barebones (add disk and RAM) or $141 with 2GB RAM and 16GB SSD.

Considering you guys don't have a hardware "partner" to push users towards, setting up some kind of supported hardware FAQ or forum would sure be handy.  There's a lot of nutty hardware out there in the direct-from-China market, including stuff like these multi-LAN boxes:

http://www.aliexpress.com/store/product/4-Gigabit-Hardware-Firewall-Router-BYPASS-Function-ROS-Security-Network-Server-IN-R24S/1501885_32452275008.html