Newbie install and configuration experience

[zenlord]

Dear all,

First of all, thank you for the OPNsense software - I have it more or less up and running by now, but as I have had some issues, I'd like to document them here. Maybe in my ignorance I have stumbled across a bug or two, and who knows maybe this leads to the bugs being squashed.

Secondly: The reason why I ordered a new PC Engines APU2C4 board and chose OPNsense is that I thought my Linksys EA3500 router had started dying on me: I received complaints of frequent drops in connections (http and sip), but today I must admit now that it seems that these drops have been caused by internal DHCP-issues: one of my devices used a fixed IP that was conflicting with a statically attributed address. Now this has been resolved, I don't see the connection drops anymore, so I hope I can mark the issue solved .

My setup: MODEM - ROUTER (only NAT) - SG200 SWITCH - internal network, including a Debian Jessie server with IPTABLES, FAIL2BAN, DHCPD and BIND9.

In this setup, changing routers should be very straightforward, but it proved not that straightforward:
1. Installation was painless (once I had the correct null modem cable) with the 'serial'-image on a USB stick and installation to an mSATA disk
2. Configuration of the Intel ethernet ports was easy: igb0=WAN=DHCPv4 + igb1=LAN=fixed IPv4
3. In the webgui, I spoofed the former Linksys MAC Address to receive my fixed IPv4 address on WAN
4. Again in the webgui, I setup port forwarding for ~15 destinations, carefully leaving the 'bind to firewall rule'-option unchanged
5. I setup disk cache as per the instructions

Still thinking I had solved the issue with the dropped packets, I was very satisfied: the old router configuration (NAT only) was manually copied to the new device, and everything was accessible. At that moment, I witnessed only:
* a constantly high CPU load (30%+, although a max of 5 low-traffic clients were active on the LAN)
* a hanging JS-script when opening the "Interfaces > LAN" -page
* the NTPD service failed quite often

24 hours later, the connection drops reared their ugly head again, and I moved the old router back into place, which seemed to solve the issue in the short term. Yesterday I found the issue, but I had made a few changes to the router:
1. I moved all DHCPD-rules to the OPNsense router and disabled the DHCPD on the Debian Jessie server
2. I reset the SG200-switch as well as some other switches
3. I unplugged a lot of cables from the switch
4. I removed the igb0-interface and moved WAN to igb2, to move it back 30 minutes later
5. I stopped spoofing the MAC address of the old router
6. I tried to login into all the managed switches and resolved the dual use of one IP address for two devices
7. I stopped the Intrusion Detection and WebProxy services
8. I rebooted the modem and the router

Then suddenly,
* the CPU load dropped to 1%-3%
* the packet loss was gone
* the NTPD service was constantly up
* the LAN-page did not have the issue with the hanging JS-script any longer

Today I learnt that, although the port forwarding rules were still 'there', the corresponding firewall rules were 'gone' - in the firewall logs all attempts to forward traffic to the local server were 'blocked by the default deny rule'. To change this, it did not suffice to change the port forward rule to set the corresponding firewall rule to 'pass' - nor did it help to add an explicit firewall rule to allow the specific traffic on a certain port. Only after removing all specific firewall rules AND port forwarding rules, I could re-add port forwarding rules with the expected consequence that such traffic would be allowed to pass through the firewall.

Firewalling and routing network traffic are not entirely new to me, but my knowledge/experience is quite high level. Still, if I may cautiously conclude:
1. the above would not have happened if I did not have any collisions on the internal network, but I think OPNsense 'sensed' these issues while it was not able (or willing to tell me what was wrong (or at least something was wrong - it even reported '0 collisions' and '0 packets lost'). If at all possible, this would be a great feature...
2. maybe I have messed up the NAT/Firewall rules myself by deleting the igb0/WAN interface, but given the consequences if the situation is restored (all rules have to be deleted and reconfigured anyway), wouldn't it be a useful action to automatically delete all NAT+FW rules that had been added to the network interface upon deletion of the network interface assignment?

Thank you for reading this long post - I hope my experiences with this distro can help making it even more newbie-proof

Kr,
Vincent