Messages

16

21.1 Legacy Series / Re: DNS doesn't work for quite a while (or at all) after rebooting

« on: June 13, 2021, 03:21:59 pm »

Manually saving in 'System: Settings: General' will force whatever DNS configuration you have to re-initialize, so it sounds like DNS isn't being correctly configured and initialized properly at boot.  To try and narrow down what's happening, perhaps you could try the following with Unbound.

• Configure 'Services: Unbound DNS: General'.  Defaults should be fine, but the most important options are likely
  
  • Enable: checked
  • Network Interfaces: All
  • Outgoing Network Interfaces: All
• Configure the system to use Unbound in 'System: Settings: General'
  
  • DNS servers: blank
  • Allow DNS server list to be overridden by DHCP/PPP on WAN: unchecked
  • Do not use the local DNS service as a nameserver for this system: unchecked

Then reboot the system, immediately start a shell session and check for the following

Is unbound running?

Code: [Select]

root@OPNsense:~ # ps auxwww | grep unbound
unbound 78629   0.0  1.2   99436 49424  -  Is   Mon20       0:17.84 /usr/local/sbin/unbound -c /var/unbound/unbound.conf
root    60108   0.0  0.1 1060900  3196  3  R+   21:08       0:00.00 grep unbound
If it's not running, check to see if the generated config file exists and contains something meaningful.  It should look something like the following

Code: [Select]

root@OPNsense:~ # head -n 20 /var/unbound/unbound.conf
##########################
# Unbound Configuration
##########################

##
# Server configuration
##
server:
chroot: /var/unbound
username: unbound
directory: /var/unbound
pidfile: /var/run/unbound.pid
root-hints: /var/unbound/root.hints
use-syslog: yes
port: 53
[...]


If it is running, will it respond to queries?

Code: [Select]

root@OPNsense:~ # drill @127.0.0.1 google.com
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 46171
;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; google.com.  IN      A

;; ANSWER SECTION:
google.com.     300     IN      A       172.217.17.110

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 1294 msec
;; SERVER: 127.0.0.1
;; WHEN: Sun Jun 13 21:09:59 2021
;; MSG SIZE  rcvd: 44

If it will respond to queries locally, the issue will likely be with firewall rules.
 

17

21.1 Legacy Series / Re: Apparently I am using 109% of my Disk Space?

« on: June 09, 2021, 05:45:11 am »

Digging into the code, flowd log files (`/var/log/flowd.log*`) appear to be managed by `/usr/local/opnsense/scripts/netflow/flowd_aggregate.py` which should be keeping a maximum of 10 files of 10 MB each.  I can't see any other process that tidies up these logs, so it's possible the daemon has fallen over and the rc script managing the process never properly recovered.

Manually running `/usr/local/opnsense/scripts/netflow/flush_all.sh all` should tidy things up.

18

21.1 Legacy Series / ERR_SSL_PROTOCOL_ERROR after reboot

« on: June 04, 2021, 03:54:57 pm »

I'm configuring a new OPNsense instance and have been rebooting the system quite frequently when testing.  Each time the system comes back up I'm faced with an `ERR_SSL_PROTOCOL_ERROR` when attempting to access the web interface.

Searching around for similar issues, I found that running

Code: [Select]

configctl webgui restart
from the console was enough to regain access to the web interface.  Granted I won't intentionally be rebooting the system very frequently.  But this is the last thing I'd want to be dealing with after an unexpected restart.

Can anyone shed some light on what's happening here?

19

21.1 Legacy Series / Re: Routing firewall traffic over VPN interfaces

« on: June 04, 2021, 03:31:21 pm »

After some more digging, the culprit seems to be (the confusingly named) Disable force gateway.  The description text ("Disable automatic rules which force local services to use the assigned interface gateway.") gives a better idea of what it does.

By default it's checked, meaning the firewall is restricted to only sending traffic through the default gateway.  Unchecking the option results in several rules being created which allow traffic egress through all interfaces with an upstream gateway.

Perhaps the name of the option could be changed.  Or, at the very least, make the firewall logging more obvious when this option is enabled.  Currently there doesn't seem to be any indication whatsoever that traffic is being blocked.

Now the next problem: a race condition between Unbound and OpenVPN/WireGuard where unbound.conf gets generated before the VPN interfaces come up...

20

21.1 Legacy Series / [Solved] Routing firewall traffic over VPN interfaces

« on: June 04, 2021, 08:38:13 am »

I'm in the process of setting up a new OPNsense installation and have so far configured

• several subnets on multiple VLANs
• a number of OpenVPN and WireGuard interfaces in client mode
• firewall and NAT rules to enable selective traffic routing for hosts in the VLAN subnets over the VPN links

For hosts on the various LAN segments, everything is working as it should.  The problem is getting OPNsense itself to use these interfaces for outbound traffic, with the specific use case of having Unbound use the WireGuard and OpenVPN interfaces for all outbound requests; something I have working in other pfSense installations.

As the traffic is originating from the firewall itself, no outbound NAT should be required.  However, when troubleshooting this issue, I tried creating NAT rules and it did not made a difference.  The firewall logs do not show any traffic explicitly being blocked, not even a 'default deny'.  I even turned on logging for every single firewall rule in the system, which only resulted in a wall of 'let out anything from firewall host itself' messages.

`pftop` shows the following in these scenarios:

1. Configuring Unbound to explicitly use the WAN interface, then running `drill google.com` on the firewall

Code: [Select]

udp      Out 127.0.0.1:58049          127.0.0.1:53               MULTIPLE:SINGLE       00:00:13  00:00:17        2      116
udp      Out <WAN_IP>:24639        198.97.190.53:53           MULTIPLE:SINGLE       00:00:08  00:00:22        2      802
udp      Out <WAN_IP>:62446        202.12.27.33:53            MULTIPLE:SINGLE       00:00:05  00:00:25        2     1181
udp      Out 127.0.0.1:36195          127.0.0.1:53               MULTIPLE:SINGLE       00:00:04  00:00:26        2      164
udp      Out 127.0.0.1:54392          127.0.0.1:53               MULTIPLE:SINGLE       00:00:37  00:00:00        2      136
udp      Out <WAN_IP>:29434        193.0.14.129:53            MULTIPLE:SINGLE       00:00:05  00:00:25        2     1181
udp      Out <WAN_IP>:4269         198.97.190.53:53           MULTIPLE:SINGLE       00:00:05  00:00:25        2      701
udp      Out <WAN_IP>:24497        192.112.36.4:53            MULTIPLE:SINGLE       00:00:05  00:00:25        2      701
udp      Out <WAN_IP>:50107        65.22.160.17:53            MULTIPLE:SINGLE       00:00:05  00:00:25        2      415

2. Configuring Unbound to explicitly use the the OpenVPN and WireGuard interfaces, then running `drill google.com` on the firewall

Code: [Select]

PR       DIR SRC                      DEST                             STATE                AGE       EXP     PKTS    BYTES
udp      Out <OVPN1_IP>:60017           193.0.14.129:53              SINGLE:NO_TRAFFIC   00:00:15  00:00:45        1       56
udp      Out <OVPN2_IP>:42628           192.5.5.241:53               SINGLE:NO_TRAFFIC   00:00:10  00:00:50        1       56
udp      Out <WG1_IP>:40007           192.33.4.12:53               SINGLE:NO_TRAFFIC   00:00:09  00:00:51        1       56
udp      Out <OVPN1_IP>:53387           199.7.91.13:53               SINGLE:NO_TRAFFIC   00:00:09  00:00:51        1       56
udp      Out <OVPN1_IP>:57877           199.7.83.42:53               SINGLE:NO_TRAFFIC   00:00:07  00:00:53        1       56
udp      Out <OVPN2_IP>:8751            199.7.83.42:53               SINGLE:NO_TRAFFIC   00:00:06  00:00:54        1       56
udp      Out <OVPN1_IP>:18903           199.9.14.201:53              SINGLE:NO_TRAFFIC   00:00:06  00:00:54        1       56
udp      Out <OVPN2_IP>:10430           192.5.5.241:53               SINGLE:NO_TRAFFIC   00:00:05  00:00:55        1       56
udp      Out <OVPN2_IP>:63408           192.5.5.241:53               SINGLE:NO_TRAFFIC   00:00:04  00:00:56        1       56
udp      Out <OVPN2_IP>:44057           202.12.27.33:53              SINGLE:NO_TRAFFIC   00:00:04  00:00:56        1       56
udp      Out <WG1_IP>:52863           198.97.190.53:53             SINGLE:NO_TRAFFIC   00:00:03  00:00:57        1       56
udp      Out <OVPN1_IP>:54187           202.12.27.33:53              SINGLE:NO_TRAFFIC   00:00:03  00:00:57        1       56
udp      Out <WG1_IP>:36030           192.58.128.30:53             SINGLE:NO_TRAFFIC   00:00:01  00:00:59        1       56
udp      Out <WG1_IP>:43964           192.203.230.10:53            SINGLE:NO_TRAFFIC   00:00:00  00:01:00        1       56
udp      Out <WG1_IP>:16896           198.41.0.4:53                SINGLE:NO_TRAFFIC   00:00:14  00:00:46        1       56
udp      Out <WG1_IP>:44421           198.41.0.4:53                SINGLE:NO_TRAFFIC   00:00:14  00:00:46        1       56
udp      Out <OVPN1_IP>:51771           192.58.128.30:53             SINGLE:NO_TRAFFIC   00:00:13  00:00:47        1       56
3. Running `curl -L http://1.1.1.1` on the firewall

Code: [Select]

tcp      Out <WAN_IP>:5919         1.1.1.1:80               FIN_WAIT_2:FIN_WAIT_2   00:00:04  00:01:26       10      938

4. Running `curl -L --interface ovpnc1 http://1.1.1.1` on the firewall

Code: [Select]

tcp      Out <OVPN_IP>:24788           1.1.1.1:80                 SYN_SENT:CLOSED       00:00:07  00:00:26        3      180

Have I overlooked a simple configuration option?  If so, I would really appreciate if someone could point me in the right direction.