Messages

Hi,

after running update from 20.1 to 20.7 and after that 20.7.1 firewall started behaving differently.

1. DNS stopped working
2. strange logs from unrelated filter rules were assigned to other ip ranges
3. ping ips e.g. 8.8.8.8 from lan stopped working event when firewall iteself was connected to internet, with working dns
4/ firewall log stopped updating right after reboot
- after digging through logs I found corrupted filter.log, after last timestamp which corelated with last line in web gui, thera was binary data in txt log file

After dissecting backup xml and manual parcial application of working parts I found offending section, which was filter rules.
Even after manualy adding basic vlan rules, the firewall again stopped updating in web gui

Running in Qotom appliance e.g. x64 pc

working and not working diff:

Code Select Expand


--- /conf/backup/config-1598529328.7681.xml 2020-08-27 13:55:28.768757000 +0200
+++ /conf/config.xml 2020-08-27 18:03:17.561776000 +0200
@@ -1205,162 +1205,6 @@
         <any/>
       </destination>
     </rule>
-    <rule>
-      <type>pass</type>
-      <interface>lan</interface>
-      <ipprotocol>inet</ipprotocol>
-      <statetype>keep state</statetype>
-      <descr>vlan 90 KaresWIfi to lan allow</descr>
-      <direction>in</direction>
-      <log>1</log>
-      <quick>1</quick>
-      <source>
-        <network>opt2ip</network>
-      </source>
-      <destination>
-        <network>lan</network>
-      </destination>
-      <updated>
-        <username>root@192.168.50.111</username>
-        <time>1598528372.929</time>
-        <description>/firewall_rules_edit.php made changes</description>
-      </updated>
-      <created>
-        <username>root@192.168.50.111</username>
-        <time>1598528372.929</time>
-        <description>/firewall_rules_edit.php made changes</description>
-      </created>
-    </rule>
-    <rule>
-      <type>pass</type>
-      <interface>lan</interface>
-      <ipprotocol>inet</ipprotocol>
-      <statetype>keep state</statetype>
-      <descr>vlan 80 Kares Guest o lan allow</descr>
-      <direction>in</direction>
-      <log>1</log>
-      <quick>1</quick>
-      <source>
-        <network>opt3ip</network>
-      </source>
-      <destination>
-        <network>lan</network>
-      </destination>
-      <updated>
-        <username>root@192.168.50.111</username>
-        <time>1598528401.006</time>
-        <description>/firewall_rules_edit.php made changes</description>
-      </updated>
-      <created>
-        <username>root@192.168.50.111</username>
-        <time>1598528401.006</time>
-        <description>/firewall_rules_edit.php made changes</description>
-      </created>
-    </rule>
-    <rule>
-      <type>pass</type>
-      <interface>opt2</interface>
-      <ipprotocol>inet</ipprotocol>
-      <statetype>keep state</statetype>
-      <descr>to vlan 90 allowed by list</descr>
-      <direction>in</direction>
-      <log>1</log>
-      <quick>1</quick>
-      <source>
-        <address>Allow_vlan_net</address>
-      </source>
-      <destination>
-        <any>1</any>
-      </destination>
-      <updated>
-        <username>root@192.168.50.111</username>
-        <time>1598528637.0106</time>
-        <description>/firewall_rules_edit.php made changes</description>
-      </updated>
-      <created>
-        <username>root@192.168.50.222</username>
-        <time>1598477577.3571</time>
-        <description>/firewall_rules_edit.php made changes</description>
-      </created>
-    </rule>
-    <rule>
-      <type>pass</type>
-      <interface>opt2</interface>
-      <ipprotocol>inet</ipprotocol>
-      <statetype>keep state</statetype>
-      <descr>lan to  vlan 90 allowed all</descr>
-      <direction>in</direction>
-      <log>1</log>
-      <quick>1</quick>
-      <source>
-        <network>lan</network>
-      </source>
-      <destination>
-        <any>1</any>
-      </destination>
-      <updated>
-        <username>root@192.168.50.111</username>
-        <time>1598528624.247</time>
-        <description>/firewall_rules_edit.php made changes</description>
-      </updated>
-      <created>
-        <username>root@192.168.50.111</username>
-        <time>1598528549.786</time>
-        <description>/firewall_rules_edit.php made changes</description>
-      </created>
-    </rule>
-    <rule>
-      <type>pass</type>
-      <interface>opt3</interface>
-      <ipprotocol>inet</ipprotocol>
-      <statetype>keep state</statetype>
-      <descr>to vlan 80 allowed by list</descr>
-      <direction>in</direction>
-      <log>1</log>
-      <quick>1</quick>
-      <source>
-        <address>Allow_vlan_net</address>
-      </source>
-      <destination>
-        <any>1</any>
-      </destination>
-      <updated>
-        <username>root@192.168.50.111</username>
-        <time>1598528652.0203</time>
-        <description>/firewall_rules_edit.php made changes</description>
-      </updated>
-      <created>
-        <username>root@192.168.50.111</username>
-        <time>1598528652.0203</time>
-        <description>/firewall_rules_edit.php made changes</description>
-      </created>
-    </rule>
-    <rule>
-      <type>pass</type>
-      <interface>opt3</interface>
-      <ipprotocol>inet</ipprotocol>
-      <statetype>keep state</statetype>
-      <descr>lan to  vlan 80 allowed all</descr>
-      <direction>in</direction>
-      <log>1</log>
-      <quick>1</quick>
-      <source>
-        <network>lan</network>
-      </source>
-      <destination>
-        <any>1</any>
-      </destination>
-      <updated>
-        <username>root@192.168.50.111</username>
-        <time>1598528609.4877</time>
-        <description>/firewall_rules_edit.php made changes</description>
-      </updated>
-      <created>
-        <username>root@192.168.50.111</username>
-        <time>1598528609.4877</time>
-        <description>/firewall_rules_edit.php made changes</description>
-      </created>
-    </rule>
   </filter>
   <rrd>
     <enable/>
@@ -1416,9 +1260,9 @@
     <column_count>2</column_count>
   </widgets>
   <revision>
-    <username>root@192.168.50.111</username>
-    <time>1598528782.2619</time>
-    <description>Gateways: removed gateway 0</description>
+    <username>(system)</username>
+    <time>1598472328.7505</time>
+    <description>/usr/local/opnsense/mvc/script/run_migrations.php made changes</description>
   </revision>
   <OPNsense>
     <monit version="1.0.8">
@@ -2559,8 +2403,8 @@
       </client>
     </wireguard>
     <Interfaces>
-      <vxlans/>
-      <loopbacks/>
+      <vxlans version="1.0.1"/>
+      <loopbacks version="1.0.0"/>
     </Interfaces>
     <MDNSRepeater version="1.0.0">
       <enabled>0</enabled>
@@ -2721,5 +2565,5 @@
       </timerange>
     </schedule>
   </schedules>
-  <staticroutes/>
+  <staticroutes version="1.0.0"/>
</opnsense>


Edit1: missing firewall log was caused by not starting/stopped syslog, see here:https://forum.opnsense.org/index.php?topic=18587.0
solution: upgrade to 20.7.2 or

Code Select Expand


pkg add -f https://pkg.opnsense.org/FreeBSD:12:amd64/20.7/misc/syslog-ng327-3.27.1_2.txz


Edit2: dns problems were caused by wrong IPV4 upstream GW for LAN,  in old config I had LAN_GW for lan, which was not working, changing the setting to auto-detect  corrected the blocking off all TCP

After clean install od 19.7.4 I could upgrade to 19.7.5 and everything worked (both on 19.7.4. and 19.7.5), also without 0.0.0.0/0 static route, in this post https://forum.opnsense.org/index.php?topic=11341.msg66960#msg66960 is diff of my notworking config and working config

OK, after clean install of 19.7.4 and importing my backuped config the issue reappears, so I did factory reset, and manual basic setup for WAN and LAN to get internet going, and now i can update my firewall...
After update i tried reimporting my old config - statis dhcp leasis, certs, vpn wg config etc and after full working setup i didd diff of those conf file, see below

Lines from original nonfuncional are shown as - (minus)

Code Select Expand


--- ./config-OPNsense.local-orig_nefunkcna.xml 2019-10-11 19:34:03.087841648 +0200
+++ ./config-OPNsense.local-openvpn_a_wg_certificates_rules.xml 2019-10-11 20:33:40.980630000 +0200
@@ -1,5 +1,6 @@
<?xml version="1.0"?>
<opnsense>
+  <trigger_initial_wizard/>
   <theme>opnsense</theme>
   <sysctl>
     <item>
@@ -52,15 +53,6 @@
     </item>
     <item>
       <descr>
-        Redirect attacks are the purposeful mass-issuing of ICMP type 5 packets. In a normal network, redirects
-        to the end stations should not be required. This option enables the NIC to drop all inbound ICMP redirect
-        packets without returning a response.
-      </descr>
-      <tunable>net.inet.icmp.drop_redirect</tunable>
-      <value>default</value>
-    </item>
-    <item>
-      <descr>
         This option turns off the logging of redirect packets because there is no limit and this could fill
         up your logs consuming your whole hard drive.
       </descr>
@@ -73,11 +65,6 @@
       <value>default</value>
     </item>
     <item>
-      <descr>Enable sending IPv4 redirects</descr>
-      <tunable>net.inet.ip.redirect</tunable>
-      <value>default</value>
-    </item>
-    <item>
       <descr>Enable sending IPv6 redirects</descr>
       <tunable>net.inet6.ip6.redirect</tunable>
       <value>default</value>
@@ -193,20 +180,37 @@
       <value>default</value>
     </item>
     <item>
-      <tunable>hint.sdhci_pci.0.disabled</tunable>
-      <value>1</value>
-      <descr>hint.sdhci_pci.0.disabled</descr>
+      <descr>Hide processes running as other groups</descr>
+      <tunable>security.bsd.see_other_gids</tunable>
+      <value>default</value>
+    </item>
+    <item>
+      <descr>Hide processes running as other users</descr>
+      <tunable>security.bsd.see_other_uids</tunable>
+      <value>default</value>
+    </item>
+    <item>
+      <descr>Enable/disable sending of ICMP redirects in response to IP packets for which a better,
+        and for the sender directly reachable, route and next hop is known.
+      </descr>
+      <tunable>net.inet.ip.redirect</tunable>
+      <value>0</value>
     </item>
     <item>
-      <tunable>hint.sdhci_pci.1.disabled</tunable>
+      <descr>
+        Redirect attacks are the purposeful mass-issuing of ICMP type 5 packets. In a normal network, redirects
+        to the end stations should not be required. This option enables the NIC to drop all inbound ICMP redirect
+        packets without returning a response.
+      </descr>
+      <tunable>net.inet.icmp.drop_redirect</tunable>
       <value>1</value>
-      <descr>hint.sdhci_pci.1.disabled</descr>
     </item>
   </sysctl>
   <system>
     <optimization>normal</optimization>
     <hostname>OPNsense</hostname>
     <domain>local</domain>
+    <dnsallowoverride>1</dnsallowoverride>
     <group>
       <name>admins</name>
       <description>System Administrators</description>
@@ -251,13 +255,18 @@
     <disablenatreflection>yes</disablenatreflection>
     <usevirtualterminal>1</usevirtualterminal>
     <disableconsolemenu>1</disableconsolemenu>
+    <disablevlanhwfilter>2</disablevlanhwfilter>
+    <disablechecksumoffloading>1</disablechecksumoffloading>
+    <disablesegmentationoffloading>1</disablesegmentationoffloading>
+    <disablelargereceiveoffloading>1</disablelargereceiveoffloading>
+    <ipv6allow/>
     <powerd_ac_mode>hadp</powerd_ac_mode>
     <powerd_battery_mode>hadp</powerd_battery_mode>
     <powerd_normal_mode>hadp</powerd_normal_mode>
     <bogons>
       <interval>monthly</interval>
     </bogons>
-    <kill_states>1</kill_states>
+    <kill_states/>
     <backupcount>60</backupcount>
     <crypto_hardware>aesni</crypto_hardware>
     <pf_share_forward>1</pf_share_forward>
@@ -294,14 +303,10 @@
     <dns6gw>none</dns6gw>
     <dns7gw>none</dns7gw>
     <dns8gw>none</dns8gw>
-    <rulesetoptimization>basic</rulesetoptimization>
-    <maximumstates/>
-    <maximumfrags/>
-    <aliasesresolveinterval/>
-    <maximumtableentries>500000</maximumtableentries>
-    <prefer_ipv4>1</prefer_ipv4>
+    <serialspeed>115200</serialspeed>
+    <primaryconsole>video</primaryconsole>
     <firmware>
-      <mirror>https://opnsense.ieji.de</mirror>
+      <plugins>os-debug,os-clamav,os-iperf,os-telegraf,os-wireguard</plugins>
     </firmware>
   </system>
   <interfaces>
@@ -546,28 +551,7 @@
     </lan>
   </dhcpd>
   <unbound>
-    <dnssecstripped>1</dnssecstripped>
-    <domainoverrides/>
-    <custom_options/>
     <enable>1</enable>
-    <regdhcp>1</regdhcp>
-    <noreglladdr6>1</noreglladdr6>
-    <regdhcpstatic>1</regdhcpstatic>
-    <txtsupport>1</txtsupport>
-    <cache_max_ttl/>
-    <cache_min_ttl/>
-    <incoming_num_tcp>10</incoming_num_tcp>
-    <infra_cache_numhosts>10000</infra_cache_numhosts>
-    <infra_host_ttl>900</infra_host_ttl>
-    <jostle_timeout>200</jostle_timeout>
-    <log_verbosity>3</log_verbosity>
-    <msgcachesize>4</msgcachesize>
-    <num_queries_per_thread>4096</num_queries_per_thread>
-    <outgoing_num_tcp>10</outgoing_num_tcp>
-    <unwanted_reply_threshold/>
-    <dnssec>1</dnssec>
-    <prefetch>1</prefetch>
-    <forwarding>1</forwarding>
   </unbound>
   <snmpd>
     <syslocation/>

@@ -2223,92 +1955,16 @@
     <IDS version="1.0.3">
       <rules/>
       <userDefinedRules/>
-      <files>
-        <file uuid="2715ab15-1bbd-4ee8-bd0f-dd8cbac2726d">
-          <filename>emerging-current_events.rules</filename>
-          <filter/>
-          <enabled>1</enabled>
-        </file>
-        <file uuid="ca45d589-ab2e-44ba-9756-f6d0d87bcaeb">
-          <filename>emerging-chat.rules</filename>
-          <filter/>
-          <enabled>1</enabled>
-        </file>
-        <file uuid="72335001-da84-4663-803f-45b4c30205f4">
-          <filename>emerging-attack_response.rules</filename>
-          <filter/>
-          <enabled>1</enabled>
-        </file>
-        <file uuid="6e68401f-2b7e-4b0b-bb6f-8188375371a3">
-          <filename>emerging-activex.rules</filename>
-          <filter/>
-          <enabled>1</enabled>
-        </file>
-        <file uuid="9c37c3c8-e587-4687-8a22-a5a718c1c052">
-          <filename>dshield.rules</filename>
-          <filter/>
-          <enabled>1</enabled>
-        </file>
-        <file uuid="4f2ac0d7-799d-4dbe-bc63-e7fa0d269d37">
-          <filename>drop.rules</filename>
-          <filter/>
-          <enabled>1</enabled>
-        </file>
-        <file uuid="518b9785-c982-40ee-b089-0b21c75a913e">
-          <filename>compromised.rules</filename>
-          <filter/>
-          <enabled>1</enabled>
-        </file>
-        <file uuid="1e13ef1d-a5f1-4310-9acd-167e92a72276">
-          <filename>ciarmy.rules</filename>
-          <filter/>
-          <enabled>1</enabled>
-        </file>
-        <file uuid="73080d3d-f9c6-407d-87e4-c08b7b1278d9">
-          <filename>botcc.portgrouped.rules</filename>
-          <filter/>
-          <enabled>1</enabled>
-        </file>
-        <file uuid="089d519e-58db-476c-b014-7d970c3d30d1">
-          <filename>botcc.rules</filename>
-          <filter/>
-          <enabled>1</enabled>
-        </file>
-        <file uuid="5d6d7f89-2c2d-40f9-9bbd-28f7d379a7b0">
-          <filename>abuse.ch.urlhaus.rules</filename>
-          <filter/>
-          <enabled>1</enabled>
-        </file>
-        <file uuid="ac3b3cda-f06b-418e-92b2-64f58964abc5">
-          <filename>abuse.ch.sslipblacklist.rules</filename>
-          <filter/>
-          <enabled>1</enabled>
-        </file>
-        <file uuid="4af3e6b8-96e5-4075-9b3d-bc5a35f91e64">
-          <filename>abuse.ch.sslblacklist.rules</filename>
-          <filter/>
-          <enabled>1</enabled>
-        </file>
-        <file uuid="710adde0-f7c4-4414-9f26-2a4f87b97709">
-          <filename>abuse.ch.feodotracker.rules</filename>
-          <filter/>
-          <enabled>1</enabled>
-        </file>
-        <file uuid="f6a857d4-f674-43bd-b6dc-209c3d1de498">
-          <filename>abuse.ch.dyre_sslipblacklist.rules</filename>
-          <filter/>
-          <enabled>1</enabled>
-        </file>
-      </files>
+      <files/>
       <fileTags/>
       <general>
-        <enabled>1</enabled>
-        <ips>1</ips>
-        <promisc>1</promisc>
+        <enabled>0</enabled>
+        <ips>0</ips>
+        <promisc>0</promisc>
         <interfaces>wan</interfaces>
         <homenet>192.168.0.0/16,10.0.0.0/8,172.16.0.0/12</homenet>
         <defaultPacketSize/>
-        <UpdateCron>aefe4747-196a-4558-bb1a-50aed2436c0d</UpdateCron>
+        <UpdateCron/>
         <AlertLogrotate>W0D23</AlertLogrotate>
         <AlertSaveLogs>4</AlertSaveLogs>
         <MPMAlgo>ac</MPMAlgo>

-  <crl/>
-  <staticroutes version="1.0.0">
-    <route uuid="5a1145ee-7c5a-4cc2-8099-028324c0b997">
-      <network>1.1.1.1/0</network>
-      <gateway>WAN_DHCP</gateway>
-      <descr>check firmware workaround</descr>
-      <disabled>1</disabled>
-    </route>
-    <route uuid="0694bc6c-eea2-4a3b-b917-eb455cf5bd5f">
-      <network>1.1.1.1/0</network>
-      <gateway>LAN_GWv4</gateway>
-      <descr>check firmware workaround</descr>
-      <disabled>1</disabled>
-    </route>
-    <route uuid="080a2466-96d5-4cc4-a61d-0e5a203fb475">
-      <network>8.8.8.8/0</network>
-      <gateway>LAN_GWv4</gateway>
-      <descr>check firmware workaround</descr>
-      <disabled>1</disabled>
-    </route>
-    <route uuid="26c5234b-3f89-4b72-a62c-d8df3e49af55">
-      <network>8.8.8.8/0</network>
-      <gateway>WAN_DHCP</gateway>
-      <descr>check firmware workaround</descr>
-      <disabled>1</disabled>
-    </route>
-    <route uuid="75b0d450-e839-4a57-95cd-13345480f017">
-      <network>0.0.0.0/0</network>
-      <gateway>WAN_DHCP</gateway>
-      <descr/>
-      <disabled>0</disabled>
-    </route>
+  <staticroutes>
+    <route/>
   </staticroutes>
</opnsense>



Hi, it looks like similar problem I am getting,

problem looks like localhost / firewall host itself cannot reach internet, some users reported succesfull resolution by adding static route 0.0.0.0/0 to Wan interface - this resolves dns issues.

In my case ping works but connection timeouts - fetch, curl, pkg can connect to repository web but that is all, after that the connection is stalled, which is my case also, seems that it is also present in 19.7.4, (did clean install yesterday) and the issue persist, wil do rule after rule manual setup after factory reset today and will see...

Another similar topics, from older version 19.1 too




[url]https://github.com/opnsense/update/issues/49]https://forum.opnsense.org/index.php?topic=11341.0[u/rl]


[url]https://github.com/opnsense/update/issues/49

and

https://forum.opnsense.org/index.php?topic=10201.0

and older

https://forum.opnsense.org/index.php?topic=10843.0

I have similar problem, for other examples and troubleshooting see:

https://forum.opnsense.org/index.php?topic=11341.0

and

https://github.com/opnsense/update/issues/49

and

https://forum.opnsense.org/index.php?topic=10201.0

and older

https://forum.opnsense.org/index.php?topic=10843.0

I have the same/ similar problem, I am stuck on 19.1 version and cannot update, while testing I first got dns erros, by adding 0.0.0.0/0 route per sugestions  on this forum https://forum.opnsense.org/index.php?topic=11341.msg56947#msg56947, ping can now resolve hosts from firewall localhost, but fetch, curl all stall/freeze after connecting, fetch with -vvv options stalls at resolving github.com:433, curl with -vvv option can connect to correct address (firewall log shows connection) but will stall after that.

Code Select Expand


# curl --tcp-nodelay -4 -vvvvv -o kernel-19.7.3-amd64.txz -k https://pkg.opnsense.org/FreeBSD:11:amd64/19.7/sets/kernel-19.7.3-amd64.txz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 212.32.245.132...
* TCP_NODELAY set
  0     0    0     0    0     0      0      0 --:--:--  0:01:14 --:--:--     0* connect to 212.32.245.132 port 443 failed: Operation timed out
* Failed to connect to pkg.opnsense.org port 443: Operation timed out
* Closing connection 0
curl: (7) Failed to connect to pkg.opnsense.org port 443: Operation timed out



Code Select Expand


# fetch -vvv https://pkg.opnsense.org/FreeBSD:11:amd64/19.7/sets/kernel-19.7.3-amd64.txz
resolving server address: pkg.opnsense.org:443
failed to connect to pkg.opnsense.org:443
fetch: https://pkg.opnsense.org/FreeBSD:11:amd64/19.7/sets/kernel-19.7.3-amd64.txz: No route to host


Ping to both host (opnsense pkg mirror and github, google etc.) is working but tcp connection stalls, no log in firewall...

Code Select Expand


# ping github.com
PING github.com (140.82.118.3): 56 data bytes
64 bytes from 140.82.118.3: icmp_seq=0 ttl=53 time=33.061 ms
64 bytes from 140.82.118.3: icmp_seq=1 ttl=53 time=33.093 ms
^C
--- github.com ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 33.061/33.077/33.093/0.016 ms

# ping pkg.opnsense.org
PING pkg.opnsense.org (212.32.245.132): 56 data bytes
64 bytes from 212.32.245.132: icmp_seq=0 ttl=53 time=42.606 ms
64 bytes from 212.32.245.132: icmp_seq=1 ttl=53 time=43.134 ms
64 bytes from 212.32.245.132: icmp_seq=2 ttl=53 time=42.611 ms
^C
--- pkg.opnsense.org ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 42.606/42.784/43.134/0.248 ms


At first I tried the bootstrap script (https://github.com/opnsense/update/blob/master/bootstrap/opnsense-bootstrap.sh), but after connection problems the script deleted my pkg config...without verifying  if download succeded...