Random freeze of the firewall

[mimino]

Looking at past posts it seems that the newest driver version (1.95) is or was OKish. Do you know if there is a way to restart the ethernet interface without rebooting? I can watchdog the interfaces and if something is wrong do a "service netif restart" or similar. That would be faster than rebooting... My freebsd skills are close to nill.

I would probably try to determine who's at fault first, i.e. connect a monitor and see what's happening. I've seen drivers causing hard locks, the whole system freezes, in which case no ehternet restart can help or can even be performed.

[guest19757]

I agree, Hard lock, kernel panic, does FreeBSB panic? That's a hard lock

[golfvert]

Thank you for your answers.
The "small" issue is that the opnsense unit has only HDMI output and my TV is 10 meters away!! So moving the 43" screen is going to be interesting. I should buy a basic HDMI monitor

[chemlud]

I had a similar issue yesterday, box was up for 12 days or so. Suddenly GUI not reachable, no traffic. I connected via serial console and saw this after entering reboot:

Code: [Select]

reboot
Password:
ovpns5: link state changed to DOWN
ovpns2: link state changed to DOWN
ovpns1: link state changed to DOWN
em0: link state changed to DOWN
250.176545 [ 663] netmap_obj_free           ouch, double free on buffer 281
250.183112 [ 663] netmap_obj_free           ouch, double free on buffer 3144
250.189915 [ 663] netmap_obj_free           ouch, double free on buffer 151
250.196622 [ 663] netmap_obj_free           ouch, double free on buffer 3704
250.203417 [ 663] netmap_obj_free           ouch, double free on buffer 369
250.210126 [ 663] netmap_obj_free           ouch, double free on buffer 3598
250.216927 [ 663] netmap_obj_free           ouch, double free on buffer 767
250.223635 [ 663] netmap_obj_free           ouch, double free on buffer 52
250.230258 [ 663] netmap_obj_free           ouch, double free on buffer 3178
250.237057 [ 663] netmap_obj_free           ouch, double free on buffer 146
250.243765 [ 663] netmap_obj_free           ouch, double free on buffer 270
250.250482 [ 663] netmap_obj_free           ouch, double free on buffer 3143
250.257272 [ 663] netmap_obj_free           ouch, double free on buffer 720
250.263983 [ 663] netmap_obj_free           ouch, double free on buffer 831
250.270693 [ 663] netmap_obj_free           ouch, double free on buffer 641
250.277405 [ 663] netmap_obj_free           ouch, double free on buffer 3791
250.284201 [ 663] netmap_obj_free           ouch, double free on buffer 3830
250.291001 [ 663] netmap_obj_free           ouch, double free on buffer 3315
250.297800 [ 663] netmap_obj_free           ouch, double free on buffer 292
250.304516 [ 663] netmap_obj_free           ouch, double free on buffer 3347
250.311306 [ 663] netmap_obj_free           ouch, double free on buffer 3817
250.318103 [ 663] netmap_obj_free           ouch, double free on buffer 3636
250.324906 [ 663] netmap_obj_free           ouch, double free on buffer 699
250.331615 [ 663] netmap_obj_free           ouch, double free on buffer 3230
250.338413 [ 663] netmap_obj_free           ouch, double free on buffer 3189
250.345209 [ 663] netmap_obj_free           ouch, double free on buffer 735
250.351924 [ 663] netmap_obj_free           ouch, double free on buffer 3809
250.358714 [ 663] netmap_obj_free           ouch, double free on buffer 550
250.365429 [ 663] netmap_obj_free           ouch, double free on buffer 3691
250.372221 [ 663] netmap_obj_free           ouch, double free on buffer 3716
250.379020 [ 663] netmap_obj_free           ouch, double free on buffer 3840
250.385822 [ 663] netmap_obj_free           ouch, double free on buffer 767
250.392526 [ 663] netmap_obj_free           ouch, double free on buffer 921
250.399243 [ 663] netmap_obj_free           ouch, double free on buffer 3087
250.406041 [ 663] netmap_obj_free           ouch, double free on buffer 4004
250.412841 [ 663] netmap_obj_free           ouch, double free on buffer 131
250.419544 [ 663] netmap_obj_free           ouch, double free on buffer 460
250.426259 [ 663] netmap_obj_free           ouch, double free on buffer 3772
250.433059 [ 663] netmap_obj_free           ouch, double free on buffer 84
250.439678 [ 663] netmap_obj_free           ouch, double free on buffer 3697
250.446476 [ 663] netmap_obj_free           ouch, double free on buffer 833
250.453191 [ 663] netmap_obj_free           ouch, double free on buffer 3982
250.459988 [ 663] netmap_obj_free           ouch, double free on buffer 4056
250.466791 [ 663] netmap_obj_free           ouch, double free on buffer 463
250.473522 [ 663] netmap_obj_free           ouch, double free on buffer 1332
250.480303 [ 663] netmap_obj_free           ouch, double free on buffer 2317
250.487099 [ 663] netmap_obj_free           ouch, double free on buffer 2304
250.493894 [ 663] netmap_obj_free           ouch, double free on buffer 2181
250.500694 [ 663] netmap_obj_free           ouch, double free on buffer 1472
250.507492 [ 663] netmap_obj_free           ouch, double free on buffer 1591
250.514286 [ 663] netmap_obj_free           ouch, double free on buffer 1572
250.521086 [ 663] netmap_obj_free           ouch, double free on buffer 2161
250.527890 [ 663] netmap_obj_free           ouch, double free on buffer 2543
250.534684 [ 663] netmap_obj_free           ouch, double free on buffer 1194
250.541481 [ 663] netmap_obj_free           ouch, double free on buffer 1577
250.548282 [ 663] netmap_obj_free           ouch, double free on buffer 2246
250.555082 [ 663] netmap_obj_free           ouch, double free on buffer 2746
250.561882 [ 663] netmap_obj_free           ouch, double free on buffer 3039
250.568681 [ 663] netmap_obj_free           ouch, double free on buffer 2541
250.575480 [ 663] netmap_obj_free           ouch, double free on buffer 2543
250.582278 [ 663] netmap_obj_free           ouch, double free on buffer 2277
250.589074 [ 663] netmap_obj_free           ouch, double free on buffer 2119
250.595875 [ 663] netmap_obj_free           ouch, double free on buffer 1182
250.602672 [ 663] netmap_obj_free           ouch, double free on buffer 2454
250.609473 [ 663] netmap_obj_free           ouch, double free on buffer 2218
250.616272 [ 663] netmap_obj_free           ouch, double free on buffer 1362
250.623074 [ 663] netmap_obj_free           ouch, double free on buffer 1727
250.629873 [ 663] netmap_obj_free           ouch, double free on buffer 2247
250.636670 [ 663] netmap_obj_free           ouch, double free on buffer 2089
250.643471 [ 663] netmap_obj_free           ouch, double free on buffer 2810
250.650267 [ 663] netmap_obj_free           ouch, double free on buffer 2526
250.657064 [ 663] netmap_obj_free           ouch, double free on buffer 1827
250.663864 [ 663] netmap_obj_free           ouch, double free on buffer 2414
250.670667 [ 663] netmap_obj_free           ouch, double free on buffer 2763
250.677475 [ 663] netmap_obj_free           ouch, double free on buffer 2782
250.684264 [ 663] netmap_obj_free           ouch, double free on buffer 1612
250.691061 [ 663] netmap_obj_free           ouch, double free on buffer 1216
250.697862 [ 663] netmap_obj_free           ouch, double free on buffer 3007
250.704659 [ 663] netmap_obj_free           ouch, double free on buffer 1713
250.711459 [ 663] netmap_obj_free           ouch, double free on buffer 3018
250.718252 [ 663] netmap_obj_free           ouch, double free on buffer 1263
250.725051 [ 663] netmap_obj_free           ouch, double free on buffer 2392
250.731854 [ 663] netmap_obj_free           ouch, double free on buffer 1553
250.738651 [ 663] netmap_obj_free           ouch, double free on buffer 1454
250.745451 [ 663] netmap_obj_free           ouch, double free on buffer 2290
250.752249 [ 663] netmap_obj_free           ouch, double free on buffer 1286
250.759049 [ 663] netmap_obj_free           ouch, double free on buffer 2353
250.765843 [ 663] netmap_obj_free           ouch, double free on buffer 2474
250.772644 [ 663] netmap_obj_free           ouch, double free on buffer 1398
250.779447 [ 663] netmap_obj_free           ouch, double free on buffer 2885
250.786246 [ 663] netmap_obj_free           ouch, double free on buffer 2970
250.793047 [ 663] netmap_obj_free           ouch, double free on buffer 1057
250.799846 [ 663] netmap_obj_free           ouch, double free on buffer 2967
250.806645 [ 663] netmap_obj_free           ouch, double free on buffer 2441
250.813442 [ 663] netmap_obj_free           ouch, double free on buffer 1364
250.820240 [ 663] netmap_obj_free           ouch, double free on buffer 3069
250.827041 [ 663] netmap_obj_free           ouch, double free on buffer 1241
250.833838 [ 663] netmap_obj_free           ouch, double free on buffer 1807
250.840636 [ 663] netmap_obj_free           ouch, double free on buffer 2247
250.847436 [ 663] netmap_obj_free           ouch, double free on buffer 2199
250.854237 [ 663] netmap_obj_free           ouch, double free on buffer 2418
250.861038 [ 663] netmap_obj_free           ouch, double free on buffer 1808
250.867828 [ 663] netmap_obj_free           ouch, double free on buffer 1833
250.874629 [ 663] netmap_obj_free           ouch, double free on buffer 2054
250.881431 [ 663] netmap_obj_free           ouch, double free on buffer 1700
250.888231 [ 663] netmap_obj_free           ouch, double free on buffer 2508
250.895027 [ 663] netmap_obj_free           ouch, double free on buffer 2889
250.901828 [ 663] netmap_obj_free           ouch, double free on buffer 1219
250.908623 [ 663] netmap_obj_free           ouch, double free on buffer 2280
250.915421 [ 663] netmap_obj_free           ouch, double free on buffer 1131
250.922220 [ 663] netmap_obj_free           ouch, double free on buffer 793
250.928933 [ 663] netmap_obj_free           ouch, double free on buffer 280
250.935643 [ 663] netmap_obj_free           ouch, double free on buffer 4078
250.942444 [ 663] netmap_obj_free           ouch, double free on buffer 557
250.949150 [ 663] netmap_obj_free           ouch, double free on buffer 3556
250.955950 [ 663] netmap_obj_free           ouch, double free on buffer 366
250.962664 [ 663] netmap_obj_free           ouch, double free on buffer 3728
250.969457 [ 663] netmap_obj_free           ouch, double free on buffer 741
250.976172 [ 663] netmap_obj_free           ouch, double free on buffer 3855
250.982970 [ 663] netmap_obj_free           ouch, double free on buffer 97
250.989591 [ 663] netmap_obj_free           ouch, double free on buffer 669
250.996305 [ 663] netmap_obj_free           ouch, double free on buffer 3697
251.003100 [ 663] netmap_obj_free           ouch, double free on buffer 558
251.009816 [ 663] netmap_obj_free           ouch, double free on buffer 991
251.016525 [ 663] netmap_obj_free           ouch, double free on buffer 125
251.023242 [ 663] netmap_obj_free           ouch, double free on buffer 3571
251.030040 [ 663] netmap_obj_free           ouch, double free on buffer 680
251.036758 [ 663] netmap_obj_free           ouch, double free on buffer 4036
251.043558 [ 663] netmap_obj_free           ouch, double free on buffer 699
251.050271 [ 663] netmap_obj_free           ouch, double free on buffer 3962
251.057071 [ 663] netmap_obj_free           ouch, double free on buffer 3901
251.063870 [ 663] netmap_obj_free           ouch, double free on buffer 62
251.070497 [ 663] netmap_obj_free           ouch, double free on buffer 36
251.077124 [ 663] netmap_obj_free           ouch, double free on buffer 218
251.083840 [ 663] netmap_obj_free           ouch, double free on buffer 3931
251.090637 [ 663] netmap_obj_free           ouch, double free on buffer 3737
251.097441 [ 663] netmap_obj_free           ouch, double free on buffer 147
251.104156 [ 663] netmap_obj_free           ouch, double free on buffer 687
251.110868 [ 663] netmap_obj_free           ouch, double free on buffer 3823
251.117670 [ 663] netmap_obj_free           ouch, double free on buffer 131
251.124385 [ 663] netmap_obj_free           ouch, double free on buffer 420
251.131096 [ 663] netmap_obj_free           ouch, double free on buffer 562
251.137811 [ 663] netmap_obj_free           ouch, double free on buffer 3521
251.144617 [ 663] netmap_obj_free           ouch, double free on buffer 224
251.151325 [ 663] netmap_obj_free           ouch, double free on buffer 3946
251.158126 [ 663] netmap_obj_free           ouch, double free on buffer 334
251.164839 [ 663] netmap_obj_free           ouch, double free on buffer 3385
251.171640 [ 663] netmap_obj_free           ouch, double free on buffer 125
251.178355 [ 663] netmap_obj_free           ouch, double free on buffer 617
251.185070 [ 663] netmap_obj_free           ouch, double free on buffer 332
251.191786 [ 663] netmap_obj_free           ouch, double free on buffer 3446
251.198587 [ 663] netmap_obj_free           ouch, double free on buffer 99
251.205212 [ 663] netmap_obj_free           ouch, double free on buffer 3552
251.212011 [ 663] netmap_obj_free           ouch, double free on buffer 3659
251.218812 [ 663] netmap_obj_free           ouch, double free on buffer 30
251.225442 [ 663] netmap_obj_free           ouch, double free on buffer 3904
251.232239 [ 663] netmap_obj_free           ouch, double free on buffer 785
251.238956 [ 663] netmap_obj_free           ouch, double free on buffer 537
251.245670 [ 663] netmap_obj_free           ouch, double free on buffer 3082
251.252470 [ 663] netmap_obj_free           ouch, double free on buffer 24
251.259095 [ 663] netmap_obj_free           ouch, double free on buffer 1026
251.265897 [ 663] netmap_obj_free           ouch, double free on buffer 3332
251.272701 [ 663] netmap_obj_free           ouch, double free on buffer 561
251.279412 [ 663] netmap_obj_free           ouch, double free on buffer 3699
251.286214 [ 663] netmap_obj_free           ouch, double free on buffer 1012
251.293015 [ 663] netmap_obj_free           ouch, double free on buffer 3995
251.299818 [ 663] netmap_obj_free           ouch, double free on buffer 3104
251.306624 [ 663] netmap_obj_free           ouch, double free on buffer 951
251.313336 [ 663] netmap_obj_free           ouch, double free on buffer 3262
251.320133 [ 663] netmap_obj_free           ouch, double free on buffer 33
...
254.995333 [ 663] netmap_obj_free           ouch, double free on buffer 27916
255.002226 [ 663] netmap_obj_free           ouch, double free on buffer 34416


(had to skip some thousand lines...)

...afterwards it rebooted.

The minute before I recognized that the Suricata log (alerts still up and running!) had stopped a day ago (but update had happend), after reboot the log was up-to-date again.

[golfvert]

Hello,
I have tried a different solution! I have replaced the box with a spare one I had which is (luckily) using Intel Ethernet interfaces. I have the exact same config.  I will see how it goes.
GV

[chemlud]

...my box has 4 igb and 1 em (WAN) interfaces...

[chemlud]

Mine is running Surricata IPS, what about the others?

[guest19757]

Hello,

I only have Suricata in alert mode, so far no freezes observed here with 2 interfaces WAN and LAN. As I mentioned earlier, Suricata IPS using 'inline mode', is not well supported with virtio drivers. I'm not yet finished configuring Suricata atm to enable IPS.

Regards

[chemlud]

her the sense is real-world, not virtual...

[golfvert]

I am almost running the bare minimum (only dhcpv4 and telegraf) on a physical box, now using Intel ethernet.
So far so good. 3 days up and running. I will wait for another 10 days (I never had 2 weeks without freeze) and then add one by one additional services (suricata...) and will report back.

[chemlud]

Dunno if related, but Surricata on 18.7.9 stopped logging (not alerting) on 03-JAN and resumed logging only after reboot for 18.7.10

Code: [Select]

Jan 11 09:36:45 suricata: [100203] <Notice> -- This is Suricata version 4.1.2 RELEASE
Jan 11 09:34:23 suricata: [100211] <Notice> -- Stats for 'igb2+': pkts: 74461884, drop: 0 (0.00%), invalid chksum: 0
Jan 11 09:34:23 suricata: [100211] <Notice> -- Stats for 'igb2': pkts: 38273808, drop: 0 (0.00%), invalid chksum: 0
Jan 11 09:34:23 suricata: [100211] <Notice> -- Stats for 'igb1+': pkts: 7547098, drop: 0 (0.00%), invalid chksum: 0
Jan 11 09:34:23 suricata: [100211] <Notice> -- Stats for 'igb1': pkts: 19380430, drop: 0 (0.00%), invalid chksum: 0
Jan 11 09:34:23 suricata: [100211] <Notice> -- Stats for 'igb0+': pkts: 41225338, drop: 0 (0.00%), invalid chksum: 0
Jan 11 09:34:23 suricata: [100211] <Notice> -- Stats for 'igb0': pkts: 20916919, drop: 0 (0.00%), invalid chksum: 0
Jan 11 09:34:23 suricata: [100211] <Notice> -- Stats for 'em0+': pkts: 57803719, drop: 0 (0.00%), invalid chksum: 0
Jan 11 09:34:23 suricata: [100211] <Notice> -- Stats for 'em0': pkts: 101705617, drop: 0 (0.00%), invalid chksum: 53
Jan 11 09:34:21 suricata: [100211] <Notice> -- Signal Received. Stopping engine.
Jan 3 22:38:20 suricata: [100211] <Notice> -- all 17 packet processing threads, 4 management threads initialized, engine started.
Jan 3 22:37:45 suricata: [100211] <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "drop http $HOME_NET any -> $EXTERNAL_NET any (msg:"URLhaus Known malware download URL detected"; flow:established,from_client; content:"GET"; http_method; content:"/url=http://skolnickassoc.com/us/clients/12_18|26|data=02|01||ab83990e8fdf459eec1208d65b94410c|4157b39d533a41f78314898c4d2ff33b|0|0|636797089635556280|26|sdata=z4owmgexijowl/f/5lzprvhbwxntiqnowpzesax1bmq=|26|reserved=0"; http_uri; depth:218; isdataat:!1,relative; content:"na01.safelinks.protection.outlook.com"; http_host; depth:37; isdataat:!1,relative; metadata:created_at 2018_12_06; reference:url, urlhaus.abuse.ch/url/90467/; classtype:trojan-activity;sid:80953567; rev:1;)^M" from file /usr/local/etc/suricata/opnsense.rules/abuse.ch.urlhaus.rules at line 10455
Jan 3 22:37:45
Jan 3 22:37:40 suricata: [100211] <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "drop http $HOME_NET any -> $EXTERNAL_NET any (msg:"URLhaus Known malware download URL detected"; flow:established,from_client; content:"GET"; http_method; content:"/ucid=1jpbpdcslphmb1fsao4mjhnk7jdopxawk|26|data=02|01|taison.tongbram@non.schneider-electric.com|5f3ba8dcd3ae43e07a3b08d65f77f329|6e51e1adc54b4b39b5980ffe9ae68fef|0|0|636801366149796273|26|sdata=3aqqim7nvgzilbiuwfxr5fmqtsixnarr+wnqogx6aoy=|26|reserved=0"; http_uri; depth:253; isdataat:!1,relative; content:"docs.google.com"; http_host; depth:15; isdataat:!1,relative; metadata:created_at 2018_12_11; reference:url, urlhaus.abuse.ch/url/93192/; classtype:trojan-activity;sid:80956292; rev:1;)^M" from file /usr/local/etc/suricata/opnsense.rules/abuse.ch.urlhaus.rules at line 7772
Jan 3 22:37:40
Jan 3 22:37:37 suricata: [100211] <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "drop http $HOME_NET any -> $EXTERNAL_NET any (msg:"URLhaus Known malware download URL detected"; flow:established,from_client; content:"GET"; http_method; content:"/url=http://shipus.net/bfew-9mhkdwksydk1xh_uqduhmzs-bap|26|data=01|01|trevor.stone@nvoicepay.com|9f5aca7dba7f48c4f5f708d661d8473b|8ad0e60b834b4e40bdbd2b43fea3fa1f|0|26|sdata=ah5crxi3khfqcw6jpqhfbugjiryra0dugzcxblqaxmg=|26|reserved=0/"; http_uri; depth:233; isdataat:!1,relative; content:"na01.safelinks.protection.outlook.com"; http_host; depth:37; isdataat:!1,relative; metadata:created_at 2018_12_14; reference:url, urlhaus.abuse.ch/url/95303/; classtype:trojan-activity;sid:80958403; rev:1;)^M" from file /usr/local/etc/suricata/opnsense.rules/abuse.ch.urlhaus.rules at line 5766
Jan 3 22:37:37
Had a look in the general logs for what might have happened on 03-JAN 22 22:38, but in the GUI I only get the first some hundred log entries...
: