Traffic stopped several hours after upgrade to 20.7.2 with nm_rxsync_prologue fa

[dp]

Upgraded the firewalls last night to 20.7.2. HA setup with broke failover(another story/bug). All was well and traffic was flowing without issue. Then at 9:42 the firewall stopped passing traffic. My boss had to go pull the cables on the active firewall to force it to failover to the HA unit. At this point I am concerned that I will have the same failure in the backup unit.

The only plugins I am running are Sensei and Mail Backup. No IDS. The only information I can find on this entry is tied to netmap incompatibility with the NIC. The card is a four port HP card that reports as an Intel Pro 1000.

From the general logs:

2020-09-23T09:45:19   /flowd_aggregate.py[71076]   vacuum src_addr_000300.sqlite
2020-09-23T09:42:28   kernel   548.444820 [1787] netmap_ring_reinit called for em0 RX0
2020-09-23T09:42:28   kernel   548.444791 [1742] nm_rxsync_prologue em0 RX0: fail 'head < kring->nr_hwcur || head > kring->nr_hwtail' h 301 c 301 t 300 rh 301 rc 301 rt 300 hc 300 ht 300
2020-09-23T09:39:50   /flowd_aggregate.py[71076]   vacuum src_addr_details_086400.sqlite
2020-09-23T01:54:49   opnsense[46459]   /usr/local/etc/rc.filter_synchronize: Filter sync successfully completed with https://10.101.0.16/xmlrpc.php.
2020-09-23T01:54:48   kernel   arp: 00:24:81:7e:1c:c3 is using my IP address 164.106.234.141 on lagg1!

[dp]

Heard through another channel that apparently this is due to an issue in the kernel with netmap.

[Ricardo]

If I were you, I would be very careful using opnsense in corporate environment (a.k.a at work). Its very cool and nerdy to use it at home: if its breaks (as it does during major upgrades, and sometimes even during minor upgrades, but so is life), you reboot it or reinstall it, so not a big deal, and life goes on!
But if you may lose your job because you were careless not testing it properly in a test environment before upgrading your production device, well then its no longer just a hobby firewall.

[franco]

If I were Ricardo, I would subtly try to threaten people with the risk of losing their jobs. Just saying that this particular recommendation is very shitty. Please think twice before posting bullshit and instead try to help people. ¯\_(ツ)_/¯

As for "nm_rxsync_prologue" this is a Netmap message so you are using IPS or Sensei.

You can try to update to 20.7.3 and install the netmap test kernel on top:

# opnsense-update -kr 20.7.3-netmap
# opnsense-shell reboot


Cheers,
Franco

[dp]

Franco,

Yep, it was Sensei and a netmap issue. After a few tries and version upgrades it now looks stable and working good. Went several rounds with the Sensei folks doing some testing for them and debug logs and looks like all is well.

Ricardo,

The foolish man is one who hide the facts from management. I have the complete buy in from management as they are willing to take the risk of an outage in exchange for the savings we gain each year. As noted above we are also using Sensei in preparation for new features, namely traffic shaping by application. Although Sensei is a pay product at my size it is still a fraction of the cost of the other players in the market.

One may say that using open source is not wise in a production environment but it is so weird that I have had far more issues (many never resolved) from the big name ransom players. Canned answer, "We can't replicate that in our test environment". And then they go on to do everything but tell me I am imagining it even when they see it happen. How do you explain that to management for a company you are shelling out $50k plus a year for support!

Doug

[allebone]

Most of us are using opnsense in work environments here. Reverting is normally pretty easy. I think if you have an ha pair you can upgrade just 1 and wait a few days before upgrading the second ha in the pair if I am not mistaken.