21.1.3 - Firmware Audit - Health - get "checksum mismatch"

[tofflock]

I updated two firewalls this morning from 21.1.2 to 21.1.3.  One FW in France, one FW in UK.
Both updates appeared to go fine and both systems came back up and got their ipsec VPN connected.

I then ran the audit->health on both systems.
The French system reported everything ok.

The UK system reported checksum errors in 13 files from the python37-3.7.10 package.  The files were:

-rw-r-----  1 root  wheel  3397 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/io.cpython-37.pyc
-rw-r-----  1 root  wheel  29786 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/os.cpython-37.pyc
-rw-r-----  1 root  wheel  10417 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/posixpath.cpython-37.pyc
-rw-r-----  1 root  wheel  16538 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/site.cpython-37.pyc
-rw-r-----  1 root  wheel  4332 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/stat.cpython-37.pyc
-rw-r-----  1 root  wheel  37921 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/threading.cpython-37.pyc
-rw-r-----  1 root  wheel  3587 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/token.cpython-37.pyc
-rw-r-----  1 root  wheel  17819 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/tokenize.cpython-37.pyc
-rw-r-----  1 root  wheel  19610 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/traceback.cpython-37.pyc
-rw-r-----  1 root  wheel  8964 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/types.cpython-37.pyc
-rw-r-----  1 root  wheel  51017 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/typing.cpython-37.pyc
-rw-r-----  1 root  wheel  13824 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/warnings.cpython-37.pyc
-rw-r-----  1 root  wheel  19557 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/weakref.cpython-37.pyc

Note - all files in the same directory, and with the date & time from when I did the update to 21.1.3.

I kept copies of these files, but not the other 513 files in the same directory.

I then reinstalled the python37 package:

python37   3.7.10   111MiB   OPNsense   PSFL   Interpreted object-oriented programming language

see also

https://forum.opnsense.org/index.php?topic=21972.0

I then reran audit->health check again and this time there were no errors.  I got a listing of the files that had problems:

-rw-r--r--  1 root  wheel  3397 Mar  8 23:55 /usr/local/lib/python3.7/__pycache__/io.cpython-37.pyc
-rw-r--r--  1 root  wheel  29786 Mar  8 23:55 /usr/local/lib/python3.7/__pycache__/os.cpython-37.pyc
-rw-r--r--  1 root  wheel  10417 Mar  8 23:55 /usr/local/lib/python3.7/__pycache__/posixpath.cpython-37.pyc
-rw-r--r--  1 root  wheel  16538 Mar  8 23:55 /usr/local/lib/python3.7/__pycache__/site.cpython-37.pyc
-rw-r--r--  1 root  wheel  4332 Mar  8 23:55 /usr/local/lib/python3.7/__pycache__/stat.cpython-37.pyc
-rw-r--r--  1 root  wheel  37921 Mar  8 23:55 /usr/local/lib/python3.7/__pycache__/threading.cpython-37.pyc
-rw-r--r--  1 root  wheel  3587 Mar  8 23:56 /usr/local/lib/python3.7/__pycache__/token.cpython-37.pyc
-rw-r--r--  1 root  wheel  17819 Mar  8 23:56 /usr/local/lib/python3.7/__pycache__/tokenize.cpython-37.pyc
-rw-r--r--  1 root  wheel  19610 Mar  8 23:56 /usr/local/lib/python3.7/__pycache__/traceback.cpython-37.pyc
-rw-r--r--  1 root  wheel  8964 Mar  8 23:56 /usr/local/lib/python3.7/__pycache__/types.cpython-37.pyc
-rw-r--r--  1 root  wheel  51017 Mar  8 23:56 /usr/local/lib/python3.7/__pycache__/typing.cpython-37.pyc
-rw-r--r--  1 root  wheel  13824 Mar  8 23:56 /usr/local/lib/python3.7/__pycache__/warnings.cpython-37.pyc
-rw-r--r--  1 root  wheel  19557 Mar  8 23:56 /usr/local/lib/python3.7/__pycache__/weakref.cpython-37.pyc

The file lengths match between these two listings, however, the date stamps differ.  I was puzzled by the differences so decided to look for the "corruptions".  So far, I've analysed only the first file (io.cpython-37.pyc) and it contained 48 corrupted bytes:

!da 5a!1f 1e!1f 1e!da 5a!19 18!1a 19!1b 1a!1c 1b
!1e 1d!1f 1e!1f 1e!1f 1e!20 1f!19 18!1a 19!1b 1a
!1c 1b!1e 1d!1f 1e!1f 1e!1f 1e!20 1f!19 18!1a 19
!1b 1a!1c 1b!1e 1d!1f 1e!1f 1e!1f 1e!20 1f!1e 1d
!da 5a!1c 1b!da 5a!da 5a!1a 19!1d 1c!da 5a!22 20
!23 21!24 22!da 5a!25 23!1f 1e!1f 1e!1f 1e!20 1f

After each "!" there are two bytes.  The first byte is from the "good" file, the second byte is from the corrupted file.  There isn't a consistent corruption between the good byte & the bad.  The majority of errors are the effect of subtracting 1 from the good byte (NB that's not the same as a one-bit error in my book) before storing it.  However, there are a few instances where 2 is subtracted, and a few more where the error is the upper four bits.

I'm not happy about 48 byte errors in a single file, and there's another 12 files to analyse.

I thought about a problem in the package file, so I checked the python37-3.7.10.tgz package file on the mirrors.  There are 24 mirror sites listed.  20 of those would accept my request for a copy of the file.  19 of those files were identical.  The one that differed was from "fourdots.com" and was truncated such that I only got 0.5% of it.  I'm certain that my system didn't install that file because I would have had many more errors, and probably pkg would have barfed, anyway.  So, not a problem with the package.

Franco mentioned "filing system errors" in the reference above.  I would like to pin it down further.
I've wondered about a disk error - it's a small SSD (KINGSTON SMS200S330G mSATA) that's been running for just over 4 years & 4 months.

Does anyone know why the corruped files had today's date, whereas after I had reinstalled the package, the the files had the original dates?

I've just done another check, this time on the French firewall - which didn't produce any errors.  All the files in the relevant directory ( /usr/local/lib/python3.7/__pycache__ ) have exactly the same date+time stamp as the (fresh) UK ones if one compensates for the different timezones.  So I'm beginning to think that the errors happened while the files were being extracted from the package. 

But why?

I'm not expecting definitive answers, but if anyone has any hints or suggestions, I'd be interested to hear them.

Thanks for reading...

PeterF

P.S.  What did we do before the audit->health option appeared???  It certainly proved useful today 

[franco]

Glad you are enjoying the health audit

We only accept signed package updates so whatever lands on your disk is verified. The package manager itself calculates checksums on package creation time and these are the checksums that fail comparison here.

I am not sure if Python deliberately recompiles those files or if it's a disk corruption, but at least from the basics the corruption appears on the target system and not earlier (as witnessed by the reinstall providing a clean audit).


Cheers,
Franco

[JohnnyBeee]

I have the same errors:
python37-3.7.10: checksum mismatch for /usr/local/lib/python3.7/__pycache__/inspect.cpython-37.pyc
python37-3.7.10: checksum mismatch for /usr/local/lib/python3.7/__pycache__/mimetypes.cpython-37.pyc
python37-3.7.10: checksum mismatch for /usr/local/lib/python3.7/__pycache__/subprocess.cpython-37.pyc
python37-3.7.10: checksum mismatch for /usr/local/lib/python3.7/__pycache__/typing.cpython-37.pyc
python37-3.7.10: checksum mismatch for /usr/local/lib/python3.7/__pycache__/zipfile.cpython-37.pyc

I am not an expert, so my questions are simple:
Is this anything I should worry about? If yes, how can/should I fix this?

[AdSchellevis]

Python recompiles before startup, I'm not 100% sure why the standard package contains pyc files to be honest, usually these aren't part of the delivery as far as I know.

So at a fist glance I wouldn't be too worried, it might be something worth looking into on our end at a later moment. (either exclude from check or cleanse before packaging, the latter is likely more of an upstream question)


Best regards,

Ad