Messages

This was still bugging me enough that I went ahead and tracked down the call chain.

The filter update script imports dns.asyncresolver, which in turn imports trio (/usr/local/lib/python3.9/site-packages/trio/_core/_thread_cache.py).  Trio needs the pthread library, so it calls ctypes.util.find_library("pthread"), which in turn calls _findLib_gcc.

_findLib_gcc in /usr/local/lib/python3.9/ctypes/util.py is what ends up invoking the c compiler, it apparently tries to compile a test program with the library in question to determine if the library exists.  This does seem like unwanted behavior on a firewall, but I'm not 100% sure how to get rid of it given the call chain.

@meyergru I completely agree, but short of one of the maintainers / designers of the table update script chiming in (or someone digging through all the nested Python calls and classes) we simply won't know why it's done this way.  The only thing I can figure is it's a leftover from pfSense days, thus potentially a relic from when these machines were too slow to process the update for thousands of states outside of a compiled program.

To close the loop on this, I found the same thing on another OPNSense box (hung ld process) and spent a little time digging around.  I'm thinking this is a harmless side effect of the way OPNSense "compiles" the rules after an alias update, and the only reason I even saw the compiler call is because /tmp was full on both boxes.  That led to a zero byte output file and a hung linker.

In case anyone is interested, this is why I'm thinking it's probably harmless / normal internal system operation (and a bit of a cheat sheet for anyone else that might run across a similar oddity and want to dig deeper):

Code Select Expand

pargs 94768
94768:  ld.lld
argv[0]: /usr/bin/ld
argv[1]: --eh-frame-hdr
argv[2]: -dynamic-linker
argv[3]: /libexec/ld-elf.so.1
argv[4]: --enable-new-dtags
argv[5]: -o
argv[6]: /tmp/tmp1jcrcfb7
argv[7]: /usr/lib/crt1.o
argv[8]: /usr/lib/crti.o
argv[9]: /usr/lib/crtbegin.o
argv[10]: -L/usr/lib
argv[11]: -t
argv[12]: -lpthread
argv[13]: -lgcc
argv[14]: --as-needed
argv[15]: -lgcc_s
argv[16]: --no-as-needed
argv[17]: -lc
argv[18]: -lgcc
argv[19]: --as-needed
argv[20]: -lgcc_s
argv[21]: --no-as-needed
argv[22]: /usr/lib/crtend.o
argv[23]: /usr/lib/crtn.o
OK, that seems fairly normal....this is invoked by 'cc':

Code Select Expand

/usr/bin/cc -Wl,-t -o /tmp/tmp1jcrcfb7 -lpthread
So something is invoking the compiler in a manner where it is reading from stdin and writing to a temporary binary file.  Let's go up another level and look....

Code Select Expand

root 94216 ... /usr/local/bin/python3 /usr/local/opnsense/scripts/filter/update_tables.py (python3.9)

Code Select Expand

procstat -f 94216
  PID COMM                FD T V FLAGS    REF  OFFSET PRO NAME
94216 python3.9         text v r r-------   -       - -   /usr/local/bin/python3.9
94216 python3.9          cwd v d r-------   -       - -   /root
94216 python3.9         root v d r-------   -       - -   /
94216 python3.9            0 p - rw------   4       0 -   -
94216 python3.9            1 v c -w------   2       0 -   /dev/null
94216 python3.9            2 p - rw------   2       0 -   -
94216 python3.9            3 v r r-------  14    9567 -   /usr/local/sbin/pluginctl
94216 python3.9            4 v r rw------  14  512073 -   /conf/config.xml
94216 python3.9            5 v r rw------   1       0 -   /tmp/tmp1jcrcfb7
94216 python3.9            6 p - rw------   2       0 -   -
Aha!  The python script wants to interact with the output file, and it's quite obviously in the middle of reading from 'config.xml', presumably for an update of some kind.  Doesn't look much like malware at this distance, and in fact explains why alias updates sometimes fail and require a reboot of the box to apply.

That's just the apparent process call chain (ps hierarchy).  The update process was fired by cron.  At the end of the day, it's unusual enough to see a C compiler being invoked on a security appliance that I want to know if it's intentional behavior on a stock OPNsense install or not.

Just a quick question for those who know more about the table update internals than I do...noticed something odd and just want to know if it's normal.

Logged into a development box this morning and saw a hung linker job, which in and of itself isn't a big deal (the linker does hang from time to time), but what was odd was that it seems to have been called with the following chain:

Code Select Expand

/usr/local/bin/python3 /usr/local/opnsense/scripts/filter/update_tables.py
/usr/local/bin/flock -n -E 0 -o /tmp/filter_update_tables.lock /usr/local/opnsense/scripts/filter/update_tables.py
/usr/bin/cc -Wl,-t -o /tmp/tmpz1a1k7md -lpthread
/usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --enable-new-dtags -o /tmp/tmpz1a1k7md /usr/lib/crt1.o
Does the filter update script, through some dependency somewhere, end up trying to compile C code, or are we looking at a potentially compromised upstream rule server somewhere?