Just a quick question for those who know more about the table update internals than I do...noticed something odd and just want to know if it's normal.
Logged into a development box this morning and saw a hung linker job, which in and of itself isn't a big deal (the linker does hang from time to time), but what was odd was that it seems to have been called with the following chain:
/usr/local/bin/python3 /usr/local/opnsense/scripts/filter/update_tables.py
/usr/local/bin/flock -n -E 0 -o /tmp/filter_update_tables.lock /usr/local/opnsense/scripts/filter/update_tables.py
/usr/bin/cc -Wl,-t -o /tmp/tmpz1a1k7md -lpthread
/usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --enable-new-dtags -o /tmp/tmpz1a1k7md /usr/lib/crt1.o
Does the filter update script, through some dependency somewhere, end up trying to compile C code, or are we looking at a potentially compromised upstream rule server somewhere?
I'm missing a bit of context: what sort of "chain" are you posting? A ps process hierarchy? A shell history? Something else?
Nothing obvious from that block. It appears rather technical/clinical in the way the commands are run which would suggest this could be normal.
But, again, context matters.
Cheers,
Franco
That's just the apparent process call chain (ps hierarchy). The update process was fired by cron. At the end of the day, it's unusual enough to see a C compiler being invoked on a security appliance that I want to know if it's intentional behavior on a stock OPNsense install or not.
It is, but then again Python objects are created on the fly and I'm confident I don't know enough about how it works.
Cheers,
Franco
To close the loop on this, I found the same thing on another OPNSense box (hung ld process) and spent a little time digging around. I'm thinking this is a harmless side effect of the way OPNSense "compiles" the rules after an alias update, and the only reason I even saw the compiler call is because /tmp was full on both boxes. That led to a zero byte output file and a hung linker.
In case anyone is interested, this is why I'm thinking it's probably harmless / normal internal system operation (and a bit of a cheat sheet for anyone else that might run across a similar oddity and want to dig deeper):
pargs 94768
94768: ld.lld
argv[0]: /usr/bin/ld
argv[1]: --eh-frame-hdr
argv[2]: -dynamic-linker
argv[3]: /libexec/ld-elf.so.1
argv[4]: --enable-new-dtags
argv[5]: -o
argv[6]: /tmp/tmp1jcrcfb7
argv[7]: /usr/lib/crt1.o
argv[8]: /usr/lib/crti.o
argv[9]: /usr/lib/crtbegin.o
argv[10]: -L/usr/lib
argv[11]: -t
argv[12]: -lpthread
argv[13]: -lgcc
argv[14]: --as-needed
argv[15]: -lgcc_s
argv[16]: --no-as-needed
argv[17]: -lc
argv[18]: -lgcc
argv[19]: --as-needed
argv[20]: -lgcc_s
argv[21]: --no-as-needed
argv[22]: /usr/lib/crtend.o
argv[23]: /usr/lib/crtn.o
OK, that seems fairly normal....this is invoked by 'cc':
/usr/bin/cc -Wl,-t -o /tmp/tmp1jcrcfb7 -lpthread
So something is invoking the compiler in a manner where it is reading from stdin and writing to a temporary binary file. Let's go up another level and look....
root 94216 ... /usr/local/bin/python3 /usr/local/opnsense/scripts/filter/update_tables.py (python3.9)
procstat -f 94216
PID COMM FD T V FLAGS REF OFFSET PRO NAME
94216 python3.9 text v r r------- - - - /usr/local/bin/python3.9
94216 python3.9 cwd v d r------- - - - /root
94216 python3.9 root v d r------- - - - /
94216 python3.9 0 p - rw------ 4 0 - -
94216 python3.9 1 v c -w------ 2 0 - /dev/null
94216 python3.9 2 p - rw------ 2 0 - -
94216 python3.9 3 v r r------- 14 9567 - /usr/local/sbin/pluginctl
94216 python3.9 4 v r rw------ 14 512073 - /conf/config.xml
94216 python3.9 5 v r rw------ 1 0 - /tmp/tmp1jcrcfb7
94216 python3.9 6 p - rw------ 2 0 - -
Aha! The python script wants to interact with the output file, and it's quite obviously in the middle of reading from 'config.xml', presumably for an update of some kind. Doesn't look much like malware at this distance, and in fact explains why alias updates sometimes fail and require a reboot of the box to apply.
Just out of curiosity: I still cannot imagine why a compiler run should be neccessary to update the aliases from config.xml. The file has to be parsed, probably some URLs fetched, the output of which has to be parsed again (probably XML or JSON) and then fed into pf.
Using a C compiler in that process feels a little heavyweight...
@meyergru I completely agree, but short of one of the maintainers / designers of the table update script chiming in (or someone digging through all the nested Python calls and classes) we simply won't know why it's done this way. The only thing I can figure is it's a leftover from pfSense days, thus potentially a relic from when these machines were too slow to process the update for thousands of states outside of a compiled program.
But if that were not dynamically created, it could have been provided as a binary... and I see no need for a dynamically created program there.
pluginctl is PHP.
This was still bugging me enough that I went ahead and tracked down the call chain.
The filter update script imports dns.asyncresolver, which in turn imports trio (/usr/local/lib/python3.9/site-packages/trio/_core/_thread_cache.py). Trio needs the pthread library, so it calls ctypes.util.find_library("pthread"), which in turn calls _findLib_gcc.
_findLib_gcc in /usr/local/lib/python3.9/ctypes/util.py is what ends up invoking the c compiler, it apparently tries to compile a test program with the library in question to determine if the library exists. This does seem like unwanted behavior on a firewall, but I'm not 100% sure how to get rid of it given the call chain.