Rogue speed test service using a lot of bandwidth

[bobshark]

Hi, I observed a crazy amount of traffic in my provider and noticed my wan metrics had a spike every 5 minutes. Source ip is the wan ipv4 and 6. 6 is disabled on all my internal nets and I also do not see any traffic on the other interfaces so I suspect it’s running in the firewall. Blocking port 8080 on wan out worked but I would like to find the root cause. I can’t find any plugin installed that may be the cause. Any idea how I can track it down?

[Seimus]

If it happens every 5min check the Toptalkers in Traffic to see if its really the FW.
And if its really the FW check what plugins are installed and cronjobs are set.
Check as well what packages are installed directly on the device.

Code: [Select]

pkg info –all

Regards,
S.

[bobshark]

Checked the packages and I do not see anything suspicious.

FW logs: https://drive.google.com/file/d/1zMAxAUhyT3bvBUryjqerWsDQ0bPpieYB/view?usp=sharing Note how the source ip is wan  in ipv6 and 4.

Note before the firewall rule how only the wan interface had the spikes: https://drive.google.com/file/d/1zpLlNUyno5ZncSlS8hbj6hAgOxKEcuIA/view?usp=sharing (in / out as it seems its a up / down test). Not sure how can it be a different host if its just on the wan side?

Installed packages:

Code: [Select]

abseil-20240722.0              Abseil Common Libraries (C++)
acme.sh-3.0.7_2                ACME protocol client written in shell
beep-1.0_2                     Beeps a certain duration and pitch out of the PC Speaker
bind-tools-9.18.28             Command line tools from BIND: delv, dig, host, nslookup...
boost-libs-1.84.0              Free portable C++ libraries (without Boost.Python)
ca_root_nss-3.93               Root certificate bundle from the Mozilla Project
choparp-20150613_1             Simple proxy arp daemon
cpdup-1.22_1                   Comprehensive filesystem mirroring and backup program
cpustats-0.1                   Gather system statistics
curl-8.9.1_1                   Command line tool and library for transferring data with URLs
cyrus-sasl-2.1.28_4            RFC 2222 SASL (Simple Authentication and Security Layer)
cyrus-sasl-gssapi-2.1.28       SASL GSSAPI authentication plugin
ddclient-3.11.2_1              Update dynamic DNS entries
dhcp6c-20240820                OPNsense WIDE-DHCPv6 client
dhcrelay-0.6                   OpenBSD dhcrelay daemons
dnsmasq-2.90_2,1               Lightweight DNS forwarder, DHCP, and TFTP server
dpinger-3.3                    IP device monitoring tool
e2fsprogs-libuuid-1.47.1       UUID library from e2fsprogs package
easy-rsa-3.1.7,1               Small RSA key management package based on openssl
expat-2.6.2                    XML 1.0 parser written in C
expiretable-0.6_3              Utility to remove entries from the pf(4) table based on their age
filterlog-0.7_1                Parse pflog(4) output
flock-2.37.2_1                 Manage locks from shell scripts
flowd-0.9.1_5                  Small, fast, and secure NetFlow collector
fstrm-0.6.1_1                  Implementation of the Frame Streams data transport protocol in C
gettext-runtime-0.22.5         GNU gettext runtime libraries and programs
glib-2.80.4,2                  Some useful routines of C programming (current stable version)
gmp-6.3.0                      Free library for arbitrary precision arithmetic
hostapd-2.11                   IEEE 802.11 AP, IEEE 802.1X/WPA/WPA2/EAP/RADIUS Authenticator
hyperscan-5.4.2                High-performance multiple regex matching library
icu-74.2_1,1                   International Components for Unicode (from IBM)
ifinfo-13.0_1                  Interface statistics reader
iftop-1.0.p4_1                 Display bandwidth usage on an interface by host
indexinfo-0.3.1                Utility to regenerate the GNU info page index
isc-dhcp44-server-4.4.3P1_1    ISC Dynamic Host Configuration Protocol server
ivykis-0.43.2                  Asynchronous I/O readiness notification library
jansson-2.14                   C library for encoding, decoding, and manipulating JSON data
json-c-0.17                    JSON (JavaScript Object Notation) implementation in C
jsoncpp-1.9.5                  JSON reader and writer library for C++
kea-2.6.1                      Alternative DHCP implementation by ISC
krb5-1.21.3                    MIT implementation of RFC 4120 network authentication service
ldns-1.8.4                     Library for programs conforming to DNS RFCs and drafts
libargon2-20190702_1           Memory hard password hashing program and library
libcbor-0.11.0                 CBOR protocol implementation for C and others
libcjson-1.7.18_2              Ultralightweight JSON parser in ANSI C
libedit-3.1.20240517,1         Command line editor library
libevent-2.1.12                API for executing callback functions on events or timeouts
libffi-3.4.6                   Foreign Function Interface
libfido2-1.15.0                Provides library functionality for FIDO 2.0
libgcrypt-1.11.0               General purpose cryptographic library based on the code from GnuPG
libgpg-error-1.50              Common error values for all GnuPG components
libiconv-1.17_1                Character set conversion library
libidn2-2.3.7                  Implementation of IDNA2008 internationalized domain names
libltdl-2.4.7                  System independent dlopen wrapper
liblz4-1.10.0,1                LZ4 compression library, lossless and very fast
libmcrypt-2.5.8_4              Multi-cipher cryptographic library (used in PHP)
libnet-1.3,1                   C library for creating IP packets
libnghttp2-1.62.1              HTTP/2.0 C Library
libpfctl-0.12                  Library for interaction with pf(4)
libpsl-0.21.5_1                C library to handle the Public Suffix List
libsodium-1.0.19               Library to build higher-level cryptographic tools
libucl-0.9.2                   Universal configuration library parser
libunistring-1.2               Unicode string library
libuv-1.48.0                   Multi-platform support library with a focus on asynchronous I/O
libxml2-2.11.8                 XML parser library for GNOME
libxslt-1.1.37_1               XML stylesheet transformation library
libyaml-0.2.5                  YAML 1.1 parser and emitter written in C
lighttpd-1.4.76                Secure, fast, compliant, and flexible Web Server
log4cplus-2.1.1                Logging library for C++
lzo2-2.10_1                    Portable speedy, lossless data compression library
mdns-repeater-1.11_1           Multicast DNS repeater to re-broadcast mDNS packets across interfaces
monit-5.33.0_1                 Unix system management and proactive monitoring
mpd5-5.9_18                    Multi-link PPP daemon based on netgraph(4)
mpdecimal-4.0.0                C/C++ arbitrary precision decimal floating point libraries
net-snmp-5.9.4_6,1             Extendable SNMP implementation
nettle-3.10_1                  Low-level cryptographic library
nspr-4.35                      Platform-neutral API for system level and libc like functions
nss-3.103                      Libraries to support development of security-enabled applications
ntp-4.2.8p18                   The Network Time Protocol Distribution
oniguruma-6.9.9                Regular expressions library compatible with POSIX/GNU/Perl
openldap26-client-2.6.8        Open source LDAP client implementation
openssh-portable-9.8.p1_1,1    The portable version of OpenBSD's OpenSSH
openssl-3.0.14,1               TLSv1.3 capable SSL and crypto library
openvpn-2.6.12                 Secure IP/Ethernet tunnel daemon
opnsense-24.7.2                OPNsense community release
opnsense-installer-24.7        OPNsense installer scripts
opnsense-lang-23.7.11          OPNsense translations
opnsense-update-24.7.2         OPNsense update utilities
os-acme-client-4.5             ACME Client
os-ddclient-1.23               Dynamic DNS client
os-mdns-repeater-1.1_1         Proxy multicast DNS between networks
os-net-snmp-1.5_5              Net-SNMP is a daemon for the SNMP protocol
os-qemu-guest-agent-1.2        QEMU Guest Agent for OPNsense
os-squid-1.0_4                 Squid is a caching proxy for the web
os-telegraf-1.12.11            Agent for collecting metrics and data
p5-Data-Validate-IP-0.27       Common data validation methods for IPs
p5-NetAddr-IP-4.079            Work with IPv4 and IPv6 addresses and subnets
pam_opnsense-24.1              OPNsense shared authentication system using PAM
pcre2-10.43                    Perl Compatible Regular Expressions library, version 2
perl5-5.36.3_2                 Practical Extraction and Report Language
pftop-0.10_1                   Utility for real-time display of statistics for pf
php82-8.2.22                   PHP Scripting Language (8.2.X branch)
php82-ctype-8.2.22             The ctype shared extension for php
php82-curl-8.2.22              The curl shared extension for php
php82-dom-8.2.22               The dom shared extension for php
php82-filter-8.2.22            The filter shared extension for php
php82-gettext-8.2.22           The gettext shared extension for php
php82-google-api-php-client-2.4.0 Google APIs Client Library for PHP
php82-ldap-8.2.22              The ldap shared extension for php
php82-mbstring-8.2.22          The mbstring shared extension for php
php82-pcntl-8.2.22             The pcntl shared extension for php
php82-pdo-8.2.22               The pdo shared extension for php
php82-pear-1.10.13             PEAR framework for PHP
php82-pear-Crypt_CHAP-1.5.0_1  PEAR class for generating CHAP packets
php82-pecl-mcrypt-1.0.7        PHP extension for mcrypt, removed in PHP 7.2
php82-pecl-radius-1.4.0b1_2    Radius client library for PHP
php82-phalcon-5.8.0            Phalcon PHP Framework written in C-language
php82-phpseclib-3.0.41         PHP Secure Communications Library
php82-session-8.2.22           The session shared extension for php
php82-simplexml-8.2.22         The simplexml shared extension for php
php82-sockets-8.2.22           The sockets shared extension for php
php82-sqlite3-8.2.22           The sqlite3 shared extension for php
php82-xml-8.2.22               The xml shared extension for php
php82-zlib-8.2.22              The zlib shared extension for php
pkcs11-helper-1.29.0_3         Helper library for multiple PKCS#11 providers
pkg-1.19.2_1                   Package manager
protobuf-27.3,1                Data interchange format library
protobuf-c-1.4.1_3             Code generator and libraries to use Protocol Buffers from pure C
py311-Babel-2.16.0             Collection of tools for internationalizing Python applications
py311-Jinja2-3.1.4             Fast and easy to use stand-alone template engine
py311-aioquic-1.2.0            Implementation of QUIC and HTTP/3
py311-anyio-4.4.0              High level compatibility layer for multiple asynchronous event loop implementations
py311-async_generator-1.10     Tiny library to add async generators to Python 3.5
py311-attrs-24.1.0             Python attributes without boilerplate
py311-beautifulsoup-4.12.3     HTML/XML Parser for Python
py311-boto3-1.34.155           AWS SDK for Python
py311-botocore-1.34.155        Low-level, core functionality of boto 3
py311-bottleneck-1.3.8_1       Collection of fast NumPy array functions written in Cython
py311-certifi-2024.7.4         Mozilla SSL certificates
py311-cffi-1.17.0              Foreign Function Interface for Python calling C code
py311-charset-normalizer-3.3.2_1 Real First Universal Charset Detector
py311-cryptography-42.0.8_1,1  Cryptographic recipes and primitives for Python developers
py311-dns-lexicon-3.17.0_1     Manipulate DNS records on various DNS providers in a standardized way
py311-dnspython-2.6.1,1        DNS toolkit for Python
py311-duckdb-1.0.0             In-process SQL OLAP database management system
py311-filelock-3.15.4          Platform independent file lock
py311-h11-0.14.0               Pure-Python, bring-your-own-I/O implementation of HTTP/1.1
py311-h2-4.1.0                 HTTP/2 State-Machine based protocol implementation
py311-hpack-4.0.0              HTTP/2 header encoding (HPACK) logic implementation
py311-html5lib-1.1             Python based HTML parser based on the HTML5 spec
py311-httpcore-1.0.5           Minimal low-level HTTP client
py311-httpx-0.27.0_1           Next generation HTTP client
py311-hyperframe-6.0.0         Python module that decodes binary streams into HTTP/2 frames
py311-idna-3.7                 Internationalized Domain Names in Applications (IDNA)
py311-jmespath-1.0.1_1         JSON Matching Expressions
py311-lxml-4.9.3               Pythonic binding for the libxml2 and libxslt libraries
py311-markupsafe-2.1.5_1       Implements XML/HTML/XHTML Markup safe string for Python
py311-netaddr-1.3.0            Manipulation of IPv4, IPv6, CIDR, EUI and MAC network addresses
py311-numexpr-2.10.1           Fast numerical array expression evaluator for Python and NumPy
py311-numpy-1.25.0_7,1         The New Numeric Extension to Python
py311-openssl-24.1.0,1         Python interface to the OpenSSL library
py311-outcome-1.3.0_1          Capture the outcome of Python function calls
py311-packaging-24.1           Core utilities for Python packages
py311-pandas-2.0.3_2,1         Flexible, high-performance data analysis in Python
py311-pyasn1-0.6.0             ASN.1 toolkit for Python
py311-pyasn1-modules-0.4.0     Collection of ASN.1 data structures for py-pyasn1
py311-pycparser-2.22           C parser in Python
py311-pylsqpack-0.3.18         Python wrapper for the ls-qpack QPACK library
py311-pyotp-2.9.0_1            Python One Time Password Library
py311-pysocks-1.7.1_1          Python SOCKS module
py311-python-dateutil-2.9.0    Extensions to the standard Python datetime module
py311-pytz-2024.1,1            World Timezone Definitions for Python
py311-pyyaml-6.0.1             Python YAML parser
py311-requests-2.32.3          Python HTTP for Humans
py311-requests-file-2.0.0      Transport adapter for use with the Requests Python library
py311-s3transfer-0.10.2        Amazon S3 Transfer Manager for Python
py311-service-identity-24.1.0  Service identity verification for pyOpenSSL & cryptography
py311-setuptools-63.1.0_1      Python packages installer
py311-six-1.16.0_1             Python 2 and 3 compatibility utilities
py311-sniffio-1.3.1            Sniff out which async library your code is running under
py311-socksio-1.0.0_1          Sans-I/O implementation of SOCKS4, SOCKS4A, and SOCKS5
py311-sortedcontainers-2.4.0   Python Sorted Container Types: SortedList, SortedDict, and SortedSet
py311-soupsieve-2.0.1          CSS4 selector implementation for BeautifulSoup
py311-sqlite3-3.11.9_7         Standard Python binding to the SQLite3 library (Python 3.11)
py311-tldextract-3.3.1         Separate the TLD from the registered domain and subdomains of a URL
py311-trio-0.26.2              Library for async concurrency and I/O
py311-tzdata-2024.1            Provider of IANA time zone data
py311-ujson-5.10.0             Ultra fast JSON encoder and decoder for Python
py311-urllib3-1.26.19,1        HTTP library with thread-safe connection pooling, file post, and more
py311-vici-5.9.11              Native Python interface for strongSwan's VICI protocol
py311-webencodings-0.5.1       Character encoding aliases for legacy web content
python311-3.11.9_1             Interpreted object-oriented programming language
qemu-guest-agent-9.0.2         QEMU CPU Emulator (guest agent)
radvd-2.19_4                   Linux/BSD IPv6 router advertisement daemon
readline-8.2.10                Library for editing command lines as they are typed
rrdtool-1.9.0                  Round Robin Database Tools
samplicator-1.3.8.r1_1         Receives UDP datagrams and redistributes them to a set of receivers
socat-1.8.0.0_3                Multipurpose relay and more
sqlite3-3.46.0,1               SQL database engine in a C library
squid-6.10                     HTTP Caching Proxy
squid-langpack-7.0.0.20240307  Language-specific error documents for Squid web cache
strongswan-5.9.14              Open Source IKEv2 IPsec-based VPN solution
sudo-1.9.15p5_4                Allow others to run commands as root
suricata-7.0.6                 High Performance Network IDS, IPS and Security Monitoring engine
syslog-ng-4.8.0_1              Powerful syslogd replacement
telegraf-1.31.3                Time-series data collection
unbound-1.21.0                 Validating, recursive, and caching DNS resolver
wpa_supplicant-2.11_1          Supplicant (client) for WPA/802.1x protocols
zip-3.0_2                      Create/update ZIP files compatible with PKZIP



[Seimus]

Indeed this looks like ports and server for Speedtests.

However keep in mind you will see the source in OUT direction as your WAN, cause you are NATing the private LAN addresses behind the Public WAN IP on your OPNsense.

So, this doesn't means that OPNsense itself is doing it. Its highly possible something behind it can cause this. You can actually check the top-talker.

Reporting > Traffic > Toptalker
here in the menu on the right choose all your interfaces and observe who is the IP connecting to the destination of Speedtest servers for destination port 8080.

Regards,
S.

[bobshark]

Attaching the top talker view in text and graph. Note how the graph shows the increase in traffic but not the text. I still can't determine the source beyond the firewall.

Few other steps done:

- Created a new VM and installed opnsense again while restoring the config (trying to see if it was some manual cron or process I did at some point)
- Packet capture on the firewall interfaces: I only see WAN source ip, ipv4 and ipv6. Not only that but the capture shows no packets on any interface beyond wan (capturing port 8080 renders nothing on any interface except wan)
- Checked cron (only ping_hosts.sh file but its every 4 mins)

[bobshark]

Attaching cron

[bobshark]

I even added one rule per interface to block any outbound to port 8080 except wan and no hits either. I just can't think of anything beyond the firewall its doing this at this point

[meyergru]

I think you missed a part of the OpnSense cron tabs, which are created for user "nobody" and can be seen from the command line via "crontab -u nobody -l" or in the web UI under "System: Settings: Cron".

[bobshark]

There is one but it’s disabled and runs daily (auto update acme)

[Seimus]

From the TopTalkers,

The only IP that shows high Throughput during the time you experience this is an IPv6 IP, I assume its the FW itself as you mentioned on start?

Honestly this is wierd, if you dont have the speedtest plugin installed nor running anything via CRON, its a mystery  why its happening.

Did you try to check the top output during the time when this happens? If some process/daemon doesn't pop out during the interval?

By pure logic the FW can not by itself just go and run periodical speedtest if its not set to to it.

Regards,
S.