Looking for testers Q-Feeds plugin

[Patrick M. Hausen]

The scanner looks like a fantastic addition but my first test suggests that at least over IPv6 it does not quite work, yet.

I scanned both the external IPv4 and the external IPv6 address of my OPNsense. I run a Caddy reverse proxy for all public services. The IPv4 scan correctly identified open ports 80 and 443 while the IPv6 scan not only finished in a couple of seconds but also resulted in nothing whatsoever.

Thinking about it - I'll go check if maybe your scanner was blocked by either your own lists or one of the free ones I also use.

EDIT: no blocked packets. So something is wrong with the IPv6 scan it seems.

[Q-Feeds]

The scanner looks like a fantastic addition but my first test suggests that at least over IPv6 it does not quite work, yet.

I scanned both the external IPv4 and the external IPv6 address of my OPNsense. I run a Caddy reverse proxy for all public services. The IPv4 scan correctly identified open ports 80 and 443 while the IPv6 scan not only finished in a couple of seconds but also resulted in nothing whatsoever.

Thinking about it - I'll go check if maybe your scanner was blocked by either your own lists or one of the free ones I also use.

EDIT: no blocked packets. So something is wrong with the IPv6 scan it seems.

Thanks for your extensive testing once again ! Seems we got work to do! :)

EDIT: Fixed now for IPv6

[vk2him]

I've tested this today in my home and seems to be working ok - here's some feedback

• The main website https://qfeeds.com/ has a typo in this section
  .. "We offers detection and response services against phishing  .." the word "offers" should be "offer"
• I added the Blocklist "https://api.qfeeds.com/api?feed_type=malware_domains&api" into Adguard Home and it was working fine, however it blocked the solar monitoring website https://pvoutput.org
  I made a false positive report advising that I was using the malware domains list in AdGuard Home, and Support closed it saying it's not on the list. I double checked I set it correctly, which I had, and submitted another false positive report. Support closed it again and wrote back saying it must be an issue on my end as pvoutput.org isn't on your IP list and not in your Domains lists and to try force reloading the list. I logged another false positive after I disabled all the other AGH blocklists and force reloaded the qfeeds malware list. I also pointed out that issue is with the blocklist in Adguard Home which is using MALWARE Domains and that if I create a whitelist in AGH for pvoutput.org, I can access it, and I was able to access it until I loaded the Qfeeds blocklist - I'm still awaiting a reply
• When logging the false positive reports, I noticed that if I entered a single quote ' in my report, for example won't,  after I saved the report it displayed the HTML number ' instead won't   
• It would be good if a false positive report could be added to/reopened rather than needing to keep adding a new report as I had to keep repeating all the information from the previous ones there were (incorrectly) closed
• I was considering a Plus subscription, however Patrick reported that the scanner isn't working properly, so we need to wait a week to try again as we can only test one IP per week - can this be relaxed until you fix the issue with the scanner?
• In the TIP dashboard, it would be great if clicking on the panels My API Keys, Available Feeds and My API Calls were hyperlinks to those sections

Edited to add

• How can I enable monit to monitor qfeeds ?

Many thanks

[Q-Feeds]

I've tested this today in my home and seems to be working ok - here's some feedback

• The main website https://qfeeds.com/ has a typo in this section
  .. "We offers detection and response services against phishing  .." the word "offers" should be "offer"
• I added the Blocklist "https://api.qfeeds.com/api?feed_type=malware_domains&api" into Adguard Home and it was working fine, however it blocked the solar monitoring website https://pvoutput.org
  I made a false positive report advising that I was using the malware domains list in AdGuard Home, and Support closed it saying it's not on the list. I double checked I set it correctly, which I had, and submitted another false positive report. Support closed it again and wrote back saying it must be an issue on my end as pvoutput.org isn't on your IP list and not in your Domains lists and to try force reloading the list. I logged another false positive after I disabled all the other AGH blocklists and force reloaded the qfeeds malware list. I also pointed out that issue is with the blocklist in Adguard Home which is using MALWARE Domains and that if I create a whitelist in AGH for pvoutput.org, I can access it, and I was able to access it until I loaded the Qfeeds blocklist - I'm still awaiting a reply
• When logging the false positive reports, I noticed that if I entered a single quote ' in my report, for example won't,  after I saved the report it displayed the HTML number ' instead won't 
• It would be good if a false positive report could be added to/reopened rather than needing to keep adding a new report as I had to keep repeating all the information from the previous ones there were (incorrectly) closed
• I was considering a Plus subscription, however Patrick reported that the scanner isn't working properly, so we need to wait a week to try again as we can only test one IP per week - can this be relaxed until you fix the issue with the scanner?
• In the TIP dashboard, it would be great if clicking on the panels My API Keys, Available Feeds and My API Calls were hyperlinks to those sections

Edited to add

• How can I enable monit to monitor qfeeds ?

Many thanks

Dear vk2him,

Thank you for your great feedback and suggestions!

• Thanks, fixed it right away!
• I had a look at the reports but unfortunately couldn't reproduce this either. In our backend, which contains over 80 million IOCs, this domain doesn't exist, I also couldn't find it in the OSINT or Premium feeds. If you download the list directly in your browser, do you see the domain then? And or is there anyone else on this forum who experiences connection issues with pvoutput.org? Obviously I'm using all of our blocklists but I can connect without any issues.
• Thanks for pointing out the encoding issue, it's been fixed.
• Good idea, you can now reopen cases and also edit your initial submissions.
• The IPv6 issue is solved now. We'll reconsider the scan limits later, but for now, while we're monitoring infrastructure load, you can scan multiple IPs once per week. The allowed IPs are based on where you've connected from (via TIP login or API calls), to help prevent abuse.
• We've added direct links to the Logs and API Keys sections, great suggestion! For the Available Feeds section, we'll add a link once some new planned pages are ready.

Unfortunately, I can't assist directly with Monit configuration, but maybe someone else in the community can share some insights.

Kind regards,

David

[vk2him]

Dear vk2him,

Thank you for your great feedback and suggestions!

• Thanks, fixed it right away!
• I had a look at the reports but unfortunately couldn't reproduce this either. In our backend, which contains over 80 million IOCs, this domain doesn't exist, I also couldn't find it in the OSINT or Premium feeds. If you download the list directly in your browser, do you see the domain then? And or is there anyone else on this forum who experiences connection issues with pvoutput.org? Obviously I'm using all of our blocklists but I can connect without any issues.
• Thanks for pointing out the encoding issue, it's been fixed.
• Good idea, you can now reopen cases and also edit your initial submissions.
• The IPv6 issue is solved now. We'll reconsider the scan limits later, but for now, while we're monitoring infrastructure load, you can scan multiple IPs once per week. The allowed IPs are based on where you've connected from (via TIP login or API calls), to help prevent abuse.
• We've added direct links to the Logs and API Keys sections, great suggestion! For the Available Feeds section, we'll add a link once some new planned pages are ready.

[/list]

Unfortunately, I can't assist directly with Monit configuration, but maybe someone else in the community can share some insights.

Kind regards,

David

Thanks for the reply and fixing the items I pointed out.

Regarding pvoutput.org - today I am able to connect to it, and I wasn't able to find it on the list after I downloaded it, so that's strange.

Anyway, I'm seeing something strange at the moment - my Events tab isn't showing anything and it was yesterday.

I tried browing to a site that is in the IP filter table and the livelogs show Qfeeds blocked it, however The events tab is blank? I tried this yesterday and it appeared in live logs and the events tab?

[passeri]

I am curious to know what to make of this.

I saw in event logs that a string of outgoing connections were attempted from one machine. The first set was to 63.141.128.3 and the second to 162.247.243.29. I tried to look up each of these in TIP (Plus licence) to receive the reply "An error occurred while searching. Please try again or contact support if the problem persists." which it did.

dig showed NXDOMAIN of course. At virustotal.com each of these addresses was flagged by the SOCRadar Abusix list as malware but clean in the other 94 analyses shown by virustotal.

Is the likely source for these a pixel in a spam message or some web page? I have spam fairly heavily controlled so such a message rarely becomes visible to execute.

Why does the threat lookup so rarely respond with information, this not being the first time it has simply said there was an error?

[Q-Feeds]

Dear vk2him,

Thank you for your great feedback and suggestions!

• Thanks, fixed it right away!
• I had a look at the reports but unfortunately couldn't reproduce this either. In our backend, which contains over 80 million IOCs, this domain doesn't exist, I also couldn't find it in the OSINT or Premium feeds. If you download the list directly in your browser, do you see the domain then? And or is there anyone else on this forum who experiences connection issues with pvoutput.org? Obviously I'm using all of our blocklists but I can connect without any issues.
• Thanks for pointing out the encoding issue, it's been fixed.
• Good idea, you can now reopen cases and also edit your initial submissions.
• The IPv6 issue is solved now. We'll reconsider the scan limits later, but for now, while we're monitoring infrastructure load, you can scan multiple IPs once per week. The allowed IPs are based on where you've connected from (via TIP login or API calls), to help prevent abuse.
• We've added direct links to the Logs and API Keys sections, great suggestion! For the Available Feeds section, we'll add a link once some new planned pages are ready.

[/list]

Unfortunately, I can't assist directly with Monit configuration, but maybe someone else in the community can share some insights.

Kind regards,

David

Thanks for the reply and fixing the items I pointed out.

Regarding pvoutput.org - today I am able to connect to it, and I wasn't able to find it on the list after I downloaded it, so that's strange.

Anyway, I'm seeing something strange at the moment - my Events tab isn't showing anything and it was yesterday.

I tried browing to a site that is in the IP filter table and the livelogs show Qfeeds blocked it, however The events tab is blank? I tried this yesterday and it appeared in live logs and the events tab?

That's very interesting, but we're glad that issue is solved now. Now regarding the events tab, that's an interesting find as well. Just to be sure, you haven't disabled logging on the rules? And you do see blocks in the dashboard widget?

Does this command dump logs ? "/usr/local/opnsense/scripts/qfeeds/qfeedsctl.py logs"

[Q-Feeds]

I am curious to know what to make of this.

I saw in event logs that a string of outgoing connections were attempted from one machine. The first set was to 63.141.128.3 and the second to 162.247.243.29. I tried to look up each of these in TIP (Plus licence) to receive the reply "An error occurred while searching. Please try again or contact support if the problem persists." which it did.

dig showed NXDOMAIN of course. At virustotal.com each of these addresses was flagged by the SOCRadar Abusix list as malware but clean in the other 94 analyses shown by virustotal.

Is the likely source for these a pixel in a spam message or some web page? I have spam fairly heavily controlled so such a message rarely becomes visible to execute.

Why does the threat lookup so rarely respond with information, this not being the first time it has simply said there was an error?

Well to start, threat lookup should always show information on hits you've experienced. There are very rare cases that it doesn't if a IOC has been deleted from our database and the feed wasn't updated yet, but after looking into it that wasn't the case here. We've made some improvements to the lookup functionality which should result in faster lookups and no more error messages.

Regarding the IPs you were trying to lookup; 162.247.243.29 is known for an IP from newrelic. This is indeed a kind of tracking pixel but there were also known events were they're scanning the internet for unknown reasons and even possibly exploiting https://nvd.nist.gov/vuln/detail/CVE-2020-11910. 63.141.128.3 is known for brute force tries, interesting to see that you experience outbound connections to it.

[passeri]

interesting to see that you experience outbound connections to it

That "interesting" could be carrying a lot of freight. The attempted connections were for a short period then ceased. The machine which sourced them has Sophos Premium running on it and no open ports. The router which trapped them is internal, not at the edge, looking only at outgoing traffic, Community key for Q-feeds. The Plus key is on the edge so it saw nothing of this.

I tried the threat lookups again. They worked in Safari, not in Mullvad (Firefox), "network error". Everything is latest versions.

[vk2him]

That's very interesting, but we're glad that issue is solved now. Now regarding the events tab, that's an interesting find as well. Just to be sure, you haven't disabled logging on the rules? And you do see blocks in the dashboard widget?

Does this command dump logs ? "/usr/local/opnsense/scripts/qfeeds/qfeedsctl.py logs"

The rules have the logging enabled - I shared a screenshot in my previous reply of the live logs showing my test was blocked. Yes the dashboard widget shows a large blocked number.

The command gives an error:

Code Select Expand

root@OPNsense:~ # /usr/local/opnsense/scripts/qfeeds/qfeedsctl.py logs
Traceback (most recent call last):
  File "/usr/local/opnsense/scripts/qfeeds/qfeedsctl.py", line 50, in <module>
    for msg in getattr(actions, action)():
  File "/usr/local/opnsense/scripts/qfeeds/lib/__init__.py", line 187, in logs
    yield ujson.dumps({'rows': PFLogCrawler(feeds).find()})
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/opnsense/scripts/qfeeds/lib/log.py", line 75, in find
    result.append(self._parse_log_line(line))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/opnsense/scripts/qfeeds/lib/log.py", line 64, in _parse_log_line
    return [parts[1], fw_line[4], fw_line[7]] + [x for x in fw_line if is_ip_address(x)]
                      ~~~~~~~^^^
IndexError: list index out of range
Thanks