Has the OPNsense team applied for Claude Mythos access yet?

[drosophila]

Judging from it's most recent response, the SPAMbot has lost the plot now. So much for AI. XD

Yeah, a fuzzer can be useful, and AI tests may effectively be a "stateful fuzzer". Still the old saying "don't believe the hype" stands strong and it's certainly nothing that must be done now, immediately. Let the dust settle and see what remains.

And I prefer spaghetti code to the risotto code we get today. With spaghetti code, you can at least scroll and find (by /, F3, whatever) directly because you know which file it is in. With risotto, you have to search through countless files for a one-liner that merely sets some parameters in something that you then need to find in yet another file, in another directory, rinse, repeat. Yes, this can be fixed by good indexing that can tell the difference between declaration, implementation and calls, if it has a quick way to access it without digging through countless sub-menus. To stay on topic: maybe AI would work for that as well, but it's not needed. The only benefit would be that it could generate a meaningful summary of what the respective function actually does.

[OPNenthu]

I see a lot of focus on software bugs but is there a similar push to work out issues in silicon?  Or are those proprietary secrets too precious to ever let these AIs sniff around in? 🤔

[connervt]

I work in semiconductor manufacturing (25+ years now).  The "silicon" you are talking about, CPU/GPU/SoC CMOS logic, has some of the most guarded IP (Intellectual Property) that you can imagine.  The companies offering LLM AI also make semiconductor chips, so there is no way they will ever let AI get a whiff of them.

That said, there is extensive use of computer algorithmic review during all phases of development, manufacturing and test.  Companies such as Cadence Design Systems have been offering tools to the semiconductor industry for decades.

[serving_myself]

Not on your radar at the moment.


Cheers,
Franco

Hello Franco,

Has something changed after these past few months?

I'd like to voice an opinion on this subject as a (home) user of OPNsense.

Major vendors of commercial network security products have all joined the party. Of course, from the outside, it may seem like a marketing/PR stunt ("let's join to show everyone how serious we're about security", "X joined so we also have to, otherwise X's sales will say to the customers that we aren't serious enough about security", etc.). I can't know what were the incentives at the start of it, but it's certainly not the case today.

During the past few months, the number of security-related software updates in some relevant commercial software products increased significantly. I know it is taken very seriously inside those companies.

Leaving aside the political, economic, and social aspects of AI as a phenomenon, in my opinion, ignoring its direct impact on the security/tech landscape is not pragmatic.

I don't know what requirements a FOSS project like OPNsense would have to meet to get involved in Glasswing (and I appreciate the amount of work this can potentially generate for maintainers), but my suggestion would be to at least have a look and try to get access.
After all, if getting access itself is not a lot of work, the worst that could happen is that all you get is useless false positives and/or minor things not worth fixing (immediately). In return you will learn the current baseline for frontier AI security capabilities in the context of OPNsense codebase and get a confirmation that (at least at the moment) OPNsense is safe in this brave new world. And us, users, will also get a peace of mind.

Of course, I don't have your full perspective, so it'd be valuable to at least understand why not.

[franco]

> Major vendors of commercial network security products have all joined the party.

Thanks for showing us how your marketing strategy works, Mr. 1-post account.

The mafia has similar tactics you know: everyone has protection around this neighborhood -- would be a shame if something happened to your little open source project.


Cheers,
Franco

[pfry]

[...]I'd like to voice an opinion on this subject as a (home) user of OPNsense.[...]

As with most open-source projects, feel free to put your money where your mouth is. Folks could always donate an account/access. I imagine a direct (monetary) donation would be more welcome (and useful), though.

I think VRF/FIB support would be nice. Enough to pay for it? Heh.

[serving_myself]

> Thanks for showing us how your marketing strategy works, Mr. 1-post account.

The mafia has similar tactics you know: everyone has protection around this neighborhood -- would be a shame if something happened to your little open source project.

And you made your conclusion based on ... what exactly? A single datapoint - the number of posts I have on this forum?
Looks like a joke some stats prof would tell as an intro.

For the reference, here's a couple of random posts from the same account name on other forums related to home-labbing and self-hosting:
https://forum.level1techs.com/t/nvme-namespaces-little-known-cool-features-of-most-nvme-and-user-programmable-endurance/172660/26?u=serving_myself
https://www.truenas.com/community/threads/how-to-change-link_power_management_policy-and-make-it-persistent.109390/post-755532

And if you tried a simple search for this account name you'd probably find a few other posts/boards.
Just because I'm usually a reader and not a poster doesn't mean that I'm a "marketing bot" or whatever you've assumed about me.

I politely asked you for an opinion, and you brushed me off as a bot for no reason.

P.S.: notwithstanding another logical twist - looking at the current member list of that "mafia project", how likely it is to see an army of bots running around the Internet and trying to sway smaller individual FOSS project like this one, especially considering that (I assume, of course) you probably haven't heard of such widespread tactics in this context? I see it as "almost certainly not".

[serving_myself]

As with most open-source projects, feel free to put your money where your mouth is. Folks could always donate an account/access. I imagine a direct (monetary) donation would be more welcome (and useful), though.

It's not something one can "donate". The access to this thing is strictly gated through this Glasswing project membership. If you follow the field you know that Anthropics only publicly released the nerfed-down over-refusing version of Mythos (Fable).
From what I heard, there's a way to get free credits for the use of (public) Claude models for FOSS projects anyway.

The access to "the real thing" is provided through orgs like Linux Foundation and Apache, but apparently based on Anthropic expanding it recently, the criteria must've been also expanded. In my understanding, OPNsense is on the smaller side, but it arguably is a part of what can be considered a critical security infra. Thus, the access request itself wouldn't be baseless.

If the maintainers would've said "we applied, they refused" or "the costs are exorbitant" I'd perfectly understand these arguments. But that hasn't been the case so far. At least the "marketing mafia" makes it look like there's a lot of freebies for FOSS projects in this space that they can just use. At least for now.

[Monviech (Cedrik)]

Reminds me of Cartmanland in South Park.

The gated thing acquires enormous perceived value precisely because you can't just use it. If Anthropic simply exposed "Mythos Security Scanner" to everyone tomorrow, a good chunk of the mystique would probably disappear after people had fed it a few large codebases and discovered the usual mixture of impressive findings, mundane observations, and garbage.

"My AI tool is the most dangerous" seems to attract the most venture capital, more examples are "Rogue AI Agents hacked something oh no so dangerous" like when OpenAI or Anthropic or who else need more money again.

There is a very convenient incentive structure around these stories. Normally, "our autonomous system behaved unexpectedly and compromised a real target" would sound like a catastrophic control failure. In the current AI market, essentially the same story can be packaged as evidence of capability:

Look how powerful our model is — we barely contained it!

[serving_myself]

Arguing about marketing ploys can be fun indeed, but completely impractical (unless you have access to a board-member of some frontier lab who's suddenly ready to get candid in public about this). Thus, that's not what I'm arguing about.

The more down-to-earth questions that interest me are:
1. Is there utility? Does it make sense to use those tools to find vulns in the code or it is a completely useless exercise?
2. How cost/benefit analysis changes this? Is it prohibitively expensive, or laying hands on these tools is impossible in practice, or there are some other obstacles/costs that make this non-viable?

My (speculative, using implicit evidence) argument was that the answers to these questions are "1. yes, there's one" and "2. not by much".

If from your POV the perceived utility is 0 because the real capability is so far below the advertised one, then it'd be a position, of course. But then there's a counter-argument of "how do you know if you haven't tried yet?".
Your own mention of "impressive findings" among "mundane observations, and garbage" tells me that it might be not as straightforward as this.

That's the questions I was looking to get answers for from you.


P.S.: Maybe I'm too early to the party. A year ago there were still a lot of skeptics (inc. in my circle) who were saying that agents will never be able to write proper code. Now most of them don't even read most of the code that an agent generates, except for the most critical paths. Maybe we'll just have to wait for the Chinese to release something that will force US labs to show their frontier to the public so everyone could try and see. If it all proves a failure, oddly enough I'll be among those celebrating this fact.

[Monviech (Cedrik)]

To reflect on it I use Chatgpt, Claude and self hosted LLMs and also other tools like Stable Diffusion since years now, incrementally seeing how these tools evolve.

Yet I dont pretend to be an expert on the matter, I dont use MCP or autonomous agentic AI yet.

More compute and larger context + time means AI will be able to find more things.

If it's one tool or another doesn't matter that much in the grand scheme of things by how fast it's still evolving.

Im not attaching exceptional significance to one gated Anthropic product because Anthropic says it's exceptionally capable.