Messages

1

24.7 Production Series / [working now] 24.7 HA seemingly noworky™️ ... (not sure why tho)

« on: July 13, 2024, 07:07:23 am »

Hm.
well... maybe?
so I ran

Code: [Select]

opnsense-patch 07b96bcon both nodes, validated by:

Code: [Select]

[root@primary /var/log]# grep -c allow_url_fopen  /usr/local/opnsense/service/templates/OPNsense/WebGui/php.ini
0
[root@standby /var/log]#  grep -c allow_url_fopen  /usr/local/opnsense/service/templates/OPNsense/WebGui/php.ini
0

and restarted allthethings™️, which exhibited no change in behavior.

with the HA mode configured to 24.7

in https://both-my-firewalls/system_advanced_admin.php

I adjusted the deployment type to 'development'
I disabled http compression,
I enabled access logging,

and kicked off

/usr/local/etc/rc.filter_synchronize

again on the primary node:

Code: [Select]

[root@primary /usr/local]# /usr/local/etc/rc.filter_synchronize

Deprecated: Creation of dynamic property SimpleXMLRPC_Client::$url is deprecated in /usr/local/etc/inc/XMLRPC_Client.inc on line 92
send >>>
Host: 10.18.100.2
User-Agent: XML_RPC
Content-Type: text/xml
Content-Length: 117
Authorization: Basic da-REDACTED-Q=
<?xml version="1.0"?>
<methodCall>
<methodName>opnsense.firmware_version</methodName>
<params>
</params></methodCall>received >>>

Deprecated: Creation of dynamic property IXR_Message::$currentTag is deprecated in /usr/local/opnsense/contrib/IXR/IXR_Library.php on line 239

Warning: Cannot modify header information - headers already sent by (output started at /usr/local/opnsense/contrib/IXR/IXR_Library.php:239) in /usr/local/opnsense/contrib/IXR/IXR_Library.php on line 464

Warning: Cannot modify header information - headers already sent by (output started at /usr/local/opnsense/contrib/IXR/IXR_Library.php:239) in /usr/local/opnsense/contrib/IXR/IXR_Library.php on line 465

Warning: Cannot modify header information - headers already sent by (output started at /usr/local/opnsense/contrib/IXR/IXR_Library.php:239) in /usr/local/opnsense/contrib/IXR/IXR_Library.php on line 466

Warning: Cannot modify header information - headers already sent by (output started at /usr/local/opnsense/contrib/IXR/IXR_Library.php:239) in /usr/local/opnsense/contrib/IXR/IXR_Library.php on line 467
<?xml version="1.0"?>
<methodResponse>
  <params>
    <param>
      <value>
      <struct>
  <member><name>base</name><value><struct>
  <member><name>version</name><value><string>24.7.b</string></value></member>
</struct></value></member>
  <member><name>firmware</name><value><struct>
  <member><name>version</name><value><string>24.7.b_114</string></value></member>
</struct></value></member>
  <member><name>kernel</name><value><struct>
  <member><name>version</name><value><string>24.7.b</string></value></member>
</struct></value></member>
</struct>
      </value>
    </param>
  </params>
</methodResponse>
error >>>
parse error. not well formed[root@primary /usr/local]#
I then set the deployment type of the standby node to 'production', leaving the primary node in 'development'

and switched the HA mode 24.1 and kicked off
/usr/local/etc/rc.filter_synchronize on the primary node again, which seemed to work!!!
or at least... it returned:

Code: [Select]

[root@primary /usr/local]# /usr/local/etc/rc.filter_synchronize

Deprecated: Creation of dynamic property SimpleXMLRPC_Client::$url is deprecated in /usr/local/etc/inc/XMLRPC_Client.inc on line 92

Deprecated: Creation of dynamic property IXR_Message::$currentTag is deprecated in /usr/local/opnsense/contrib/IXR/IXR_Library.php on line 239

Deprecated: Creation of dynamic property SimpleXMLRPC_Client::$url is deprecated in /usr/local/etc/inc/XMLRPC_Client.inc on line 92

Deprecated: Creation of dynamic property IXR_Message::$currentTag is deprecated in /usr/local/opnsense/contrib/IXR/IXR_Library.php on line 239

Deprecated: Creation of dynamic property SimpleXMLRPC_Client::$url is deprecated in /usr/local/etc/inc/XMLRPC_Client.inc on line 92

Deprecated: Creation of dynamic property IXR_Message::$currentTag is deprecated in /usr/local/opnsense/contrib/IXR/IXR_Library.php on line 239
[root@primary /usr/local]#


I checked the https://primary.node/status_habackup.php

page, and it showed the standby information...

I then switched the HA mode to 24.7...  and kicked off another synchronization and it still seemed to work

I re-enabled http compression.... and kicked off another synchronization....
it still works....

there was a crashreporter event I sent along from both the active/standby nodes, with a mention of this topic, in case it sheds any light...

HOPEFULLY this helps?

I dunno what the "fix" was...
perhaps there was some odd config value that switching to development mode sidestepped, and was remediated in synchronization... but I really dunno...

if there is anything log-wise or diagnostic-wise that would be helpful for me to provide, I'm happy to.
bizarre.

2

Development and Code Review / Re: modal opened with 'clone $thing' has a misleading title of 'edit $clonedthing'

« on: July 12, 2024, 10:01:01 am »

Nod.. is this likely something you’ll wanna internationalize?
Not an ask, I just dunno if it’s toil, toil relief, carrot, stick,  or terrible idea 😊

3

High availability / Re: Strange behavior, unable to get CARP to work properly

« on: July 12, 2024, 03:36:55 am »

have you tried changing the native vlan id on the switchports to see if that is related to the problem?

4

Development and Code Review / Re: modal opened with 'clone $thing' has a misleading title of 'edit $clonedthing'

« on: July 12, 2024, 03:32:25 am »

Created https://github.com/opnsense/core/issues/7613 as requested

5

Development and Code Review / Re: configctl commands list?

« on: July 12, 2024, 03:24:03 am »

THANK YOU.

that will likely REALLY help (HOPEFULLY more than just me) next time I'm trying to remember syntax.

... now.... what did I do with my keys...

6

Development and Code Review / Re: modal opened with 'clone $thing' has a misleading title of 'edit $clonedthing'

« on: July 11, 2024, 09:19:14 am »

awesome! wilco! jus thought it best to ask here before doing so.

7

Development and Code Review / Re: configctl commands list?

« on: July 11, 2024, 09:18:30 am »

hiya!
thanks for your thoughts, and  yep… I agree…

the thought/ desire I had was more along the lines of
‘does it make sense to include a pointer to “configctl configd actions” when configctl is run with -h’ or something to remind those of us with poor memory how to find that output again.

i’m aware it’s not intended to be super user friendly…

and if you make it TOO user friendly you run the risk of asking for more borderline sane requests like this one

what would you think would be a good reminder/hint/clue here?

thanks again.
I sincerely appreciate your work.

8

Development and Code Review / modal opened with 'clone $thing' has a misleading title of 'edit $clonedthing'

« on: July 10, 2024, 10:57:18 pm »

hai all!

This is something of a widespread behavior within opnsense, and so bringing it up here as it feels like it's prolly a low-level behavioral issue not scoped to any one service or facet of the ui

When a user clicks on 'clone $thing' (rule, server, alias, firewall rule, dhcp reservation, haproxy config, etc)
(read 'pretty much anything that has a clone button')

The resulting modal popup has a title which conveys that the user is editing the original resource, rather than creating a new one from it.

This looks like its controlled by jquery, and the label seems like it would be targetable, thus if one clicks on 'clone' the title could read 'Clone $objectType' or 'Cloning $originalObjectTitle'  rather than 'Edit $objectType'

While it's true, one is TECHNICALLY editing an item of that type, it's misleading, as the window appears the same as if one clicked the 'edit $thing' button...

I'm sure there's a good reason for this behavior, but I'm having a hard time intuiting it...
Can someone more familiar with the code help me see what I'm missing?

9

Development and Code Review / configctl commands list?

« on: July 10, 2024, 09:58:33 pm »

the configctl tool is pretty cool!

however,
it's a little obtuse to use at the moment, due to a lack of

'heres what I can do'
guidance.

There is no inbuilt mechanism to query configctl about what faces it has, and what options those faces respond to.

There is no hinting as to where one might look in attempt to deduce this for themselves.

The user is expected to know this through osmosis.

I'm sure that as someone who works with it day in and day out, these papercuts aren't really noticeable..

However, for someone who logs into their firewall at 0400 because something's busted, half awake, and needing to restore homeostasis as expediently as possible, this problem's significance potentiates immensely.


I recognize that configctl's abilities are somewhat dynamic, based on the services installed and the scripts they feed to configctl...

however... pointing the user to `look in $directory for a list of configctl subcommands`

or coalescing the data at package install time, and saving it somepace for subsequent recall on invocation
doesn't intuitively feel especially burdensome, given the benefit a user would derive after the dreaded 0-dark-thirty page...

Thoughts?

10

Hardware and Performance / [DOS] Service reconfiguration logic doesn't take flapping into account

« on: July 10, 2024, 09:50:35 pm »

Howdy all!

Due to an unrelated issue discussed in another thread,
(In 24.7beta my where Standby fw and active fw no longer recognize each other's existence... )
I currently have my standby soft powered off.

HOWEVER, my standby seems to keep wanting to flap its heartbeat link.

(which is annoying, and outside the scope of opnsense, however it does bring a related problem to light...)

Several services are configured to re-jiggle themselves upon interface activation change....

HOWEVER, there doesn't appear to be any sort of 'flapping' logic to prevent a wobbly interface from causing secondary problems.

I'd think it reasonable for opnsense to implement some sort of guard around service state change, such that if an interface is wobbling and flapping often enough that re-running the 'OH! A NIC CHANGED STATE! I SHOULD DO A THING!' is essentially DOS's itself..

some sort of 'cooldown' timer, wherein the system wants to avoid restarting services or responding to state change due to $specific-hardware....  might be helpful?

perhaps with an incremental backoff, such that if interfaceX keeps flapping, after N times, its flaps will be ignored more and more of the time...

"interface flapping detected on wobblingNic0.... ongoing since .... Delaying actions for ($timer*wobblerate) for wobbling to settle" 
sort of thing...

Associating a backoff timer per device, along with a systemic timer, may be a valuable way to penalize the wobbler the most, whilst still responding timely to other random events that may occur.
(it's not nic5's fault that nic0 is wobbling)

granted, this isn't a problem that opnsense can CONTROL.

HOWEVER, OPNsense **CAN** control how it responds... and in the case of a badly behaving interface, I believe OPNsense may benefit from a bit more 'protect system stability in response to unforseen upstream events' robustification

Thoughts?

11

24.7 Production Series / [Worky™️ now] 24.7 HA seemingly Noworky™️ ... (not sure why tho)

« on: July 10, 2024, 01:04:18 am »

Howdy all!

I've updated my firewall pair to 24.7 by doing a config-import upgrade with the vga iso beta image.

Looks like there's gonna be some really cool stuff ahead!

I'm seeing an odd problem, however, with my hosts, in that the backup is no longer seen by the primary.

carp still seems to be working.

with sync compatibility either at 24.1 or 24.7, I don't experience a change in behavior.
I have the heartbeat crossover interface specified as the synchronization interface....

on the primary node, I have the synchronize peer ip set to the ip of the standby, and the
( synchronize config remote sytem username / remote system password ) values set.
These are empty on the standby.

I tried creating an API key for the sync user and adding it to the config on the standby, and using either of the components as the remote system password... but it didn't change the behavior at all

I can connect to the standby's heartbeat IP from the primary over the heartbeat interface via icmp, 80/443

Code: [Select]

primarynode# /usr/local/etc/rc.filter_synchronize
send >>>
Host: 10.18.100.2
User-Agent: XML_RPC
Content-Type: text/xml
Content-Length: 117
Authorization: Basic da-REDACTED-Q=
<?xml version="1.0"?>
<methodCall>
<methodName>opnsense.firmware_version</methodName>
<params>
</params></methodCall>received >>>
error >>>
fetch error. remote host down?
.

Code: [Select]

[root@primary /home/syncuser]# ssh 10.18.100.2 hostname
standby.mydomain.com
[root@primary /home/syncuser]#

I thought the problem might be SSL related, so I minted a trusted ssl cert which had the IPs of the hosts in the cert
but got the same issue....

so I disabled SSL on the web interface....

same issue.

I tried removing the synchronize IP so they try with multicast (with both 24.1 and 24.7 compatibility)
... but didn't notice a change in behavior


What diagnostics might I try to help identify the issue?

is this a known issue?

12

High availability / Re: Seeking guidance on moving WAN phy on a live ha firewall pair

« on: May 02, 2024, 05:28:13 pm »

I'd **HOPED** it would be this simple; but you know how it goes.... there's almosty always .....
Wibbly bits.


'this mostly works, unless you have HA configured alongside _service-here_ and....'

kinds of things

(For example,  if you set the heartbeat crossover interface to have an MTU > 1500 things can go kinda pear shaped)


appreciate the $.03. hopefully it'll be as straight forward as we want it to be

13

High availability / previously working: -> The backup firewall is not accessible or not configured.

« on: May 01, 2024, 08:53:28 pm »

[[solved]]

SO... setting the deployment type to 'production' as opposed to 'development' resolved the problem.

my guess is that right now, something in the php deprecation emissions causes the ha sync process to fail...

I'd think that this aughtn't happen... but we don't live in a perfect world...

so yeah. if you're running an HA pair, and you're running into problems, try switching the deployment type
( found in system -> settings -> administration -> deployment ((towards the bottom)) )
back to production and see if it resolves your issue

14

High availability / Seeking guidance on moving WAN phy on a live ha firewall pair

« on: May 01, 2024, 08:47:32 pm »

Hai all!

SO!
snazzy new internet upgrade happens... woohoo....
however now my firewall pair's wan interface is no longer fast enough to consume the additional bandwidth.

Fortunately, my firewall pair **DOES** have available and unutilized interfaces which ARE capable (copper 10G)
(I'm using the 10G SFP interfaces on it currently)

so in specific:

• igb0 **Heartbeat**
• igb1 **current WAN**
• igb2 **secondary unconfigured wan ((project for tomorrowland/not in scope here))**
• igb3 **virtual endpoint vlan**
• ixl0 *currently unused*
• ixl1 *currently unused*
• ixl2 **LAGG0.0**
• ixl3 **LAGG0.1**
• lagg0 (many vlan interfaces for internal traffic)

so both firewalls have the above rough topology.

essentially I want to identify the least problematic way to accomplish the goal of

move everything interacting with the physical connection 'igb1' to the physical connection 'ixl0'

on the firewall pair.

I can certainly cease using the standby for a bit
(ie power it off to prevent wobbly bits from making things harder than they need to be while reconfiguring)

but I'm not sure what the best way forward is...

I have a couple ideas, but before grabbing the scissors, blindfold, and running shoes, It felt prudent to reach out here and ask what others' experience has been.

anyone have any guidance or experience they care to share?

15

High availability / Re: How to configure OPNsense HA on a Multi-WAN network

« on: May 01, 2024, 08:28:24 pm »

Where are you at on your journey?
  Do you have HA set up already?
  Do you have multi-wan set up already?

What are you stumbling on/ wondering?