Why does Netflow V9 packets include OUT_BYTES and OUT_PACKETS that are always zero?

[krby]

I have Netflow setup to send v9 flows. When I look at the templates sent from OPNsense I see that IN_BYTES/IN_PACKETS as well OUT_BYTES/OUT_PACKETS are defined?

But when I get actual flows, OUT_BYTES and OUT_PACKETS are always 0. Why include them if they're always 0? The reason this is a problem is I'm trying to parse with goflow or goflow2 packages for golang and both of them will prefer OUT_BYTES/PACKETS to IN. See: https://github.com/netsampler/goflow2/issues/186

Is it intentional to include the OUT_* stats but never populate them?

[krby]

I did some digging and confirmed sys/netgraph/netflow/netflow_v9.c always sets OUT_BYTES and OUT_PACKETS to 0.

Code Select Expand

       
rec->src_addr = fed->r.r_src.s_addr;
rec->dst_addr = fed->r.r_dst.s_addr;
...
rec->i_packets  = htonl(fed->packets);
rec->i_octets   = htonl(fed->bytes);
rec->o_packets  = htonl(0);
rec->o_octets   = htonl(0);
rec->first    = htonl(MILLIUPTIME(fed->first));
rec->last     = htonl(MILLIUPTIME(fed->last));

Why is it done this way? I've seen other collectors that only support unidirectional flows and they just don't include OUT_BYTES/OUT_PACKETS

[lilsense]

Let me GROK that for you:

If you're seeing `OUT_BYTES` and `OUT_PACKETS` as always zero in the NetFlow v9 flow records from OPNsense, despite these fields being defined in the templates, there are several potential reasons for this behavior. Below, I'll explain why this might be happening and why these fields are still included in the templates, along with steps to diagnose and potentially resolve the issue.
### Why `OUT_BYTES` and `OUT_PACKETS` Are Always Zero
1. **NetFlow Configuration in OPNsense**:
  - OPNsense's NetFlow exporter, based on the `ng_netflow` module in FreeBSD, can be configured to capture either ingress, egress, or both directions of traffic. If your configuration is set to capture only **ingress** traffic (default in many setups), the `OUT_BYTES` and `OUT_PACKETS` fields will be zero because no egress data is being collected.
  - Check the NetFlow settings in OPNsense under **Reporting > NetFlow**. Look at the interface settings to see if egress capturing is enabled. If only ingress is selected, the exporter will not populate `OUT_BYTES` and `OUT_PACKETS`.
2. **Traffic Direction and NAT**:
  - In OPNsense, Network Address Translation (NAT) can affect how flows are recorded. If traffic is being NAT'd (e.g., from LAN to WAN), the firewall may only record flows in one direction (typically ingress) to avoid double-counting traffic. This is a common optimization in NetFlow setups, as noted in OPNsense documentation, where capturing both directions can lead to duplicate data for NAT'd flows.
  - If your setup involves NAT, the exporter might intentionally omit egress data to simplify analysis, resulting in `OUT_BYTES` and `OUT_PACKETS` being zero.
3. **Interface Monitoring**:
  - The fields `OUT_BYTES` and `OUT_PACKETS` are populated based on the traffic leaving the monitored interface. If the monitored interfaces (e.g., LAN or WAN) are not seeing outbound traffic or if the exporter is not configured to monitor the interfaces where outbound traffic occurs, these fields will remain zero.
  - Verify which interfaces are being monitored in the NetFlow settings. For example, if you're only monitoring the LAN interface for ingress traffic, outbound traffic (e.g., to WAN) may not be captured.
4. **Collector or Template Misinterpretation**:
  - Some NetFlow collectors may misinterpret or fail to properly parse the flow records, leading to `OUT_BYTES` and `OUT_PACKETS` appearing as zero. Ensure your collector (e.g., nfdump, SolarWinds, or OPNsense's Insight) is fully compatible with NetFlow v9 and correctly processes the templates sent by OPNsense.
  - Check the collector's logs for any errors about missing or unprocessed templates, as this can cause data to be misreported.
5. **NetFlow v9 Template Behavior**:
  - In NetFlow v9, templates are sent periodically to describe the structure of flow records, and they may include fields like `OUT_BYTES` and `OUT_PACKETS` even if no data is currently being reported for those fields. This is standard behavior, as the template defines the *potential* data that can be sent, not necessarily what is always populated. If egress data is not being captured due to configuration, these fields will remain zero but are still included in the template for compatibility and future use.
### Why Are `OUT_BYTES` and `OUT_PACKETS` Included in Templates?
- **Template Design**: NetFlow v9 templates are designed to be flexible and reusable. OPNsense includes `OUT_BYTES` and `OUT_PACKETS` in the templates to support scenarios where egress traffic is monitored. Even if your current configuration doesn't populate these fields, the template is structured to accommodate both ingress and egress data, ensuring compatibility with collectors that expect these fields.
- **Future-Proofing**: Including these fields allows you to enable egress capturing later without changing the template structure, avoiding disruptions to the collector. If you reconfigure OPNsense to capture egress traffic, the same template can be used, and `OUT_BYTES`/`OUT_PACKETS` will start being populated.
- **Standard Compliance**: The `OUT_BYTES` and `OUT_PACKETS` fields (Field IDs 23 and 24 in NetFlow v9) are part of the standard NetFlow v9 specification. Including them in the template aligns with best practices and ensures interoperability with various collectors.
- **Bidirectional Analysis**: Many network monitoring use cases require bidirectional flow data to analyze traffic patterns fully. Including these fields in the template supports such use cases, even if they're not currently active in your setup.
### Steps to Diagnose and Resolve
1. **Verify NetFlow Configuration**:
  - Go to **Reporting > NetFlow** in the OPNsense web interface.
  - Check the **Interfaces** setting to see if egress capturing is enabled. You may need to explicitly enable "Export flows from this interface" for egress on the relevant interfaces (e.g., WAN or LAN).
  - Ensure the "Direction" setting includes both ingress and egress if you want bidirectional data.
2. **Check Interface Traffic**:
  - Confirm that the interfaces being monitored have outbound traffic. For example, if you're monitoring the LAN interface, ensure it's seeing traffic leaving the interface (e.g., to the WAN or another network).
  - Use OPNsense's traffic graphs or packet captures to verify outbound traffic on the monitored interfaces.
3. **Inspect Flow Records**:
  - Use a tool like `nfdump` or Wireshark to capture and inspect the raw NetFlow packets from OPNsense. Look at the flow records to confirm whether `OUT_BYTES` and `OUT_PACKETS` are indeed zero or if the collector is misinterpreting the data.
  - Example `nfdump` command to view flows:
    ```bash
    nfdump -r <flow-file> -o extended
    ```
    Check for non-zero values in the `out bytes` and `out packets` columns.
4. **Test Egress Traffic**:
  - Generate test traffic (e.g., using `ping`, `wget`, or file transfers) to ensure outbound traffic is flowing through the monitored interface. Then check if `OUT_BYTES` and `OUT_PACKETS` are populated in the flow records.
5. **Collector Compatibility**:
  - Verify that your NetFlow collector supports NetFlow v9 and bidirectional flows. Some older collectors may not handle `OUT_BYTES`/`OUT_PACKETS` correctly.
  - Update the collector to the latest version or test with a different collector (e.g., nfdump or FlowViewer) to rule out collector issues.
6. **Consider Disabling NAT Optimization**:
  - If NAT is causing OPNsense to suppress egress data to avoid double-counting, you can experiment with disabling this optimization. In the NetFlow settings, look for options like "Disable NAT" or adjust the flow capture to include egress explicitly. Be cautious, as this may lead to double-counting in some scenarios.
7. **Consult OPNsense Documentation or Community**:
  - Check the OPNsense documentation for NetFlow (available under **Reporting** in the official docs) or the OPNsense forum for similar issues. Users have reported cases where egress data wasn't captured due to specific interface or NAT configurations.
### Additional Notes
- **Performance Impact**: Capturing both ingress and egress flows increases the load on the OPNsense system and the volume of flow data sent to the collector. If egress data isn't critical for your use case, keeping it disabled (resulting in zero `OUT_BYTES`/`OUT_PACKETS`) can reduce resource usage.
- **Alternative Tools**: If bidirectional data is critical, consider using other OPNsense features like the Traffic Insight tool or external collectors like nProbe, which can provide more granular control over flow data.
If you'd like, I can guide you through specific configuration changes in OPNsense or help analyze flow data from your collector. Could you share:
- Which interfaces are configured for NetFlow?
- Whether you've enabled ingress, egress, or both in the settings?
- The NetFlow collector you're using?
This will help pinpoint the issue further.

[franco]

People don't even read my replies sometimes... how are they supposed to read this long and inconclusive answer on the fact that the code will never set it to anything other than zero in the first place as the OP points out (if that is true, I haven't checked)?  ;)


Cheers,
Franco

[shermdog]

That's my goflow issue :)

It is indeed set to zero in the FreeBSD source.  Why? Who knows.  But that's how it's been for at least the last 15 years.

[franco]

Just for reference the commit is https://github.com/opnsense/src/commit/5dcd9c106

If I read this correctly softflowd also does not fill out bytes/packets:

https://github.com/irino/softflowd/blob/0260261a2302ce635cea1c02c9ea72d7b808d8f6/netflow9.c#L51-L78

There's probably something in the standard that defines this as something implementations cannot provide so it's left out.


Cheers,
Franco

[krby]

It would be fine to leave the fields out. Including them and setting them to zero makes little sense.

The GROK/LLM answer wasn't useful. I know the code is doing it. I'm saying it is not correct to do so.