这是indexloc提供的服务,不要输入任何密码
Skip to content

Conversation

@nv-aefimov
Copy link
Contributor

This changeset adds the ability to emit JSON with a new --json flag.

The first few commits are cleanup from pending merge requests on upstream.
The first JSON-related commit that adds the ability to emit JSON for the top-level
context object doesn't provide much useful information - it simply emits the existing
text output as an object value. Each following commit incrementally converts more of
the objects in this tool to support JSON output, so the --json flag becomes increasingly
useful as the commits in this series progress.

The existing text output remains unchanged up to whitespace - there are a few bugs with
leading/trailing whitespace. Some are preserved, others are not. But the non-whitespace
output should be byte-for-byte identical. You can verify this with diff -w. If
changing the output slightly is acceptable, several commits can be somewhat simplified.
Note: I checked this a few months ago - there may have been some changes introduced
upstream since then that slightly change the output format.

32     HWS: Emit modify_actions as a list of actions
31     HWS: Emit dr_ste as a dict
30     HWS: Make definer fields printable
29     HWS: Make dr_parse_matcher_match_template printable
28     HWS: Make dr_parse_matcher_resources printable
27     HWS: Make dr_parse_matcher_attr printable
26     HWS: Make dr_parse_matcher_action_template printable
25     HWS: Make dr_parse_context_caps printable
24     HWS: Make dr_parse_context_attr printable
23     HWS: Make dr_parse_context_send_ring printable
22     HWS: Make dr_rule printable
21     HWS: Return parsed rules list from dr_parse_rules
20     HWS: Simplify dr_parse_rule tree_print
19     HWS: Make dr_matcher printable
18     HWS: Fix typo in dr_matcher
17     HWS: Make dr_table printable
16     HWS: Make dr_parse_context_send_engine printable
15     HWS: Make dr_context printable
14     HWS: Add CLI flag to emit JSON
13     HWS: Define base class for Printable objects
12     HWS: Add function to print object with indentation
11     HWS: Clean up dr_csv_get_obj
10     HWS: Simplify dr_parse_table.fix_data
9      HWS: Set the minimum Python version to 3.9
8      HWS: Clean up print_unsupported_obj_list
7      HWS: Add error checking to the CSV parser
6      HWS & SWS: Improve boolean comparison style
5      HWS & SWS:Improve equality comparison style
4      HWS: Remove tmp_file from config
3      HWS: Store tmp file path as Path object
2      HWS: Open the binary file in read-only mode
1      HWS: Keep the binary file open between each read

This has 2 advantages:
1) Performance: fewer syscalls due to no longer repeatedly closing and
 reopening the same file
2) Exception safety, guaranteeing that this file will be closed at the
 end of the `with` clause

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
We do not write to this file, so there is no reason to open it
in `r+` (read + update) mode.

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
No need to convert it to a string - convert it only where necessary.

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Now that we always open() the file path directly, we don't need to keep
a reference to the file descriptor.

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Replace "== None" with "is None". According to PEP8, this is the correct
way to compare to singletons like None. No behavioural changes.

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Equality comparisons with boolean literals are unnecessary and are
considered bad style as per PEP8. Replace them with the more pythonic
implicit conversion to boolean in if statements. No behavioural changes.

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
If the CSV file is malformed, the parser will print an error message and
skip the malformed line rather than crashing.

Also add type annotations to the parsing function.

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Stop using a global variable for the unsupported object list. Instead,
return and pass the value as a local variable into the function. No
behavioural changes - the printed output is exactly the same. If
changing the output format is fine, then the join call can be removed.

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
The script now checks the Python version and exits if it's lower than
the minimum supported version. The alternative is cryptic stack traces
that users may not understand - it is inevitable that we use *some*
features that are not available in older versions of Python. The printed
error looks like this:
```
> python38 hws/mlx_hw_steering_parser.py
Error: mlx_hw_steering_parser.py requires Python 3.9 or newer
```

Before this changeset, the minimum supported version was at least 3.6
due to using f-strings. We can validate this by running
`pipx run vermin -vvv hws/mlx_hw_steering_parser.py`.

With this change, we set the minimum to 3.9. This should be
reasonable as 3.9 is currently the oldest supported version. It was
released on 2020-10-05 and will be EOL on 2025-10. The DOCA dev
container is based on Ubuntu 22.04, which has Python 3.10.

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Remove the redundant checks for None values. `dict.get()` already has a
parameter for the default value if the key is not found in the
dictionary.

Note that this isn't exactly equivalent to the original if the
retrieved value really is `None`. However, this should not happen with
the current code.

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Make the global constant UPPERCASE to make it clear where the data comes
from in the dr_csv_get_obj function. Use the res_type variable instead
of splitting the line twice.

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Up to this point, object formatting was always been handled by the
tree_print method of each object. This is not ideal because it returns
only a string, which would mean that emitting the object as JSON would
require a 2nd function to duplicate much of the logic in that function.
To prevent that, we want to be able to get the object as a dict, and be
able to format that dict as either JSON (introduced later in this commit
series) or as a string (existing functionality).

This commit adds the first step in that direction: a function that
pretty-prints an object as a string. This function corresponds almost
exactly to the current format of the tree_print method, so it is
suitable as a drop-in replacement. The output is not char-for-char
identical, but it should be close enough that human readers should not
notice a difference.

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Add a small base class to keep track of which classes we've converted to
support emitting as JSON.

The dump_obj method provides a single known interface for emitting an
object as a dict. The sturcture of the fields is not the same as the
struture we wish to print (to preserve the existing structure of the
text output), so it has a parameter to control whether to transform the
object to make it suitable for printing. This transformation is done
within the method rather than by the caller since the caller doesn't
have visiblity into the type of the object. Rather than passing around a
function with the returned dict, just do the transformation in the
method itself.

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
This commit simply emits the existing text-based tree as a string in a
JSON object. The following commits will incrementally convert the
objects inside the parse context from text-only to dictionary-based
representations. The dictionary-based representations can then be used
to either print the object as tree_print does now, or emit the object as
JSON for machine processing.

The text-based output remains identical.

The current JSON output is simply:
```
{"context": "..."}
```

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
This will enable formatting the data as JSON. Since dr_context is the
top-level object, update the main() code to call the new dump_obj
function instead of tree_print. It is a drop-in replacement for
tree_print when transform_for_print is true, so this should have almost
no user-visible changes.

In this commit, most objects are emitted as string values in the JSON
dictionary. Only the top-level context is actually emitted as an object.

The current JSON output looks like:
```
{
    "data": {
        "mlx5dr_debug_res_type": "4000",
        "id": "0x55dc5151a4e0",
        "hws_support": "True",
        "dev_name": "mlx5_0",
        "debug_version": "1.0.NV_HWS_25.04"
    },
    "attr": "Attr: ctx_id 0x55dc5151a4e0, pd_num 22, queues 5, queue_size 1024, vhca_id 0\n",
    "caps": "...",
    "send_engine": [
        "Send_engine 0: ...",
        "Send_engine 1: ...",
        ...
    ],
    "tables": [
        "Table 0x7fccd5dbfc40: ...",
        "Table 0x7fb453ad2520: ...",
        ...
    ]
}
```

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
The text-based output remains identical.
The JSON output of send_engine looks like this:

```
[
    {
        "data": {
            "mlx5dr_debug_res_type": "4003",
            "ctx_id": "0x55dc5151a4e0",
            "id": "0",
            "used_entries": "0",
            "th_entries": "1024",
            "rings": "1",
            "num_entries": "1024",
            "err": "0",
            "ci": "1",
            "pi": "1",
            "completed_mask": "1023"
        },
        "send_ring": [
            "Send_ring 0: ...",
            ...
        ]
    },
    ...
]
```

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
The text output is identical.

The JSON output looks like this:
```
{
    "data": {
        "mlx5dr_debug_res_type": "4100",
        "id": "0x7fb453ad3c30",
        "ctx_id": "0x55dc5151a4e0",
        "ft_id": "0x9b",
        "type": "FDB_UNIFIED",
        "fw_ft_type": "4",
        "level": "1",
        "local_ft_id": "0x0",
        "rx_icm_addr": "0x80000508",
        "tx_icm_addr": "0x80000509",
        "local_rx_icm_addr": "0x0",
        "local_tx_icm_addr": "0x0",
        "miss_tbl": "0x7fb453ad2520",
        "ib_port": "0",
        "vport": "0",
        "other_vport": "0"
    },
    "matchers": [...]
}
```

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Also add a return type annotation.

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
As usual, the text output is unchanged.
The JSON output looks like this (example with statistics):

```
{
    "data": {
        "mlx5dr_debug_res_type": "4200",
        "id": "0x7fccd5dbf060",
        "tbl_id": "0x7fccd4bcd550",
        "num_of_mt": "1",
        "end_ft_id": "0x82",
        "col_matcher_id": "0x0",
        "match_rtc_0_id": "53",
        "match_ste_0_id": "52",
        "match_rtc_1_id": "53",
        "match_ste_1_id": "52",
        "action_rtc_0_id": "0",
        "action_ste_0_id": "-1",
        "action_rtc_1_id": "0",
        "action_ste_1_id": "-1",
        "aliased_rtc_0_id": "0",
        "rx_icm_addr": "0x80000444",
        "tx_icm_addr": "0x80000445"
    },
    "attr": "...",
    "matcher_resources": "Resources: ... \n",
    "statistics": "Statistics: ... \n",
    "match_templates": [
        "Match_template 0x7fccd5dbef70: ... \n"
    ],
    "action_templates": [
        "Action_template 0x7fccd5dbf160: ... \n"
    ],
    "rules": "..."
}
```

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Inline the nested function since it only has 1 call site.

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
This will enable formatting the data as JSON.

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
This will enable formatting the data as JSON.
For example:
```
{
    "tbl_type": "FDB_UNIFIED",
    "stes": [
        "STE 0xc5803000: ...",
        ...
    ]
}
```

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Emit the send_ring as a object that is printable or emittable as JSON.
The text output should be identical to the previous version.

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Example JSON output:

```
{
    "mlx5dr_debug_res_type": "4204",
    "id": "0x7fb453ad4540",
    "matcher_id": "0x7fb453ad40f0",
    "only_term": "0",
    "num_of_action_stes": "2",
    "num_of_actions": "3",
    "action_combinations": [
        "L2_TO_TNL_L2",
        "(null)",
        "CTR"
    ]
}
```

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
For example:
```
{
    "mlx5dr_debug_res_type": "4201",
    "matcher_id": "0x7fccd5dbff50",
    "priority": "0x1",
    "mode": "RULE",
    "sz_row_log": "0",
    "sz_col_log": "0",
    "use_rule_idx": "0",
    "flow_src": "default",
    "insertion": "HASH",
    "distribution": "HASH",
    "match": "Default",
    "isolated": "",
    "log_sz": "0X0"
}
```

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
For example, a matcher with col_matcher_id:
```
{
    "match_rtc_0_id": "38",
    "match_ste_0_id": "37",
    "match_rtc_1_id": "38",
    "match_ste_1_id": "37",
    "action_rtc_0_id": "37",
    "action_ste_0_id": "36",
    "action_rtc_1_id": "37",
    "action_ste_1_id": "36",
    "col_matcher_data": {
        "mlx5dr_debug_res_type": "4200",
        "id": "0x7fccd5dba200",
        "tbl_id": "0x7fccd4b968b0",
        "num_of_mt": "1",
        "end_ft_id": "0x74",
        "col_matcher_id": "0x0",
        "match_rtc_0_id": "39",
        "match_ste_0_id": "38",
        "match_rtc_1_id": "39",
        "match_ste_1_id": "38",
        "action_rtc_0_id": "0",
        "action_ste_0_id": "-1",
        "action_rtc_1_id": "0",
        "action_ste_1_id": "-1",
        "aliased_rtc_0_id": "0",
        "rx_icm_addr": "0x800003d5",
        "tx_icm_addr": "0x800003d6"
    }
}
```

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
This is not 100% identical to the previous output, but it's very close.
For example, before:
```
Match_template 0x7fb453ad41f0:
    Match fields:
        ipv6_addr_dst_o: 0xffffffffffffffffffffffffffffffff
     flags 0, fc_sz 4, fcr_sz 0, fcc_sz 0
```

After:
```
Match_template 0x7fb453ad41f0:
    match_fields
        ipv6_addr_dst_o: 0xffffffffffffffffffffffffffffffff
    flags 0, fc_sz 4, fcr_sz 0, fcc_sz 0
```

And in JSON form:
```
{
    "data": {
        "mlx5dr_debug_res_type": "4202",
        "id": "0x7fb453ad41f0",
        "matcher_id": "0x7fb453ad40f0",
        "fc_sz": "4",
        "flags": "0",
        "fcr_sz": "0",
        "fcc_sz": "0"
    },
    "match_fields": "ipv6_addr_dst_o: 0xffffffffffffffffffffffffffffffff"
}
```

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
The text format is very similar, but not identical. If there is a
formatter available, it is always invoked to produce a human-readable
interpretation of the fields.

The JSON format intentionally leaves the values as ints rather than
converting to hexadecimal to make machine processing easier.

Text format before:
```
protocol_next_header_o: 0xff
src_port_o: 0xffff
dst_port_o: 0xffff
src_ip_o: 0xffffffff
dst_ip_o: 0xffffffff
```

Text format after:
```
protocol_next_header_o: 0xff
src_port_o: 0xffff
dst_port_o: 0xffff
src_ip_o: 255.255.255.255 (0xffffffff)
dst_ip_o: 255.255.255.255 (0xffffffff)
```

JSON format after:
```
{
    "protocol_next_header_o": 255,
    "src_port_o": 65535,
    "dst_port_o": 65535,
    "src_ip_o": 4294967295,
    "dst_ip_o": 4294967295
}
```

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Emitting the info as a dict will enable formatting the data as JSON. We
can't make dr_ste Printable because the dump_object function needs to
know whether the STE is the last one in the array, so the function
signature won't match.

Example JSON output:

```
{
    "data": {
        "mlx5dr_debug_res_type": "4930",
        "id": "0xc0000030",
        "fw_ste_id": "71",
        "raw_ste": "0x..."
    },
    "fields": "Tag:\n    ipv6_addr_dst_o: 0x20010db000332211\n",
    "miss": "Miss address: 0x80000511\n",
    "actions": "...",
    "raw_ste": "..."
}
```

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>

HWS: Emit dr_ste actions as dict

The JSON output for STEs now looks like this:

```
{
    "data": {
        "mlx5dr_debug_res_type": "4930",
        "id": "0xc0000030",
        "fw_ste_id": "71",
        "raw_ste": "0x..."
    },
    "fields": {
        "ipv6_address_63_32_dst_o": 536939952,
        "ipv6_address_31_0_dst_o": 3351057
    },
    "miss": {
        "gvmi": 0,
        "gvmi_str": "0",
        "index": 2147484945
    },
    "actions": {...}
}
```

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Unfortunately, we cannot easily emit modifyHeader patterns as a dict
since they are in their parsed form in the CSV file and we don't want to
break anything.

Modify_header actions looks like this:
```
[
    "",
    {
        "type": "Modify action list",
        "pat_idx": 8454172,
        "num_of_actions": 6,
        "arg_idx": 655361
    },
    "    Set: field: smac_15_0_o (0x9) length: 0x10 left_shifter: 0x10 (06093010)\n",
    "    Set: field: dmac_15_0_o (0x1) length: 0x10 left_shifter: 0x10 (06013010)\n",
    "    Add: field: time_to_live_hop_limit_o (0xe) length: 0x8 left_shifter: 0x8 (070e2808)\n",
    "    Set: field: smac_47_16_o (0x8) length: 0x20 left_shifter: 0x0 (06082000)\n",
    "    Set: field: dmac_47_16_o (0x0) length: 0x20 left_shifter: 0x0 (06002000)\n",
    "     (0000000000000000)\n"
]
```

and other actions looks like this:

```
[
    "",
    {
        "type": "Set",
        "field": "regc_3 (0x93)",
        "length": 1,
        "left_shifter": 0,
        "inline_data": 1
    }
]
```

Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant