-
Notifications
You must be signed in to change notification settings - Fork 28
HWS: Add JSON output #159
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
nv-aefimov
wants to merge
32
commits into
Mellanox:master
Choose a base branch
from
nv-aefimov:dev/aefimov/json-2
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
HWS: Add JSON output #159
+692
−469
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This has 2 advantages: 1) Performance: fewer syscalls due to no longer repeatedly closing and reopening the same file 2) Exception safety, guaranteeing that this file will be closed at the end of the `with` clause Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
We do not write to this file, so there is no reason to open it in `r+` (read + update) mode. Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
No need to convert it to a string - convert it only where necessary. Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Now that we always open() the file path directly, we don't need to keep a reference to the file descriptor. Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Replace "== None" with "is None". According to PEP8, this is the correct way to compare to singletons like None. No behavioural changes. Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Equality comparisons with boolean literals are unnecessary and are considered bad style as per PEP8. Replace them with the more pythonic implicit conversion to boolean in if statements. No behavioural changes. Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
If the CSV file is malformed, the parser will print an error message and skip the malformed line rather than crashing. Also add type annotations to the parsing function. Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Stop using a global variable for the unsupported object list. Instead, return and pass the value as a local variable into the function. No behavioural changes - the printed output is exactly the same. If changing the output format is fine, then the join call can be removed. Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
The script now checks the Python version and exits if it's lower than the minimum supported version. The alternative is cryptic stack traces that users may not understand - it is inevitable that we use *some* features that are not available in older versions of Python. The printed error looks like this: ``` > python38 hws/mlx_hw_steering_parser.py Error: mlx_hw_steering_parser.py requires Python 3.9 or newer ``` Before this changeset, the minimum supported version was at least 3.6 due to using f-strings. We can validate this by running `pipx run vermin -vvv hws/mlx_hw_steering_parser.py`. With this change, we set the minimum to 3.9. This should be reasonable as 3.9 is currently the oldest supported version. It was released on 2020-10-05 and will be EOL on 2025-10. The DOCA dev container is based on Ubuntu 22.04, which has Python 3.10. Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Remove the redundant checks for None values. `dict.get()` already has a parameter for the default value if the key is not found in the dictionary. Note that this isn't exactly equivalent to the original if the retrieved value really is `None`. However, this should not happen with the current code. Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Make the global constant UPPERCASE to make it clear where the data comes from in the dr_csv_get_obj function. Use the res_type variable instead of splitting the line twice. Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Up to this point, object formatting was always been handled by the tree_print method of each object. This is not ideal because it returns only a string, which would mean that emitting the object as JSON would require a 2nd function to duplicate much of the logic in that function. To prevent that, we want to be able to get the object as a dict, and be able to format that dict as either JSON (introduced later in this commit series) or as a string (existing functionality). This commit adds the first step in that direction: a function that pretty-prints an object as a string. This function corresponds almost exactly to the current format of the tree_print method, so it is suitable as a drop-in replacement. The output is not char-for-char identical, but it should be close enough that human readers should not notice a difference. Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Add a small base class to keep track of which classes we've converted to support emitting as JSON. The dump_obj method provides a single known interface for emitting an object as a dict. The sturcture of the fields is not the same as the struture we wish to print (to preserve the existing structure of the text output), so it has a parameter to control whether to transform the object to make it suitable for printing. This transformation is done within the method rather than by the caller since the caller doesn't have visiblity into the type of the object. Rather than passing around a function with the returned dict, just do the transformation in the method itself. Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
This commit simply emits the existing text-based tree as a string in a
JSON object. The following commits will incrementally convert the
objects inside the parse context from text-only to dictionary-based
representations. The dictionary-based representations can then be used
to either print the object as tree_print does now, or emit the object as
JSON for machine processing.
The text-based output remains identical.
The current JSON output is simply:
```
{"context": "..."}
```
Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
This will enable formatting the data as JSON. Since dr_context is the
top-level object, update the main() code to call the new dump_obj
function instead of tree_print. It is a drop-in replacement for
tree_print when transform_for_print is true, so this should have almost
no user-visible changes.
In this commit, most objects are emitted as string values in the JSON
dictionary. Only the top-level context is actually emitted as an object.
The current JSON output looks like:
```
{
"data": {
"mlx5dr_debug_res_type": "4000",
"id": "0x55dc5151a4e0",
"hws_support": "True",
"dev_name": "mlx5_0",
"debug_version": "1.0.NV_HWS_25.04"
},
"attr": "Attr: ctx_id 0x55dc5151a4e0, pd_num 22, queues 5, queue_size 1024, vhca_id 0\n",
"caps": "...",
"send_engine": [
"Send_engine 0: ...",
"Send_engine 1: ...",
...
],
"tables": [
"Table 0x7fccd5dbfc40: ...",
"Table 0x7fb453ad2520: ...",
...
]
}
```
Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
The text-based output remains identical.
The JSON output of send_engine looks like this:
```
[
{
"data": {
"mlx5dr_debug_res_type": "4003",
"ctx_id": "0x55dc5151a4e0",
"id": "0",
"used_entries": "0",
"th_entries": "1024",
"rings": "1",
"num_entries": "1024",
"err": "0",
"ci": "1",
"pi": "1",
"completed_mask": "1023"
},
"send_ring": [
"Send_ring 0: ...",
...
]
},
...
]
```
Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
The text output is identical.
The JSON output looks like this:
```
{
"data": {
"mlx5dr_debug_res_type": "4100",
"id": "0x7fb453ad3c30",
"ctx_id": "0x55dc5151a4e0",
"ft_id": "0x9b",
"type": "FDB_UNIFIED",
"fw_ft_type": "4",
"level": "1",
"local_ft_id": "0x0",
"rx_icm_addr": "0x80000508",
"tx_icm_addr": "0x80000509",
"local_rx_icm_addr": "0x0",
"local_tx_icm_addr": "0x0",
"miss_tbl": "0x7fb453ad2520",
"ib_port": "0",
"vport": "0",
"other_vport": "0"
},
"matchers": [...]
}
```
Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Also add a return type annotation. Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
As usual, the text output is unchanged.
The JSON output looks like this (example with statistics):
```
{
"data": {
"mlx5dr_debug_res_type": "4200",
"id": "0x7fccd5dbf060",
"tbl_id": "0x7fccd4bcd550",
"num_of_mt": "1",
"end_ft_id": "0x82",
"col_matcher_id": "0x0",
"match_rtc_0_id": "53",
"match_ste_0_id": "52",
"match_rtc_1_id": "53",
"match_ste_1_id": "52",
"action_rtc_0_id": "0",
"action_ste_0_id": "-1",
"action_rtc_1_id": "0",
"action_ste_1_id": "-1",
"aliased_rtc_0_id": "0",
"rx_icm_addr": "0x80000444",
"tx_icm_addr": "0x80000445"
},
"attr": "...",
"matcher_resources": "Resources: ... \n",
"statistics": "Statistics: ... \n",
"match_templates": [
"Match_template 0x7fccd5dbef70: ... \n"
],
"action_templates": [
"Action_template 0x7fccd5dbf160: ... \n"
],
"rules": "..."
}
```
Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Inline the nested function since it only has 1 call site. Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
This will enable formatting the data as JSON. Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
This will enable formatting the data as JSON.
For example:
```
{
"tbl_type": "FDB_UNIFIED",
"stes": [
"STE 0xc5803000: ...",
...
]
}
```
Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Emit the send_ring as a object that is printable or emittable as JSON. The text output should be identical to the previous version. Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Example JSON output:
```
{
"mlx5dr_debug_res_type": "4204",
"id": "0x7fb453ad4540",
"matcher_id": "0x7fb453ad40f0",
"only_term": "0",
"num_of_action_stes": "2",
"num_of_actions": "3",
"action_combinations": [
"L2_TO_TNL_L2",
"(null)",
"CTR"
]
}
```
Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
For example:
```
{
"mlx5dr_debug_res_type": "4201",
"matcher_id": "0x7fccd5dbff50",
"priority": "0x1",
"mode": "RULE",
"sz_row_log": "0",
"sz_col_log": "0",
"use_rule_idx": "0",
"flow_src": "default",
"insertion": "HASH",
"distribution": "HASH",
"match": "Default",
"isolated": "",
"log_sz": "0X0"
}
```
Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
For example, a matcher with col_matcher_id:
```
{
"match_rtc_0_id": "38",
"match_ste_0_id": "37",
"match_rtc_1_id": "38",
"match_ste_1_id": "37",
"action_rtc_0_id": "37",
"action_ste_0_id": "36",
"action_rtc_1_id": "37",
"action_ste_1_id": "36",
"col_matcher_data": {
"mlx5dr_debug_res_type": "4200",
"id": "0x7fccd5dba200",
"tbl_id": "0x7fccd4b968b0",
"num_of_mt": "1",
"end_ft_id": "0x74",
"col_matcher_id": "0x0",
"match_rtc_0_id": "39",
"match_ste_0_id": "38",
"match_rtc_1_id": "39",
"match_ste_1_id": "38",
"action_rtc_0_id": "0",
"action_ste_0_id": "-1",
"action_rtc_1_id": "0",
"action_ste_1_id": "-1",
"aliased_rtc_0_id": "0",
"rx_icm_addr": "0x800003d5",
"tx_icm_addr": "0x800003d6"
}
}
```
Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
This is not 100% identical to the previous output, but it's very close.
For example, before:
```
Match_template 0x7fb453ad41f0:
Match fields:
ipv6_addr_dst_o: 0xffffffffffffffffffffffffffffffff
flags 0, fc_sz 4, fcr_sz 0, fcc_sz 0
```
After:
```
Match_template 0x7fb453ad41f0:
match_fields
ipv6_addr_dst_o: 0xffffffffffffffffffffffffffffffff
flags 0, fc_sz 4, fcr_sz 0, fcc_sz 0
```
And in JSON form:
```
{
"data": {
"mlx5dr_debug_res_type": "4202",
"id": "0x7fb453ad41f0",
"matcher_id": "0x7fb453ad40f0",
"fc_sz": "4",
"flags": "0",
"fcr_sz": "0",
"fcc_sz": "0"
},
"match_fields": "ipv6_addr_dst_o: 0xffffffffffffffffffffffffffffffff"
}
```
Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
The text format is very similar, but not identical. If there is a
formatter available, it is always invoked to produce a human-readable
interpretation of the fields.
The JSON format intentionally leaves the values as ints rather than
converting to hexadecimal to make machine processing easier.
Text format before:
```
protocol_next_header_o: 0xff
src_port_o: 0xffff
dst_port_o: 0xffff
src_ip_o: 0xffffffff
dst_ip_o: 0xffffffff
```
Text format after:
```
protocol_next_header_o: 0xff
src_port_o: 0xffff
dst_port_o: 0xffff
src_ip_o: 255.255.255.255 (0xffffffff)
dst_ip_o: 255.255.255.255 (0xffffffff)
```
JSON format after:
```
{
"protocol_next_header_o": 255,
"src_port_o": 65535,
"dst_port_o": 65535,
"src_ip_o": 4294967295,
"dst_ip_o": 4294967295
}
```
Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Emitting the info as a dict will enable formatting the data as JSON. We
can't make dr_ste Printable because the dump_object function needs to
know whether the STE is the last one in the array, so the function
signature won't match.
Example JSON output:
```
{
"data": {
"mlx5dr_debug_res_type": "4930",
"id": "0xc0000030",
"fw_ste_id": "71",
"raw_ste": "0x..."
},
"fields": "Tag:\n ipv6_addr_dst_o: 0x20010db000332211\n",
"miss": "Miss address: 0x80000511\n",
"actions": "...",
"raw_ste": "..."
}
```
Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
HWS: Emit dr_ste actions as dict
The JSON output for STEs now looks like this:
```
{
"data": {
"mlx5dr_debug_res_type": "4930",
"id": "0xc0000030",
"fw_ste_id": "71",
"raw_ste": "0x..."
},
"fields": {
"ipv6_address_63_32_dst_o": 536939952,
"ipv6_address_31_0_dst_o": 3351057
},
"miss": {
"gvmi": 0,
"gvmi_str": "0",
"index": 2147484945
},
"actions": {...}
}
```
Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Unfortunately, we cannot easily emit modifyHeader patterns as a dict
since they are in their parsed form in the CSV file and we don't want to
break anything.
Modify_header actions looks like this:
```
[
"",
{
"type": "Modify action list",
"pat_idx": 8454172,
"num_of_actions": 6,
"arg_idx": 655361
},
" Set: field: smac_15_0_o (0x9) length: 0x10 left_shifter: 0x10 (06093010)\n",
" Set: field: dmac_15_0_o (0x1) length: 0x10 left_shifter: 0x10 (06013010)\n",
" Add: field: time_to_live_hop_limit_o (0xe) length: 0x8 left_shifter: 0x8 (070e2808)\n",
" Set: field: smac_47_16_o (0x8) length: 0x20 left_shifter: 0x0 (06082000)\n",
" Set: field: dmac_47_16_o (0x0) length: 0x20 left_shifter: 0x0 (06002000)\n",
" (0000000000000000)\n"
]
```
and other actions looks like this:
```
[
"",
{
"type": "Set",
"field": "regc_3 (0x93)",
"length": 1,
"left_shifter": 0,
"inline_data": 1
}
]
```
Signed-off-by: Alexander Efimov <aefimov@nvidia.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This changeset adds the ability to emit JSON with a new
--jsonflag.The first few commits are cleanup from pending merge requests on upstream.
The first JSON-related commit that adds the ability to emit JSON for the top-level
context object doesn't provide much useful information - it simply emits the existing
text output as an object value. Each following commit incrementally converts more of
the objects in this tool to support JSON output, so the
--jsonflag becomes increasinglyuseful as the commits in this series progress.
The existing text output remains unchanged up to whitespace - there are a few bugs with
leading/trailing whitespace. Some are preserved, others are not. But the non-whitespace
output should be byte-for-byte identical. You can verify this with
diff -w. Ifchanging the output slightly is acceptable, several commits can be somewhat simplified.
Note: I checked this a few months ago - there may have been some changes introduced
upstream since then that slightly change the output format.