+
Skip to content

Conversation

kirillgarbar
Copy link
Contributor

@kirillgarbar kirillgarbar commented Sep 15, 2025

Summary by Sourcery

Enable creating zero-copy locks across multiple or all replicas by adding new CLI options, updating internal replica discovery and zookeeper path handling, and improving remote table querying

New Features:

  • Add --replicas and --all-replicas options to the create-zero-copy-locks command

Enhancements:

  • Iterate over replicated tables and specified replicas to create locks using a new helper that retrieves zookeeper paths and validates active replicas
  • Filter tables by ReplicatedMergeTree engine by default and error when no replicated tables match the filters
  • Introduce get_remote_table_for_hosts utility to wrap remote/remoteSecure functions for querying remote replicas in list_parts and clean_object_storage
  • Refactor zero-copy internals to pass zookeeper_path explicitly when constructing part lock paths

Tests:

  • Add feature scenarios to validate create-zero-copy-locks with multiple and all replicas, including examples for different replica selection modes

Copy link
Contributor

sourcery-ai bot commented Sep 15, 2025

Reviewer's Guide

This PR enhances the create-zero-copy-locks command with multi-replica and all-replicas support, refactors zookeeper path handling and remote table utilities, and updates feature tests to cover the new replica selection options.

Sequence diagram for create-zero-copy-locks with multi-replica support

sequenceDiagram
    participant User
    participant CLI
    participant TableInfo
    participant ZKHelper
    participant LockCreator

    User->>CLI: Run create-zero-copy-locks (with --replicas or --all-replicas)
    CLI->>TableInfo: list_tables (filter for Replicated tables)
    CLI->>ZKHelper: _get_replicas_and_zk_path (get zk_path, replicas)
    ZKHelper-->>CLI: Return zk_path, list of replicas
    loop For each replica in replicas
        CLI->>LockCreator: create_zero_copy_locks(..., replica, zk_path)
    end
Loading

ER diagram for table and replica relationship in zero-copy locks

erDiagram
    TABLE {
        string database
        string name
        string engine
    }
    REPLICA {
        string fqdn
        string zookeeper_path
    }
    TABLE ||--o{ REPLICA : has
    REPLICA ||--o{ ZERO_COPY_LOCK : creates
    ZERO_COPY_LOCK {
        string part_name
        string path_in_zk
    }
Loading

Class diagram for updated replica and zookeeper path handling

classDiagram
    class CLI {
        +create_zk_locks_command(..., replicas, all_replicas)
    }
    class ZKHelper {
        +_get_replicas_and_zk_path(ctx, table, replicas, all_replicas)
    }
    class LockCreator {
        +create_zero_copy_locks(ctx, disk, table_info, partition_id, part_id, replica, zk_path, dry_run)
    }
    class PartPathHelper {
        +_get_part_path_in_zk(part, zookeeper_path, replica)
    }
    CLI --> ZKHelper
    CLI --> LockCreator
    LockCreator --> PartPathHelper
Loading

Class diagram for get_remote_table_for_hosts utility

classDiagram
    class Utils {
        +get_remote_table_for_hosts(ctx, table, replicas) : str
    }
    class ClickhouseClient {
        +user : str
        +password : str
        +check_port(port)
    }
    Utils --> ClickhouseClient
Loading

File-Level Changes

Change Details Files
Extend create-zero-copy-locks command to support multiple replicas and all-replicas flags
  • Add a new Replica selection options group with --replicas and --all-replicas flags
  • Remove the single --replica option
  • Filter only ReplicatedMergeTree tables and loop over each replica when creating locks
  • Introduce _get_replicas_and_zk_path helper for fetching zk path and replica list
ch_tools/chadmin/cli/zookeeper_group.py
Update feature tests for replica selection scenarios
  • Add Scenario Outline for --replicas and --all-replicas combinations
  • Update existing tests to use --replicas flag
  • Include a non-replicated MergeTree table to verify filtering
tests/features/chadmin_zero_copy.feature
Support remote table functions in list_parts and clean_object_storage
  • Add get_remote_table_for_hosts util to generate remote()/remoteSecure() calls
  • Enable list_parts to accept a remote_replica argument and pass sensitive credentials
  • Refactor clean_object_storage to use get_remote_table_for_hosts instead of manual port checks
ch_tools/chadmin/internal/part.py
ch_tools/chadmin/internal/utils.py
ch_tools/common/commands/clean_object_storage.py
Refactor zero-copy path construction to use provided zookeeper path
  • Add zookeeper_path parameter to create_zero_copy_locks signature
  • Simplify _get_part_path_in_zk to construct paths using the passed zk path and replica
ch_tools/chadmin/internal/zero_copy.py

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey there - I've reviewed your changes and they look great!

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location> `ch_tools/chadmin/cli/zookeeper_group.py:644` </location>
<code_context>
-        create_zero_copy_locks(
-            ctx, disk, table_info, partition_id, part_id, replica, dry_run
-        )
+    replicas_ = replicas.split(",")
+    for replica in replicas_:
+        if replica not in replicas_list:
</code_context>

<issue_to_address>
**suggestion:** Splitting replicas by comma does not trim whitespace, which may cause issues with user input.

Use `[r.strip() for r in replicas.split(",")]` to ensure whitespace is removed from each replica, preventing issues with user input containing spaces.

```suggestion
    replicas_ = [r.strip() for r in replicas.split(",")]
```
</issue_to_address>

### Comment 2
<location> `ch_tools/chadmin/internal/utils.py:96-97` </location>
<code_context>
+    else:
+        raise RuntimeError("For using remote() table function tcp port must be defined")
+
+    replicas_str = ",".join(replicas)
+    return f"{remote_clause}('{replicas_str}', {table}, '{user_name}', '{{user_password}}')"
+
</code_context>

<issue_to_address>
**suggestion (bug_risk):** Joining replicas without handling empty or invalid entries may result in malformed remote table function calls.

Validate that the replicas list is non-empty and all entries are valid hostnames before joining.

```suggestion
    import re

    if not replicas or not all(isinstance(replica, str) and replica.strip() for replica in replicas):
        raise ValueError("Replicas list must be non-empty and contain valid hostnames.")

    hostname_regex = re.compile(
        r"^(?=.{1,253}$)(?!-)[A-Za-z0-9-]{1,63}(?<!-)(\.(?!-)[A-Za-z0-9-]{1,63}(?<!-))*$"
    )
    for replica in replicas:
        if not hostname_regex.match(replica):
            raise ValueError(f"Invalid hostname in replicas list: '{replica}'")

    replicas_str = ",".join(replicas)
    return f"{remote_clause}('{replicas_str}', {table}, '{user_name}', '{{user_password}}')"
```
</issue_to_address>

### Comment 3
<location> `ch_tools/chadmin/internal/utils.py:81` </location>
<code_context>
         )


+def get_remote_table_for_hosts(ctx: Context, table: str, replicas: list[str]) -> str:
+    """
+    Get remote/remoteSecure function for a table with given remote replicas.
</code_context>

<issue_to_address>
**suggestion (review_instructions):** Please ensure that the construction of the remote/remoteSecure table function matches ClickHouse's documented behavior, including argument order and escaping.

ClickHouse's remote/remoteSecure table functions require careful attention to argument order and escaping, especially for user/password and host lists. Please double-check that the generated string matches the documented format and handles edge cases (e.g., hostnames with special characters, empty user/password).

<details>
<summary>Review instructions:</summary>

**Path patterns:** `**/*.py`

**Instructions:**
Code that deals with ClickHouse should take into account its implementation https://github.com/ClickHouse/ClickHouse

</details>
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

create_zero_copy_locks(
ctx, disk, table_info, partition_id, part_id, replica, dry_run
)
replicas_ = replicas.split(",")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: Splitting replicas by comma does not trim whitespace, which may cause issues with user input.

Use [r.strip() for r in replicas.split(",")] to ensure whitespace is removed from each replica, preventing issues with user input containing spaces.

Suggested change
replicas_ = replicas.split(",")
replicas_ = [r.strip() for r in replicas.split(",")]

)


def get_remote_table_for_hosts(ctx: Context, table: str, replicas: list[str]) -> str:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (review_instructions): Please ensure that the construction of the remote/remoteSecure table function matches ClickHouse's documented behavior, including argument order and escaping.

ClickHouse's remote/remoteSecure table functions require careful attention to argument order and escaping, especially for user/password and host lists. Please double-check that the generated string matches the documented format and handles edge cases (e.g., hostnames with special characters, empty user/password).

Review instructions:

Path patterns: **/*.py

Instructions:
Code that deals with ClickHouse should take into account its implementation https://github.com/ClickHouse/ClickHouse

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants
点击 这是indexloc提供的php浏览器服务,不要输入任何密码和下载