Test PR #6
Open
Test PR #6
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Reviewer's GuideRefactor database migration flow to use Zookeeper transactions in place of a temporary database approach, introduce helper routines for node and metadata management, update CLI command flow, enhance table and database metadata handling, and adjust feature tests. Sequence Diagram: Database Migration - New First Replica PathsequenceDiagram
participant ActorUser as User
participant CMD as migrate_engine_command
participant CDN as migration.create_database_nodes
participant MFR as migration.migrate_as_first_replica
participant ULMFR as migration._update_local_metadata_first_replica
participant ZK as Zookeeper
participant CH as ClickHouse
participant FS as LocalFileSystem
ActorUser->>CMD: Run chadmin database migrate-engine --database DBNAME
CMD->>CDN: create_database_nodes(DBNAME)
CDN->>ZK: Check/Create /clickhouse path
CDN->>ZK: TXN: Create common ZK nodes for DBNAME (e.g., /log, /replicas, /metadata)
alt create_database_nodes succeeds (DBNAME not yet in ZK)
CMD->>MFR: migrate_as_first_replica(DBNAME)
MFR->>ZK: _generate_counter()
MFR->>ZK: TXN Start
MFR->>ZK: Create /clickhouse/DBNAME/first_replica_database_name
MFR->>ZK: Create /clickhouse/DBNAME/log/query-{counter} structure
MFR->>ZK: Create /clickhouse/DBNAME/replicas/{shard}|{replica} structure
MFR->>ZK: Create /clickhouse/DBNAME/metadata/{table} for each table
MFR->>ZK: TXN Commit
MFR->>ULMFR: _update_local_metadata_first_replica(DBNAME)
ULMFR->>CH: DETACH DATABASE DBNAME
ULMFR->>FS: Update DBNAME.sql (set Replicated engine properties)
ULMFR->>CH: ATTACH DATABASE DBNAME
else NodeExistsError from create_database_nodes
CMD-->>ActorUser: (Proceed to non-first replica flow)
end
Class Diagram: Updated Metadata Handling and Migration LogicclassDiagram
class DatabaseMetadata {
+database_name: String
+database_engine: DatabaseEngine
+replica_path: String
+shard: String
+replica_name: String
+update_metadata_file()
+set_replicated() // New method
}
class migration_module {
<<Module>>
+migrate_as_first_replica(ctx, migrating_database) // Refactored
+migrate_as_non_first_replica(ctx, migrating_database) // Refactored
+create_database_nodes(ctx, migrating_database) // New
+is_table_schema_equal(ctx, db_name, table_name, path) // New
+_update_local_metadata_first_replica(ctx, migrating_database) // New
+_check_tables_consistent(ctx, database_name, tables_info) // New
+_generate_counter(ctx, zk, migrating_database) // New
+_check_result_txn(results) // New
+get_shard_and_replica_from_macros(ctx) // New
+_create_first_replica_database_name(ctx, txn, migrating_database) // New
+_create_query_node(ctx, txn, migrating_database, counter) // New
+_create_database_replica(ctx, txn, migrating_database) // New
+_create_database_metadata_nodes(ctx, txn, migrating_database) // New
+_attach_dbs(ctx, dbs) // New
+_get_host_id(ctx, migrating_database, replica) // New
+_change_tables_uuid(ctx, tables, database_name) bool // Modified (new return type, internal logic)
+_get_tables_info_and_detach(ctx, database_name) dict // Modified (used in new flow)
+_detach_dbs(ctx, dbs)
}
class table_module {
<<Module>>
+read_local_table_metadata(ctx, table_local_metadata_path) String // New
+change_table_uuid(ctx, database, table, table_local_metadata_path, new_uuid, attached) // Modified (internal path construction)
}
migration_module ..> DatabaseMetadata : Uses
migration_module ..> table_module : Uses
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
There was a problem hiding this comment.
Hey @Alex-Burmak - I've reviewed your changes - here's some feedback:
- Consider refactoring the repeated ZooKeeper transaction building (multiple txn.create calls) into a data‐driven helper or builder to reduce duplication and improve readability.
- Extract the commit‐and‐check‐result pattern into a single helper to DRY up the repeated txn.commit() and _check_result_txn logic.
- Replace the hardcoded UUID substring logic in is_table_schema_equal with a dedicated parser or regex to make the intent clearer and avoid magic numbers.
Here's what I looked at during the review
- 🟡 General issues: 1 issue found
- 🟢 Security: all looks good
- 🟢 Review instructions: all looks good
- 🟢 Testing: all looks good
- 🟢 Documentation: all looks good
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| txn.create( | ||
| path=format_path( | ||
| ctx, | ||
| f"/clickhouse/{migrating_database}/log/query-0000000001/finished/{shard}|{replica}", |
There was a problem hiding this comment.
issue: Use dynamic counter instead of hard-coded query index
Pass the dynamic counter to this function and use it in the path to ensure it matches the generated query ID.
| logging.info( | ||
| "New metadata for node from mapping table:\n{}\n===", target_metadata | ||
| ) | ||
| txn.create( |
There was a problem hiding this comment.
issue (code-quality): Use f-string instead of string concatenation [×4] (use-fstring-for-concatenation)
| metadata_path = row["metadata_path"] | ||
|
|
||
| if match_str_ch_version(get_version(ctx), "25.1"): | ||
| metadata_path = CLICKHOUSE_PATH + "/" + metadata_path |
There was a problem hiding this comment.
suggestion (code-quality): Use f-string instead of string concatenation [×2] (use-fstring-for-concatenation)
Suggested change
| metadata_path = CLICKHOUSE_PATH + "/" + metadata_path | |
| metadata_path = f"{CLICKHOUSE_PATH}/{metadata_path}" |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary by Sourcery
Implement ZooKeeper-based ClickHouse database migration with separate first and non-first replica flows, refactor CLI and metadata handling, introduce schema consistency checks and extend BDD tests for migration scenarios
New Features:
Enhancements:
chadmin migrateCLI to use create_database_nodes and improve error handling for replica determinationTests:
Chores: