Improve logging of rebalancer and recovery #586
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
6b1057d to
64cc837
Compare
Serpentian
reviewed
Aug 20, 2025
Serpentian
reviewed
Aug 20, 2025
5a8b3f8 to
f5c25f7
Compare
Serpentian
reviewed
Aug 25, 2025
Serpentian
reviewed
Aug 25, 2025
04c506f to
ccff54f
Compare
Serpentian
requested changes
Sep 15, 2025
46add65 to
a1c095b
Compare
Serpentian
requested changes
Sep 30, 2025
Before this patch "Finish bucket recovery step ..." logs were printed at the end of recovery even if no buckets were successfully recovered. It led to unnecessary log records. This patch fixes the issue by adding an additional check for the number of recovered buckets. Part of tarantool#212 NO_DOC=bugfix
871197d to
489b425
Compare
489b425 to
fce8f28
Compare
Serpentian
reviewed
Oct 13, 2025
This patch introduces logging of buckets' ids which were recovered during recovery stage of storage. Part of tarantool#212 NO_DOC=bugfix
This patch adds rebalancer routes' logging. The log file now includes information about the source storage, the number of buckets, and the destination storage where the buckets will be moved. Since the rebalancer service has changed logging of routes that were sent, we change the `rebalancer/rebalancer.test.lua` and `rebalancer/stress_add_remove_several_rs.test.lua` tests. Part of tarantool#212 NO_DOC=bugfix
Before this patch the function `rebalancer_download_states` didn't return information about replicaset from which the states could not be downloaded. As a result, the log "Some buckets are not active ..." lacks of valuable information about unhealthy replicaset. Now, we return `(replicaset.id, nil)` instead of `nil` in case when rebalancer can't download state from this replicaset. Also we add replicaset.id in "Some buckets are not active ..." log. Also we change `rebalancer/rebalancer.test.lua` test which expected the old "Some buckets are not active" log without replicaset.id. Closes tarantool#212 NO_DOC=bugfix
fce8f28 to
dcfa98a
Compare
Serpentian
reviewed
Oct 20, 2025
| local hanged_bucket_id_2 = vtest.storage_first_bucket(g.replica_1_a) | ||
| start_bucket_move(g.replica_1_a, g.replica_2_a, hanged_bucket_id_2) | ||
| t.helpers.retrying({}, function() | ||
| t.assert(g.replica_1_a:grep_log('Error during recovery of bucket 1')) |
There was a problem hiding this comment.
Nit: it's better to wakeup the recovery there in the loop. And in other places, please. E.g. in the test_rebalancer_routes_logging rebalancer should be waken up, when waiting for the Balanced ok
| {timeout = consts.REBALANCER_GET_STATE_TIMEOUT}) | ||
| if state == nil then | ||
| return | ||
| return nil, replicaset.id |
There was a problem hiding this comment.
This should be carefully rebased, now error is returned from the rebalancer_download_states. I'd propose to introduce the new error code, which will say, that the storage doesn't have all bucket in proper state, and set err there if it's nil (it'll happen, in case the function really returned nil).
In that err (or any other there) we'll just set the err.replicaset, smth like that:
Line 1653 in b4adaea
And we'll print the whole err, not just the replicaset's id
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before this patch "Finish bucket recovery step ..." logs were printed at
the end of recovery even if no buckets were successfully recovered, it led
to unnecessary log entries. This patch fixes the issue by adding an
additional check for the number of recovered buckets.
Closes #212
NO_DOC=bugfix