Flaky 02730_dictionary_hashed_load_factor_element_count #84468
Description
I've only found 3 instances of the test failing by itself:
2025-03-07 03:30:58 Reason: result differs with reference:
2025-03-07 03:30:58 --- /repo/tests/queries/0_stateless/02730_dictionary_hashed_load_factor_element_count.reference 2025-03-07 03:07:26.812016513 +0500
2025-03-07 03:30:58 +++ /repo/tests/queries/0_stateless/02730_dictionary_hashed_load_factor_element_count.stdout 2025-03-07 03:30:58.365948586 +0500
2025-03-07 03:30:58 @@ -1,2 +1,2 @@
2025-03-07 03:30:58 dict_sharded 1 1000000 0.4768
2025-03-07 03:30:58 -dict_sharded_multi 5 1000000 0.4768
2025-03-07 03:30:58 +dict_sharded_multi 5 997672 0.4757
2025-03-07 03:30:58
2025-03-07 03:30:58
2025-03-07 03:30:58 Settings used in the test: --max_insert_threads 1 --group_by_two_level_threshold 1 --group_by_two_level_threshold_bytes 50000000 --distributed_aggregation_memory_efficient 1 --fsync_metadata 1 --output_format_parallel_formatting 0 --input_format_parallel_parsing 1 --min_chunk_bytes_for_parallel_parsing 3654019 --max_read_buffer_size 523180 --prefer_localhost_replica 0 --max_block_size 77132 --max_joined_block_size_rows 39975 --max_threads 1 --optimize_append_index 0 --optimize_if_chain_to_multiif 1 --optimize_if_transform_strings_to_enum 0 --optimize_read_in_order 0 --optimize_or_like_chain 0 --optimize_substitute_columns 0 --enable_multiple_prewhere_read_steps 1 --read_in_order_two_level_merge_threshold 67 --optimize_aggregation_in_order 0 --aggregation_in_order_max_block_bytes 15402392 --use_uncompressed_cache 0 --min_bytes_to_use_direct_io 2993450232 --min_bytes_to_use_mmap_io 10737418240 --local_filesystem_read_method pread --remote_filesystem_read_method read --local_filesystem_read_prefetch 0 --filesystem_cache_segments_batch_size 100 --read_from_filesystem_cache_if_exists_otherwise_bypass_cache 0 --throw_on_error_from_cache_on_write_operations 1 --remote_filesystem_read_prefetch 0 --allow_prefetched_read_pool_for_remote_filesystem 1 --filesystem_prefetch_max_memory_usage 64Mi --filesystem_prefetches_limit 0 --filesystem_prefetch_min_bytes_for_single_read_task 8Mi --filesystem_prefetch_step_marks 0 --filesystem_prefetch_step_bytes 100Mi --compile_aggregate_expressions 1 --compile_sort_description 1 --merge_tree_coarse_index_granularity 3 --optimize_distinct_in_order 0 --max_bytes_before_remerge_sort 272436228 --min_compress_block_size 503324 --max_compress_block_size 2702456 --merge_tree_compact_parts_min_granules_to_multibuffer_read 1 --optimize_sorting_by_input_stream_properties 1 --http_response_buffer_size 2879053 --http_wait_end_of_query True --enable_memory_bound_merging_of_aggregation_results 1 --min_count_to_compile_expression 3 --min_count_to_compile_aggregate_expression 0 --min_count_to_compile_sort_description 0 --session_timezone America/Mazatlan --use_page_cache_for_disks_without_file_cache True --page_cache_inject_eviction False --merge_tree_read_split_ranges_into_intersecting_and_non_intersecting_injection_probability 0.84 --prefer_external_sort_block_bytes 1 --cross_join_min_rows_to_compress 0 --cross_join_min_bytes_to_compress 100000000 --min_external_table_block_size_bytes 1 --max_parsing_threads 0 --optimize_functions_to_subcolumns 1 --parallel_replicas_local_plan 0 --query_plan_join_swap_table auto --enable_vertical_final 0 --optimize_extract_common_expressions 1 --use_query_condition_cache 0 --max_bytes_before_external_sort 0 --max_bytes_before_external_group_by 0 --max_bytes_ratio_before_external_sort 0.1 --max_bytes_ratio_before_external_group_by 0.2
2025-03-07 03:30:58
2025-03-07 03:30:58 MergeTree settings used in test: --ratio_of_defaults_for_sparse_serialization 0.0 --prefer_fetch_merged_part_size_threshold 10737418240 --vertical_merge_algorithm_min_rows_to_activate 304161 --vertical_merge_algorithm_min_columns_to_activate 1 --allow_vertical_merges_from_compact_to_wide_parts 0 --min_merge_bytes_to_use_direct_io 507248432 --index_granularity_bytes 4524382 --merge_max_block_size 15749 --index_granularity 13448 --min_bytes_for_wide_part 0 --marks_compress_block_size 25849 --primary_key_compress_block_size 66575 --replace_long_file_name_to_hash 0 --max_file_name_length 128 --min_bytes_for_full_part_storage 73234873 --compact_parts_max_bytes_to_buffer 194486728 --compact_parts_max_granules_to_buffer 67 --compact_parts_merge_max_bytes_to_prefetch_part 13468075 --cache_populated_by_fetch 1 --concurrent_part_removal_threshold 0 --old_parts_lifetime 393 --prewarm_mark_cache 0 --use_const_adaptive_granularity 1 --enable_index_granularity_compression 0 --enable_block_number_column 1 --enable_block_offset_column 0 --use_primary_key_cache 1 --prewarm_primary_key_cache 1
2025-03-07 03:30:58
2025-03-07 03:30:58 Database: test_bjfxml7p
2025-05-07 11:09:46 Reason: result differs with reference:
2025-05-07 11:09:46 --- /repo/tests/queries/0_stateless/02730_dictionary_hashed_load_factor_element_count.reference 2025-05-07 10:46:49.141867234 +0300
2025-05-07 11:09:46 +++ /repo/tests/queries/0_stateless/02730_dictionary_hashed_load_factor_element_count.stdout 2025-05-07 11:09:46.079485961 +0300
2025-05-07 11:09:46 @@ -1,2 +1,2 @@
2025-05-07 11:09:46 dict_sharded 1 1000000 0.4768
2025-05-07 11:09:46 -dict_sharded_multi 5 1000000 0.4768
2025-05-07 11:09:46 +dict_sharded_multi 5 997208 0.4755
2025-05-07 11:09:46
2025-05-07 11:09:46
2025-05-07 11:09:46 Settings used in the test: --max_insert_threads 3 --group_by_two_level_threshold 1818 --group_by_two_level_threshold_bytes 3109236 --distributed_aggregation_memory_efficient 0 --fsync_metadata 1 --output_format_parallel_formatting 0 --input_format_parallel_parsing 0 --min_chunk_bytes_for_parallel_parsing 17679467 --max_read_buffer_size 1035423 --prefer_localhost_replica 0 --max_block_size 91055 --max_joined_block_size_rows 25414 --max_threads 3 --optimize_append_index 1 --use_hedged_requests 1 --optimize_if_chain_to_multiif 1 --optimize_if_transform_strings_to_enum 0 --optimize_read_in_order 1 --optimize_or_like_chain 1 --optimize_substitute_columns 0 --enable_multiple_prewhere_read_steps 1 --read_in_order_two_level_merge_threshold 79 --optimize_aggregation_in_order 0 --aggregation_in_order_max_block_bytes 48073045 --use_uncompressed_cache 1 --min_bytes_to_use_direct_io 10737418240 --min_bytes_to_use_mmap_io 10737418240 --local_filesystem_read_method mmap --remote_filesystem_read_method read --local_filesystem_read_prefetch 0 --filesystem_cache_segments_batch_size 10 --read_from_filesystem_cache_if_exists_otherwise_bypass_cache 1 --throw_on_error_from_cache_on_write_operations 0 --remote_filesystem_read_prefetch 1 --allow_prefetched_read_pool_for_remote_filesystem 1 --filesystem_prefetch_max_memory_usage 64Mi --filesystem_prefetches_limit 0 --filesystem_prefetch_min_bytes_for_single_read_task 1Mi --filesystem_prefetch_step_marks 50 --filesystem_prefetch_step_bytes 100Mi --compile_aggregate_expressions 1 --compile_sort_description 0 --merge_tree_coarse_index_granularity 2 --optimize_distinct_in_order 1 --max_bytes_before_remerge_sort 431102917 --min_compress_block_size 450787 --max_compress_block_size 2813080 --merge_tree_compact_parts_min_granules_to_multibuffer_read 32 --optimize_sorting_by_input_stream_properties 1 --http_response_buffer_size 275801 --http_wait_end_of_query False --enable_memory_bound_merging_of_aggregation_results 1 --min_count_to_compile_expression 3 --min_count_to_compile_aggregate_expression 3 --min_count_to_compile_sort_description 0 --session_timezone America/Mazatlan --use_page_cache_for_disks_without_file_cache True --page_cache_inject_eviction False --merge_tree_read_split_ranges_into_intersecting_and_non_intersecting_injection_probability 0.8 --prefer_external_sort_block_bytes 0 --cross_join_min_rows_to_compress 100000000 --cross_join_min_bytes_to_compress 0 --min_external_table_block_size_bytes 0 --max_parsing_threads 1 --optimize_functions_to_subcolumns 1 --parallel_replicas_local_plan 0 --query_plan_join_swap_table false --enable_vertical_final 1 --optimize_extract_common_expressions 0 --use_query_condition_cache 0 --secondary_indices_enable_bulk_filtering 1 --use_skip_indexes_if_final 1 --max_bytes_before_external_sort 10737418240 --max_bytes_before_external_group_by 10737418240 --max_bytes_ratio_before_external_sort 0 --max_bytes_ratio_before_external_group_by 0 --use_skip_indexes_if_final_exact_mode 1
2025-05-07 11:09:46
2025-05-07 11:09:46 MergeTree settings used in test: --ratio_of_defaults_for_sparse_serialization 0.0 --prefer_fetch_merged_part_size_threshold 3587224740 --vertical_merge_algorithm_min_rows_to_activate 747950 --vertical_merge_algorithm_min_columns_to_activate 100 --allow_vertical_merges_from_compact_to_wide_parts 1 --min_merge_bytes_to_use_direct_io 6458459814 --index_granularity_bytes 17441018 --merge_max_block_size 23330 --index_granularity 44285 --min_bytes_for_wide_part 386287558 --marks_compress_block_size 14893 --primary_key_compress_block_size 17983 --replace_long_file_name_to_hash 1 --max_file_name_length 62 --min_bytes_for_full_part_storage 0 --compact_parts_max_bytes_to_buffer 155696204 --compact_parts_max_granules_to_buffer 253 --compact_parts_merge_max_bytes_to_prefetch_part 19038870 --cache_populated_by_fetch 0 --concurrent_part_removal_threshold 8 --old_parts_lifetime 202 --prewarm_mark_cache 0 --use_const_adaptive_granularity 1 --enable_index_granularity_compression 1 --enable_block_number_column 1 --enable_block_offset_column 1 --use_primary_key_cache 0 --prewarm_primary_key_cache 0
2025-05-07 11:09:46
2025-05-07 11:09:46 Database: test_xb1dw5gq
2025-07-24 01:12:52 Reason: result differs with reference:
2025-07-24 01:12:52 --- /home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/tests/queries/0_stateless/02730_dictionary_hashed_load_factor_element_count.reference 2025-07-24 01:02:42.855837056 +1000
2025-07-24 01:12:52 +++ /home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/tests/queries/0_stateless/02730_dictionary_hashed_load_factor_element_count.stdout 2025-07-24 01:12:51.959695161 +1000
2025-07-24 01:12:52 @@ -1,2 +1,2 @@
2025-07-24 01:12:52 dict_sharded 1 1000000 0.4768
2025-07-24 01:12:52 -dict_sharded_multi 5 1000000 0.4768
2025-07-24 01:12:52 +dict_sharded_multi 5 997709 0.4757
2025-07-24 01:12:52
2025-07-24 01:12:52
2025-07-24 01:12:52 Settings used in the test: --max_insert_threads 1 --group_by_two_level_threshold 1 --group_by_two_level_threshold_bytes 1 --distributed_aggregation_memory_efficient 0 --fsync_metadata 1 --output_format_parallel_formatting 1 --input_format_parallel_parsing 0 --min_chunk_bytes_for_parallel_parsing 12515848 --max_read_buffer_size 688405 --prefer_localhost_replica 1 --max_block_size 84236 --max_joined_block_size_rows 31719 --max_threads 3 --optimize_append_index 0 --use_hedged_requests 1 --optimize_if_chain_to_multiif 0 --optimize_if_transform_strings_to_enum 1 --optimize_read_in_order 1 --optimize_or_like_chain 1 --optimize_substitute_columns 1 --enable_multiple_prewhere_read_steps 0 --read_in_order_two_level_merge_threshold 14 --optimize_aggregation_in_order 0 --aggregation_in_order_max_block_bytes 17081875 --use_uncompressed_cache 1 --min_bytes_to_use_direct_io 1 --min_bytes_to_use_mmap_io 10737418240 --local_filesystem_read_method pread_threadpool --remote_filesystem_read_method read --local_filesystem_read_prefetch 1 --filesystem_cache_segments_batch_size 3 --read_from_filesystem_cache_if_exists_otherwise_bypass_cache 1 --throw_on_error_from_cache_on_write_operations 0 --remote_filesystem_read_prefetch 0 --allow_prefetched_read_pool_for_remote_filesystem 1 --filesystem_prefetch_max_memory_usage 64Mi --filesystem_prefetches_limit 0 --filesystem_prefetch_min_bytes_for_single_read_task 1Mi --filesystem_prefetch_step_marks 50 --filesystem_prefetch_step_bytes 100Mi --compile_expressions 1 --compile_aggregate_expressions 0 --compile_sort_description 0 --merge_tree_coarse_index_granularity 21 --optimize_distinct_in_order 1 --max_bytes_before_remerge_sort 837357390 --min_compress_block_size 2796468 --max_compress_block_size 2410605 --merge_tree_compact_parts_min_granules_to_multibuffer_read 53 --optimize_sorting_by_input_stream_properties 0 --http_response_buffer_size 5816702 --http_wait_end_of_query True --enable_memory_bound_merging_of_aggregation_results 0 --min_count_to_compile_expression 3 --min_count_to_compile_aggregate_expression 3 --min_count_to_compile_sort_description 0 --session_timezone America/Hermosillo --use_page_cache_for_disks_without_file_cache True --page_cache_inject_eviction False --merge_tree_read_split_ranges_into_intersecting_and_non_intersecting_injection_probability 0.43 --prefer_external_sort_block_bytes 100000000 --cross_join_min_rows_to_compress 1 --cross_join_min_bytes_to_compress 1 --min_external_table_block_size_bytes 100000000 --max_parsing_threads 1 --optimize_functions_to_subcolumns 0 --parallel_replicas_local_plan 1 --query_plan_join_swap_table auto --enable_vertical_final 0 --optimize_extract_common_expressions 1 --use_async_executor_for_materialized_views 1 --use_query_condition_cache 0 --secondary_indices_enable_bulk_filtering 1 --use_skip_indexes_if_final 1 --max_bytes_before_external_sort 0 --max_bytes_before_external_group_by 0 --max_bytes_ratio_before_external_sort 0.14 --max_bytes_ratio_before_external_group_by 0.0 --use_skip_indexes_if_final_exact_mode 1
2025-07-24 01:12:52
2025-07-24 01:12:52 MergeTree settings used in test: --ratio_of_defaults_for_sparse_serialization 1.0 --prefer_fetch_merged_part_size_threshold 1403372642 --vertical_merge_algorithm_min_rows_to_activate 1000000 --vertical_merge_algorithm_min_columns_to_activate 57 --allow_vertical_merges_from_compact_to_wide_parts 0 --min_merge_bytes_to_use_direct_io 1 --index_granularity_bytes 14388523 --merge_max_block_size 19960 --index_granularity 45865 --min_bytes_for_wide_part 1073741824 --marks_compress_block_size 42210 --primary_key_compress_block_size 77014 --replace_long_file_name_to_hash 0 --max_file_name_length 128 --min_bytes_for_full_part_storage 536870912 --compact_parts_max_bytes_to_buffer 315746724 --compact_parts_max_granules_to_buffer 1 --compact_parts_merge_max_bytes_to_prefetch_part 16062054 --cache_populated_by_fetch 0 --concurrent_part_removal_threshold 88 --old_parts_lifetime 10 --prewarm_mark_cache 1 --use_const_adaptive_granularity 1 --enable_index_granularity_compression 0 --enable_block_number_column 1 --enable_block_offset_column 0 --use_primary_key_cache 0 --prewarm_primary_key_cache 0
2025-07-24 01:12:52
2025-07-24 01:12:52 Database: test_cq2v3p70
When looking into it it seems like a bug, not a flaky test: #84263 (comment)
- The source table has loaded 1M rows.
dict_shardeddictionary loads 1M rows.- Occasionally (literally 3 times in thousands of runs)
dict_sharded_multiloads less rows than expected.
The only difference between the 2 dictionaries is that _multi has more columns so it might be a coincidence.
The 3 failures happened under the TSAN runs but with different configs, which might also be a coincidence.
I've even run the tests thousands of times locally and it hasn't failed, even with the system under heavy load.