2026-03-20 18:14:24,955 [ 4644 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:53, check_args_and_update_paths) 2026-03-20 18:14:24,956 [ 4644 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:79, check_args_and_update_paths) 2026-03-20 18:14:24,956 [ 4644 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:90, check_args_and_update_paths) 2026-03-20 18:14:24,956 [ 4644 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/ClickHouse/ClickHouse/ci/tmp/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:92, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_swmcii --privileged --dns-search='.' --memory=30709026816 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/ci/tmp/clickhouse:/clickhouse --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=e7a502a8ddfb733e8385_amd -e DOCKER_HELPER_TAG=5dc43a6382f0055b4d8e_amd -e DOCKER_BASE_TAG=9c789665fa1f2c3b60a8_amd -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d716376396_amd -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=a2be58f091e1427a1e32_amd -e DOCKER_MYSQL_JAVA_CLIENT_TAG=f9bb23fd3568fb51064b_amd -e DOCKER_MYSQL_JS_CLIENT_TAG=6a2cfd28cf040f57b0aa_amd -e DOCKER_ARROWFLIGHT_SERVER_TAG=8ec78de57d59a65d2bfb_amd -e DOCKER_MYSQL_PHP_CLIENT_TAG=2e3bd01324fab1b1e95d_amd -e DOCKER_NGINX_DAV_TAG=aa33b6ad5665b723c6e4_amd -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=9f8b31d2e09ab8aaf242_amd -e DOCKER_PYTHON_BOTTLE_TAG=426daa4ec42501225a52_amd -e DOCKER_BASE_WITH_UNITY_CATALOG_TAG=b2310980ad8a5093f676_amd -e DOCKER_BASE_WITH_HMS_TAG=859811a46af684b6fc6b_amd -e DOCKER_MYSQL_DOTNET_CLIENT_TAG=2efb0784cb8ed9630acd_amd -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 1 --count 5 -rfEps --run-id=0 --color=no --durations=0 --report-log=flaky0_0.jsonl --report-log-exclude-logs-on-passed-tests test_s3_cluster/test.py --random-order -vvv " altinityinfra/integration-tests-runner:75b9a401cb1d94a0286c_amd '. Unable to find image 'altinityinfra/integration-tests-runner:75b9a401cb1d94a0286c_amd' locally 75b9a401cb1d94a0286c_amd: Pulling from altinityinfra/integration-tests-runner 60d98d907669: Pulling fs layer ed9b52f32429: Pulling fs layer 9e40f5c5c563: Pulling fs layer 78eac1ec92bf: Pulling fs layer 4f4fb700ef54: Pulling fs layer 167c3b6a68eb: Pulling fs layer 4de93e11a551: Pulling fs layer 61111994dd2e: Pulling fs layer 597ef5e86392: Pulling fs layer 4a8c24950dc1: Pulling fs layer bf36b48af1f2: Pulling fs layer 167c3b6a68eb: Waiting 6ed3d4cac62d: Pulling fs layer 16fe4d2441cb: Pulling fs layer baf04471f756: Pulling fs layer 4f4fb700ef54: Waiting 6e81d8b3573f: Pulling fs layer baf04471f756: Waiting 32d2b1221562: Pulling fs layer 597ef5e86392: Waiting 23beb18f5a32: Pulling fs layer cee0a232e8df: Pulling fs layer aa1eaece7e64: Pulling fs layer 4de93e11a551: Waiting 23beb18f5a32: Waiting cee0a232e8df: Waiting 285af1f2d705: Pulling fs layer 32d2b1221562: Waiting 71f5dc65a9dc: Pulling fs layer 3fe0badf7a57: Pulling fs layer 2871093841bb: Pulling fs layer 6ed3d4cac62d: Waiting 3fe0badf7a57: Waiting ac9e392cf437: Pulling fs layer 285af1f2d705: Waiting 4a8c24950dc1: Waiting 2871093841bb: Waiting ac9e392cf437: Waiting bf36b48af1f2: Waiting ac8aa3555032: Pulling fs layer 6a6d993275cc: Pulling fs layer 32eefb2cd325: Pulling fs layer 6e81d8b3573f: Waiting 6a6d993275cc: Waiting 3d887e1ab7b3: Pulling fs layer ac8aa3555032: Waiting c356e54edd32: Pulling fs layer 32eefb2cd325: Waiting d7855536ed1d: Pulling fs layer 3d887e1ab7b3: Waiting c356e54edd32: Waiting 505e7e1bbae8: Pulling fs layer d7855536ed1d: Waiting aa1eaece7e64: Waiting ed9b52f32429: Verifying Checksum 60d98d907669: Verifying Checksum 60d98d907669: Download complete 9e40f5c5c563: Verifying Checksum 9e40f5c5c563: Download complete 4f4fb700ef54: Verifying Checksum 4f4fb700ef54: Download complete 167c3b6a68eb: Download complete 4de93e11a551: Verifying Checksum 4de93e11a551: Download complete 78eac1ec92bf: Verifying Checksum 78eac1ec92bf: Download complete 60d98d907669: Pull complete ed9b52f32429: Pull complete 4a8c24950dc1: Verifying Checksum 4a8c24950dc1: Download complete 9e40f5c5c563: Pull complete 597ef5e86392: Verifying Checksum 597ef5e86392: Download complete 61111994dd2e: Verifying Checksum 61111994dd2e: Download complete 6ed3d4cac62d: Download complete 16fe4d2441cb: Download complete baf04471f756: Verifying Checksum baf04471f756: Download complete 6e81d8b3573f: Verifying Checksum 6e81d8b3573f: Download complete 23beb18f5a32: Verifying Checksum 23beb18f5a32: Download complete 32d2b1221562: Verifying Checksum 32d2b1221562: Download complete cee0a232e8df: Verifying Checksum cee0a232e8df: Download complete 78eac1ec92bf: Pull complete 4f4fb700ef54: Pull complete 167c3b6a68eb: Pull complete 4de93e11a551: Pull complete aa1eaece7e64: Verifying Checksum aa1eaece7e64: Download complete 71f5dc65a9dc: Verifying Checksum 71f5dc65a9dc: Download complete 3fe0badf7a57: Verifying Checksum 3fe0badf7a57: Download complete bf36b48af1f2: Verifying Checksum bf36b48af1f2: Download complete 61111994dd2e: Pull complete 285af1f2d705: Verifying Checksum 285af1f2d705: Download complete 597ef5e86392: Pull complete 4a8c24950dc1: Pull complete ac9e392cf437: Verifying Checksum ac9e392cf437: Download complete 6a6d993275cc: Verifying Checksum 6a6d993275cc: Download complete 32eefb2cd325: Verifying Checksum 32eefb2cd325: Download complete 3d887e1ab7b3: Download complete c356e54edd32: Download complete d7855536ed1d: Verifying Checksum d7855536ed1d: Download complete 505e7e1bbae8: Verifying Checksum ac8aa3555032: Verifying Checksum ac8aa3555032: Download complete bf36b48af1f2: Pull complete 6ed3d4cac62d: Pull complete 16fe4d2441cb: Pull complete baf04471f756: Pull complete 6e81d8b3573f: Pull complete 32d2b1221562: Pull complete 23beb18f5a32: Pull complete cee0a232e8df: Pull complete aa1eaece7e64: Pull complete 285af1f2d705: Pull complete 71f5dc65a9dc: Pull complete 3fe0badf7a57: Pull complete 2871093841bb: Pull complete ac9e392cf437: Pull complete ac8aa3555032: Pull complete 6a6d993275cc: Pull complete 32eefb2cd325: Pull complete 3d887e1ab7b3: Pull complete c356e54edd32: Pull complete d7855536ed1d: Pull complete 505e7e1bbae8: Pull complete Digest: sha256:22f263ac55e92c59c5d84186fe96825e7faa827f180215bc9805e7330c61505e Status: Downloaded newer image for altinityinfra/integration-tests-runner:75b9a401cb1d94a0286c_amd Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache Using --random-order-bucket=module Using --random-order-seed=422553 rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: timeout-2.3.1, repeat-0.9.3, order-1.0.0, reportlog-0.4.0, xdist-3.5.0, random-order-1.1.1 timeout: 900.0s timeout method: signal timeout func_only: False created: 1/1 worker 1 worker [115 items] scheduling tests via LoadFileScheduling test_s3_cluster/test.py::test_hive_partitioning[0-2-5] [gw0] [ 0%] PASSED test_s3_cluster/test.py::test_hive_partitioning[0-2-5] test_s3_cluster/test.py::test_count[1-5] [gw0] [ 1%] PASSED test_s3_cluster/test.py::test_count[1-5] test_s3_cluster/test.py::test_hive_partitioning[0-4-5] [gw0] [ 2%] PASSED test_s3_cluster/test.py::test_hive_partitioning[0-4-5] test_s3_cluster/test.py::test_wrong_cluster[2-5] [gw0] [ 3%] PASSED test_s3_cluster/test.py::test_wrong_cluster[2-5] test_s3_cluster/test.py::test_cluster_with_named_collection[2-5] [gw0] [ 4%] PASSED test_s3_cluster/test.py::test_cluster_with_named_collection[2-5] test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[2-5] [gw0] [ 5%] PASSED test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[2-5] test_s3_cluster/test.py::test_remote_hedged[3-5] [gw0] [ 6%] PASSED test_s3_cluster/test.py::test_remote_hedged[3-5] test_s3_cluster/test.py::test_count[2-5] [gw0] [ 6%] PASSED test_s3_cluster/test.py::test_count[2-5] test_s3_cluster/test.py::test_unset_skip_unavailable_shards[3-5] [gw0] [ 7%] PASSED test_s3_cluster/test.py::test_unset_skip_unavailable_shards[3-5] test_s3_cluster/test.py::test_union_all[2-5] [gw0] [ 8%] PASSED test_s3_cluster/test.py::test_union_all[2-5] test_s3_cluster/test.py::test_joins[2-5] [gw0] [ 9%] PASSED test_s3_cluster/test.py::test_joins[2-5] test_s3_cluster/test.py::test_cluster_format_detection[4-5] [gw0] [ 10%] PASSED test_s3_cluster/test.py::test_cluster_format_detection[4-5] test_s3_cluster/test.py::test_cluster_with_header[2-5] [gw0] [ 11%] PASSED test_s3_cluster/test.py::test_cluster_with_header[2-5] test_s3_cluster/test.py::test_distributed_s3_table_engine[5-5] [gw0] [ 12%] PASSED test_s3_cluster/test.py::test_distributed_s3_table_engine[5-5] test_s3_cluster/test.py::test_select_all[5-5] [gw0] [ 13%] PASSED test_s3_cluster/test.py::test_select_all[5-5] test_s3_cluster/test.py::test_object_storage_remote_initiator[3-5] [gw0] [ 13%] PASSED test_s3_cluster/test.py::test_object_storage_remote_initiator[3-5] test_s3_cluster/test.py::test_joins[5-5] [gw0] [ 14%] PASSED test_s3_cluster/test.py::test_joins[5-5] test_s3_cluster/test.py::test_graceful_shutdown[5-5] [gw0] [ 15%] PASSED test_s3_cluster/test.py::test_graceful_shutdown[5-5] test_s3_cluster/test.py::test_ambiguous_join[2-5] [gw0] [ 16%] PASSED test_s3_cluster/test.py::test_ambiguous_join[2-5] test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[4-5] [gw0] [ 17%] PASSED test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[4-5] test_s3_cluster/test.py::test_cluster_hosts_limit[3-5] [gw0] [ 18%] PASSED test_s3_cluster/test.py::test_cluster_hosts_limit[3-5] test_s3_cluster/test.py::test_distributed_s3_table_engine[2-5] [gw0] [ 19%] PASSED test_s3_cluster/test.py::test_distributed_s3_table_engine[2-5] test_s3_cluster/test.py::test_skip_unavailable_shards[3-5] [gw0] [ 20%] PASSED test_s3_cluster/test.py::test_skip_unavailable_shards[3-5] test_s3_cluster/test.py::test_unset_skip_unavailable_shards[2-5] [gw0] [ 20%] PASSED test_s3_cluster/test.py::test_unset_skip_unavailable_shards[2-5] test_s3_cluster/test.py::test_cluster_format_detection[3-5] [gw0] [ 21%] PASSED test_s3_cluster/test.py::test_cluster_format_detection[3-5] test_s3_cluster/test.py::test_cluster_default_expression[3-5] [gw0] [ 22%] PASSED test_s3_cluster/test.py::test_cluster_default_expression[3-5] test_s3_cluster/test.py::test_remote_no_hedged[4-5] [gw0] [ 23%] PASSED test_s3_cluster/test.py::test_remote_no_hedged[4-5] test_s3_cluster/test.py::test_select_all[4-5] [gw0] [ 24%] PASSED test_s3_cluster/test.py::test_select_all[4-5] test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[3-5] [gw0] [ 25%] PASSED test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[3-5] test_s3_cluster/test.py::test_cluster_with_named_collection[5-5] [gw0] [ 26%] PASSED test_s3_cluster/test.py::test_cluster_with_named_collection[5-5] test_s3_cluster/test.py::test_ambiguous_join[1-5] [gw0] [ 26%] PASSED test_s3_cluster/test.py::test_ambiguous_join[1-5] test_s3_cluster/test.py::test_cluster_with_header[1-5] [gw0] [ 27%] PASSED test_s3_cluster/test.py::test_cluster_with_header[1-5] test_s3_cluster/test.py::test_remote_no_hedged[2-5] [gw0] [ 28%] PASSED test_s3_cluster/test.py::test_remote_no_hedged[2-5] test_s3_cluster/test.py::test_remote_no_hedged[3-5] [gw0] [ 29%] PASSED test_s3_cluster/test.py::test_remote_no_hedged[3-5] test_s3_cluster/test.py::test_select_all[3-5] [gw0] [ 30%] PASSED test_s3_cluster/test.py::test_select_all[3-5] test_s3_cluster/test.py::test_graceful_shutdown[2-5] [gw0] [ 31%] PASSED test_s3_cluster/test.py::test_graceful_shutdown[2-5] test_s3_cluster/test.py::test_cluster_default_expression[1-5] [gw0] [ 32%] PASSED test_s3_cluster/test.py::test_cluster_default_expression[1-5] test_s3_cluster/test.py::test_cluster_hosts_limit[5-5] [gw0] [ 33%] PASSED test_s3_cluster/test.py::test_cluster_hosts_limit[5-5] test_s3_cluster/test.py::test_hive_partitioning[1-3-5] [gw0] [ 33%] PASSED test_s3_cluster/test.py::test_hive_partitioning[1-3-5] test_s3_cluster/test.py::test_cluster_format_detection[1-5] [gw0] [ 34%] PASSED test_s3_cluster/test.py::test_cluster_format_detection[1-5] test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[1-5] [gw0] [ 35%] PASSED test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[1-5] test_s3_cluster/test.py::test_wrong_cluster[4-5] [gw0] [ 36%] PASSED test_s3_cluster/test.py::test_wrong_cluster[4-5] test_s3_cluster/test.py::test_hive_partitioning[0-3-5] [gw0] [ 37%] PASSED test_s3_cluster/test.py::test_hive_partitioning[0-3-5] test_s3_cluster/test.py::test_graceful_shutdown[3-5] [gw0] [ 38%] PASSED test_s3_cluster/test.py::test_graceful_shutdown[3-5] test_s3_cluster/test.py::test_distributed_s3_table_engine[3-5] [gw0] [ 39%] PASSED test_s3_cluster/test.py::test_distributed_s3_table_engine[3-5] test_s3_cluster/test.py::test_skip_unavailable_shards[2-5] [gw0] [ 40%] PASSED test_s3_cluster/test.py::test_skip_unavailable_shards[2-5] test_s3_cluster/test.py::test_count_macro[3-5] [gw0] [ 40%] PASSED test_s3_cluster/test.py::test_count_macro[3-5] test_s3_cluster/test.py::test_cluster_with_header[4-5] [gw0] [ 41%] PASSED test_s3_cluster/test.py::test_cluster_with_header[4-5] test_s3_cluster/test.py::test_ambiguous_join[3-5] [gw0] [ 42%] PASSED test_s3_cluster/test.py::test_ambiguous_join[3-5] test_s3_cluster/test.py::test_object_storage_remote_initiator[4-5] [gw0] [ 43%] PASSED test_s3_cluster/test.py::test_object_storage_remote_initiator[4-5] test_s3_cluster/test.py::test_joins[1-5] [gw0] [ 44%] PASSED test_s3_cluster/test.py::test_joins[1-5] test_s3_cluster/test.py::test_hive_partitioning[0-1-5] [gw0] [ 45%] PASSED test_s3_cluster/test.py::test_hive_partitioning[0-1-5] test_s3_cluster/test.py::test_remote_hedged[1-5] [gw0] [ 46%] PASSED test_s3_cluster/test.py::test_remote_hedged[1-5] test_s3_cluster/test.py::test_remote_hedged[2-5] [gw0] [ 46%] PASSED test_s3_cluster/test.py::test_remote_hedged[2-5] test_s3_cluster/test.py::test_unset_skip_unavailable_shards[5-5] [gw0] [ 47%] PASSED test_s3_cluster/test.py::test_unset_skip_unavailable_shards[5-5] test_s3_cluster/test.py::test_cluster_with_named_collection[1-5] [gw0] [ 48%] PASSED test_s3_cluster/test.py::test_cluster_with_named_collection[1-5] test_s3_cluster/test.py::test_cluster_with_named_collection[4-5] [gw0] [ 49%] PASSED test_s3_cluster/test.py::test_cluster_with_named_collection[4-5] test_s3_cluster/test.py::test_select_all[2-5] [gw0] [ 50%] PASSED test_s3_cluster/test.py::test_select_all[2-5] test_s3_cluster/test.py::test_count[4-5] [gw0] [ 51%] PASSED test_s3_cluster/test.py::test_count[4-5] test_s3_cluster/test.py::test_graceful_shutdown[4-5] [gw0] [ 52%] PASSED test_s3_cluster/test.py::test_graceful_shutdown[4-5] test_s3_cluster/test.py::test_wrong_cluster[5-5] [gw0] [ 53%] PASSED test_s3_cluster/test.py::test_wrong_cluster[5-5] test_s3_cluster/test.py::test_unset_skip_unavailable_shards[1-5] [gw0] [ 53%] PASSED test_s3_cluster/test.py::test_unset_skip_unavailable_shards[1-5] test_s3_cluster/test.py::test_cluster_with_header[3-5] [gw0] [ 54%] PASSED test_s3_cluster/test.py::test_cluster_with_header[3-5] test_s3_cluster/test.py::test_union_all[1-5] [gw0] [ 55%] PASSED test_s3_cluster/test.py::test_union_all[1-5] test_s3_cluster/test.py::test_joins[3-5] [gw0] [ 56%] PASSED test_s3_cluster/test.py::test_joins[3-5] test_s3_cluster/test.py::test_remote_no_hedged[1-5] [gw0] [ 57%] PASSED test_s3_cluster/test.py::test_remote_no_hedged[1-5] test_s3_cluster/test.py::test_cluster_format_detection[5-5] [gw0] [ 58%] PASSED test_s3_cluster/test.py::test_cluster_format_detection[5-5] test_s3_cluster/test.py::test_ambiguous_join[4-5] [gw0] [ 59%] PASSED test_s3_cluster/test.py::test_ambiguous_join[4-5] test_s3_cluster/test.py::test_graceful_shutdown[1-5] [gw0] [ 60%] FAILED test_s3_cluster/test.py::test_graceful_shutdown[1-5] test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[5-5] [gw0] [ 60%] PASSED test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[5-5] test_s3_cluster/test.py::test_hive_partitioning[1-4-5] [gw0] [ 61%] PASSED test_s3_cluster/test.py::test_hive_partitioning[1-4-5] test_s3_cluster/test.py::test_remote_hedged[4-5] [gw0] [ 62%] PASSED test_s3_cluster/test.py::test_remote_hedged[4-5] test_s3_cluster/test.py::test_hive_partitioning[0-5-5] [gw0] [ 63%] PASSED test_s3_cluster/test.py::test_hive_partitioning[0-5-5] test_s3_cluster/test.py::test_ambiguous_join[5-5] [gw0] [ 64%] PASSED test_s3_cluster/test.py::test_ambiguous_join[5-5] test_s3_cluster/test.py::test_cluster_default_expression[2-5] [gw0] [ 65%] PASSED test_s3_cluster/test.py::test_cluster_default_expression[2-5] test_s3_cluster/test.py::test_count[5-5] [gw0] [ 66%] PASSED test_s3_cluster/test.py::test_count[5-5] test_s3_cluster/test.py::test_skip_unavailable_shards[1-5] [gw0] [ 66%] PASSED test_s3_cluster/test.py::test_skip_unavailable_shards[1-5] test_s3_cluster/test.py::test_count_macro[1-5] [gw0] [ 67%] PASSED test_s3_cluster/test.py::test_count_macro[1-5] test_s3_cluster/test.py::test_remote_no_hedged[5-5] [gw0] [ 68%] PASSED test_s3_cluster/test.py::test_remote_no_hedged[5-5] test_s3_cluster/test.py::test_distributed_s3_table_engine[4-5] [gw0] [ 69%] PASSED test_s3_cluster/test.py::test_distributed_s3_table_engine[4-5] test_s3_cluster/test.py::test_union_all[4-5] [gw0] [ 70%] PASSED test_s3_cluster/test.py::test_union_all[4-5] test_s3_cluster/test.py::test_select_all[1-5] [gw0] [ 71%] PASSED test_s3_cluster/test.py::test_select_all[1-5] test_s3_cluster/test.py::test_distributed_s3_table_engine[1-5] [gw0] [ 72%] PASSED test_s3_cluster/test.py::test_distributed_s3_table_engine[1-5] test_s3_cluster/test.py::test_count_macro[2-5] [gw0] [ 73%] PASSED test_s3_cluster/test.py::test_count_macro[2-5] test_s3_cluster/test.py::test_union_all[5-5] [gw0] [ 73%] PASSED test_s3_cluster/test.py::test_union_all[5-5] test_s3_cluster/test.py::test_count_macro[5-5] [gw0] [ 74%] PASSED test_s3_cluster/test.py::test_count_macro[5-5] test_s3_cluster/test.py::test_unset_skip_unavailable_shards[4-5] [gw0] [ 75%] PASSED test_s3_cluster/test.py::test_unset_skip_unavailable_shards[4-5] test_s3_cluster/test.py::test_cluster_with_header[5-5] [gw0] [ 76%] PASSED test_s3_cluster/test.py::test_cluster_with_header[5-5] test_s3_cluster/test.py::test_object_storage_remote_initiator[5-5] [gw0] [ 77%] PASSED test_s3_cluster/test.py::test_object_storage_remote_initiator[5-5] test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[4-5] [gw0] [ 78%] PASSED test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[4-5] test_s3_cluster/test.py::test_hive_partitioning[1-2-5] [gw0] [ 79%] PASSED test_s3_cluster/test.py::test_hive_partitioning[1-2-5] test_s3_cluster/test.py::test_hive_partitioning[1-5-5] [gw0] [ 80%] PASSED test_s3_cluster/test.py::test_hive_partitioning[1-5-5] test_s3_cluster/test.py::test_cluster_hosts_limit[2-5] [gw0] [ 80%] PASSED test_s3_cluster/test.py::test_cluster_hosts_limit[2-5] test_s3_cluster/test.py::test_cluster_hosts_limit[4-5] [gw0] [ 81%] PASSED test_s3_cluster/test.py::test_cluster_hosts_limit[4-5] test_s3_cluster/test.py::test_count_macro[4-5] [gw0] [ 82%] PASSED test_s3_cluster/test.py::test_count_macro[4-5] test_s3_cluster/test.py::test_count[3-5] [gw0] [ 83%] PASSED test_s3_cluster/test.py::test_count[3-5] test_s3_cluster/test.py::test_cluster_hosts_limit[1-5] [gw0] [ 84%] PASSED test_s3_cluster/test.py::test_cluster_hosts_limit[1-5] test_s3_cluster/test.py::test_cluster_format_detection[2-5] [gw0] [ 85%] PASSED test_s3_cluster/test.py::test_cluster_format_detection[2-5] test_s3_cluster/test.py::test_joins[4-5] [gw0] [ 86%] PASSED test_s3_cluster/test.py::test_joins[4-5] test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[1-5] [gw0] [ 86%] PASSED test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[1-5] test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[5-5] [gw0] [ 87%] PASSED test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[5-5] test_s3_cluster/test.py::test_object_storage_remote_initiator[1-5] [gw0] [ 88%] PASSED test_s3_cluster/test.py::test_object_storage_remote_initiator[1-5] test_s3_cluster/test.py::test_wrong_cluster[3-5] [gw0] [ 89%] PASSED test_s3_cluster/test.py::test_wrong_cluster[3-5] test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[3-5] [gw0] [ 90%] PASSED test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[3-5] test_s3_cluster/test.py::test_cluster_default_expression[4-5] [gw0] [ 91%] PASSED test_s3_cluster/test.py::test_cluster_default_expression[4-5] test_s3_cluster/test.py::test_remote_hedged[5-5] [gw0] [ 92%] PASSED test_s3_cluster/test.py::test_remote_hedged[5-5] test_s3_cluster/test.py::test_cluster_with_named_collection[3-5] [gw0] [ 93%] PASSED test_s3_cluster/test.py::test_cluster_with_named_collection[3-5] test_s3_cluster/test.py::test_union_all[3-5] [gw0] [ 93%] PASSED test_s3_cluster/test.py::test_union_all[3-5] test_s3_cluster/test.py::test_cluster_default_expression[5-5] [gw0] [ 94%] PASSED test_s3_cluster/test.py::test_cluster_default_expression[5-5] test_s3_cluster/test.py::test_skip_unavailable_shards[5-5] [gw0] [ 95%] PASSED test_s3_cluster/test.py::test_skip_unavailable_shards[5-5] test_s3_cluster/test.py::test_object_storage_remote_initiator[2-5] [gw0] [ 96%] PASSED test_s3_cluster/test.py::test_object_storage_remote_initiator[2-5] test_s3_cluster/test.py::test_wrong_cluster[1-5] [gw0] [ 97%] PASSED test_s3_cluster/test.py::test_wrong_cluster[1-5] test_s3_cluster/test.py::test_skip_unavailable_shards[4-5] [gw0] [ 98%] PASSED test_s3_cluster/test.py::test_skip_unavailable_shards[4-5] test_s3_cluster/test.py::test_hive_partitioning[1-1-5] [gw0] [ 99%] PASSED test_s3_cluster/test.py::test_hive_partitioning[1-1-5] test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[2-5] [gw0] [100%] PASSED test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[2-5] =================================== FAILURES =================================== _________________________ test_graceful_shutdown[1-5] __________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_graceful_shutdown(started_cluster): node = started_cluster.instances["s0_0_0"] node_to_shutdown = started_cluster.instances["s0_1_0"] expected = TSV("64\tBar\t8\n56\tFoo\t8\n") num_lock = threading.Lock() errors = 0 def query_cycle(): nonlocal errors try: i = 0 while i < 10: i += 1 # Query time 3-4 seconds # Processing single object 1-2 seconds result = node.query(f""" SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', '{minio_secret_key}', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 """) with num_lock: if TSV(result) != expected: errors += 1 if errors >= 1: break except QueryRuntimeException: with num_lock: errors += 1 threads = [] for _ in range(10): thread = threading.Thread(target=query_cycle) thread.start() threads.append(thread) time.sleep(0.2) time.sleep(3) node_to_shutdown.query("SYSTEM STOP SWARM MODE") # enough time to complete processing of objects, started before "SYSTEM STOP SWARM MODE" time.sleep(3) node_to_shutdown.stop_clickhouse(kill=True) for thread in threads: thread.join() node_to_shutdown.start_clickhouse() > assert errors == 0 E assert 1 == 0 test_s3_cluster/test.py:1255: AssertionError ----------------------------- Captured stderr call ----------------------------- Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SYSTEM STOP SWARM MODE on s0_1_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 run container_id:roottests3cluster-gw0-s0_1_0-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottests3cluster-gw0-s0_1_0-1 bash -c ps -C clickhouse] Stdout: PID TTY TIME CMD Stdout: 3198 ? 00:00:28 clickhouse run container_id:roottests3cluster-gw0-s0_1_0-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill -9 clickhouse'] Command:[docker exec -u root roottests3cluster-gw0-s0_1_0-1 bash -c pkill -9 clickhouse] run container_id:roottests3cluster-gw0-s0_1_0-1 detach:False nothrow:True cmd: ['bash', '-c', "pgrep -f '^[^ ]*clickhouse'"] Command:[docker exec roottests3cluster-gw0-s0_1_0-1 bash -c pgrep -f '^[^ ]*clickhouse'] Stdout:3198 run container_id:roottests3cluster-gw0-s0_1_0-1 detach:False nothrow:True cmd: ['bash', '-c', "pgrep -f '^[^ ]*clickhouse'"] Command:[docker exec roottests3cluster-gw0-s0_1_0-1 bash -c pgrep -f '^[^ ]*clickhouse'] Exitcode:1 run container_id:roottests3cluster-gw0-s0_1_0-1 detach:False nothrow:True cmd: ['bash', '-c', "pgrep -f '^[^ ]*clickhouse'"] Command:[docker exec roottests3cluster-gw0-s0_1_0-1 bash -c pgrep -f '^[^ ]*clickhouse'] Exitcode:1 No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottests3cluster-gw0-s0_1_0-1/exec HTTP/11" 201 74 http://localhost:None "POST /v1.46/exec/4d29132b861546effef03bb07110c873b3cf70f8e2abce0024d54579c4e68a8d/start HTTP/11" 200 0 http://localhost:None "GET /v1.46/exec/4d29132b861546effef03bb07110c873b3cf70f8e2abce0024d54579c4e68a8d/json HTTP/11" 200 586 run container_id:roottests3cluster-gw0-s0_1_0-1 detach:False nothrow:True cmd: ['bash', '-c', "pgrep -f '^[^ ]*clickhouse'"] Command:[docker exec roottests3cluster-gw0-s0_1_0-1 bash -c pgrep -f '^[^ ]*clickhouse'] Stdout:4002 Clickhouse process running. run container_id:roottests3cluster-gw0-s0_1_0-1 detach:False nothrow:True cmd: ['bash', '-c', "pgrep -f '^[^ ]*clickhouse'"] Command:[docker exec roottests3cluster-gw0-s0_1_0-1 bash -c pgrep -f '^[^ ]*clickhouse'] Stdout:4002 Executing query select 20 on s0_1_0 Executing query select 20 on s0_1_0 Executing query select 20 on s0_1_0 Executing query select 20 on s0_1_0 ------------------------------ Captured log call ------------------------------- 2026-03-20 18:22:33.958000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:34.158000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:34.359000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:34.560000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:34.763000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:34.963000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:35.164000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:35.365000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:35.566000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:35.767000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:37.385000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:37.585000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:37.791000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:37.987000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:38.139000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:38.340000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:38.590000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:38.741000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:38.941000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:38.968000 [ 519 ] DEBUG : Executing query SYSTEM STOP SWARM MODE on s0_1_0 (cluster.py:4196, query) 2026-03-20 18:22:39.144000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:40.863000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:41.011000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:41.217000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:41.415000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:41.565000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:41.816000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:42.016000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:42.217000 [ 519 ] DEBUG : Executing query SELECT sum(value),name,sum(sleep(1)+1) as sleep FROM s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/graceful/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'value UInt32, name String') GROUP BY name ORDER BY name SETTINGS max_threads=2 on s0_0_0 (cluster.py:4196, query) 2026-03-20 18:22:42.287000 [ 519 ] DEBUG : run container_id:roottests3cluster-gw0-s0_1_0-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2362, exec_in_container) 2026-03-20 18:22:42.288000 [ 519 ] DEBUG : Command:[docker exec -u root roottests3cluster-gw0-s0_1_0-1 bash -c ps -C clickhouse] (cluster.py:161, run_and_check) 2026-03-20 18:22:42.371000 [ 519 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:185, run_and_check) 2026-03-20 18:22:42.372000 [ 519 ] DEBUG : Stdout: 3198 ? 00:00:28 clickhouse (cluster.py:185, run_and_check) 2026-03-20 18:22:42.372000 [ 519 ] DEBUG : run container_id:roottests3cluster-gw0-s0_1_0-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill -9 clickhouse'] (cluster.py:2362, exec_in_container) 2026-03-20 18:22:42.372000 [ 519 ] DEBUG : Command:[docker exec -u root roottests3cluster-gw0-s0_1_0-1 bash -c pkill -9 clickhouse] (cluster.py:161, run_and_check) 2026-03-20 18:22:42.443000 [ 519 ] DEBUG : run container_id:roottests3cluster-gw0-s0_1_0-1 detach:False nothrow:True cmd: ['bash', '-c', "pgrep -f '^[^ ]*clickhouse'"] (cluster.py:2362, exec_in_container) 2026-03-20 18:22:42.444000 [ 519 ] DEBUG : Command:[docker exec roottests3cluster-gw0-s0_1_0-1 bash -c pgrep -f '^[^ ]*clickhouse'] (cluster.py:161, run_and_check) 2026-03-20 18:22:42.496000 [ 519 ] DEBUG : Stdout:3198 (cluster.py:185, run_and_check) 2026-03-20 18:22:43.497000 [ 519 ] DEBUG : run container_id:roottests3cluster-gw0-s0_1_0-1 detach:False nothrow:True cmd: ['bash', '-c', "pgrep -f '^[^ ]*clickhouse'"] (cluster.py:2362, exec_in_container) 2026-03-20 18:22:43.497000 [ 519 ] DEBUG : Command:[docker exec roottests3cluster-gw0-s0_1_0-1 bash -c pgrep -f '^[^ ]*clickhouse'] (cluster.py:161, run_and_check) 2026-03-20 18:22:43.559000 [ 519 ] DEBUG : Exitcode:1 (cluster.py:189, run_and_check) 2026-03-20 18:22:46.500000 [ 519 ] DEBUG : run container_id:roottests3cluster-gw0-s0_1_0-1 detach:False nothrow:True cmd: ['bash', '-c', "pgrep -f '^[^ ]*clickhouse'"] (cluster.py:2362, exec_in_container) 2026-03-20 18:22:46.501000 [ 519 ] DEBUG : Command:[docker exec roottests3cluster-gw0-s0_1_0-1 bash -c pgrep -f '^[^ ]*clickhouse'] (cluster.py:161, run_and_check) 2026-03-20 18:22:46.559000 [ 519 ] DEBUG : Exitcode:1 (cluster.py:189, run_and_check) 2026-03-20 18:22:46.560000 [ 519 ] DEBUG : No clickhouse process running. Start new one. (cluster.py:4562, start_clickhouse) 2026-03-20 18:22:46.563000 [ 519 ] DEBUG : http://localhost:None "POST /v1.46/containers/roottests3cluster-gw0-s0_1_0-1/exec HTTP/11" 201 74 (connectionpool.py:546, _make_request) 2026-03-20 18:22:46.586000 [ 519 ] DEBUG : http://localhost:None "POST /v1.46/exec/4d29132b861546effef03bb07110c873b3cf70f8e2abce0024d54579c4e68a8d/start HTTP/11" 200 0 (connectionpool.py:546, _make_request) 2026-03-20 18:22:46.588000 [ 519 ] DEBUG : http://localhost:None "GET /v1.46/exec/4d29132b861546effef03bb07110c873b3cf70f8e2abce0024d54579c4e68a8d/json HTTP/11" 200 586 (connectionpool.py:546, _make_request) 2026-03-20 18:22:47.590000 [ 519 ] DEBUG : run container_id:roottests3cluster-gw0-s0_1_0-1 detach:False nothrow:True cmd: ['bash', '-c', "pgrep -f '^[^ ]*clickhouse'"] (cluster.py:2362, exec_in_container) 2026-03-20 18:22:47.590000 [ 519 ] DEBUG : Command:[docker exec roottests3cluster-gw0-s0_1_0-1 bash -c pgrep -f '^[^ ]*clickhouse'] (cluster.py:161, run_and_check) 2026-03-20 18:22:47.634000 [ 519 ] DEBUG : Stdout:4002 (cluster.py:185, run_and_check) 2026-03-20 18:22:47.635000 [ 519 ] DEBUG : Clickhouse process running. (cluster.py:4576, start_clickhouse) 2026-03-20 18:22:47.635000 [ 519 ] DEBUG : run container_id:roottests3cluster-gw0-s0_1_0-1 detach:False nothrow:True cmd: ['bash', '-c', "pgrep -f '^[^ ]*clickhouse'"] (cluster.py:2362, exec_in_container) 2026-03-20 18:22:47.635000 [ 519 ] DEBUG : Command:[docker exec roottests3cluster-gw0-s0_1_0-1 bash -c pgrep -f '^[^ ]*clickhouse'] (cluster.py:161, run_and_check) 2026-03-20 18:22:47.693000 [ 519 ] DEBUG : Stdout:4002 (cluster.py:185, run_and_check) 2026-03-20 18:22:47.693000 [ 519 ] DEBUG : Executing query select 20 on s0_1_0 (cluster.py:4196, query) 2026-03-20 18:22:48.461000 [ 519 ] DEBUG : Executing query select 20 on s0_1_0 (cluster.py:4196, query) 2026-03-20 18:22:49.179000 [ 519 ] DEBUG : Executing query select 20 on s0_1_0 (cluster.py:4196, query) 2026-03-20 18:22:49.895000 [ 519 ] DEBUG : Executing query select 20 on s0_1_0 (cluster.py:4196, query) =============================== warnings summary =============================== ../../../usr/local/lib/python3.10/dist-packages/paramiko/pkey.py:100 ../../../usr/local/lib/python3.10/dist-packages/paramiko/pkey.py:100 /usr/local/lib/python3.10/dist-packages/paramiko/pkey.py:100: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from cryptography.hazmat.primitives.ciphers.algorithms in 48.0.0. "cipher": algorithms.TripleDES, ../../../usr/local/lib/python3.10/dist-packages/paramiko/transport.py:259 ../../../usr/local/lib/python3.10/dist-packages/paramiko/transport.py:259 /usr/local/lib/python3.10/dist-packages/paramiko/transport.py:259: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from cryptography.hazmat.primitives.ciphers.algorithms in 48.0.0. "class": algorithms.TripleDES, -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ------------------ generated report log file: flaky0_0.jsonl ------------------- ============================== slowest durations =============================== 63.69s teardown test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[2-5] 52.29s setup test_s3_cluster/test.py::test_hive_partitioning[0-2-5] 49.06s call test_s3_cluster/test.py::test_graceful_shutdown[4-5] 48.39s call test_s3_cluster/test.py::test_graceful_shutdown[5-5] 48.37s call test_s3_cluster/test.py::test_graceful_shutdown[2-5] 48.24s call test_s3_cluster/test.py::test_graceful_shutdown[3-5] 16.26s call test_s3_cluster/test.py::test_graceful_shutdown[1-5] 8.88s call test_s3_cluster/test.py::test_hive_partitioning[0-2-5] 7.31s call test_s3_cluster/test.py::test_hive_partitioning[1-4-5] 7.21s call test_s3_cluster/test.py::test_hive_partitioning[1-1-5] 7.16s call test_s3_cluster/test.py::test_hive_partitioning[1-5-5] 7.06s call test_s3_cluster/test.py::test_hive_partitioning[1-2-5] 6.95s call test_s3_cluster/test.py::test_cluster_default_expression[5-5] 6.91s call test_s3_cluster/test.py::test_hive_partitioning[0-1-5] 6.89s call test_s3_cluster/test.py::test_cluster_default_expression[1-5] 6.86s call test_s3_cluster/test.py::test_hive_partitioning[0-5-5] 6.85s call test_s3_cluster/test.py::test_hive_partitioning[1-3-5] 6.84s call test_s3_cluster/test.py::test_cluster_default_expression[3-5] 6.84s call test_s3_cluster/test.py::test_cluster_default_expression[4-5] 6.81s call test_s3_cluster/test.py::test_hive_partitioning[0-3-5] 6.80s call test_s3_cluster/test.py::test_cluster_default_expression[2-5] 6.54s call test_s3_cluster/test.py::test_hive_partitioning[0-4-5] 6.31s call test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[5-5] 6.26s call test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[1-5] 6.16s call test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[4-5] 6.01s call test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[3-5] 5.96s call test_s3_cluster/test.py::test_cluster_hosts_limit[1-5] 5.96s call test_s3_cluster/test.py::test_cluster_hosts_limit[4-5] 5.86s call test_s3_cluster/test.py::test_cluster_hosts_limit[2-5] 5.66s call test_s3_cluster/test.py::test_cluster_hosts_limit[5-5] 5.48s call test_s3_cluster/test.py::test_joins[2-5] 5.46s call test_s3_cluster/test.py::test_cluster_hosts_limit[3-5] 5.13s call test_s3_cluster/test.py::test_distributed_s3_table_engine[3-5] 5.11s call test_s3_cluster/test.py::test_object_storage_remote_initiator[2-5] 5.11s call test_s3_cluster/test.py::test_joins[4-5] 5.07s call test_s3_cluster/test.py::test_distributed_s3_table_engine[1-5] 5.06s call test_s3_cluster/test.py::test_joins[3-5] 4.96s call test_s3_cluster/test.py::test_object_storage_remote_initiator[1-5] 4.92s call test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[2-5] 4.91s call test_s3_cluster/test.py::test_object_storage_remote_initiator[5-5] 4.88s call test_s3_cluster/test.py::test_distributed_s3_table_engine[2-5] 4.87s call test_s3_cluster/test.py::test_distributed_s3_table_engine[4-5] 4.86s call test_s3_cluster/test.py::test_object_storage_remote_initiator[4-5] 4.86s call test_s3_cluster/test.py::test_joins[1-5] 4.78s call test_s3_cluster/test.py::test_distributed_s3_table_engine[5-5] 4.76s call test_s3_cluster/test.py::test_joins[5-5] 4.31s call test_s3_cluster/test.py::test_object_storage_remote_initiator[3-5] 4.07s call test_s3_cluster/test.py::test_cluster_format_detection[3-5] 3.97s call test_s3_cluster/test.py::test_cluster_format_detection[2-5] 3.84s call test_s3_cluster/test.py::test_cluster_format_detection[4-5] 3.83s call test_s3_cluster/test.py::test_cluster_format_detection[1-5] 3.82s call test_s3_cluster/test.py::test_cluster_format_detection[5-5] 2.85s call test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[2-5] 2.76s call test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[3-5] 2.60s call test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[5-5] 2.55s call test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[1-5] 2.40s call test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[4-5] 1.92s call test_s3_cluster/test.py::test_cluster_with_header[3-5] 1.89s call test_s3_cluster/test.py::test_ambiguous_join[4-5] 1.87s call test_s3_cluster/test.py::test_cluster_with_header[5-5] 1.79s call test_s3_cluster/test.py::test_skip_unavailable_shards[1-5] 1.79s call test_s3_cluster/test.py::test_ambiguous_join[5-5] 1.79s call test_s3_cluster/test.py::test_ambiguous_join[3-5] 1.79s call test_s3_cluster/test.py::test_ambiguous_join[2-5] 1.77s call test_s3_cluster/test.py::test_cluster_with_header[2-5] 1.74s call test_s3_cluster/test.py::test_unset_skip_unavailable_shards[3-5] 1.74s call test_s3_cluster/test.py::test_skip_unavailable_shards[4-5] 1.74s call test_s3_cluster/test.py::test_unset_skip_unavailable_shards[5-5] 1.74s call test_s3_cluster/test.py::test_unset_skip_unavailable_shards[2-5] 1.70s call test_s3_cluster/test.py::test_cluster_with_named_collection[3-5] 1.69s call test_s3_cluster/test.py::test_skip_unavailable_shards[2-5] 1.69s call test_s3_cluster/test.py::test_skip_unavailable_shards[3-5] 1.69s call test_s3_cluster/test.py::test_unset_skip_unavailable_shards[4-5] 1.69s call test_s3_cluster/test.py::test_skip_unavailable_shards[5-5] 1.67s call test_s3_cluster/test.py::test_cluster_with_header[4-5] 1.65s call test_s3_cluster/test.py::test_cluster_with_named_collection[5-5] 1.65s call test_s3_cluster/test.py::test_cluster_with_named_collection[1-5] 1.64s call test_s3_cluster/test.py::test_unset_skip_unavailable_shards[1-5] 1.62s call test_s3_cluster/test.py::test_cluster_with_header[1-5] 1.60s call test_s3_cluster/test.py::test_cluster_with_named_collection[4-5] 1.59s call test_s3_cluster/test.py::test_ambiguous_join[1-5] 1.45s call test_s3_cluster/test.py::test_cluster_with_named_collection[2-5] 1.24s call test_s3_cluster/test.py::test_count_macro[1-5] 1.24s call test_s3_cluster/test.py::test_count_macro[5-5] 1.19s call test_s3_cluster/test.py::test_remote_no_hedged[5-5] 1.19s call test_s3_cluster/test.py::test_remote_no_hedged[2-5] 1.19s call test_s3_cluster/test.py::test_union_all[5-5] 1.19s call test_s3_cluster/test.py::test_union_all[3-5] 1.19s call test_s3_cluster/test.py::test_count_macro[2-5] 1.14s call test_s3_cluster/test.py::test_remote_no_hedged[3-5] 1.14s call test_s3_cluster/test.py::test_remote_hedged[1-5] 1.14s call test_s3_cluster/test.py::test_remote_hedged[5-5] 1.14s call test_s3_cluster/test.py::test_remote_hedged[2-5] 1.13s call test_s3_cluster/test.py::test_union_all[2-5] 1.13s call test_s3_cluster/test.py::test_select_all[3-5] 1.13s call test_s3_cluster/test.py::test_count_macro[4-5] 1.13s call test_s3_cluster/test.py::test_count_macro[3-5] 1.09s call test_s3_cluster/test.py::test_select_all[1-5] 1.09s call test_s3_cluster/test.py::test_union_all[1-5] 1.09s call test_s3_cluster/test.py::test_remote_no_hedged[4-5] 1.09s call test_s3_cluster/test.py::test_select_all[2-5] 1.09s call test_s3_cluster/test.py::test_remote_hedged[4-5] 1.09s call test_s3_cluster/test.py::test_remote_no_hedged[1-5] 1.08s call test_s3_cluster/test.py::test_union_all[4-5] 1.08s call test_s3_cluster/test.py::test_count[3-5] 1.08s call test_s3_cluster/test.py::test_count[1-5] 1.08s call test_s3_cluster/test.py::test_count[5-5] 1.04s call test_s3_cluster/test.py::test_count[2-5] 1.04s call test_s3_cluster/test.py::test_select_all[5-5] 1.04s call test_s3_cluster/test.py::test_select_all[4-5] 0.98s call test_s3_cluster/test.py::test_count[4-5] 0.98s call test_s3_cluster/test.py::test_remote_hedged[3-5] 0.74s call test_s3_cluster/test.py::test_wrong_cluster[2-5] 0.58s call test_s3_cluster/test.py::test_wrong_cluster[5-5] 0.53s call test_s3_cluster/test.py::test_wrong_cluster[4-5] 0.53s call test_s3_cluster/test.py::test_wrong_cluster[1-5] 0.53s call test_s3_cluster/test.py::test_wrong_cluster[3-5] 0.00s teardown test_s3_cluster/test.py::test_hive_partitioning[0-2-5] 0.00s setup test_s3_cluster/test.py::test_hive_partitioning[0-4-5] 0.00s setup test_s3_cluster/test.py::test_hive_partitioning[1-4-5] 0.00s setup test_s3_cluster/test.py::test_remote_no_hedged[4-5] 0.00s setup test_s3_cluster/test.py::test_cluster_with_named_collection[1-5] 0.00s setup test_s3_cluster/test.py::test_count_macro[1-5] 0.00s setup test_s3_cluster/test.py::test_count[3-5] 0.00s setup test_s3_cluster/test.py::test_distributed_s3_table_engine[2-5] 0.00s setup test_s3_cluster/test.py::test_object_storage_remote_initiator[2-5] 0.00s setup test_s3_cluster/test.py::test_joins[4-5] 0.00s setup test_s3_cluster/test.py::test_remote_hedged[3-5] 0.00s setup test_s3_cluster/test.py::test_cluster_hosts_limit[4-5] 0.00s setup test_s3_cluster/test.py::test_hive_partitioning[1-5-5] 0.00s teardown test_s3_cluster/test.py::test_union_all[1-5] 0.00s setup test_s3_cluster/test.py::test_skip_unavailable_shards[1-5] 0.00s teardown test_s3_cluster/test.py::test_cluster_hosts_limit[3-5] 0.00s teardown test_s3_cluster/test.py::test_unset_skip_unavailable_shards[4-5] 0.00s teardown test_s3_cluster/test.py::test_graceful_shutdown[2-5] 0.00s teardown test_s3_cluster/test.py::test_graceful_shutdown[4-5] 0.00s setup test_s3_cluster/test.py::test_union_all[3-5] 0.00s setup test_s3_cluster/test.py::test_joins[3-5] 0.00s setup test_s3_cluster/test.py::test_graceful_shutdown[4-5] 0.00s setup test_s3_cluster/test.py::test_cluster_with_named_collection[5-5] 0.00s setup test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[4-5] 0.00s setup test_s3_cluster/test.py::test_wrong_cluster[5-5] 0.00s setup test_s3_cluster/test.py::test_cluster_with_header[4-5] 0.00s setup test_s3_cluster/test.py::test_cluster_default_expression[1-5] 0.00s setup test_s3_cluster/test.py::test_remote_no_hedged[2-5] 0.00s setup test_s3_cluster/test.py::test_cluster_with_header[2-5] 0.00s setup test_s3_cluster/test.py::test_distributed_s3_table_engine[3-5] 0.00s setup test_s3_cluster/test.py::test_remote_no_hedged[3-5] 0.00s setup test_s3_cluster/test.py::test_union_all[1-5] 0.00s setup test_s3_cluster/test.py::test_cluster_format_detection[2-5] 0.00s setup test_s3_cluster/test.py::test_select_all[1-5] 0.00s setup test_s3_cluster/test.py::test_unset_skip_unavailable_shards[1-5] 0.00s setup test_s3_cluster/test.py::test_skip_unavailable_shards[5-5] 0.00s setup test_s3_cluster/test.py::test_cluster_format_detection[4-5] 0.00s setup test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[3-5] 0.00s setup test_s3_cluster/test.py::test_cluster_with_header[5-5] 0.00s setup test_s3_cluster/test.py::test_joins[5-5] 0.00s setup test_s3_cluster/test.py::test_count[2-5] 0.00s setup test_s3_cluster/test.py::test_remote_hedged[4-5] 0.00s setup test_s3_cluster/test.py::test_skip_unavailable_shards[3-5] 0.00s setup test_s3_cluster/test.py::test_count[4-5] 0.00s setup test_s3_cluster/test.py::test_cluster_with_named_collection[3-5] 0.00s setup test_s3_cluster/test.py::test_cluster_default_expression[2-5] 0.00s setup test_s3_cluster/test.py::test_select_all[2-5] 0.00s setup test_s3_cluster/test.py::test_select_all[5-5] 0.00s setup test_s3_cluster/test.py::test_distributed_s3_table_engine[5-5] 0.00s setup test_s3_cluster/test.py::test_union_all[4-5] 0.00s setup test_s3_cluster/test.py::test_wrong_cluster[4-5] 0.00s setup test_s3_cluster/test.py::test_cluster_hosts_limit[3-5] 0.00s setup test_s3_cluster/test.py::test_count_macro[5-5] 0.00s setup test_s3_cluster/test.py::test_ambiguous_join[5-5] 0.00s setup test_s3_cluster/test.py::test_ambiguous_join[3-5] 0.00s setup test_s3_cluster/test.py::test_ambiguous_join[4-5] 0.00s setup test_s3_cluster/test.py::test_remote_no_hedged[5-5] 0.00s teardown test_s3_cluster/test.py::test_count[2-5] 0.00s setup test_s3_cluster/test.py::test_wrong_cluster[1-5] 0.00s setup test_s3_cluster/test.py::test_distributed_s3_table_engine[1-5] 0.00s setup test_s3_cluster/test.py::test_hive_partitioning[1-2-5] 0.00s setup test_s3_cluster/test.py::test_remote_hedged[5-5] 0.00s setup test_s3_cluster/test.py::test_graceful_shutdown[1-5] 0.00s setup test_s3_cluster/test.py::test_cluster_default_expression[3-5] 0.00s setup test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[2-5] 0.00s teardown test_s3_cluster/test.py::test_cluster_hosts_limit[2-5] 0.00s setup test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[5-5] 0.00s setup test_s3_cluster/test.py::test_cluster_with_named_collection[4-5] 0.00s teardown test_s3_cluster/test.py::test_joins[2-5] 0.00s teardown test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[2-5] 0.00s teardown test_s3_cluster/test.py::test_graceful_shutdown[3-5] 0.00s setup test_s3_cluster/test.py::test_select_all[4-5] 0.00s teardown test_s3_cluster/test.py::test_hive_partitioning[1-2-5] 0.00s teardown test_s3_cluster/test.py::test_hive_partitioning[1-4-5] 0.00s teardown test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[1-5] 0.00s teardown test_s3_cluster/test.py::test_count_macro[4-5] 0.00s teardown test_s3_cluster/test.py::test_count[5-5] 0.00s teardown test_s3_cluster/test.py::test_count[1-5] 0.00s teardown test_s3_cluster/test.py::test_joins[5-5] 0.00s setup test_s3_cluster/test.py::test_unset_skip_unavailable_shards[5-5] 0.00s teardown test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[5-5] 0.00s setup test_s3_cluster/test.py::test_hive_partitioning[0-3-5] 0.00s teardown test_s3_cluster/test.py::test_remote_no_hedged[2-5] 0.00s teardown test_s3_cluster/test.py::test_cluster_with_header[2-5] 0.00s setup test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[1-5] 0.00s teardown test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[3-5] 0.00s teardown test_s3_cluster/test.py::test_union_all[4-5] 0.00s setup test_s3_cluster/test.py::test_graceful_shutdown[5-5] 0.00s teardown test_s3_cluster/test.py::test_count_macro[3-5] 0.00s teardown test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[4-5] 0.00s teardown test_s3_cluster/test.py::test_remote_no_hedged[1-5] 0.00s setup test_s3_cluster/test.py::test_hive_partitioning[1-1-5] 0.00s setup test_s3_cluster/test.py::test_union_all[2-5] 0.00s teardown test_s3_cluster/test.py::test_cluster_format_detection[5-5] 0.00s teardown test_s3_cluster/test.py::test_hive_partitioning[0-5-5] 0.00s setup test_s3_cluster/test.py::test_remote_no_hedged[1-5] 0.00s teardown test_s3_cluster/test.py::test_remote_hedged[3-5] 0.00s teardown test_s3_cluster/test.py::test_distributed_s3_table_engine[4-5] 0.00s teardown test_s3_cluster/test.py::test_union_all[5-5] 0.00s teardown test_s3_cluster/test.py::test_distributed_s3_table_engine[2-5] 0.00s setup test_s3_cluster/test.py::test_ambiguous_join[1-5] 0.00s setup test_s3_cluster/test.py::test_remote_hedged[1-5] 0.00s setup test_s3_cluster/test.py::test_cluster_with_named_collection[2-5] 0.00s teardown test_s3_cluster/test.py::test_cluster_with_header[3-5] 0.00s teardown test_s3_cluster/test.py::test_object_storage_remote_initiator[3-5] 0.00s teardown test_s3_cluster/test.py::test_cluster_hosts_limit[1-5] 0.00s setup test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[2-5] 0.00s teardown test_s3_cluster/test.py::test_distributed_s3_table_engine[5-5] 0.00s teardown test_s3_cluster/test.py::test_unset_skip_unavailable_shards[5-5] 0.00s setup test_s3_cluster/test.py::test_count[1-5] 0.00s teardown test_s3_cluster/test.py::test_cluster_with_header[1-5] 0.00s setup test_s3_cluster/test.py::test_joins[2-5] 0.00s teardown test_s3_cluster/test.py::test_cluster_format_detection[2-5] 0.00s setup test_s3_cluster/test.py::test_hive_partitioning[0-5-5] 0.00s teardown test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[1-5] 0.00s teardown test_s3_cluster/test.py::test_skip_unavailable_shards[1-5] 0.00s teardown test_s3_cluster/test.py::test_cluster_default_expression[3-5] 0.00s teardown test_s3_cluster/test.py::test_cluster_format_detection[3-5] 0.00s teardown test_s3_cluster/test.py::test_cluster_with_named_collection[3-5] 0.00s teardown test_s3_cluster/test.py::test_wrong_cluster[5-5] 0.00s teardown test_s3_cluster/test.py::test_joins[4-5] 0.00s teardown test_s3_cluster/test.py::test_ambiguous_join[5-5] 0.00s teardown test_s3_cluster/test.py::test_ambiguous_join[4-5] 0.00s setup test_s3_cluster/test.py::test_hive_partitioning[0-1-5] 0.00s setup test_s3_cluster/test.py::test_hive_partitioning[1-3-5] 0.00s setup test_s3_cluster/test.py::test_unset_skip_unavailable_shards[3-5] 0.00s setup test_s3_cluster/test.py::test_cluster_format_detection[1-5] 0.00s teardown test_s3_cluster/test.py::test_select_all[2-5] 0.00s setup test_s3_cluster/test.py::test_cluster_default_expression[5-5] 0.00s teardown test_s3_cluster/test.py::test_object_storage_remote_initiator[5-5] 0.00s setup test_s3_cluster/test.py::test_cluster_format_detection[3-5] 0.00s setup test_s3_cluster/test.py::test_ambiguous_join[2-5] 0.00s setup test_s3_cluster/test.py::test_distributed_s3_table_engine[4-5] 0.00s setup test_s3_cluster/test.py::test_select_all[3-5] 0.00s setup test_s3_cluster/test.py::test_joins[1-5] 0.00s teardown test_s3_cluster/test.py::test_graceful_shutdown[5-5] 0.00s teardown test_s3_cluster/test.py::test_cluster_format_detection[4-5] 0.00s teardown test_s3_cluster/test.py::test_select_all[4-5] 0.00s setup test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[3-5] 0.00s setup test_s3_cluster/test.py::test_cluster_hosts_limit[2-5] 0.00s teardown test_s3_cluster/test.py::test_count[4-5] 0.00s teardown test_s3_cluster/test.py::test_cluster_default_expression[4-5] 0.00s teardown test_s3_cluster/test.py::test_graceful_shutdown[1-5] 0.00s teardown test_s3_cluster/test.py::test_count_macro[1-5] 0.00s teardown test_s3_cluster/test.py::test_cluster_with_named_collection[4-5] 0.00s setup test_s3_cluster/test.py::test_graceful_shutdown[2-5] 0.00s teardown test_s3_cluster/test.py::test_cluster_default_expression[5-5] 0.00s setup test_s3_cluster/test.py::test_remote_hedged[2-5] 0.00s setup test_s3_cluster/test.py::test_graceful_shutdown[3-5] 0.00s teardown test_s3_cluster/test.py::test_remote_no_hedged[4-5] 0.00s teardown test_s3_cluster/test.py::test_cluster_with_header[4-5] 0.00s teardown test_s3_cluster/test.py::test_select_all[1-5] 0.00s setup test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[4-5] 0.00s teardown test_s3_cluster/test.py::test_object_storage_remote_initiator[2-5] 0.00s setup test_s3_cluster/test.py::test_cluster_with_header[1-5] 0.00s setup test_s3_cluster/test.py::test_count_macro[3-5] 0.00s teardown test_s3_cluster/test.py::test_remote_hedged[5-5] 0.00s setup test_s3_cluster/test.py::test_count_macro[4-5] 0.00s setup test_s3_cluster/test.py::test_cluster_default_expression[4-5] 0.00s setup test_s3_cluster/test.py::test_wrong_cluster[2-5] 0.00s teardown test_s3_cluster/test.py::test_cluster_with_named_collection[1-5] 0.00s setup test_s3_cluster/test.py::test_skip_unavailable_shards[2-5] 0.00s setup test_s3_cluster/test.py::test_count[5-5] 0.00s setup test_s3_cluster/test.py::test_unset_skip_unavailable_shards[4-5] 0.00s setup test_s3_cluster/test.py::test_cluster_format_detection[5-5] 0.00s setup test_s3_cluster/test.py::test_cluster_hosts_limit[5-5] 0.00s setup test_s3_cluster/test.py::test_object_storage_remote_initiator[3-5] 0.00s teardown test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[4-5] 0.00s teardown test_s3_cluster/test.py::test_hive_partitioning[0-1-5] 0.00s teardown test_s3_cluster/test.py::test_hive_partitioning[0-3-5] 0.00s setup test_s3_cluster/test.py::test_object_storage_remote_initiator[1-5] 0.00s setup test_s3_cluster/test.py::test_skip_unavailable_shards[4-5] 0.00s teardown test_s3_cluster/test.py::test_skip_unavailable_shards[5-5] 0.00s teardown test_s3_cluster/test.py::test_remote_hedged[2-5] 0.00s setup test_s3_cluster/test.py::test_object_storage_remote_initiator[4-5] 0.00s teardown test_s3_cluster/test.py::test_wrong_cluster[2-5] 0.00s teardown test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[5-5] 0.00s setup test_s3_cluster/test.py::test_unset_skip_unavailable_shards[2-5] 0.00s teardown test_s3_cluster/test.py::test_hive_partitioning[1-1-5] 0.00s teardown test_s3_cluster/test.py::test_hive_partitioning[1-5-5] 0.00s teardown test_s3_cluster/test.py::test_cluster_with_named_collection[5-5] 0.00s teardown test_s3_cluster/test.py::test_joins[3-5] 0.00s teardown test_s3_cluster/test.py::test_hive_partitioning[1-3-5] 0.00s teardown test_s3_cluster/test.py::test_union_all[2-5] 0.00s teardown test_s3_cluster/test.py::test_union_all[3-5] 0.00s teardown test_s3_cluster/test.py::test_remote_no_hedged[5-5] 0.00s setup test_s3_cluster/test.py::test_wrong_cluster[3-5] 0.00s setup test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[1-5] 0.00s teardown test_s3_cluster/test.py::test_unset_skip_unavailable_shards[3-5] 0.00s teardown test_s3_cluster/test.py::test_cluster_hosts_limit[4-5] 0.00s setup test_s3_cluster/test.py::test_object_storage_remote_initiator[5-5] 0.00s setup test_s3_cluster/test.py::test_union_all[5-5] 0.00s teardown test_s3_cluster/test.py::test_object_storage_remote_initiator[4-5] 0.00s teardown test_s3_cluster/test.py::test_cluster_with_named_collection[2-5] 0.00s setup test_s3_cluster/test.py::test_count_macro[2-5] 0.00s setup test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[5-5] 0.00s teardown test_s3_cluster/test.py::test_wrong_cluster[3-5] 0.00s setup test_s3_cluster/test.py::test_cluster_with_header[3-5] 0.00s teardown test_s3_cluster/test.py::test_cluster_format_detection[1-5] 0.00s teardown test_s3_cluster/test.py::test_remote_hedged[1-5] 0.00s teardown test_s3_cluster/test.py::test_skip_unavailable_shards[2-5] 0.00s teardown test_s3_cluster/test.py::test_count_macro[2-5] 0.00s teardown test_s3_cluster/test.py::test_hive_partitioning[0-4-5] 0.00s setup test_s3_cluster/test.py::test_cluster_hosts_limit[1-5] 0.00s teardown test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[3-5] 0.00s teardown test_s3_cluster/test.py::test_count_macro[5-5] 0.00s teardown test_s3_cluster/test.py::test_distributed_s3_table_engine[3-5] 0.00s teardown test_s3_cluster/test.py::test_select_all[3-5] 0.00s teardown test_s3_cluster/test.py::test_ambiguous_join[2-5] 0.00s teardown test_s3_cluster/test.py::test_cluster_with_header[5-5] 0.00s teardown test_s3_cluster/test.py::test_select_all[5-5] 0.00s teardown test_s3_cluster/test.py::test_wrong_cluster[1-5] 0.00s teardown test_s3_cluster/test.py::test_cluster_default_expression[1-5] 0.00s teardown test_s3_cluster/test.py::test_joins[1-5] 0.00s teardown test_s3_cluster/test.py::test_remote_hedged[4-5] 0.00s teardown test_s3_cluster/test.py::test_ambiguous_join[1-5] 0.00s teardown test_s3_cluster/test.py::test_ambiguous_join[3-5] 0.00s teardown test_s3_cluster/test.py::test_cluster_hosts_limit[5-5] 0.00s teardown test_s3_cluster/test.py::test_remote_no_hedged[3-5] 0.00s teardown test_s3_cluster/test.py::test_wrong_cluster[4-5] 0.00s teardown test_s3_cluster/test.py::test_unset_skip_unavailable_shards[1-5] 0.00s teardown test_s3_cluster/test.py::test_object_storage_remote_initiator[1-5] 0.00s teardown test_s3_cluster/test.py::test_distributed_s3_table_engine[1-5] 0.00s teardown test_s3_cluster/test.py::test_cluster_default_expression[2-5] 0.00s teardown test_s3_cluster/test.py::test_skip_unavailable_shards[3-5] 0.00s teardown test_s3_cluster/test.py::test_unset_skip_unavailable_shards[2-5] 0.00s teardown test_s3_cluster/test.py::test_skip_unavailable_shards[4-5] 0.00s teardown test_s3_cluster/test.py::test_count[3-5] =========================== short test summary info ============================ FAILED test_s3_cluster/test.py::test_graceful_shutdown[1-5] - assert 1 == 0 PASSED test_s3_cluster/test.py::test_hive_partitioning[0-2-5] PASSED test_s3_cluster/test.py::test_count[1-5] PASSED test_s3_cluster/test.py::test_hive_partitioning[0-4-5] PASSED test_s3_cluster/test.py::test_wrong_cluster[2-5] PASSED test_s3_cluster/test.py::test_cluster_with_named_collection[2-5] PASSED test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[2-5] PASSED test_s3_cluster/test.py::test_remote_hedged[3-5] PASSED test_s3_cluster/test.py::test_count[2-5] PASSED test_s3_cluster/test.py::test_unset_skip_unavailable_shards[3-5] PASSED test_s3_cluster/test.py::test_union_all[2-5] PASSED test_s3_cluster/test.py::test_joins[2-5] PASSED test_s3_cluster/test.py::test_cluster_format_detection[4-5] PASSED test_s3_cluster/test.py::test_cluster_with_header[2-5] PASSED test_s3_cluster/test.py::test_distributed_s3_table_engine[5-5] PASSED test_s3_cluster/test.py::test_select_all[5-5] PASSED test_s3_cluster/test.py::test_object_storage_remote_initiator[3-5] PASSED test_s3_cluster/test.py::test_joins[5-5] PASSED test_s3_cluster/test.py::test_graceful_shutdown[5-5] PASSED test_s3_cluster/test.py::test_ambiguous_join[2-5] PASSED test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[4-5] PASSED test_s3_cluster/test.py::test_cluster_hosts_limit[3-5] PASSED test_s3_cluster/test.py::test_distributed_s3_table_engine[2-5] PASSED test_s3_cluster/test.py::test_skip_unavailable_shards[3-5] PASSED test_s3_cluster/test.py::test_unset_skip_unavailable_shards[2-5] PASSED test_s3_cluster/test.py::test_cluster_format_detection[3-5] PASSED test_s3_cluster/test.py::test_cluster_default_expression[3-5] PASSED test_s3_cluster/test.py::test_remote_no_hedged[4-5] PASSED test_s3_cluster/test.py::test_select_all[4-5] PASSED test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[3-5] PASSED test_s3_cluster/test.py::test_cluster_with_named_collection[5-5] PASSED test_s3_cluster/test.py::test_ambiguous_join[1-5] PASSED test_s3_cluster/test.py::test_cluster_with_header[1-5] PASSED test_s3_cluster/test.py::test_remote_no_hedged[2-5] PASSED test_s3_cluster/test.py::test_remote_no_hedged[3-5] PASSED test_s3_cluster/test.py::test_select_all[3-5] PASSED test_s3_cluster/test.py::test_graceful_shutdown[2-5] PASSED test_s3_cluster/test.py::test_cluster_default_expression[1-5] PASSED test_s3_cluster/test.py::test_cluster_hosts_limit[5-5] PASSED test_s3_cluster/test.py::test_hive_partitioning[1-3-5] PASSED test_s3_cluster/test.py::test_cluster_format_detection[1-5] PASSED test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[1-5] PASSED test_s3_cluster/test.py::test_wrong_cluster[4-5] PASSED test_s3_cluster/test.py::test_hive_partitioning[0-3-5] PASSED test_s3_cluster/test.py::test_graceful_shutdown[3-5] PASSED test_s3_cluster/test.py::test_distributed_s3_table_engine[3-5] PASSED test_s3_cluster/test.py::test_skip_unavailable_shards[2-5] PASSED test_s3_cluster/test.py::test_count_macro[3-5] PASSED test_s3_cluster/test.py::test_cluster_with_header[4-5] PASSED test_s3_cluster/test.py::test_ambiguous_join[3-5] PASSED test_s3_cluster/test.py::test_object_storage_remote_initiator[4-5] PASSED test_s3_cluster/test.py::test_joins[1-5] PASSED test_s3_cluster/test.py::test_hive_partitioning[0-1-5] PASSED test_s3_cluster/test.py::test_remote_hedged[1-5] PASSED test_s3_cluster/test.py::test_remote_hedged[2-5] PASSED test_s3_cluster/test.py::test_unset_skip_unavailable_shards[5-5] PASSED test_s3_cluster/test.py::test_cluster_with_named_collection[1-5] PASSED test_s3_cluster/test.py::test_cluster_with_named_collection[4-5] PASSED test_s3_cluster/test.py::test_select_all[2-5] PASSED test_s3_cluster/test.py::test_count[4-5] PASSED test_s3_cluster/test.py::test_graceful_shutdown[4-5] PASSED test_s3_cluster/test.py::test_wrong_cluster[5-5] PASSED test_s3_cluster/test.py::test_unset_skip_unavailable_shards[1-5] PASSED test_s3_cluster/test.py::test_cluster_with_header[3-5] PASSED test_s3_cluster/test.py::test_union_all[1-5] PASSED test_s3_cluster/test.py::test_joins[3-5] PASSED test_s3_cluster/test.py::test_remote_no_hedged[1-5] PASSED test_s3_cluster/test.py::test_cluster_format_detection[5-5] PASSED test_s3_cluster/test.py::test_ambiguous_join[4-5] PASSED test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[5-5] PASSED test_s3_cluster/test.py::test_hive_partitioning[1-4-5] PASSED test_s3_cluster/test.py::test_remote_hedged[4-5] PASSED test_s3_cluster/test.py::test_hive_partitioning[0-5-5] PASSED test_s3_cluster/test.py::test_ambiguous_join[5-5] PASSED test_s3_cluster/test.py::test_cluster_default_expression[2-5] PASSED test_s3_cluster/test.py::test_count[5-5] PASSED test_s3_cluster/test.py::test_skip_unavailable_shards[1-5] PASSED test_s3_cluster/test.py::test_count_macro[1-5] PASSED test_s3_cluster/test.py::test_remote_no_hedged[5-5] PASSED test_s3_cluster/test.py::test_distributed_s3_table_engine[4-5] PASSED test_s3_cluster/test.py::test_union_all[4-5] PASSED test_s3_cluster/test.py::test_select_all[1-5] PASSED test_s3_cluster/test.py::test_distributed_s3_table_engine[1-5] PASSED test_s3_cluster/test.py::test_count_macro[2-5] PASSED test_s3_cluster/test.py::test_union_all[5-5] PASSED test_s3_cluster/test.py::test_count_macro[5-5] PASSED test_s3_cluster/test.py::test_unset_skip_unavailable_shards[4-5] PASSED test_s3_cluster/test.py::test_cluster_with_header[5-5] PASSED test_s3_cluster/test.py::test_object_storage_remote_initiator[5-5] PASSED test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[4-5] PASSED test_s3_cluster/test.py::test_hive_partitioning[1-2-5] PASSED test_s3_cluster/test.py::test_hive_partitioning[1-5-5] PASSED test_s3_cluster/test.py::test_cluster_hosts_limit[2-5] PASSED test_s3_cluster/test.py::test_cluster_hosts_limit[4-5] PASSED test_s3_cluster/test.py::test_count_macro[4-5] PASSED test_s3_cluster/test.py::test_count[3-5] PASSED test_s3_cluster/test.py::test_cluster_hosts_limit[1-5] PASSED test_s3_cluster/test.py::test_cluster_format_detection[2-5] PASSED test_s3_cluster/test.py::test_joins[4-5] PASSED test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[1-5] PASSED test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[5-5] PASSED test_s3_cluster/test.py::test_object_storage_remote_initiator[1-5] PASSED test_s3_cluster/test.py::test_wrong_cluster[3-5] PASSED test_s3_cluster/test.py::test_distributed_insert_select_with_replicated[3-5] PASSED test_s3_cluster/test.py::test_cluster_default_expression[4-5] PASSED test_s3_cluster/test.py::test_remote_hedged[5-5] PASSED test_s3_cluster/test.py::test_cluster_with_named_collection[3-5] PASSED test_s3_cluster/test.py::test_union_all[3-5] PASSED test_s3_cluster/test.py::test_cluster_default_expression[5-5] PASSED test_s3_cluster/test.py::test_skip_unavailable_shards[5-5] PASSED test_s3_cluster/test.py::test_object_storage_remote_initiator[2-5] PASSED test_s3_cluster/test.py::test_wrong_cluster[1-5] PASSED test_s3_cluster/test.py::test_skip_unavailable_shards[4-5] PASSED test_s3_cluster/test.py::test_hive_partitioning[1-1-5] PASSED test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference[2-5] ============ 1 failed, 114 passed, 4 warnings in 678.84s (0:11:18) ============= Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 531, in subprocess.check_call(cmd, shell=True, bufsize=0) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_swmcii --privileged --dns-search='.' --memory=30709026816 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/ci/tmp/clickhouse:/clickhouse --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=e7a502a8ddfb733e8385_amd -e DOCKER_HELPER_TAG=5dc43a6382f0055b4d8e_amd -e DOCKER_BASE_TAG=9c789665fa1f2c3b60a8_amd -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d716376396_amd -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=a2be58f091e1427a1e32_amd -e DOCKER_MYSQL_JAVA_CLIENT_TAG=f9bb23fd3568fb51064b_amd -e DOCKER_MYSQL_JS_CLIENT_TAG=6a2cfd28cf040f57b0aa_amd -e DOCKER_ARROWFLIGHT_SERVER_TAG=8ec78de57d59a65d2bfb_amd -e DOCKER_MYSQL_PHP_CLIENT_TAG=2e3bd01324fab1b1e95d_amd -e DOCKER_NGINX_DAV_TAG=aa33b6ad5665b723c6e4_amd -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=9f8b31d2e09ab8aaf242_amd -e DOCKER_PYTHON_BOTTLE_TAG=426daa4ec42501225a52_amd -e DOCKER_BASE_WITH_UNITY_CATALOG_TAG=b2310980ad8a5093f676_amd -e DOCKER_BASE_WITH_HMS_TAG=859811a46af684b6fc6b_amd -e DOCKER_MYSQL_DOTNET_CLIENT_TAG=2efb0784cb8ed9630acd_amd -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 1 --count 5 -rfEps --run-id=0 --color=no --durations=0 --report-log=flaky0_0.jsonl --report-log-exclude-logs-on-passed-tests test_s3_cluster/test.py --random-order -vvv " altinityinfra/integration-tests-runner:75b9a401cb1d94a0286c_amd ' returned non-zero exit status 1.