S3 Test Run Report

DateSep 03, 2025 0:54
Duration1h 1m
Framework TestFlows 2.0.250110.1002922

Artifacts

Test artifacts can be found at https://altinity-build-artifacts.s3.amazonaws.com/index.html#998/ad3281d439d6d71588fea15e0147825155625faf/regression/aarch64/with_analyzer/zookeeper/without_thread_fuzzer/s32/gcs/

Attributes

projectAltinity/ClickHouse
project.id159717931
packagehttps://altinity-build-artifacts.s3.amazonaws.com/PRs/998/ad3281d439d6d71588fea15e0147825155625faf/build_arm_release/clickhouse-common-static_25.6.5.20166.altinityantalya_arm64.deb
version25.6.5.20166.altinityantalya
user.namestrtgbb
repositoryhttps://github.com/Altinity/clickhouse-regression
commit.hash38b4f3c4cbcf7b38c97e16793c210a1496075af7
job.nameS3 (gcs-2)
job.retry1
job.urlhttps://github.com/Altinity/ClickHouse/actions/runs/17418173257
archaarch64
localTrue
clickhouse_versionNone
clickhouse_pathhttps://altinity-build-artifacts.s3.amazonaws.com/PRs/998/ad3281d439d6d71588fea15e0147825155625faf/build_arm_release/clickhouse-common-static_25.6.5.20166.altinityantalya_arm64.deb
as_binaryFalse
base_osNone
keeper_pathNone
zookeeper_versionNone
use_keeperFalse
stressFalse
collect_service_logsTrue
thread_fuzzerFalse
with_analyzerTrue
reuse_envFalse
storages['gcs']
minio_uriSecret(name='minio_uri')
minio_root_userSecret(name='minio_root_user')
minio_root_passwordSecret(name='minio_root_password')
aws_s3_bucketSecret(name='aws_s3_bucket')
aws_s3_regionSecret(name='aws_s3_region')
aws_s3_key_idSecret(name='aws_s3_key_id')
aws_s3_access_keySecret(name='aws_s3_access_key')
gcs_uriSecret(name='gcs_uri')
gcs_key_idSecret(name='gcs_key_id')
gcs_key_secretSecret(name='gcs_key_secret')
azure_account_nameSecret(name='azure_account_name')
azure_storage_keySecret(name='azure_storage_key')
azure_containerSecret(name='azure_container')

Summary

65.5%OK
22.8%Known
9%Fail

Statistics

Units Skip OK Fail Error XFail XError
Modules
1
1
Features
8
2
3
3
Scenarios
70
35
2
31
2
Examples
11
2
2
7
Steps
9914
9
9872
25
8

Fails

Test NameResultMessage
/s3Fail 1h 1m
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 813, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 171, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 348, in capture_s3_args
    return func(self, s3_args=s3_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 785, in regression
    Feature(test=gcs_regression)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 746, in gcs_regression
    Feature(test=load("s3.tests.table_function_performance", "gcs"))(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 137, in gcs
    outline(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 118, in outline
    Scenario(run=s3_create_many_files)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 30, in s3_create_many_files
    r = node.query(
        ^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1195, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Received exception from server (version 25.6.5):
Code: 36. DB::Exception: Received from localhost:9000. DB::Exception: One input format is only suitable for tables with a single column of type UInt8 but the number of columns is 2: While executing S3(_table_function.s3)Source. (BAD_ARGUMENTS)
(query: SELECT _path FROM s3('[masked]:Secret(name='gcs_uri')many_files_benchmark/id=556649/*', '[masked]:Secret(name='gcs_key_id')','[masked]:Secret(name='gcs_key_secret')', 'One') FORMAT TabSeparated
)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 1195 in 'query'

1187\|                  assert message in r.output, error(r.output)
1188\|  
1189\|          if not ignore_exception:
1190\|              if message is None or "Exception:" not in message:
1191\|                  with Then("check if output has exception") if steps else NullStep():
1192\|                      if "Exception:" in r.output:
1193\|                          if raise_on_exception:
1194\|                              raise QueryRuntimeException(r.output)
1195\|>                         assert False, error(r.output)
1196\|  
1197\|          return r
1198\|
/s3/gcsFail 1h 1m
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 813, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 171, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 348, in capture_s3_args
    return func(self, s3_args=s3_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 785, in regression
    Feature(test=gcs_regression)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 746, in gcs_regression
    Feature(test=load("s3.tests.table_function_performance", "gcs"))(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 137, in gcs
    outline(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 118, in outline
    Scenario(run=s3_create_many_files)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 30, in s3_create_many_files
    r = node.query(
        ^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1195, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Received exception from server (version 25.6.5):
Code: 36. DB::Exception: Received from localhost:9000. DB::Exception: One input format is only suitable for tables with a single column of type UInt8 but the number of columns is 2: While executing S3(_table_function.s3)Source. (BAD_ARGUMENTS)
(query: SELECT _path FROM s3('[masked]:Secret(name='gcs_uri')many_files_benchmark/id=556649/*', '[masked]:Secret(name='gcs_key_id')','[masked]:Secret(name='gcs_key_secret')', 'One') FORMAT TabSeparated
)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 1195 in 'query'

1187\|                  assert message in r.output, error(r.output)
1188\|  
1189\|          if not ignore_exception:
1190\|              if message is None or "Exception:" not in message:
1191\|                  with Then("check if output has exception") if steps else NullStep():
1192\|                      if "Exception:" in r.output:
1193\|                          if raise_on_exception:
1194\|                              raise QueryRuntimeException(r.output)
1195\|>                         assert False, error(r.output)
1196\|  
1197\|          return r
1198\|
/s3/gcs/part 2Fail 59m 27s
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 813, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 171, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 348, in capture_s3_args
    return func(self, s3_args=s3_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 785, in regression
    Feature(test=gcs_regression)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 746, in gcs_regression
    Feature(test=load("s3.tests.table_function_performance", "gcs"))(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 137, in gcs
    outline(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 118, in outline
    Scenario(run=s3_create_many_files)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 30, in s3_create_many_files
    r = node.query(
        ^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1195, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Received exception from server (version 25.6.5):
Code: 36. DB::Exception: Received from localhost:9000. DB::Exception: One input format is only suitable for tables with a single column of type UInt8 but the number of columns is 2: While executing S3(_table_function.s3)Source. (BAD_ARGUMENTS)
(query: SELECT _path FROM s3('[masked]:Secret(name='gcs_uri')many_files_benchmark/id=556649/*', '[masked]:Secret(name='gcs_key_id')','[masked]:Secret(name='gcs_key_secret')', 'One') FORMAT TabSeparated
)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 1195 in 'query'

1187\|                  assert message in r.output, error(r.output)
1188\|  
1189\|          if not ignore_exception:
1190\|              if message is None or "Exception:" not in message:
1191\|                  with Then("check if output has exception") if steps else NullStep():
1192\|                      if "Exception:" in r.output:
1193\|                          if raise_on_exception:
1194\|                              raise QueryRuntimeException(r.output)
1195\|>                         assert False, error(r.output)
1196\|  
1197\|          return r
1198\|
/s3/gcs/part 2/table function performanceFail 40s 522ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 813, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 171, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 348, in capture_s3_args
    return func(self, s3_args=s3_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 785, in regression
    Feature(test=gcs_regression)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 746, in gcs_regression
    Feature(test=load("s3.tests.table_function_performance", "gcs"))(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 137, in gcs
    outline(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 118, in outline
    Scenario(run=s3_create_many_files)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 30, in s3_create_many_files
    r = node.query(
        ^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1195, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Received exception from server (version 25.6.5):
Code: 36. DB::Exception: Received from localhost:9000. DB::Exception: One input format is only suitable for tables with a single column of type UInt8 but the number of columns is 2: While executing S3(_table_function.s3)Source. (BAD_ARGUMENTS)
(query: SELECT _path FROM s3('[masked]:Secret(name='gcs_uri')many_files_benchmark/id=556649/*', '[masked]:Secret(name='gcs_key_id')','[masked]:Secret(name='gcs_key_secret')', 'One') FORMAT TabSeparated
)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 1195 in 'query'

1187\|                  assert message in r.output, error(r.output)
1188\|  
1189\|          if not ignore_exception:
1190\|              if message is None or "Exception:" not in message:
1191\|                  with Then("check if output has exception") if steps else NullStep():
1192\|                      if "Exception:" in r.output:
1193\|                          if raise_on_exception:
1194\|                              raise QueryRuntimeException(r.output)
1195\|>                         assert False, error(r.output)
1196\|  
1197\|          return r
1198\|
/s3/gcs/part 2/table function performance/setupFail 893ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 813, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 171, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 348, in capture_s3_args
    return func(self, s3_args=s3_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 785, in regression
    Feature(test=gcs_regression)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 746, in gcs_regression
    Feature(test=load("s3.tests.table_function_performance", "gcs"))(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 137, in gcs
    outline(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 118, in outline
    Scenario(run=s3_create_many_files)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 30, in s3_create_many_files
    r = node.query(
        ^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1195, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Received exception from server (version 25.6.5):
Code: 36. DB::Exception: Received from localhost:9000. DB::Exception: One input format is only suitable for tables with a single column of type UInt8 but the number of columns is 2: While executing S3(_table_function.s3)Source. (BAD_ARGUMENTS)
(query: SELECT _path FROM s3('[masked]:Secret(name='gcs_uri')many_files_benchmark/id=556649/*', '[masked]:Secret(name='gcs_key_id')','[masked]:Secret(name='gcs_key_secret')', 'One') FORMAT TabSeparated
)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 1195 in 'query'

1187\|                  assert message in r.output, error(r.output)
1188\|  
1189\|          if not ignore_exception:
1190\|              if message is None or "Exception:" not in message:
1191\|                  with Then("check if output has exception") if steps else NullStep():
1192\|                      if "Exception:" in r.output:
1193\|                          if raise_on_exception:
1194\|                              raise QueryRuntimeException(r.output)
1195\|>                         assert False, error(r.output)
1196\|  
1197\|          return r
1198\|
/s3/gcs/part 2/table function performance/wildcardFail 39s 624ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 813, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 171, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 348, in capture_s3_args
    return func(self, s3_args=s3_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 785, in regression
    Feature(test=gcs_regression)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 746, in gcs_regression
    Feature(test=load("s3.tests.table_function_performance", "gcs"))(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 137, in gcs
    outline(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 121, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 96, in wildcard
    r = node.query(
        ^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1195, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Received exception from server (version 25.6.5):
Code: 117. DB::Exception: Received from localhost:9000. DB::Exception: A hive partitioned file can't contain only partition columns. Try reading it with `use_hive_partitioning=0`. (INCORRECT_DATA)
(query: SELECT median(d) FROM s3('[masked]:Secret(name='gcs_uri')many_files_benchmark/id=522029/*', '[masked]:Secret(name='gcs_key_id')','[masked]:Secret(name='gcs_key_secret')', 'CSV', 'd UInt64') FORMAT TabSeparated
)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 1195 in 'query'

1187\|                  assert message in r.output, error(r.output)
1188\|  
1189\|          if not ignore_exception:
1190\|              if message is None or "Exception:" not in message:
1191\|                  with Then("check if output has exception") if steps else NullStep():
1192\|                      if "Exception:" in r.output:
1193\|                          if raise_on_exception:
1194\|                              raise QueryRuntimeException(r.output)
1195\|>                         assert False, error(r.output)
1196\|  
1197\|          return r
1198\|

Known Fails

Test NameResultMessage
/s3/gcs/part 2/combinatoric table/engine=VersionedCollapsingMergeTree,replicated=True,n_cols=2000,n_tables=3,part_type=wideXFail 4m 19s
Needs investigation, rows not appearing
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 813, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 171, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 348, in capture_s3_args
    return func(self, s3_args=s3_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 785, in regression
    Feature(test=gcs_regression)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 735, in gcs_regression
    Feature(test=load("s3.tests.combinatoric_table", "feature"))(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/combinatoric_table.py", line 182, in feature
    Scenario(title, test=check_table_combination)(**table_config)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/combinatoric_table.py", line 149, in check_table_combination
    retry(assert_row_count, timeout=60, delay=5)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py", line 1754, in assert_row_count
    assert rows == actual_count, error()
           ^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert rows == actual_count, error()

Assertion values
  assert rows == actual_count, error()
         ^ is 1500
  assert rows == actual_count, error()
                 ^ is 0
  assert rows == actual_count, error()
              ^ is = False
  assert rows == actual_count, error()
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py', line 1754 in 'assert_row_count'

1746\|  
1747\|  @TestStep(Then)
1748\|  def assert_row_count(self, node, table_name: str, rows: int = 1000000):
1749\|      """Assert that the number of rows in a table is as expected."""
1750\|      if node is None:
1751\|          node = current().context.node
1752\|  
1753\|      actual_count = get_row_count(node=node, table_name=table_name)
1754\|>     assert rows == actual_count, error()
1755\|  
1756\|  
1757\|  @TestStep(Then)
/s3/gcs/part 2/combinatoric table/engine=AggregatingMergeTree,replicated=True,n_cols=2000,n_tables=3,part_type=compactXError 8m 1s
Times out, needs investigation
ExpectTimeoutError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/venv/lib/python3.12/site-packages/testflows/uexpect/uexpect.py", line 223, in read
    d = self.queue.get(timeout=timeleft)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/queue.py", line 179, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/venv/lib/python3.12/site-packages/testflows/uexpect/uexpect.py", line 187, in expect
    data = self.read(timeout=min(timeleft, 0.1), raise_exception=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/venv/lib/python3.12/site-packages/testflows/uexpect/uexpect.py", line 235, in read
    raise TimeoutError(timeout)
testflows.uexpect.uexpect.TimeoutError: Timeout 0.076s

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 813, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 171, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 348, in capture_s3_args
    return func(self, s3_args=s3_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 785, in regression
    Feature(test=gcs_regression)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 735, in gcs_regression
    Feature(test=load("s3.tests.combinatoric_table", "feature"))(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/combinatoric_table.py", line 182, in feature
    Scenario(title, test=check_table_combination)(**table_config)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/combinatoric_table.py", line 117, in check_table_combination
    table = create_test_table(
            ^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/combinatoric_table.py", line 81, in create_test_table
    yield create_table(
          ^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/tables.py", line 482, in create_table
    node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1123, in query
    r = self.cluster.bash(None)(command, *args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
testflows.uexpect.uexpect.ExpectTimeoutError: Timeout 300.000s for '(bash# )\|(\n)'
/s3/gcs/part 2/combinatoric table/engine=SummingMergeTree,replicated=True,n_cols=2000,n_tables=3,part_type=compactXError 8m 0s
Times out, needs investigation
ExpectTimeoutError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/venv/lib/python3.12/site-packages/testflows/uexpect/uexpect.py", line 223, in read
    d = self.queue.get(timeout=timeleft)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/queue.py", line 179, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/venv/lib/python3.12/site-packages/testflows/uexpect/uexpect.py", line 187, in expect
    data = self.read(timeout=min(timeleft, 0.1), raise_exception=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/venv/lib/python3.12/site-packages/testflows/uexpect/uexpect.py", line 235, in read
    raise TimeoutError(timeout)
testflows.uexpect.uexpect.TimeoutError: Timeout 0.091s

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 813, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 171, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 348, in capture_s3_args
    return func(self, s3_args=s3_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 785, in regression
    Feature(test=gcs_regression)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 735, in gcs_regression
    Feature(test=load("s3.tests.combinatoric_table", "feature"))(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/combinatoric_table.py", line 182, in feature
    Scenario(title, test=check_table_combination)(**table_config)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/combinatoric_table.py", line 117, in check_table_combination
    table = create_test_table(
            ^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/combinatoric_table.py", line 81, in create_test_table
    yield create_table(
          ^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/tables.py", line 482, in create_table
    node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1123, in query
    r = self.cluster.bash(None)(command, *args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
testflows.uexpect.uexpect.ExpectTimeoutError: Timeout 300.000s for '(bash# )\|(\n)'

Results

Test Name Result Duration
/s3 Fail 1h 1m
/s3/gcs Fail 1h 1m
/s3/gcs/part 2 Fail 59m 27s
/s3/gcs/part 2/combinatoric table OK 38m 6s
/s3/gcs/part 2/combinatoric table/engine=MergeTree,replicated=True,n_cols=2000,n_tables=3,part_type=compact OK 35s 566ms
/s3/gcs/part 2/combinatoric table/engine=MergeTree,replicated=False,n_cols=500,n_tables=1,part_type=wide OK 5m 7s
/s3/gcs/part 2/combinatoric table/engine=ReplacingMergeTree,replicated=True,n_cols=500,n_tables=3,part_type=unspecified OK 29s 852ms
/s3/gcs/part 2/combinatoric table/engine=ReplacingMergeTree,replicated=False,n_cols=2000,n_tables=1,part_type=compact OK 4s 422ms
/s3/gcs/part 2/combinatoric table/engine=CollapsingMergeTree,replicated=True,n_cols=10,n_tables=1,part_type=wide OK 24s 584ms
/s3/gcs/part 2/combinatoric table/engine=CollapsingMergeTree,replicated=False,n_cols=2000,n_tables=3,part_type=unspecified OK 13s 538ms
/s3/gcs/part 2/combinatoric table/engine=VersionedCollapsingMergeTree,replicated=True,n_cols=2000,n_tables=3,part_type=wide XFail 4m 19s
/s3/gcs/part 2/combinatoric table/engine=VersionedCollapsingMergeTree,replicated=False,n_cols=10,n_tables=3,part_type=compact OK 11s 803ms
/s3/gcs/part 2/combinatoric table/engine=AggregatingMergeTree,replicated=True,n_cols=2000,n_tables=3,part_type=compact XError 8m 1s
/s3/gcs/part 2/combinatoric table/engine=AggregatingMergeTree,replicated=False,n_cols=500,n_tables=1,part_type=unspecified OK 4s 37ms
/s3/gcs/part 2/combinatoric table/engine=SummingMergeTree,replicated=True,n_cols=2000,n_tables=3,part_type=compact XError 8m 0s
/s3/gcs/part 2/combinatoric table/engine=SummingMergeTree,replicated=False,n_cols=500,n_tables=1,part_type=compact OK 4s 373ms
/s3/gcs/part 2/combinatoric table/engine=MergeTree,replicated=True,n_cols=10,n_tables=3,part_type=unspecified OK 5m 9s
/s3/gcs/part 2/combinatoric table/engine=ReplacingMergeTree,replicated=True,n_cols=10,n_tables=3,part_type=wide OK 1m 12s
/s3/gcs/part 2/combinatoric table/engine=CollapsingMergeTree,replicated=True,n_cols=500,n_tables=3,part_type=compact OK 35s 189ms
/s3/gcs/part 2/combinatoric table/engine=VersionedCollapsingMergeTree,replicated=True,n_cols=500,n_tables=1,part_type=unspecified OK 11s 182ms
/s3/gcs/part 2/combinatoric table/engine=AggregatingMergeTree,replicated=True,n_cols=10,n_tables=3,part_type=wide OK 1m 13s
/s3/gcs/part 2/combinatoric table/engine=SummingMergeTree,replicated=True,n_cols=10,n_tables=3,part_type=wide OK 1m 13s
/s3/gcs/part 2/combinatoric table/engine=SummingMergeTree,replicated=True,n_cols=10,n_tables=1,part_type=unspecified OK 9s 765ms
/s3/gcs/part 2/zero copy replication Skip 1ms
/s3/gcs/part 2/backup OK 3m 3s
/s3/gcs/part 2/backup/local and s3 disk OK 59s 403ms
/s3/gcs/part 2/backup/local and s3 volumes OK 56s 188ms
/s3/gcs/part 2/backup/s3 disk OK 1m 6s
/s3/gcs/part 2/orphans Skip 1ms
/s3/gcs/part 2/settings OK 17m 37s
/s3/gcs/part 2/settings/setting combinations OK 16m 52s
/s3/gcs/part 2/table function performance Fail 40s 522ms
/s3/gcs/part 2/table function performance/setup Fail 893ms
/s3/gcs/part 2/table function performance/wildcard Fail 39s 624ms

Generated by TestFlows Open-Source Test Framework v2.0.250110.1002922