Copyright 2025, Altinity Inc.. All Rights Reserved. All information contained herein is, and remains the property of Altinity Inc.. Any dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Altinity Inc..

S3 Test Run Report

DateApr 02, 2025 16:45
Duration2h 31m
Framework TestFlows 2.0.250110.1002922

Artifacts

Test artifacts can be found at https://altinity-build-artifacts.s3.amazonaws.com/index.html#711/b4ea76ee0ee067975b3943ba13d654c5832bbe7c/regression/aarch64/with_analyzer/zookeeper/without_thread_fuzzer/s3/gcs/

Attributes

projectAltinity/ClickHouse
project.id159717931
packagehttps://s3.amazonaws.com/altinity-build-artifacts/PRs/711/b4ea76ee0ee067975b3943ba13d654c5832bbe7c/package_aarch64/clickhouse-common-static_24.12.2.20224.altinityantalya_arm64.deb
version24.12.2.20224.altinityantalya
user.namearthurpassos
repositoryhttps://github.com/Altinity/clickhouse-regression
commit.hashbd31e738c0cedaca253d15a05ed245c41b6e0b6a
job.nameS3
job.retry1
job.urlhttps://github.com/Altinity/ClickHouse/actions/runs/14221757734
archaarch64
localTrue
clickhouse_versionNone
clickhouse_pathhttps://s3.amazonaws.com/altinity-build-artifacts/PRs/711/b4ea76ee0ee067975b3943ba13d654c5832bbe7c/package_aarch64/clickhouse-common-static_24.12.2.20224.altinityantalya_arm64.deb
as_binaryFalse
base_osNone
keeper_pathNone
zookeeper_versionNone
use_keeperFalse
stressFalse
collect_service_logsTrue
thread_fuzzerFalse
with_analyzerTrue
reuse_envFalse
storages['gcs']
minio_uriSecret(name='minio_uri')
minio_root_userSecret(name='minio_root_user')
minio_root_passwordSecret(name='minio_root_password')
aws_s3_bucketSecret(name='aws_s3_bucket')
aws_s3_regionSecret(name='aws_s3_region')
aws_s3_key_idSecret(name='aws_s3_key_id')
aws_s3_access_keySecret(name='aws_s3_access_key')
gcs_uriSecret(name='gcs_uri')
gcs_key_idSecret(name='gcs_key_id')
gcs_key_secretSecret(name='gcs_key_secret')
azure_account_nameSecret(name='azure_account_name')
azure_storage_keySecret(name='azure_storage_key')
azure_containerSecret(name='azure_container')

Summary

84.8%OK
10.6%Known
1.3%Fail

Statistics

Units Skip OK Fail Error XFail XError
Modules
1
1
Features
13
2
9
2
Scenarios
194
11
140
1
39
3
Checks
54
54
Examples
78
77
1
Steps
21698
30
21638
16
14

Fails

Test NameResultMessage
/s3Fail 2h 31m
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 694, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 171, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 348, in capture_s3_args
    return func(self, s3_args=s3_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 666, in regression
    Feature(test=gcs_regression)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 627, in gcs_regression
    Feature(test=load("s3.tests.table_function_performance", "gcs"))(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 135, in gcs
    outline(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 119, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 104, in wildcard
    assert t_elapsed < expected_time, error()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert t_elapsed < expected_time, error()

Assertion values
  assert t_elapsed < expected_time, error()
         ^ is 120.02076482772827
  assert t_elapsed < expected_time, error()
                     ^ is 120
  assert t_elapsed < expected_time, error()
                   ^ is = False
  assert t_elapsed < expected_time, error()
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py', line 104 in 'wildcard'

 96\|                  timeout=expected_time,
 97\|              )
 98\|              t_elapsed = time.time() - t_start
 99\|              metric(f"wildcard pattern='{wildcard}', i={i}", t_elapsed, "s")
100\|              is_number = r.output.strip().replace(".", "").isdigit()
101\|              assert is_number == expect_result, error(
102\|                  f"Expected a number, got {r.output}"
103\|              )
104\|>             assert t_elapsed < expected_time, error()
105\|  
106\|  
107\|  @TestOutline(Feature)
/s3/gcsFail 2h 31m
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 694, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 171, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 348, in capture_s3_args
    return func(self, s3_args=s3_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 666, in regression
    Feature(test=gcs_regression)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 627, in gcs_regression
    Feature(test=load("s3.tests.table_function_performance", "gcs"))(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 135, in gcs
    outline(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 119, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 104, in wildcard
    assert t_elapsed < expected_time, error()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert t_elapsed < expected_time, error()

Assertion values
  assert t_elapsed < expected_time, error()
         ^ is 120.02076482772827
  assert t_elapsed < expected_time, error()
                     ^ is 120
  assert t_elapsed < expected_time, error()
                   ^ is = False
  assert t_elapsed < expected_time, error()
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py', line 104 in 'wildcard'

 96\|                  timeout=expected_time,
 97\|              )
 98\|              t_elapsed = time.time() - t_start
 99\|              metric(f"wildcard pattern='{wildcard}', i={i}", t_elapsed, "s")
100\|              is_number = r.output.strip().replace(".", "").isdigit()
101\|              assert is_number == expect_result, error(
102\|                  f"Expected a number, got {r.output}"
103\|              )
104\|>             assert t_elapsed < expected_time, error()
105\|  
106\|  
107\|  @TestOutline(Feature)
/s3/gcs/table function performanceFail 13m 16s
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 694, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 171, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 348, in capture_s3_args
    return func(self, s3_args=s3_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 666, in regression
    Feature(test=gcs_regression)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 627, in gcs_regression
    Feature(test=load("s3.tests.table_function_performance", "gcs"))(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 135, in gcs
    outline(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 119, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 104, in wildcard
    assert t_elapsed < expected_time, error()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert t_elapsed < expected_time, error()

Assertion values
  assert t_elapsed < expected_time, error()
         ^ is 120.02076482772827
  assert t_elapsed < expected_time, error()
                     ^ is 120
  assert t_elapsed < expected_time, error()
                   ^ is = False
  assert t_elapsed < expected_time, error()
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py', line 104 in 'wildcard'

 96\|                  timeout=expected_time,
 97\|              )
 98\|              t_elapsed = time.time() - t_start
 99\|              metric(f"wildcard pattern='{wildcard}', i={i}", t_elapsed, "s")
100\|              is_number = r.output.strip().replace(".", "").isdigit()
101\|              assert is_number == expect_result, error(
102\|                  f"Expected a number, got {r.output}"
103\|              )
104\|>             assert t_elapsed < expected_time, error()
105\|  
106\|  
107\|  @TestOutline(Feature)
/s3/gcs/table function performance/wildcardFail 13m 15s
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 694, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 171, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 348, in capture_s3_args
    return func(self, s3_args=s3_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 666, in regression
    Feature(test=gcs_regression)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 627, in gcs_regression
    Feature(test=load("s3.tests.table_function_performance", "gcs"))(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 135, in gcs
    outline(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 119, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py", line 104, in wildcard
    assert t_elapsed < expected_time, error()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert t_elapsed < expected_time, error()

Assertion values
  assert t_elapsed < expected_time, error()
         ^ is 120.02076482772827
  assert t_elapsed < expected_time, error()
                     ^ is 120
  assert t_elapsed < expected_time, error()
                   ^ is = False
  assert t_elapsed < expected_time, error()
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_performance.py', line 104 in 'wildcard'

 96\|                  timeout=expected_time,
 97\|              )
 98\|              t_elapsed = time.time() - t_start
 99\|              metric(f"wildcard pattern='{wildcard}', i={i}", t_elapsed, "s")
100\|              is_number = r.output.strip().replace(".", "").isdigit()
101\|              assert is_number == expect_result, error(
102\|                  f"Expected a number, got {r.output}"
103\|              )
104\|>             assert t_elapsed < expected_time, error()
105\|  
106\|  
107\|  @TestOutline(Feature)

Known Fails

Test NameResultMessage
/s3/gcs/invalid table function/invalid pathXError 30s 54ms
https://github.com/ClickHouse/ClickHouse/issues/59084
ExpectTimeoutError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/venv/lib/python3.12/site-packages/testflows/uexpect/uexpect.py", line 223, in read
    d = self.queue.get(timeout=timeleft)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/queue.py", line 179, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/venv/lib/python3.12/site-packages/testflows/uexpect/uexpect.py", line 187, in expect
    data = self.read(timeout=min(timeleft, 0.1), raise_exception=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/venv/lib/python3.12/site-packages/testflows/uexpect/uexpect.py", line 235, in read
    raise TimeoutError(timeout)
testflows.uexpect.uexpect.TimeoutError: Timeout 0.042s

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 694, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 171, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 348, in capture_s3_args
    return func(self, s3_args=s3_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 666, in regression
    Feature(test=gcs_regression)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 610, in gcs_regression
    Feature(test=load("s3.tests.table_function_invalid", "gcs"))(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_invalid.py", line 411, in gcs
    outline(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_invalid.py", line 395, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_invalid.py", line 114, in invalid_path
    insert_to_s3_function_invalid(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_invalid.py", line 34, in insert_to_s3_function_invalid
    node.query(query, message=message, exitcode=exitcode, timeout=timeout)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1140, in query
    r = self.cluster.bash(self.name)(command, *args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
testflows.uexpect.uexpect.ExpectTimeoutError: Timeout 30.000s for '(bash# )\|(\n)'
/s3/gcs/invalid table function/invalid wildcardXFail 1s 125ms
doesn't work <25.1 https://github.com/ClickHouse/ClickHouse/issues/75492
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 694, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 171, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 348, in capture_s3_args
    return func(self, s3_args=s3_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 666, in regression
    Feature(test=gcs_regression)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 610, in gcs_regression
    Feature(test=load("s3.tests.table_function_invalid", "gcs"))(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_invalid.py", line 411, in gcs
    outline(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_invalid.py", line 395, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_invalid.py", line 158, in invalid_wildcard
    insert_from_s3_function_invalid(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/table_function_invalid.py", line 66, in insert_from_s3_function_invalid
    node.query(query, message=message, exitcode=exitcode, timeout=timeout)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1172, in query
    assert r.exitcode == exitcode, error(r.output)
           ^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 243
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 1172 in 'query'

1164\|                      **kwargs,
1165\|                  )
1166\|  
1167\|          if no_checks:
1168\|              return r
1169\|  
1170\|          if exitcode is not None:
1171\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
1172\|>                 assert r.exitcode == exitcode, error(r.output)
1173\|  
1174\|          if message is not None:
1175\|              with (
/s3/gcs/disk/cacheXFail 1ms
Under development for 22.8 and newer.
None
/s3/gcs/disk/cache defaultXFail 4ms
Under development for 22.8 and newer.
None
/s3/gcs/disk/cache pathXFail 785us
Under development for 22.8 and newer.
None
/s3/gcs/disk/low cardinality offsetXFail 50s 403ms
https://github.com/ClickHouse/ClickHouse/pull/44875
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 694, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 171, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 348, in capture_s3_args
    return func(self, s3_args=s3_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 666, in regression
    Feature(test=gcs_regression)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 611, in gcs_regression
    Feature(test=load("s3.tests.disk", "gcs"))(uri=uri, bucket_prefix=bucket_prefix)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk.py", line 2560, in gcs
    disk_tests(uri=uri, bucket_prefix=bucket_prefix)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk.py", line 2538, in disk_tests
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk.py", line 2500, in low_cardinality_offset
    assert output == "23999\n", error()
           ^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert output == "23999\n", error()

Assertion values
  assert output == "23999\n", error()
         ^ is '23999'
  assert output == "23999\n", error()
                ^ is = False

  assert output == "23999\n", error()
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk.py', line 2500 in 'low_cardinality_offset'

2492\|                      "1",
2493\|                  ),
2494\|                  (
2495\|                      "merge_tree_min_bytes_for_concurrent_read_for_remote_filesystem",
2496\|                      "1",
2497\|                  ),
2498\|              ],
2499\|          ).output
2500\|>         assert output == "23999\n", error()
2501\|  
2502\|  
2503\|  @TestFeature
/s3/gcs/disk/no restartXFail 1ms
https://github.com/ClickHouse/ClickHouse/issues/58924
None
/s3/gcs/invalid disk/cache path conflictXFail 1ms
Under development for 22.8 and newer.
None
/s3/gcs/combinatoric table/engine=VersionedCollapsingMergeTree,replicated=True,n_cols=2000,n_tables=3,part_type=wideXFail 4m 23s
Needs investigation, rows not appearing
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 694, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 171, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 348, in capture_s3_args
    return func(self, s3_args=s3_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 666, in regression
    Feature(test=gcs_regression)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 616, in gcs_regression
    Feature(test=load("s3.tests.combinatoric_table", "feature"))(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/combinatoric_table.py", line 182, in feature
    Scenario(title, test=check_table_combination)(**table_config)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/combinatoric_table.py", line 149, in check_table_combination
    retry(assert_row_count, timeout=60, delay=5)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py", line 1668, in assert_row_count
    assert rows == actual_count, error()
           ^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert rows == actual_count, error()

Assertion values
  assert rows == actual_count, error()
         ^ is 1500
  assert rows == actual_count, error()
                 ^ is 0
  assert rows == actual_count, error()
              ^ is = False
  assert rows == actual_count, error()
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py', line 1668 in 'assert_row_count'

1660\|  
1661\|  @TestStep(Then)
1662\|  def assert_row_count(self, node, table_name: str, rows: int = 1000000):
1663\|      """Assert that the number of rows in a table is as expected."""
1664\|      if node is None:
1665\|          node = current().context.node
1666\|  
1667\|      actual_count = get_row_count(node=node, table_name=table_name)
1668\|>     assert rows == actual_count, error()
1669\|  
1670\|  
1671\|  @TestStep(Then)
/s3/gcs/combinatoric table/engine=AggregatingMergeTree,replicated=True,n_cols=2000,n_tables=3,part_type=compactXError 8m 2s
Times out, needs investigation
ExpectTimeoutError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/venv/lib/python3.12/site-packages/testflows/uexpect/uexpect.py", line 223, in read
    d = self.queue.get(timeout=timeleft)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/queue.py", line 179, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/venv/lib/python3.12/site-packages/testflows/uexpect/uexpect.py", line 187, in expect
    data = self.read(timeout=min(timeleft, 0.1), raise_exception=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/venv/lib/python3.12/site-packages/testflows/uexpect/uexpect.py", line 235, in read
    raise TimeoutError(timeout)
testflows.uexpect.uexpect.TimeoutError: Timeout 0.018s

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 694, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 171, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 348, in capture_s3_args
    return func(self, s3_args=s3_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 666, in regression
    Feature(test=gcs_regression)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 616, in gcs_regression
    Feature(test=load("s3.tests.combinatoric_table", "feature"))(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/combinatoric_table.py", line 182, in feature
    Scenario(title, test=check_table_combination)(**table_config)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/combinatoric_table.py", line 117, in check_table_combination
    table = create_test_table(
            ^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/combinatoric_table.py", line 81, in create_test_table
    yield create_table(
          ^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/tables.py", line 482, in create_table
    node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1116, in query
    r = self.cluster.bash(None)(command, *args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
testflows.uexpect.uexpect.ExpectTimeoutError: Timeout 300.000s for '(bash# )\|(\n)'
/s3/gcs/combinatoric table/engine=SummingMergeTree,replicated=True,n_cols=2000,n_tables=3,part_type=compactXError 8m 2s
Times out, needs investigation
ExpectTimeoutError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/venv/lib/python3.12/site-packages/testflows/uexpect/uexpect.py", line 223, in read
    d = self.queue.get(timeout=timeleft)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/queue.py", line 179, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/venv/lib/python3.12/site-packages/testflows/uexpect/uexpect.py", line 187, in expect
    data = self.read(timeout=min(timeleft, 0.1), raise_exception=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/venv/lib/python3.12/site-packages/testflows/uexpect/uexpect.py", line 235, in read
    raise TimeoutError(timeout)
testflows.uexpect.uexpect.TimeoutError: Timeout 0.012s

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 694, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 171, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 348, in capture_s3_args
    return func(self, s3_args=s3_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 666, in regression
    Feature(test=gcs_regression)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 616, in gcs_regression
    Feature(test=load("s3.tests.combinatoric_table", "feature"))(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/combinatoric_table.py", line 182, in feature
    Scenario(title, test=check_table_combination)(**table_config)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/combinatoric_table.py", line 117, in check_table_combination
    table = create_test_table(
            ^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/combinatoric_table.py", line 81, in create_test_table
    yield create_table(
          ^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/tables.py", line 482, in create_table
    node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1116, in query
    r = self.cluster.bash(None)(command, *args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
testflows.uexpect.uexpect.ExpectTimeoutError: Timeout 300.000s for '(bash# )\|(\n)'

Results

Test Name Result Duration
/s3 Fail 2h 31m
/s3/gcs Fail 2h 31m
/s3/gcs/sanity OK 1m 5s
/s3/gcs/sanity/sanity OK 15s 774ms
/s3/gcs/table function OK 1m 43s
/s3/gcs/table function/auto OK 8s 991ms
/s3/gcs/table function/compression OK 8s 412ms
/s3/gcs/table function/credentials OK 861ms
/s3/gcs/table function/data format OK 14s 830ms
/s3/gcs/table function/measure file size Skip 1ms
/s3/gcs/table function/multipart OK 7s 627ms
/s3/gcs/table function/multiple columns OK 1s 262ms
/s3/gcs/table function/partition OK 1s 422ms
/s3/gcs/table function/remote host filter OK 48s 341ms
/s3/gcs/table function/syntax OK 1s 36ms
/s3/gcs/table function/wildcard OK 8s 106ms
/s3/gcs/invalid table function OK 34s 807ms
/s3/gcs/invalid table function/empty path OK 250ms
/s3/gcs/invalid table function/empty structure OK 247ms
/s3/gcs/invalid table function/invalid bucket OK 202ms
/s3/gcs/invalid table function/invalid compression OK 256ms
/s3/gcs/invalid table function/invalid credentials OK 480ms
/s3/gcs/invalid table function/invalid format OK 493ms
/s3/gcs/invalid table function/invalid path XError 30s 54ms
/s3/gcs/invalid table function/invalid region Skip 835us
/s3/gcs/invalid table function/invalid structure OK 1s 673ms
/s3/gcs/invalid table function/invalid wildcard XFail 1s 125ms
/s3/gcs/disk OK 40m 26s
/s3/gcs/disk/access OK 51s 450ms
/s3/gcs/disk/access skip check OK 54s 526ms
/s3/gcs/disk/add storage OK 1m 48s
/s3/gcs/disk/alter move OK 1m 2s
/s3/gcs/disk/alter on cluster modify ttl OK 1m 54s
/s3/gcs/disk/cache XFail 1ms
/s3/gcs/disk/cache default XFail 4ms
/s3/gcs/disk/cache path XFail 785us
/s3/gcs/disk/compact parts OK 50s 393ms
/s3/gcs/disk/config over restart OK 1m 16s
/s3/gcs/disk/default move factor OK 1m 24s
/s3/gcs/disk/delete OK 3m 24s
/s3/gcs/disk/download appropriate disk OK 1m 26s
/s3/gcs/disk/drop sync OK 56s 902ms
/s3/gcs/disk/environment credentials Skip 1ms
/s3/gcs/disk/exports OK 52s 265ms
/s3/gcs/disk/generic url Skip 23ms
/s3/gcs/disk/imports OK 52s 373ms
/s3/gcs/disk/log OK 3m 3s
/s3/gcs/disk/low cardinality offset XFail 50s 403ms
/s3/gcs/disk/max single part upload size syntax OK 54s 616ms
/s3/gcs/disk/mergetree OK 4m 9s
/s3/gcs/disk/mergetree collapsing OK 1m 4s
/s3/gcs/disk/mergetree versionedcollapsing OK 1m 2s
/s3/gcs/disk/metadata OK 1m 1s
/s3/gcs/disk/min bytes for seek syntax OK 52s 802ms
/s3/gcs/disk/multiple storage OK 1m 2s
/s3/gcs/disk/multiple storage query OK 1m 3s
/s3/gcs/disk/no restart XFail 1ms
/s3/gcs/disk/perform ttl move on insert OK 1m 45s
/s3/gcs/disk/perform ttl move on insert default OK 1m 3s
/s3/gcs/disk/performance ttl move OK 1m 14s
/s3/gcs/disk/remote host filter OK 1m 48s
/s3/gcs/disk/specific url Skip 1ms
/s3/gcs/disk/syntax OK 1m 2s
/s3/gcs/disk/wide parts OK 50s 938ms
/s3/gcs/invalid disk OK 3m 48s
/s3/gcs/invalid disk/access default OK 9s 687ms
/s3/gcs/invalid disk/access failed OK 10s 709ms
/s3/gcs/invalid disk/access failed skip check OK 50s 587ms
/s3/gcs/invalid disk/cache path conflict XFail 1ms
/s3/gcs/invalid disk/empty endpoint OK 10s 723ms
/s3/gcs/invalid disk/invalid endpoint OK 2m 5s
/s3/gcs/invalid disk/invalid type OK 21s 381ms
/s3/gcs/alter OK 16m 28s
/s3/gcs/alter/normal OK 4m 7s
/s3/gcs/alter/normal/attach from OK 13s 894ms
/s3/gcs/alter/normal/columns OK 19s 500ms
/s3/gcs/alter/normal/detach OK 26s 343ms
/s3/gcs/alter/normal/drop OK 50s 355ms
/s3/gcs/alter/normal/fetch OK 31s 112ms
/s3/gcs/alter/normal/freeze OK 22s 31ms
/s3/gcs/alter/normal/index OK 10s 672ms
/s3/gcs/alter/normal/move to table OK 19s 487ms
/s3/gcs/alter/normal/order by OK 7s 432ms
/s3/gcs/alter/normal/projection OK 12s 464ms
/s3/gcs/alter/normal/replace OK 17s 837ms
/s3/gcs/alter/normal/sample by OK 7s 396ms
/s3/gcs/alter/normal/update delete OK 8s 755ms
/s3/gcs/alter/encrypted OK 5m 25s
/s3/gcs/alter/encrypted/attach from OK 15s 13ms
/s3/gcs/alter/encrypted/columns OK 18s 146ms
/s3/gcs/alter/encrypted/detach OK 36s 767ms
/s3/gcs/alter/encrypted/drop OK 1m 12s
/s3/gcs/alter/encrypted/fetch OK 37s 798ms
/s3/gcs/alter/encrypted/freeze OK 30s 848ms
/s3/gcs/alter/encrypted/index OK 16s 115ms
/s3/gcs/alter/encrypted/move to table OK 29s 317ms
/s3/gcs/alter/encrypted/order by OK 8s 380ms
/s3/gcs/alter/encrypted/projection OK 17s 46ms
/s3/gcs/alter/encrypted/replace OK 20s 950ms
/s3/gcs/alter/encrypted/sample by OK 8s 1ms
/s3/gcs/alter/encrypted/update delete OK 14s 450ms
/s3/gcs/alter/zero copy OK 2m 10s
/s3/gcs/alter/zero copy/attach from OK 11s 452ms
/s3/gcs/alter/zero copy/columns OK 17s 990ms
/s3/gcs/alter/zero copy/detach Skip 1ms
/s3/gcs/alter/zero copy/drop Skip 1ms
/s3/gcs/alter/zero copy/fetch Skip 848us
/s3/gcs/alter/zero copy/freeze OK 20s 629ms
/s3/gcs/alter/zero copy/index OK 10s 696ms
/s3/gcs/alter/zero copy/move to table OK 17s 380ms
/s3/gcs/alter/zero copy/order by OK 7s 440ms
/s3/gcs/alter/zero copy/projection OK 10s 222ms
/s3/gcs/alter/zero copy/replace OK 18s 516ms
/s3/gcs/alter/zero copy/sample by OK 7s 374ms
/s3/gcs/alter/zero copy/update delete OK 8s 466ms
/s3/gcs/alter/zero copy encrypted OK 3m 2s
/s3/gcs/alter/zero copy encrypted/attach from OK 17s 615ms
/s3/gcs/alter/zero copy encrypted/columns OK 18s 396ms
/s3/gcs/alter/zero copy encrypted/detach Skip 1ms
/s3/gcs/alter/zero copy encrypted/drop Skip 1ms
/s3/gcs/alter/zero copy encrypted/fetch Skip 914us
/s3/gcs/alter/zero copy encrypted/freeze OK 30s 659ms
/s3/gcs/alter/zero copy encrypted/index OK 15s 365ms
/s3/gcs/alter/zero copy encrypted/move to table OK 29s 940ms
/s3/gcs/alter/zero copy encrypted/order by OK 8s 544ms
/s3/gcs/alter/zero copy encrypted/projection OK 17s 507ms
/s3/gcs/alter/zero copy encrypted/replace OK 20s 604ms
/s3/gcs/alter/zero copy encrypted/sample by OK 8s 818ms
/s3/gcs/alter/zero copy encrypted/update delete OK 15s 480ms
/s3/gcs/combinatoric table OK 37m 46s
/s3/gcs/combinatoric table/engine=MergeTree,replicated=True,n_cols=2000,n_tables=3,part_type=compact OK 42s 495ms
/s3/gcs/combinatoric table/engine=MergeTree,replicated=False,n_cols=500,n_tables=1,part_type=wide OK 4m 46s
/s3/gcs/combinatoric table/engine=ReplacingMergeTree,replicated=True,n_cols=500,n_tables=3,part_type=unspecified OK 40s 498ms
/s3/gcs/combinatoric table/engine=ReplacingMergeTree,replicated=False,n_cols=2000,n_tables=1,part_type=compact OK 6s 924ms
/s3/gcs/combinatoric table/engine=CollapsingMergeTree,replicated=True,n_cols=10,n_tables=1,part_type=wide OK 23s 485ms
/s3/gcs/combinatoric table/engine=CollapsingMergeTree,replicated=False,n_cols=2000,n_tables=3,part_type=unspecified OK 21s 381ms
/s3/gcs/combinatoric table/engine=VersionedCollapsingMergeTree,replicated=True,n_cols=2000,n_tables=3,part_type=wide XFail 4m 23s
/s3/gcs/combinatoric table/engine=VersionedCollapsingMergeTree,replicated=False,n_cols=10,n_tables=3,part_type=compact OK 11s 286ms
/s3/gcs/combinatoric table/engine=AggregatingMergeTree,replicated=True,n_cols=2000,n_tables=3,part_type=compact XError 8m 2s
/s3/gcs/combinatoric table/engine=AggregatingMergeTree,replicated=False,n_cols=500,n_tables=1,part_type=unspecified OK 6s 858ms
/s3/gcs/combinatoric table/engine=SummingMergeTree,replicated=True,n_cols=2000,n_tables=3,part_type=compact XError 8m 2s
/s3/gcs/combinatoric table/engine=SummingMergeTree,replicated=False,n_cols=500,n_tables=1,part_type=compact OK 6s 975ms
/s3/gcs/combinatoric table/engine=MergeTree,replicated=True,n_cols=10,n_tables=3,part_type=unspecified OK 4m 17s
/s3/gcs/combinatoric table/engine=ReplacingMergeTree,replicated=True,n_cols=10,n_tables=3,part_type=wide OK 1m 13s
/s3/gcs/combinatoric table/engine=CollapsingMergeTree,replicated=True,n_cols=500,n_tables=3,part_type=compact OK 38s 707ms
/s3/gcs/combinatoric table/engine=VersionedCollapsingMergeTree,replicated=True,n_cols=500,n_tables=1,part_type=unspecified OK 12s 735ms
/s3/gcs/combinatoric table/engine=AggregatingMergeTree,replicated=True,n_cols=10,n_tables=3,part_type=wide OK 1m 12s
/s3/gcs/combinatoric table/engine=SummingMergeTree,replicated=True,n_cols=10,n_tables=3,part_type=wide OK 1m 13s
/s3/gcs/combinatoric table/engine=SummingMergeTree,replicated=True,n_cols=10,n_tables=1,part_type=unspecified OK 10s 733ms
/s3/gcs/zero copy replication Skip 2ms
/s3/gcs/backup OK 3m 25s
/s3/gcs/backup/local and s3 disk OK 1m 9s
/s3/gcs/backup/local and s3 volumes OK 1m 3s
/s3/gcs/backup/s3 disk OK 1m 12s
/s3/gcs/orphans Skip 1ms
/s3/gcs/settings OK 28m 52s
/s3/gcs/settings/setting combinations OK 28m 1s
/s3/gcs/table function performance Fail 13m 16s
/s3/gcs/table function performance/setup OK 807ms
/s3/gcs/table function performance/wildcard Fail 13m 15s

Generated by TestFlows Open-Source Test Framework v2.0.250110.1002922