Copyright 2025, Altinity Inc.. All Rights Reserved. All information contained herein is, and remains the property of Altinity Inc.. Any dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Altinity Inc..

Tiered Storage Test Run Report

DateJun 27, 2025 13:09
Duration1h 57m
Framework TestFlows 2.0.250110.1002922

Artifacts

Test artifacts can be found at https://altinity-build-artifacts.s3.amazonaws.com/index.html#0/d1f73c287081665b9e9ca2767a633a3cc8a41f08/regression/x86_64/with_analyzer/zookeeper/without_thread_fuzzer/tiered_storage/s3gcs/

Attributes

projectAltinity/ClickHouse
project.id159717931
packagehttps://s3.amazonaws.com/altinity-build-artifacts/24.8/d1f73c287081665b9e9ca2767a633a3cc8a41f08/package_release/clickhouse-common-static_24.8.14.10501.altinitytest_amd64.deb
version24.8.14.10501.altinitytest
user.nameEnmk
repositoryhttps://github.com/Altinity/clickhouse-regression
commit.hashe3c00be97a045aa04e9d1a6ec50cc64f4c387b70
job.nameTieredStorage
job.retry1
job.urlhttps://github.com/Altinity/ClickHouse/actions/runs/15925737638
archx86_64
localTrue
clickhouse_versionNone
clickhouse_pathhttps://s3.amazonaws.com/altinity-build-artifacts/24.8/d1f73c287081665b9e9ca2767a633a3cc8a41f08/package_release/clickhouse-common-static_24.8.14.10501.altinitytest_amd64.deb
as_binaryFalse
base_osNone
keeper_pathNone
zookeeper_versionNone
use_keeperFalse
stressFalse
collect_service_logsTrue
thread_fuzzerFalse
with_analyzerTrue
reuse_envFalse
with_minioFalse
with_s3amazonFalse
with_s3gcsTrue
aws_s3_access_keySecret(name='aws_s3_access_key')
aws_s3_key_idSecret(name='aws_s3_key_id')
aws_s3_uriSecret(name='aws_s3_uri')
gcs_uriSecret(name='gcs_uri')
gcs_key_idSecret(name='gcs_key_id')
gcs_key_secretSecret(name='gcs_key_secret')

Summary

96.7%OK
3.3%Known

Statistics

Units OK Fail XFail Retried
Modules
1
1
Features
1
1
Scenarios
94
90
4
Checks
92
91
1
Examples
84
80
4
Steps
12521
12280
239
2

Known Fails

Test NameResultMessage
/tiered storage/with s3gcs/attach or replace partition different policiesXFail 515ms
known issue
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/regression.py", line 436, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/../helpers/argparser.py", line 171, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/regression.py", line 425, in regression
    Feature(name, test=feature)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/regression.py", line 224, in feature
    Scenario(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/../tiered_storage/tests/attach_or_replace_partition_different_policies.py", line 54, in scenario
    node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/../helpers/cluster.py", line 1180, in query
    assert message in r.output, error(r.output)
           ^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert message in r.output, error(r.output)

Assertion values
  assert message in r.output, error(r.output)
         ^ is 'Exception:'
  assert message in r.output, error(r.output)
                    ^ is 
  assert message in r.output, error(r.output)
                    ^ is = ''
  assert message in r.output, error(r.output)
                 ^ is = False
  assert message in r.output, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/../helpers/cluster.py', line 1180 in 'query'

1172\|                  assert r.exitcode == exitcode, error(r.output)
1173\|  
1174\|          if message is not None:
1175\|              with (
1176\|                  Then(f"output should contain message", description=message)
1177\|                  if steps
1178\|                  else NullStep()
1179\|              ):
1180\|>                 assert message in r.output, error(r.output)
1181\|  
1182\|          if not ignore_exception:
1183\|              if message is None or "Exception:" not in message:
/tiered storage/with s3gcs/double move while selectXFail 4s 696ms
not yet supported
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/regression.py", line 436, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/../helpers/argparser.py", line 171, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/regression.py", line 425, in regression
    Feature(name, test=feature)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/regression.py", line 295, in feature
    Scenario(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/../tiered_storage/tests/double_move_while_select.py", line 120, in scenario
    assert r == [
           ^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r == [
      "jbod1" if positive else "external"
  ], error()

Assertion values
  assert r == [
         ^ is ['jbod1']
      "jbod1" if positive else "external"
  ], error()
  assert r == [
      "jbod1" if positive else "external"
                 ^ is 0
  ], error()
  assert r == [
      "jbod1" if positive else "external"
      ^ is = 'external'
  ], error()
  assert r == [
              ^ is = ['external']
      "jbod1" if positive else "external"
  ], error()
  assert r == [
           ^ is = False
    @@ -1 +1 @@
    -['jbod1']
    +['external']
      "jbod1" if positive else "external"
  ], error()
  assert r == [
  ^ is False
      "jbod1" if positive else "external"
  ], error()

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/../tiered_storage/tests/double_move_while_select.py', line 122 in 'scenario'

114\|                                      .output.strip()
115\|                                      .splitlines()
116\|                                  )
117\|                                  with Then(
118\|                                      "if select locked old part on the destination disk the move shall fail"
119\|                                  ):
120\|                                      assert r == [
121\|                                          "jbod1" if positive else "external"
122\|>                                     ], error()
123\|                          finally:
124\|                              with Finally("I join the thread"):
125\|                                  task.result(timeout=60)
/tiered storage/with s3gcs/manual move with downtimeXFail 750ms
https://altinity.atlassian.net/browse/CH-124
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/regression.py", line 436, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/../helpers/argparser.py", line 171, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/regression.py", line 425, in regression
    Feature(name, test=feature)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/regression.py", line 307, in feature
    Scenario(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/../tiered_storage/tests/manual_move_with_downtime.py", line 80, in scenario
    node.command(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/../helpers/cluster.py", line 292, in command
    return self.cluster.command(self.name, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/../helpers/cluster.py", line 2256, in command
    assert r.exitcode == exitcode, error(r.output)
           ^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  Mv: target '/external/store/*/*/detached/.' is not a directory

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 1
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 0
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/../helpers/cluster.py', line 2256 in 'command'

2248\|                  Then(f"exitcode should be {exitcode}", format_name=False)
2249\|                  if steps
2250\|                  else NullStep()
2251\|              ):
2252\|                  if str(exitcode).startswith("!="):
2253\|                      exitcode = int(str(exitcode).split("!=", 1)[-1].strip())
2254\|                      assert r.exitcode != exitcode, error(r.output)
2255\|                  else:
2256\|>                     assert r.exitcode == exitcode, error(r.output)
2257\|  
2258\|          if messages is None:
2259\|              messages = []

Results

Test Name Result Duration
/tiered storage OK 1h 57m
/tiered storage/with s3gcs OK 1h 55m
/tiered storage/with s3gcs/startup and queries OK 36s 271ms
/tiered storage/with s3gcs/metadata OK 5s 308ms
/tiered storage/with s3gcs/no changes to queries OK 7s 16ms
/tiered storage/with s3gcs/volume configuration either max_data_part_size_bytes or max_data_part_size_ratio OK 15s 4ms
/tiered storage/with s3gcs/system tables OK 197ms
/tiered storage/with s3gcs/system․detached_parts OK 1s 718ms
/tiered storage/with s3gcs/attach or replace partition different policies XFail 515ms
/tiered storage/with s3gcs/query parser OK 605ms
/tiered storage/with s3gcs/keep free space OK 11s 365ms
/tiered storage/with s3gcs/no warning about zero max data part size OK 550ms
/tiered storage/with s3gcs/round robin OK 1s 226ms
/tiered storage/with s3gcs/max data part size OK 6s 860ms
/tiered storage/with s3gcs/jbod overflow OK 40s 219ms
/tiered storage/with s3gcs/start stop moves OK 17s 769ms
/tiered storage/with s3gcs/mutate to another disk OK 9s 352ms
/tiered storage/with s3gcs/alter table policy OK 24s 95ms
/tiered storage/with s3gcs/simple replication and moves OK 52s 638ms
/tiered storage/with s3gcs/download appropriate disk OK 8s 803ms
/tiered storage/with s3gcs/download appropriate disk advanced OK 19s 616ms
/tiered storage/with s3gcs/download appropriate disk max data part size OK 24s 79ms
/tiered storage/with s3gcs/rename OK 6s 190ms
/tiered storage/with s3gcs/freeze OK 4s 910ms
/tiered storage/with s3gcs/double move while select XFail 4s 696ms
/tiered storage/with s3gcs/background move OK 4m 29s
/tiered storage/with s3gcs/alter move OK 31m 2s
/tiered storage/with s3gcs/manual move with downtime XFail 750ms
/tiered storage/with s3gcs/merge parts different volumes OK 14s 336ms
/tiered storage/with s3gcs/merge parts different volumes no space OK 5s 876ms
/tiered storage/with s3gcs/ttl moves OK 1h 8m
/tiered storage/with s3gcs/change config norestart OK 5m 36s

Generated by TestFlows Open-Source Test Framework v2.0.250110.1002922