Swarms Test Run Report

DateApr 13, 2026 15:19
Duration30m 41s
Framework TestFlows 2.0.250110.1002922

Artifacts

Test artifacts can be found at https://altinity-build-artifacts.s3.amazonaws.com/index.html#REFs/antalya-26.3/803f9b027095ed2c6ad66a2e08d3de6a0c1cc03e/regression/

Attributes

projectAltinity/ClickHouse
project.id159717931
user.namestrtgbb
version26.3.3.20001.altinityantalya
packagehttps://altinity-build-artifacts.s3.amazonaws.com/REFs/antalya-26.3/803f9b027095ed2c6ad66a2e08d3de6a0c1cc03e/build_arm_binary/clickhouse
repositoryhttps://github.com/Altinity/clickhouse-regression
commit.hashc7897a6a858a9ef9c7b3c519e7291cfd3c2ec646
job.nameswarms
job.retry1
job.urlhttps://github.com/Altinity/ClickHouse/actions/runs/24348689732
archaarch64
localTrue
clickhouse_versionNone
clickhouse_pathhttps://altinity-build-artifacts.s3.amazonaws.com/REFs/antalya-26.3/803f9b027095ed2c6ad66a2e08d3de6a0c1cc03e/build_arm_binary/clickhouse
as_binaryFalse
base_osNone
keeper_pathNone
zookeeper_versionNone
use_keeperFalse
stressFalse
collect_service_logsTrue
thread_fuzzerFalse
with_analyzerTrue
reuse_envFalse
cicdTrue
minio_uriSecret(name='minio_uri')
minio_root_userSecret(name='minio_root_user')
minio_root_passwordSecret(name='minio_root_password')

Summary

12.9%OK
87.1%Fail

Statistics

Units Skip OK Fail
Modules
1
1
Features
10
1
9
Scenarios
20
3
17
Steps
2201
14
2122
65

Fails

Test NameResultMessage
/swarmsFail 30m 41s
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 109, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 179, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 450, in capture_minio_args
    return func(self, minio_args=minio_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 102, in regression
    Feature(test=load("swarms.tests.feature", "feature"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/feature.py", line 8, in feature
    Feature(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/swarm_sanity.py", line 464, in feature
    Scenario(test=cluster_with_only_one_observer_node)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/swarm_sanity.py", line 94, in cluster_with_only_one_observer_node
    s3_steps.read_data_with_s3_table_function(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/steps/s3_steps.py", line 73, in read_data_with_s3_table_function
    result = node.query(
             ^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py", line 1151, in query
    assert r.exitcode == exitcode, error(r.output)
           ^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  Code: 552. DB::Exception: Unrecognized option '--object_storage_cluster'. (UNRECOGNIZED_ARGUMENTS)

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 40
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 189
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py', line 1151 in 'query'

1143\|                      **kwargs,
1144\|                  )
1145\|  
1146\|          if no_checks:
1147\|              return r
1148\|  
1149\|          if exitcode is not None:
1150\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
1151\|>                 assert r.exitcode == exitcode, error(r.output)
1152\|  
1153\|          if message is not None:
1154\|              with Then(f"output should contain message", description=message) if steps else NullStep():
/swarms/featureFail 26m 51s
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 109, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 179, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 450, in capture_minio_args
    return func(self, minio_args=minio_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 102, in regression
    Feature(test=load("swarms.tests.feature", "feature"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/feature.py", line 8, in feature
    Feature(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/swarm_sanity.py", line 464, in feature
    Scenario(test=cluster_with_only_one_observer_node)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/swarm_sanity.py", line 94, in cluster_with_only_one_observer_node
    s3_steps.read_data_with_s3_table_function(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/steps/s3_steps.py", line 73, in read_data_with_s3_table_function
    result = node.query(
             ^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py", line 1151, in query
    assert r.exitcode == exitcode, error(r.output)
           ^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  Code: 552. DB::Exception: Unrecognized option '--object_storage_cluster'. (UNRECOGNIZED_ARGUMENTS)

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 40
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 189
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py', line 1151 in 'query'

1143\|                      **kwargs,
1144\|                  )
1145\|  
1146\|          if no_checks:
1147\|              return r
1148\|  
1149\|          if exitcode is not None:
1150\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
1151\|>                 assert r.exitcode == exitcode, error(r.output)
1152\|  
1153\|          if message is not None:
1154\|              with Then(f"output should contain message", description=message) if steps else NullStep():
/swarms/feature/swarm sanityFail 3m 14s
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 109, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 179, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 450, in capture_minio_args
    return func(self, minio_args=minio_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 102, in regression
    Feature(test=load("swarms.tests.feature", "feature"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/feature.py", line 8, in feature
    Feature(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/swarm_sanity.py", line 464, in feature
    Scenario(test=cluster_with_only_one_observer_node)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/swarm_sanity.py", line 94, in cluster_with_only_one_observer_node
    s3_steps.read_data_with_s3_table_function(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/steps/s3_steps.py", line 73, in read_data_with_s3_table_function
    result = node.query(
             ^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py", line 1151, in query
    assert r.exitcode == exitcode, error(r.output)
           ^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  Code: 552. DB::Exception: Unrecognized option '--object_storage_cluster'. (UNRECOGNIZED_ARGUMENTS)

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 40
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 189
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py', line 1151 in 'query'

1143\|                      **kwargs,
1144\|                  )
1145\|  
1146\|          if no_checks:
1147\|              return r
1148\|  
1149\|          if exitcode is not None:
1150\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
1151\|>                 assert r.exitcode == exitcode, error(r.output)
1152\|  
1153\|          if message is not None:
1154\|              with Then(f"output should contain message", description=message) if steps else NullStep():
/swarms/feature/swarm sanity/cluster with only one observer nodeFail 20s 199ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 109, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 179, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 450, in capture_minio_args
    return func(self, minio_args=minio_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 102, in regression
    Feature(test=load("swarms.tests.feature", "feature"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/feature.py", line 8, in feature
    Feature(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/swarm_sanity.py", line 464, in feature
    Scenario(test=cluster_with_only_one_observer_node)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/swarm_sanity.py", line 94, in cluster_with_only_one_observer_node
    s3_steps.read_data_with_s3_table_function(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/steps/s3_steps.py", line 73, in read_data_with_s3_table_function
    result = node.query(
             ^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py", line 1151, in query
    assert r.exitcode == exitcode, error(r.output)
           ^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  Code: 552. DB::Exception: Unrecognized option '--object_storage_cluster'. (UNRECOGNIZED_ARGUMENTS)

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 40
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 189
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py', line 1151 in 'query'

1143\|                      **kwargs,
1144\|                  )
1145\|  
1146\|          if no_checks:
1147\|              return r
1148\|  
1149\|          if exitcode is not None:
1150\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
1151\|>                 assert r.exitcode == exitcode, error(r.output)
1152\|  
1153\|          if message is not None:
1154\|              with Then(f"output should contain message", description=message) if steps else NullStep():
/swarms/feature/swarm sanity/check scale up and downFail 34s 632ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 109, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 179, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 450, in capture_minio_args
    return func(self, minio_args=minio_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 102, in regression
    Feature(test=load("swarms.tests.feature", "feature"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/feature.py", line 8, in feature
    Feature(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/swarm_sanity.py", line 468, in feature
    Scenario(test=check_scale_up_and_down)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/swarm_sanity.py", line 226, in check_scale_up_and_down
    result = s3_steps.read_data_with_s3_table_function(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/steps/s3_steps.py", line 73, in read_data_with_s3_table_function
    result = node.query(
             ^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py", line 1163, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 552. DB::Exception: Unrecognized option '--object_storage_cluster'. (UNRECOGNIZED_ARGUMENTS)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py', line 1163 in 'query'

1155\|                  assert message in r.output, error(r.output)
1156\|  
1157\|          if not ignore_exception:
1158\|              if message is None or "Exception:" not in message:
1159\|                  with Then("check if output has exception") if steps else NullStep():
1160\|                      if "Exception:" in r.output:
1161\|                          if raise_on_exception:
1162\|                              raise QueryRuntimeException(r.output)
1163\|>                         assert False, error(r.output)
1164\|  
1165\|          return r
1166\|
/swarms/feature/swarm sanity/swarm examplesFail 59s 635ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 109, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 179, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 450, in capture_minio_args
    return func(self, minio_args=minio_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 102, in regression
    Feature(test=load("swarms.tests.feature", "feature"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/feature.py", line 8, in feature
    Feature(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/swarm_sanity.py", line 472, in feature
    Scenario(test=swarm_examples)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/swarm_sanity.py", line 363, in swarm_examples
    result = s3_steps.read_data_with_s3_table_function(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/steps/s3_steps.py", line 73, in read_data_with_s3_table_function
    result = node.query(
             ^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py", line 1163, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 552. DB::Exception: Unrecognized option '--object_storage_cluster'. (UNRECOGNIZED_ARGUMENTS)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py', line 1163 in 'query'

1155\|                  assert message in r.output, error(r.output)
1156\|  
1157\|          if not ignore_exception:
1158\|              if message is None or "Exception:" not in message:
1159\|                  with Then("check if output has exception") if steps else NullStep():
1160\|                      if "Exception:" in r.output:
1161\|                          if raise_on_exception:
1162\|                              raise QueryRuntimeException(r.output)
1163\|>                         assert False, error(r.output)
1164\|  
1165\|          return r
1166\|
/swarms/feature/swarm sanity/cluster with one observer node and one swarm nodeFail 33s 773ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 109, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 179, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 450, in capture_minio_args
    return func(self, minio_args=minio_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 102, in regression
    Feature(test=load("swarms.tests.feature", "feature"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/feature.py", line 8, in feature
    Feature(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/swarm_sanity.py", line 476, in feature
    Scenario(test=cluster_with_one_observer_node_and_one_swarm_node)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/swarm_sanity.py", line 138, in cluster_with_one_observer_node_and_one_swarm_node
    result = read_data_with_icebergS3_table_function(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../iceberg/tests/steps/icebergS3.py", line 127, in read_data_with_icebergS3_table_function
    result = node.query(
             ^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py", line 1163, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 552. DB::Exception: Unrecognized option '--object_storage_cluster'. (UNRECOGNIZED_ARGUMENTS)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py', line 1163 in 'query'

1155\|                  assert message in r.output, error(r.output)
1156\|  
1157\|          if not ignore_exception:
1158\|              if message is None or "Exception:" not in message:
1159\|                  with Then("check if output has exception") if steps else NullStep():
1160\|                      if "Exception:" in r.output:
1161\|                          if raise_on_exception:
1162\|                              raise QueryRuntimeException(r.output)
1163\|>                         assert False, error(r.output)
1164\|  
1165\|          return r
1166\|
/swarms/feature/cluster discoveryFail 2m 47s
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 109, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 179, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 450, in capture_minio_args
    return func(self, minio_args=minio_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 102, in regression
    Feature(test=load("swarms.tests.feature", "feature"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/feature.py", line 14, in feature
    Feature(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/cluster_discovery.py", line 353, in feature
    Scenario(test=multiple_discovery_sections)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/cluster_discovery.py", line 153, in multiple_discovery_sections
    result = s3_steps.read_data_with_s3_table_function(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/steps/s3_steps.py", line 73, in read_data_with_s3_table_function
    result = node.query(
             ^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py", line 1163, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 552. DB::Exception: Unrecognized option '--object_storage_cluster'. (UNRECOGNIZED_ARGUMENTS)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py', line 1163 in 'query'

1155\|                  assert message in r.output, error(r.output)
1156\|  
1157\|          if not ignore_exception:
1158\|              if message is None or "Exception:" not in message:
1159\|                  with Then("check if output has exception") if steps else NullStep():
1160\|                      if "Exception:" in r.output:
1161\|                          if raise_on_exception:
1162\|                              raise QueryRuntimeException(r.output)
1163\|>                         assert False, error(r.output)
1164\|  
1165\|          return r
1166\|
/swarms/feature/cluster discovery/multiple discovery sectionsFail 50s 12ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 109, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 179, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 450, in capture_minio_args
    return func(self, minio_args=minio_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 102, in regression
    Feature(test=load("swarms.tests.feature", "feature"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/feature.py", line 14, in feature
    Feature(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/cluster_discovery.py", line 353, in feature
    Scenario(test=multiple_discovery_sections)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/cluster_discovery.py", line 153, in multiple_discovery_sections
    result = s3_steps.read_data_with_s3_table_function(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/steps/s3_steps.py", line 73, in read_data_with_s3_table_function
    result = node.query(
             ^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py", line 1163, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 552. DB::Exception: Unrecognized option '--object_storage_cluster'. (UNRECOGNIZED_ARGUMENTS)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py', line 1163 in 'query'

1155\|                  assert message in r.output, error(r.output)
1156\|  
1157\|          if not ignore_exception:
1158\|              if message is None or "Exception:" not in message:
1159\|                  with Then("check if output has exception") if steps else NullStep():
1160\|                      if "Exception:" in r.output:
1161\|                          if raise_on_exception:
1162\|                              raise QueryRuntimeException(r.output)
1163\|>                         assert False, error(r.output)
1164\|  
1165\|          return r
1166\|
/swarms/feature/cluster discovery/multiple pathsFail 48s 207ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 109, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 179, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 450, in capture_minio_args
    return func(self, minio_args=minio_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 102, in regression
    Feature(test=load("swarms.tests.feature", "feature"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/feature.py", line 14, in feature
    Feature(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/cluster_discovery.py", line 356, in feature
    Scenario(test=multiple_paths)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/cluster_discovery.py", line 243, in multiple_paths
    result = s3_steps.read_data_with_s3_table_function(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/steps/s3_steps.py", line 73, in read_data_with_s3_table_function
    result = node.query(
             ^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py", line 1163, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 552. DB::Exception: Unrecognized option '--object_storage_cluster'. (UNRECOGNIZED_ARGUMENTS)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py', line 1163 in 'query'

1155\|                  assert message in r.output, error(r.output)
1156\|  
1157\|          if not ignore_exception:
1158\|              if message is None or "Exception:" not in message:
1159\|                  with Then("check if output has exception") if steps else NullStep():
1160\|                      if "Exception:" in r.output:
1161\|                          if raise_on_exception:
1162\|                              raise QueryRuntimeException(r.output)
1163\|>                         assert False, error(r.output)
1164\|  
1165\|          return r
1166\|
/swarms/feature/cluster discovery/check cluster discovery with wrong cluster nameFail 33s 598ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 109, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 179, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 450, in capture_minio_args
    return func(self, minio_args=minio_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 102, in regression
    Feature(test=load("swarms.tests.feature", "feature"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/feature.py", line 14, in feature
    Feature(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/cluster_discovery.py", line 359, in feature
    Scenario(test=check_cluster_discovery_with_wrong_cluster_name)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/cluster_discovery.py", line 322, in check_cluster_discovery_with_wrong_cluster_name
    result = s3_steps.read_data_with_s3Cluster_table_function(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/steps/s3_steps.py", line 132, in read_data_with_s3Cluster_table_function
    result = node.query(query, exitcode=exitcode, message=message)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py", line 1151, in query
    assert r.exitcode == exitcode, error(r.output)
           ^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  Received exception from server (version 26.3.3):
Code: 32. DB::Exception: Received from localhost:9000. DB::Exception: Attempt to read after eof: while receiving packet from clickhouse2:9000, 172.18.0.10, local address: 172.18.0.6:50658: While executing Remote. (ATTEMPT_TO_READ_AFTER_EOF)
(query: SELECT hostName(), count() 
            FROM s3Cluster('swarm', 'http://minio:9000/warehouse/data/data/**/**.parquet', '[masked]:Secret(name='minio_root_user')', '[masked]:Secret(name='minio_root_password')')
             GROUP BY hostName() FORMAT TabSeparated
)

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 32
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 33
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py', line 1151 in 'query'

1143\|                      **kwargs,
1144\|                  )
1145\|  
1146\|          if no_checks:
1147\|              return r
1148\|  
1149\|          if exitcode is not None:
1150\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
1151\|>                 assert r.exitcode == exitcode, error(r.output)
1152\|  
1153\|          if message is not None:
1154\|              with Then(f"output should contain message", description=message) if steps else NullStep():
/swarms/feature/node failureFail 3m 48s
AssertionError
Traceback (most recent call last):
  File "/usr/lib/python3.12/threading.py", line 1030, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.12/threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/node_failure.py", line 77, in run_long_query
    result = node.query(
             ^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py", line 1151, in query
    assert r.exitcode == exitcode, error(r.output)
           ^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  Received exception from server (version 26.3.3):
Code: 115. DB::Exception: Received from localhost:9000. DB::Exception: Unknown setting 'object_storage_cluster'. (UNKNOWN_SETTING)
(query: SELECT count(), hostName() 
            FROM datalakecatalog_db_6a0eb937_374d_11f1_b181_92000764b166.`namespace_6a0ec70e_374d_11f1_8587_92000764b166.table_6a0ec750_374d_11f1_a365_92000764b166` 
            WHERE NOT ignore(sleepEachRow(1)) 
            GROUP BY hostName()
            SETTINGS 
                object_storage_cluster='static_swarm_cluster', 
                max_threads=1


)

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 115
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 138
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py', line 1151 in 'query'

1143\|                      **kwargs,
1144\|                  )
1145\|  
1146\|          if no_checks:
1147\|              return r
1148\|  
1149\|          if exitcode is not None:
1150\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
1151\|>                 assert r.exitcode == exitcode, error(r.output)
1152\|  
1153\|          if message is not None:
1154\|              with Then(f"output should contain message", description=message) if steps else NullStep():
/swarms/feature/node failure/check restart swarm nodeFail 3s 186ms
AssertionError
Traceback (most recent call last):
  File "/usr/lib/python3.12/threading.py", line 1030, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.12/threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/node_failure.py", line 77, in run_long_query
    result = node.query(
             ^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py", line 1151, in query
    assert r.exitcode == exitcode, error(r.output)
           ^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  Received exception from server (version 26.3.3):
Code: 115. DB::Exception: Received from localhost:9000. DB::Exception: Unknown setting 'object_storage_cluster'. (UNKNOWN_SETTING)
(query: SELECT count(), hostName() 
            FROM datalakecatalog_db_6a0eb937_374d_11f1_b181_92000764b166.`namespace_6a0ec70e_374d_11f1_8587_92000764b166.table_6a0ec750_374d_11f1_a365_92000764b166` 
            WHERE NOT ignore(sleepEachRow(1)) 
            GROUP BY hostName()
            SETTINGS 
                object_storage_cluster='static_swarm_cluster', 
                max_threads=1


)

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 115
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 138
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py', line 1151 in 'query'

1143\|                      **kwargs,
1144\|                  )
1145\|  
1146\|          if no_checks:
1147\|              return r
1148\|  
1149\|          if exitcode is not None:
1150\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
1151\|>                 assert r.exitcode == exitcode, error(r.output)
1152\|  
1153\|          if message is not None:
1154\|              with Then(f"output should contain message", description=message) if steps else NullStep():
/swarms/feature/node failure/check restart clickhouse on swarm nodeFail 18s 969ms
AssertionError
Traceback (most recent call last):
  File "/usr/lib/python3.12/threading.py", line 1030, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.12/threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/node_failure.py", line 77, in run_long_query
    result = node.query(
             ^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py", line 1151, in query
    assert r.exitcode == exitcode, error(r.output)
           ^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  Received exception from server (version 26.3.3):
Code: 115. DB::Exception: Received from localhost:9000. DB::Exception: Unknown setting 'object_storage_cluster'. (UNKNOWN_SETTING)
(query: SELECT count(), hostName() 
            FROM datalakecatalog_db_6bf53c74_374d_11f1_b7e1_92000764b166.`namespace_6bf560b1_374d_11f1_ac0c_92000764b166.table_6bf56114_374d_11f1_acd6_92000764b166` 
            WHERE NOT ignore(sleepEachRow(1)) 
            GROUP BY hostName()
            SETTINGS 
                object_storage_cluster='static_swarm_cluster', 
                max_threads=1


)

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 115
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 32
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py', line 1151 in 'query'

1143\|                      **kwargs,
1144\|                  )
1145\|  
1146\|          if no_checks:
1147\|              return r
1148\|  
1149\|          if exitcode is not None:
1150\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
1151\|>                 assert r.exitcode == exitcode, error(r.output)
1152\|  
1153\|          if message is not None:
1154\|              with Then(f"output should contain message", description=message) if steps else NullStep():
/swarms/feature/node failure/network failureFail 4s 804ms
AssertionError
Traceback (most recent call last):
  File "/usr/lib/python3.12/threading.py", line 1030, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.12/threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/node_failure.py", line 77, in run_long_query
    result = node.query(
             ^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py", line 1151, in query
    assert r.exitcode == exitcode, error(r.output)
           ^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  Received exception from server (version 26.3.3):
Code: 115. DB::Exception: Received from localhost:9000. DB::Exception: Unknown setting 'object_storage_cluster'. (UNKNOWN_SETTING)
(query: SELECT count(), hostName() 
            FROM datalakecatalog_db_774425e5_374d_11f1_8317_92000764b166.`namespace_774439b6_374d_11f1_a19c_92000764b166.table_774439fb_374d_11f1_a51b_92000764b166` 
            WHERE NOT ignore(sleepEachRow(1)) 
            GROUP BY hostName()
            SETTINGS 
                object_storage_cluster='static_swarm_cluster', 
                max_threads=1


)

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 115
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 138
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py', line 1151 in 'query'

1143\|                      **kwargs,
1144\|                  )
1145\|  
1146\|          if no_checks:
1147\|              return r
1148\|  
1149\|          if exitcode is not None:
1150\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
1151\|>                 assert r.exitcode == exitcode, error(r.output)
1152\|  
1153\|          if message is not None:
1154\|              with Then(f"output should contain message", description=message) if steps else NullStep():
/swarms/feature/node failure/swarm out of disk spaceFail 37s 786ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 109, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 179, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 450, in capture_minio_args
    return func(self, minio_args=minio_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 102, in regression
    Feature(test=load("swarms.tests.feature", "feature"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/feature.py", line 17, in feature
    Feature(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/node_failure.py", line 465, in feature
    Scenario(test=swarm_out_of_disk_space)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/node_failure.py", line 253, in swarm_out_of_disk_space
    run_long_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/node_failure.py", line 77, in run_long_query
    result = node.query(
             ^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py", line 1163, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Received exception from server (version 26.3.3):
Code: 115. DB::Exception: Received from localhost:9000. DB::Exception: Unknown setting 'object_storage_cluster'. (UNKNOWN_SETTING)
(query: SELECT count(), hostName() 
            FROM datalakecatalog_db_7a211133_374d_11f1_9d56_92000764b166.`namespace_7a2122f9_374d_11f1_97db_92000764b166.table_7a212350_374d_11f1_8273_92000764b166` 
            WHERE NOT ignore(sleepEachRow(1)) 
            GROUP BY hostName()
            SETTINGS 
                object_storage_cluster='static_swarm_cluster', 
                max_threads=1


)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py', line 1163 in 'query'

1155\|                  assert message in r.output, error(r.output)
1156\|  
1157\|          if not ignore_exception:
1158\|              if message is None or "Exception:" not in message:
1159\|                  with Then("check if output has exception") if steps else NullStep():
1160\|                      if "Exception:" in r.output:
1161\|                          if raise_on_exception:
1162\|                              raise QueryRuntimeException(r.output)
1163\|>                         assert False, error(r.output)
1164\|  
1165\|          return r
1166\|
/swarms/feature/node failure/initiator out of disk spaceFail 36s 132ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 109, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 179, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 450, in capture_minio_args
    return func(self, minio_args=minio_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 102, in regression
    Feature(test=load("swarms.tests.feature", "feature"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/feature.py", line 17, in feature
    Feature(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/node_failure.py", line 471, in feature
    Scenario(test=initiator_out_of_disk_space)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/node_failure.py", line 291, in initiator_out_of_disk_space
    run_long_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/node_failure.py", line 77, in run_long_query
    result = node.query(
             ^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py", line 1163, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Received exception from server (version 26.3.3):
Code: 115. DB::Exception: Received from localhost:9000. DB::Exception: Unknown setting 'object_storage_cluster'. (UNKNOWN_SETTING)
(query: SELECT count(), hostName() 
            FROM datalakecatalog_db_90a6db63_374d_11f1_aee0_92000764b166.`namespace_90a70bbc_374d_11f1_9adb_92000764b166.table_90a70c8e_374d_11f1_aaaf_92000764b166` 
            WHERE NOT ignore(sleepEachRow(1)) 
            GROUP BY hostName()
            SETTINGS 
                object_storage_cluster='static_swarm_cluster', 
                max_threads=1


)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py', line 1163 in 'query'

1155\|                  assert message in r.output, error(r.output)
1156\|  
1157\|          if not ignore_exception:
1158\|              if message is None or "Exception:" not in message:
1159\|                  with Then("check if output has exception") if steps else NullStep():
1160\|                      if "Exception:" in r.output:
1161\|                          if raise_on_exception:
1162\|                              raise QueryRuntimeException(r.output)
1163\|>                         assert False, error(r.output)
1164\|  
1165\|          return r
1166\|
/swarms/feature/node failure/cpu overloadFail 1m 3s
AssertionError
Traceback (most recent call last):
  File "/usr/lib/python3.12/threading.py", line 1030, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.12/threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/node_failure.py", line 77, in run_long_query
    result = node.query(
             ^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py", line 1163, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Received exception from server (version 26.3.3):
Code: 115. DB::Exception: Received from localhost:9000. DB::Exception: Unknown setting 'object_storage_cluster'. (UNKNOWN_SETTING)
(query: SELECT count(), hostName() 
            FROM datalakecatalog_db_a63081cb_374d_11f1_9e4e_92000764b166.`namespace_a63096db_374d_11f1_97ae_92000764b166.table_a63097c5_374d_11f1_9ab0_92000764b166` 
            WHERE NOT ignore(sleepEachRow(0.5)) 
            GROUP BY hostName()
            SETTINGS 
                object_storage_cluster='static_swarm_cluster', 
                max_threads=1


)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py', line 1163 in 'query'

1155\|                  assert message in r.output, error(r.output)
1156\|  
1157\|          if not ignore_exception:
1158\|              if message is None or "Exception:" not in message:
1159\|                  with Then("check if output has exception") if steps else NullStep():
1160\|                      if "Exception:" in r.output:
1161\|                          if raise_on_exception:
1162\|                              raise QueryRuntimeException(r.output)
1163\|>                         assert False, error(r.output)
1164\|  
1165\|          return r
1166\|
/swarms/feature/node failure/cpu overload all swarm nodesFail 1m 3s
AssertionError
Traceback (most recent call last):
  File "/usr/lib/python3.12/threading.py", line 1030, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.12/threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/node_failure.py", line 77, in run_long_query
    result = node.query(
             ^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py", line 1151, in query
    assert r.exitcode == exitcode, error(r.output)
           ^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  Received exception from server (version 26.3.3):
Code: 115. DB::Exception: Received from localhost:9000. DB::Exception: Unknown setting 'object_storage_cluster'. (UNKNOWN_SETTING)
(query: SELECT count(), hostName() 
            FROM datalakecatalog_db_cc014428_374d_11f1_99ed_92000764b166.`namespace_cc016f80_374d_11f1_929a_92000764b166.table_cc016fea_374d_11f1_b980_92000764b166` 
            WHERE NOT ignore(sleepEachRow(0.1)) 
            GROUP BY hostName()
            SETTINGS 
                object_storage_cluster='static_swarm_cluster', 
                max_threads=1


)

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 115
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 23
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py', line 1151 in 'query'

1143\|                      **kwargs,
1144\|                  )
1145\|  
1146\|          if no_checks:
1147\|              return r
1148\|  
1149\|          if exitcode is not None:
1150\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
1151\|>                 assert r.exitcode == exitcode, error(r.output)
1152\|  
1153\|          if message is not None:
1154\|              with Then(f"output should contain message", description=message) if steps else NullStep():
/swarms/feature/swarm joinsFail 11m 2s
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 109, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 179, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 450, in capture_minio_args
    return func(self, minio_args=minio_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 102, in regression
    Feature(test=load("swarms.tests.feature", "feature"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/feature.py", line 20, in feature
    Feature(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/joins.py", line 716, in feature
    Feature(test=join_clause)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/joins.py", line 494, in join_clause
    iceberg_engine.create_experimental_iceberg_database(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../iceberg/tests/steps/iceberg_engine.py", line 181, in create_experimental_iceberg_database
    return create_experimental_iceberg_database_with_rest_catalog(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../iceberg/tests/steps/iceberg_engine.py", line 89, in create_experimental_iceberg_database_with_rest_catalog
    node.query(query, exitcode=exitcode, message=message)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py", line 1163, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 210. DB::NetException: Connection refused (localhost:9000). (NETWORK_ERROR)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py', line 1163 in 'query'

1155\|                  assert message in r.output, error(r.output)
1156\|  
1157\|          if not ignore_exception:
1158\|              if message is None or "Exception:" not in message:
1159\|                  with Then("check if output has exception") if steps else NullStep():
1160\|                      if "Exception:" in r.output:
1161\|                          if raise_on_exception:
1162\|                              raise QueryRuntimeException(r.output)
1163\|>                         assert False, error(r.output)
1164\|  
1165\|          return r
1166\|
/swarms/feature/swarm joins/join clauseFail 175ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 109, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 179, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 450, in capture_minio_args
    return func(self, minio_args=minio_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 102, in regression
    Feature(test=load("swarms.tests.feature", "feature"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/feature.py", line 20, in feature
    Feature(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/joins.py", line 716, in feature
    Feature(test=join_clause)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/joins.py", line 494, in join_clause
    iceberg_engine.create_experimental_iceberg_database(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../iceberg/tests/steps/iceberg_engine.py", line 181, in create_experimental_iceberg_database
    return create_experimental_iceberg_database_with_rest_catalog(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../iceberg/tests/steps/iceberg_engine.py", line 89, in create_experimental_iceberg_database_with_rest_catalog
    node.query(query, exitcode=exitcode, message=message)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py", line 1163, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 210. DB::NetException: Connection refused (localhost:9000). (NETWORK_ERROR)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py', line 1163 in 'query'

1155\|                  assert message in r.output, error(r.output)
1156\|  
1157\|          if not ignore_exception:
1158\|              if message is None or "Exception:" not in message:
1159\|                  with Then("check if output has exception") if steps else NullStep():
1160\|                      if "Exception:" in r.output:
1161\|                          if raise_on_exception:
1162\|                              raise QueryRuntimeException(r.output)
1163\|>                         assert False, error(r.output)
1164\|  
1165\|          return r
1166\|
/swarms/feature/swarm join sanityFail 350ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 109, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 179, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 450, in capture_minio_args
    return func(self, minio_args=minio_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 102, in regression
    Feature(test=load("swarms.tests.feature", "feature"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/feature.py", line 23, in feature
    Feature(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/swarm_join_sanity.py", line 200, in feature
    Scenario(test=join_subquery)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/swarm_join_sanity.py", line 26, in join_subquery
    iceberg_engine.create_experimental_iceberg_database(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../iceberg/tests/steps/iceberg_engine.py", line 181, in create_experimental_iceberg_database
    return create_experimental_iceberg_database_with_rest_catalog(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../iceberg/tests/steps/iceberg_engine.py", line 89, in create_experimental_iceberg_database_with_rest_catalog
    node.query(query, exitcode=exitcode, message=message)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py", line 1163, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 210. DB::NetException: Connection refused (localhost:9000). (NETWORK_ERROR)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py', line 1163 in 'query'

1155\|                  assert message in r.output, error(r.output)
1156\|  
1157\|          if not ignore_exception:
1158\|              if message is None or "Exception:" not in message:
1159\|                  with Then("check if output has exception") if steps else NullStep():
1160\|                      if "Exception:" in r.output:
1161\|                          if raise_on_exception:
1162\|                              raise QueryRuntimeException(r.output)
1163\|>                         assert False, error(r.output)
1164\|  
1165\|          return r
1166\|
/swarms/feature/swarm join sanity/join subqueryFail 174ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 109, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 179, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 450, in capture_minio_args
    return func(self, minio_args=minio_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 102, in regression
    Feature(test=load("swarms.tests.feature", "feature"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/feature.py", line 23, in feature
    Feature(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/swarm_join_sanity.py", line 200, in feature
    Scenario(test=join_subquery)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/swarm_join_sanity.py", line 26, in join_subquery
    iceberg_engine.create_experimental_iceberg_database(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../iceberg/tests/steps/iceberg_engine.py", line 181, in create_experimental_iceberg_database
    return create_experimental_iceberg_database_with_rest_catalog(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../iceberg/tests/steps/iceberg_engine.py", line 89, in create_experimental_iceberg_database_with_rest_catalog
    node.query(query, exitcode=exitcode, message=message)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py", line 1163, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 210. DB::NetException: Connection refused (localhost:9000). (NETWORK_ERROR)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py', line 1163 in 'query'

1155\|                  assert message in r.output, error(r.output)
1156\|  
1157\|          if not ignore_exception:
1158\|              if message is None or "Exception:" not in message:
1159\|                  with Then("check if output has exception") if steps else NullStep():
1160\|                      if "Exception:" in r.output:
1161\|                          if raise_on_exception:
1162\|                              raise QueryRuntimeException(r.output)
1163\|>                         assert False, error(r.output)
1164\|  
1165\|          return r
1166\|
/swarms/feature/swarm join sanity/join with clauseFail 175ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 109, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 179, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 450, in capture_minio_args
    return func(self, minio_args=minio_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 102, in regression
    Feature(test=load("swarms.tests.feature", "feature"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/feature.py", line 23, in feature
    Feature(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/swarm_join_sanity.py", line 205, in feature
    Scenario(test=join_with_clause)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/swarm_join_sanity.py", line 109, in join_with_clause
    iceberg_engine.create_experimental_iceberg_database(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../iceberg/tests/steps/iceberg_engine.py", line 181, in create_experimental_iceberg_database
    return create_experimental_iceberg_database_with_rest_catalog(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../iceberg/tests/steps/iceberg_engine.py", line 89, in create_experimental_iceberg_database_with_rest_catalog
    node.query(query, exitcode=exitcode, message=message)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py", line 1163, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 210. DB::NetException: Connection refused (localhost:9000). (NETWORK_ERROR)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py', line 1163 in 'query'

1155\|                  assert message in r.output, error(r.output)
1156\|  
1157\|          if not ignore_exception:
1158\|              if message is None or "Exception:" not in message:
1159\|                  with Then("check if output has exception") if steps else NullStep():
1160\|                      if "Exception:" in r.output:
1161\|                          if raise_on_exception:
1162\|                              raise QueryRuntimeException(r.output)
1163\|>                         assert False, error(r.output)
1164\|  
1165\|          return r
1166\|
/swarms/feature/swarm unionFail 5m 3s
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 109, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 179, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 450, in capture_minio_args
    return func(self, minio_args=minio_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 102, in regression
    Feature(test=load("swarms.tests.feature", "feature"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/feature.py", line 26, in feature
    Feature(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/union.py", line 337, in feature
    swarm_steps.add_node_to_swarm(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/steps/swarm_steps.py", line 136, in add_node_to_swarm
    change_clickhouse_config(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/common.py", line 713, in add_config
    check_preprocessed_config_is_updated()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/common.py", line 637, in check_preprocessed_config_is_updated
    assert exitcode == 0, error()
           ^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert exitcode == 0, error()

Assertion values
  assert exitcode == 0, error()
         ^ is 1
  assert exitcode == 0, error()
                  ^ is = False
  assert exitcode == 0, error()
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/common.py', line 637 in 'check_preprocessed_config_is_updated'

629\|          if settings.debug:
630\|              node.command(
631\|                  f"cat /var/lib/clickhouse/preprocessed_configs/{config.preprocessed_name}"
632\|              )
633\|  
634\|          if after_removal:
635\|              assert exitcode == 1, error()
636\|          else:
637\|>             assert exitcode == 0, error()
638\|  
639\|      def wait_for_config_to_be_loaded(user=None):
640\|          """Wait for config to be loaded."""
/swarms/feature/featureFail 18s 400ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 109, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 179, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 450, in capture_minio_args
    return func(self, minio_args=minio_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 102, in regression
    Feature(test=load("swarms.tests.feature", "feature"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/feature.py", line 29, in feature
    Feature(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/object_storage_cluster_profile_events.py", line 271, in feature
    Scenario(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/object_storage_cluster_profile_events.py", line 164, in object_storage_cluster_profile_events_with_overloaded_node
    iceberg_engine.read_data_from_clickhouse_iceberg_table(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../iceberg/tests/steps/iceberg_engine.py", line 337, in read_data_from_clickhouse_iceberg_table
    result = node.query(
             ^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py", line 1163, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 552. DB::Exception: Unrecognized option '--object_storage_cluster'. (UNRECOGNIZED_ARGUMENTS)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py', line 1163 in 'query'

1155\|                  assert message in r.output, error(r.output)
1156\|  
1157\|          if not ignore_exception:
1158\|              if message is None or "Exception:" not in message:
1159\|                  with Then("check if output has exception") if steps else NullStep():
1160\|                      if "Exception:" in r.output:
1161\|                          if raise_on_exception:
1162\|                              raise QueryRuntimeException(r.output)
1163\|>                         assert False, error(r.output)
1164\|  
1165\|          return r
1166\|
/swarms/feature/feature/object storage cluster profile events with overloaded nodeFail 18s 399ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 109, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 179, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 450, in capture_minio_args
    return func(self, minio_args=minio_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 102, in regression
    Feature(test=load("swarms.tests.feature", "feature"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/feature.py", line 29, in feature
    Feature(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/object_storage_cluster_profile_events.py", line 271, in feature
    Scenario(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/object_storage_cluster_profile_events.py", line 164, in object_storage_cluster_profile_events_with_overloaded_node
    iceberg_engine.read_data_from_clickhouse_iceberg_table(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../iceberg/tests/steps/iceberg_engine.py", line 337, in read_data_from_clickhouse_iceberg_table
    result = node.query(
             ^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py", line 1163, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 552. DB::Exception: Unrecognized option '--object_storage_cluster'. (UNRECOGNIZED_ARGUMENTS)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/cluster.py', line 1163 in 'query'

1155\|                  assert message in r.output, error(r.output)
1156\|  
1157\|          if not ignore_exception:
1158\|              if message is None or "Exception:" not in message:
1159\|                  with Then("check if output has exception") if steps else NullStep():
1160\|                      if "Exception:" in r.output:
1161\|                          if raise_on_exception:
1162\|                              raise QueryRuntimeException(r.output)
1163\|>                         assert False, error(r.output)
1164\|  
1165\|          return r
1166\|

Results

Test Name Result Duration
/swarms Fail 30m 41s
/swarms/feature Fail 26m 51s
/swarms/feature/swarm sanity Fail 3m 14s
/swarms/feature/swarm sanity/create swarm cluster OK 46s 134ms
/swarms/feature/swarm sanity/cluster with only one observer node Fail 20s 199ms
/swarms/feature/swarm sanity/check scale up and down Fail 34s 632ms
/swarms/feature/swarm sanity/swarm examples Fail 59s 635ms
/swarms/feature/swarm sanity/cluster with one observer node and one swarm node Fail 33s 773ms
/swarms/feature/invalid configuration OK 36s 54ms
/swarms/feature/invalid configuration/wrong key OK 36s 53ms
/swarms/feature/cluster discovery Fail 2m 47s
/swarms/feature/cluster discovery/cluster registration deregistration OK 35s 598ms
/swarms/feature/cluster discovery/multiple discovery sections Fail 50s 12ms
/swarms/feature/cluster discovery/multiple paths Fail 48s 207ms
/swarms/feature/cluster discovery/check cluster discovery with wrong cluster name Fail 33s 598ms
/swarms/feature/node failure Fail 3m 48s
/swarms/feature/node failure/check restart swarm node Fail 3s 186ms
/swarms/feature/node failure/check restart clickhouse on swarm node Fail 18s 969ms
/swarms/feature/node failure/network failure Fail 4s 804ms
/swarms/feature/node failure/swarm out of disk space Fail 37s 786ms
/swarms/feature/node failure/initiator out of disk space Fail 36s 132ms
/swarms/feature/node failure/cpu overload Fail 1m 3s
/swarms/feature/node failure/cpu overload all swarm nodes Fail 1m 3s
/swarms/feature/swarm joins Fail 11m 2s
/swarms/feature/swarm joins/join clause Fail 175ms
/swarms/feature/swarm join sanity Fail 350ms
/swarms/feature/swarm join sanity/join subquery Fail 174ms
/swarms/feature/swarm join sanity/join with clause Fail 175ms
/swarms/feature/swarm union Fail 5m 3s
/swarms/feature/feature Fail 18s 400ms
/swarms/feature/feature/object storage cluster profile events with overloaded node Fail 18s 399ms

Generated by TestFlows Open-Source Test Framework v2.0.250110.1002922