Swarms Test Run Report

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

FULL OUTER JOIN is not supported in allow mode https://github.com/ClickHouse/ClickHouse/issues/89996

AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 139, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 180, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../helpers/argparser.py", line 451, in capture_minio_args
    return func(self, minio_args=minio_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/regression.py", line 132, in regression
    Feature(test=load("swarms.tests.feature", "feature"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/feature.py", line 32, in feature
    Feature(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/task_rescheduling.py", line 181, in feature
    Scenario(test=rescheduling_with_bucket_granularity)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/task_rescheduling.py", line 168, in rescheduling_with_bucket_granularity
    assert sent_to_matched + sent_to_non_matched > 1, error(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  @TestScenario
  def rescheduling_with_bucket_granularity(
      self,
      minio_root_user,
      minio_root_[masked]:Secret(name='minio_root_password'),
      cluster_name="static_swarm_cluster",
      node=None,
  ):
      """Check bucket-split task rescheduling after replica kill.

      This is the core regression scenario for #1486: one file is split into many bucket
      tasks. If rescheduling uses getPath() instead of getIdentifier(), tasks can collide
      and rows can be lost after replica failure.
      """
      if node is None:
          node = self.context.node

      expected_total = 200000
      row_group_size = 1000
      log_comment = f"bucket_reschedule_{getuid()}"

      with Given("create one parquet file with many row groups"):
          parquet_url = s3_steps.create_parquet_with_many_row_groups(
              node=node,
              s3_access_key_id=minio_root_user,
              s3_secret_access_key=minio_root_[masked]:Secret(name='minio_root_password'),
              rows=expected_total,
              row_group_size=row_group_size,
          )

      table_expression = (
          f"s3('{parquet_url}', '{minio_root_user}', '{minio_root_[masked]:Secret(name='minio_root_password')}', 'Parquet')"
      )

      with Then(
          "run select with bucket granularity and kill clickhouse on a swarm node"
      ):
          with Pool() as pool:
              Step(
                  "run long query",
                  test=run_long_query,
                  parallel=True,
                  executor=pool,
              )(
                  node=node,
                  clickhouse_iceberg_table_name=table_expression,
                  cluster_name=cluster_name,
                  sleep_each_row=0.00005,
                  max_threads=1,
                  lock_object_storage_task_distribution_ms=2,
                  cluster_table_function_split_granularity="bucket",
                  # Force one bucket per distributed task (smallest possible batch)
                  # so the pre-fix bucket-identity collision in
                  # rescheduleTasksFromReplica is reliably exercised when a replica
                  # is killed mid-query (see Altinity/ClickHouse#1486).
                  cluster_table_function_buckets_batch_size=1,
                  log_comment=log_comment,
                  expected_total=expected_total,
              )
              Step(
                  "kill clickhouse on random swarm node",
                  test=actions.restart_clickhouse_on_random_swarm_node,
                  parallel=True,
                  executor=pool,
              )(delay=20, signal="KILL", delay_before_execution=2)
              join()

      with And("verify bucket split was active in query profile events"):
          for log_node in (self.context.node, self.context.node2, self.context.node3):
              log_node.query("SYSTEM FLUSH LOGS")

          metrics = node.query(
              f"""
                  SELECT
                      sumIf(ProfileEvents['ObjectStorageClusterSentToMatchedReplica'], is_initial_query = 1),
                      sumIf(ProfileEvents['ObjectStorageClusterSentToNonMatchedReplica'], is_initial_query = 1)
                  FROM system.query_log
                  WHERE log_comment = '{log_comment}'
                    AND type = 'QueryFinish'
                  FORMAT TabSeparated
              """
          ).output.strip()

          sent_to_matched, sent_to_non_matched = [
              int(v) for v in metrics.split("\t")
          ]

          processed_tasks_total = 0
          for replica_node in (self.context.node2, self.context.node3):
              processed_tasks = replica_node.query(
                  f"""
                      SELECT sum(ProfileEvents['ObjectStorageClusterProcessedTasks'])
                      FROM system.query_log
                      WHERE log_comment = '{log_comment}'
                        AND type = 'QueryFinish'
                        AND is_initial_query = 0
                      FORMAT TabSeparated
                  """
              ).output.strip()
              processed_tasks_total += int(processed_tasks or "0")

          # With one source file, values > 1 indicate bucket-level splitting.
          assert sent_to_matched + sent_to_non_matched > 1, error(

Description
  Bucket split was not active: expected more than one distributed task.

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/swarms/../swarms/tests/task_rescheduling.py', line 168 in 'rescheduling_with_bucket_granularity'

160\|                        AND type = 'QueryFinish'
161\|                        AND is_initial_query = 0
162\|                      FORMAT TabSeparated
163\|                  """
164\|              ).output.strip()
165\|              processed_tasks_total += int(processed_tasks or "0")
166\|  
167\|          # With one source file, values > 1 indicate bucket-level splitting.
168\|>         assert sent_to_matched + sent_to_non_matched > 1, error(
169\|              "Bucket split was not active: expected more than one distributed task."
170\|          )
171\|          assert processed_tasks_total > 1, error(