Copyright 2025, Altinity Inc.. All Rights Reserved. All information contained herein is, and remains the property of Altinity Inc.. Any dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Altinity Inc..

Authentication Test Run Report

DateSep 03, 2025 0:53
Duration30m 41s
Framework TestFlows 2.0.250110.1002922

Artifacts

Test artifacts can be found at https://altinity-build-artifacts.s3.amazonaws.com/index.html#998/ad3281d439d6d71588fea15e0147825155625faf/regression/aarch64/with_analyzer/zookeeper/without_thread_fuzzer/ldap/authentication/

Attributes

projectAltinity/ClickHouse
project.id159717931
packagehttps://altinity-build-artifacts.s3.amazonaws.com/PRs/998/ad3281d439d6d71588fea15e0147825155625faf/build_arm_release/clickhouse-common-static_25.6.5.20166.altinityantalya_arm64.deb
version25.6.5.20166.altinityantalya
user.namestrtgbb
repositoryhttps://github.com/Altinity/clickhouse-regression
commit.hash38b4f3c4cbcf7b38c97e16793c210a1496075af7
job.nameLDAP (authentication)
job.retry1
job.urlhttps://github.com/Altinity/ClickHouse/actions/runs/17418173257
archaarch64
localTrue
clickhouse_versionNone
clickhouse_pathhttps://altinity-build-artifacts.s3.amazonaws.com/PRs/998/ad3281d439d6d71588fea15e0147825155625faf/build_arm_release/clickhouse-common-static_25.6.5.20166.altinityantalya_arm64.deb
as_binaryFalse
base_osNone
keeper_pathNone
zookeeper_versionNone
use_keeperFalse
stressFalse
collect_service_logsTrue
thread_fuzzerFalse
with_analyzerTrue
reuse_envFalse

Summary

93.7%OK
6.3%Known

Statistics

Units OK Fail XFail
Features
8
8
Scenarios
93
89
4
Examples
25
21
4
Steps
6337
6321
16

Known Fails

Test NameResultMessage
/authentication/connection protocols/tls cipher suiteXFail 20s 170ms
can't get it to work
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 110, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/argparser.py", line 171, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 102, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 242, in tls_cipher_suite
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 438, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 1195, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user4: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 1195 in 'query'

1187\|                  assert message in r.output, error(r.output)
1188\|  
1189\|          if not ignore_exception:
1190\|              if message is None or "Exception:" not in message:
1191\|                  with Then("check if output has exception") if steps else NullStep():
1192\|                      if "Exception:" in r.output:
1193\|                          if raise_on_exception:
1194\|                              raise QueryRuntimeException(r.output)
1195\|>                         assert False, error(r.output)
1196\|  
1197\|          return r
1198\|
/authentication/connection protocols/tls require cert default demandXFail 20s 182ms
can't be tested with self-signed certificates
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 110, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/argparser.py", line 171, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 102, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 199, in tls_require_cert_default_demand
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 438, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 1195, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user2: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 1195 in 'query'

1187\|                  assert message in r.output, error(r.output)
1188\|  
1189\|          if not ignore_exception:
1190\|              if message is None or "Exception:" not in message:
1191\|                  with Then("check if output has exception") if steps else NullStep():
1192\|                      if "Exception:" in r.output:
1193\|                          if raise_on_exception:
1194\|                              raise QueryRuntimeException(r.output)
1195\|>                         assert False, error(r.output)
1196\|  
1197\|          return r
1198\|
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is providedXFail 20s 433ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 110, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/argparser.py", line 171, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 105, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1228, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1203, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1184, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 1179, in query
    assert r.exitcode == exitcode, error(r.output)
           ^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 1179 in 'query'

1171\|                      **kwargs,
1172\|                  )
1173\|  
1174\|          if no_checks:
1175\|              return r
1176\|  
1177\|          if exitcode is not None:
1178\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
1179\|>                 assert r.exitcode == exitcode, error(r.output)
1180\|  
1181\|          if message is not None:
1182\|              with (
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is providedXFail 20s 378ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 110, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/argparser.py", line 171, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 105, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1228, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1203, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1184, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 1179, in query
    assert r.exitcode == exitcode, error(r.output)
           ^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 1179 in 'query'

1171\|                      **kwargs,
1172\|                  )
1173\|  
1174\|          if no_checks:
1175\|              return r
1176\|  
1177\|          if exitcode is not None:
1178\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
1179\|>                 assert r.exitcode == exitcode, error(r.output)
1180\|  
1181\|          if message is not None:
1182\|              with (

Results

Test Name Result Duration
/authentication OK 30m 41s
/authentication/sanity OK 309ms
/authentication/multiple servers OK 18s 56ms
/authentication/connection protocols OK 6m 42s
/authentication/connection protocols/plain text OK 20s 58ms
/authentication/connection protocols/plain text with custom port OK 20s 19ms
/authentication/connection protocols/starttls OK 1m 20s
/authentication/connection protocols/starttls with custom port OK 20s 92ms
/authentication/connection protocols/tls OK 1m 20s
/authentication/connection protocols/tls cipher suite XFail 20s 170ms
/authentication/connection protocols/tls enable tls default yes OK 20s 162ms
/authentication/connection protocols/tls minimum protocol version OK 1m 40s
/authentication/connection protocols/tls require cert default demand XFail 20s 182ms
/authentication/connection protocols/tls with custom port OK 20s 214ms
/authentication/server config OK 4m 0s
/authentication/server config/auth dn value OK 20s 188ms
/authentication/server config/empty ca cert dir OK 20s 157ms
/authentication/server config/empty ca cert file OK 20s 201ms
/authentication/server config/empty host OK 20s 188ms
/authentication/server config/empty server name OK 2s 80ms
/authentication/server config/invalid auth dn prefix OK 20s 213ms
/authentication/server config/invalid auth dn suffix OK 20s 222ms
/authentication/server config/invalid enable tls value OK 20s 202ms
/authentication/server config/invalid host OK 20s 264ms
/authentication/server config/invalid port OK 20s 237ms
/authentication/server config/invalid tls require cert value OK 20s 287ms
/authentication/server config/invalid verification cooldown value OK 12s 498ms
/authentication/server config/missing host OK 19s 206ms
/authentication/server config/syntax OK 4s 558ms
/authentication/user config OK 1m 18s
/authentication/user config/empty server name OK 5s 531ms
/authentication/user config/empty server not defined OK 20s 217ms
/authentication/user config/empty user name OK 6s 618ms
/authentication/user config/ldap and password OK 5s 592ms
/authentication/user config/multiple servers OK 20s 290ms
/authentication/user config/valid user config OK 20s 236ms
/authentication/user authentications OK 16m 55s
/authentication/user authentications/rbac=False OK 10m 56s
/authentication/user authentications/rbac=False/add user to ldap and login OK 15s 807ms
/authentication/user authentications/rbac=False/empty username and empty password OK 91ms
/authentication/user authentications/rbac=False/invalid long username and valid short password OK 15s 771ms
/authentication/user authentications/rbac=False/invalid username and valid password OK 15s 832ms
/authentication/user authentications/rbac=False/login after clickhouse server is restarted OK 27s 177ms
/authentication/user authentications/rbac=False/login after ldap server is restarted OK 15s 478ms
/authentication/user authentications/rbac=False/login after user cn changed in ldap OK 15s 743ms
/authentication/user authentications/rbac=False/login after user is deleted from ldap OK 15s 696ms
/authentication/user authentications/rbac=False/login after user password changed in ldap OK 15s 761ms
/authentication/user authentications/rbac=False/parallel login OK 26s 68ms
/authentication/user authentications/rbac=False/valid ascii username and utf8 password OK 15s 673ms
/authentication/user authentications/rbac=False/valid long username and short password OK 15s 687ms
/authentication/user authentications/rbac=False/valid short username and invalid long password OK 15s 645ms
/authentication/user authentications/rbac=False/valid short username and long password OK 15s 667ms
/authentication/user authentications/rbac=False/valid username and invalid empty password OK 15s 632ms
/authentication/user authentications/rbac=False/valid username and invalid password OK 15s 653ms
/authentication/user authentications/rbac=False/valid username and password OK 15s 702ms
/authentication/user authentications/rbac=False/valid username and password invalid server OK 15s 641ms
/authentication/user authentications/rbac=False/valid username with valid empty password OK 15s 693ms
/authentication/user authentications/rbac=False/valid utf8 username and ascii password OK 15s 723ms
/authentication/user authentications/rbac=False/verification cooldown OK 5m 32s
/authentication/user authentications/rbac=False/verification cooldown/default verification cooldown value OK 20s 290ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is provided XFail 20s 433ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value cn change OK 20s 319ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value ldap unavailable OK 21s 175ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value password change OK 19s 333ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown performance OK 1m 42s
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 31s 956ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 31s 970ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server host parameter change OK 31s 988ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server port parameter change OK 32s 6ms
/authentication/user authentications/rbac=True OK 5m 59s
/authentication/user authentications/rbac=True/add user to ldap and login OK 257ms
/authentication/user authentications/rbac=True/empty username and empty password OK 72ms
/authentication/user authentications/rbac=True/invalid long username and valid short password OK 227ms
/authentication/user authentications/rbac=True/invalid username and valid password OK 218ms
/authentication/user authentications/rbac=True/login after clickhouse server is restarted OK 11s 596ms
/authentication/user authentications/rbac=True/login after ldap server is restarted OK 1s 114ms
/authentication/user authentications/rbac=True/login after user cn changed in ldap OK 329ms
/authentication/user authentications/rbac=True/login after user is deleted from ldap OK 325ms
/authentication/user authentications/rbac=True/login after user password changed in ldap OK 393ms
/authentication/user authentications/rbac=True/parallel login OK 12s 675ms
/authentication/user authentications/rbac=True/valid ascii username and utf8 password OK 250ms
/authentication/user authentications/rbac=True/valid long username and short password OK 247ms
/authentication/user authentications/rbac=True/valid short username and invalid long password OK 222ms
/authentication/user authentications/rbac=True/valid short username and long password OK 236ms
/authentication/user authentications/rbac=True/valid username and invalid empty password OK 215ms
/authentication/user authentications/rbac=True/valid username and invalid password OK 220ms
/authentication/user authentications/rbac=True/valid username and password OK 245ms
/authentication/user authentications/rbac=True/valid username and password invalid server OK 160ms
/authentication/user authentications/rbac=True/valid username with valid empty password OK 216ms
/authentication/user authentications/rbac=True/valid utf8 username and ascii password OK 245ms
/authentication/user authentications/rbac=True/verification cooldown OK 5m 26s
/authentication/user authentications/rbac=True/verification cooldown/default verification cooldown value OK 19s 198ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is provided XFail 20s 378ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value cn change OK 20s 216ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value ldap unavailable OK 20s 13ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value password change OK 19s 217ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown performance OK 1m 39s
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 31s 781ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 31s 873ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server host parameter change OK 31s 844ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server port parameter change OK 31s 819ms
/authentication/multiple authentications methods OK 6s 51ms
/authentication/multiple authentications methods/ldap with other auth methods OK 1s 545ms

Generated by TestFlows Open-Source Test Framework v2.0.250110.1002922