hipscatalog_gen.cluster package

Submodules

hipscatalog_gen.cluster.runtime module

Dask cluster setup and teardown utilities.

class ClusterRuntime(cluster, client, diagnostics_mode)[source]

Bases: object

Runtime handles for the Dask cluster.

Parameters:

  • cluster (Any)

  • client (Client)

  • diagnostics_mode (str)

cluster: Any
client: Client
diagnostics_mode: str
setup_cluster(cfg, report_dir, log_fn)[source]

Create and configure the Dask cluster and diagnostics context.

Parameters:

  • cfg (ClusterCfg) – Cluster configuration (local or SLURM) with worker counts and memory limits.

  • report_dir (Path) – Directory where per-step diagnostics reports are written.

  • log_fn (Callable[[str, bool], None]) – Logging callback (message, always).

Returns:

A pair (runtime, diag_ctx_factory), where:

  • runtime: ClusterRuntime with cluster/client handles and flags.

  • diag_ctx_factory: callable label -> context manager used as with diag_ctx_factory("step_name"): around pipeline steps.

Return type:

Tuple[ClusterRuntime, Callable[[str], ContextManager[Any]]]

Raises:

ImportError – If mode='slurm' is set but dask-jobqueue is not available.

shutdown_cluster(runtime)[source]

Gracefully close client and cluster.

Parameters:

runtime (ClusterRuntime) – ClusterRuntime with cluster and client handles.

Return type:

None

Module contents

Cluster orchestration helpers for hipscatalog-gen.

class ClusterRuntime(cluster, client, diagnostics_mode)[source]

Bases: object

Runtime handles for the Dask cluster.

Parameters:

  • cluster (Any)

  • client (Client)

  • diagnostics_mode (str)

cluster: Any
client: Client
diagnostics_mode: str
setup_cluster(cfg, report_dir, log_fn)[source]

Create and configure the Dask cluster and diagnostics context.

Parameters:

  • cfg (ClusterCfg) – Cluster configuration (local or SLURM) with worker counts and memory limits.

  • report_dir (Path) – Directory where per-step diagnostics reports are written.

  • log_fn (Callable[[str, bool], None]) – Logging callback (message, always).

Returns:

A pair (runtime, diag_ctx_factory), where:

  • runtime: ClusterRuntime with cluster/client handles and flags.

  • diag_ctx_factory: callable label -> context manager used as with diag_ctx_factory("step_name"): around pipeline steps.

Return type:

Tuple[ClusterRuntime, Callable[[str], ContextManager[Any]]]

Raises:

ImportError – If mode='slurm' is set but dask-jobqueue is not available.

shutdown_cluster(runtime)[source]

Gracefully close client and cluster.

Parameters:

runtime (ClusterRuntime) – ClusterRuntime with cluster and client handles.

Return type:

None