sqltrack package#

Module contents#

class sqltrack.Client(config_path: str = None, **kwargs)[source]#

Bases: object

Creates and manages psycopg.Connection objects when used as a context manager:

client = Client(...)
with client.connect() as conn
    with conn.cursor() as cursor:
        ...

Alternatively, if you don’t need to use the connection directly, you can also get a cursor:

client = Client(...)
with client.cursor() as cursor:
    ...

Connection parameters are given as kwargs. Common options are user, dbname, host, and schema (a shorthand for setting the search_path option). For the full list of available parameters, see https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS

Parameters passed from Python take priority, but they may also be passed as environment variables SQLTRACK_DSN_<PARAM> (e.g., SQLTRACK_DSN_USER), or loaded from a config file, by default ./sqltrack.conf.

Experiment and Run objects obtain connections as required. Nested contexts reuse the same connection (reentrant), so they can be used to avoid connecting to the database multiple times over a short period. E.g., the following snippet will connect only once, with the caveat that everything happens within the same transaction:

def do_queries(client, ...):
    with client.cursor() as cursor:
        cursor.execute(...)
        ...

client = Client(...)
with client.connect():
    do_queries(client, ...)
    do_queries(client, ...)
    do_queries(client, ...)

Parameters:

config_path – Path to config file, defaults to SQLTRACK_CONFIG_PATH environment variable, and finally ./sqltrack.conf
kwargs – Connection parameters

commit()[source]#: Convenience function to call commit on the DB connection. Raises RuntimeError when not connected.

connect() → Connection[source]#

Context manager that connects to the DB. Use in with statement:

with client.connect() as conn:
    ... connection things ...
    with client.cursor() as cursor:
        ... cursor things ...
    ... connection things ...

The connection is closed and any changes comitted when the with block ends.

Nested contexts reuse the same connection (reentrant), so they can be used to avoid connecting to the database multiple times over a short period. E.g., the following snippet will connect only once, with the caveat that everything happens within the same transaction:

def do_queries(client, ...):
    with client.cursor() as cursor:
        cursor.execute(...)
        ...

client = Client(...)
with client.connect():
    do_queries(client, ...)
    do_queries(client, ...)
    do_queries(client, ...)

cursor() → Cursor[source]#

Connect to the DB and return a cursor. Use in with statement:

with client.cursor() as cursor:
    ... cursor things ...

The connection is closed and any changes comitted when the with block ends.

rollback()[source]#: Convenience function to call rollback on the DB connection. Raises RuntimeError when not connected.

class sqltrack.Experiment(client: Client, experiment_id: int | None = None, name: str | None = None, comment: str | None = None, tags: Iterable[str] | None = None)[source]#

Bases: object

Helper class to create experiments, as well as runs for experiments.

Note

Parameters comment and tags are ignored if the experiment already exists.

Parameters:

client – Client object to use
experiment_id – ID of the experiment; may be None if name is given
name – name of the experiment; may be None if experiment_id is given
comment – text comment; See experiment_set_comment() for details
tags – list of experiment tags; See experiment_set_tags() for details

add_link(kind: str, to_id: int)[source]#: Add a link to another experiment.

add_tags(*tags: str)[source]#: Add tags to the experiment.

get_run(run_id: int | str | None = None, status: str | None = None, comment: str | None = None, tags: Iterable[str] | None = None, args: 'auto' | dict | None = None, env: 'auto' | dict | None = None, updated: 'auto' | datetime | None = None) → Run[source]#: Get a Run object, see its documentation for more details.

remove_link(kind: str, to_id: int)[source]#: Remove a link to another experiment.

remove_tags(*tags: str)[source]#: Remove tags from the experiment.

set_comment(comment: str)[source]#: Set the experiment comment.

set_name(name: str)[source]#: Set the experiment name.

set_tags(*tags: str)[source]#: Set tags of the experiment.

Bases: object

Helper class to manage runs.

Note

If run_id is None, a new run with an unused ID is created.

Parameters:

client – Client object to use
experiment_id – ID of the experiment; may be None if an existing run_id is given
run_id – ID of the run; if None an unused ID is chosen; if string then load ID from that environment variable
comment – just some text comment; See run_set_comment() for details
tags – list of run tags; See run_set_tags() for details
args – run parameters; See run_set_args() for details
env – environment variables; See run_set_env() for details
updated – update timestamp; See run_set_updated() for details

add_args(args: 'auto' | dict | None = 'auto')[source]#: Add parameters to the run. See run_add_args() for details.

add_env(env: 'auto' | dict | None = 'auto')[source]#: Add environment variables to the run. See run_add_env() for details.

add_link(kind: str, to_id: int)[source]#: Add a link to another run.

add_metrics(step: int = 0, progress: float = 0.0, set_updated: 'auto' | datetime | None = 'auto', **metrics)[source]#: Add metrics to the run. See run_add_metrics() for details.

add_tags(*tags: str)[source]#: Add tags to the run.

remove_args(args: 'auto' | dict | None = 'auto')[source]#: Remove parameters from the run. See run_remove_args() for details.

remove_env(env: 'auto' | dict | None = 'auto')[source]#: Remove environment variables from the run. See run_remove_env() for details.

remove_link(kind: str, to_id: int)[source]#: Remove a link to another run.

remove_tags(*tags: str)[source]#: Remove tags from the run.

set_args(args: 'auto' | dict | None = 'auto')[source]#: Set the run parameters. See run_set_args() for details.

set_comment(comment: str)[source]#: Set the run comment.

set_created(dt: 'auto' | datetime | None = 'auto')[source]#: Set time_created for the run. See run_set_created() for details.

set_env(env: 'auto' | dict | None = 'auto')[source]#: Set the run environment. See run_set_env() for details.

set_started(dt: 'auto' | datetime | None = 'auto')[source]#: Set time_started for the run. See run_set_started() for details.

set_status(status: str)[source]#: Set the run status. See run_set_status() for details.

set_tags(*tags: str)[source]#: Set the run tags.

set_updated(dt: 'auto' | datetime | None = 'auto')[source]#: Set time_updated for the run. See run_set_updated() for details.

Start the run, setting its status to RUNNING, among other values like time_started, depending on parameters.

Parameters:

terminated – status in case SIGTERM is received during the run; see also sqltrack.sigterm
started – what to do about time_started; See run_set_started() for details
updated – what to do about time_updated; See run_set_updated() for details
env – control what to do about the run’s env; See run_set_env() for details

stop(status='COMPLETED', updated: 'auto' | datetime | None = 'auto')[source]#

Stop the run, setting its status to COMPLETED and time_started to now (default), depending on parameters.

Parameters:

status – status to set (default: COMPLETED); See run_set_status() for details
updated – what to do about time_updated; See run_set_updated() for details

A context manager to track the execution of the run. This is equivalent to calling start and stop separately with the appropriate status value.

Parameters:

normal – status in case the run completes normally
exception – status in case an exception occurs
interrupt – status in case SIGINT is received during the run
terminated – status in case SIGTERM is received during the run; see also sqltrack.sigterm
started – what to do about time_started; See run_set_started() for details
updated – what to do about time_updated; See run_set_started() for details
env – control what to do about the run’s env; See run_set_env() for details

sqltrack.experiment_add_link(client: Client, from_id: int, kind: str, to_id: int)[source]#: Add a link between two experiments.

sqltrack.experiment_add_tags(client: Client, experiment_id: int, *tags: str)[source]#: Add tags to an experiment.

sqltrack.experiment_remove_link(client: Client, from_id: int, kind: str, to_id: int)[source]#: Remove a link between two experiments.

sqltrack.experiment_remove_tags(client: Client, experiment_id: int, *tags: str)[source]#: Remove tags from an experiment.

sqltrack.experiment_set_comment(client: Client, experiment_id: int, comment: str)[source]#: Set an experiment comment.

sqltrack.experiment_set_name(client: Client, experiment_id: int, name: str | None)[source]#: Set an experiment name.

sqltrack.experiment_set_tags(client: Client, experiment_id: int, *tags: str)[source]#: Set tags of an experiment.

sqltrack.run_add_args(client: Client, run_id: int, args: 'auto' | dict | None = 'auto')[source]#

Add some parameters to an existing run. See sqltrack.args.detect_args() for details on detection in auto mode.

Parameters:

client – the Client to use
run_id – which existing run to update
args – the parameters, may be 'auto' (detect parameters), dict, or None (do nothing)

sqltrack.run_add_env(client: Client, run_id: int, env: 'auto' | dict | None = 'auto')[source]#

Add some environment variables to an existing run.

Parameters:

client – the Client to use
run_id – which existing run to update
env – the environment, may be 'auto' (set to os.environ), dict, or None (do nothing)

sqltrack.run_add_link(client: Client, from_id: int, kind: str, to_id: int)[source]#: Add a link between two runs.

sqltrack.run_add_metrics(client: Client, run_id: int, step: int = 0, progress: float = 0.0, **metrics)[source]#: Add metrics to a run.

sqltrack.run_add_tags(client: Client, run_id: int, *tags: str)[source]#

Add tags to an existing run.

Parameters:

client – the Client to use
run_id – which existing run to update
tags – tags to add

sqltrack.run_from_env(client: Client) → Run[source]#: Get the Run object defined by environment variables. The experiment is defined by at least one or both of SQLTRACK_EXPERIMENT_NAME and SQLTRACK_EXPERIMENT_ID, and optionally SQLTRACK_RUN_ID.

sqltrack.run_remove_args(client: Client, run_id: int, *args: str)[source]#

Remove some parameters from an existing run.

Parameters:

client – the Client to use
run_id – which existing run to update
args – names to remove from parameters

sqltrack.run_remove_env(client: Client, run_id: int, *env: str)[source]#

Remove some environment variables from an existing run.

Parameters:

client – the Client to use
run_id – which existing run to update
env – names to remove from env

sqltrack.run_remove_link(client: Client, from_id: int, kind: str, to_id: int)[source]#: Remove a link between two runs.

sqltrack.run_remove_tags(client: Client, run_id: int, *tags: str)[source]#

Remove tags from an existing run.

Parameters:

client – the Client to use
run_id – which existing run to update
tags – tags to remove

sqltrack.run_set_args(client: Client, run_id: int, args: 'auto' | dict | None = 'auto')[source]#

Set run parameters. See sqltrack.args.detect_args() for details on detection in auto mode.

Parameters:

client – the Client to use
run_id – which existing run to update
args – the parameters, may be 'auto' (detect parameters), dict, or None (do nothing)

sqltrack.run_set_comment(client: Client, run_id: int, comment: str | None)[source]#: Set the comment for an existing run. If comment is None, do nothing.

sqltrack.run_set_created(client: Client, run_id: int, dt: 'auto' | datetime | None = 'auto')[source]#

Set time_created for an existing run.

Parameters:

client – the Client to use
run_id – which existing run to update
dt – the timestamp, may be 'auto' (set to now), timezone aware datetime, or None (do nothing)

sqltrack.run_set_env(client: Client, run_id: int, env: 'auto' | dict | None = 'auto')[source]#

Set run environment.

Parameters:

client – the Client to use
run_id – which existing run to update
env – the environment, may be 'auto' (set to os.environ), dict, or None (do nothing)

sqltrack.run_set_started(client: Client, run_id: int, dt: 'auto' | datetime | None = 'auto')[source]#

Set time_started for an existing run.

Parameters:

client – the Client to use
run_id – which existing run to update
dt – the timestamp, may be 'auto' (set to now), timezone aware datetime, or None (do nothing)

sqltrack.run_set_status(client: Client, run_id: int, status: str | None)[source]#

Set a run status to one of the following:

BOOT_FAIL
CANCELLED
CONFIGURING
COMPLETED
COMPLETING
DEADLINE
FAILED
NODE_FAIL
OUT_OF_MEMORY
PENDING
PREEMPTED
RESV_DEL_HOLD
REQUEUE_FED
REQUEUE_HOLD
REQUEUED
RESIZING
REVOKED
RUNNING
SIGNALING
SPECIAL_EXIT
STAGE_OUT
STOPPED
SUSPENDED
TIMEOUT

Does nothing if status is None.

sqltrack.run_set_tags(client: Client, run_id: int, *tags: str)[source]#

Set the tags of an existing run.

Parameters:

client – the Client to use
run_id – which existing run to update
tags – the tags

sqltrack.run_set_updated(client: Client, run_id: int, dt: 'auto' | datetime | None = 'auto')[source]#

Set time_updated for an existing run.

Parameters:

client – the Client to use
run_id – which existing run to update
dt – the timestamp, may be 'auto' (set to now), timezone aware datetime, or None (do nothing)