sqltrack package#
Module contents#
- class sqltrack.Client(config_path: str = None, **kwargs)[source]#
Bases:
object
Creates and manages
psycopg.Connection
objects when used as a context manager:client = Client(...) with client.connect() as conn with conn.cursor() as cursor: ...
Alternatively, if you don’t need to use the connection directly, you can also get a cursor:
client = Client(...) with client.cursor() as cursor: ...
Connection parameters are given as
kwargs
. Common options areuser
,dbname
,host
, andschema
(a shorthand for setting thesearch_path
option). For the full list of available parameters, see https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDSParameters passed from Python take priority, but they may also be passed as environment variables
SQLTRACK_DSN_<PARAM>
(e.g.,SQLTRACK_DSN_USER
), or loaded from a config file, by default./sqltrack.conf
.Experiment and Run objects obtain connections as required. Nested contexts reuse the same connection (reentrant), so they can be used to avoid connecting to the database multiple times over a short period. E.g., the following snippet will connect only once, with the caveat that everything happens within the same transaction:
def do_queries(client, ...): with client.cursor() as cursor: cursor.execute(...) ... client = Client(...) with client.connect(): do_queries(client, ...) do_queries(client, ...) do_queries(client, ...)
- Parameters:
config_path – Path to config file, defaults to
SQLTRACK_CONFIG_PATH
environment variable, and finally./sqltrack.conf
kwargs – Connection parameters
- commit()[source]#
Convenience function to call commit on the DB connection. Raises
RuntimeError
when not connected.
- connect() Connection [source]#
Context manager that connects to the DB. Use in with statement:
with client.connect() as conn: ... connection things ... with client.cursor() as cursor: ... cursor things ... ... connection things ...
The connection is closed and any changes comitted when the with block ends.
Nested contexts reuse the same connection (reentrant), so they can be used to avoid connecting to the database multiple times over a short period. E.g., the following snippet will connect only once, with the caveat that everything happens within the same transaction:
def do_queries(client, ...): with client.cursor() as cursor: cursor.execute(...) ... client = Client(...) with client.connect(): do_queries(client, ...) do_queries(client, ...) do_queries(client, ...)
- cursor() Cursor [source]#
Connect to the DB and return a cursor. Use in with statement:
with client.cursor() as cursor: ... cursor things ...
The connection is closed and any changes comitted when the with block ends.
- rollback()[source]#
Convenience function to call rollback on the DB connection. Raises
RuntimeError
when not connected.
- class sqltrack.Experiment(client: Client, experiment_id: int | None = None, name: str | None = None, comment: str | None = None, tags: Iterable[str] | None = None)[source]#
Bases:
object
Helper class to create experiments, as well as runs for experiments.
Note
Parameters comment and tags are ignored if the experiment already exists.
- Parameters:
client – Client object to use
experiment_id – ID of the experiment; may be None if name is given
name – name of the experiment; may be None if experiment_id is given
comment – text comment; See
experiment_set_comment()
for detailstags – list of experiment tags; See
experiment_set_tags()
for details
- get_run(run_id: int | str | None = None, status: str | None = None, comment: str | None = None, tags: Iterable[str] | None = None, args: 'auto' | dict | None = None, env: 'auto' | dict | None = None, updated: 'auto' | datetime | None = None) Run [source]#
Get a Run object, see its documentation for more details.
- class sqltrack.Run(client: Client, experiment_id: int | None = None, run_id: int | str | None = None, status: str | None = None, comment: str | None = None, tags: Iterable[str] | None = None, args: 'auto' | dict | None = None, env: 'auto' | dict | None = None, updated: 'auto' | datetime | None = None)[source]#
Bases:
object
Helper class to manage runs.
Note
If
run_id
is None, a new run with an unused ID is created.- Parameters:
client – Client object to use
experiment_id – ID of the experiment; may be None if an existing run_id is given
run_id – ID of the run; if None an unused ID is chosen; if string then load ID from that environment variable
comment – just some text comment; See
run_set_comment()
for detailstags – list of run tags; See
run_set_tags()
for detailsargs – run parameters; See
run_set_args()
for detailsenv – environment variables; See
run_set_env()
for detailsupdated – update timestamp; See
run_set_updated()
for details
- add_args(args: 'auto' | dict | None = 'auto')[source]#
Add parameters to the run. See
run_add_args()
for details.
- add_env(env: 'auto' | dict | None = 'auto')[source]#
Add environment variables to the run. See
run_add_env()
for details.
- add_metrics(step: int = 0, progress: float = 0.0, set_updated: 'auto' | datetime | None = 'auto', **metrics)[source]#
Add metrics to the run. See
run_add_metrics()
for details.
- remove_args(args: 'auto' | dict | None = 'auto')[source]#
Remove parameters from the run. See
run_remove_args()
for details.
- remove_env(env: 'auto' | dict | None = 'auto')[source]#
Remove environment variables from the run. See
run_remove_env()
for details.
- set_args(args: 'auto' | dict | None = 'auto')[source]#
Set the run parameters. See
run_set_args()
for details.
- set_created(dt: 'auto' | datetime | None = 'auto')[source]#
Set time_created for the run. See
run_set_created()
for details.
- set_env(env: 'auto' | dict | None = 'auto')[source]#
Set the run environment. See
run_set_env()
for details.
- set_started(dt: 'auto' | datetime | None = 'auto')[source]#
Set time_started for the run. See
run_set_started()
for details.
- set_status(status: str)[source]#
Set the run status. See
run_set_status()
for details.
- set_updated(dt: 'auto' | datetime | None = 'auto')[source]#
Set time_updated for the run. See
run_set_updated()
for details.
- start(terminated='CANCELLED', started: 'auto' | datetime | None = 'auto', updated: 'auto' | datetime | None = 'auto', args: 'auto' | dict | None = None, env: 'auto' | dict | None = None)[source]#
Start the run, setting its status to RUNNING, among other values like
time_started
, depending on parameters.- Parameters:
terminated – status in case SIGTERM is received during the run; see also
sqltrack.sigterm
started – what to do about time_started; See
run_set_started()
for detailsupdated – what to do about time_updated; See
run_set_updated()
for detailsenv – control what to do about the run’s env; See
run_set_env()
for details
- stop(status='COMPLETED', updated: 'auto' | datetime | None = 'auto')[source]#
Stop the run, setting its status to COMPLETED and
time_started
to now (default), depending on parameters.- Parameters:
status – status to set (default: COMPLETED); See
run_set_status()
for detailsupdated – what to do about time_updated; See
run_set_updated()
for details
- track(normal='COMPLETED', exception='FAILED', interrupt='CANCELLED', terminated='CANCELLED', started: 'auto' | datetime | None = 'auto', updated: 'auto' | datetime | None = 'auto', args: 'auto' | dict | None = None, env: 'auto' | dict | None = None)[source]#
A context manager to track the execution of the run. This is equivalent to calling start and stop separately with the appropriate status value.
- Parameters:
normal – status in case the run completes normally
exception – status in case an exception occurs
interrupt – status in case SIGINT is received during the run
terminated – status in case SIGTERM is received during the run; see also
sqltrack.sigterm
started – what to do about time_started; See
run_set_started()
for detailsupdated – what to do about time_updated; See
run_set_started()
for detailsenv – control what to do about the run’s env; See
run_set_env()
for details
- sqltrack.experiment_add_relationship(client: Client, from_id: int, kind: str, to_id: int)[source]#
Add a relationship between two experiments.
- sqltrack.experiment_add_tags(client: Client, experiment_id: int, *tags: str)[source]#
Add tags to an experiment.
- sqltrack.experiment_remove_relationship(client: Client, from_id: int, kind: str, to_id: int)[source]#
Remove a relationship between two experiments.
- sqltrack.experiment_remove_tags(client: Client, experiment_id: int, *tags: str)[source]#
Remove tags from an experiment.
- sqltrack.experiment_set_comment(client: Client, experiment_id: int, comment: str)[source]#
Set an experiment comment.
- sqltrack.experiment_set_name(client: Client, experiment_id: int, name: str | None)[source]#
Set an experiment name.
- sqltrack.experiment_set_tags(client: Client, experiment_id: int, *tags: str)[source]#
Set tags of an experiment.
- sqltrack.run_add_args(client: Client, run_id: int, args: 'auto' | dict | None = 'auto')[source]#
Add some parameters to an existing run. See
sqltrack.args.detect_args()
for details on detection in auto mode.- Parameters:
client – the Client to use
run_id – which existing run to update
args – the parameters, may be
'auto'
(detect parameters),dict
, or None (do nothing)
- sqltrack.run_add_env(client: Client, run_id: int, env: 'auto' | dict | None = 'auto')[source]#
Add some environment variables to an existing run.
- Parameters:
client – the Client to use
run_id – which existing run to update
env – the environment, may be
'auto'
(set toos.environ
),dict
, or None (do nothing)
- sqltrack.run_add_metrics(client: Client, run_id: int, step: int = 0, progress: float = 0.0, **metrics)[source]#
Add metrics to a run.
- sqltrack.run_add_relationship(client: Client, from_id: int, kind: str, to_id: int)[source]#
Add a relationship between two runs.
- sqltrack.run_add_tags(client: Client, run_id: int, *tags: str)[source]#
Add tags to an existing run.
- Parameters:
client – the Client to use
run_id – which existing run to update
tags – tags to add
- sqltrack.run_from_env(client: Client) Run [source]#
Get the Run object defined by environment variables. The experiment is defined by at least one or both of
SQLTRACK_EXPERIMENT_NAME
andSQLTRACK_EXPERIMENT_ID
, and optionallySQLTRACK_RUN_ID
.
- sqltrack.run_remove_args(client: Client, run_id: int, *args: str)[source]#
Remove some parameters from an existing run.
- Parameters:
client – the Client to use
run_id – which existing run to update
args – names to remove from parameters
- sqltrack.run_remove_env(client: Client, run_id: int, *env: str)[source]#
Remove some environment variables from an existing run.
- Parameters:
client – the Client to use
run_id – which existing run to update
env – names to remove from env
- sqltrack.run_remove_relationship(client: Client, from_id: int, kind: str, to_id: int)[source]#
Remove a relationship between two runs.
- sqltrack.run_remove_tags(client: Client, run_id: int, *tags: str)[source]#
Remove tags from an existing run.
- Parameters:
client – the Client to use
run_id – which existing run to update
tags – tags to remove
- sqltrack.run_set_args(client: Client, run_id: int, args: 'auto' | dict | None = 'auto')[source]#
Set run parameters. See
sqltrack.args.detect_args()
for details on detection in auto mode.- Parameters:
client – the Client to use
run_id – which existing run to update
args – the parameters, may be
'auto'
(detect parameters),dict
, or None (do nothing)
- sqltrack.run_set_comment(client: Client, run_id: int, comment: str | None)[source]#
Set the comment for an existing run. If comment is None, do nothing.
- sqltrack.run_set_created(client: Client, run_id: int, dt: 'auto' | datetime | None = 'auto')[source]#
Set time_created for an existing run.
- Parameters:
client – the Client to use
run_id – which existing run to update
dt – the timestamp, may be
'auto'
(set to now), timezone awaredatetime
, or None (do nothing)
- sqltrack.run_set_env(client: Client, run_id: int, env: 'auto' | dict | None = 'auto')[source]#
Set run environment.
- Parameters:
client – the Client to use
run_id – which existing run to update
env – the environment, may be
'auto'
(set toos.environ
),dict
, or None (do nothing)
- sqltrack.run_set_started(client: Client, run_id: int, dt: 'auto' | datetime | None = 'auto')[source]#
Set time_started for an existing run.
- Parameters:
client – the Client to use
run_id – which existing run to update
dt – the timestamp, may be
'auto'
(set to now), timezone awaredatetime
, or None (do nothing)
- sqltrack.run_set_status(client: Client, run_id: int, status: str | None)[source]#
- Set a run status to one of the following:
BOOT_FAIL
CANCELLED
CONFIGURING
COMPLETED
COMPLETING
DEADLINE
FAILED
NODE_FAIL
OUT_OF_MEMORY
PENDING
PREEMPTED
RESV_DEL_HOLD
REQUEUE_FED
REQUEUE_HOLD
REQUEUED
RESIZING
REVOKED
RUNNING
SIGNALING
SPECIAL_EXIT
STAGE_OUT
STOPPED
SUSPENDED
TIMEOUT
Does nothing if status is None.
- sqltrack.run_set_tags(client: Client, run_id: int, *tags: str)[source]#
Set the tags of an existing run.
- Parameters:
client – the Client to use
run_id – which existing run to update
tags – the tags
- sqltrack.run_set_updated(client: Client, run_id: int, dt: 'auto' | datetime | None = 'auto')[source]#
Set time_updated for an existing run.
- Parameters:
client – the Client to use
run_id – which existing run to update
dt – the timestamp, may be
'auto'
(set to now), timezone awaredatetime
, or None (do nothing)