Science Data Challenge 2 scoring service CLI¶
This package contains two simple command line interface (CLI) applications for interaction with the SDC2 online scoring service:
sdc2-score
: A tool for upload and check the status of submissions.sdc2-score-admin
: A tool managing SDC2 users and groups.
Installation¶
This package can be installed with:
pip install ska-sdc2-scoring-utils
Using the sdc2-score
command¶
Create a submission and generate its SUBMISSION_ID using
sdc2-score create CATALOGUE_FILE
Retrieve information about the submission using
sdc2-score info SUBMISSION_ID
All submissions belonging to a team can be printed using
sdc2-score ls
By default, all commands are run against the ‘full’ dataset. The -c flag can be used to run commands against the ‘dev’ dataset.
Credentials:¶
Credentials can either be specified using the command line options as follows:
sdc2-score create -u USER -p PASSWORD CATALOGUE_FILE
or are loaded from the following environment variables:
SDC2_SCORER_USER
SDC2_SCORER_PASSWORD
If both are supplied, the command line option flags are used.
Summary usage:
Usage:
sdc2-score leaderboard [--sub-info] [--score-info] [--raw] [-c VERSION]
sdc2-score create CATALOGUE_FILE [-u USERNAME] [-p PASSWORD] [-c VERSION] [--hidden]
sdc2-score info SUBMISSION_ID [-c VERSION] [--raw]
sdc2-score ls [-u USERNAME] [-p PASSWORD] [-n LIMIT] [--score-info] [--raw] [-c VERSION]
sdc2-score -h | --help | --version
Options:
--sub-info Display additional submission information.
--score-info Display additional score information.
--raw Display the raw response from the SDC scoring service.
--hidden If passed, submission will be hidden from leaderboards
-c VERSION Set the catalogue version (1.dev, 1.ldev, 1.full) [default: 1.full].
-n LIMIT Maximum number of submissions to list [default: 20].
-u USER Scoring service username.
-p PASSWORD Scoring service user password.
-h, --help Show this screen.
--version Show the version.
Arguments
CATALOGUE_FILE Path to catalogue file to submit.
SUBMISSION_ID Submission ID, minimally matched.
Using the sdc2-score-admin
command¶
The admin client is intended for SDC2 challenge administrators and requires that the following environment variables are set:
SDC2_SCORER_ADMIN_USER
SDC2_SCORER_ADMIN_PASSWORD
Once these are set, the basic usage settings are as follows:
Usage:
sdc2-score-admin group ls [-u] [--raw]
sdc2-score-admin group add GROUP_NAME...
sdc2-score-admin group rm GROUP_ID...
sdc2-score-admin user ls [--raw]
sdc2-score-admin user add FIRST_NAME LAST_NAME USERNAME EMAIL GROUP [PASSWORD]
sdc2-score-admin user rm USER_ID...
sdc2-score-admin submission ls [-n LIMIT] [-c VERSION] [--raw]
sdc2-score-admin submission rm SUBMISSION_ID [-c VERSION]
sdc2-score-admin (-h | --help | --version)
Options:
-u Show users in the group.
--raw Display unformatted API response JSON object.
-n LIMIT Maximum number of submissions to show [default: 20].
-c VERSION Catalogue version (1.full, 1.dev) [default: 1.full].
-h, --help Show this screen.
--version Show the version.
Arguments:
EMAIL Email address of the user.
FIRST_NAME User's first name.
GROUP_ID Group name or ID, minimally matched.
GROUP_NAME Name of the group.
LAST_NAME User's last name.
PASSWORD Password for the user. Optional, autogenerated if not set.
SUBMISSION_ID UUID for a submission, minimally matched
USER_ID User name or ID, minimally matched.
USERNAME User's user name
Developer notes¶
For local development, commands can also be run and tested using:
python -m ska_sdc2_scoring_utils.sdc2_score
python -m ska_sdc2_scoring_utils.sdc2_score_admin