Download OpenAPI specification:Download
This is a draft of the NMDC Runtime API. The resource layout currently covers aspects of workflow execution and automation, and is intended to facilitate discussion as more of the API is developed.
A site corresponds to a physical place that may participate in job execution.
A site may register data objects and capabilties with NMDC. It may claim jobs to execute, and it may update job operations with execution info.
A site must be able to service requests for any data objects it has registered.
A site may expose a "put object" custom method for authorized users. This method facilitates an operation to upload an object to the site and have the site register that object with the runtime system.
id required | string (Id) |
capability_ids | Array of strings (Capability Ids) Default: [] |
{- "id": "string",
- "capability_ids": [ ]
}
{- "id": "string",
- "capability_ids": [ ],
- "clients": [ ]
}
site_id required | string (Site Id) |
[- "string"
]
[- {
- "name": "string",
- "description": "string",
- "id": "string",
- "created_at": "2019-08-24T14:15:22Z"
}
]
site_id required | string (Site Id) |
aliases | Array of strings (Aliases) |
description | string (Description) |
mime_type | string (Mime Type) ^\w+/[-+.\w]+$ |
name | string (Name) ^[A-Za-z0-9._\-]+$ |
{- "aliases": [
- "string"
], - "description": "string",
- "mime_type": "string",
- "name": "string"
}
{- "id": "string",
- "done": false,
- "expire_time": "2019-08-24T14:15:22Z",
- "result": {
- "aliases": [
- "string"
], - "description": "string",
- "mime_type": "string",
- "name": "string",
- "access_methods": [
- {
- "access_id": "string",
- "access_url": {
- "headers": {
- "property1": "string",
- "property2": "string"
},
}, - "region": "string",
- "type": "s3"
}
], - "checksums": [
- {
- "checksum": "string",
- "type": "string"
}
], - "contents": [
], - "created_time": "2019-08-24T14:15:22Z",
- "size": 0,
- "updated_time": "2019-08-24T14:15:22Z",
- "version": "string"
}, - "metadata": {
- "model": "string",
- "object_id": "string",
- "site_id": "string",
- "expires_in_seconds": 0
}
}
site_id required | string (Site Id) |
access_id | string (Access Id) non-empty |
object (AccessURL) | |
region | string (Region) |
type | string Default: "https" Enum: "s3" "gs" "ftp" "gsiftp" "globus" "htsget" "https" "file" An enumeration. |
{- "access_id": "string",
- "access_url": {
- "headers": {
- "property1": "string",
- "property2": "string"
},
}, - "region": "string",
- "type": "s3"
}
{- "headers": {
- "property1": "string",
- "property2": "string"
},
}
Endpoints for user identification.
Currently, accounts for use of the runtime API are created manually by system administrators.
grant_type | string (Grant Type) ^password$|^client_credentials$ |
username | string (Username) |
password | string (Password) |
scope | string (Scope) Default: "" |
client_id | string (Client Id) |
client_secret | string (Client Secret) |
{- "access_token": "string",
- "token_type": "string",
- "expires": {
- "days": 1,
- "hours": 0,
- "minutes": 0
}
}
{- "username": "string",
- "email": "string",
- "full_name": "string",
- "site_admin": [ ],
- "disabled": true
}
A workflow is a template for creating jobs.
Workflow jobs are typically created by the system via trigger associations between workflows and object types. A workflow may also require certain capabilities of sites in order for those sites to claim workflow jobs.
A workflow may require an executing site to have particular capabilities.
These capabilities go beyond the simple ability to access the data object resources registered with the runtime system. Sites register their capabilities, and sites are only able to claim workflow jobs if they are known to have the capabilities required by the workflow.
An object type is an object annotation that is useful for triggering workflows.
A data object may be annotated with one or more types, which in turn can be associated with workflows through trigger resources.
The data-object type system may be used to trigger workflow jobs on a subset of data objects when a new version of a workflow is deployed. This could be done by minting a special object type for the occasion, annotating the subset of data objects with that type, and registering the association of object type to workflow via a trigger resource.
A trigger is an association between a workflow and a data object type.
When a data object is annotated with a type, perhaps shortly after object registration, the NMDC Runtime will check, via trigger associations, for potential new jobs to create for any workflows.
A job is a resource that isolates workflow configuration from execution.
Rather than directly creating a workflow operation by supplying a workflow ID along with configuration, NMDC creates a job that pairs a workflow with configuration. Then, a site can claim a job ID, allowing the site to execute the intended workflow without additional configuration.
A job can have multiple executions, and a workflow's executions are precisely the executions of all jobs created for that workflow.
A site that already has a compatible job execution result can preempt the unnecessary creation of a job by pre-claiming it. This will return like a claim, and now the site can register known data object inputs for the job without the risk of the runtime system creating a claimable job of the pre-claimed type.
filter | string (Filter) |
max_page_size | integer (Max Page Size) Default: 20 |
page_token | string (Page Token) |
{- "resources": [
- {
- "workflow": {
- "name": "string",
- "description": "string",
- "capability_ids": [
- "string"
], - "id": "string",
- "created_at": "2019-08-24T14:15:22Z"
}, - "name": "string",
- "description": "string",
- "id": "string",
- "created_at": "2019-08-24T14:15:22Z",
- "config": { }
}
], - "next_page_token": "string"
}
{- "workflow": {
- "name": "string",
- "description": "string",
- "capability_ids": [
- "string"
], - "id": "string",
- "created_at": "2019-08-24T14:15:22Z"
}, - "name": "string",
- "description": "string",
- "id": "string",
- "created_at": "2019-08-24T14:15:22Z",
- "config": { }
}
A Data Repository Service (DRS) object represents content necessary for a workflow job to execute, and/or output from a job execution.
An object may be a blob, analogous to a file, or a bundle, analogous to a folder. Sites register objects, and sites must ensure that these objects are accessible to the NMDC data broker.
An object may be associated with one or more object types, useful for triggering workflows.
filter | string (Filter) |
max_page_size | integer (Max Page Size) Default: 20 |
page_token | string (Page Token) |
{- "resources": [
- {
- "aliases": [
- "string"
], - "description": "string",
- "mime_type": "string",
- "name": "string",
- "access_methods": [
- {
- "access_id": "string",
- "access_url": {
- "headers": {
- "property1": "string",
- "property2": "string"
},
}, - "region": "string",
- "type": "s3"
}
], - "checksums": [
- {
- "checksum": "string",
- "type": "string"
}
], - "contents": [
], - "created_time": "2019-08-24T14:15:22Z",
- "size": 0,
- "updated_time": "2019-08-24T14:15:22Z",
- "version": "string",
- "id": "string",
}
], - "next_page_token": "string"
}
Create a new DrsObject.
You may create a blob or a bundle.
A blob is like a file - it's a single blob of bytes, so there there is no contents
array,
only one or more access_methods
.
A bundle is like a folder - it's a gathering of other objects (blobs and/or bundles) in a
contents
array, and access_methods
is optional because a data consumer can fetch each of
the bundle contents individually.
At least one checksum is required. The names of supported checksum types are given by
the set of Python 3.8 hashlib.algorithms_guaranteed
:
blake2b | blake2s | md5 | sha1 | sha224 | sha256 | sha384 | sha3_224 | sha3_256 | sha3_384 | sha3_512 | sha512 | shake_128 | shake_256
Each provided access_method
needs either an access_url
or an access_id
.
aliases | Array of strings (Aliases) |
description | string (Description) |
mime_type | string (Mime Type) ^\w+/[-+.\w]+$ |
name | string (Name) ^[A-Za-z0-9._\-]+$ |
Array of objects (Access Methods) | |
required | Array of objects (Checksums) |
Array of objects (Contents) | |
created_time required | string <date-time> (Created Time) |
size required | integer (Size) >= 0 |
updated_time | string <date-time> (Updated Time) |
version | string (Version) |
{- "aliases": [
- "string"
], - "description": "string",
- "mime_type": "string",
- "name": "string",
- "access_methods": [
- {
- "access_id": "string",
- "access_url": {
- "headers": {
- "property1": "string",
- "property2": "string"
},
}, - "region": "string",
- "type": "s3"
}
], - "checksums": [
- {
- "checksum": "string",
- "type": "string"
}
], - "contents": [
], - "created_time": "2019-08-24T14:15:22Z",
- "size": 0,
- "updated_time": "2019-08-24T14:15:22Z",
- "version": "string"
}
{- "aliases": [
- "string"
], - "description": "string",
- "mime_type": "string",
- "name": "string",
- "access_methods": [
- {
- "access_id": "string",
- "access_url": {
- "headers": {
- "property1": "string",
- "property2": "string"
},
}, - "region": "string",
- "type": "s3"
}
], - "checksums": [
- {
- "checksum": "string",
- "type": "string"
}
], - "contents": [
], - "created_time": "2019-08-24T14:15:22Z",
- "size": 0,
- "updated_time": "2019-08-24T14:15:22Z",
- "version": "string",
- "id": "string",
}
{- "aliases": [
- "string"
], - "description": "string",
- "mime_type": "string",
- "name": "string",
- "access_methods": [
- {
- "access_id": "string",
- "access_url": {
- "headers": {
- "property1": "string",
- "property2": "string"
},
}, - "region": "string",
- "type": "s3"
}
], - "checksums": [
- {
- "checksum": "string",
- "type": "string"
}
], - "contents": [
], - "created_time": "2019-08-24T14:15:22Z",
- "size": 0,
- "updated_time": "2019-08-24T14:15:22Z",
- "version": "string",
- "id": "string",
}
{- "aliases": [
- "string"
], - "description": "string",
- "mime_type": "string",
- "name": "string",
- "access_methods": [
- {
- "access_id": "string",
- "access_url": {
- "headers": {
- "property1": "string",
- "property2": "string"
},
}, - "region": "string",
- "type": "s3"
}
], - "checksums": [
- {
- "checksum": "string",
- "type": "string"
}
], - "contents": [
], - "created_time": "2019-08-24T14:15:22Z",
- "size": 0,
- "updated_time": "2019-08-24T14:15:22Z",
- "version": "string",
- "id": "string",
}
{- "aliases": [
- "string"
], - "description": "string",
- "mime_type": "string",
- "name": "string",
- "access_methods": [
- {
- "access_id": "string",
- "access_url": {
- "headers": {
- "property1": "string",
- "property2": "string"
},
}, - "region": "string",
- "type": "s3"
}
], - "checksums": [
- {
- "checksum": "string",
- "type": "string"
}
], - "contents": [
], - "created_time": "2019-08-24T14:15:22Z",
- "size": 0,
- "updated_time": "2019-08-24T14:15:22Z",
- "version": "string",
- "id": "string",
}
Redirect to /objects/{object_id}.
object_id required | string (Object Id) ^[A-Za-z0-9._~\-]+$ |
{- "aliases": [
- "string"
], - "description": "string",
- "mime_type": "string",
- "name": "string",
- "access_methods": [
- {
- "access_id": "string",
- "access_url": {
- "headers": {
- "property1": "string",
- "property2": "string"
},
}, - "region": "string",
- "type": "s3"
}
], - "checksums": [
- {
- "checksum": "string",
- "type": "string"
}
], - "contents": [
], - "created_time": "2019-08-24T14:15:22Z",
- "size": 0,
- "updated_time": "2019-08-24T14:15:22Z",
- "version": "string",
- "id": "string",
}
object_id required | string (Object Id) |
[- "string"
]
{- "aliases": [
- "string"
], - "description": "string",
- "mime_type": "string",
- "name": "string",
- "access_methods": [
- {
- "access_id": "string",
- "access_url": {
- "headers": {
- "property1": "string",
- "property2": "string"
},
}, - "region": "string",
- "type": "s3"
}
], - "checksums": [
- {
- "checksum": "string",
- "type": "string"
}
], - "contents": [
], - "created_time": "2019-08-24T14:15:22Z",
- "size": 0,
- "updated_time": "2019-08-24T14:15:22Z",
- "version": "string",
- "id": "string",
- "types": [
- "string"
]
}
An operation is a resource for tracking the execution of a job.
When a job is claimed by a site for execution, an operation resource is created.
An operation is akin to a "promise" or "future" in that it should eventually resolve to either a successful result, i.e. an execution resource, or to an error.
An operation is parameterized to return a result type, and a metadata type for storing progress information, that are both particular to the job type.
Operations may be paused, resumed, and/or cancelled.
Operations may expire, i.e. not be stored indefinitely. In this case, it is recommended that execution resources have longer lifetimes / not expire, so that information about successful results of operations are available.
filter | string (Filter) |
max_page_size | integer (Max Page Size) Default: 20 |
page_token | string (Page Token) |
{- "resources": [
- {
- "id": "string",
- "done": false,
- "expire_time": "2019-08-24T14:15:22Z",
- "result": null,
- "metadata": null
}
], - "next_page_token": "string"
}
op_id required | string (Op Id) |
done | boolean (Done) Default: false |
any or OperationError (object) (Result) | |
metadata | any (Metadata) |
{- "done": false,
- "result": null,
- "metadata": null
}
{- "id": "string",
- "done": false,
- "expire_time": "2019-08-24T14:15:22Z",
- "result": null,
- "metadata": null
}
A query is an operation (find, update, etc.) against the metadata store.
Metadata -- for studies, biosamples, omics processing, etc. -- is used by sites to execute jobs, as the parameterization of job executions may depend not only on the content of data objects, but also on objects' associated metadata.
Also, the function of many workflows is to extract or produce new metadata. Such metadata products should be registered as data objects, and they may also be supplied by sites to the runtime system as an update query (if the latter is not done, the runtime system will sense the new metadata and issue an update query).
name | string (Name) |
description | string (Description) |
save | boolean (Save) Default: false |
query | any (Query) |
{- "name": "string",
- "description": "string",
- "save": false,
- "query": null
}
{- "name": "string",
- "description": "string",
- "save": false,
- "query": null,
- "id": "string",
- "created_at": "2019-08-24T14:15:22Z",
- "last_ran": "2019-08-24T14:15:22Z",
- "result": null,
- "error": null
}
{- "name": "string",
- "description": "string",
- "save": false,
- "query": null,
- "id": "string",
- "created_at": "2019-08-24T14:15:22Z",
- "last_ran": "2019-08-24T14:15:22Z",
- "result": null,
- "error": null
}
{- "name": "string",
- "description": "string",
- "save": false,
- "query": null,
- "id": "string",
- "created_at": "2019-08-24T14:15:22Z",
- "last_ran": "2019-08-24T14:15:22Z",
- "result": null,
- "error": null
}
{- "name": "string",
- "description": "string",
- "save": false,
- "query": null,
- "id": "string",
- "created_at": "2019-08-24T14:15:22Z",
- "last_ran": "2019-08-24T14:15:22Z",
- "result": null,
- "error": null
}
Generate one or more identifiers.
Leaving populator
blank will set it to the site ID of the request client.
populator | string (Populator) Default: "" |
naa | string (Naa) Default: "nmdc" Value: "nmdc" |
shoulder | string (Shoulder) >= 2 characters ^(?P<shoulder>(fk0|fk1|fk2|fk3|fk4|fk5|fk6|fk... Default: "fk0" |
number | integer (Number) Default: 1 |
{- "populator": "",
- "naa": "nmdc",
- "shoulder": "fk0",
- "number": 1
}
[- "string"
]
i required | string (I) (?P<naa>(nmdc)):(?P<shoulder>(fk0|fk1|fk2|fk3... |
o | string Default: "set" Enum: "set" "addToSet" "rm" "purge" An enumeration. |
a | string (A) |
v | any (V) |
[- {
- "i": "string",
- "o": "set",
- "a": "string",
- "v": null
}
]
[- { }
]