Kubeflow – Reference

Docs: Component Specification

Mon, 01 Jan 0001 00:00:00 +0000

This specification describes the container component data model for Kubeflow Pipelines. The data model is serialized to a file in YAML format for sharing.

Below are the main parts of the component definition:

Metadata: Name, description, and other metadata.
Interface (inputs and outputs): Name, type, default value.
Implementation: How to run the component, given the input arguments.

Example of a component specification

A component specification takes the form of a YAML file, component.yaml. Below is an example:

name: xgboost4j - Train classifier
description: Trains a boosted tree ensemble classifier using xgboost4j

inputs:
- {name: Training data}
- {name: Rounds, type: Integer, default: '30', help: Number of training rounds}

outputs:
- {name: Trained model, type: XGBoost model, help: Trained XGBoost model}

implementation:
  container:
    image: gcr.io/ml-pipeline/xgboost-classifier-train@sha256:b3a64d57
    command: [
      /ml/train.py,
      --train-set, {inputPath: Training data},
      --rounds,    {inputValue: Rounds},
      --out-model, {outputPath: Trained model},
    ]

See some examples of real-world component specifications.

Detailed specification (ComponentSpec)

This section describes the ComponentSpec.

Metadata

name: Human-readable name of the component.
description: Description of the component.
metadata: Standard object’s metadata:
- annotations: A string key-value map used to add information about the component. Currently, the annotations get translated to Kubernetes annotations when the component task is executed on Kubernetes. Current limitation: the key cannot contain more that one slash ("/"). See more information in the Kubernetes user guide.
- labels: Deprecated. Use annotations.

Interface

inputs and outputs: Specifies the list of inputs/outputs and their properties. Each input or output has the following properties:
- name: Human-readable name of the input/output. Name must be unique inside the inputs or outputs section, but an output may have the same name as an input.
- description: Human-readable description of the input/output.
- default: Specifies the default value for an input. Only valid for inputs.
- type: Specifies the type of input/output. The types are used as hints for pipeline authors and can be used by the pipeline system/UI to validate arguments and connections between components. Basic types are String, Integer, Float, and Bool. See the full list of types defined by the Kubeflow Pipelines SDK.

Implementation

implementation: Specifies how to execute the component instance. There are two implementation types, container and graph. (The latter is not in scope for this document.) In future we may introduce more implementation types like daemon or K8sResource.
- container: Describes the Docker container that implements the component. A portable subset of the Kubernetes Container v1 spec.
  - image: Name of the Docker image.
  - command: Entrypoint array. The Docker image’s ENTRYPOINT is used if this is not provided. Each item is either a string or a placeholder. The most common placeholders are {inputValue: Input name}, {inputPath: Input name} and {outputPath: Output name}.
  - args: Arguments to the entrypoint. The Docker image’s CMD is used if this is not provided. Each item is either a string or a placeholder. The most common placeholders are {inputValue: Input name}, {inputPath: Input name} and {outputPath: Output name}.
  - env: Map of environment variables to set in the container.
  - fileOutputs: Legacy property that is only needed in cases where the container always stores the output data in some hard-coded non-configurable local location. This property specifies a map between some outputs and local file paths where the program writes the output data files. Only needed for components that have hard-coded output paths. Such containers need to be fixed by modifying the program or adding a wrapper script that copies the output to a configurable location. Otherwise the component may be incompatible with future storage systems.

You can set all other Kubernetes container properties when you use the component inside a pipeline.

Using placeholders for command-line arguments

Consuming input by value

The {inputValue: <Input name>} placeholder is replaced by the value of the input argument:

In component.yaml:

command: [program.py, --rounds, {inputValue: Rounds}]

In the pipeline code:
```
task1 = component1(rounds=150)
```
Resulting command-line code (showing the value of the input argument that has replaced the placeholder):
```
program.py --rounds 150
```

Consuming input by file

The {inputPath: <Input name>} placeholder is replaced by the (auto-generated) local file path where the system has put the argument data passed for the “Input name” input.

In component.yaml:

command: [program.py, --train-set, {inputPath: training_data}]

In the pipeline code:

task2 = component1(training_data=some_task1.outputs['some_data'])

Resulting command-line code (the placeholder is replaced by the generated path):
```
program.py --train-set /inputs/train_data/data
```

Producing outputs

The {outputPath: <Output name>} placeholder is replaced by a (generated) local file path where the component program is supposed to write the output data. The parent directories of the path may or may not not exist. Your program must handle both cases without error.

In component.yaml:

command: [program.py, --out-model, {outputPath: trained_model}]

In the pipeline code:

task1 = component1()
# You can now pass `task1.outputs['trained_model']` to other components as argument.

Resulting command-line code (the placeholder is replaced by the generated path):
```
program.py --out-model /outputs/trained_model/data
```

Docs: Pipelines API Reference

Mon, 01 Jan 0001 00:00:00 +0000

Kubeflow Pipelines API

Version: 0.5.1

This file contains REST API specification for Kubeflow Pipelines. The file is autogenerated from the swagger definition.

Default request content-types: application/json

Default response content-types: application/json

Schemes: http, https

Summary

Tag: RunService

Operation	Description
GET /apis/v1beta1/runs	Finds all runs.
POST /apis/v1beta1/runs	Creates a new run.
DELETE /apis/v1beta1/runs/{id}	Deletes a run.
POST /apis/v1beta1/runs/{id}:archive	Archives a run.
POST /apis/v1beta1/runs/{id}:unarchive	Restores an archived run.
GET /apis/v1beta1/runs/{run_id}	Finds a specific run by ID.
GET /apis/v1beta1/runs/{run_id}/nodes/{node_id}/artifacts/{artifact_name}:read	Finds a run's artifact data.
POST /apis/v1beta1/runs/{run_id}/retry	Re-initiates a failed or terminated run.
POST /apis/v1beta1/runs/{run_id}/terminate	Terminates an active run.
POST /apis/v1beta1/runs/{run_id}:reportMetrics	ReportRunMetrics reports metrics of a run. Each metric is reported in its own transaction, so this API accepts partial failures. Metric can be uniquely identified by (run_id, node_id, name). Duplicate reporting will be ignored by the API. First reporting wins.

Tag: JobService

Operation	Description
GET /apis/v1beta1/jobs	Finds all jobs.
POST /apis/v1beta1/jobs	Creates a new job.
GET /apis/v1beta1/jobs/{id}	Finds a specific job by ID.
DELETE /apis/v1beta1/jobs/{id}	Deletes a job.
POST /apis/v1beta1/jobs/{id}/disable	Stops a job and all its associated runs. The job is not deleted.
POST /apis/v1beta1/jobs/{id}/enable	Restarts a job that was previously stopped. All runs associated with the job will continue.

Tag: PipelineService

Operation	Description
GET /apis/v1beta1/pipeline_versions	Lists all pipeline versions of a given pipeline.
POST /apis/v1beta1/pipeline_versions	Adds a pipeline version to the specified pipeline.
GET /apis/v1beta1/pipeline_versions/{version_id}	Gets a pipeline version by pipeline version ID.
DELETE /apis/v1beta1/pipeline_versions/{version_id}	Deletes a pipeline version by pipeline version ID. If the deleted pipeline version is the default pipeline version, the pipeline's default version changes to the pipeline's most recent pipeline version. If there are no remaining pipeline versions, the pipeline will have no default version. Examines the run_service_api.ipynb notebook to learn more about creating a run using a pipeline version (https://github.com/kubeflow/pipelines/blob/master/tools/benchmarks/run_service_api.ipynb).
GET /apis/v1beta1/pipeline_versions/{version_id}/templates	Returns a YAML template that contains the specified pipeline version's description, parameters and metadata.
GET /apis/v1beta1/pipelines	Finds all pipelines.
POST /apis/v1beta1/pipelines	Creates a pipeline.
GET /apis/v1beta1/pipelines/{id}	Finds a specific pipeline by ID.
DELETE /apis/v1beta1/pipelines/{id}	Deletes a pipeline and its pipeline versions.
GET /apis/v1beta1/pipelines/{id}/templates	Returns a single YAML template that contains the description, parameters, and metadata associated with the pipeline provided.

Tag: ExperimentService

Operation	Description
GET /apis/v1beta1/experiments	Finds all experiments. Supports pagination, and sorting on certain fields.
POST /apis/v1beta1/experiments	Creates a new experiment.
GET /apis/v1beta1/experiments/{id}	Finds a specific experiment by ID.
DELETE /apis/v1beta1/experiments/{id}	Deletes an experiment without deleting the experiment's runs and jobs. To avoid unexpected behaviors, delete an experiment's runs and jobs before deleting the experiment.
POST /apis/v1beta1/experiments/{id}:archive	Archives an experiment and the experiment's runs and jobs.
POST /apis/v1beta1/experiments/{id}:unarchive	Restores an archived experiment. The experiment's archived runs and jobs will stay archived.

Tag: PipelineUploadService

Operation	Description
POST /apis/v1beta1/pipelines/upload
POST /apis/v1beta1/pipelines/upload_version

Security

Bearer

Type: apiKey

Name:: authorization
In:: header

Paths

Finds all experiments. Supports pagination, and sorting on certain fields.

GET /apis/v1beta1/experiments

Tags: ExperimentService


page_token	A page token to request the next page of results. The token is acquried from the nextPageToken field of the response from the previous ListExperiment call or can be omitted when fetching the first page.	query	string
page_size	The number of experiments to be listed per page. If there are more experiments than this number, the response message will contain a nextPageToken field you can use to fetch the next page.	query	integer (int32)
sort_by	Can be format of "field_name", "field_name asc" or "field_name des" Ascending by default.	query	string
filter	A url-encoded, JSON-serialized Filter protocol buffer (see [filter.proto](https://github.com/kubeflow/pipelines/ blob/master/backend/api/filter.proto)).	query	string
resource_reference_key.type	The type of the resource that referred to.	query	string , x ∈ { UNKNOWN_RESOURCE_TYPE (default) , EXPERIMENT , JOB , PIPELINE , PIPELINE_VERSION , NAMESPACE }
resource_reference_key.id	The ID of the resource that referred to.	query	string

Uses default content-types: application/json

200 OK: A successful response.

apiListExperimentsResponse
default: apiStatus

Creates a new experiment.

POST /apis/v1beta1/experiments

Tags: ExperimentService

Uses default content-types: application/json

The experiment to be created.

apiExperiment

Uses default content-types: application/json

200 OK: A successful response.

apiExperiment
default: apiStatus

Deletes an experiment without deleting the experiment's runs and jobs. To avoid unexpected behaviors, delete an experiment's runs and jobs before deleting the experiment.

DELETE /apis/v1beta1/experiments/{id}

Tags: ExperimentService


id	The ID of the experiment to be deleted.	path	string

Uses default content-types: application/json

200 OK

A successful response.

default

apiStatus

Finds a specific experiment by ID.

GET /apis/v1beta1/experiments/{id}

Tags: ExperimentService


id	The ID of the experiment to be retrieved.	path	string

Uses default content-types: application/json

200 OK: A successful response.

apiExperiment
default: apiStatus

Archives an experiment and the experiment's runs and jobs.

POST /apis/v1beta1/experiments/{id}:archive

Tags: ExperimentService


id	The ID of the experiment to be archived.	path	string

Uses default content-types: application/json

200 OK

A successful response.

default

apiStatus

Restores an archived experiment. The experiment's archived runs and jobs will stay archived.

POST /apis/v1beta1/experiments/{id}:unarchive

Tags: ExperimentService


id	The ID of the experiment to be restored.	path	string

Uses default content-types: application/json

200 OK

A successful response.

default

apiStatus

Finds all jobs.

GET /apis/v1beta1/jobs

Tags: JobService


page_token	A page token to request the next page of results. The token is acquried from the nextPageToken field of the response from the previous ListJobs call or can be omitted when fetching the first page.	query	string
page_size	The number of jobs to be listed per page. If there are more jobs than this number, the response message will contain a nextPageToken field you can use to fetch the next page.	query	integer (int32)
sort_by	Can be format of "field_name", "field_name asc" or "field_name des". Ascending by default.	query	string
resource_reference_key.type	The type of the resource that referred to.	query	string , x ∈ { UNKNOWN_RESOURCE_TYPE (default) , EXPERIMENT , JOB , PIPELINE , PIPELINE_VERSION , NAMESPACE }
resource_reference_key.id	The ID of the resource that referred to.	query	string
filter	A url-encoded, JSON-serialized Filter protocol buffer (see [filter.proto](https://github.com/kubeflow/pipelines/ blob/master/backend/api/filter.proto)).	query	string

Uses default content-types: application/json

200 OK: A successful response.

apiListJobsResponse
default: apiStatus

Creates a new job.

POST /apis/v1beta1/jobs

Tags: JobService

Uses default content-types: application/json

The job to be created

apiJob

Uses default content-types: application/json

200 OK: A successful response.

apiJob
default: apiStatus

Deletes a job.

DELETE /apis/v1beta1/jobs/{id}

Tags: JobService


id	The ID of the job to be deleted	path	string

Uses default content-types: application/json

200 OK

A successful response.

default

apiStatus

Finds a specific job by ID.

GET /apis/v1beta1/jobs/{id}

Tags: JobService


id	The ID of the job to be retrieved	path	string

Uses default content-types: application/json

200 OK: A successful response.

apiJob
default: apiStatus

Stops a job and all its associated runs. The job is not deleted.

POST /apis/v1beta1/jobs/{id}/disable

Tags: JobService


id	The ID of the job to be disabled	path	string

Uses default content-types: application/json

200 OK

A successful response.

default

apiStatus

Restarts a job that was previously stopped. All runs associated with the job will continue.

POST /apis/v1beta1/jobs/{id}/enable

Tags: JobService


id	The ID of the job to be enabled	path	string

Uses default content-types: application/json

200 OK

A successful response.

default

apiStatus

Lists all pipeline versions of a given pipeline.

GET /apis/v1beta1/pipeline_versions

Tags: PipelineService


resource_key.type	The type of the resource that referred to.	query	string , x ∈ { UNKNOWN_RESOURCE_TYPE (default) , EXPERIMENT , JOB , PIPELINE , PIPELINE_VERSION , NAMESPACE }
resource_key.id	The ID of the resource that referred to.	query	string
page_size	The number of pipeline versions to be listed per page. If there are more pipeline versions than this number, the response message will contain a nextPageToken field you can use to fetch the next page.	query	integer (int32)
page_token	A page token to request the next page of results. The token is acquried from the nextPageToken field of the response from the previous ListPipelineVersions call or can be omitted when fetching the first page.	query	string
sort_by	Can be format of "field_name", "field_name asc" or "field_name des" Ascending by default.	query	string
filter	A base-64 encoded, JSON-serialized Filter protocol buffer (see filter.proto).	query	string

Uses default content-types: application/json

200 OK: A successful response.

apiListPipelineVersionsResponse
default: apiStatus

Adds a pipeline version to the specified pipeline.

POST /apis/v1beta1/pipeline_versions

Tags: PipelineService

Uses default content-types: application/json

ResourceReference inside PipelineVersion specifies the pipeline that this version belongs to.

apiPipelineVersion

Uses default content-types: application/json

200 OK: A successful response.

apiPipelineVersion
default: apiStatus

Deletes a pipeline version by pipeline version ID. If the deleted pipeline version is the default pipeline version, the pipeline's default version changes to the pipeline's most recent pipeline version. If there are no remaining pipeline versions, the pipeline will have no default version. Examines the run_service_api.ipynb notebook to learn more about creating a run using a pipeline version (https://github.com/kubeflow/pipelines/blob/master/tools/benchmarks/run_service_api.ipynb).

DELETE /apis/v1beta1/pipeline_versions/{version_id}

Tags: PipelineService


version_id	The ID of the pipeline version to be deleted.	path	string

Uses default content-types: application/json

200 OK

A successful response.

default

apiStatus

Gets a pipeline version by pipeline version ID.

GET /apis/v1beta1/pipeline_versions/{version_id}

Tags: PipelineService


version_id	The ID of the pipeline version to be retrieved.	path	string

Uses default content-types: application/json

200 OK: A successful response.

apiPipelineVersion
default: apiStatus

Returns a YAML template that contains the specified pipeline version's description, parameters and metadata.

GET /apis/v1beta1/pipeline_versions/{version_id}/templates

Tags: PipelineService


version_id	The ID of the pipeline version whose template is to be retrieved.	path	string

Uses default content-types: application/json

200 OK: A successful response.

apiGetTemplateResponse
default: apiStatus

Finds all pipelines.

GET /apis/v1beta1/pipelines

Tags: PipelineService


page_token	A page token to request the next page of results. The token is acquried from the nextPageToken field of the response from the previous ListPipelines call.	query	string
page_size	The number of pipelines to be listed per page. If there are more pipelines than this number, the response message will contain a valid value in the nextPageToken field.	query	integer (int32)
sort_by	Can be format of "field_name", "field_name asc" or "field_name des" Ascending by default.	query	string
filter	A url-encoded, JSON-serialized Filter protocol buffer (see [filter.proto](https://github.com/kubeflow/pipelines/ blob/master/backend/api/filter.proto)).	query	string

Uses default content-types: application/json

200 OK: A successful response.

apiListPipelinesResponse
default: apiStatus

Creates a pipeline.

POST /apis/v1beta1/pipelines

Tags: PipelineService

Uses default content-types: application/json

apiPipeline

Uses default content-types: application/json

200 OK: A successful response.

apiPipeline
default: apiStatus

POST /apis/v1beta1/pipelines/upload

Tags: PipelineUploadService

multipart/form-data


uploadfile	The pipeline to upload. Maximum size of 32MB is supported.	formData	file
name		query	string
description		query	string

application/json

200 OK: apiPipeline
default: apiStatus

POST /apis/v1beta1/pipelines/upload_version

Tags: PipelineUploadService

multipart/form-data


uploadfile	The pipeline to upload. Maximum size of 32MB is supported.	formData	file
name		query	string
pipelineid		query	string

application/json

200 OK: apiPipelineVersion
default: apiStatus

Deletes a pipeline and its pipeline versions.

DELETE /apis/v1beta1/pipelines/{id}

Tags: PipelineService


id	The ID of the pipeline to be deleted.	path	string

Uses default content-types: application/json

200 OK

A successful response.

default

apiStatus

Finds a specific pipeline by ID.

GET /apis/v1beta1/pipelines/{id}

Tags: PipelineService


id	The ID of the pipeline to be retrieved.	path	string

Uses default content-types: application/json

200 OK: A successful response.

apiPipeline
default: apiStatus

Returns a single YAML template that contains the description, parameters, and metadata associated with the pipeline provided.

GET /apis/v1beta1/pipelines/{id}/templates

Tags: PipelineService


id	The ID of the pipeline whose template is to be retrieved.	path	string

Uses default content-types: application/json

200 OK: A successful response.

apiGetTemplateResponse
default: apiStatus

Finds all runs.

GET /apis/v1beta1/runs

Tags: RunService


page_token	A page token to request the next page of results. The token is acquried from the nextPageToken field of the response from the previous ListRuns call or can be omitted when fetching the first page.	query	string
page_size	The number of runs to be listed per page. If there are more runs than this number, the response message will contain a nextPageToken field you can use to fetch the next page.	query	integer (int32)
sort_by	Can be format of "field_name", "field_name asc" or "field_name des" (Example, "name asc" or "id des"). Ascending by default.	query	string
resource_reference_key.type	The type of the resource that referred to.	query	string , x ∈ { UNKNOWN_RESOURCE_TYPE (default) , EXPERIMENT , JOB , PIPELINE , PIPELINE_VERSION , NAMESPACE }
resource_reference_key.id	The ID of the resource that referred to.	query	string
filter	A url-encoded, JSON-serialized Filter protocol buffer (see [filter.proto](https://github.com/kubeflow/pipelines/ blob/master/backend/api/filter.proto)).	query	string

Uses default content-types: application/json

200 OK: A successful response.

apiListRunsResponse
default: apiStatus

Creates a new run.

POST /apis/v1beta1/runs

Tags: RunService

Uses default content-types: application/json

apiRun

Uses default content-types: application/json

200 OK: A successful response.

apiRunDetail
default: apiStatus

Deletes a run.

DELETE /apis/v1beta1/runs/{id}

Tags: RunService


id	The ID of the run to be deleted.	path	string

Uses default content-types: application/json

200 OK

A successful response.

default

apiStatus

Archives a run.

POST /apis/v1beta1/runs/{id}:archive

Tags: RunService


id	The ID of the run to be archived.	path	string

Uses default content-types: application/json

200 OK

A successful response.

default

apiStatus

Restores an archived run.

POST /apis/v1beta1/runs/{id}:unarchive

Tags: RunService


id	The ID of the run to be restored.	path	string

Uses default content-types: application/json

200 OK

A successful response.

default

apiStatus

Finds a specific run by ID.

GET /apis/v1beta1/runs/{run_id}

Tags: RunService


run_id	The ID of the run to be retrieved.	path	string

Uses default content-types: application/json

200 OK: A successful response.

apiRunDetail
default: apiStatus

Finds a run's artifact data.

GET /apis/v1beta1/runs/{run_id}/nodes/{node_id}/artifacts/{artifact_name}:read

Tags: RunService


run_id	The ID of the run.	path	string
node_id	The ID of the running node.	path	string
artifact_name	The name of the artifact.	path	string

Uses default content-types: application/json

200 OK: A successful response.

apiReadArtifactResponse
default: apiStatus

Re-initiates a failed or terminated run.

POST /apis/v1beta1/runs/{run_id}/retry

Tags: RunService


run_id	The ID of the run to be retried.	path	string

Uses default content-types: application/json

200 OK

A successful response.

default

apiStatus

Terminates an active run.

POST /apis/v1beta1/runs/{run_id}/terminate

Tags: RunService


run_id	The ID of the run to be terminated.	path	string

Uses default content-types: application/json

200 OK

A successful response.

default

apiStatus

ReportRunMetrics reports metrics of a run. Each metric is reported in its own transaction, so this API accepts partial failures. Metric can be uniquely identified by (run_id, node_id, name). Duplicate reporting will be ignored by the API. First reporting wins.

POST /apis/v1beta1/runs/{run_id}:reportMetrics

Tags: RunService

Uses default content-types: application/json

apiReportRunMetricsRequest


run_id	Required. The parent run ID of the metric.	path	string

Uses default content-types: application/json

200 OK: A successful response.

apiReportRunMetricsResponse
default: apiStatus

Schema definitions

apiCronSchedule: object

start_time: string (date-time)
end_time: string (date-time)
cron: string

apiExperiment: object

id: string: Output. Unique experiment ID. Generated by API server.
name: string: Required input field. Unique experiment name provided by user.
description: string
created_at: string (date-time): Output. The time that the experiment created.
resource_references: object[]: Optional input field. Specify which resource this run belongs to. For Experiment, the only valid resource reference is a single Namespace.

apiResourceReference
storage_state: ExperimentStorageState: Output. Specifies whether this experiment is in archived or available state.

apiGetTemplateResponse: object

template: string: The template of the pipeline specified in a GetTemplate request, or of a pipeline version specified in a GetPipelinesVersionTemplate request.

apiJob: object

id: string: Output. Unique run ID. Generated by API server.
name: string: Required input field. Job name provided by user. Not unique.
description: string
pipeline_spec: apiPipelineSpec: Required input field. Describing what the pipeline manifest and parameters to use for the scheduled job.
resource_references: object[]: Optional input field. Specify which resource this job belongs to.

apiResourceReference
service_account: string: Optional input field. Specify which Kubernetes service account this job uses.
max_concurrency: string (int64)
trigger: apiTrigger: Required input field. Specify how a run is triggered. Support cron mode or periodic mode.
mode: JobMode
created_at: string (date-time): Output. The time this job is created.
updated_at: string (date-time): Output. The last time this job is updated.
status: string
error: string: In case any error happens retrieving a job field, only job ID and the error message is returned. Client has the flexibility of choosing how to handle error. This is especially useful during listing call.
enabled: boolean (boolean): Input. Whether the job is enabled or not.
no_catchup: boolean (boolean): Optional input field. Whether the job should catch up if behind schedule. If true, the job will only schedule the latest interval if behind schedule. If false, the job will catch up on each past interval.

apiListExperimentsResponse: object

experiments: object[]: A list of experiments returned.

apiExperiment
total_size: integer (int32): The total number of experiments for the given query.
next_page_token: string: The token to list the next page of experiments.

apiListJobsResponse: object

jobs: object[]: A list of jobs returned.

apiJob
total_size: integer (int32): The total number of jobs for the given query.
next_page_token: string: The token to list the next page of jobs.

apiListPipelinesResponse: object

pipelines: object[]: apiPipeline
total_size: integer (int32): The total number of pipelines for the given query.
next_page_token: string: The token to list the next page of pipelines.

apiListPipelineVersionsResponse: object

versions: object[]: apiPipelineVersion
next_page_token: string: The token to list the next page of pipeline versions.
total_size: integer (int32): The total number of pipeline versions for the given query.

apiListRunsResponse: object

runs: object[]: apiRun
total_size: integer (int32): The total number of runs for the given query.
next_page_token: string: The token to list the next page of runs.

apiParameter: object

name: string
value: string

apiPeriodicSchedule: object

start_time: string (date-time)
end_time: string (date-time)
interval_second: string (int64)

apiPipeline: object

id: string: Output. Unique pipeline ID. Generated by API server.
created_at: string (date-time): Output. The time this pipeline is created.
name: string: Optional input field. Pipeline name provided by user. If not specified, file name is used as pipeline name.
description: string: Optional input field. Describing the purpose of the job.
parameters: object[]: Output. The input parameters for this pipeline. TODO(jingzhang36): replace this parameters field with the parameters field inside PipelineVersion when all usage of the former has been changed to use the latter.

apiParameter
url: apiUrl: The URL to the source of the pipeline. This is required when creating the pipeine through CreatePipeline API. TODO(jingzhang36): replace this url field with the code_source_urls field inside PipelineVersion when all usage of the former has been changed to use the latter.
error: string: In case any error happens retrieving a pipeline field, only pipeline ID and the error message is returned. Client has the flexibility of choosing how to handle error. This is especially useful during listing call.
default_version: apiPipelineVersion

apiPipelineRuntime: object

pipeline_manifest: string: Output. The runtime JSON manifest of the pipeline, including the status of pipeline steps and fields need for UI visualization etc.
workflow_manifest: string: Output. The runtime JSON manifest of the argo workflow. This is deprecated after pipeline_runtime_manifest is in use.

apiPipelineSpec: object

pipeline_id: string: Optional input field. The ID of the pipeline user uploaded before.
pipeline_name: string: Optional output field. The name of the pipeline. Not empty if the pipeline id is not empty.
workflow_manifest: string: Optional input field. The marshalled raw argo JSON workflow. This will be deprecated when pipeline_manifest is in use.
pipeline_manifest: string: Optional input field. The raw pipeline JSON spec.
parameters: object[]: The parameter user provide to inject to the pipeline JSON. If a default value of a parameter exist in the JSON, the value user provided here will replace.

apiParameter

apiPipelineVersion: object

id: string: Output. Unique version ID. Generated by API server.
name: string: Optional input field. Version name provided by user.
created_at: string (date-time): Output. The time this pipeline version is created.
parameters: object[]: Output. The input parameters for this pipeline.

apiParameter
code_source_url: string: Input. Optional. Pipeline version code source.
package_url: apiUrl: Input. Required. Pipeline version package url. Whe calling CreatePipelineVersion API method, need to provide one package file location.
resource_references: object[]: Input. Required. E.g., specify which pipeline this pipeline version belongs to.

apiResourceReference

apiReadArtifactResponse: object

data: string (byte): The bytes of the artifact content.

apiRelationship: string , x ∈ { UNKNOWN_RELATIONSHIP (default) , OWNER , CREATOR }

apiReportRunMetricsRequest: object

run_id: string: Required. The parent run ID of the metric.
metrics: object[]: List of metrics to report.

apiRunMetric

apiReportRunMetricsResponse: object

results: object[]: ReportRunMetricsResponseReportRunMetricResult

apiResourceKey: object

type: apiResourceType: The type of the resource that referred to.
id: string: The ID of the resource that referred to.

apiResourceReference: object

key: apiResourceKey
name: string: The name of the resource that referred to.
relationship: apiRelationship: Required field. The relationship from referred resource to the object.

apiResourceType: string , x ∈ { UNKNOWN_RESOURCE_TYPE (default) , EXPERIMENT , JOB , PIPELINE , PIPELINE_VERSION , NAMESPACE }

apiRun: object

id: string: Output. Unique run ID. Generated by API server.
name: string: Required input field. Name provided by user, or auto generated if run is created by scheduled job. Not unique.
storage_state: RunStorageState: Output. Specify whether this run is in archived or available mode.
description: string
pipeline_spec: apiPipelineSpec: Required input field. Describing what the pipeline manifest and parameters to use for the run.
resource_references: object[]: Optional input field. Specify which resource this run belongs to. When creating a run from a particular pipeline version, the pipeline version can be specified here.

apiResourceReference
service_account: string: Optional input field. Specify which Kubernetes service account this run uses.
created_at: string (date-time): Output. The time that the run created.
scheduled_at: string (date-time): Output. When this run is scheduled to run. This could be different from created_at. For example, if a run is from a backfilling job that was supposed to run 2 month ago, the scheduled_at is 2 month ago, v.s. created_at is the current time.
finished_at: string (date-time): Output. The time this run is finished.
status: string
error: string: In case any error happens retrieving a run field, only run ID and the error message is returned. Client has the flexibility of choosing how to handle error. This is especially useful during listing call.
metrics: object[]: Output. The metrics of the run. The metrics are reported by ReportMetrics API.

apiRunMetric

apiRunDetail: object

run: apiRun
pipeline_runtime: apiPipelineRuntime

apiRunMetric: object

name: string: Required. The user defined name of the metric. It must between 1 and 63 characters long and must conform to the following regular expression: [a-z]([-a-z0-9]*[a-z0-9])?.
node_id: string: Required. The runtime node ID which reports the metric. The node ID can be found in the RunDetail.workflow.Status. Metric with same (node_id, name) are considerd as duplicate. Only the first reporting will be recorded. Max length is 128.
number_value: number (double): The number value of the metric.
format: RunMetricFormat: The display format of metric.

apiStatus: object

error: string
code: integer (int32)
details: object[]: protobufAny

apiTrigger: object

Trigger defines what starts a pipeline run.

cron_schedule: apiCronSchedule
periodic_schedule: apiPeriodicSchedule

apiUrl: object

pipeline_url: string: URL of the pipeline definition or the pipeline version definition.

ExperimentStorageState: string , x ∈ { STORAGESTATE_UNSPECIFIED (default) , STORAGESTATE_AVAILABLE , STORAGESTATE_ARCHIVED }

JobMode: string , x ∈ { UNKNOWN_MODE (default) , ENABLED , DISABLED }

Required input.

DISABLED: The job won't schedule any run if disabled.

protobufAny: object

Any contains an arbitrary serialized protocol buffer message along with a URL that describes the type of the serialized message.

Protobuf library provides support to pack/unpack Any values in the form of utility functions or additional generated methods of the Any type.

Example 1: Pack and unpack a message in C++.

Foo foo = ...;
Any any;
any.PackFrom(foo);
...
if (any.UnpackTo(&foo)) {
  ...
}

Example 2: Pack and unpack a message in Java.

Foo foo = ...;
Any any = Any.pack(foo);
...
if (any.is(Foo.class)) {
  foo = any.unpack(Foo.class);
}

Example 3: Pack and unpack a message in Python.

foo = Foo(...)
any = Any()
any.Pack(foo)
...
if any.Is(Foo.DESCRIPTOR):
  any.Unpack(foo)
  ...

Example 4: Pack and unpack a message in Go

 foo := &pb.Foo{...}
 any, err := ptypes.MarshalAny(foo)
 ...
 foo := &pb.Foo{}
 if err := ptypes.UnmarshalAny(any, foo); err != nil {
   ...
 }

The pack methods provided by protobuf library will by default use 'type.googleapis.com/full.type.name' as the type URL and the unpack methods only use the fully qualified type name after the last '/' in the type URL, for example "foo.bar.com/x/y.z" will yield type name "y.z".

JSON

The JSON representation of an Any value uses the regular representation of the deserialized, embedded message, with an additional field @type which contains the type URL. Example:

package google.profile;
message Person {
  string first_name = 1;
  string last_name = 2;
}

                            {
  "@type": "type.googleapis.com/google.profile.Person",
  "firstName": <string>,
  "lastName": <string>
}

If the embedded message type is well-known and has a custom JSON representation, that representation will be embedded adding a field value which holds the custom JSON in addition to the @type field. Example (for message [google.protobuf.Duration][]):

{
  "@type": "type.googleapis.com/google.protobuf.Duration",
  "value": "1.212s"
}

type_url: string

A URL/resource name that uniquely identifies the type of the serialized protocol buffer message. The last segment of the URL's path must represent the fully qualified name of the type (as in path/google.protobuf.Duration). The name should be in a canonical form (e.g., leading "." is not accepted).

In practice, teams usually precompile into the binary all types that they expect it to use in the context of Any. However, for URLs which use the scheme http, https, or no scheme, one can optionally set up a type server that maps type URLs to message definitions as follows:

If no scheme is provided, https is assumed.
An HTTP GET on the URL must yield a [google.protobuf.Type][] value in binary format, or produce an error.
Applications are allowed to cache lookup results based on the URL, or have them precompiled into a binary to avoid any lookup. Therefore, binary compatibility needs to be preserved on changes to types. (Use versioned type names to manage breaking changes.)

Note: this functionality is not currently available in the official protobuf release, and it is not used for type URLs beginning with type.googleapis.com.

Schemes other than http, https (or the empty scheme) might be used with implementation specific semantics.

value: string (byte)

Must be a valid serialized protocol buffer of the above specified type.

ReportRunMetricsResponseReportRunMetricResult: object

metric_name: string: Output. The name of the metric.
metric_node_id: string: Output. The ID of the node which reports the metric.
status: ReportRunMetricsResponseReportRunMetricResultStatus: Output. The status of the metric reporting.
message: string: Output. The detailed message of the error of the reporting.

ReportRunMetricsResponseReportRunMetricResultStatus: string , x ∈ { UNSPECIFIED (default) , OK , INVALID_ARGUMENT , DUPLICATE_REPORTING , INTERNAL_ERROR }

UNSPECIFIED: Default value if not present.
OK: Indicates successful reporting.
INVALID_ARGUMENT: Indicates that the payload of the metric is invalid.
DUPLICATE_REPORTING: Indicates that the metric has been reported before.
INTERNAL_ERROR: Indicates that something went wrong in the server.

RunMetricFormat: string , x ∈ { UNSPECIFIED (default) , RAW , PERCENTAGE }

UNSPECIFIED: Default value if not present.
RAW: Display value as its raw format.
PERCENTAGE: Display value in percentage format.

RunStorageState: string , x ∈ { STORAGESTATE_AVAILABLE (default) , STORAGESTATE_ARCHIVED }

Docs: Pipelines SDK Reference

Mon, 01 Jan 0001 00:00:00 +0000

See the generated reference docs for the Python SDK (hosted on Read the Docs).