Kubeflow – Pipelines

Docs: Pipelines Quickstart

Mon, 01 Jan 0001 00:00:00 +0000

Beta

This Kubeflow component has beta status. See the Kubeflow versioning policies. The Kubeflow team is interested in your feedback about the usability of the feature.

Use this guide if you want to get a simple pipeline running quickly in Kubeflow Pipelines.

This quickstart guide shows you how to use one of the samples that come with the Kubeflow Pipelines installation and are visible on the Kubeflow Pipelines user interface (UI). You can use this guide as an introduction to the Kubeflow Pipelines UI.
The end-to-end tutorial shows you how to prepare and compile a pipeline, upload it to Kubeflow Pipelines, then run it.

Deploy Kubeflow and open the pipelines UI

Follow these steps to deploy Kubeflow and open the pipelines dashboard:

Follow the guide to deploying Kubeflow on GCP.

Due to kubeflow/pipelines#1700, the container builder in Kubeflow Pipelines currently prepares credentials for Google Cloud Platform (GCP) only. As a result, the container builder supports only Google Container Registry. However, you can store the container images on other registries, provided you set up the credentials correctly to fetchthe image.
When Kubeflow is running, access the Kubeflow UI at a URL of the form https://<deployment-name>.endpoints.<project>.cloud.goog/, as described in the setup guide. The Kubeflow UI looks like this:

If you skipped the Cloud IAP option when deploying Kubeflow, or if you haven’t yet set up your Kubeflow endpoint, you can access Kubeflow via kubectl and port-forwarding:
1. Install kubectl if you haven’t already done so, by running the following command on the command line: gcloud components install kubectl. For more information, see the kubectl documentation.
2. Run kubectl port-forward -n kubeflow svc/ml-pipeline-ui 8080:80 and go to http://localhost:8080/.
Click Pipelines to access the pipelines UI. The pipelines UI looks like this:

Run a basic pipeline

The pipelines UI offers a few samples that you can use to try out pipelines quickly. The steps below show you how to run a basic sample that includes some Python operations, but doesn’t include a machine learning (ML) workload:

Click the name of the sample, [Sample] Basic - Parallel Execution, on the pipelines UI:
Click Create experiment:
Follow the prompts to create an experiment and then create a run. The sample supplies default values for all the parameters you need. The following screenshot assumes you’ve already created an experiment named My experiment and are now creating a run named My first run:
Click Start to create the run.
Click the name of the run on the experiments dashboard:
Explore the graph and other aspects of your run by clicking on the components of the graph and the other UI elements:

You can find the source code for the basic parallel join sample in the Kubeflow Pipelines repo.

Run an ML pipeline

This section shows you how to run the XGBoost sample available from the pipelines UI. Unlike the basic sample described above, the XGBoost sample does include ML components. Before running this sample, you need to set up some GCP services for use by the sample.

Follow these steps to set up the necessary GCP services and run the sample:

In addition to the standard GCP APIs that you need for Kubeflow (see the GCP setup guide), ensure that the following APIs are enabled:
- Cloud Storage
- Dataproc
Create a Cloud Storage bucket to hold the results of the pipeline run.
- Your bucket name must be unique across all of Cloud Storage.
- Each time you create a new run for this pipeline, Kubeflow creates a unique directory within the output bucket, so the output of each run does not override the output of the previous run.
Click the name of the sample, [Sample] ML - XGBoost - Training with Confusion Matrix, on the pipelines UI:
Click Create experiment.
Follow the prompts to create an experiment and then create a run. Supply the following run parameters:
- output: The Cloud Storage bucket that you created earlier to hold the results of the pipeline run.
- project: Your GCP project ID.
The sample supplies the values for the other parameters:
- region: The GCP geographical region in which the training and evaluation data is stored.
- train-data: Cloud Storage path to the training data.
- eval-data: Cloud Storage path to the evaluation data.
- schema: Cloud Storage path to a JSON file describing the format of the CSV files that contain the training and evaluation data.
- target: Column name of the target variable.
- rounds: The number of rounds for XGBoost training.
- workers: Number of workers used for distributed training.
- true-label: Column to be used for text representation of the label output by the model.
The following partial screenshot shows the run parameters, including the two parameters that you must supply:
Click Start to create the run.
Click the name of the run on the experiments dashboard.
Explore the graph and other aspects of your run by clicking on the components of the graph and the other UI elements. The following screenshot shows the graph when the pipeline has finished running:

You can find the source code for the XGBoost training sample in the Kubeflow Pipelines repo.

Clean up your GCP environment

As you work through this guide, your project uses billable components of GCP. To minimise costs, follow these steps to clean up resources when you’ve finished with them:

Visit Deployment Manager to delete your deployment and related resources.
Delete your Cloud Storage bucket when you’ve finished examining the output of the pipeline.

Next steps

Learn more about the important concepts in Kubeflow Pipelines.
This page showed you how to run some of the examples supplied in the Kubeflow Pipelines UI. Next, you may want to run a pipeline from a notebook, or compile and run a sample from the code. See the guide to experimenting with the Kubeflow Pipelines samples.
Build your own machine-learning pipelines with the Kubeflow Pipelines SDK.

Docs: Installing Pipelines

Mon, 01 Jan 0001 00:00:00 +0000

Docs: Understanding Pipelines

Mon, 01 Jan 0001 00:00:00 +0000

Docs: Building Pipelines with the SDK

Mon, 01 Jan 0001 00:00:00 +0000

Docs: Caching

Mon, 01 Jan 0001 00:00:00 +0000

Starting from Kubeflow Pipelines 0.4, Kubeflow Pipelines supports step caching capabilities in both standalone deployment and GCP hosted deployment.

Before you start

This guide tells you the basic concepts of Kubeflow Pipelines step caching and how to use it. This guide assumes that you already have Kubeflow Pipelines installed or want to use standalone or GCP hosted deployment options in the Kubeflow Pipelines deployment guide to deploy Kubeflow Pipelines.

What is step caching?

Kubeflow Pipelines caching provides step-level output caching. And caching is enabled by default for all pipelines submitted through the KFP backend and UI. The exception is pipelines authored using TFX SDK which has its own caching mechanism. The cache key calculation is based on the component (base image, command-line, code), arguments passed to the component (values or artifacts) and any additional customizations. If the component is exactly the same and the arguments are exactly the same as in some previous execution, then the task can be skipped and the outputs of the old step can be used. The cache reuse behavior can be controlled and the pipeline author can specify the maximum staleness of the cached data considered for reuse. With caching enabled, the system can skip a step that has already been executed which saves time and money.

Disabling/enabling caching

Cache is enabled by default after Kubeflow Pipelines 0.4. These are instructions on disabling and enabling cache service:

Configure access to your Kubeflow cluster

Use the following instructions to configure kubectl with access to your Kubeflow cluster.

To check if you have kubectl installed, run the following command:
```
which kubectl
```
The response should be something like this:
```
/usr/bin/kubectl
```
If you do not have kubectl installed, follow the instructions in the guide to installing and setting up kubectl.
Follow the guide to configuring access to Kubernetes clusters.

Disabling caching in your Kubeflow Pipelines deployment:

Make sure mutatingwebhookconfiguration exists in your cluster:

export NAMESPACE=<Namespace where KFP is installed>
kubectl get mutatingwebhookconfiguration cache-webhook -n ${NAMESPACE}

Change mutatingwebhookconfiguration rules:

kubectl patch mutatingwebhookconfiguration cache-webhook -n ${NAMESPACE} --type='json' -p='[{"op":"replace", "path": "/webhooks/0/rules/0/operations/0", "value": "DELETE"}]'

Enabling caching

Make sure mutatingwebhookconfiguration exists in your cluster:

export NAMESPACE=<Namespace where KFP is installed>
kubectl get mutatingwebhookconfiguration cache-webhook -n ${NAMESPACE}

Change back mutatingwebhookconfiguration rules:

kubectl patch mutatingwebhookconfiguration cache-webhook -n ${NAMESPACE} --type='json' -p='[{"op":"replace", "path": "/webhooks/0/rules/0/operations/0", "value": "CREATE"}]'

Managing caching staleness

The cache is enabled by default and if you ever executed same component with the same arguments, any new execution of the component will be skipped and the outputs will be taken from the cache. For some scenarios, the cached output data of some components might become too stale for use after some time. To control the maximum staleness of the reused cached data, you can set the step’s max_cache_staleness parameter. The max_cache_staleness is in RFC3339 Duration format (so 30 days = “P30D”). By default the max_cache_staleness is set to infinity so any old cached data will be reused.

Set max_cache_staleness to 30 days for a step:

def some_pipeline():
      # task is a target step in a pipeline
      task = some_op()
      task.execution_options.caching_strategy.max_cache_staleness = "P30D"

Ideally, the component code should be pure and deterministic in the sense that it produces same outputs given same inputs. If your component is not deterministic (for example, it returns a different random number on every invocation) you might want to disable caching for the tasks created from this component by setting max_cache_staleness to 0:

def some_pipeline():
      # task is a target step in a pipeline
      task_never_use_cache = some_op()
      task_never_use_cache.execution_options.caching_strategy.max_cache_staleness = "P0D"

A better solution would be to make the component deterministic. If the component uses random number generation, you can expose the RNG seed as a component input. If the component fetches some changing data you can add a timestamp or date input.

Docs: Upgrading and Reinstalling

Mon, 01 Jan 0001 00:00:00 +0000

Starting from Kubeflow v0.5, Kubeflow Pipelines persists the pipeline data in permanent storage volumes. Kubeflow Pipelines therefore supports the following capabilities:

Reinstall: You can delete a cluster and create a new cluster, specifying the existing storage volumes to retrieve the original data in the new cluster.
Upgrade (limited support):

The full Kubeflow deployment currently supports upgrading in alpha status with limited support. Check the following sources for progress updates:
- Issue kubeflow/kubeflow #3727.
- Kubeflow upgrade guide.

Before you start

This guide tells you how to reinstall Kubeflow Pipelines as part of a full Kubeflow deployment. This guide therefore assumes that you want to use one of the options in the Kubeflow deployment guide to deploy Kubeflow Pipelines with Kubeflow.

Note the following alternatives:

Instead of the full Kubeflow deployment, you can use Kubeflow Pipelines Standalone, which does support upgrading. See how to upgrade the Kubeflow Pipelines Standalone deployment.
If you’re using Kubeflow Pipelines on Google Cloud Platform (GCP), see how to upgrade or reinstall Kubeflow Pipelines on GCP.

Kubeflow Pipelines data storage

Kubeflow Pipelines creates and manages the following data related to your machine learning pipeline:

Metadata: Experiments, jobs, runs, etc. Kubeflow Pipelines stores the pipeline metadata in a MySQL database.
Artifacts: Pipeline packages, metrics, views, etc. Kubeflow Pipelines stores the artifacts in a Minio server.

Kubeflow Pipelines uses the Kubernetes PersistentVolume (PV) subsystem to provision the MySQL database and the Minio server. You can specify your own preferred PV.

Deploying Kubeflow

This section describes how to deploy Kubeflow in a way that ensures you can use the Kubeflow Pipelines reinstallation capability.

If you don’t need custom storage and are happy with the default PVs that Kubeflow provides, you can follow the Kubeflow deployment guide without doing anything extra. The deployment process uses the Kubernetes default StorageClass to provision the PVs for you.
If you want to specify a custom PV:
1. Create your Kubernetes cluster if you don’t already have one. See the Kubernetes documentation.
2. Create two PVs in your Kubernetes cluster with your preferred storage type. See the Kubernetes guide to PVs.
3. Follow the Kubeflow deployment guide, but note the following changes to the standard procedure.
4. Before running the kfctl apply command:
  - Edit ${KF_DIR}/kustomize/minio/overlays/minioPd/params.env and specify the PV for the Minio server:
```
...
minioPd=[YOUR-PRE-CREATED-MINIO-PV-NAME]
...
```
  - Edit ${KF_DIR}/kustomize/mysql/overlays/mysqlPd/params.env and specify the PV for the MySQL database:
```
...
mysqlPd=[YOUR-PRE-CREATED-MYSQL-PV-NAME]
...
```
5. Run the kfctl apply command to deploy Kubeflow as usual:
```
kfctl apply -V -f ${CONFIG_FILE}
```

Reinstalling Kubeflow Pipelines

You can delete a Kubeflow cluster and create a new one, specifying your existing storage to retrieve the original data in the new cluster.

Notes:

You must use command-line deployment. You cannot reinstall Kubeflow Pipelines using the web interface.
When you do kfctl apply or kfctl build, you should use a different deployment name from your existing deployment name. Using a different name ensures that your data is safe in case of a deployment failure. This guide defines the deployment name in the ${KF_NAME} environment variable.
The reinstallation steps are the same as for a standard Kubeflow installation, except that you must use the same PV definitions as in your previous deployment to create the PV in the new cluster.

To reinstall Kubeflow Pipelines:

Create two PVs in your Kubernetes cluster, using the same PV definitions as in your previous deployment. See the Kubernetes guide to PVs.
Follow the Kubeflow deployment guide, but note the following changes to the standard procedure.
Set a different ${KF_NAME} name from your existing ${KF_NAME}.
Before running the kfctl apply command:
- Edit ${KF_DIR}/kustomize/minio/overlays/minioPd/params.env and specify the PV for the Minio server:
```
...
minioPd=[YOUR-PRE-CREATED-MINIO-PV-NAME]
...
```
- Edit ${KF_DIR}/kustomize/mysql/overlays/mysqlPd/params.envand specify the PV for the MySQL database:
```
...
mysqlPd=[YOUR-PRE-CREATED-MYSQL-PV-NAME]
...
```
Run the kfctl apply command to deploy Kubeflow as usual:
```
kfctl apply -V -f ${CONFIG_FILE}
```

You should now have a new Kubeflow deployment that uses the same pipelines data storage as your previous deployment. Follow any remaining steps in the Kubeflow deployment guide to check your deployment, depending on the deployment option you chose.

Docs: Samples and Tutorials

Mon, 01 Jan 0001 00:00:00 +0000

Docs: Troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

This page presents some hints for troubleshooting specific problems that you may encounter.

Diagnosing problems in your Kubeflow Pipelines environment

For help diagnosing environment issues that affect Kubeflow Pipelines, run the kfp diagnose_me command-line tool.

The kfp diagnose_me CLI reports on the configuration of your local development environment, Kubernetes cluster, or Google Cloud environment. Use this command to help resolve issues like the following:

Python library dependencies
Trouble accessing resources or APIs using Kubernetes secrets
Trouble accessing Persistent Volume Claims

To use the kfp diagnose_me CLI, follow these steps:

Install the Kubeflow Pipelines SDK.
Follow the guide to configuring access to Kubernetes clusters, to update your kubeconfig file with appropriate credentials and endpoint information to access your Kubeflow cluster. If your Kubeflow Pipelines cluster is hosted on a cloud provider like Google Cloud, use your cloud provider’s instructions for configuring access to your Kubernetes cluster.
Run the kfp diagnose_me command.
Analyze the results to troubleshoot your environment.

Troubleshooting the Kubeflow Pipelines SDK

The following sections describe how to resolve issues that can occur when installing or using the Kubeflow Pipelines SDK.

Error: Could not find a version that satisfies the requirement kfp

This error indicates that you have not installed the kfp package in your Python3 environment. Follow the instructions in the Kubeflow Pipelines SDK installation guide, if you have not already installed the SDK.

If you have already installed the Kubeflow Pipelines SDK, check that you have Python 3.5 or higher:

python3 -V

The response should be something like the following:

Python 3.7.3

If you do not have Python 3.5 or higher, you can download Python from the Python Software Foundation.

kfp or dsl-compile command not found

If your install the Kubeflow Pipelines SDK with the --user flag, you may get the following error when using the kfp or dsl-compile command-line tools.

bash: kfp: command not found

This error occurs because installing the Kubeflow Pipelines SDK with --user stores kfp and dsl-compile in your ~/.local/bin directory. In some Linux distributions, the ~/.local/bin directory is not part of the $PATH environment variable.

You can resolve this issue by using one of the following options:

Add export $PATH=$PATH:~/.local/bin to the end of your ~/.bashrc file. Then restart your terminal session or run source ~/.bashrc.
Run the kfp and dsl-compile commands as ~/.local/bin/kfp and ~/.local/bin/dsl-compile.

Docs: Reference

Mon, 01 Jan 0001 00:00:00 +0000