Kubeflow – Pipelines on GCP

Docs: Authenticating Pipelines to GCP

Mon, 01 Jan 0001 00:00:00 +0000

This page describes authentication for Kubeflow Pipelines to GCP. Available options listed below have different tradeoffs. You should choose the one that fits your use-case.

Compute Engine default service account is easy to set up, but overgrants permission if enabled access to “cloud-platform” scope. Therefore, it is not suitable for a shared GCP project.
Workload Identity takes more efforts to set up, but allows fine-grained permission control. It is recommended for production use-cases.
Google service account keys stored as Kubernetes secrets is the legacy approach and no longer recommended in GKE. However, it’s the only option to use GCP APIs when your cluster is an anthos or on-prem cluster.

NOTE: AI Platform Pipelines only supports Compute Engine default service account out of the box. If you want custom configurations, recommend using Pipelines Standalone instead. For details, please refer to AI Platform Pipelines documentation.

Before you begin

Installation Options for Kubeflow Pipelines introduces options to install Pipelines. Be aware that authentication support and cluster setup instructions will vary depending on the option you installed Kubeflow Pipelines with.

Compute Engine default service account

This is good for trying out Kubeflow Pipelines, because it is easy to set up, but does not support permission separation for workloads in the cluster.

NOTE: Using pipelines with Compute Engine default service account is not supported in Full Kubeflow deployment.

Cluster setup to use Compute Engine default service account

By default, your GKE nodes use Compute Engine default service account. If you allowed cloud-platform scope when creating the cluster, Kubeflow Pipelines can authenticate to GCP and manage resources in your project without further configuration.

Use one of the following options to create a GKE cluster that uses the Compute Engine default service account:

If you followed instructions in Setting up AI Platform Pipelines and checked Allow access to the following Cloud APIs, your cluster is already using Compute Engine default service account.
In Google Cloud Console UI, you can enable it in Create a Kubernetes cluster -> default-pool -> Security -> Accesss Scopes -> Allow full access to all Cloud APIs like the following:
Using gcloud CLI, you can enable it with --scopes cloud-platform like the following:

gcloud container clusters create cluster-name \
  --scopes cloud-platform

Please refer to gcloud container clusters create command documentation for other available options.

Authoring pipelines to use default service account

Pipelines don’t need any specific changes to authenticate to GCP, it will use the default service account transparently.

However, you must update existing pipelines that use the use_gcp_secret kfp sdk operator. Remove the use_gcp_secret usage to let your pipeline authenticate to Google Cloud using the default service account.

Securing the cluster with fine-grained GCP permission control

Workload Identity

Workload Identity is the recommended way for your GKE applications to consume services provided by Google APIs. You accomplish this by configuring a Kubernetes service account to act as a Google service account. Any Pods running as the Kubernetes service account then use the Google service account to authenticate to cloud services.

Referenced from Workload Identity Documentation. Please read this doc for:

A detailed introduction to Workload Identity.
Instructions to enable it on your cluster.
Whether its limitations affect your adoption.

Terminology

This document distinguishes between Kubernetes service accounts (KSAs) and Google service accounts (GSAs). KSAs are Kubernetes resources, while GSAs are specific to Google Cloud. Other documentation usually refers to both of them as just “service accounts”.

Authoring pipelines to use Workload Identity

Pipelines don’t need any specific changes to authenticate to Google Cloud. With Workload Identity, pipelines run as the Google service account that is bound to the KSA.

However, existing pipelines that use use_gcp_secret kfp sdk operator need to remove the use_gcp_secret usage to use the bound GSA. You can also continue to use use_gcp_secret in a cluster with Workload Identity enabled and use_gcp_secret will take precedence for those workloads.

Cluster setup to use Workload Identity for Pipelines Standalone

1. Create your cluster with Workload Identity enabled

In Google Cloud Console UI, you can enable Workload Identity in Create a Kubernetes cluster -> Security -> Enable Workload Identity like the following:
Using gcloud CLI, you can enable it with:

gcloud beta container clusters create cluster-name \
  --release-channel regular \
  --workload-pool=project-id.svc.id.goog

References:

2. Deploy Kubeflow Pipelines

Deploy via Pipelines Standalone as usual.

3. Bind Workload Identities for KSAs used by Kubeflow Pipelines

The following helper bash scripts bind Workload Identities for KSAs used by Kubeflow Pipelines:

gcp-workload-identity-setup.sh helps you create GSAs and bind them to KSAs used by pipelines workloads. This script provides an interactive command line dialog with explanation messages.
wi-utils.sh alternatively provides minimal utility bash functions that let you customize your setup. The minimal utilities make it easy to read and use programmatically.

For example, to get a default setup using gcp-workload-identity-setup.sh, you can

$ curl -O https://raw.githubusercontent.com/kubeflow/pipelines/master/manifests/kustomize/gcp-workload-identity-setup.sh
$ chmod +x ./gcp-workload-identity-setup.sh
$ ./gcp-workload-identity-setup.sh
# This prints the command's usage example and introduction.
# Then you can run the command with required parameters.
# Command output will tell you which GSAs and Workload Identity bindings have been
# created.

4. Configure IAM permissions of used GSAs

If you used gcp-workload-identity-setup.sh to bind Workload Identities for your cluster, you can simply add the following IAM bindings:

Give GSA <cluster-name>-kfp-system@<project-id>.iam.gserviceaccount.com Storage Object Viewer role to let UI load data in GCS in the same project.
Give GSA <cluster-name>-kfp-user@<project-id>.iam.gserviceaccount.com any permissions your pipelines need. For quick tryouts, you can give it Project Editor role for all permissions.

If you configured bindings by yourself, here are GCP permission requirements for KFP KSAs:

Pipelines use pipeline-runner KSA. Configure IAM permissions of the GSA bound to this KSA to allow pipelines use GCP APIs.
Pipelines UI uses ml-pipeline-ui KSA. Pipelines Visualization Server uses ml-pipeline-visualizationserver KSA. If you need to view artifacts and visualizations stored in Google Cloud Storage (GCS) from pipelines UI, you should add Storage Object Viewer permission (or the minimal required permission) to their bound GSAs.

Cluster setup to use Workload Identity for Full Kubeflow

Public Kubeflow v1.0.1 release hasn’t been configured out of box. The fix has been merged, but not yet released publically. After the fix is released, if you deployed Kubeflow following the GCP instructions Workload Identity will have already been configured properly for Kubeflow Pipelines.

If you want to use Workload Identity with pipelines on Kubeflow v1.0.1 or before, I recommend running the following commands to patch your deployment:

export NAMESPACE=kubeflow # Replace with your kubeflow's namespace if it's been customized.
kubectl patch deployment -n ${NAMESPACE} ml-pipeline --patch '{"spec": {"template": {"spec": {"containers": [{"name": "ml-pipeline-api-server", "env": [{"name": "DEFAULTPIPELINERUNNERSERVICEACCOUNT", "value": "kf-user"}]}]}}}}'
kubectl patch clusterrolebinding -n ${NAMESPACE} pipeline-runner --patch '{"subjects": [{"kind": "ServiceAccount", "name": "kf-user", "namespace": "'$NAMESPACE'"}]}'

Pipelines use kf-user KSA by default which is different from Kubeflow Standalone.

Google service account keys stored as Kubernetes secrets

It is recommended to use Workload Identity for easier and secure management, but you can also choose to use GSA keys.

Authoring pipelines to use GSA keys

Each pipeline step describes a container that is run independently. If you want to grant access for a single step to use one of your service accounts, you can use kfp.gcp.use_gcp_secret(). Examples for how to use this function can be found in the Kubeflow examples repo.

Cluster setup to use use_gcp_secret for Full Kubeflow

You don’t need to do anything. Full Kubeflow deployment has already deployed the user-gcp-sa secret for you.

Cluster setup to use use_gcp_secret for Pipelines Standalone

Pipelines Standalone require your manual setup for the user-gcp-sa secret used by use_gcp_secret.

Instructions to set up the secret:

First download the GCE VM service account token (refer to GCP documentation for more information):

gcloud iam service-accounts keys create application_default_credentials.json \
  --iam-account [SA-NAME]@[PROJECT-ID].iam.gserviceaccount.com

Run:

kubectl create secret -n [your-namespace] generic user-gcp-sa \
  --from-file=user-gcp-sa.json=application_default_credentials.json

Docs: Upgrading and Reinstalling

Mon, 01 Jan 0001 00:00:00 +0000

Starting from Kubeflow v0.5, Kubeflow Pipelines persists the pipeline data in permanent storage volumes. Kubeflow Pipelines therefore supports the following capabilities:

Reinstall: You can delete a cluster and create a new cluster, specifying the existing storage volumes to retrieve the original data in the new cluster. This guide tells you how to reinstall Kubeflow Pipelines as part of a full Kubeflow deployment.
Upgrade (limited support):

The full Kubeflow deployment currently supports upgrading in Alpha status with limited support. Check the following sources for progress updates:
- Issue kubeflow/kubeflow #3727.
- Kubeflow upgrade guide.

Before you start

This guide tells you how to reinstall Kubeflow Pipelines as part of a full Kubeflow deployment on Google Kubernetes Engine (GKE). See the Kubeflow deployment guide.

Instead of the full Kubeflow deployment, you can use Kubeflow Pipelines Standalone or GCP Hosted ML Pipelines (Alpha), which support different options for upgrading and reinstalling. See the Kubeflow Pipelines installation options.

Kubeflow Pipelines data storage

Kubeflow Pipelines creates and manages the following data related to your machine learning pipeline:

Metadata: Experiments, jobs, runs, etc. Kubeflow Pipelines stores the pipeline metadata in a MySQL database.
Artifacts: Pipeline packages, metrics, views, etc. Kubeflow Pipelines stores the artifacts in a Minio server.

Kubeflow Pipelines uses the Kubernetes PersistentVolume (PV) subsystem to provision the MySQL database and the Minio server. On GCP, Kubeflow Pipelines creates a Google Compute Engine Persistent Disk (PD) and mounts it as a PV.

After deploying Kubeflow on GCP, you can see two entries in the GCP Deployment Manager, one for the cluster deployment and one for the storage deployment:

The entry with the suffix -storage creates one PD for the metadata store and one for the artifact store:

Reinstalling Kubeflow Pipelines

You can delete a Kubeflow cluster and create a new one, specifying your existing storage to retrieve the original data in the new cluster.

Notes:

You must use command-line deployment. You cannot reinstall Kubeflow Pipelines using the web interface.
When you do kfctl apply or kfctl build, you should use a different deployment name from your existing deployment name. Otherwise, kfctl will delete your data in the existing PDs. This guide defines the deployment name in the ${KF_NAME} environment variable.

To reinstall Kubeflow Pipelines:

Follow the command-line deployment instructions, but note the following changes in the procedure.
Set a different ${KF_NAME} name from your existing ${KF_NAME}.
Before running the kfctl apply command:
- Edit ${KF_DIR}/gcp_config/storage-kubeflow.yaml and set the following flag to skip creating new storage:
```
...
createPipelinePersistentStorage: false
...
```
- Edit ${KF_DIR}/kustomize/minio/overlays/minioPd/params.env and specify the PD that your existing deployment uses for the Minio server:
```
...
minioPd=[NAME-OF-ARTIFACT-STORAGE-DISK]
...
```
- Edit ${KF_DIR}/kustomize/mysql/overlays/mysqlPd/params.env and specify the PD that your existing deployment uses for the MySQL database:
```
...
mysqlPd=[NAME-OF-METADATA-STORAGE-DISK]
...
```
Run the kfctl apply command to deploy Kubeflow as usual:
```
kfctl apply -V -f ${CONFIG_FILE}
```

You should now have a new Kubeflow deployment that uses the same pipelines data storage as your previous deployment. Follow the steps in the deployment guide to check your deployment.

Docs: Enabling GPU and TPU

Mon, 01 Jan 0001 00:00:00 +0000

This page describes how to enable GPU or TPU for a pipeline on GKE by using the Pipelines DSL language.

Prerequisites

To enable GPU and TPU on your Kubeflow cluster, follow the instructions on how to customize the GKE cluster for Kubeflow before setting up the cluster.

Configure ContainerOp to consume GPUs

After enabling the GPU, the Kubeflow setup script installs a default GPU pool with type nvidia-tesla-k80 with auto-scaling enabled. The following code consumes 2 GPUs in a ContainerOp.

import kfp.dsl as dsl
gpu_op = dsl.ContainerOp(name='gpu-op', ...).set_gpu_limit(2)

The code above will be compiled into Kubernetes Pod spec:

container:
  ...
  resources:
    limits:
      nvidia.com/gpu: "2"

If the cluster has multiple node pools with different GPU types, you can specify the GPU type by the following code.

import kfp.dsl as dsl
gpu_op = dsl.ContainerOp(name='gpu-op', ...).set_gpu_limit(2)
gpu_op.add_node_selector_constraint('cloud.google.com/gke-accelerator', 'nvidia-tesla-p4')

The code above will be compiled into Kubernetes Pod spec:

container:
  ...
  resources:
    limits:
      nvidia.com/gpu: "2"
nodeSelector:
  cloud.google.com/gke-accelerator: nvidia-tesla-p4

Check the GKE GPU guide to learn more about GPU settings.

Configure ContainerOp to consume TPUs

Use the following code to configure ContainerOp to consume TPUs on GKE:

import kfp.dsl as dsl
import kfp.gcp as gcp
tpu_op = dsl.ContainerOp(name='tpu-op', ...).apply(gcp.use_tpu(
  tpu_cores = 8, tpu_resource = 'v2', tf_version = '1.12'))

The above code uses 8 v2 TPUs with TF version to be 1.12. The code above will be compiled into Kubernetes Pod spec:

container:
  ...
  resources:
    limits:
      cloud-tpus.google.com/v2: "8"
  metadata:
    annotations:
      tf-version.cloud-tpus.google.com: "1.12"

See the GKE TPU Guide to learn more about TPU settings.

Docs: Using Preemptible VMs and GPUs on GCP

Mon, 01 Jan 0001 00:00:00 +0000

This document describes how to configure preemptible virtual machines (preemptible VMs) and GPUs on preemptible VM instances (preemptible GPUs) for your workflows running on Kubeflow Pipelines on Google Cloud Platform (GCP).

Introduction

Preemptible VMs are Compute Engine VM instances that last a maximum of 24 hours and provide no availability guarantees. The pricing of preemptible VMs is lower than that of standard Compute Engine VMs.

GPUs attached to preemptible instances (preemptible GPUs) work like normal GPUs but persist only for the life of the instance.

Using preemptible VMs and GPUs can reduce costs on GCP. In addition to using preemptible VMs, your Google Kubernetes Engine (GKE) cluster can autoscale based on current workloads.

This guide assumes that you have already deployed Kubeflow Pipelines. If not, follow the guide to deploying Kubeflow on GCP.

Using preemptible VMs with Kubeflow Pipelines

In summary, the steps to schedule a pipeline to run on preemptible VMs are as follows:

Create a node pool in your cluster that contains preemptible VMs.
Configure your pipelines to run on the preemptible VMs.

The following sections contain more detail about the above steps.

1. Create a node pool with preemptible VMs

Use the gcloud command to create a node pool. The following example includes placeholders to illustrate the important configurations:

gcloud container node-pools create PREEMPTIBLE_CPU_POOL \
	--cluster=CLUSTER_NAME \
    --enable-autoscaling --max-nodes=MAX_NODES --min-nodes=MIN_NODES \
    --preemptible \
    --node-taints=preemptible=true:NoSchedule \
    --service-account=DEPLOYMENT_NAME-vm@PROJECT_NAME.iam.gserviceaccount.com

Where:

PREEMPTIBLE_CPU_POOL is the name of the node pool.
CLUSTER_NAME is the name of the GKE cluster.
MAX_NODES and MIN_NODES are the maximum and minimum number of nodes for the GKE autoscaling functionality.
DEPLOYMENT_NAME is the name of your Kubeflow deployment. If you used the CLI to deploy Kubeflow, this name is the value of the ${KF_NAME} environment variable. If you used the deployment UI, this name is the value you specified as the deployment name.
PROJECT_NAME is the name of your GCP project.

Below is an example of the command:

gcloud container node-pools create preemptible-cpu-pool \
	--cluster=user-4-18 \
    --enable-autoscaling --max-nodes=4 --min-nodes=0 \
    --preemptible \
    --node-taints=preemptible=true:NoSchedule \
    --service-account=user-4-18-vm@ml-pipeline-project.iam.gserviceaccount.com

2. Schedule your pipeline to run on the preemptible VMs

After configuring a node pool with preemptible VMs, you must configure your pipelines to run on the preemptible VMs.

In the DSL code for your pipeline, add the following to the ContainerOp instance:

.apply(gcp.use_preemptible_nodepool())

The above function works for both methods of generating the ContainerOp:

The ContainerOp generated from kfp.components.func_to_container_op.
The ContainerOp generated from the task factory function, which is loaded by components.load_component_from_url.

Note:

Call .set_retry(#NUM_RETRY) on your ContainerOp to retry the task after the task is preempted.
If you modified the node taint when creating the node pool, pass the same node toleration to the use_preemptible_nodepool() function.
use_preemptible_nodepool() also accepts a parameter hard_constraint. When the hard_constraint is True, the system will strictly schedule the task in preemptible VMs. When the hard_constraint is False, the system will try to schedule the task in preemptible VMs. If it cannot find the preemptible VMs, or the preemptible VMs are busy, the system will schedule the task in normal VMs.

For example:

import kfp.dsl as dsl
import kfp.gcp as gcp

class FlipCoinOp(dsl.ContainerOp):
  """Flip a coin and output heads or tails randomly."""

  def __init__(self):
    super(FlipCoinOp, self).__init__(
      name='Flip',
      image='python:alpine3.6',
      command=['sh', '-c'],
      arguments=['python -c "import random; result = \'heads\' if random.randint(0,1) == 0 '
                 'else \'tails\'; print(result)" | tee /tmp/output'],
      file_outputs={'output': '/tmp/output'})

@dsl.pipeline(
  name='pipeline flip coin',
  description='shows how to use dsl.Condition.'
)

def flipcoin():
  flip = FlipCoinOp().apply(gcp.use_preemptible_nodepool())

if __name__ == '__main__':
  import kfp.compiler as compiler
  compiler.Compiler().compile(flipcoin, __file__ + '.zip')

Using preemptible GPUs with Kubeflow Pipelines

This guide assumes that you have already deployed Kubeflow Pipelines. In summary, the steps to schedule a pipeline to run with preemptible GPUs are as follows:

Make sure you have enough GPU quota.
Create a node pool in your GKE cluster that contains preemptible VMs with preemptible GPUs.
Configure your pipelines to run on the preemptible VMs with preemptible GPUs.

The following sections contain more detail about the above steps.

1. Make sure you have enough GPU quota

Add GPU quota to your GCP project. The GCP documentation lists the availability of GPUs across regions. To check the available quota for resources in your project, go to the Quotas page in the GCP Console.

2. Create a node pool of preemptible VMs with preemptible GPUs

Use the gcloud command to create a node pool. The following example includes placeholders to illustrate the important configurations:

gcloud container node-pools create PREEMPTIBLE_GPU_POOL \
    --cluster=CLUSTER_NAME \
    --enable-autoscaling --max-nodes=MAX_NODES --min-nodes=MIN_NODES \
    --preemptible \
    --node-taints=preemptible=true:NoSchedule \
    --service-account=DEPLOYMENT_NAME-vm@PROJECT_NAME.iam.gserviceaccount.com \
    --accelerator=type=GPU_TYPE,count=GPU_COUNT

Where:

PREEMPTIBLE_GPU_POOL is the name of the node pool.
CLUSTER_NAME is the name of the GKE cluster.
MAX_NODES and MIN_NODES are the maximum and minimum number of nodes for the GKE autoscaling functionality.
DEPLOYMENT_NAME is the name of your Kubeflow deployment. If you used the CLI to deploy Kubeflow, this name is the value of the ${KF_NAME} environment variable. If you used the deployment UI, this name is the value you specified as the deployment name.
PROJECT_NAME is the name of your GCP project.
GPU_TYPE is the type of GPU.
GPU_COUNT is the number of GPUs.

Below is an example of the command:

gcloud container node-pools create preemptible-gpu-pool \
    --cluster=user-4-18 \
    --enable-autoscaling --max-nodes=4 --min-nodes=0 \
    --preemptible \
    --node-taints=preemptible=true:NoSchedule \
    --service-account=user-4-18-vm@ml-pipeline-project.iam.gserviceaccount.com \
    --accelerator=type=nvidia-tesla-t4,count=2

3. Schedule your pipeline to run on the preemptible VMs with preemptible GPUs

In the DSL code for your pipeline, add the following to the ContainerOp instance:

.apply(gcp.use_preemptible_nodepool()

The above function works for both methods of generating the ContainerOp:

The ContainerOp generated from kfp.components.func_to_container_op.
The ContainerOp generated from the task factory function, which is loaded by components.load_component_from_url.

Note:

Call .set_gpu_limit(#NUM_GPUs, GPU_VENDOR) on your ContainerOp to specify the GPU limit (for example, 1) and vendor (for example, 'nvidia').
Call .set_retry(#NUM_RETRY) on your ContainerOp to retry the task after the task is preempted.
If you modified the node taint when creating the node pool, pass the same node toleration to the use_preemptible_nodepool() function.
use_preemptible_nodepool() also accepts a parameter hard_constraint. When the hard_constraint is True, the system will strictly schedule the task in preemptible VMs. When the hard_constraint is False, the system will try to schedule the task in preemptible VMs. If it cannot find the preemptible VMs, or the preemptible VMs are busy, the system will schedule the task in normal VMs.

For example:

import kfp.dsl as dsl
import kfp.gcp as gcp

class FlipCoinOp(dsl.ContainerOp):
  """Flip a coin and output heads or tails randomly."""

  def __init__(self):
    super(FlipCoinOp, self).__init__(
      name='Flip',
      image='python:alpine3.6',
      command=['sh', '-c'],
      arguments=['python -c "import random; result = \'heads\' if random.randint(0,1) == 0 '
                 'else \'tails\'; print(result)" | tee /tmp/output'],
      file_outputs={'output': '/tmp/output'})

@dsl.pipeline(
  name='pipeline flip coin',
  description='shows how to use dsl.Condition.'
)

def flipcoin():
  flip = FlipCoinOp().set_gpu_limit(1, 'nvidia').apply(gcp.use_preemptible_nodepool())
if __name__ == '__main__':
  import kfp.compiler as compiler
  compiler.Compiler().compile(flipcoin, __file__ + '.zip')

Comparison with Cloud AI Platform Training service

Cloud AI Platform Training is a GCP machine learning (ML) training service that supports distributed training and hyperparameter tuning, and requires no complex GKE configuration. Cloud AI Platform Training charges the Compute Engine costs only for the runtime of the job.

The table below compares Cloud AI Platform Training with Kubeflow Pipelines running preemptible VMs or GPUs:

	Cloud AI Platform Training	Kubeflow Pipelines with preemption
Configuration	No GKE configuration	Requires GKE configuration
Cost	Compute Engine costs for the job lifetime	Lower price with preemptible VMs/GPUs/TPUs
Accelerator	Supports various VM types, GPUs, and CPUs	Support various VM types, GPUs, and CPUs
Scalability	Automates resource provisioning and supports distributed training	Requires manual configuration such as GKE autoscaler and distributed training workflow
Features	Out-of-box support for hyperparameter tuning	Do-it-yourself hyperparameter tuning with Katib

Next steps

Explore further options for customizing Kubeflow on GCP.
See how to build pipelines with the SDK.