Kubeflow – Documentation

Docs: Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

The Kubeflow project is dedicated to making deployments of machine learning (ML) workflows on Kubernetes simple, portable and scalable. Our goal is not to recreate other services, but to provide a straightforward way to deploy best-of-breed open-source systems for ML to diverse infrastructures. Anywhere you are running Kubernetes, you should be able to run Kubeflow.

Getting started with Kubeflow

Read the Kubeflow overview for an introduction to the Kubeflow architecture and to see how you can use Kubeflow to manage your ML workflow.

Follow the getting-started guide to set up your environment and install Kubeflow.

Watch the following video which provides an introduction to Kubeflow.

What is Kubeflow?

Kubeflow is the machine learning toolkit for Kubernetes. Learn about Kubeflow use cases.

To use Kubeflow, the basic workflow is:

Download and run the Kubeflow deployment binary.
Customize the resulting configuration files.
Run the specified script to deploy your containers to your specific environment.

You can adapt the configuration to choose the platforms and services that you want to use for each stage of the ML workflow: data preparation, model training, prediction serving, and service management.

You can choose to deploy your Kubernetes workloads locally, on-premises, or to a cloud environment.

Read the Kubeflow overview for more details.

The Kubeflow mission

Our goal is to make scaling machine learning (ML) models and deploying them to production as simple as possible, by letting Kubernetes do what it’s great at:

Easy, repeatable, portable deployments on a diverse infrastructure (for example, experimenting on a laptop, then moving to an on-premises cluster or to the cloud)
Deploying and managing loosely-coupled microservices
Scaling based on demand

Because ML practitioners use a diverse set of tools, one of the key goals is to customize the stack based on user requirements (within reason) and let the system take care of the “boring stuff”. While we have started with a narrow set of technologies, we are working with many different projects to include additional tooling.

Ultimately, we want to have a set of simple manifests that give you an easy to use ML stack anywhere Kubernetes is already running, and that can self configure based on the cluster it deploys into.

History

Kubeflow started as an open sourcing of the way Google ran TensorFlow internally, based on a pipeline called TensorFlow Extended. It began as just a simpler way to run TensorFlow jobs on Kubernetes, but has since expanded to be a multi-architecture, multi-cloud framework for running entire machine learning pipelines.

Roadmaps

To see what’s coming up in future versions of Kubeflow, refer to the Kubeflow roadmap.

The following components also have roadmaps:

Getting involved

There are many ways to contribute to Kubeflow, and we welcome contributions! Read the contributor’s guide to get started on the code, and get to know the community in the community guide.

Docs: Overview

Mon, 01 Jan 0001 00:00:00 +0000

Kubeflow supports two model serving systems that allow multi-framework model serving: KFServing and Seldon Core. Alternatively, you can use a standalone model serving system. This page gives an overview of the options, so that you can choose the framework that best supports your model serving requirements.

Multi-framework serving with KFServing or Seldon Core

KFServing and Seldon Core are both open source systems that allow multi-framework model serving. The following table compares KFServing and Seldon Core. A check mark (✓) indicates that the system (KFServing or Seldon Core) supports the feature specified in that row.

Feature	Sub-feature	KFServing	Seldon Core
Framework	TensorFlow	✓ sample	✓ docs
	XGBoost	✓ sample	✓ docs
	scikit-learn	✓ sample	✓ docs
	NVIDIA Triton Inference Server	✓ sample	✓ docs
	ONNX	✓ sample
	PyTorch	✓ sample	✓
Graph	Transformers	✓ sample	✓ docs
	Combiners	Roadmap	✓ sample
	Routers including MAB	Roadmap	✓ docs
Analytics	Explanations	✓ sample	✓ docs
Scaling	Knative	✓ sample
	GPU AutoScaling	✓ sample
	HPA	✓	✓ docs
Custom	Container	✓ sample	✓ docs
	Language Wrappers		✓ Python, Java, R
	Multi-Container		✓ docs
Rollout	Canary	✓ sample	✓ docs
	Shadow		✓
Istio		✓	✓

Notes:

KFServing and Seldon Core share some technical features, including explainability (using Seldon Alibi Explain) and payload logging, as well as other areas.
A commercial product, Seldon Deploy, supports both KFServing and Seldon in production.
KFServing is part of the Kubeflow project ecosystem. Seldon Core is an external project supported within Kubeflow.

Further information:

KFServing:
Seldon Core

TensorFlow Serving

For TensorFlow models you can use TensorFlow Serving for real-time prediction. However, if you plan to use multiple frameworks, you should consider KFServing or Seldon Core as described above.

NVIDIA Triton Inference Server

NVIDIA Triton Inference Server is a REST and GRPC service for deep-learning inferencing of TensorRT, TensorFlow, Pytorch, ONNX and Caffe2 models. The server is optimized to deploy machine learning algorithms on both GPUs and CPUs at scale. Triton Inference Server was previously known as TensorRT Inference Server.

You can use NVIDIA Triton Inference Server as a standalone system, but you should consider KFServing as described above. KFServing includes support for NVIDIA Triton Inference Server.

BentoML

BentoML is an open-source platform for high-performance ML model serving. It makes building production API endpoint for your ML model easy and supports all major machine learning training frameworks, including Tensorflow, Keras, PyTorch, XGBoost, scikit-learn and etc.

BentoML comes with a high-performance API model server with adaptive micro-batching support, which achieves the advantage of batch processing in online serving. It also provides model management and model deployment functionality, giving ML teams an end-to-end model serving workflow, with DevOps best practices baked in.

Docs: Overview of Deployment on Existing Clusters

Mon, 01 Jan 0001 00:00:00 +0000

Follow these instructions if you want to install Kubeflow on an existing Kubernetes cluster. Some clouds and Kubernetes distributions provide Kubeflow specific instructions for getting the most out of their Kubernetes. If your existing Kubernetes cluster is from one of those, consider following those instructions.

Minimum system requirements

The Kubernetes cluster must meet the following minimum requirements:

Your cluster must include at least one worker node with a minimum of:
- 4 CPU
- 50 GB storage
- 12 GB memory

The recommended Kubernetes version is 1.14. Kubeflow has been validated and tested on Kubernetes 1.14.
- Your cluster must run at least Kubernetes version 1.11.
- Kubeflow does not work on Kubernetes 1.16.
- Older versions of Kubernetes may not be compatible with the latest Kubeflow versions. The following matrix provides information about compatibility between Kubeflow and Kubernetes versions.

Kubernetes Versions	Kubeflow 0.4	Kubeflow 0.5	Kubeflow 0.6	Kubeflow 0.7	Kubeflow 1.0
1.11	compatible	compatible	incompatible	incompatible	incompatible
1.12	compatible	compatible	incompatible	incompatible	incompatible
1.13	compatible	compatible	incompatible	incompatible	incompatible
1.14	compatible	compatible	compatible	compatible	compatible
1.15	incompatible	compatible	compatible	compatible	compatible
1.16	incompatible	incompatible	incompatible	incompatible	no known issues
1.17	incompatible	incompatible	incompatible	incompatible	no known issues
1.18	incompatible	incompatible	incompatible	incompatible	no known issues

incompatible: the combination does not work at all
compatible: all Kubeflow features have been tested and verified for the Kubernetes version
no known issues: the combination has not been fully tested but there are no repoted issues

Kubeflow Deployment Configurations

The following table lists the options for installing Kubeflow on an existing Kubernetes cluster and links to detailed instructions. These solutions are vendor neutral and are governed by consensus within the Kubeflow community.

Deployment config	Description
kfctl_k8s_istio.yaml	This config creates a vanilla deployment of Kubeflow with all its core components without any external dependencies. The deployment can be customized based on your environment needs. Follow instructions: Kubeflow Deployment with kfctl_k8s_istio
kfctl_istio_dex.yaml	This config creates a Kubeflow deployment with all its core components, and uses Dex and Istio for vendor-neutral authentication. Follow instructions: Multi-user, auth-enabled Kubeflow with kfctl_istio_dex

Docs: Reference Overview

Mon, 01 Jan 0001 00:00:00 +0000

TFJob

TFJob is a Kubernetes custom resource that you can use to run TensorFlow training jobs on Kubernetes. For help with using TFJob with Kubeflow, see the user guide.

API references:

v1
v1beta2

PyTorchJob

PyTorchJob is a Kubernetes custom resource that you can use to run PyTorch training jobs on Kubernetes. For help with using PyTorch with Kubeflow, see the user guide.

API references:

v1
v1beta2

MPIJob

MPIJob is a Kubernetes custom resource that you can use to run allreduce-style distributed training jobs on Kubernetes. For help with using MPIJob with Kubeflow, see the user guide.

API references:

v1alpha2

Notebook

Notebook CRD is a Kubernetes custom resource that you can use to manage Jupyter Notebook servers on Kubernetes. For help with using notebooks with Kubeflow, see the user guide.

API references:

Docs: Set up a GCP Project

Mon, 01 Jan 0001 00:00:00 +0000

Follow these steps to set up your GCP project:

Select or create a project on the GCP Console.
Make sure that you have the owner role for the project. The deployment process creates various service accounts with appropriate roles in order to enable seamless integration with GCP services. This process requires that you have the owner role for the project in order to deploy Kubeflow.
Make sure that billing is enabled for your project. See the guide to modifying a project’s billing settings.
Go to the following pages on the GCP Console and ensure that the specified APIs are enabled:
- Compute Engine API
- Kubernetes Engine API
- Identity and Access Management (IAM) API
- Deployment Manager API
- Cloud Resource Manager API
- Cloud Filestore API
- AI Platform Training & Prediction API
- Cloud Build API (It’s required if you plan to use Fairing in your Kubeflow cluster)
If you are using the GCP Free Tier or the 12-month trial period with $300 credit, note that you can’t run the default GCP installation of Kubeflow, because the free tier does not offer enough resources. You need to upgrade to a paid account.

For more information, see the following issues:
- kubeflow/website #1065 reports the problem.
- kubeflow/kubeflow #3936 requests a Kubeflow configuration to work with a free trial project.
Read the GCP guide to resource quotas to understand the quotas on resource usage that Compute Engine enforces, and to learn how to check your quota and how to request an increase in quota.

You do not need a running GKE cluster. The deployment process creates a cluster for you.

Next steps

Set up an OAuth credential if you want to use Cloud Identity-Aware Proxy (Cloud IAP). Cloud IAP is recommended for production deployments or deployments with access to sensitive data. You can skip this step if you want to test Kubeflow in a non-production environment.
Choose one of the following ways to deploy Kubeflow:
- Using the UI. This option provides a simple way to deploy Kubeflow.
- Using the CLI. This option provides more control over the deployment process.

Docs: Use Cases

Mon, 01 Jan 0001 00:00:00 +0000

The end goal of every organization is to have their machine learning (ML) model successfully running in production and generating value to the business. But what does it take to reach that point? Before a model ends up in production, there are potentially many steps required to build and deploy an ML model: data loading, verification, splitting, processing, feature engineering, model training and verification, hyperparameter tuning, and model serving. In addition, ML models can require more observation than traditional applications, because your data inputs can drift over time. Manually rebuilding models and data sets is time consuming and error prone. To simplify these requirements and challenges, we introduce Kubeflow.

Deploying and managing a complex ML system at scale

Kubeflow is a scalable, portable, distributed ML platform that runs on Kubernetes, meaning that all capabilities of Kubernetes are available to a Kubeflow user. With Kubeflow you can manage the entire AI organization at scale and still be able to maintain the same quality of control. Kubeflow and Kubernetes provide consistent and efficient operations and optimized infrastructure. This means that your top researchers have more time to focus on the valuable tasks of developing domain specific intellectual property rather than debugging DevOps configuration issues. Kubeflow’s core and ecosystem critical user journeys (CUJs) provide software solutions for end-to-end workflows i.e. build, train and deploy and/or develop a model and create, run and explore a pipeline.

Experimentation with training an ML model

Rapid experimentation is critical to building high quality machine learning models quickly. Kubeflow offers a user-friendly interface (UI) that allows you to track and compare experiments. You can decide later on which experiment was the best and use it as a main source for your future steps. On top of that Kubeflow 1.0 provides stable software sub-systems for model training including Jupyter notebooks, popular ML training operators such as Tensorflow and Pytorch that run efficiently and securely in Kubernetes isolated namespaces. The ML training operators simplify configuration and operations of scaling ML training tasks. In addition, Kubeflow has delivered Critical User Journeys(CUJs), such as the build, train and deploy, which provide end-to-end workflows that speed development. You can read more about the CUJs in the Kubeflow roadmap.

End to end hybrid and multi-cloud ML workloads

The development of ML models can require hybrid and multi-cloud portability and secure sharing between teams, clusters and clouds. Kubeflow is supported by all major cloud providers and available for on-premises installation. If you need to develop on your laptop, train with GPU on your on-prem cluster and serve in the cloud, Kubeflow provides the portability to support fast experimentation, rapid training and robust deployment in the same or different environments with minimal operational overhead.

Tuning the model hyperparameters during training

During the model development part hyperparameters are often hard to tune. Tuning hyperparameters is critical for model performance and accuracy. Manually configuring hyperparameters is time consuming. Kubeflow’s hyperparameter tuner (Katib) provides an automated way to match your objectives. This automation can save days of model testing compute time (freeing up valuable infrastructure), and speed the delivery of improved models.

Continuous integration and deployment (CI/CD) for ML

Kubeflow currently doesn’t have a dedicated tool for this purpose. But our users have been using the Pipelines component and it worked really well for them. Kubeflow Pipelines can be used to create reproducible workflows. These workflows automate the steps needed to build a ML workflow, which delivers consistency, saves iteration time, and helps in debugging, auditability and compliance requirements.

Next steps

See these docs for more information on the topics covered above:

Docs: AWS for Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

Please refer to the deployment section in the AWS docs for information on setting up your AWS environment and deploying Kubeflow on Amazon Elastic Kubernetes Service (Amazon EKS).

Docs: Azure for Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

Please refer to the deployment section in the Azure docs for information on setting up your Azure environment and deploying Kubeflow on an existing Azure Resource Group or Cluster for AKS (Azure Kubernetes Service).

Docs: Google Cloud for Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

For details on setting up your GCP environment and deploying Kubeflow on Kubernetes Engine (GKE), refer to the deployment section of the Kubeflow GCP documentation.

Docs: IBM Cloud for Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

Please refer to the installation section in the IBM docs for information on setting up your IKS environment and deploying Kubeflow on IBM Cloud Kubernetes Service (IKS).

Docs: KFServing

Mon, 01 Jan 0001 00:00:00 +0000

Beta

This Kubeflow component has beta status. See the Kubeflow versioning policies. The Kubeflow team is interested in your feedback about the usability of the feature.

KFServing enables serverless inferencing on Kubernetes and provides performant, high abstraction interfaces for common machine learning (ML) frameworks like TensorFlow, XGBoost, scikit-learn, PyTorch, and ONNX to solve production model serving use cases.

You can use KFServing to do the following:

Provide a Kubernetes Custom Resource Definition for serving ML models on arbitrary frameworks.
Encapsulate the complexity of autoscaling, networking, health checking, and server configuration to bring cutting edge serving features like GPU autoscaling, scale to zero, and canary rollouts to your ML deployments.
Enable a simple, pluggable, and complete story for your production ML inference server by providing prediction, pre-processing, post-processing and explainability out of the box.

Our strong community contributions help KFServing to grow. We have a Technical Steering Committee driven by Google, IBM, Microsoft, Seldon, and Bloomberg. Browse the KFServing GitHub repo to give us feedback!

Install with Kubeflow

KFServing works with Kubeflow 0.7. Kustomize installation files are located in the manifests repo.

Examples

Sample notebooks

We frequently add examples to our GitHub repo.

Learn more

Join our working group for meeting invitations and discussion.
Read the docs.
API docs.
Roadmap.
KFServing 101 slides.

Prerequisites

Knative Serving (v0.8.0 +) and Istio (v1.1.7+) should be available on your Kubernetes cluster.

KFServing installation using kubectl

The following commands install KFServing 0.2.2, using a yaml file in GitHub repo. See here for other available releases. Alternatively, you can clone the GitHub repo and run kubectl on top of it.

TAG=0.2.2
CONFIG_URI=https://raw.githubusercontent.com/kubeflow/kfserving/master/install/$TAG/kfserving.yaml
kubectl apply -f ${CONFIG_URI}

Use

Install the SDK.
```
pip install kfserving
```
Follow the example to use the KFServing SDK to create, patch, roll out, and delete a KFServing instance.

Contribute

Developer guide.

Docs: Kubeflow Deployment with kfctl_k8s_istio

Mon, 01 Jan 0001 00:00:00 +0000

This configuration creates a vanilla deployment of Kubeflow with all its core components without any external dependencies. The deployment can be customized based on your environment needs.

Before you start

This Kubeflow deployment requires a default StorageClass with a dynamic volume provisioner. Verify the provisioner field of your default StorageClass definition. If you don’t have a provisioner, ensure that you have configured volume provisioning in your Kubernetes cluster appropriately as mentioned below.

Configuring your installation with kfctl_k8s_istio.v1.0.2.yaml has an option you should consider:

Disabling istio installation - If your Kubernetes cluster has an existing Istio installation you may choose to not install Istio by removing the applications istio-crds and istio-install in the configuration file kfctl_k8s_istio.v1.0.2.yaml

Prepare your environment

Follow these steps to download the kfctl binary for the Kubeflow CLI and set some handy environment variables:

Download the kfctl v1.0.2 release from the Kubeflow releases page.

Unpack the tar ball:

tar -xvf kfctl_v1.0.2_<platform>.tar.gz

Create environment variables to make the deployment process easier:

# The following command is optional. It adds the kfctl binary to your path.
# If you don't add kfctl to your path, you must use the full path
# each time you run kfctl.
# Use only alphanumeric characters or - in the directory name.
export PATH=$PATH:"<path-to-kfctl>"

# Set KF_NAME to the name of your Kubeflow deployment. You also use this
# value as directory name when creating your configuration directory.
# For example, your deployment name can be 'my-kubeflow' or 'kf-test'.
export KF_NAME=<your choice of name for the Kubeflow deployment>

# Set the path to the base directory where you want to store one or more 
# Kubeflow deployments. For example, /opt/.
# Then set the Kubeflow application directory for this deployment.
export BASE_DIR=<path to a base directory>
export KF_DIR=${BASE_DIR}/${KF_NAME}

# Set the configuration file to use when deploying Kubeflow.
# The following configuration installs Istio by default. Comment out 
# the Istio components in the config file to skip Istio installation. 
# See https://github.com/kubeflow/kubeflow/pull/3663
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_k8s_istio.v1.0.2.yaml"

Notes:

${KF_NAME} - The name of your Kubeflow deployment. If you want a custom deployment name, specify that name here. For example, my-kubeflow or kf-test. The value of KF_NAME must consist of lower case alphanumeric characters or ‘-’, and must start and end with an alphanumeric character. The value of this variable cannot be greater than 25 characters. It must contain just a name, not a directory path. You also use this value as directory name when creating the directory where your Kubeflow configurations are stored, that is, the Kubeflow application directory.
${KF_DIR} - The full path to your Kubeflow application directory.
${CONFIG_URI} - The GitHub address of the configuration YAML file that you want to use to deploy Kubeflow. The URI used in this guide is https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_k8s_istio.v1.0.2.yaml. When you run kfctl apply or kfctl build (see the next step), kfctl creates a local version of the configuration YAML file which you can further customize if necessary.

Set up and deploy Kubeflow

To set up and deploy Kubeflow using the default settings, run the kfctl apply command:

mkdir -p ${KF_DIR}
cd ${KF_DIR}
kfctl apply -V -f ${CONFIG_URI}

Check the resources deployed in namespace kubeflow:

kubectl -n kubeflow get all

Alternatively, set up your configuration for later deployment

If you want to customize your configuration before deploying Kubeflow, you can set up your configuration files first, then edit the configuration, then deploy Kubeflow:

Run the kfctl build command to set up your configuration:

mkdir -p ${KF_DIR}
cd ${KF_DIR}
kfctl build -V -f ${CONFIG_URI}

Edit the configuration files, as described in the guide to customizing your Kubeflow deployment.
Set an environment variable pointing to your local configuration file:
```
export CONFIG_FILE=${KF_DIR}/kfctl_k8s_istio.v1.0.2.yaml
```
Run the kfctl apply command to deploy Kubeflow:
```
kfctl apply -V -f ${CONFIG_FILE}
```

Access the Kubeflow user interface (UI)

After Kubeflow is deployed, the Kubeflow Dashboard can be accessed via istio-ingressgateway service. If loadbalancer is not available in your environment, NodePort or Port forwarding can be used to access the Kubeflow Dashboard. Refer to Ingress Gateway guide.

Delete Kubeflow

Run the following commands to delete your deployment and reclaim all resources:

cd ${KF_DIR}
# If you want to delete all the resources, run:
kfctl delete -f ${CONFIG_FILE}

Understanding the deployment process

The kfctl deployment process includes the following commands:

kfctl build - (Optional) Creates configuration files defining the various resources in your deployment. You only need to run kfctl build if you want to edit the resources before running kfctl apply.
kfctl apply - Creates or updates the resources.
kfctl delete - Deletes the resources.

Application layout

Your Kubeflow application directory ${KF_DIR} contains the following files and directories:

${CONFIG_FILE} is a YAML file that defines configurations related to your Kubeflow deployment.
- This file is a copy of the GitHub-based configuration YAML file that you used when deploying Kubeflow: https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_k8s_istio.v1.0.2.yaml.
- When you run kfctl apply or kfctl build, kfctl creates a local version of the configuration file, ${CONFIG_FILE}, which you can further customize if necessary.
kustomize is a directory that contains the kustomize packages for Kubeflow applications. See how Kubeflow uses kustomize.
- The directory is created when you run kfctl build or kfctl apply.
- You can customize the Kubernetes resources by modifying the manifests and running kfctl apply again.

We recommend that you check in the contents of your ${KF_DIR} directory into source control.

Provisioning of Persistent Volumes in Kubernetes

Note that you can skip this step if you have a dynamic volume provisioner already installed in your cluster.

If you don’t have one:

You can choose to create PVs manually after deployment of Kubeflow.
Or install a dynamic volume provisioner like Local Path Provisioner. Ensure that the StorageClass used by this provisioner is the default StorageClass.

Troubleshooting

Persistent Volume Claims are in Pending State

Check if PersistentVolumeClaims get Bound to PersistentVolumes.

kubectl -n kubeflow get pvc

If the PersistentVolumeClaims (PVCs) are in Pending state after deployment and they are not bound to PersistentVolumes (PVs), you may have to either manually create PVs for each PVC in your Kubernetes Cluster or an alternative is to set up dynamic volume provisioning to create PVs on demand and redeploy Kubeflow after deleting existing PVCs.

Next steps

Run a sample machine learning workflow.
Get started with Kubeflow Pipelines.

Docs: Set up OAuth for Cloud IAP

Mon, 01 Jan 0001 00:00:00 +0000

If you want to use Cloud Identity-Aware Proxy (Cloud IAP) when deploying Kubeflow on GCP, then you must follow these instructions to create an OAuth client for use with Kubeflow.

You can skip the instructions on this page if you want to use basic authentication (username and password) with Kubeflow instead of Cloud IAP. Cloud IAP is recommended for production deployments or deployments with access to sensitive data.

Follow the steps below to create an OAuth client ID that identifies Cloud IAP when requesting access to a user’s email account. Kubeflow uses the email address to verify the user’s identity.

Set up your OAuth consent screen:
- In the Application name box, enter the name of your application. The example below uses the name “Kubeflow”.
- Under Support email, select the email address that you want to display as a public contact. You must use either your email address or a Google Group that you own.
- If you see Authorized domains, enter
```
<project>.cloud.goog
```
  - where <project> is your Google Cloud Platform (GCP) project ID.
  - If you are using your own domain, such as acme.com, you should add that as well
  - The Authorized domains option appears only for certain project configurations. If you don’t see the option, then there’s nothing you need to set.
- Click Save.
- Here’s an example of the completed form:
On the credentials screen:
- Click Create credentials, and then click OAuth client ID.
- Under Application type, select Web application.
- In the Name box enter any name for your OAuth client ID. This is not the name of your application nor the name of your Kubeflow deployment. It’s just a way to help you identify the OAuth client ID.
Click Create. A dialog box appears, like the one below:
Copy the client ID shown in the dialog box, because you need the client ID in the next step.
On the Create credentials screen, find your newly created OAuth credential and click the pencil icon to edit it:
In the Authorized redirect URIs box, enter the following (if it’s not already present in the list of authorized redirect URIs):
```
https://iap.googleapis.com/v1/oauth/clientIds/<CLIENT_ID>:handleRedirect
```
- <CLIENT_ID> is the OAuth client ID that you copied from the dialog box in step four. It looks like XXX.apps.googleusercontent.com.
- Note that the URI is not dependent on the Kubeflow deployment or endpoint. Multiple Kubeflow deployments can share the same OAuth client without the need to modify the redirect URIs.
Press Enter/Return to add the URI. Check that the URI now appears as a confirmed item under Authorized redirect URIs. (The URI should no longer be editable.)

Here’s an example of the completed form:
Click Save.
Make note that you can find your OAuth client credentials in the credentials section of the GCP Console. You need to retrieve the client ID and client secret later when you’re ready to enable Cloud IAP.

Next steps

Choose one of the following ways to deploy Kubeflow:

Using the UI. This option provides a simple way to deploy Kubeflow.
Using the CLI. This option provides more control over the deployment process.

Docs: Contributing to Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

Welcome to the Kubeflow project!

Getting started as a Kubeflow contributor

This document is the single source of truth for how to contribute to the code base. We’d love to accept your patches and contributions to this project. There are just a few small guidelines you need to follow.

Sign the CLA

Contributions to this project must be accompanied by a Contributor License Agreement (CLA). You (or your employer) retain the copyright to your contribution. This gives us permission to use and redistribute your contributions as part of the project. Head over to https://cla.developers.google.com/ to see your current agreements on file or to sign a new one.

You generally only need to submit a CLA once, so if you’ve already submitted one (even if it was for a different project), you probably don’t need to do it again.

Follow the code of conduct

Please make sure to read and observe our Code of Conduct and inclusivity document.

Consider participating in Kubeflow user research

Maggie Lynn, a user experience researcher, is conducting user studies to inform future developments for Kubeflow. These typically involve a one hour study session conducted online with a thank you gift for providing your feedback. As a member of the Kubeflow community, your feedback and expertise are extremely valuable to us, so if you have time in the next month, please consider participating. To gather your interest, availability, and some basic information about you, please fill out this form where you’ll find out more details about this research opportunity: https://goo.gl/forms/sv5sRo3UfsgeUEjK2

Joining the community

Follow these instructions if you want to

Become a member of the Kubeflow GitHub org (see below)
Become part of the Kubeflow build cop or release teams
Be recognized as an individual or organization contributing to Kubeflow

Joining the Kubeflow GitHub Org

Before asking to join the community, we ask that you first make a small number of contributions to demonstrate your intent to continue contributing to Kubeflow.

There are are a number of ways to contribute to Kubeflow

Submit PRs
File issues reporting bugs or providing feedback
Answer questions on Slack or GitHub issues

You can use this table to see how many contributions you’ve made

Note: This only counts GitHub related ways of contributing

When you are ready to join

Send a PR adding yourself as a member in org.yaml
After the PR is merged an admin will send you an invitation
- This is a manual process that’s generally run a couple times a week
- If a week passes without receiving an invitation reach out on kubeflow#community

Companies/organizations

If you would like your company or organization to be acknowledged for contributing to Kubeflow or participating in the community (being a user counts) please send a PR adding the relevant info to member_organizations.yaml.

If you want your employee’s GitHub contributions to be attributed to your company please ask them to set the company field in their GitHub profile.

Community discussions

There are many ways to contribute! Join one of our communication channels, attend a community meeting, get to know the community. Read the details in our community guide.

Your first contribution

Find something to work on

Help is always welcome! For example, documentation (like the text you are reading now) can always use improvement. There’s always code that can be clarified and variables or functions that can be renamed or commented. There’s always a need for more test coverage. You get the idea - if you ever see something you think should be fixed, you should own it. Here is how you get started.

Starter issues

To find Kubeflow issues that make good entry points:

Start with issues labeled good first issue. For example, see the good first issues in the kubeflow/website repository for doc updates, and in the kubeflow/kubeflow repository for updates to the core Kubeflow code.
For issues that require deeper knowledge of one or more technical aspects, look at issues labeled help wanted. For example, see these issues in the kubeflow/kubeflow repository
Examine the issues in any of the Kubeflow repositories.

Owners files and PR workflow

Our PR workflow is nearly identical to Kubernetes'. Most of these instructions are a modified version of Kubernetes' contributors and owners guides.

Overview of OWNERS files

OWNERS files are used to designate responsibility over different parts of the Kubeflow codebase. Today, we use them to assign the reviewer and approver roles used in our two-phase code review process. Our OWNERS files were inspired by Chromium OWNERS files, which in turn inspired GitHub’s CODEOWNERS files.

The velocity of a project that uses code review is limited by the number of people capable of reviewing code. The quality of a person’s code review is limited by their familiarity with the code under review. Our goal is to address both of these concerns through the prudent use and maintenance of OWNERS files

OWNERS

Each directory that contains a unit of independent code or content may also contain an OWNERS file. This file applies to everything within the directory, including the OWNERS file itself, sibling files, and child directories.

OWNERS files are in YAML format and support the following keys:

approvers: a list of GitHub usernames or aliases that can /approve a PR
labels: a list of GitHub labels to automatically apply to a PR
options: a map of options for how to interpret this OWNERS file, currently only one:
- no_parent_owners: defaults to false if not present; if true, exclude parent OWNERS files. Allows the use case where a/deep/nested/OWNERS file prevents a/OWNERS file from having any effect on a/deep/nested/bit/of/code
reviewers: a list of GitHub usernames or aliases that are good candidates to /lgtm a PR

All users are expected to be assignable. In GitHub terms, this means they are either collaborators of the repo, or members of the organization to which the repo belongs.

A typical OWNERS file looks like:

approvers:
  - alice
  - bob     # this is a comment
reviewers:
  - alice
  - carol   # this is another comment
  - sig-foo # this is an alias

OWNERS_ALIASES

Each repo may contain at its root an OWNERS_ALIAS file.

OWNERS_ALIAS files are in YAML format and support the following keys:

aliases: a mapping of alias name to a list of GitHub usernames

We use aliases for groups instead of GitHub Teams, because changes to GitHub Teams are not publicly auditable.

A sample OWNERS_ALIASES file looks like:

aliases:
  sig-foo:
    - david
    - erin
  sig-bar:
    - bob
    - frank

GitHub usernames and aliases listed in OWNERS files are case-insensitive.

The code review process

The author submits a PR
Phase 0: Automation suggests reviewers and approvers for the PR
- Determine the set of OWNERS files nearest to the code being changed
- Choose at least two suggested reviewers, trying to find a unique reviewer for every leaf OWNERS file, and request their reviews on the PR
- Choose suggested approvers, one from each OWNERS file, and list them in a comment on the PR
Phase 1: Humans review the PR
- Reviewers look for general code quality, correctness, sane software engineering, style, etc.
- Anyone in the organization can act as a reviewer with the exception of the individual who opened the PR
- If the code changes look good to them, a reviewer types /lgtm in a PR comment or review; if they change their mind, they /lgtm cancel
- Once a reviewer has /lgtm‘ed, prow (@k8s-ci-robot) applies an lgtm label to the PR
Phase 2: Humans approve the PR
- The PR author /assign’s all suggested approvers to the PR, and optionally notifies them (eg: “pinging @foo for approval”)
- Only people listed in the relevant OWNERS files, either directly or through an alias, can act as approvers, including the individual who opened the PR
- Approvers look for holistic acceptance criteria, including dependencies with other features, forwards/backwards compatibility, API and flag definitions, etc
- If the code changes look good to them, an approver types /approve in a PR comment or review; if they change their mind, they /approve cancel
- prow (@k8s-ci-robot) updates its comment in the PR to indicate which approvers still need to approve
- Once all approvers (one from each of the previously identified OWNERS files) have approved, prow (@k8s-ci-robot) applies an approved label
Phase 3: Automation merges the PR:
- If all of the following are true:
  - All required labels are present (eg: lgtm, approved)
  - Any blocking labels are missing (eg: there is no do-not-merge/hold, needs-rebase)
- And if any of the following are true:
  - there are no presubmit prow jobs configured for this repo
  - there are presubmit prow jobs configured for this repo, and they all pass after automatically being re-run one last time
- Then the PR will automatically be merged

Quirks of the process

There are a number of behaviors we’ve observed that while possible are discouraged, as they go against the intent of this review process. Some of these could be prevented in the future, but this is the state of today.

An approver’s /lgtm is simultaneously interpreted as an /approve
- While a convenient shortcut for some, it can be surprising that the same command is interpreted in one of two ways depending on who the commenter is
- Instead, explicitly write out /lgtm and /approve to help observers, or save the /lgtm for a reviewer
- This goes against the idea of having at least two sets of eyes on a PR, and may be a sign that there are too few reviewers (who aren’t also approver)
Technically, anyone who is a member of the Kubeflow GitHub organization can drive-by /lgtm a PR
- Drive-by reviews from non-members are encouraged as a way of demonstrating experience and intent to become a collaborator or reviewer
- Drive-by /lgtm’s from members may be a sign that our OWNERS files are too small, or that the existing reviewers are too unresponsive
- This goes against the idea of specifying reviewers in the first place, to ensure that author is getting actionable feedback from people knowledgeable with the code
Reviewers, and approvers are unresponsive
- This causes a lot of frustration for authors who often have little visibility into why their PR is being ignored
- Many reviewers and approvers are so overloaded by GitHub notifications that @mention’ing is unlikely to get a quick response
- If an author /assign’s a PR, reviewers and approvers will be made aware of it on their PR dashboard
- An author can work around this by manually reading the relevant OWNERS files, /unassign‘ing unresponsive individuals, and /assign‘ing others
- This is a sign that our OWNERS files are stale; pruning the reviewers and approvers lists would help with this
Authors are unresponsive
- This costs a tremendous amount of attention as context for an individual PR is lost over time
- This hurts the project in general as its general noise level increases over time
- Instead, close PR’s that are untouched after too long (we currently have a bot do this after 90 days)

Automation using OWNERS files

`prow`

Prow receives events from GitHub, and reacts to them. It is effectively stateless. The following pieces of prow are used to implement the code review process above.

cmd: tide
- per-repo configuration:
  - labels: list of labels required to be present for merge (eg: lgtm)
  - missingLabels: list of labels required to be missing for merge (eg: do-not-merge/hold)
  - reviewApprovedRequired: defaults to false; when true, require that there must be at least one approved pull request review present for merge
  - merge_method: defaults to merge; when squash or rebase, use that merge method instead when clicking a PR’s merge button
- merges PR’s once they meet the appropriate criteria as configured above
- if there are any presubmit prow jobs for the repo the PR is against, they will be re-run one final time just prior to merge
plugin: assign
- assigns GitHub users in response to /assign comments on a PR
- unassigns GitHub users in response to /unassign comments on a PR
plugin: approve
- per-repo configuration:
  - issue_required: defaults to false; when true, require that the PR description link to an issue, or that at least one approver issues a /approve no-isse
  - implicit_self_approve: defaults to false; when true, if the PR author is in relevant OWNERS files, act as if they have implicitly /approve’d
- adds the approved label once an approver for each of the required OWNERS files has /approve’d
- comments as required OWNERS files are satisfied
- removes outdated approval status comments
plugin: blunderbuss
- determines reviewers and requests their reviews on PR’s
plugin: lgtm
- adds the lgtm label when a reviewer comments /lgtm on a PR
- the PR author may not /lgtm their own PR
pkg: k8s.io/test-infra/prow/repoowners
- parses OWNERS and OWNERS_ALIAS files
- if the no_parent_owners option is encountered, parent owners are excluded from having any influence over files adjacent to or underneath of the current OWNERS file

Maintaining OWNERS files

OWNERS files should be regularly maintained.

We encourage people to self-nominate or self-remove from OWNERS files via PR’s. Ideally in the future we could use metrics-driven automation to assist in this process.

We should strive to:

grow the number of OWNERS files
add new people to OWNERS files
ensure OWNERS files only contain org members and repo collaborators
ensure OWNERS files only contain people are actively contributing to or reviewing the code they own
remove inactive people from OWNERS files

Bad examples of OWNERS usage:

directories that lack OWNERS files, resulting in too many hitting root OWNERS
OWNERS files that have a single person as both approver and reviewer
OWNERS files that haven’t been touched in over 6 months
OWNERS files that have non-collaborators present

Good examples of OWNERS usage:

there are more reviewers than approvers
the approvers are not in the reviewers section
OWNERS files that are regularly updated (at least once per release)

Docs: Deploy using UI

Mon, 01 Jan 0001 00:00:00 +0000

This page provides instructions for using the Kubeflow deployment web app to deploy Kubeflow on GCP. The deployment web app currently supports Kubeflow v1.0.0.

For more control over your deployment, see the guide to deployment using the CLI. The CLI supports Kubeflow v1.0.2 and later versions.

Before you start

Check the following requirements before installing Kubeflow:

Make sure that your GCP project meets the minimum requirements described in the project setup guide.
If you want to use Cloud Identity-Aware Proxy (Cloud IAP) for access control, follow the guide to setting up OAuth credentials. Cloud IAP is recommended for production deployments or deployments with access to sensitive data.

Deploy Kubeflow

Here’s a partial screenshot of the deployment user interface (UI):

Follow these steps to open the deployment UI and deploy Kubeflow on GCP:

Open https://deploy.kubeflow.cloud/ in your web browser. You should see a form like the one in the above screenshot.
Sign in to your browser using an account that has the owner role for your GCP project.
Complete the following fields on the form:
- Project ID: Enter your GCP project ID.
- Deployment name: Enter a short name that you can use to recognize this deployment of Kubeflow. The maximum length for the deployment name is 25 characters.
- Choose how to connect to Kubeflow service: You can choose one of the following options:
  - Login with GCP IAP: Choose this option if you want to use Cloud Identity-Aware Proxy (Cloud IAP) for access control. Cloud IAP is the best option for production deployments or deployments with access to sensitive data. See more details below.
  - Login with Username Password: Warning: This option is deprecated in Kubeflow 1.0 and will be removed in the next version. We recommend switching to IAP.
- GKE zone: Enter the GCP zone in which to create your deployment. The default is us-central-1a.
- Kubeflow version: Choose one of the available versions of Kubeflow. You can see all the versions on the Kubeflow releases page. If you need a version that does not show on the deployment UI, you need to deploy Kubeflow using the CLI.
- Share Anonymous Usage Report: Check this option to allow Kubeflow to report usage data using Spartakus. Spartakus does not report any personal information. The default is to enable the reporting of usage data.
Click Create Deployment.
Watch for the deployment updates in the information box at the bottom of the deployment UI.

Authenticating with Cloud IAP

This section contains details about using Cloud IAP to control access to Kubeflow. Cloud IAP is the best option for production deployments or deployments with access to sensitive data.

Follow the guide to setting up OAuth credentials.
Choose the Login with GCP IAP option on the Kubeflow deployment UI.
Enter your IAP OAuth client ID and IAP OAuth client secret into the corresponding fields on the deployment UI.
Complete the rest of the form as described above.

Kubeflow will be available at the following URI:

https://<deployment_name>.endpoints.<project>.cloud.goog/

It can take 10-15 minutes for the URI to become available. You can watch for updates in the information box on the deployment UI. If the deployment takes longer than expected, click Kubeflow Service Endpoint to try accessing your Kubeflow URI.

Authenticating with username and password

No longer supported

Basic authentication is not supported in Kubeflow v1.0.0 and will be removed entirely in the next version. We highly recommend switching to deploying Kubeflow with IAP.

Next steps

Run a full ML workflow on Kubeflow, using the end-to-end MNIST tutorial or the GitHub issue summarization example.
See how to delete your Kubeflow deployment using the CLI or the GCP Console.
See how to customize your Kubeflow deployment.
See how to upgrade Kubeflow and how to upgrade or reinstall a Kubeflow Pipelines deployment.
Troubleshoot any issues you may find.

Docs: Chainer Training

Mon, 01 Jan 0001 00:00:00 +0000

Alpha

This Kubeflow component has alpha status with limited support. See the Kubeflow versioning policies. The Kubeflow team is interested in your feedback about the usability of the feature.

Chainer is not supported in Kubeflow versions greater than v0.6. See the Kubeflow v0.6 documentation for earlier support for Chainer training.

Docs: Community

Mon, 01 Jan 0001 00:00:00 +0000

In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to make participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.

The Kubeflow community is guided by our Code of Conduct, which we encourage everybody to read before participating. We hold our leaders accountable for the guidelines in this document.

Welcome to all Google Summer of Code (GSoC) participants

The Kubeflow community is delighted to be part of Google Summer of Code 2020. Community mentors look forward to working with students on their GSoC projects.

Visit the Kubeflow GSoC page to find handy information and links for GSoC students and mentors.

Community discussions

There are many ways to contribute! Join one of our communication channels, attend a community meeting, get to know the community, discuss updates, suggest exciting new integrations.

Community meetings

Meeting calendar (iCal version).

Meeting notes.

If your group has a regular meeting, talk to @ewilderj about getting it added to the calendar.

Kubeflow community call

The project team holds a weekly community call on Tuesdays. This call alternates weekly between US East/EMEA and US West/APAC friendly times. Joining the kubeflow-discuss mailing list will automatically send you calendar invitations for the meetings, or you can subscribe to the community meeting calendar above.

Agenda, notes, and a reminder of the next call are sent to the kubeflow-discuss mailing list.

Slack community and channels

The Kubeflow Slack workspace is kubeflow.slack.com. To join, click this invitation to our Slack workspace.

The Kubeflow Slack workspace offers several channels. Here are a few examples:

Topic	Slack channel
General discussion	#general
Community meeting chat	#community
TF Operator (GitHub)	#tf-operator
KFServing (GitHub)	#kfserving
Pipelines (GitHub)	#kubeflow-pipelines
Examples (GitHub)	#kubeflow-examples
Documentation (GitHub)	#website
Product management	#product-management
Google Summer of Code (GSoC)	#gsoc

Mailing lists

The primary mailing list (email group) is kubeflow-discuss.

More detail about the Kubeflow mailing lists:

Topic	Mailing list
General discussion	kubeflow-discuss
TF Operator (GitHub)	tf-operator

Who should consider contributing to Kubeflow?

Folks who want to add support for other ML frameworks (e.g. PyTorch, XGBoost, scikit-learn)
Folks who want to bring more Kubernetes magic to ML (e.g. ISTIO integration for prediction)
Folks who want to make Kubeflow a richer ML platform (e.g. support for ML pipelines, hyperparameter tuning)
Folks who want to tune Kubeflow for their particular Kubernetes distribution or Cloud
Folks who want to write tutorials or blog posts showing how to use Kubeflow to solve ML problems

For details on contributing please look at the contributor’s guide.

Docs: Deploy using CLI

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to use the kfctl command line interface (CLI) to deploy Kubeflow on GCP. The command line deployment gives you more control over the deployment process and configuration than you get if you use the deployment UI. If you’re looking for a simpler deployment procedure, see how to deploy Kubeflow using the deployment UI.

Before you start

Before installing Kubeflow on the command line:

Ensure you have installed the following tools:
- kubectl.
- gcloud. If you already have gcloud installed, run gcloud components update to get the latest version of all your installed Cloud SDK components.
If you’re using Cloud Shell, enable boost mode.
Make sure that your GCP project meets the minimum requirements described in the project setup guide.
Follow the guide setting up OAuth credentials. to create OAuth credentials for Cloud Identity-Aware Proxy (Cloud IAP).

Prepare your environment

Follow these steps to download the kfctl binary for the Kubeflow CLI and set some handy environment variables:

Download the kfctl v1.0.2 release from the kfctl releases page.

Unpack the tar ball:

tar -xvf kfctl_v1.0.2_<platform>.tar.gz

Log in. You only need to run this command once:
```
gcloud auth login
```
Create user credentials. You only need to run this command once:
```
gcloud auth application-default login
```

Configure gcloud default values for zone and project

# Set your GCP project ID and the zone where you want to create 
# the Kubeflow deployment:
export PROJECT=<your GCP project ID>
export ZONE=<your GCP zone>

gcloud config set project ${PROJECT}    
gcloud config set compute/zone ${ZONE}

kfctl by default uses the gcloud defaults for zone and project
You can override this by explicitly setting zone and project in your KFDef file

Select the KFDef spec to use as the basis for your deployment

export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_gcp_iap.v1.0.2.yaml"

Create environment variables containing the OAuth client ID and secret that you created earlier
```
export CLIENT_ID=<CLIENT_ID from OAuth page>
export CLIENT_SECRET=<CLIENT_SECRET from OAuth page>
```
- The CLIENT_ID and CLIENT_SECRET can be obtained from the Cloud Console by selecting APIs & Services -> Credentials
Pick a name KF_NAME for your Kubeflow deployment and directory for your configuration.
```
export KF_NAME=<your choice of name for the Kubeflow deployment>
export BASE_DIR=<path to a base directory>
export KF_DIR=${BASE_DIR}/${KF_NAME}
```
- For example, your kubeflow deployment name might be ‘my-kubeflow’ or ‘kf-test’.
- Set base directory where you want to store one or more Kubeflow deployments. For example, ${HOME}/kf_deployments.
(Optional) Add the kfctl binary to your path. If you don’t add kfctl to your path, you must use the full path each time you run kfctl.
```
export PATH=$PATH:<path to your kfctl file>
```

Notes:

${PROJECT} - The project ID of the GCP project where you want Kubeflow deployed.
${ZONE} - The GCP zone where you want to create the Kubeflow deployment. You can see a list of zones in the Compute Engine documentation. If you plan to use accelerators, you must choose a zone that supports the type you want. See the guide to customizing your Kubeflow deployment.
${CONFIG_URI} - The GitHub address of the configuration YAML file that you want to use to deploy Kubeflow. For GCP deployments, the recommended configuration is:
```
https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_gcp_iap.v1.0.2.yaml
```
When you run kfctl apply or kfctl build (see the next step), kfctl creates a local version of the configuration YAML file which you can further customize if necessary.
${KF_NAME} - The name of your Kubeflow deployment. If you want a custom deployment name, specify that name here. For example, my-kubeflow or kf-test. The value of KF_NAME must consist of lower case alphanumeric characters or ‘-’, and must start and end with an alphanumeric character. The value of this variable cannot be greater than 25 characters. It must contain just a name, not a directory path. You also use this value as directory name when creating the directory where your Kubeflow configurations are stored, that is, the Kubeflow application directory.
${KF_DIR} - The full path to your Kubeflow application directory.

Deploying Kubeflow

To deploy Kubeflow using the default settings, run the kfctl apply command:

mkdir -p ${KF_DIR}
cd ${KF_DIR}
kfctl apply -V -f ${CONFIG_URI}

kfctl will try to populate the KFDef spec with various defaults automatically

project and zone will be set based on your gcloud config defaults
the name for the deployment will be inferred from the directory ${KF_DIR}
You can override these values by modifying your KFDef spec before running the build and apply commands

You can follow the instructions in the next section to override these defaults.

Customizing your Kubeflow deployment

The process outlined in the previous step configures Kubeflow with various defaults. You can follow the instructions below to have greater control.

Download the KFDef file to your local directory to allow modifications
```
mkdir -p ${KF_DIR}
cd ${KF_DIR}
curl -L -o ${CONFIG_FILE} https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_gcp_iap.v1.0.2.yaml
```
- CONFIG_FILE should be the name you would like to use for your local config file; e.g. “kfdef.yaml”
Edit the KFDef spec in the yaml file. The following snippet shows you how to set values in the configuration file using yq:
```
yq w -i ${CONFIG_FILE} 'spec.plugins[0].spec.project' ${PROJECT}
yq w -i ${CONFIG_FILE} 'spec.plugins[0].spec.zone' ${ZONE}
yq w -i ${CONFIG_FILE} 'metadata.name' ${KF_NAME}
```
- PROJECT: The GCP project to deploy in
- ZONE: The zone to deploy in
- KF_NAME: The name used for your deployment.
Run the kfctl build command to generate kustomize and GCP Deployment manager configuration files for your deployment:
```
cd ${KF_DIR}
kfctl build -V -f ${CONFIG_FILE}
```
To customize your GKE cluster modify the deployment manager configuration files in the directory ${KF_DIR}/gcp_config.
- For more information refer to:
  - customizing your Kubeflow deployment
  - deployment manager docs.
To customize individual Kubeflow applications modify the Kustomize manifests in the directory ${KF_DIR}/kustomize
- For more information please refer to the kustomize docs.
Run the kfctl apply command to deploy Kubeflow:
```
kfctl apply -V -f ${CONFIG_FILE}
```

Check your deployment

Follow these steps to verify the deployment:

The deployment process creates a separate deployment for your data storage. After running kfctl apply you should notice two new deployments:
- {KF_NAME}-storage: This deployment has persistent volumes for your pipelines.
- {KF_NAME}: This deployment has all the components of Kubeflow, including a GKE cluster named ${KF_NAME} with Kubeflow installed.
When the deployment finishes, check the resources installed in the namespace kubeflow in your new cluster. To do this from the command line, first set your kubectl credentials to point to the new cluster:
```
gcloud container clusters get-credentials ${KF_NAME} --zone ${ZONE} --project ${PROJECT}
```
Then see what’s installed in the kubeflow namespace of your GKE cluster:
```
kubectl -n kubeflow get all
```

Access the Kubeflow user interface (UI)

Follow these steps to access the Kubeflow central dashboard:

Enter the following URI into your browser address bar. It can take 20 minutes for the URI to become available:

https://<KF_NAME>.endpoints.<project-id>.cloud.goog/

You can run the following command to get the URI for your deployment:

kubectl -n istio-system get ingress
NAME            HOSTS                                                      ADDRESS         PORTS   AGE
envoy-ingress   your-kubeflow-name.endpoints.your-gcp-project.cloud.goog   34.102.232.34   80      5d13h

The following command sets an environment variable named HOST to the URI:

export HOST=$(kubectl -n istio-system get ingress envoy-ingress -o=jsonpath={.spec.rules[0].host})

Follow the instructions on the UI to create a namespace. See the guide to creation of profiles.

Notes:

It can take 20 minutes for the URI to become available. Kubeflow needs to provision a signed SSL certificate and register a DNS name.
If you own or manage the domain or a subdomain with Cloud DNS then you can configure this process to be much faster. See kubeflow/kubeflow#731.

Understanding the deployment process

This section gives you more details about the kfctl configuration and deployment process, so that you can customize your Kubeflow deployment if necessary.

kfctl process and configuration

The kfctl deployment process includes the following commands:

kfctl build - (Optional) Creates configuration files defining the various resources in your deployment. You only need to run kfctl build if you want to edit the resources before running kfctl apply. See the guide to customizing your Kubeflow deployment.
kfctl apply - Creates or updates the resources.
kfctl delete - Deletes the resources.

The kfctl deployment process applies default values to certain properties as follows:

Email address: kfctl attempts to fetch your email address from your Cloud SDK configuration. You can run gcloud config list to see the default email address, which the command output lists as the account. If kfctl can’t find a valid email address, you must use the flag --email <your email address> to pass a valid email address. This email address becomes an administrator in the configuration of your Kubeflow deployment.
GCP project ID: kfctl attempts to fetch your project ID from your Cloud SDK configuration. You can run gcloud config list to see your active project ID.
GCP zone: kfctl attempts to fetch the zone from your Cloud SDK configuration. You can run gcloud config list to see your active zone.
Kubeflow deployment name: kfctl defaults to the name of the directory where you run the kfctl build or kfctl apply command.

You can also explicitly set the following values in your ${CONFIG_FILE} configuration file:

Kubeflow deployment name
GCP project
GCP zone
Email address

The following snippet shows you how to set values in the configuration file using yq:

yq w -i ${CONFIG_FILE} 'spec.plugins[0].spec.project' ${PROJECT}
yq w -i ${CONFIG_FILE} 'spec.plugins[0].spec.zone' ${ZONE}
yq w -i ${CONFIG_FILE} 'metadata.name' ${KF_NAME}

Application layout

Your Kubeflow application directory ${KF_DIR} contains the following files and directories:

${CONFIG_FILE} is a YAML file that defines configurations related to your Kubeflow deployment.
- This file is a copy of the GitHub-based configuration YAML file that you used when deploying Kubeflow: kfctl_gcp_iap.v1.0.0.yaml.
- When you run kfctl apply or kfctl build, kfctl creates a local version of the configuration file, ${CONFIG_FILE}, which you can further customize if necessary.
gcp_config is a directory that contains Deployment Manager configuration files defining your GCP infrastructure.
- The directory is created when you run kfctl build or kfctl apply.
- You can modify these configurations to customize your GCP infrastructure. After modifying a configuration, run kfctl apply again.
kustomize is a directory that contains the kustomize packages for Kubeflow applications. See how Kubeflow uses kustomize.
- The directory is created when you run kfctl build or kfctl apply.
- You can customize the Kubernetes resources by modifying the manifests and running kfctl apply again.

We recommend that you check in the contents of your ${KF_DIR} directory into source control.

GCP service accounts

The kfctl deployment process creates three service accounts in your GCP project. These service accounts follow the principle of least privilege. The service accounts are:

${KF_NAME}-admin is used for some admin tasks like configuring the load balancers. The principle is that this account is needed to deploy Kubeflow but not needed to actually run jobs.
${KF_NAME}-user is intended to be used by training jobs and models to access GCP resources (Cloud Storage, BigQuery, etc.). This account has a much smaller set of privileges compared to admin.
${KF_NAME}-vm is used only for the virtual machine (VM) service account. This account has the minimal permissions needed to send metrics and logs to Stackdriver.

Basic authentication (deprecated)

No longer supported

Basic authentication is not supported in Kubeflow v1.0.0 and will be removed entirely in the next version. We highly recommend switching to deploying Kubeflow with IAP.

Next steps

Run a full ML workflow on Kubeflow, using the end-to-end MNIST tutorial or the GitHub issue summarization Pipelines example.
See how to delete your Kubeflow deployment using the CLI.
See how to customize your Kubeflow deployment.
See how to upgrade Kubeflow and how to upgrade or reinstall a Kubeflow Pipelines deployment.
Troubleshoot any issues you may find.

Docs: Install Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to use the kfctl CLI to deploy Kubeflow on Amazon Web Services (AWS).

Prerequisites

Install kubectl
Install and configure the AWS Command Line Interface (AWS CLI):
- Install the AWS Command Line Interface.
- Configure the AWS CLI by running the following command: aws configure.
- Enter your Access Keys (Access Key ID and Secret Access Key).
- Enter your preferred AWS Region and default output options.
Install eksctl (version 0.1.31 or newer) and the aws-iam-authenticator.

EKS cluster

There’re many ways to provision EKS cluster, using AWS EKS CLI, CloudFormation or Terraform, AWS CDK or eksctl. Here, we highly recommend you to create an EKS cluster using eksctl.

You are required to have an existing Amazon Elastic Kubernetes Service (Amazon EKS) cluster before moving the next step.

The installation tool uses the eksctl command and doesn’t support the --profile option in that command. If you need to switch role, use the aws sts assume-role commands. See the AWS guide to using temporary security credentials to request access to AWS resources.

Prepare your environment

In order to deploy Kubeflow on your existing Amazon EKS cluster, you need to provide AWS_CLUSTER_NAME, cluster region and worker roles.

Follow these steps to download the kfctl binary for the Kubeflow CLI and set some handy environment variables:

Download the kfctl v1.0.2 release from the Kubeflow releases page.

Unpack the tar ball:

tar -xvf kfctl_v1.0.2_<platform>.tar.gz

Create environment variables to make the deployment process easier:

# Add kfctl to PATH, to make the kfctl binary easier to use.
export PATH=$PATH:"<path to kfctl>"

# Use the following kfctl configuration file for the AWS setup without authentication:
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_aws.v1.0.2.yaml"

# Alternatively, use the following kfctl configuration if you want to enable
# authentication, authorization and multi-user:
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_aws_cognito.v1.0.2.yaml"

# Set an environment variable for your AWS cluster name, and set the name
# of the Kubeflow deployment to the same as the cluster name.
export AWS_CLUSTER_NAME=<YOUR EKS CLUSTER NAME>
export KF_NAME=${AWS_CLUSTER_NAME}

# Set the path to the base directory where you want to store one or more
# Kubeflow deployments. For example, /opt/.
# Then set the Kubeflow application directory for this deployment.
export BASE_DIR=<path to a base directory>
export KF_DIR=${BASE_DIR}/${KF_NAME}

Notes:

${CONFIG_URI} - The GitHub address of the configuration YAML file that you want to use to deploy Kubeflow. For AWS deployments, the following configurations are available:
- https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_aws.v1.0.2.yaml
- https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_aws_cognito.v1.0.2.yaml
  
  When you run kfctl apply or kfctl build (see the next step), kfctl creates a local version of the configuration YAML file which you can further customize if necessary.
${KF_NAME} - The name of your Kubeflow deployment. You should set this value to be the same as your AWS cluster name. The value of KF_NAME must consist of lower case alphanumeric characters or ‘-’, and must start and end with an alphanumeric character. The value of this variable cannot be greater than 25 characters. It must contain just a name, not a directory path. This value also becomes the name of the directory where your Kubeflow configurations are stored, that is, the Kubeflow application directory.
${KF_DIR} - The full path to your Kubeflow application directory.

Set up your Kubeflow configuration

Download your configuration files, so that you can customize the configuration before deploying Kubeflow:

mkdir -p ${KF_DIR}
cd ${KF_DIR}

wget -O kfctl_aws.yaml $CONFIG_URI
export CONFIG_FILE=${KF_DIR}/kfctl_aws.yaml

Configure Kubeflow

In v1.0.1, Kubeflow supports to use AWS IAM Roles for Service Account to fine grain control AWS service access. kfctl will create two roles kf-admin-${cluster_name} and kf-user-${cluster_name} and Kubernetes service account kf-admin and kf-user under kubeflow namespace. kf-admin-${cluster_name} will be assumed by components like alb-ingress-controller, profile-controller or any Kubeflow control plane components which need to talk to AWS services. kf-user-${cluster_name} can be used by user’s application.

This is only available on EKS, for DIY Kubernetes on AWS, check out aws/amazon-eks-pod-identity-webhook to setup webhook.

Traditional way to attach IAM policies to node group role is still working, feel free choose the way you like to use.

Option 1: Use IAM For Service Account

kfctl will help create or reuse IAM OIDC Identity Provider, create role and handle trust relationship binding with Kubernetes Service Accounts.

Note: By default, we don’t attach any policies to kf-user-${cluster_name}, you can attach policies based on your need.

Add enablePodIamPolicy: true in your ${CONFIG_FILE} file:

region: us-west-2
enablePodIamPolicy: true

# you can delete following roles settings.
#roles:
#- eksctl-kubeflow-example-nodegroup-ng-185-NodeInstanceRole-1DDJJXQBG9EM6

Check IAM Role For Service Account for more usage.

Option 2: Use Node Group Role

Replace the AWS cluster name in your ${CONFIG_FILE} file, by changing the value kubeflow-aws to ${AWS_CLUSTER_NAME} in multiple locations in the file. For example, use this sed command:
```
sed -i'.bak' -e 's/kubeflow-aws/'"$AWS_CLUSTER_NAME"'/' ${CONFIG_FILE}
```
Retrieve the AWS Region and IAM role name for your worker nodes. To get the IAM role name for your Amazon EKS worker node, run the following command:
```
aws iam list-roles \
    | jq -r ".Roles[] \
    | select(.RoleName \
    | startswith(\"eksctl-$AWS_CLUSTER_NAME\") and contains(\"NodeInstanceRole\")) \
    .RoleName"

eksctl-kubeflow-example-nodegroup-ng-185-NodeInstanceRole-1DDJJXQBG9EM6
```
Note: The above command assumes that you used eksctl to create your cluster. If you use other provisioning tools to create your worker node groups, find the role that is associated with your worker nodes in the Amazon EC2 console.
Change cluster region and worker role names in your ${CONFIG_FILE} file:

region: us-west-2
roles:
- eksctl-kubeflow-example-nodegroup-ng-185-NodeInstanceRole-1DDJJXQBG9EM6

If you have multiple node groups, you will see corresponding number of node group roles. In that case, please provide the role names as an array.

Deploy Kubeflow

Run the following commands to initialize the Kubeflow cluster:
```
cd ${KF_DIR}
kfctl apply -V -f ${CONFIG_FILE}
```
Important!!! By default, these scripts create an AWS Application Load Balancer for Kubeflow that is open to public. This is good for development testing and for short term use, but we do not recommend that you use this configuration for production workloads.

To secure your installation, Follow the instructions to add authentication and authorization.
Wait for all the resources to become ready in the kubeflow namespace.
```
kubectl -n kubeflow get all
```

Access Kubeflow central dashboard

If you are using https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_aws_cognito.v1.0.2.yaml, run following command to get Kubeflow service endpoint host name and copy link in browser.

kubectl get ingress -n istio-system

NAMESPACE      NAME            HOSTS   ADDRESS                                                             PORTS   AGE
istio-system   istio-ingress   *       a743484b-istiosystem-istio-2af2-xxxxxx.us-west-2.elb.amazonaws.com   80      1h

This deployment may take 3-5 minutes to become ready. Verify that the address works by opening it in your preferred Internet browser.

If you are using https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_aws.v1.0.2.yaml, the Kubeflow Dashboard can be accessed via istio-ingressgateway service.

You can run following command to port forward to local, then open http://localhost:8080 in browser.

kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80

Check more details Ingress Gateway guide

To expose Kubeflow with a LoadBalancer Service, just change the type of the istio-ingressgateway Service to LoadBalancer.

kubectl patch service -n istio-system istio-ingressgateway -p '{"spec": {"type": "LoadBalancer"}}'

While the change is being applied, you can watch the service until below command prints a value under the EXTERNAL-IP column:

kubectl get -w -n istio-system svc/istio-ingressgateway

The external IP should be accessible by visiting http://. Note that above installation instructions do not create any protection for the external endpoint so it will be accessible to anyone without any authentication. To secure your installation, use https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_aws_cognito.v1.0.2.yaml and follow the instructions to add authentication and authorization.

Post Installation

Kubeflow provides multi-tenancy support and user are not able to create notebooks in kubeflow, default namespace.

The first time you visit the cluster, you can ceate a namespace anonymous to use. If you want to create different users, you can create Profile and then kubectl apply -f profile.yaml. Profile controller will create new namespace and service account which is allowed to create notebook in that namespace.

apiVersion: kubeflow.org/v1beta1
kind: Profile
metadata:
  name: aws-sample-user
spec:
  owner:
    kind: User
    name: aws-sample-user

Check Multi-Tenancy in Kubeflow for more details.

Understanding the deployment process

The kfctl deployment process is controlled by the following commands:

kfctl build - (Optional) Creates configuration files defining the various resources in your deployment. You only need to run kfctl build if you want to edit the resources before running kfctl apply.
kfctl apply - Creates or updates the resources.
kfctl delete - Deletes the resources.

App layout

Your Kubeflow app directory ${KF_DIR} contains the following files and directories:

${CONFIG_FILE} is a YAML file that defines configurations related to your Kubeflow deployment.
- This file is a copy of the GitHub-based configuration YAML file that you used when deploying Kubeflow.
- When you run kfctl apply or kfctl build, kfctl creates a local version of the configuration file, ${CONFIG_FILE}, which you can further customize if necessary.
aws_config is a directory that contains a sample eksctl cluster configuration file that defines the AWS cluster and policy files to attach to your node group roles.
- You can modify the cluster_config.yaml and cluster_features.yaml files to customize your AWS infrastructure.
kustomize is a directory that contains the kustomize packages for Kubeflow applications.
- The directory is created when you run kfctl build or kfctl apply.
- You can customize the Kubernetes resources (modify the manifests and run kfctl apply again).

The provisioning scripts can either bring up a new cluster and install Kubeflow on it, or you can install Kubeflow on your existing cluster. We recommend that you create a new cluster for better isolation.

If you experience any issues running these scripts, see the troubleshooting guidance for more information.

Docs: Install Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to use the kfctl binary to deploy Kubeflow on Azure.

Prerequisites

Install kubectl
Install and configure the Azure Command Line Interface (Az)
- Log in with az login
(Optional) Install Docker
- For Windows and WSL: Guide
- For other OS: Docker Desktop

You do not need to have an existing Azure Resource Group or Cluster for AKS (Azure Kubernetes Service). You can create a cluster in the deployment process.

Understanding the deployment process

The deployment process is controlled by the following commands:

build - (Optional) Creates configuration files defining the various resources in your deployment. You only need to run kfctl build if you want to edit the resources before running kfctl apply.
apply - Creates or updates the resources.
delete - Deletes the resources.

App layout

Your Kubeflow application directory ${KF_DIR} contains the following files and directories:

${CONFIG_FILE} is a YAML file that defines configurations related to your Kubeflow deployment.
- This file is a copy of the GitHub-based configuration YAML file that you used when deploying Kubeflow. For example, https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_k8s_istio.v1.0.2.yaml.
- When you run kfctl apply or kfctl build, kfctl creates a local version of the configuration file, ${CONFIG_FILE}, which you can further customize if necessary.
kustomize is a directory that contains the kustomize packages for Kubeflow applications.
- The directory is created when you run kfctl build or kfctl apply.
- You can customize the Kubernetes resources (modify the manifests and run kfctl apply again).

If you experience any issues running these scripts, see the troubleshooting guidance for more information.

Azure setup

az login

Initial cluster setup for new cluster

Create a resource group:

az group create -n <RESOURCE_GROUP_NAME> -l <LOCATION>

Example variables:

RESOURCE_GROUP_NAME=KubeTest
LOCATION=westus

Create a specifically defined cluster:

az aks create -g <RESOURCE_GROUP_NAME> -n <NAME> -s <AGENT_SIZE> -c <AGENT_COUNT> -l <LOCATION> --generate-ssh-keys

Example variables:

NAME=KubeTestCluster
AGENT_SIZE=Standard_D4s_v3
AGENT_COUNT=2
Use the same resource group and name from the previous step

NOTE: If you are using a GPU based AKS cluster (For example: AGENT_SIZE=Standard_NC6), you also need to install the NVidia drivers on the cluster nodes before you can use GPUs with Kubeflow.

Kubeflow installation

Run the following commands to set up and deploy Kubeflow.

Create user credentials. You only need to run this command once.
```
 az aks get-credentials -n <NAME> -g <RESOURCE_GROUP_NAME>
```
Download the kfctl v1.0.2 release from the Kubeflow releases page.

Unpack the tar ball

 tar -xvf kfctl_v1.0.2_<platform>.tar.gz

Run the following commands to set up and deploy Kubeflow. The code below includes an optional command to add the binary kfctl to your path. If you don’t add the binary to your path, you must use the full path to the kfctl binary each time you run it.

# The following command is optional, to make kfctl binary easier to use.
export PATH=$PATH:<path to where kfctl was unpacked>

# Set KF_NAME to the name of your Kubeflow deployment. This also becomes the
# name of the directory containing your configuration.
# For example, your deployment name can be 'my-kubeflow' or 'kf-test'.
export KF_NAME=<your choice of name for the Kubeflow deployment>

# Set the path to the base directory where you want to store one or more 
# Kubeflow deployments. For example, /opt/.
# Then set the Kubeflow application directory for this deployment.
export BASE_DIR=<path to a base directory>
export KF_DIR=${BASE_DIR}/${KF_NAME}

# Set the configuration file to use, such as the file specified below:
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_k8s_istio.v1.0.2.yaml"

# Generate and deploy Kubeflow:
mkdir -p ${KF_DIR}
cd ${KF_DIR}
kfctl apply -V -f ${CONFIG_URI}

${KF_NAME} - The name of your Kubeflow deployment. If you want a custom deployment name, specify that name here. For example, my-kubeflow or kf-test. The value of KF_NAME must consist of lower case alphanumeric characters or ‘-’, and must start and end with an alphanumeric character. The value of this variable cannot be greater than 25 characters. It must contain just a name, not a directory path. This value also becomes the name of the directory where your Kubeflow configurations are stored, that is, the Kubeflow application directory.
${KF_DIR} - The full path to your Kubeflow application directory.

Check the resources deployed correctly in namespace kubeflow
```
 kubectl get all -n kubeflow
```
Open Kubeflow Dashboard

The default installation does not create an external endpoint but you can use port-forwarding to visit your cluster. Run the following command and visit http://localhost:8080.

kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80

To open the dashboard to a public IP address, you should first implement a solution to prevent unauthorized access. You can read more about Azure authentication options from Access Control for Azure Deployment.

Additional information

You can find general information about Kubeflow configuration in the guide to configuring Kubeflow with kfctl and kustomize.

Docs: Install Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to use the kfctl binary to deploy Kubeflow on IBM Cloud.

Prerequisites

Installing the IBM Cloud developer tools

If you already have ibmcloud installed with the latest ibmcloud ks (Kubernetes Service) plug-in, you can skip these steps.

Download and install the ibmcloud command line tool: https://cloud.ibm.com/docs/cli?topic=cloud-cli-getting-started#overview

Install the Kubernetes Service plug-in:

ibmcloud plugin install container-service

Authorize ibmcloud:
```
ibmcloud login
```

Setting environment variables

To simplify the command lines for this walkthrough, you need to define a few environment variables.

Set CLUSTER_NAME and CLUSTER_ZONE variables:
```
export CLUSTER_NAME=kubeflow
export CLUSTER_ZONE=dal13
```
- CLUSTER_NAME must be lowercase and unique among any other Kubernetes clusters in the specified CLUSTER_ZONE.
- CLUSTER_ZONE identifies the location where CLUSTER_NAME will be created. Run ibmcloud ks locations to list supported IBM Cloud Kubernetes Service locations. For example, choose dal13 to create CLUSTER_NAME in the Dallas (US) data center.

Creating a IBM Cloud Kubernetes cluster

To make sure the cluster is large enough to host all the Knative and Istio components, the recommended configuration for a cluster is:

Kubernetes version 1.15
4 vCPU nodes with 16GB memory (b2c.4x16)

Create a Kubernetes cluster on IKS with the required specifications:
```
ibmcloud ks cluster create classic \
  --flavor b2c.4x16 \
  --name $CLUSTER_NAME \
  --zone=$CLUSTER_ZONE \
  --workers=3
```
If you’re starting in a fresh account with no public and private VLANs, they are created automatically for you. If you already have VLANs configured in your account, get them via ibmcloud ks vlans --zone $CLUSTER_ZONE and include the public/private VLAN id in the cluster create command:
```
ibmcloud ks cluster create classic \
  --machine-type=b2c.4x16 \
  --name=$CLUSTER_NAME \
  --zone=$CLUSTER_ZONE \
  --workers=3 \
  --private-vlan $PRIVATE_VLAN_ID \
  --public-vlan $PUBLIC_VLAN_ID 
```
Wait until your Kubernetes cluster is deployed:
```
ibmcloud ks clusters | grep $CLUSTER_NAME
```
It can take a while for your cluster to be deployed. Repeat the above command until the state of your cluster is “normal”.

Point kubectl to the cluster:

ibmcloud ks cluster config --cluster $CLUSTER_NAME

Make sure all nodes are up:
```
kubectl get nodes
```
Make sure all the nodes are in Ready state. You are now ready to install Istio into your cluster.

IBM Cloud Block Storage Setup

By default, IBM Cloud Kubernetes cluster uses IBM Cloud File Storage based on NFS as the default storage class. File Storage is designed to run RWX (read-write multiple nodes) workloads with proper security built around it. Therefore, File Storage does not allow fsGroup securityContext which is needed for DEX and Kubeflow Jupyter Server.

IBM Cloud Block Storage provides a fast way to store data and satisfy many of the Kubeflow persistent volume requirements such as fsGroup out of the box and optimized RWO (read-write single node) which is used on all Kubeflow’s persistent volume claim.

Therefore, we strongly recommend to set up IBM Cloud Block Storage as the default storage class so that you can get the best experience from Kubeflow.

Follow the instructions to install the Helm version 3 client on your local machine.
Add the IBM Cloud Helm chart repository to the cluster where you want to use the IBM Cloud Block Storage plug-in.
```
helm repo add iks-charts https://icr.io/helm/iks-charts
helm repo update
```

Install the IBM Cloud Block Storage plug-in. When you install the plug-in, pre-defined block storage classes are added to your cluster.

helm install 1.6.0 iks-charts/ibmcloud-block-storage-plugin -n kube-system

Example output:

NAME: 1.6.0
LAST DEPLOYED: Thu Feb 27 11:41:35 2020
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing: ibmcloud-block-storage-plugin.   Your release is named: 1.6.0
...

Verify that the installation was successful.

kubectl get pod -n kube-system | grep block

Verify that the storage classes for Block Storage were added to your cluster.
```
kubectl get storageclasses | grep block
```

Set the Block Storage as the default storageclass.

kubectl patch storageclass ibmc-block-gold -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
kubectl patch storageclass ibmc-file-bronze -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
   
# Check the default storageclass is block storage
kubectl get storageclass | grep \(default\)

Example output:

ibmc-block-gold (default)   ibm.io/ibmc-block   65s

Understanding the Kubeflow deployment process

The deployment process is controlled by the following commands:

build - (Optional) Creates configuration files defining the various resources in your deployment. You only need to run kfctl build if you want to edit the resources before running kfctl apply.
apply - Creates or updates the resources.
delete - Deletes the resources.

App layout

Your Kubeflow application directory ${KF_DIR} contains the following files and directories:

${CONFIG_FILE} is a YAML file that defines configurations related to your Kubeflow deployment.
- This file is a copy of the GitHub-based configuration YAML file that you used when deploying Kubeflow. For example, https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_k8s_istio.v1.0.2.yaml.
- When you run kfctl apply or kfctl build, kfctl creates a local version of the configuration file, ${CONFIG_FILE}, which you can further customize if necessary.
kustomize is a directory that contains the kustomize packages for Kubeflow applications.
- The directory is created when you run kfctl build or kfctl apply.
- You can customize the Kubernetes resources (modify the manifests and run kfctl apply again).

Kubeflow installation

Run the following commands to set up and deploy Kubeflow.

Download the kfctl v1.0.2 release from the Kubeflow releases page.

Unpack the tar ball

tar -xvf kfctl_v1.0.2_<platform>.tar.gz

# The following command is optional, to make kfctl binary easier to use.
export PATH=$PATH:<path to where kfctl was unpacked>

# Set KF_NAME to the name of your Kubeflow deployment. This also becomes the
# name of the directory containing your configuration.
# For example, your deployment name can be 'my-kubeflow' or 'kf-test'.
export KF_NAME=<your choice of name for the Kubeflow deployment>

# Set the path to the base directory where you want to store one or more 
# Kubeflow deployments. For example, /opt/.
# Then set the Kubeflow application directory for this deployment.
export BASE_DIR=<path to a base directory>
export KF_DIR=${BASE_DIR}/${KF_NAME}

# Set the configuration file to use, such as the file specified below:
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/master/kfdef/kfctl_ibm.yaml"

# Generate and deploy Kubeflow:
mkdir -p ${KF_DIR}
cd ${KF_DIR}
kfctl apply -V -f ${CONFIG_URI}

${KF_NAME} - The name of your Kubeflow deployment. If you want a custom deployment name, specify that name here. For example, my-kubeflow or kf-test. The value of KF_NAME must consist of lower case alphanumeric characters or ‘-’, and must start and end with an alphanumeric character. The value of this variable cannot be greater than 25 characters. It must contain just a name, not a directory path. This value also becomes the name of the directory where your Kubeflow configurations are stored, that is, the Kubeflow application directory.
${KF_DIR} - The full path to your Kubeflow application directory.

Check the resources deployed correctly in namespace kubeflow
```
kubectl get all -n kubeflow
```
Open Kubeflow Dashboard. The default installation does not create an external endpoint but you can use port-forwarding to visit your cluster. Run the following command and visit http://localhost:8080.
```
kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80
```

In case you want to expose the Kubeflow Dashboard over an external IP, you can change the type of the ingress gateway. To do that, you can edit the service:

 kubectl edit -n istio-system svc/istio-ingressgateway

From that file, replace type: NodePort with type: LoadBalancer and save.

While the change is being applied, you can watch the service until below command prints a value under the EXTERNAL-IP column:

 kubectl get -w -n istio-system svc/istio-ingressgateway

Additional information

You can find general information about Kubeflow configuration in the guide to configuring Kubeflow with kfctl and kustomize.

Docs: Install Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to use the kfctl CLI to deploy Kubeflow 0.7 on an existing OpenShift 4.2 cluster.

Prerequisites

OpenShift 4 cluster

You need to have access to an OpenShift 4 cluster as cluster-admin to be able to deploy Kubeflow.
You can use Code Ready Containers (CRC) to run a local cluster, use try.openshift.com to create a new cluster or use an existing cluster.
Install oc command-line tool to communicate with the cluster.

Code Ready Containers

If you are using Code Ready Containers, you need to make sure you have enough resources configured for the VM:

Recommended:

16 GB memory
6 CPU
45 GB disk space

Minimal:

10 GB memory
6 CPU
30 GB disk space (default for CRC)

Installing Kubeflow

Use the following steps to install Kubeflow 0.7 on OpenShift 4.2.

Clone the [opendatahub/manifests] (https://github.com/opendatahub-io/manifests) repository. This repository defaults to the v0.7.0-branch-openshift branch.
```
git clone https://github.com/opendatahub-io/manifests.git
cd manifests
```

Build the deployment configuration using the OpenShift KFDef file and local downloaded manifests.

At the time this document was written, Kubeflow issue #4678 prevents downloading the manifests during a build process. Update the manifest repo URI. Copy the KFDef file to the Kubeflow application directory. And finally build the configuration.

# update the manifest repo URI
sed -i 's#uri: .*#uri: '$PWD'#' ./kfdef/kfctl_openshift.yaml

# set the Kubeflow application diretory for this deployment, for example /opt/openshift-kfdef
export KF_DIR=<path-to-kfdef>
mkdir -p ${KF_DIR}
cp ./kfdef/kfctl_openshift.yaml ${KF_DIR}
   
# build deployment configuration
cd ${KF_DIR}
kfctl build --file=kfctl_openshift.yaml

Apply the generated deployment configuration.
```
kfctl apply --file=kfctl_openshift.yaml
```

Wait until all the pods are running.

$ oc get pods -n kubeflow
NAME                                                           READY     STATUS             RESTARTS   AGE
argo-ui-7c584fc474-k5blx                                       1/1       Running            0          3m46s
centraldashboard-678f74d985-rblnm                              1/1       Running            0          3m41s
jupyter-web-app-deployment-57977c6965-2qznb                    1/1       Running            0          3m37s
katib-controller-fddbb4864-fdzf5                               1/1       Running            1          3m4s
katib-db-6b9b5bc446-6pbtp                                      1/1       Running            0          3m3s
katib-manager-7797db7f7c-p5ztb                                 1/1       Running            1          3m3s
katib-ui-5bdbb97475-585rp                                      1/1       Running            0          3m2s
metadata-db-c88c9bf6f-5ddbz                                    1/1       Running            0          3m30s
metadata-deployment-969879b6c-swvqf                            1/1       Running            0          3m30s
metadata-envoy-deployment-69766744b5-75t5l                     1/1       Running            0          3m29s
metadata-grpc-deployment-578956fc6d-msvj5                      1/1       Running            3          3m29s
metadata-ui-57f9b8d667-dckm4                                   1/1       Running            0          3m28s
minio-784784b9bb-bqslk                                         1/1       Running            0          2m56s
ml-pipeline-687969b966-wx6jd                                   1/1       Running            0          2m59s
ml-pipeline-ml-pipeline-visualizationserver-57997bdc64-jw6l4   1/1       Running            0          2m37s
ml-pipeline-persistenceagent-b74f6455b-z9nzw                   1/1       Running            0          2m51s
...

The command below looks up the URL of Kubeflow user interface assigned by the OpenShift cluster. You can open the printed URL in your broser to access the Kubeflow user interface.
```
oc get routes -n istio-system istio-ingressgateway -o jsonpath='http://{.spec.host}/'
```

Next steps

Learn about the changes made to Kubeflow manifests to enable deployment on OpenShift
See how to upgrade Kubeflow and how to upgrade or reinstall a Kubeflow Pipelines deployment.
See how to uninstall your Kubeflow deployment using the CLI.

Docs: Multi-user, auth-enabled Kubeflow with kfctl_existing_arrikto

Mon, 01 Jan 0001 00:00:00 +0000

If you were using the kfctl_existing_arrikto configuration in Kubeflow v0.7 or earlier, you should use kfctl_istio_dex.v1.0.2.yaml in Kubeflow v1.0.2. Follow the instructions in the guide to kfctl_istio_dex.

Docs: Multi-user, auth-enabled Kubeflow with kfctl_istio_dex

Mon, 01 Jan 0001 00:00:00 +0000

Follow these instructions if you want to install Kubeflow on an existing Kubernetes cluster.

This installation of Kubeflow is geared towards existing Kubernetes clusters and does not depend on any cloud-specific feature.

Architecture overview

In this reference architecture, we use Dex and Istio for vendor-neutral authentication.

This deployment works well for on-premises installations, where companies/organizations need LDAP/AD integration for multi-user authentication, and they don’t want to depend on any cloud-specific feature.

Read the relevant article for more info about this architecture.

Before you start

The instructions below assume that you have an existing Kubernetes cluster.

Default StorageClass for on-premises deployments

Notes on the configuration file

Configuring your installation with kfctl_istio_dex.v1.0.2.yaml has a few options you should consider:

Disabling istio installation - If your Kubernetes cluster has an existing Istio installation you may choose to not install Istio by removing the applications istio-crds and istio-install in the configuration file kfctl_istio_dex.v1.0.2.yaml.
Istio configuration for trustworthy JWTs - This configuration uses Istio version 1.3.1 with SDS enabled, which requires Kubernetes 1.13 or later. Follow Istio’s blog to add API server configurations to your Kubernetes cluster. Ensure that the TokenRequest feature flag is set to true in the cluster. For kubeadm created clusters, set the API server flags in the pod manifest at /etc/kubernetes/manifests/kube-apiserver.yaml. For example, the Istio community runs their test-infrastructure with the following API server flags:
```
"service-account-issuer": "kubernetes.default.svc"
"service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"
```
Default password in static file configuration for Dex - The configuration file kfctl_istio_dex.v1.0.2.yaml contains a default staticPasswords user with email set to admin@kubeflow.org and password 12341234. You should change this configuration or replace it with a Dex connector.

Prepare your environment

Follow these steps to download the kfctl binary for the Kubeflow CLI and set some handy environment variables:

Download the kfctl v1.0.2 release from the Kubeflow releases page.

Unpack the tar ball:

tar -xvf kfctl_<release tag>_<platform>.tar.gz

Create environment variables to make the deployment process easier:

# Add kfctl to PATH, to make the kfctl binary easier to use.
# Use only alphanumeric characters or - in the directory name.
export PATH=$PATH:"<path-to-kfctl>"

# Set the following kfctl configuration file:
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_istio_dex.v1.0.2.yaml"

# Set KF_NAME to the name of your Kubeflow deployment. You also use this
# value as directory name when creating your configuration directory.
# For example, your deployment name can be 'my-kubeflow' or 'kf-test'.
export KF_NAME=<your choice of name for the Kubeflow deployment>

# Set the path to the base directory where you want to store one or more 
# Kubeflow deployments. For example, /opt.
# Then set the Kubeflow application directory for this deployment.
export BASE_DIR=<path to a base directory>
export KF_DIR=${BASE_DIR}/${KF_NAME}

Notes:

${KF_NAME} - The name of your Kubeflow deployment. If you want a custom deployment name, specify that name here. For example, my-kubeflow or kf-test. The value of KF_NAME must consist of lower case alphanumeric characters or ‘-’, and must start and end with an alphanumeric character. The value of this variable cannot be greater than 25 characters. It must contain just a name, not a directory path. You also use this value as directory name when creating the directory where your Kubeflow configurations are stored, that is, the Kubeflow application directory.
${KF_DIR} - The full path to your Kubeflow application directory.
${CONFIG_URI} - The GitHub address of the configuration YAML file that you want to use to deploy Kubeflow. The URI used in this guide is https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_istio_dex.v1.0.2.yaml. When you run kfctl apply or kfctl build (see the next step), kfctl creates a local version of the configuration YAML file which you can further customize if necessary.

Set up and deploy Kubeflow

To set up and deploy Kubeflow using the default settings, run the kfctl apply command:

mkdir -p ${KF_DIR}
cd ${KF_DIR}

# Download the config file and change the default login credentials.
wget -O kfctl_istio_dex.yaml $CONFIG_URI
export CONFIG_FILE=${KF_DIR}/kfctl_istio_dex.yaml

# Credentials for the default user are admin@kubeflow.org:12341234
# To change them, please edit the dex-auth application parameters
# inside the KfDef file.
vim $CONFIG_FILE

kfctl apply -V -f ${CONFIG_FILE}

Alternatively, set up your configuration for later deployment

If you want to customize your configuration before deploying Kubeflow, you can set up your configuration files first, then edit the configuration, then deploy Kubeflow:

Run the kfctl build command to set up your configuration:

mkdir -p ${KF_DIR}
cd ${KF_DIR}
kfctl build -V -f ${CONFIG_URI}

Edit the configuration files, as described in the guide to customizing your Kubeflow deployment.
Set an environment variable pointing to your local configuration file:
```
export CONFIG_FILE=${KF_DIR}/kfctl_istio_dex.yaml
```
Run the kfctl apply command to deploy Kubeflow:
```
kfctl apply -V -f ${CONFIG_FILE}
```

Accessing Kubeflow

Log in as a static user

The default way of accessing Kubeflow is via port-forward. This enables you to get started quickly without imposing any requirements on your environment.

# Kubeflow will be available at localhost:8080
kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80

The credentials are the ones you specified in the KfDef file, or the default (admin@kubeflow.org:12341234). It is highly recommended to change the default credentials. To add static users or change the existing one, see the relevant section.

When you’re ready, you can expose your Kubeflow deployment with a LoadBalancer Service or an Ingress. For more information, see the expose kubeflow section.

Add static users for basic auth

To add users to basic auth, you just have to edit the Dex ConfigMap under the key staticPasswords.

# Download the dex config
kubectl get configmap dex -n auth -o jsonpath='{.data.config\.yaml}' > dex-config.yaml

# Edit the dex config with extra users.
# The password must be hashed with bcrypt with an at least 10 difficulty level.
# You can use an online tool like: https://passwordhashing.com/BCrypt

# After editing the config, update the ConfigMap
kubectl create configmap dex --from-file=config.yaml=dex-config.yaml -n auth --dry-run -oyaml | kubectl apply -f -

# Restart Dex to pick up the changes in the ConfigMap
kubectl rollout restart deployment dex -n auth

Log in with LDAP / Active Directory

As you saw in the overview, we use Dex for providing user authentication. Dex supports several authentication methods:

Static users, as described above
LDAP / Active Directory
External Identity Provider (IdP) (for example Google, LinkedIn, GitHub, …)

This section focuses on setting up Dex to authenticate with an existing LDAP database.

(Optional) If you don’t have an LDAP database, you can set one up following these instructions:

Deploy a new LDAP Server as a StatefulSet. This also deploys phpLDAPadmin, a GUI for interacting with your LDAP Server.

LDAP Server Manifest

        apiVersion: v1
        kind: Service
        metadata:
          labels:
            app: ldap
          name: ldap-service
          namespace: kubeflow
        spec:
          type: ClusterIP
          clusterIP: None
          ports:
            - port: 389
          selector:
            app: ldap
        ---
        
        apiVersion: apps/v1
        kind: StatefulSet
        metadata:
          name: ldap
          namespace: kubeflow
          labels:
            app: ldap
        spec:
          serviceName: ldap-service
          replicas: 1
          selector:
            matchLabels:
              app: ldap
          template:
            metadata:
              labels:
                app: ldap
            spec:
              containers:
                - name: ldap
                  image: osixia/openldap:1.2.4
                  volumeMounts:
                    - name: ldap-data
                      mountPath: /var/lib/ldap
                    - name: ldap-config
                      mountPath: /etc/ldap/slapd.d
                  ports:
                    - containerPort: 389
                      name: openldap
                  env:
                    - name: LDAP_LOG_LEVEL
                      value: "256"
                    - name: LDAP_ORGANISATION
                      value: "Example"
                    - name: LDAP_DOMAIN
                      value: "example.com"
                    - name: LDAP_ADMIN_PASSWORD
                      value: "admin"
                    - name: LDAP_CONFIG_PASSWORD
                      value: "config"
                    - name: LDAP_BACKEND
                      value: "mdb"
                    - name: LDAP_TLS
                      value: "false"
                    - name: LDAP_REPLICATION
                      value: "false"
                    - name: KEEP_EXISTING_CONFIG
                      value: "false"
                    - name: LDAP_REMOVE_CONFIG_AFTER_SETUP
                      value: "true"
              volumes:
                - name: ldap-config
                  emptyDir: {}
          volumeClaimTemplates:
              - metadata:
                  name: ldap-data
                spec:
                  accessModes: [ "ReadWriteOnce" ]
                  resources:
                    requests:
                      storage: 10Gi
        
        ---
        
        apiVersion: v1
        kind: Service
        metadata:
          labels:
            app: phpldapadmin
          name: phpldapadmin-service
          namespace: kubeflow
        spec:
          type: ClusterIP
          ports:
            - port: 80
          selector:
            app: phpldapadmin
        
        ---
        
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: phpldapadmin
          namespace: kubeflow
          labels:
            app: phpldapadmin
        spec:
          replicas: 1
          selector:
            matchLabels:
              app: phpldapadmin
          template:
            metadata:
              labels:
                app: phpldapadmin
            spec:
              containers:
                - name: phpldapadmin
                  image: osixia/phpldapadmin:0.8.0
                  ports:
                    - name: http-server
                      containerPort: 80
                  env:
                    - name: PHPLDAPADMIN_HTTPS
                      value: "false"
                    - name: PHPLDAPADMIN_LDAP_HOSTS
                      value : "#PYTHON2BASH:[{'ldap-service.kubeflow.svc.cluster.local': [{'server': [{'tls': False}]},{'login': [        {'bind_id': 'cn=admin,dc=example,dc=com'}]}]}]"

Seed the LDAP database with new entries.

kubectl exec -it -n kubeflow ldap-0 -- bash
ldapadd -x -D "cn=admin,dc=example,dc=com" -W
# Enter password "admin".
# Press Ctrl+D to complete after pasting the snippet below.

LDAP Seed Users and Groups

# If you used the OpenLDAP Server deployment in step 1,
# then this object already exists.
# If it doesn't, uncomment this.
#dn: dc=example,dc=com
#objectClass: dcObject
#objectClass: organization
#o: Example
#dc: example
       
dn: ou=People,dc=example,dc=com
objectClass: organizationalUnit
ou: People
       
dn: cn=Nick Kiliadis,ou=People,dc=example,dc=com
objectClass: person
objectClass: inetOrgPerson
givenName: Nick
sn: Kiliadis
cn: Nick Kiliadis
uid: nkili
mail: nkili@example.com
userpassword: 12341234
       
dn: cn=Robin Spanakopita,ou=People,dc=example,dc=com
objectClass: person
objectClass: inetOrgPerson
givenName: Robin
sn: Spanakopita
cn: Robin Spanakopita
uid: rspanakopita
mail: rspanakopita@example.com
userpassword: 43214321
       
# Group definitions.
       
dn: ou=Groups,dc=example,dc=com
objectClass: organizationalUnit
ou: Groups
       
dn: cn=admins,ou=Groups,dc=example,dc=com
objectClass: groupOfNames
cn: admins
member: cn=Nick Kiliadis,ou=People,dc=example,dc=com
       
dn: cn=developers,ou=Groups,dc=example,dc=com
objectClass: groupOfNames
cn: developers
member: cn=Nick Kiliadis,ou=People,dc=example,dc=com
member: cn=Robin Spanakopita,ou=People,dc=example,dc=com

To use your LDAP/AD server with Dex, you have to edit the Dex config. To edit the ConfigMap containing the Dex config, follow these steps:

Get the current Dex config from the corresponding Config Map.

        kubectl get configmap dex -n auth -o jsonpath='{.data.config\.yaml}' > dex-config.yaml

Add the LDAP-specific options. Here is an example to help you out. It is configured to work with the example LDAP Server you set up previously.

Dex LDAP Config Section

        connectors:
        - type: ldap
          # Required field for connector id.
          id: ldap
          # Required field for connector name.
          name: LDAP
          config:
            # Host and optional port of the LDAP server in the form "host:port".
            # If the port is not supplied, it will be guessed based on "insecureNoSSL",
            # and "startTLS" flags. 389 for insecure or StartTLS connections, 636
            # otherwise.
            host: ldap-service.kubeflow.svc.cluster.local:389
          
            # Following field is required if the LDAP host is not using TLS (port 389).
            # Because this option inherently leaks passwords to anyone on the same network
            # as dex, THIS OPTION MAY BE REMOVED WITHOUT WARNING IN A FUTURE RELEASE.
            #
            insecureNoSSL: true
          
            # If a custom certificate isn't provide, this option can be used to turn off
            # TLS certificate checks. As noted, it is insecure and shouldn't be used outside
            # of explorative phases.
            #
            insecureSkipVerify: true
          
            # When connecting to the server, connect using the ldap:// protocol then issue
            # a StartTLS command. If unspecified, connections will use the ldaps:// protocol
            #
            startTLS: false
          
            # Path to a trusted root certificate file. Default: use the host's root CA.
            # rootCA: /etc/dex/ldap.ca
            # clientCert: /etc/dex/ldap.cert
            # clientKey: /etc/dex/ldap.key
          
            # A raw certificate file can also be provided inline.
            # rootCAData: ( base64 encoded PEM file )
          
            # The DN and password for an application service account. The connector uses
            # these credentials to search for users and groups. Not required if the LDAP
            # server provides access for anonymous auth.
            # Please note that if the bind password contains a `$`, it has to be saved in an
            # environment variable which should be given as the value to `bindPW`.
            bindDN: cn=admin,dc=example,dc=com
            bindPW: admin
          
            # The attribute to display in the provided password prompt. If unset, will
            # display "Username"
            usernamePrompt: username
          
            # User search maps a username and password entered by a user to a LDAP entry.
            userSearch:
              # BaseDN to start the search from. It will translate to the query
              # "(&(objectClass=person)(uid=<username>))".
              baseDN: ou=People,dc=example,dc=com
              # Optional filter to apply when searching the directory.
              filter: "(objectClass=inetOrgPerson)"
          
              # username attribute used for comparing user entries. This will be translated
              # and combined with the other filter as "(<attr>=<username>)".
              username: uid
              # The following three fields are direct mappings of attributes on the user entry.
              # String representation of the user.
              idAttr: uid
              # Required. Attribute to map to Email.
              emailAttr: mail
              # Maps to display name of users. No default value.
              nameAttr: givenName
          
            # Group search queries for groups given a user entry.
            groupSearch:
              # BaseDN to start the search from. It will translate to the query
              # "(&(objectClass=group)(member=<user uid>))".
              baseDN: ou=Groups,dc=example,dc=com
              # Optional filter to apply when searching the directory.
              filter: "(objectClass=groupOfNames)"
          
              # Following two fields are used to match a user to a group. It adds an additional
              # requirement to the filter that an attribute in the group must match the user's
              # attribute value.
              userAttr: DN
              groupAttr: member
          
              # Represents group name.
              nameAttr: cn

Append the LDAP config section to the dex config.

        cat dex-config.yaml dex-config-ldap-partial.yaml > dex-config-final.yaml

Apply the new config.

        kubectl create configmap dex --from-file=config.yaml=dex-config-final.yaml -n auth --dry-run -oyaml | kubectl apply -f -

Restart the Dex deployment: kubectl rollout restart deployment dex -n auth

Expose Kubeflow

While port-forward is a great way to get started, it is not a long-term, production-ready solution. In this section, we explore the process of exposing your cluster to the outside world.

NOTE: It is highly recommended to change the default credentials before exposing your Kubeflow cluster. See the relevant section for how to edit Dex static users.

Secure with HTTPS

Since we are exposing our cluster to the outside world, it’s important to secure it with HTTPS. Here we will configure automatic self-signed certificates.

Edit the Istio Gateway Object and expose port 443 with HTTPS. In addition, make port 80 redirect to 443:

kubectl edit -n kubeflow gateways.networking.istio.io kubeflow-gateway

The Gateway Spec should look like the following:

spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP
    # Upgrade HTTP to HTTPS
    tls:
      httpsRedirect: true
  - hosts:
    - '*'
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      privateKey: /etc/istio/ingressgateway-certs/tls.key
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt

Expose with a LoadBalancer

If you don’t have support for LoadBalancer on your cluster, please follow the instructions below to deploy MetalLB in Layer 2 mode. (You can read more about Layer 2 mode in the MetalLB docs.)

MetalLB deployment

Deploy MetalLB:

Apply the manifest:

kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.8.1/manifests/metallb.yaml

Allocate a pool of addresses on your local network for MetalLB to use. You need at least one address for the Istio Gateway. This example assumes addresses 10.0.0.100-10.0.0.110. You must modify these addresses based on your environment.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 10.0.0.100-10.0.0.110
EOF

Ensure that MetalLB works as expected (optional):

Create a dummy service:

kubectl create service loadbalancer nginx --tcp=80:80
service/nginx created

Ensure that MetalLB has allocated an IP address for the service:

kubectl describe service nginx
...
Events:
  Type    Reason       Age   From                Message
  ----    ------       ----  ----                -------
  Normal  IPAllocated  69s   metallb-controller  Assigned IP "10.0.0.101"

Check the corresponding MetalLB logs:

kubectl logs -n metallb-system -l component=controller
...
{"caller":"service.go:98","event":"ipAllocated","ip":"10.0.0.101","msg":"IP address assigned by controller","service":"default/nginx","ts":"2019-08-09T15:12:09.376779263Z"}

Create a pod that will be exposed with the service:

kubectl run nginx --image nginx --restart=Never -l app=nginx
pod/nginx created

Ensure that MetalLB has assigned a node to announce the allocated IP address:

kubectl describe service nginx
...
Events:
  Type    Reason       Age   From                Message
  ----    ------       ----  ----                -------
   Normal  nodeAssigned  4s    metallb-speaker     announcing from node "node-2"

Check the corresponding MetalLB logs:

kubectl logs -n metallb-system -l component=speaker
...
{"caller":"main.go:246","event":"serviceAnnounced","ip":"10.0.0.101","msg":"service has IP, announcing","pool":"default","protocol":"layer2","service":"default/nginx","ts":"2019-08-09T15:14:02.433876894Z"}

Check that MetalLB responds to ARP requests for the allocated IP address:

arping -I eth0 10.0.0.101
...
ARPING 10.0.0.101 from 10.0.0.204 eth0
Unicast reply from 10.0.0.101 [6A:13:5A:D2:65:CB]  2.619ms

Check the corresponding MetalLB logs:

kubectl logs -n metallb-system -l component=speaker
...
{"caller":"arp.go:102","interface":"eth0,"ip":"10.0.0.101","msg":"got ARP request for service IP, sending response","responseMAC":"6a:13:5a:d2:65:cb","senderIP":"10.0.0.204","senderMAC":"9a:1f:7c:95:ca:dc","ts":"2019-08-09T15:14:52.912056021Z"}

Verify that everything works as expected:

curl http://10.0.0.101
...
<p><em>Thank you for using nginx.</em></p>
...

Clean up:

kubectl delete service nginx
kubectl delete pod nginx

To expose Kubeflow with a LoadBalancer Service, just change the type of the istio-ingressgateway Service to LoadBalancer.

kubectl patch service -n istio-system istio-ingressgateway -p '{"spec": {"type": "LoadBalancer"}}'

After that, get the LoadBalancer’s IP or Hostname from its status and create the necessary certificate.

kubectl get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0]}'

Create the Certificate with cert-manager:

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: istio-ingressgateway-certs
  namespace: istio-system
spec:
  commonName: istio-ingressgateway.istio-system.svc
  # Use ipAddresses if your LoadBalancer issues an IP
  ipAddresses:
  - <LoadBalancer IP>
  # Use dnsNames if your LoadBalancer issues a hostname (eg on AWS)
  dnsNames:
  - <LoadBalancer HostName>
  isCA: true
  issuerRef:
    kind: ClusterIssuer
    name: kubeflow-self-signing-issuer
  secretName: istio-ingressgateway-certs

After applying the above Certificate, cert-manager will generate the TLS certificate inside the istio-ingressgateway-certs secrets. The istio-ingressgateway-certs secret is mounted on the istio-ingressgateway deployment and used to serve HTTPS.

Navigate to https://<LoadBalancer Address>/ and start using Kubeflow.

Delete Kubeflow

Run the following commands to delete your deployment and reclaim all resources:

cd ${KF_DIR}
# If you want to delete all the resources, run:
kfctl delete -f ${CONFIG_FILE}

Understanding the deployment process

The kfctl deployment process includes the following commands:

kfctl build - (Optional) Creates configuration files defining the various resources in your deployment. You only need to run kfctl build if you want to edit the resources before running kfctl apply.
kfctl apply - Creates or updates the resources.
kfctl delete - Deletes the resources.

Application layout

Your Kubeflow application directory ${KF_DIR} contains the following files and directories:

${CONFIG_FILE} is a YAML file that defines configurations related to your Kubeflow deployment.
- This file is a copy of the GitHub-based configuration YAML file that you used when deploying Kubeflow: https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_istio_dex.v1.0.2.yaml
- When you run kfctl apply or kfctl build, kfctl creates a local version of the configuration file, ${CONFIG_FILE}, which you can further customize if necessary.
kustomize is a directory that contains the kustomize packages for Kubeflow applications. See how Kubeflow uses kustomize.
- The directory is created when you run kfctl build or kfctl apply.
- You can customize the Kubernetes resources by modifying the manifests and running kfctl apply again.

We recommend that you check in the contents of your ${KF_DIR} directory into source control.

Provisioning of Persistent Volumes in Kubernetes

Note that you can skip this step if you have a dynamic volume provisioner already installed in your cluster.

If you don’t have one:

You can choose to create PVs manually after deployment of Kubeflow.
Or install a dynamic volume provisioner like Local Path Provisioner. Ensure that the StorageClass used by this provisioner is the default StorageClass.

Troubleshooting

Persistent Volume Claims are in Pending State

Check if PersistentVolumeClaims get Bound to PersistentVolumes.

kubectl -n kubeflow get pvc

Kubeflow dashboard is not available

If the Kubeflow dashboard is not available at https://<kubeflow address> ensure that:

the virtual services have been created:
```
kubectl get virtualservices -n kubeflow
kubectl get virtualservices -n kubeflow centraldashboard -o yaml
```
If not, then kfctl has aborted for some reason, and not completed successfully.

OIDC auth service redirects you to Dex:

curl -k https://<kubeflow address>/ -v
...
< HTTP/2 302
< content-type: text/html; charset=utf-8
< location:
/dex/auth?client_id=kubeflow-authservice-oidc&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=openid+profile+email+groups&state=vSCMnJ2D
< date: Fri, 09 Aug 2019 14:33:21 GMT
< content-length: 181
< x-envoy-upstream-service-time: 0
< server: istio-envoy

Some additional debugging information:

OIDC AuthService logs:

kubectl logs -n istio-system -l app=authservice

Dex logs:

kubectl logs -n auth -l app=dex

Istio ingress-gateway logs:

kubectl logs -n istio-system -l istio=ingressgateway

Istio ingressgateway service:

kubectl get service -n istio-system istio-ingressgateway -o yaml

MetalLB logs:

kubectl logs -n metallb-system -l component=speaker
...
{"caller":"arp.go:102","interface":"br100","ip":"10.0.0.100","msg":"got ARP request for service IP, sending response","responseMAC":"62:41:bd:5f:cc:0d","senderIP":"10.0.0.204","senderMAC":"9a:1f:7c:95:ca:dc","ts":"2019-07-31T13:19:19.7082836Z"}

kubectl logs -n metallb-system  -l component=controller
...
{"caller":"service.go:98","event":"ipAllocated","ip":"10.0.0.100","msg":"IP address assigned by controller","service":"istio-system/istio-ingressgateway","ts":"2019-07-31T12:17:46.234638607Z"}

Please join the Kubeflow Slack to report any issues, request help, and give us feedback on this config.

Next steps

Run a sample machine learning workflow.
Get started with Kubeflow Pipelines.

Docs: Google Summer of Code

Mon, 01 Jan 0001 00:00:00 +0000

The Kubeflow community is delighted to be part of Google Summer of Code (GSoC) 2020. Community mentors look forward to working with students on their GSoC projects.

Top links:

Information for students

The GSoC student application phase is now closed and student selection is in progress. See the GSoC timeline.

You’re still welcome to explore Kubeflow. Kubeflow welcomes contributions at any time, whether within or outside the framework of GSoC. See the community page for Slack channels, mailing lists, meetings, and other ways to connect with the Kubeflow community.

For more information about GSoC, see the GSoC docs and guides.

Information for mentors

Thanks to all mentors who’re interested in helping a GSoC student with their Kubeflow project.

To get started:

Read the Google Summer of Code mentor guide and the roles and responsibilities to understand what is expected of you over the next 6 months.
Bookmark the GSoC timeline and be prepared to take action in time for the milestones shown on the timeline, and to help your students meet the deadlines.
Watch the 5 minute YouTube video: Being a Great GSoC Mentor. The video was shot with GSoC veteran org admins and mentors. It contains helpful information about the program and how to participate successfully at each step in GSoC.
If you’re not registered as a mentor, email kubeflow-gsoc-admin@kubeflow.org to indicate your interest in mentoring a student during GSoC.
When you receive an invitation, follow the link in the email to register on the GSoC program site.
Examine the student proposals. Click want to mentor on the proposals that you’d like to mentor.

Start chatting to students:

Join the #gsoc channel on Kubeflow Slack and answer students' questions.
Respond to students' questions on the Kubeflow mentors mailing list (kubeflow-gsoc-mentors@kubeflow.org).

For more information about GSoC, see the GSoC docs and guides.

Mailing lists

These are the Kubeflow mailing lists related to GSoC:

kubeflow-gsoc-mentors@kubeflow.org: The mentor mailing list for Kubeflow GSoC projects. Students can email this group to receive feedback on project proposals. Mentors and org administrators can use this group to discuss Kubeflow GSoC matters. Group members are Kubeflow GSoC mentors and org administrators. Messages are visible to sender, CCs, and group members.
kubeflow-gsoc-admin@kubeflow.org: The organization administrator mailing list for Kubeflow’s participation in GSoC. Students and mentors can email this group to raise Kubeflow GSoC matters with the administrators. Group members are Kubeflow GSoC org administrators only. Messages are visible to sender, CCs, and group members.
kubeflow-discuss: Kubeflow’s primarily mailing list and discussion group. See the community page for more information.

Docs: Initial cluster setup for existing cluster

Mon, 01 Jan 0001 00:00:00 +0000

Initial Setup for Existing Cluster

Get the Kubeconfig file:

az aks get-credentials -n <NAME> -g <RESOURCE_GROUP_NAME>

From here on, please see Install Kubeflow.

Docs: Initial cluster setup for existing cluster

Mon, 01 Jan 0001 00:00:00 +0000

Initial Setup for Existing Cluster

Get the Kubeconfig file:

ibmcloud ks cluster config --cluster $CLUSTER_NAME

From here on, please see Install Kubeflow.

Docs: Kubeflow Versioning Policies

Mon, 01 Jan 0001 00:00:00 +0000

This page describes the Kubeflow versioning policies and provides a version matrix for Kubeflow applications and other components.

Kubeflow versioning

Kubeflow version numbers are of the form vX.Y.Z, where X is the major version, Y is the minor version, and Z is the patch version. The versioning policy follows the Semantic Versioning terminology.

The version name vX.Y.Z refers to the version (git tag) of the kfctl release. If the version number includes an appendix -rcN, where N is a number, the appendix indicates a release candidate, which is a pre-release version of an upcoming release.

Examples of Kubeflow version numbers:

v0.7.0
v0.7.0-rc8
v1.0.0
v1.0.1

Application versioning and stable status

Starting from the release of Kubeflow v1.0, the Kubeflow community attributes stable status to those applications and components that meet a defined level of stability, supportability, and upgradability.

When you deploy Kubeflow to a Kubernetes cluster, your deployment includes a number of applications. Application versioning is independent of Kubeflow versioning. An application moves to version 1.0 when the application meets certain criteria in terms of stability, upgradability, and the provision of services such as logging and monitoring.

When an application moves to version 1.0, the Kubeflow community will decide whether to mark that version of the application as stable in the next major or minor release of Kubeflow.

Kubeflow application matrix

The following table shows the status (stable, beta, or alpha) of the applications that you can deploy to your Kubernetes cluster when you deploy Kubeflow. The applications are specified in the manifest that you use to deploy Kubeflow. The kfctl deployment tool deploys the applications strictly according to the manifest. kfctl does not decide whether to deploy or not deploy an application based on the application status.

You can use the information below to decide which of the applications you should deploy to your production system, and adjust the manifest accordingly.

Application status indicators for Kubeflow:

Stable means that the application complies with the criteria to reach application version 1.0, and that the Kubeflow community has deemed the application stable for this release of Kubeflow.
Beta means that the application is working towards a version 1.0 release and its maintainers have communicated a timeline for satisfying the criteria for the stable status.
Alpha means that the application is in the early phases of development and/or integration into Kubeflow.

The application version in the table reflects the application version in the manifest at the time when Kubeflow v1.0.2 was released. This is therefore the default version of the application that you receive when you deploy Kubeflow v1.0.2. Some applications may release later versions that you can choose to install into your Kubeflow deployment. If you need a later version of a specific application, refer to the documentation for that application.

Application	Status in Kubeflow v1.0.2	Application version in Kubeflow v1.0.2
Central dashboard: Kubeflow UI (GitHub)	Stable	1.0.0
Chainer operator (GitHub)	Alpha
Hyperparameter tuning: Katib (GitHub)	Beta	v1alpha3
KFServing (GitHub)	Beta	v0.2.2
Metadata (GitHub)	Beta	0.2.1
MPI training: MPI operator (GitHub)	Alpha
MXNet training: MXNet operator (GitHub)	Alpha
Notebook web app (GitHub)	Stable	1.0.0
Notebook controller (GitHub)	Stable	1.0.0
Pipelines (GitHub)	Beta	0.2.0
Profile Controller for multi-user isolation (GitHub)	Stable	1.0.0
PyTorch training: PyTorch operator (GitHub)	Stable	1.0.0
Seldon Core Serving (GitHub)	Stable	1.0.1
TensorFlow training: TFJob operator (GitHub)	Stable	1.0.0
XGBoost training: XGBoost operator (GitHub)	Alpha

Kubeflow SDKs and CLIs

Alongside Kubeflow v1.0.2, you may want to use one of the following Kubeflow SDKs and command-line interfaces (CLIs).

SDK / CLI	Status with Kubeflow v1.0.2	SDK/CLI version
Fairing (GitHub)	Beta	0.7.1
kfctl (GitHub )	Stable	1.0.0
Kubeflow Pipelines SDK (GitHub)	Beta	0.2.0

Support levels

The expectations for supportability and the types of support available depend on the stable status of each application or other component. For more information, see the support page.

Docs: Metadata

Mon, 01 Jan 0001 00:00:00 +0000

Beta

This Kubeflow component has beta status. See the Kubeflow versioning policies. The Kubeflow team is interested in your feedback about the usability of the feature.

The goal of the Metadata project is to help Kubeflow users understand and manage their machine learning (ML) workflows by tracking and managing the metadata that the workflows produce.

In this context, metadata means information about executions (runs), models, datasets, and other artifacts. Artifacts are the files and objects that form the inputs and outputs of the components in your ML workflow.

Installing the Metadata component

Kubeflow v0.6.1 and later versions install the Metadata component by default. You can skip this section if you are running Kubeflow v0.6.1 or later.

If you want to install the latest version of the Metadata component or to install the component as an application in your Kubernetes cluster, follow these steps:

Download the Kubeflow manifests repository:

git clone https://github.com/kubeflow/manifests

Run the following commands to deploy the services of the Metadata component:

cd manifests/metadata
kustomize build overlays/db | kubectl apply -n kubeflow -f -

Using the Metadata SDK to record metadata

The Metadata project publishes a Python SDK (API reference, source) that you can use to record your metadata.

Run the following command to install the Metadata SDK:

pip install kubeflow-metadata

Try the Metadata SDK in a sample Jupyter notebook

You can find an example of how to use the Metadata SDK in this demo notebook.

To run the notebook in your Kubeflow cluster:

Follow the guide to setting up your Jupyter notebooks in Kubeflow.
Go to the demo notebook on GitHub.
Download the notebook code by opening the Raw view of the file, then right-clicking on the content and saving the file locally as demo.ipynb.
Go back to your Jupyter notebook server in the Kubeflow UI. (If you’ve moved away from the notebooks section in Kubeflow, click Notebook Servers in the left-hand navigation panel to get back there.)
In the Jupyter notebook UI, click Upload and follow the prompts to upload the demo.ipynb notebook.
Click the notebook name (demo.ipynb) to open the notebook in your Kubeflow cluster.
Run the steps in the notebook to install and use the Metadata SDK.

When you have finished running through the steps in the demo.ipynb notebook, you can view the resulting metadata on the Kubeflow UI:

Click Artifact Store in the left-hand navigation panel on the Kubeflow UI.
On the Artifacts screen you should see the following items:

A model metadata item with the name MNIST.
A metrics metadata item with the name MNIST-evaluation.
A dataset metadata item with the name mytable-dump.

You can click the name of each item to view the details. See the section below about the Metadata UI for more details.

Learn more about the Metadata SDK

The Metadata SDK includes the following predefined types that you can use to describe your ML workflows:

DataSet to capture metadata for a dataset that forms the input into or the output of a component in your workflow.
Execution to capture metadata for an execution (run) of your ML workflow.
Metrics to capture metadata for the metrics used to evaluate an ML model.
Model to capture metadata for an ML model that your workflow produces.

Using metadata watcher to record metadata

Besides using the Python SDK to log metadata directly, you can add your own metadata watcher to watch Kubernetes resource changes and save the metadata into the metadata service.

Tracking artifacts on the Metadata UI

You can view a list of logged artifacts and the details of each individual artifact in the Artifact Store on the Kubeflow UI.

Go to Kubeflow in your browser. (If you haven’t yet opened the Kubeflow UI, find out how to access the central dashboard.)
Click Artifact Store in the left-hand navigation panel:
The Artifacts screen opens and displays a list of items for all the metadata events that your workflows have logged. You can click the name of each item to view the details.

The following examples show the items that appear when you run the demo.ipynb notebook described above:

Example of model metadata with the name “MNIST”:

<img src="/docs/images/metadata-model.png" 
  alt="Model metadata for an example MNIST model"
  class="mt-3 mb-3 border border-info rounded">

Example of metrics metadata with the name “MNIST-evaluation”:

<img src="/docs/images/metadata-metrics.png" 
  alt="Metrics metadata for an evaluation of an MNIST model"
  class="mt-3 mb-3 border border-info rounded">

Example of dataset metadata with the name “mytable-dump”:

<img src="/docs/images/metadata-dataset.png" 
  alt="Dataset metadata"
  class="mt-3 mb-3 border border-info rounded">

GRPC backend

The Kubeflow metadata deploys the gRPC service of ML Metadata (MLMD) to manage the metadata and relationships.

Kubeflow Metadata SDK saves and retrieves data via the gRPC service. Similarly, you can define your own metadata types to log and view metadata for your custom artifacts. For Python examples, you can check MLMD Python client and Kubeflow Metadata SDK source code. For Go examples, you can check the source code of the metadata watcher.

Next steps

Run the xgboost-synthetic notebook to build, train, and deploy an XGBoost model using Kubeflow Fairing and Kubeflow Pipelines with synthetic data. Examine the metadata output after running through the steps in the notebook.

Docs: Monitor Cloud IAP Setup

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Identity-Aware Proxy (Cloud IAP) is the recommended solution for accessing your Kubeflow deployment from outside the cluster, when running Kubeflow on Google Cloud Platform (GCP).

This document is a step-by-step guide to ensuring that your IAP-secured endpoint is available, and to debugging problems that may cause the endpoint to be unavailable.

Introduction

When deploying Kubeflow using the deployment UI or the command-line interface, you choose the authentication method you want to use. One of the options is Cloud IAP. This document assumes that you have already deployed Kubeflow.

Kubeflow uses the Google-managed certificate to provide an SSL certificate for the Kubeflow Ingress.

Cloud IAP gives you the following benefits:

Users can log in in using their GCP accounts.
You benefit from Google’s security expertise to protect your sensitive workloads.

Monitoring your Cloud IAP setup

Follow these instructions to monitor your Cloud IAP setup and troubleshoot any problems:

Examine the Ingress and Google Cloud Build (GCB) load balancer to make sure it is available:

kubectl -n istio-system describe ingress

Name:             envoy-ingress
Namespace:        kubeflow
Address:          35.244.132.160
Default backend:  default-http-backend:80 (10.20.0.10:8080)
Annotations:
...
Events:
   Type     Reason     Age                 From                     Message
   ----     ------     ----                ----                     -------
   Normal   ADD        12m                 loadbalancer-controller  kubeflow/envoy-ingress
   Warning  Translate  12m (x10 over 12m)  loadbalancer-controller  error while evaluating the ingress spec: could not find service "kubeflow/envoy"
   Warning  Translate  12m (x2 over 12m)   loadbalancer-controller  error while evaluating the ingress spec: error getting BackendConfig for port "8080" on service "kubeflow/envoy", err: no BackendConfig for service port exists.
   Warning  Sync       12m                 loadbalancer-controller  Error during sync: Error running backend syncing routine: received errors when updating backend service: googleapi: Error 400: The resource 'projects/code-search-demo/global/backendServices/k8s-be-32230--bee2fc38fcd6383f' is not ready, resourceNotReady
 googleapi: Error 400: The resource 'projects/code-search-demo/global/backendServices/k8s-be-32230--bee2fc38fcd6383f' is not ready, resourceNotReady
   Normal  CREATE  11m  loadbalancer-controller  ip: 35.244.132.160
...

There should be an annotation indicating that we are using managed certificate:

annotation:
  networking.gke.io/managed-certificates: gke-certificate

Any problems with creating the load balancer are reported as Kubernetes events in the results of the above describe command.

If the address isn’t set then there was a problem creating the load balancer.
The CREATE event indicates the load balancer was successfully created on the specified IP address.
The most common error is running out of GCP quota. To fix this problem, you must either increase the quota for the relevant resource on your GCP project or delete some existing resources.

Verify that a managed certificate resource is generated:
```
kubectl describe -n istio-system managedcertificate gke-certificate
```
The status field should have information about the current status of the Certificate. Eventually, certificate status should be Active.
Wait for the load balancer to report the back ends as healthy:
```
kubectl describe -n istio-system ingress envoy-ingress

...
Annotations:
 kubernetes.io/ingress.global-static-ip-name:  kubeflow-ip
 kubernetes.io/tls-acme:                       true
 certmanager.k8s.io/issuer:                    letsencrypt-prod
 ingress.kubernetes.io/backends:               {"k8s-be-31380--5e1566252944dfdb":"HEALTHY","k8s-be-32133--5e1566252944dfdb":"HEALTHY"}
...
```
Both backends should be reported as healthy. It can take several minutes for the load balancer to consider the back ends healthy.

The service with port 31380 is the one that handles Kubeflow traffic. (31380 is the default port of the service istio-ingressgateway.)

If the backend is unhealthy, check the pods in istio-system:
- kubectl get pods -n istio-system
- The istio-ingressgateway-XX pods should be running
- Check the logs of pod backend-updater-0, iap-enabler-XX to see if there is any error
- Follow the steps here to check the load balancer and backend service on GCP.
Try accessing Cloud IAP at the fully qualified domain name in your web browser:
```
https://<your-fully-qualified-domain-name>     
```
If you get SSL errors when you log in, this typically means that your SSL certificate is still propagating. Wait a few minutes and try again. SSL propagation can take up to 10 minutes.

If you do not see a login prompt and you get a 404 error, the configuration of Cloud IAP is not yet complete. Keep retrying for up to 10 minutes.
If you get an error Error: redirect_uri_mismatch after logging in, this means the list of OAuth authorized redirect URIs does not include your domain.

The full error message looks like the following example and includes the relevant links:
```
The redirect URI in the request, https://mykubeflow.endpoints.myproject.cloud.goog/_gcp_gatekeeper/authenticate, does not match the ones authorized for the OAuth client. 	
To update the authorized redirect URIs, visit: https://console.developers.google.com/apis/credentials/oauthclient/22222222222-7meeee7a9a76jvg54j0g2lv8lrsb4l8g.apps.googleusercontent.com?project=22222222222	
```
Follow the link in the error message to find the OAuth credential being used and add the redirect URI listed in the error message to the list of authorized URIs. For more information, read the guide to setting up OAuth for Cloud IAP.

Next steps

The GCP troubleshooting guide for Kubeflow.
Guide to sharing cluster access.
GCP guide to Cloud IAP.

Docs: Overview of Jupyter Notebooks in Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

This Kubeflow component has stable status. See the Kubeflow versioning policies.

There are multiple benefits of integrating Jupyter notebooks in Kubeflow for enterprise environments. These benefits include:

Integrating well with the rest of the infrastructure with respect to authentication and access control.
Enabling easier notebook sharing across the organization. Users can create notebook containers or pods directly in the cluster, rather than locally on their workstations. Admins can provide standard notebook images for their organization, and set up role-based access control (RBAC), Secrets and Credentials to manage which teams and individuals can access the notebooks.

When you bundle Jupyter notebooks in Kubeflow, you can use the Fairing library to submit training jobs using TFJob. The training job can run single node or distributed on the same Kubernetes cluster, but not inside the notebook pod itself. Submitting the job with the Fairing library makes processes like Docker containerization and pod allocation clear for data scientists.

Overall, Kubeflow-hosted notebooks are better integrated with other components while providing extensibility for notebook images.

Next steps

Learn more about setting up notebooks.

Docs: Overview of Kubeflow Fairing

Mon, 01 Jan 0001 00:00:00 +0000

Beta

This Kubeflow component has beta status. See the Kubeflow versioning policies. The Kubeflow team is interested in your feedback about the usability of the feature.

Kubeflow Fairing streamlines the process of building, training, and deploying machine learning (ML) training jobs in a hybrid cloud environment. By using Kubeflow Fairing and adding a few lines of code, you can run your ML training job locally or in the cloud, directly from Python code or a Jupyter notebook. After your training job is complete, you can use Kubeflow Fairing to deploy your trained model as a prediction endpoint.

Getting started

Use the following guides to get started with Kubeflow Fairing:

To set up your development environment, follow the guide to installing Kubeflow Fairing.
To ensure that Kubeflow Fairing can access your Kubeflow cluster, follow the guide to configuring your development environment with access to Kubeflow.
To learn more about how to use Kubeflow Fairing in your environment, follow the Kubeflow Fairing tutorials.

What is Kubeflow Fairing?

Kubeflow Fairing is a Python package that makes it easy to train and deploy ML models on Kubeflow. Kubeflow Fairing can also been extended to train or deploy on other platforms. Currently, Kubeflow Fairing has been extended to train on Google AI Platform.

Kubeflow Fairing packages your Jupyter notebook, Python function, or Python file as a Docker image, then deploys and runs the training job on Kubeflow or AI Platform. After your training job is complete, you can use Kubeflow Fairing to deploy your trained model as a prediction endpoint on Kubeflow.

The following are the goals of the Kubeflow Fairing project:

Easily package ML training jobs: Enable ML practitioners to easily package their ML model training code, and their code’s dependencies, as a Docker image.
Easily train ML models in a hybrid cloud environment: Provide a high-level API for training ML models to make it easy to run training jobs in the cloud, without needing to understand the underlying infrastructure.
Streamline the process of deploying a trained model: Make it easy for ML practitioners to deploy trained ML models to a hybrid cloud environment.

Next steps

Learn how to set up a Jupyter notebooks instance on your Kubeflow cluster.

Docs: Delete using CLI

Mon, 01 Jan 0001 00:00:00 +0000

This page shows you how to use the CLI to delete a Kubeflow deployment on Google Cloud Platform (GCP).

Before you start

This guide assumes the following settings:

The ${KF_DIR} environment variable contains the path to your Kubeflow application directory, which holds your Kubeflow configuration files. For example, /opt/my-kubeflow/.
```
export KF_DIR=<path to your Kubeflow application directory>
```

The ${CONFIG_FILE} environment variable contains the path to your Kubeflow configuration file.

export CONFIG_FILE=${KF_DIR}/kfctl_gcp_iap.v1.0.2.yaml

Or:

export CONFIG_FILE=${KF_DIR}/kfctl_gcp_basic_auth.v1.0.2.yaml

For further background about the above settings, see the guide to deploying Kubeflow with the CLI.

Deleting your deployment

Run the following commands to delete your deployment and reclaim all GCP resources:

# If you want to delete all the resources, including storage:
kfctl delete -f ${CONFIG_FILE} --delete_storage

# If you want to preserve storage, which contains metadata and information
# from Kubeflow Pipelines:
kfctl delete -f ${CONFIG_FILE}

You should consider preserving storage if you may want to relaunch Kubeflow in the future and restore the data from your pipelines.

Docs: Docs

Mon, 01 Jan 0001 00:00:00 +0000

Welcome to the Kubeflow documentation!

Introduction

The Kubeflow docs are published at www.kubeflow.org.

The source for the docs is in the kubeflow/website repo on GitHub. We use Hugo to format and generate our website, and Netlify to manage the deployment of the site.

Versioning

www.kubeflow.org points to the master branch of the docs. You can access other versions by clicking the version dropdown at top right of the website menu bar:

We create a new branch of the docs for each stable release of Kubeflow. For example, the docs for the v0.2 stable release are on published on the v0.2 website, which corresponds to the v0.2-branch on GitHub.

Contributing to the docs

We welcome updates to the docs! Please help us make them better. Small fixes, typos, bug fixes, plugging gaps—all are useful.

For help with getting started, take a look at the README.
For guidance on writing effective documentation, see the style guide.

Docs: Style Guide for the Kubeflow Docs

Mon, 01 Jan 0001 00:00:00 +0000

This style guide is for the Kubeflow documentation. The style guide helps contributors to write documentation that readers can understand quickly and correctly. The Kubeflow docs aim for:

Consistency in style and terminology, so that readers can expect certain structures and conventions. Readers don’t have to keep re-learning how to use the documentation or questioning whether they’ve understood something correctly.
Clear, concise writing so that readers can quickly find and understand the information they need.

Use standard American spelling

Use American spelling rather than Commonwealth or British spelling. Refer to Merriam-Webster’s Collegiate Dictionary, Eleventh Edition.

Use capital letters sparingly

Some hints:

Capitalize only the first letter of each heading within the page. (That is, use sentence case.)
Capitalize (almost) every word in page titles. (That is, use title case.) The little words like “and”, “in”, etc, don’t get a capital letter.
In page content, use capitals only for brand names, like Kubeflow, Kubernetes, and so on. See more about brand names below.
Don’t use capital letters to emphasize words.

Spell out abbreviations and acronyms on first use

Always spell out the full term for every abbreviation or acronym the first time you use it on the page. Don’t assume people know what an abbreviation or acronym means, even if it seems like common knowledge.

Example: “To run Kubernetes locally in a virtual machine (VM)”

Use contractions if you want to

For example, it’s fine to write “it’s” instead of “it is”.

Use full, correct brand names

When referring to a product or brand, use the full name. Capitalize the name as the product owners do in the product documentation. Do not use abbreviations even if they’re in common use, unless the product owner has sanctioned the abbreviation.

Use this	Instead of this
Kubeflow	kubeflow
Kubernetes	k8s
ksonnet	Ksonnet

Be consistent with punctuation

Use punctuation consistently within a page. For example, if you use a period (full stop) after every item in list, then use a period on all other lists on the page.

Check the other pages if you’re unsure about a particular convention. Examples:

Most pages in the Kubeflow docs use a period at the end of every list item.
There is no period at the end of the page subtitle and the subtitle need not be a full sentence. (The subtitle comes from the description in the front matter of each page.)

Use active voice rather than passive voice

Passive voice is often confusing, as it’s not clear who should perform the action.

Use active voice	Instead of passive voice
You can configure Kubeflow to	Kubeflow can be configured to
Add the directory to your path	The directory should be added to your path

Use simple present tense

Avoid future tense (“will”) and complex syntax such as conjunctive mood (“would”, “should”).

Use simple present tense	Instead of future tense or complex syntax
The following command provisions a virtual machine	The following command will provision a virtual machine
If you add this configuration element, the system is open to the Internet	If you added this configuration element, the system would be open to the Internet

Exception: Use future tense if it’s necessary to convey the correct meaning. This requirement is rare.

Address the audience directly

Using “we” in a sentence can be confusing, because the reader may not know whether they’re part of the “we” you’re describing. For example, compare the following two statements:

“In this release we’ve added many new features.”
“In this tutorial we build a flying saucer.”

The words “the developer” or “the user” can be ambiguous. For example, if the reader is building a product that also has users, then the reader does not know whether you’re referring to the reader or the users of their product.

Address the reader directly	Instead of "we", "the user", or "the developer"
Include the directory in your path	The user must make sure that the directory is included in their path
In this tutorial you build a flying saucer	In this tutorial we build a flying saucer

Use short, simple sentences

Keep sentences short. Short sentences are easier to read than long ones. Below are some tips for writing short sentences.

Use fewer words instead of many words that convey the same meaning
Use this	Instead of this
You can use	It is also possible to use
You can	You are able to

Split a single long sentence into two or more shorter ones
Use this	Instead of this
You do not need a running GKE cluster. The deployment process creates a cluster for you	You do not need a running GKE cluster, because the deployment process creates a cluster for you

Use a list instead of a long sentence showing various options
To train a model: Package your program in a Kubernetes container. Upload the container to an online registry. Submit your training job.	To train a model, you must package your program in a Kubernetes container, upload the container to an online registry, and submit your training job.

Use a list instead of a long sentence showing various options

Use this

Instead of this

To train a model:

Package your program in a Kubernetes container.
Upload the container to an online registry.
Submit your training job.

To train a model, you must package your program in a Kubernetes container, upload the container to an online registry, and submit your training job.

Avoid too much text styling

Use bold text when referring to UI controls or other UI elements.

Use code style for:

filenames, directories, and paths
inline code and commands
object field names

Avoid using bold text or capital letters for emphasis. If a page has too much textual highlighting it becomes confusing and even annoying.

Use angle brackets for placeholders

For example:

export KUBEFLOW_USERNAME=<your username>
--email <your email address>

Style your images

The Kubeflow docs recognise Bootstrap classes to style images and other content. The following code snippet shows the typical styling that makes an image show up nicely on the page:

<img src="/docs/images/my-image.png" 
  alt="My image"
  class="mt-3 mb-3 p-3 border border-info rounded">

To see some examples of styled images, take a look at the OAuth setup page. To see the markup, search for .png in the page source.

For more help, see the guide to Bootstrap image styling and the Bootstrap utilities, such as borders.

A detailed style guide

The Google Developer Documentation Style Guide contains detailed information about specific aspects of writing clear, readable, succinct documentation for a developer audience.

Next steps

Take a look at the documentation README for guidance on contributing to the Kubeflow docs.

Docs: Delete using GCP Console

Mon, 01 Jan 0001 00:00:00 +0000

This page shows you how to delete your Kubeflow deployment using Deployment Manager in the GCP Console.

Note: For best results you should use the CLI to delete Kubeflow. Deleting with Deployment Manager as described below can orphan some resources like Cloud Endpoints.

To delete your Kubeflow deployment and reclaim all related resources using the GCP Console:

Open the Deployment Manager in the GCP Console for your project. Deployment Manager lists all the available deployments in your project. Make sure that the selected project is the same as the one you used for your Kubeflow deployment.
Select the Kubeflow deployment with the deployment name you used at the time of creation and click the Delete button at the top.

This action should delete any running nodes in your deployment, delete service accounts that were created for the deployment, and reclaim all resources.

Docs: Central Dashboard

Mon, 01 Jan 0001 00:00:00 +0000

This Kubeflow component has stable status. See the Kubeflow versioning policies.

Your Kubeflow deployment includes a central dashboard that provides quick access to the Kubeflow components deployed in your cluster. The dashboard includes the following features:

Shortcuts to specific actions, a list of recent pipelines and notebooks, and metrics, giving you an overview of your jobs and cluster in one view.
A housing for the UIs of the components running in the cluster, including Pipelines, Katib, Notebooks, and more.
A registration flow that prompts new users to set up their namespace if necessary.

Overview of Kubeflow UIs

The Kubeflow UIs include the following:

Home, a central dashboard for navigation between the Kubeflow components.
Pipelines for a Kubeflow Pipelines dashboard.
Notebook Servers for Jupyter notebooks.
Katib for hyperparameter tuning.
Artifact Store for tracking of artifact metadata.
Manage Contributors for sharing user access across namespaces in the Kubeflow deployment.

The central dashboard looks like this:

Accessing the central dashboard

To access the central dashboard, you need to connect to the Istio gateway that provides access to the Kubeflow service mesh.

How you access the Istio gateway varies depending on how you’ve configured it.

URL pattern with Google Cloud Platform (GCP)

If you followed the guide to deploying Kubeflow on GCP, the Kubeflow central UI is accessible at a URL of the following pattern:

https://<application-name>.endpoints.<project-id>.cloud.goog/

The URL brings up the dashboard illustrated above.

If you deploy Kubeflow with Cloud Identity-Aware Proxy (IAP), Kubeflow uses the Let’s Encrypt service to provide an SSL certificate for the Kubeflow UI. For troubleshooting issues with your certificate, see the guide to monitoring your Cloud IAP setup.

Using kubectl and port-forwarding

If you didn’t configure Kubeflow to integrate with an identity provider then you can port-forward directly to the Istio gateway.

Port-forwarding typically does not work if any of the following are true:

You’ve deployed Kubeflow on GCP using the GCP deployment UI or the default settings with the CLI deployment. (If you want to use port forwarding, you must deploy Kubeflow on an existing Kubernetes cluster using the kfctl_k8s_istio configuration.)
You’ve configured the Istio ingress to only accept HTTPS traffic on a specific domain or IP address.
You’ve configured the Istio ingress to perform an authorization check (for example, using Cloud IAP or Dex).

You can access Kubeflow via kubectl and port-forwarding as follows:

Install kubectl if you haven’t already done so:
- If you’re using Kubeflow on GCP, run the following command on the command line: gcloud components install kubectl.
- Alternatively, follow the kubectl installation guide.

Use the following command to set up port forwarding to the Istio gateway.

export NAMESPACE=istio-system
kubectl port-forward -n ${NAMESPACE} svc/istio-ingressgateway 8080:80

Access the central navigation dashboard at:
```
http://localhost:8080/
```
Depending on how you’ve configured Kubeflow, not all UIs work behind port-forwarding to the reverse proxy.

For some web applications, you need to configure the base URL on which the app is serving.

For example, if you deployed Kubeflow with an ingress serving at https://example.mydomain.com and configured an application to be served at the URL https://example.mydomain.com/myapp, then the app may not work when served on https://localhost:8080/myapp because the paths do not match.

Next steps

Explore the contributor management option where you can set up a single namespace for a shared deployment or configure multi-tenancy for your Kubeflow deployment.
Set up your Jupyter notebooks in Kubeflow.

Docs: Component Specification

Mon, 01 Jan 0001 00:00:00 +0000

This specification describes the container component data model for Kubeflow Pipelines. The data model is serialized to a file in YAML format for sharing.

Below are the main parts of the component definition:

Metadata: Name, description, and other metadata.
Interface (inputs and outputs): Name, type, default value.
Implementation: How to run the component, given the input arguments.

Example of a component specification

A component specification takes the form of a YAML file, component.yaml. Below is an example:

name: xgboost4j - Train classifier
description: Trains a boosted tree ensemble classifier using xgboost4j

inputs:
- {name: Training data}
- {name: Rounds, type: Integer, default: '30', help: Number of training rounds}

outputs:
- {name: Trained model, type: XGBoost model, help: Trained XGBoost model}

implementation:
  container:
    image: gcr.io/ml-pipeline/xgboost-classifier-train@sha256:b3a64d57
    command: [
      /ml/train.py,
      --train-set, {inputPath: Training data},
      --rounds,    {inputValue: Rounds},
      --out-model, {outputPath: Trained model},
    ]

See some examples of real-world component specifications.

Detailed specification (ComponentSpec)

This section describes the ComponentSpec.

Metadata

name: Human-readable name of the component.
description: Description of the component.
metadata: Standard object’s metadata:
- annotations: A string key-value map used to add information about the component. Currently, the annotations get translated to Kubernetes annotations when the component task is executed on Kubernetes. Current limitation: the key cannot contain more that one slash ("/"). See more information in the Kubernetes user guide.
- labels: Deprecated. Use annotations.

Interface

inputs and outputs: Specifies the list of inputs/outputs and their properties. Each input or output has the following properties:
- name: Human-readable name of the input/output. Name must be unique inside the inputs or outputs section, but an output may have the same name as an input.
- description: Human-readable description of the input/output.
- default: Specifies the default value for an input. Only valid for inputs.
- type: Specifies the type of input/output. The types are used as hints for pipeline authors and can be used by the pipeline system/UI to validate arguments and connections between components. Basic types are String, Integer, Float, and Bool. See the full list of types defined by the Kubeflow Pipelines SDK.

Implementation

implementation: Specifies how to execute the component instance. There are two implementation types, container and graph. (The latter is not in scope for this document.) In future we may introduce more implementation types like daemon or K8sResource.
- container: Describes the Docker container that implements the component. A portable subset of the Kubernetes Container v1 spec.
  - image: Name of the Docker image.
  - command: Entrypoint array. The Docker image’s ENTRYPOINT is used if this is not provided. Each item is either a string or a placeholder. The most common placeholders are {inputValue: Input name}, {inputPath: Input name} and {outputPath: Output name}.
  - args: Arguments to the entrypoint. The Docker image’s CMD is used if this is not provided. Each item is either a string or a placeholder. The most common placeholders are {inputValue: Input name}, {inputPath: Input name} and {outputPath: Output name}.
  - env: Map of environment variables to set in the container.
  - fileOutputs: Legacy property that is only needed in cases where the container always stores the output data in some hard-coded non-configurable local location. This property specifies a map between some outputs and local file paths where the program writes the output data files. Only needed for components that have hard-coded output paths. Such containers need to be fixed by modifying the program or adding a wrapper script that copies the output to a configurable location. Otherwise the component may be incompatible with future storage systems.

You can set all other Kubernetes container properties when you use the component inside a pipeline.

Using placeholders for command-line arguments

Consuming input by value

The {inputValue: <Input name>} placeholder is replaced by the value of the input argument:

In component.yaml:

command: [program.py, --rounds, {inputValue: Rounds}]

In the pipeline code:
```
task1 = component1(rounds=150)
```
Resulting command-line code (showing the value of the input argument that has replaced the placeholder):
```
program.py --rounds 150
```

Consuming input by file

The {inputPath: <Input name>} placeholder is replaced by the (auto-generated) local file path where the system has put the argument data passed for the “Input name” input.

In component.yaml:

command: [program.py, --train-set, {inputPath: training_data}]

In the pipeline code:

task2 = component1(training_data=some_task1.outputs['some_data'])

Resulting command-line code (the placeholder is replaced by the generated path):
```
program.py --train-set /inputs/train_data/data
```

Producing outputs

The {outputPath: <Output name>} placeholder is replaced by a (generated) local file path where the component program is supposed to write the output data. The parent directories of the path may or may not not exist. Your program must handle both cases without error.

In component.yaml:

command: [program.py, --out-model, {outputPath: trained_model}]

In the pipeline code:

task1 = component1()
# You can now pass `task1.outputs['trained_model']` to other components as argument.

Resulting command-line code (the placeholder is replaced by the generated path):
```
program.py --out-model /outputs/trained_model/data
```

Docs: Configuring Kubeflow with kfctl and kustomize

Mon, 01 Jan 0001 00:00:00 +0000

This Kubeflow component has stable status. See the Kubeflow versioning policies.

Kfctl is the Kubeflow command-line interface (CLI) that you can use to install and configure Kubeflow.

Kubeflow makes use of kustomize to help customize YAML configurations. With kustomize, you can traverse a Kubernetes manifest to add, remove, or update configuration options without forking the manifest. A manifest is a YAML file containing a description of the applications that you want to include in your Kubeflow deployment.

Overview of kfctl and kustomize

This section describes how kfctl works with kustomize to set up your Kubeflow deployment. You need kustomize 2.0.3 or later.

The Kubeflow deployment process

Kfctl is the Kubeflow CLI that you can use to set up a Kubernetes cluster with Kubeflow installed, or to deploy Kubeflow to an existing Kubernetes cluster. See the Kubeflow getting-started guide for installation instructions based on your deployment scenario.

The kfctl deployment process includes the following commands:

kfctl build - (Optional) Creates configuration files defining the various resources in your deployment but does not deploy Kubeflow. You only need to run kfctl build if you want to edit the resources before running kfctl apply.
kfctl apply - Creates or updates the resources.
kfctl delete - Deletes the resources.

Specifying a configuration file when initializing your deployment

When you install Kubeflow, the deployment process uses one of a few possible YAML configuration files to bootstrap the configuration. You can see all the configuration files on GitHub.

As an example, this guide uses the kfctl_k8s_istio.yaml configuration. For more details about this configuration, see the kfctl_k8s_istio deployment guide.

Typically, you specify the configuration file with a -f <config-file> parameter when you run kfctl build or kfctl apply. The following example uses kfctl build:

# Set KF_NAME to the name of your Kubeflow deployment. You also use this
# value as directory name when creating your configuration directory.
# For example, your deployment name can be 'my-kubeflow' or 'kf-test'.
export KF_NAME=<your choice of name for the Kubeflow deployment>

# Set the path to the base directory where you want to store one or more 
# Kubeflow deployments. For example, /opt/.
# Then set the Kubeflow application directory for this deployment.
export BASE_DIR=<path to a base directory>
export KF_DIR=${BASE_DIR}/${KF_NAME}

# Set the URI of the configuration file to use when deploying Kubeflow. 
# For example:
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_k8s_istio.v1.0.2.yaml"

# Create your Kubeflow configurations:
mkdir -p ${KF_DIR}
cd ${KF_DIR}
kfctl build -V -f ${CONFIG_URI}

Kfctl has now built the configuration files in your Kubeflow application directory (see below) but has not yet deployed Kubeflow. To complete the deployment, run kfctl apply. See the next section on applying the configuration.

Applying the configuration to your Kubeflow cluster

When you first run kfctl build or kfctl apply, kfctl creates a local version of the YAML configuration file, which you can further customize if necessary.

Follow these steps to apply the configurations to your Kubeflow cluster:

Set an environment variable pointing to your local configuration file. For example, this guide uses the kfctl_k8s_istio.v1.0.2.yaml configuration. If you chose a different configuration in the previous step, you must change the file name to reflect your configuration:
```
export CONFIG_FILE=${KF_DIR}/kfctl_k8s_istio.v1.0.2.yaml
```
Apply the configurations:
```
kfctl apply -V -f ${CONFIG_FILE}
```

Your Kubeflow directory layout

Your Kubeflow application directory is the directory where you choose to store your Kubeflow configurations during deployment. This guide refers to the directory as ${KF_DIR}. The directory contains the following files and directories:

${CONFIG_FILE} is a YAML file that stores your primary Kubeflow configuration in the form of a KfDef Kubernetes object.

This file is a copy of the GitHub-based configuration YAML file that you used when deploying Kubeflow.

When you first run kfctl build or kfctl apply, kfctl creates a local version of the configuration file at ${CONFIG_FILE}, which you can further customize if necessary.

The YAML defines each Kubeflow application as a kustomize package.
<platform-name>_config is a directory that contains configurations specific to your chosen platform or cloud provider. For example, gcp_config or aws_config. This directory may or may not be present, depending on your setup.
- The directory is created when you run kfctl build or kfctl apply.
- To customize these configurations, you can modify parameters in your ${CONFIG_FILE}, and then run kfctl apply to apply the configuration to your Kubeflow cluster.
kustomize is a directory that contains Kubeflow application manifests. That is, the directory contains the kustomize packages for the Kubeflow applications that are included in your deployment.
- The directory is created when you run kfctl build or kfctl apply.
- To customize these configurations, you can modify parameters in your ${CONFIG_FILE}, and then run kfctl apply to apply the configuration to your Kubeflow cluster.

How your configuration is generated

The content of your ${CONFIG_FILE} is the result of running kustomize on the base and overlay kustomization.yaml files in the Kubeflow manifests. The overlays reflect the configuration file that you specify when running kfctl build or kfctl apply.

Below are some examples of configuration files:

kfctl_k8s_istio.yaml to install Kubeflow on an existing Kubernetes cluster.
kfctl_istio_dex.yaml to install Kubeflow on an existing Kubernetes cluster with Dex and Istio for authentication.
kfctl_gcp_iap.yaml to create a Google Kubernetes Engine (GKE) cluster with Kubeflow using Cloud Identity-Aware Proxy (Cloud IAP) for access control.

The kustomize package manager in kfctl uses the information in your ${CONFIG_FILE} to traverse the directories under the Kubeflow manifests and to create kustomize build targets based on the manifests.

Installing kustomize

Make sure that you have the minimum required version of kustomize: 2.0.3 or later.

Follow the kustomize installation guide, choosing the relevant options for your operating system. For example, if you’re on Linux:

Set some variables for the operating system:
```
export opsys=linux
```

Download the kustomize binary:

curl -s https://api.github.com/repos/kubernetes-sigs/kustomize/releases |\
grep browser_download |\
grep download/kustomize |\
grep -m 1 $opsys |\
cut -d '"' -f 4 |\
xargs curl -O -L

Unzip the compressed file

tar xzf ./kustomize_v*_${opsys}_amd64.tar.gz

Move the binary:

mkdir -p ${HOME}/bin
mv kustomize ${HOME}/bin/kustomize
chmod u+x ${HOME}/bin/kustomize

Include the kustomize command in your path:
```
export PATH=$PATH:${HOME}/bin
```

Modifying configuration before deployment

Kustomize lets you customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is.

You can use the following command to build and apply kustomize directories:

kustomize build <kustomization_directory> | kubectl apply -f -

The Kubeflow manifests repo contains kustomize build targets, each with a base directory. You can use kustomize to generate YAML output and pass it to kfctl. You can also make changes to the kustomize targets in the manifests repo as needed.

Modifying the configuration of an existing deployment

To customize the Kubeflow resources running within the cluster, you can modify parameters in your ${CONFIG_FILE} file. Then re-run kfctl apply.

For example, to modify settings for the Spartakus usage reporting tool within your Kubeflow deployment:

Edit the configuration file at ${CONFIG_FILE}.

Find and replace the parameter values for spartakus to suit your requirements:

 - kustomizeConfig:
     parameters:
     - initRequired: true
         name: usageId
         value: <randomly-generated-id>
     - initRequired: true
         name: reportUsage
         value: "true"
     repoRef:
         name: manifests
         path: common/spartakus
     name: spartakus

Regenerate and deploy your Kubeflow resources:

cd ${KF_DIR}
kfctl apply -V -f ${CONFIG_FILE}

More examples

For examples of customizing your deployment, see the guide to customizing Kubeflow on GKE.

For information about how Kubeflow uses Spartakus, see the guide to usage reporting.

More about kustomize

Below are some useful kustomize terms, from the kustomize glossary:

base: A combination of a kustomization and resource(s). Bases can be referred to by other kustomizations.
kustomization: Refers to a kustomization.yaml file, or more generally to a directory containing the kustomization.yaml file and all the relative file paths that the YAML file references.
overlay: A combination of a kustomization that refers to a base, and a patch. An overlay may have multiple bases.
patch: General instructions to modify a resource.
resource: Any valid YAML file that defines an object with a kind and a metadata/name field.
target: The argument to kustomize build. For example, kustomize build $TARGET. A target must be a path or a URL to a kustomization. A target can be a base or an overlay.
variant: The outcome of applying an overlay to a base.

Read more about kustomize in the kustomize documentation.

Docs: Data Management

Mon, 01 Jan 0001 00:00:00 +0000

Since a data scientist can build hundreds of different variants of their models, the ability to quickly create new models and save the code and data of each version is critical for faster iterations and better models. Automating workflows and keeping track of the machine learning (ML) code, packages, libraries, data sets and artifacts for each ML pipeline step requires integrated data management systems and processes.

As a leading contributor to Kubeflow, Arrikto incorporates its standards-based, scale-out storage and data management solution (Rok) with Kubeflow. Arrikto’s Rok presents a Kubernetes storage class to Kubeflow and natively integrates with the critical Kubeflow components. Rok’s native integration simplifies operations, boosts performance, and enables best practices for efficient data versioning, packaging, and secure sharing across teams and cloud boundaries.

The screenshot below shows the Snapshot Store option that Rok adds to the left-hand navigation panel in the Kubeflow UI:

To experience the value of Kubeflow and Rok, follow this hands-on tutorial.

Docs: Dockerfile Locations

Mon, 01 Jan 0001 00:00:00 +0000

Image Name	Dockerfile Location
tf_operator	https://github.com/kubeflow/tf-operator/tree/master/build/images/tf_operator
ml-pipeline/persistenceagent	https://github.com/kubeflow/pipelines/tree/master/backend
ml-pipeline/scheduledworkflow	https://github.com/kubeflow/pipelines/tree/master/backend
ml-pipeline/frontend	https://github.com/kubeflow/pipelines/blob/master/frontend/Dockerfile
katib/katib-controller	https://github.com/kubeflow/katib/tree/master/cmd/katib-controller/v1alpha3/Dockerfile
katib/katib-ui	https://github.com/kubeflow/katib/tree/master/cmd/ui/v1alpha3/Dockerfile
katib/katib-db-manager	https://github.com/kubeflow/katib/tree/master/cmd/db-manager/v1alpha3/Dockerfile
katib/suggestion-skopt	https://github.com/kubeflow/katib/blob/master/cmd/suggestion/skopt/v1alpha3/Dockerfile
katib/suggestion-chocolate	https://github.com/kubeflow/katib/blob/master/cmd/suggestion/chocolate/v1alpha3/Dockerfile
katib/suggestion-hyperopt	https://github.com/kubeflow/katib/blob/master/cmd/suggestion/hyperopt/v1alpha3/Dockerfile
katib/suggestion-hyperband	https://github.com/kubeflow/katib/blob/master/cmd/suggestion/hyperband/v1alpha3/Dockerfile
katib/suggestion-nasrl	https://github.com/kubeflow/katib/blob/master/cmd/suggestion/nas/enas/v1alpha3/Dockerfile
katib/file-metricscollector	https://github.com/kubeflow/katib/blob/master/cmd/metricscollector/v1alpha3/file-metricscollector/Dockerfile
katib/tfevent-metricscollector	https://github.com/kubeflow/katib/blob/master/cmd/metricscollector/v1alpha3/tfevent-metricscollector/Dockerfile
datawire/ambassador	https://github.com/datawire/ambassador/blob/master/builder/Dockerfile
tensorflow-1.13.1-notebook-cpu	https://github.com/kubeflow/kubeflow/blob/master/components/tensorflow-notebook-image/Dockerfile
jupyter-web-app	https://github.com/kubeflow/kubeflow/blob/master/components/jupyter-web-app/Dockerfile
profile-controller	https://github.com/kubeflow/kubeflow/tree/master/components/profile-controller
notebook-controller	https://github.com/kubeflow/kubeflow/tree/master/components/notebook-controller

Docs: Events Calendar

Mon, 01 Jan 0001 00:00:00 +0000

This is a nonexhaustive list of events (in reverse chronological order) with talks and workshops about Kubeflow. Please edit this page and send a pull request, or raise a GitHub issue, if something is missing or incorrect.

Kubeflow Doc Sprint - Sunnyvale and online, 10-12 February 2020
- Monday 10 February: Welcome; docs/code development; learning sessions.
- Tuesday 11 February: Sprint updates; docs/code development; learning sessions.
- Wednesday 12 February: Docs/code development; sprint demos.
Kubeflow Doc Sprint, 10-12 July 2019. See the results of the July 2019 doc sprint.
Google Cloud Next, San Francisco, 9-11 April, 2019
- Undoing Human Bias at Scale With Kubeflow: John Bohannon, Michelle Casbon
Kubeflow Contributor Summit 2019, Sunnyvale, CA, 12-13 March, 2019
- Kubeflow and the ML Landscape: Jeremy Lewi, Google
  - Slides
- Updates from the PM Working Group:
  - PMs: David Aronchick, Josh Bottum, Carmine Rimi
  - Slides
- Building Together: Community in Kubeflow: Thea Lamkin, Google
  - Slides
- Experiment Tracking with Kubeflow, Lukas Biewald, Weights & Biases
  - Slides
- Multi-User Environments, Kam Kasravi & Ebi Shahbazi, Intel
  - Slides
- On-Premise: Debo Dutta, Cisco
  - Slides
- MiniKF: Local Kubeflow: Vangelis Koukis, Arrikto
  - Slides
- TensorFlow 2.0 is coming!: Paige Bailey, Google
  - Slides
- Overview of Kubeflow Pipelines: Pavel Dournov, Google
  - Slides
- Feast: Feature Storage for Machine Learning: Tim Sell, Google
  - Slides
- Understanding the Earth: Machine Learning with Kubeflow Pipelines: Faustine Li, Descartes Labs
  - Slides
- Kubeflow Experiments at LinkedIn: Tengfei Mu & Abin Shahab, LinkedIn
  - Slides
- Ask the User: Data Science Panel, Moderator: Karthik Ramasamy, Google
  - Panelists: Ting Chen, Emmanuel Ameisen, Scott Leishman, Caio Soares, June Andrews, Jun Wang
- Simple, GPU-Accelerated Kubeflow Pipeline: Ananth Sankarasubramanian, NVIDIA
  - Slides
- Fairing: Matt Rickard, Google
  - Slides
- ModelDB: Open-source model management: Manasi Vartak, Verta.ai
  - Slides
- Katib & Hyperparameter Tuning: Richard Liu, Google
  - Slides
- Kubeflow User Experience: Derek Ferguson, JP Morgan Chase
  - Slides
- Kubeflow & TFX: Kevin Haas, Google
  - Slides
- Kubeflow inference on Knative: Dan Sun, Bloomberg
  - Slides
- Arena: Yang Che, Alibaba
  - Slides
  - Demo video
- Examples, Codelabs, & Demos: Michelle Casbon, Google
  - Slides
Data Day Texas, Austin, 26 January, 2019
- Kubeflow: Portable Machine Learning on Kubernetes: Michelle Casbon
  - Slides
AI NEXTCon, Seattle, 23-26 January, 2019
- Build and Manage Machine Learning Pipelines: Amy Unruh
KubeCon, Seattle, 11-13 December, 2018
- Deep Dive: Kubeflow BoF: Jeremy Lewi, David Aronchick
  - Slides
  - Video
- Eco-Friendly ML: How the Kubeflow Ecosystem Bootstrapped Itself: Peter McKinnon
  - Slides
  - Video
- Machine Learning as Code: Jay Smith
  - Slides
  - Video
- Natural Language Code Search for GitHub Using Kubeflow: Jeremy Lewi, Hamel Husain
  - Slides
  - Video
- Workshop: Kubeflow End-to-End: GitHub Issue Summarization: Amy Unruh, Michelle Casbon
  - Codelab
  - Slides
  - Video
Women in ML & Data Science, Melbourne, 5 December, 2018
- Panel: Michelle Casbon
YOW!, Melbourne, 4-7 December, 2018
- Kubeflow Explained: NLP Architectures on Kubernetes: Michelle Casbon
  - Slides
YOW!, Brisbane, 3-4 December, 2018
- Kubeflow Explained: NLP Architectures on Kubernetes: Michelle Casbon
  - Slides
YOW!, Sydney, 27-30 November, 2018
- Kubeflow Explained: NLP Architectures on Kubernetes: Michelle Casbon
  - Slides
Scale By the Bay, San Francisco, 15-17 November, 2018
- Data Engineering & AI Panel: Michelle Casbon
  - Video
KubeCon, Shanghai, 13-15 November, 2018
- A Tale of Using Kubeflow to Make the Electricity Smarter in China: Julia Han, Xin Zhang
  - Slides
  - Video
- A Year of Democratizing ML With Kubernetes & Kubeflow: David Aronchick, Fei Xue
  - Slides
  - Video
- Benchmarking Machine Learning Workloads on Kubeflow: Xinyuan Huang, Ce Gao
  - Slides
  - Video
- CI/CD Pipelines & Machine Learning: Jeremy Lewi
  - Slides
  - Video
- Kubeflow From the End User’s Perspective: Xin Zhang
  - Video
- Kubernetes CI/CD Hacks with KicroK8s and Kubeflow: Land Lu, Zhang Lei Mao
  - Slides
  - Video
- Machine Learning on Kubernetes BoF: David Aronchick
  - Video
- Operating Deep Learning Pipelines Anywhere Using Kubeflow: Jörg Schad, Gilbert Song
  - Slides
  - Video
DevFest, Seattle, 3 November, 2018
- Kubeflow End to End: Amy Unruh
Data@Scale, Boston, 25 October, 2018
- Women in Engineering Panel: Michelle Casbon
- Kubeflow: Portable Machine Learning on Kubernetes: Michelle Casbon
  - Video
Kafka Summit, San Francisco, 16-17 October, 2018
O’Reilly AI Conference, London, 08-11 October, 2018
- Machine Learning at Scale with Kubernetes: Chris Cho
Cloud-Native, Docker, and Kubernetes Summit, Dallas, 12 September, 2018
- Deploying Machine Learning Workloads in Kubernetes clusters that support GPUs: Michael Iatrou
O’Reilly Strata Data, New York, 11-13 September, 2018
- From Training to Serving: Deploying TensorFlow Models with Kubernetes: Brian Foo, Holden Karau, Jay Smith
- Kubeflow Explained: Portable Machine Learning on Kubernetes: Michelle Casbon
O’Reilly AI Conference, San Francisco, 4-7 September, 2018
- TensorFlow Days: Kubeflow: Portable Machine Learning on Kubernetes: Michelle Casbon
Open Source Summit, Vancouver, 29-31 August, 2018
- Elastic AI Pipeline with Kubeflow for Intelligent SKU Management in a Large Chinese Retailer: Xin Zhang
- Introducing Kubeflow: A System for Deploying ML/AI on Kubernetes: Trevor Grant, Holden Karau
JupyterCon, New York, 21-25 August, 2018
- Machine Learning at Scale with Kubernetes: Chris Cho
Google Next, San Francisco, 24-26 July, 2018
- AI Platform Showcase demo: Dan Sanche
  - Video
- Machine Learning Made Easy: How to Build Flexible, Portable ML Stacks with Kubeflow and Elastifile: David Aronchick, Allon Cohen
  - Video
- Spotlight Lab: Introduction to Kubeflow on Google Kubernetes Engine: Michelle Casbon
  - Codelab
- Spotlight Lab: Kubeflow End-to-End: GitHub Issue Summarization: Michelle Casbon
  - Codelab
O’Reilly Open Source Convention, Portland, 16-19 July, 2018
- TensorFlow Day: Hassle-free, scalable machine learning with Kubeflow: Barbara Fusinska
SciPy, Austin, 09-15 July, 2018
- Kubeflow: Pythonic Machine Learning at Scale on Kubernetes: David Aronchick, Paige Bailey
  - Video
Container Day, Paris, 26 June, 2018
- Keynote: Cloud Native ML with Kubeflow: David Aronchick, Chris Cho
  - Video
Dockercon, San Francisco, 12 - 15 June, 2018
- Keynote: Moby’s Cool Hacks: David Aronchick, Michelle Casbon
  - Video
QCon, São Paulo, 9-11 May, 2018
- Architecture of an NLP Deployment: Michelle Casbon
  - Video
KubeCon, Copenhagen, 2-4 May, 2018
- Interview: Building applications on Kubeflow: Jeremy Lewi
  - Video
- Keynote: Cloud Native ML on Kubernetes: David Aronchick, Vishnu Kannan
  - Video
  - Slides
- Kubeflow Deep Dive: David Aronchick, Jeremy Lewi
  - Video
  - Slides
- Kubeflow Intro: Michał Jastrzębski, Ala Raddaoui
  - Video
  - Slides
- Serving ML Models at Scale with Seldon & Kubeflow: Clive Cox
  - Video
  - Slides
- Conquering a Kubeflow Kubernetes Cluster with ksonnet, Ark, & Sonobuoy: David Aronchick, Kris Nova
  - Video
- Building ML Products with Kubeflow: Jeremy Lewi, Stephan Fabel
  - Video
  - Slides
- Compliant Data Management & Machine Learning on Kubernetes: Daniel Whitenack
  - Video
  - Slides
- Bringing Your Data Pipeline into the Machine Learning Era: Chris Gaun, Jörg Schad
  - Video
KubeCon, Austin, 6-8 December, 2017
- “Hot Dog or Not Hot Dog” at Scale - Kubernetes & Machine Learning: David Aronchick, Vishnu Kannan
  - Video

Docs: Features of Kubeflow on GCP

Mon, 01 Jan 0001 00:00:00 +0000

Running Kubeflow on GCP brings you the following features:

You use Deployment Manager to declaratively manage all non-Kubernetes resources (including the GKE cluster). Deployment Manager is easy to customize for your particular use case.
You can take advantage of GKE autoscaling to scale your cluster horizontally and vertically to meet the demands of machine learning (ML) workloads with large resource requirements.
Cloud Identity-Aware Proxy (Cloud IAP) makes it easy to securely connect to Jupyter and other web apps running as part of Kubeflow.
Stackdriver provides persistent logs to aid in debugging and troubleshooting.
You can use GPUs and Cloud TPU to accelerate your workload.

Next steps

Deploy Kubeflow if you haven’t already done so.
Run a full ML workflow on Kubeflow, using the end-to-end MNIST tutorial or the GitHub issue summarization example.

Docs: Install Kubeflow Fairing

Mon, 01 Jan 0001 00:00:00 +0000

You can use Kubeflow Fairing to build, train, and deploy machine learning (ML) models in a hybrid cloud environment directly from Python code or a Jupyter notebook. This guide describes how to install Kubeflow Fairing in your development environment for local development, or development in a hosted notebook.

Using Kubeflow Fairing with Kubeflow notebooks

Kubeflow notebook servers that are built from one of the standard Jupyter Docker images include Kubeflow Fairing and come preconfigured for using Kubeflow Fairing to run training jobs on your Kubeflow cluster.

If you use a Kubeflow notebook server that was built from a custom Jupyter Docker image as your development environment, follow the instruction on setting up Kubeflow Fairing in a hosted notebook environment.

Set up Kubeflow Fairing for local development

Follow these instructions to set up Kubeflow Fairing for local development. This guide has been tested on Linux and Mac OS X. Currently, this guide has not been tested on Windows.

Set up Python

You need Python 3.6 or later to use Kubeflow Fairing. To check if you have Python 3.6 or later installed, run the following command:
```
python3 -V
```
The response should be something like this:
```
Python 3.6.5
```
If you do not have Python 3.6 or later, you can download Python from the Python Software Foundation.
Use virtualenv to create a virtual environment to install Kubeflow Fairing in. To check if you have virtualenv installed, run the following command:
```
which virtualenv
```
The response should be something like this:
```
/usr/bin/virtualenv
```
If you do not have virtualenv, use pip3 to install virtualenv.
```
pip3 install virtualenv
```
Create a new virtual environment, and activate it.
```
virtualenv venv --python=python3
source venv/bin/activate
```

Install Kubeflow Fairing

Run the following command to install Kubeflow Fairing in your virtual environment.

pip install kubeflow-fairing

After the install is complete, the fairing python package is available. Run the following command to verify that Kubeflow Fairing is installed:

pip show kubeflow-fairing

The response should be something like this:

Name: kubeflow-fairing
Version: 0.6.0
Summary: Kubeflow Fairing Python SDK.
Home-page: https://github.com/kubeflow/fairing
Author: Kubeflow Authors
Author-email: hejinchi@cn.ibm.com
License: Apache License Version 2.0
Location: <path-to-kubeflow-fairing>
Requires: notebook, future, docker, tornado, cloudpickle, oauth2client, numpy, requests, setuptools, httplib2, google-auth, google-api-python-client, urllib3, boto3, azure, six, kubernetes, google-cloud-storage

Docker setup

Kubeflow Fairing uses Docker to package your code. Run the following command to verify if Docker is installed and running:

docker ps

If you receive the docker: command not found message, install Docker.
If you receive the Error response from daemon: Bad response from Docker engine message, restart your docker daemon.
If you are using Linux and you use sudo to access Docker, follow these steps to add your user to the docker group. Note, the docker group grants privileges equivalent to the root user. To learn more about how this affects security in your system, see the guide to the Docker daemon attack surface.

Configure Kubeflow Fairing

To configure Kubeflow Fairing with access to an environment that you would like to use for training and deployment, follow the instructions in the guide to configuring Kubeflow Fairing.

Set up Kubeflow Fairing in a hosted Jupyter notebook

Follow these instructions to set up Kubeflow Fairing in a hosted Jupyter notebook.

If you are using a Kubeflow notebook server that was built from one of the standard Jupyter Docker images, your notebooks environment has been preconfigured for training and deploying ML models with Kubeflow Fairing and no additional installation steps are required.

Prerequisites

Check the following prerequisites to verify that Kubeflow Fairing is compatible with your hosted notebook environment.

In the Jupyter notebooks user interface, click File > New > Terminal in the menu to start a new terminal session in your notebook environment.
You need Python 3.6 or later to use Kubeflow Fairing. To check if you have Python 3.6 or later installed, run the following command in your terminal session:
```
python3 -V
```
The response should be something like this:
```
Python 3.6.5
```
Kubeflow Fairing uses Docker to package your code. Run the following command in your terminal session to verify if Docker is installed and running in your notebook environment:
```
docker ps
```
- If you receive the docker: command not found message, install Docker.
- If you receive the Error response from daemon: Bad response from Docker engine message, restart your docker daemon.
- If you are using Linux and you use sudo to access Docker, follow these steps to add your user to the docker group. Note, the docker group grants privileges equivalent to the root user. To learn more about how this affects security in your system, see the guide to the Docker daemon attack surface.

Install Kubeflow Fairing

In the Jupyter notebooks user interface, click File > New > Terminal in the menu to start a new terminal session in your notebook environment.

Run the following command to install Kubeflow Fairing.

pip3 install kubeflow-fairing

After successful installation, the fairing python package should be available. Run the following command to verify that Kubeflow Fairing is installed:

pip3 show kubeflow-fairing

The response should be something like this:

Name: kubeflow-fairing
Version: 0.6.0
Summary: Kubeflow Fairing Python SDK.
Home-page: https://github.com/kubeflow/fairing
Author: Kubeflow Authors
Author-email: hejinchi@cn.ibm.com
License: Apache License Version 2.0
Location: <path-to-kubeflow-fairing>
Requires: notebook, future, docker, tornado, cloudpickle, oauth2client, numpy, requests, setuptools, httplib2, google-auth, google-api-python-client, urllib3, boto3, azure, six, kubernetes, google-cloud-storage

Configure Kubeflow Fairing

To configure Kubeflow Fairing with access to the environment you would like to use for training and deployment, follow the instructions in the guide to configuring Kubeflow Fairing.

Next steps

Configure your Kubeflow Fairing development environment with access to run training jobs remotely.
Follow the samples and tutorials to learn more about how to run training jobs remotely with Kubeflow Fairing.

Docs: Installation Options for Kubeflow Pipelines

Mon, 01 Jan 0001 00:00:00 +0000

Kubeflow Pipelines offers a few installation options. This page describes the options and the features available with each option:

A standalone Kubeflow Pipelines deployment.
Kubeflow Pipelines as part of a full Kubeflow deployment.
Alpha: GCP Hosted ML Pipelines.

Kubeflow Pipelines Standalone

Use this option to deploy Kubeflow Pipelines to an on-premises or cloud Kubernetes cluster, without the other components of Kubeflow. To deploy Kubeflow Pipelines Standalone, you use kustomize manifests only. This process makes it simpler to customize your deployment and to integrate Kubeflow Pipelines into an existing Kubernetes cluster.

Installation guide

Kubeflow Pipelines Standalone deployment guide

Interfaces

Kubeflow Pipelines UI
Kubeflow Pipelines SDK
Kubeflow Pipelines API

Notes on specific features

After deployment, your Kubernetes cluster contains Kubeflow Pipelines only. It does not include the other Kubeflow components. For example, to use a Jupyter Notebook, you must use a local notebook or a hosted notebook in a cloud service such as the AI Platform Notebooks.

Full Kubeflow deployment

Use this option to deploy Kubeflow Pipelines to your local machine, on-premises, or to a cloud, as part of a full Kubeflow installation.

Installation guide: Kubeflow installation guide

Interfaces :

Kubeflow UI
Kubeflow Pipelines UI within or outside the Kubeflow UI
Kubeflow Pipelines SDK
Kubeflow Pipelines API
Other Kubeflow APIs

Notes on specific features: After deployment, your Kubernetes cluster includes all the Kubeflow components. For example, you can use the Jupyter notebook services deployed with Kubeflow to create one or more notebook servers in your Kubeflow cluster.

GCP Hosted ML Pipelines

Alpha release

GCP Hosted ML Pipelines is currently in Alpha with limited support. The Kubeflow team is interested in any feedback you may have, in particular on the usability of the feature. To get access to the Alpha release, email kfp-mkp-alpha-feedback@google.com. You can raise any issues or discussion items in the Kubeflow Pipelines issue tracker.

Use this option to deploy Kubeflow Pipelines to Google Kubernetes Engine (GKE) from GCP Marketplace. You can deploy Kubeflow Pipelines to an existing or new GKE cluster and manage your cluster within GCP.

Installation guide

Deploy Kubeflow Pipelines from Google Cloud Marketplace

Interfaces

GCP Console for managing the Kubeflow Pipelines cluster and other GCP services.
Kubeflow Pipelines UI via the Open Pipelines Dashboard link in the GCP Console
Kubeflow Pipelines SDK in Cloud Notebooks

Notes on specific features

After deployment, your Kubernetes cluster contains Kubeflow Pipelines only. It does not include the other Kubeflow components. For example, to use a Jupyter Notebook, you can use AI Platform Notebooks.

Docs: Introduction to Katib

Mon, 01 Jan 0001 00:00:00 +0000

Beta

This Kubeflow component has beta status. See the Kubeflow versioning policies. The Kubeflow team is interested in your feedback about the usability of the feature.

Use Katib for automated tuning of your machine learning (ML) model’s hyperparameters and architecture.

This page introduces the concepts of hyperparameter tuning, neural architecture search, and the Katib system as a component of Kubeflow.

Hyperparameters and hyperparameter tuning

Hyperparameters are the variables that control the model training process. For example:

Learning rate.
Number of layers in a neural network.
Number of nodes in each layer.

Hyperparameter values are not learned. In other words, in contrast to the node weights and other training parameters, the model training process does not adjust the hyperparameter values.

Hyperparameter tuning is the process of optimizing the hyperparameter values to maximize the predictive accuracy of the model. If you don’t use Katib or a similar system for hyperparameter tuning, you need run many training jobs yourself, manually adjusting the hyperparameters to find the optimal values.

Automated hyperparameter tuning works by optimizing a target variable, also called the objective metric, that you specify in the configuration for the hyperparameter tuning job. A common metric is the model’s accuracy in the validation pass of the training job (validation-accuracy). You also specify whether you want the hyperparameter tuning job to maximize or minimize the metric.

For example, the following graph from Katib shows the level of accuracy for various combinations of hyperparameter values (learning rate, number of layers, and optimizer):

(To run the example that produced this graph, follow the getting-started guide.)

Katib runs several training jobs (known as trials) within each hyperparameter tuning job (experiment). Each trial tests a different set of hyperparameter configurations. At the end of the experiment, Katib outputs the optimized values for the hyperparameters.

Neural architecture search

Alpha version

Neural architecture search is currently in alpha with limited support. The Kubeflow team is interested in any feedback you may have, in particular with regards to usability of the feature. You can log issues and comments in the Katib issue tracker.

In addition to hyperparameter tuning, Katib offers a neural architecture search (NAS) feature. You can use the NAS to design your artificial neural network, with a goal of maximizing the predictive accuracy and performance of your model.

NAS is closely related to hyperparameter tuning. Both are subsets of automated machine learning (AutoML). While hyperparameter tuning optimizes the model’s hyperparameters, a NAS system optimizes the model’s structure, node weights, and hyperparameters.

NAS technology in general uses various techniques to find the optimal neural network design. The NAS in Katib uses the reinforcement learning technique.

You can submit Katib jobs from the command line or from the UI. (Read more about the Katib interfaces later on this page.) The following screenshot shows part of the form for submitting a NAS job from the Katib UI:

The Katib project

Katib is a Kubernetes-based system for hyperparameter tuning and neural architecture search. Katib supports a number of ML frameworks, including TensorFlow, MXNet, PyTorch, XGBoost, and others.

The Katib project is open source. The developer guide is a good starting point for developers who want to contribute to the project.

Katib interfaces

You can use the following interfaces to interact with Katib:

A web UI that you can use to submit experiments and to monitor your results. See the getting-started guide for information on how to access the UI. The Katib home page within Kubeflow looks like this:
A REST API. See the API reference on GitHub.
Command-line interfaces (CLIs):
- Kfctl is the Kubeflow CLI that you can use to install and configure Kubeflow. Read about kfctl in the guide to configuring Kubeflow.
- The Kubernetes CLI, kubectl, is useful for running commands against your Kubeflow cluster. Read about kubectl in the Kubernetes documentation.

Katib concepts

This section describes the terms used in Katib.

Experiment

An experiment is a single tuning run, also called an optimization run.

You specify configuration settings to define the experiment. The following are the main configurations:

Objective: What you want to optimize. This is the objective metric, also called the target variable. A common metric is the model’s accuracy in the validation pass of the training job (validation-accuracy). You also specify whether you want the hyperparameter tuning job to maximize or minimize the metric.
Search space: The set of all possible hyperparameter values that the hyperparameter tuning job should consider for optimization, and the constraints for each hyperparameter. Other names for search space include feasible set and solution space. For example, you may provide the names of the hyperparameters that you want to optimize. For each hyperparameter, you may provide a minimum and maximum value or a list of allowable values.
Search algorithm: The algorithm to use when searching for the optimal hyperparameter values.

For details of how to define your experiment, see the guide to running an experiment.

Suggestion

A suggestion is a set of hyperparameter values that the hyperparameter tuning process has proposed. Katib creates a trial to evaluate the suggested set of values.

Trial

A trial is one iteration of the hyperparameter tuning process. A trial corresponds to one worker job instance with a list of parameter assignments. The list of parameter assignments corresponds to a suggestion.

Each experiment runs several trials. The experiment runs the trials until it reaches either the objective or the configured maximum number of trials.

Worker job

The worker job is the process that runs to evaluate a trial and calculate its objective value.

The worker job can be one of the following types:

Kubernetes Job (does not support distributed execution).
Kubeflow TFJob (supports distributed execution).
Kubeflow PyTorchJob (supports distributed execution).

By offering the above worker job types, Katib supports multiple ML frameworks.

Next steps

Follow the getting-started guide to set up Katib and run some hyperparameter tuning examples.

Docs: Introduction to Multi-user Isolation

Mon, 01 Jan 0001 00:00:00 +0000

This Kubeflow component has stable status. See the Kubeflow versioning policies.

In a production environment, it is often necessary to share the same pool of resources across different teams and users. These different users need a reliable way to isolate and protect their own resources, without accidentally viewing or changing each other’s resources.

Kubeflow v1.0.2 supports multi-user isolation, which applies access control over namespaces and user-created resources in a deployment. This feature provides the users with the convenience of clutter-free browsing of notebooks, training jobs, serving deployments and other resources. The isolation mechanisms also prevent accidental deletion/modification of resources of other users in the deployment.

Note that the isolation support in Kubeflow doesn’t provide any hard security guarantees against malicious attempts by users to infiltrate other user’s profiles.

Key concepts

Administrator: An administrator is someone who creates and maintains the Kubeflow cluster. This person has the permission to grant access permissions to others.

User: A user is someone who has access to some set of resources in the cluster. A user needs to be granted access permissions by the administrator.

Profile: A profile is a grouping of all Kubernetes clusters owned by a user.

Current integration and limitations

The Jupyter notebooks service is the first application to be fully integrated with multi-user isolation. Access to the notebooks and the creation of notebooks is controlled by the profile access policies set by the administrator or the owners of the profiles. Resources created by the notebooks (for example, training jobs and deployments) also inherit the same access.

Metadata and Pipelines or any other applications currently don’t have full fledged integration with isolation, though they do have access to the user identity through the headers of the incoming requests. It’s up to the individual applications to use the available identity and isolation features in a way that makes sense for each application.

On Google Cloud Platform (GCP), the authentication and identify token is generated by GCP IAM and carried through the requests as a JWT Token in the request header. Other cloud providers can have a similar header to provide identity information.

For on-premises deployments, Kubeflow uses Dex as a federated OpenID connection provider and can be integrated with LDAP or Active Directory to provide authentication and identity services.

Next steps

Understand the detailed design of Kubeflow’s multi-user isolation feature.
Learn how to use multi-user isolation and profiles.

Docs: Introduction to the Pipelines SDK

Mon, 01 Jan 0001 00:00:00 +0000

Beta

This Kubeflow component has beta status. See the Kubeflow versioning policies. The Kubeflow team is interested in your feedback about the usability of the feature.

The Kubeflow Pipelines SDK provides a set of Python packages that you can use to specify and run your machine learning (ML) workflows. A pipeline is a description of an ML workflow, including all of the components that make up the steps in the workflow and how the components interact with each other.

SDK packages

The Kubeflow Pipelines SDK includes the following packages:

kfp.compiler includes classes and methods for building Docker container images for your pipeline components. Methods in this package include, but are not limited to, the following:
- kfp.compiler.Compiler.compile compiles your Python DSL code into a single static configuration (in YAML format) that the Kubeflow Pipelines service can process. The Kubeflow Pipelines service converts the static configuration into a set of Kubernetes resources for execution.
- kfp.compiler.build_docker_image builds a container image based on a Dockerfile and pushes the image to a URI. In the parameters, you provide the path to a Dockerfile containing the image specification, and the URI for the target image (for example, a container registry).
- kfp.compiler.build_python_component builds a container image for a pipeline component based on a Python function, and pushes the image to a URI. In the parameters, you provide the Python function that does the work of the pipeline component, a Docker image to use as a base image, and the URI for the target image (for example, a container registry).
kfp.components includes classes and methods for interacting with pipeline components. Methods in this package include, but are not limited to, the following:
- kfp.components.func_to_container_op converts a Python function to a pipeline component and returns a factory function. You can then call the factory function to construct an instance of a pipeline task (ContainerOp) that runs the original function in a container.
- kfp.components.load_component_from_file loads a pipeline component from a file and returns a factory function. You can then call the factory function to construct an instance of a pipeline task (ContainerOp) that runs the component container image.
- kfp.components.load_component_from_url loads a pipeline component from a URL and returns a factory function. You can then call the factory function to construct an instance of a pipeline task (ContainerOp) that runs the component container image.
kfp.dsl contains the domain-specific language (DSL) that you can use to define and interact with pipelines and components. Methods, classes, and modules in this package include, but are not limited to, the following:
- kfp.dsl.ContainerOp represents a pipeline task (op) implemented by a container image.
- kfp.dsl.PipelineParam represents a pipeline parameter that you can pass from one pipeline component to another. See the guide to pipeline parameters.
- kfp.dsl.component is a decorator for DSL functions that returns a pipeline component. (ContainerOp).
- kfp.dsl.pipeline is a decorator for Python functions that returns a pipeline.
- kfp.dsl.python_component is a decorator for Python functions that adds pipeline component metadata to the function object.
- kfp.dsl.types contains a list of types defined by the Kubeflow Pipelines SDK. Types include basic types like String, Integer, Float, and Bool, as well as domain-specific types like GCPProjectID and GCRPath. See the guide to DSL static type checking.
- kfp.dsl.ResourceOp represents a pipeline task (op) which lets you directly manipulate Kubernetes resources (create, get, apply, …).
- kfp.dsl.VolumeOp represents a pipeline task (op) which creates a new PersistentVolumeClaim (PVC). It aims to make the common case of creating a PersistentVolumeClaim fast.
- kfp.dsl.VolumeSnapshotOp represents a pipeline task (op) which creates a new VolumeSnapshot. It aims to make the common case of creating a VolumeSnapshot fast.
- kfp.dsl.PipelineVolume represents a volume used to pass data between pipeline steps. ContainerOps can mount a PipelineVolume either via the constructor’s argument pvolumes or add_pvolumes() method.
kfp.Client contains the Python client libraries for the Kubeflow Pipelines API. Methods in this package include, but are not limited to, the following:
- kfp.Client.create_experiment creates a pipeline experiment and returns an experiment object.
- kfp.Client.run_pipeline runs a pipeline and returns a run object.
- kfp.Client.pipeline_uploads.upload_pipeline_version uploads a local file to create a pipeline version. Follow an example to learn more about creating a pipeline version
kfp.notebook
KFP extension modules include classes and functions for specific platforms on which you can use Kubeflow Pipelines. Examples include utility functions for on premises, Google Cloud Platform (GCP), Amazon Web Services (AWS), and Microsoft Azure.
KFP diagnose_me modulesinclude classes and functions that help with environment diagnostic tasks.
- kfp.cli.diagnose_me.dev_env reports on diagnostic metadata from your development environment, such as your python library version.
- kfp.cli.diagnose_me.kubernetes_cluster reports on diagnostic data from your Kubernetes cluster, such as Kubernetes secrets.
- kfp.cli.diagnose_me.gcp reports on diagnostic data related to your GCP environment.

KFP CLI tool

The KFP CLI tool enables you to use a subset of the Kubeflow Pipelines SDK directly from the command line. The KFP CLI tool provides the following commands:

kfp diagnose_me runs environment diagnostic with specified parameters.
- --json - Indicates that this command must return its results as JSON. Otherwise, results are returned in human readable format.
- --namespace TEXT - Specifies the Kubernetes namespace to use. all-namespaces is the default value.
- --project-id TEXT - For GCP deployments, this value specifies the GCP project to use. If this value is not specified, the environment default is used.
kfp pipeline <COMMAND> provides the following commands to help you manage pipelines.
- get - Gets detailed information about a Kubeflow pipeline from your Kubeflow Pipelines cluster.
- list - Lists the pipelines that have been uploaded to your Kubeflow Pipelines cluster.
- upload - Uploads a pipeline to your Kubeflow Pipelines cluster.
kfp run <COMMAND> provides the following commands to help you manage pipeline runs.
- get - Displays the details of a pipeline run.
- list - Lists recent pipeline runs.
- submit - Submits a pipeline run.
kfp --endpoint <ENDPOINT> - Specifies the endpoint that the KFP CLI should connect to.

Installing the SDK

Follow the guide to installing the Kubeflow Pipelines SDK.

Building pipelines and components

This section summarizes the ways you can use the SDK to build pipelines and components:

The diagrams provide a conceptual guide to the relationships between the following concepts:

Your Python code
A pipeline component
A Docker container image
A pipeline

Creating components from existing application code

This section describes how to create a component and a pipeline outside your Python application, by creating components from existing containerized applications. This technique is useful when you have already created a TensorFlow program, for example, and you want to use it in a pipeline.

Below is a more detailed explanation of the above diagram:

Write your application code, my-app-code.py. For example, write code to transform data or train a model.
Create a Docker container image that packages your program (my-app-code.py) and upload the container image to a registry. To build a container image based on a given Dockerfile, you can use the Docker command-line interface or the kfp.compiler.build_docker_image method from the Kubeflow Pipelines SDK.
Write a component function using the Kubeflow Pipelines DSL to define your pipeline’s interactions with the component’s Docker container. Your component function must return a kfp.dsl.ContainerOp. Optionally, you can use the kfp.dsl.component decorator to enable static type checking in the DSL compiler. To use the decorator, you can add the @kfp.dsl.component annotation to your component function:
```
@kfp.dsl.component
def my_component(my_param):
  ...
  return kfp.dsl.ContainerOp(
    name='My component name',
    image='gcr.io/path/to/container/image'
  )
```
Write a pipeline function using the Kubeflow Pipelines DSL to define the pipeline and include all the pipeline components. Use the kfp.dsl.pipeline decorator to build a pipeline from your pipeline function. To use the decorator, you can add the @kfp.dsl.pipeline annotation to your pipeline function:
```
@kfp.dsl.pipeline(
  name='My pipeline',
  description='My machine learning pipeline'
)
def my_pipeline(param_1: PipelineParam, param_2: PipelineParam):
  my_step = my_component(my_param='a')
```
Compile the pipeline to generate a compressed YAML definition of the pipeline. The Kubeflow Pipelines service converts the static configuration into a set of Kubernetes resources for execution.

To compile the pipeline, you can choose one of the following options:
- Use the kfp.compiler.Compiler.compile method:
```
kfp.compiler.Compiler().compile(my_pipeline,  
  'my-pipeline.zip')
```
- Alternatively, use the dsl-compile command on the command line.
```
dsl-compile --py [path/to/python/file] --output my-pipeline.zip
```

Use the Kubeflow Pipelines SDK to run the pipeline:

client = kfp.Client()
my_experiment = client.create_experiment(name='demo')
my_run = client.run_pipeline(my_experiment.id, 'my-pipeline', 
  'my-pipeline.zip')

You can also choose to share your pipeline as follows:

Upload the pipeline zip file to the Kubeflow Pipelines UI. For more information about the UI, see the Kubeflow Pipelines quickstart guide.
Upload the pipeline zip file to a shared repository. See the reusable components and other shared resources.

More about the above workflow

For more detailed instructions, see the guide to building components and pipelines.

For an example, see the xgboost-training-cm.py pipeline sample on GitHub. The pipeline creates an XGBoost model using structured data in CSV format.

Creating components within your application code

This section describes how to create a pipeline component inside your Python application, as part of the application. The DSL code for creating a component therefore runs inside your Docker container.

Below is a more detailed explanation of the above diagram:

Write your code in a Python function. For example, write code to transform data or train a model:
```
def my_python_func(a: str, b: str) -> str:
  ...
```
Use the kfp.dsl.python_component decorator to convert your Python function into a pipeline component. To use the decorator, you can add the @kfp.dsl.python_component annotation to your function:
```
@kfp.dsl.python_component(
  name='My awesome component',
  description='Come and play',
)
def my_python_func(a: str, b: str) -> str:
  ...
```

Use kfp.compiler.build_python_component to create a container image for the component.

my_op = compiler.build_python_component(
  component_func=my_python_func,
  staging_gcs_path=OUTPUT_DIR,
  target_image=TARGET_IMAGE)

Write a pipeline function using the Kubeflow Pipelines DSL to define the pipeline and include all the pipeline components. Use the kfp.dsl.pipeline decorator to build a pipeline from your pipeline function, by adding the @kfp.dsl.pipeline annotation to your pipeline function:
```
@kfp.dsl.pipeline(
  name='My pipeline',
  description='My machine learning pipeline'
)
def my_pipeline(param_1: PipelineParam, param_2: PipelineParam):
  my_step = my_op(a='a', b='b')
```
Compile the pipeline to generate a compressed YAML definition of the pipeline. The Kubeflow Pipelines service converts the static configuration into a set of Kubernetes resources for execution.

To compile the pipeline, you can choose one of the following options:
- Use the kfp.compiler.Compiler.compile method:
```
kfp.compiler.Compiler().compile(my_pipeline,  
  'my-pipeline.zip')
```
- Alternatively, use the dsl-compile command on the command line.
```
dsl-compile --py [path/to/python/file] --output my-pipeline.zip
```

Use the Kubeflow Pipelines SDK to run the pipeline:

client = kfp.Client()
my_experiment = client.create_experiment(name='demo')
my_run = client.run_pipeline(my_experiment.id, 'my-pipeline', 
  'my-pipeline.zip')

You can also choose to share your pipeline as follows:

Upload the pipeline zip file to the Kubeflow Pipelines UI. For more information about the UI, see the Kubeflow Pipelines quickstart guide.
Upload the pipeline zip file to a shared repository. See the reusable components and other shared resources.

More about the above workflow

For an example of the above workflow, see the Jupyter notebook titled KubeFlow Pipelines container building on GitHub.

Creating lightweight components

This section describes how to create lightweight Python components that do not require you to build a container image. Lightweight components simplify prototyping and rapid development, especially in a Jupyter notebook environment.

Below is a more detailed explanation of the above diagram:

Write your code in a Python function. For example, write code to transform data or train a model:
```
def my_python_func(a: str, b: str) -> str:
  ...
```
Use kfp.components.func_to_container_op to convert your Python function into a pipeline component:
```
my_op = kfp.components.func_to_container_op(my_python_func)
```
Optionally, you can write the component to a file that you can share or use in another pipeline:
```
my_op = kfp.components.func_to_container_op(my_python_func, 
  output_component_file='my-op.component')
```
If you stored your lightweight component in a file as described in the previous step, use kfp.components.load_component_from_file to load the component:
```
my_op = kfp.components.load_component_from_file('my-op.component')
```
Write a pipeline function using the Kubeflow Pipelines DSL to define the pipeline and include all the pipeline components. Use the kfp.dsl.pipeline decorator to build a pipeline from your pipeline function, by adding the @kfp.dsl.pipeline annotation to your pipeline function:
```
@kfp.dsl.pipeline(
  name='My pipeline',
  description='My machine learning pipeline'
)
def my_pipeline(param_1: PipelineParam, param_2: PipelineParam):
  my_step = my_op(a='a', b='b')
```
Compile the pipeline to generate a compressed YAML definition of the pipeline. The Kubeflow Pipelines service converts the static configuration into a set of Kubernetes resources for execution.

To compile the pipeline, you can choose one of the following options:
- Use the kfp.compiler.Compiler.compile method:
```
kfp.compiler.Compiler().compile(my_pipeline,  
  'my-pipeline.zip')
```
- Alternatively, use the dsl-compile command on the command line.
```
dsl-compile --py [path/to/python/file] --output my-pipeline.zip
```

Use the Kubeflow Pipelines SDK to run the pipeline:

client = kfp.Client()
my_experiment = client.create_experiment(name='demo')
my_run = client.run_pipeline(my_experiment.id, 'my-pipeline', 
  'my-pipeline.zip')

More about the above workflow

For more detailed instructions, see the guide to building lightweight components.

For an example, see the Lightweight Python components - basics notebook on GitHub.

Using prebuilt, reusable components in your pipeline

A reusable component is one that someone has built and made available for others to use. To use the component in your pipeline, you need the YAML file that defines the component.

Below is a more detailed explanation of the above diagram:

Find the YAML file that defines the reusable component. For example, take a look at the reusable components and other shared resources.

Use kfp.components.load_component_from_url to load the component:

my_op = kfp.components.load_component_from_url('https://path/to/component.yaml')

Write a pipeline function using the Kubeflow Pipelines DSL to define the pipeline and include all the pipeline components. Use the kfp.dsl.pipeline decorator to build a pipeline from your pipeline function, by adding the @kfp.dsl.pipeline annotation to your pipeline function:
```
@kfp.dsl.pipeline(
  name='My pipeline',
  description='My machine learning pipeline'
)
def my_pipeline(param_1: PipelineParam, param_2: PipelineParam):
  my_step = my_op(a='a', b='b')
```
Compile the pipeline to generate a compressed YAML definition of the pipeline. The Kubeflow Pipelines service converts the static configuration into a set of Kubernetes resources for execution.

To compile the pipeline, you can choose one of the following options:
- Use the kfp.compiler.Compiler.compile method:
```
kfp.compiler.Compiler().compile(my_pipeline,  
  'my-pipeline.zip')
```
- Alternatively, use the dsl-compile command on the command line.
```
dsl-compile --py [path/to/python/file] --output my-pipeline.zip
```

Use the Kubeflow Pipelines SDK to run the pipeline:

client = kfp.Client()
my_experiment = client.create_experiment(name='demo')
my_run = client.run_pipeline(my_experiment.id, 'my-pipeline', 
  'my-pipeline.zip')

More about the above workflow

For an example, see the xgboost-training-cm.py pipeline sample on GitHub. The pipeline creates an XGBoost model using structured data in CSV format.

Next steps

Use pipeline parameters to pass data between components.
Learn how to write recursive functions in the DSL.
Build a reusable component for sharing in multiple pipelines.
Find out how to use the DSL to manipulate Kubernetes resources dynamically as steps of your pipeline.

Docs: Jupyter Notebooks

Mon, 01 Jan 0001 00:00:00 +0000

Your Kubeflow deployment includes support for spawning and managing Jupyter notebooks. See how to set up your notebooks and explore more notebook functionality.

Docs: Kubeflow on Linux

Mon, 01 Jan 0001 00:00:00 +0000

For Linux systems you have options for servers (physical or virtual) and desktops. The server options apply to the desktop as well.

Linux server

For Linux servers you can install Kubeflow natively. This is perfect for Linux hosts and virtual machines, such as VMs in OpenStack, VMware or public clouds like GCP, AWS and Azure.

MicroK8s

MicroK8s runs natively on most Linux distributions.

Follow the installation guide for Kubeflow with MicroK8s to set up MicroK8s and enable Kubeflow.

Linux desktop

Kubeflow appliance

A Kubeflow appliance is a virtual machine that has Kubeflow already installed. Once the necessary supporting software is installed no further installation steps are required.

MiniKF

MiniKF is a predefined virtual machine that installs onto VirtualBox through Vagrant. The only following applications are required to use MiniKF:

Install Vagrant
Install Virtual Box

The full set of instructions are available on the MiniKF getting started page.

Linux appliance

A Linux appliance is a virtual machine that holds the linux operating system. From there you have complete choice over Kubernetes and Kubeflow, which offers the greatest degree of flexibility. You only need to install a single application to follow this path:

Install Multipass

The instructions on Multipass and MicroK8s getting started page will complete this path.

Kubernetes appliance

A Kubernetes appliance is a virtual machine that has a Kubernetes cluster already installed. After starting the virtual machine, you need to install Kubeflow. This option gives you full control over your Kubeflow setup.

Install Minikube

Follow the instructions on deploying with MiniKube on Linux to complete this path.