Kubeflow – Further Setup and Troubleshooting

Docs: Configuring Kubeflow with kfctl and kustomize

Mon, 01 Jan 0001 00:00:00 +0000

This Kubeflow component has stable status. See the Kubeflow versioning policies.

Kfctl is the Kubeflow command-line interface (CLI) that you can use to install and configure Kubeflow.

Kubeflow makes use of kustomize to help customize YAML configurations. With kustomize, you can traverse a Kubernetes manifest to add, remove, or update configuration options without forking the manifest. A manifest is a YAML file containing a description of the applications that you want to include in your Kubeflow deployment.

Overview of kfctl and kustomize

This section describes how kfctl works with kustomize to set up your Kubeflow deployment. You need kustomize 2.0.3 or later.

The Kubeflow deployment process

Kfctl is the Kubeflow CLI that you can use to set up a Kubernetes cluster with Kubeflow installed, or to deploy Kubeflow to an existing Kubernetes cluster. See the Kubeflow getting-started guide for installation instructions based on your deployment scenario.

The kfctl deployment process includes the following commands:

kfctl build - (Optional) Creates configuration files defining the various resources in your deployment but does not deploy Kubeflow. You only need to run kfctl build if you want to edit the resources before running kfctl apply.
kfctl apply - Creates or updates the resources.
kfctl delete - Deletes the resources.

Specifying a configuration file when initializing your deployment

When you install Kubeflow, the deployment process uses one of a few possible YAML configuration files to bootstrap the configuration. You can see all the configuration files on GitHub.

As an example, this guide uses the kfctl_k8s_istio.yaml configuration. For more details about this configuration, see the kfctl_k8s_istio deployment guide.

Typically, you specify the configuration file with a -f <config-file> parameter when you run kfctl build or kfctl apply. The following example uses kfctl build:

# Set KF_NAME to the name of your Kubeflow deployment. You also use this
# value as directory name when creating your configuration directory.
# For example, your deployment name can be 'my-kubeflow' or 'kf-test'.
export KF_NAME=<your choice of name for the Kubeflow deployment>

# Set the path to the base directory where you want to store one or more 
# Kubeflow deployments. For example, /opt/.
# Then set the Kubeflow application directory for this deployment.
export BASE_DIR=<path to a base directory>
export KF_DIR=${BASE_DIR}/${KF_NAME}

# Set the URI of the configuration file to use when deploying Kubeflow. 
# For example:
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_k8s_istio.v1.0.2.yaml"

# Create your Kubeflow configurations:
mkdir -p ${KF_DIR}
cd ${KF_DIR}
kfctl build -V -f ${CONFIG_URI}

Kfctl has now built the configuration files in your Kubeflow application directory (see below) but has not yet deployed Kubeflow. To complete the deployment, run kfctl apply. See the next section on applying the configuration.

Applying the configuration to your Kubeflow cluster

When you first run kfctl build or kfctl apply, kfctl creates a local version of the YAML configuration file, which you can further customize if necessary.

Follow these steps to apply the configurations to your Kubeflow cluster:

Set an environment variable pointing to your local configuration file. For example, this guide uses the kfctl_k8s_istio.v1.0.2.yaml configuration. If you chose a different configuration in the previous step, you must change the file name to reflect your configuration:
```
export CONFIG_FILE=${KF_DIR}/kfctl_k8s_istio.v1.0.2.yaml
```
Apply the configurations:
```
kfctl apply -V -f ${CONFIG_FILE}
```

Your Kubeflow directory layout

Your Kubeflow application directory is the directory where you choose to store your Kubeflow configurations during deployment. This guide refers to the directory as ${KF_DIR}. The directory contains the following files and directories:

${CONFIG_FILE} is a YAML file that stores your primary Kubeflow configuration in the form of a KfDef Kubernetes object.

This file is a copy of the GitHub-based configuration YAML file that you used when deploying Kubeflow.

When you first run kfctl build or kfctl apply, kfctl creates a local version of the configuration file at ${CONFIG_FILE}, which you can further customize if necessary.

The YAML defines each Kubeflow application as a kustomize package.
<platform-name>_config is a directory that contains configurations specific to your chosen platform or cloud provider. For example, gcp_config or aws_config. This directory may or may not be present, depending on your setup.
- The directory is created when you run kfctl build or kfctl apply.
- To customize these configurations, you can modify parameters in your ${CONFIG_FILE}, and then run kfctl apply to apply the configuration to your Kubeflow cluster.
kustomize is a directory that contains Kubeflow application manifests. That is, the directory contains the kustomize packages for the Kubeflow applications that are included in your deployment.
- The directory is created when you run kfctl build or kfctl apply.
- To customize these configurations, you can modify parameters in your ${CONFIG_FILE}, and then run kfctl apply to apply the configuration to your Kubeflow cluster.

How your configuration is generated

The content of your ${CONFIG_FILE} is the result of running kustomize on the base and overlay kustomization.yaml files in the Kubeflow manifests. The overlays reflect the configuration file that you specify when running kfctl build or kfctl apply.

Below are some examples of configuration files:

kfctl_k8s_istio.yaml to install Kubeflow on an existing Kubernetes cluster.
kfctl_istio_dex.yaml to install Kubeflow on an existing Kubernetes cluster with Dex and Istio for authentication.
kfctl_gcp_iap.yaml to create a Google Kubernetes Engine (GKE) cluster with Kubeflow using Cloud Identity-Aware Proxy (Cloud IAP) for access control.

The kustomize package manager in kfctl uses the information in your ${CONFIG_FILE} to traverse the directories under the Kubeflow manifests and to create kustomize build targets based on the manifests.

Installing kustomize

Make sure that you have the minimum required version of kustomize: 2.0.3 or later.

Follow the kustomize installation guide, choosing the relevant options for your operating system. For example, if you’re on Linux:

Set some variables for the operating system:
```
export opsys=linux
```

Download the kustomize binary:

curl -s https://api.github.com/repos/kubernetes-sigs/kustomize/releases |\
grep browser_download |\
grep download/kustomize |\
grep -m 1 $opsys |\
cut -d '"' -f 4 |\
xargs curl -O -L

Unzip the compressed file

tar xzf ./kustomize_v*_${opsys}_amd64.tar.gz

Move the binary:

mkdir -p ${HOME}/bin
mv kustomize ${HOME}/bin/kustomize
chmod u+x ${HOME}/bin/kustomize

Include the kustomize command in your path:
```
export PATH=$PATH:${HOME}/bin
```

Modifying configuration before deployment

Kustomize lets you customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is.

You can use the following command to build and apply kustomize directories:

kustomize build <kustomization_directory> | kubectl apply -f -

The Kubeflow manifests repo contains kustomize build targets, each with a base directory. You can use kustomize to generate YAML output and pass it to kfctl. You can also make changes to the kustomize targets in the manifests repo as needed.

Modifying the configuration of an existing deployment

To customize the Kubeflow resources running within the cluster, you can modify parameters in your ${CONFIG_FILE} file. Then re-run kfctl apply.

For example, to modify settings for the Spartakus usage reporting tool within your Kubeflow deployment:

Edit the configuration file at ${CONFIG_FILE}.

Find and replace the parameter values for spartakus to suit your requirements:

 - kustomizeConfig:
     parameters:
     - initRequired: true
         name: usageId
         value: <randomly-generated-id>
     - initRequired: true
         name: reportUsage
         value: "true"
     repoRef:
         name: manifests
         path: common/spartakus
     name: spartakus

Regenerate and deploy your Kubeflow resources:

cd ${KF_DIR}
kfctl apply -V -f ${CONFIG_FILE}

More examples

For examples of customizing your deployment, see the guide to customizing Kubeflow on GKE.

For information about how Kubeflow uses Spartakus, see the guide to usage reporting.

More about kustomize

Below are some useful kustomize terms, from the kustomize glossary:

base: A combination of a kustomization and resource(s). Bases can be referred to by other kustomizations.
kustomization: Refers to a kustomization.yaml file, or more generally to a directory containing the kustomization.yaml file and all the relative file paths that the YAML file references.
overlay: A combination of a kustomization that refers to a base, and a patch. An overlay may have multiple bases.
patch: General instructions to modify a resource.
resource: Any valid YAML file that defines an object with a kind and a metadata/name field.
target: The argument to kustomize build. For example, kustomize build $TARGET. A target must be a path or a URL to a kustomization. A target can be a base or an overlay.
variant: The outcome of applying an overlay to a base.

Read more about kustomize in the kustomize documentation.

Docs: Kubeflow On-prem in a Multi-node Kubernetes Cluster

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to set up Kubeflow on premises (on-prem) in a multi-node cluster using dynamic volume provisioning.

Vanilla on-prem Kubeflow installation

In order to install Kubeflow in an on-prem Kubernetes cluster, follow the guide to installing Kubeflow on existing clusters, which works for single node and multi-node clusters.

At the end of the installation, some Persistent Volume Claims (PVCs) might be unbound. To fix this issue, see the troubleshooting section Pods stuck in Pending state.

However, when you set up Kubeflow in a multi-node cluster you might run into an additional issue: you can’t create a HostPath PersistentVolume (which exposes a filesystem directory to a Pod), because this type of PersistentVolume only works on a single node cluster.

Background on Kubernetes storage

Kubernetes defines several ways to attach Volumes to Pods.

The best practice is to decouple storage needs (Persistent Volume Claims or PVCs) from actual storage (Persistent Volumes or PVs). Kubernetes provisions PVs based on available resources. There are several types of PV.

Cloud providers define mechanisms to allocate PVs based on existing PVCs using their storage infrastructure; on-prem clusters must provision their PVs according the existing capability of the system.

In development clusters or single node clusters, you can bind PVCs to HostPath PVs, which are a particular kind of volumes that maps the Pod’s Volume to a directory of the filesystem.

This approach, however, is not a feasible solution in multi-node clusters. A Pod can be on different nodes during its lifecycle: Kubernetes can kill and restart it on another node at any time based on the resources available in the cluster. In this scenario, the migrated Pod will not find its old data after restarting on a new node.

A common solution is to mount a remote volume in the Pods so that any node in the cluster can reach it.

The rest of this document describes the following tasks:

how to associate a remote NFS volume to Kubeflow Pods so you don’t have any problems when the Pod is scheduled on different cluster nodes during its lifecycle.
how to simplify cluster administration by using dynamic provisioning, which lets Kubernetes create a different PV for each PVC without manual intervention.

The best way to avoid PVC problems is to complete both of these steps before you install Kubeflow on your Kubernetes cluster. However, if you have already installed Kubeflow on your cluster and want to resolve PVC problems, complete the steps and then read the In case of existing Kubeflow installation section at the end of this document.

NFS Persistent Volumes

NFS is a popular distributed filesystem commonly used in Unix operating systems.

You can use an NFS server to create PVs where Pods can write their data.

In order to do this you must provide an NFS server with an IP reachable from inside the Kubernetes cluster.

If an NFS volume is not available to your cluster, you can transform one of the cluster’s nodes into an NFS server with the following commands:

sudo apt install nfs-common
sudo apt install nfs-kernel-server
sudo mkdir /nfsroot

Than you need to configure /etc/exports to share that directory:

/nfsroot 192.168.0.0/16(rw,no_root_squash,no_subtree_check)

Notice that 192.168.0.0 is the nodes' Classless Inter-Domain Routing (CIDR), not the Kubernetes CIDR.

NFS Client

Each node of the cluster must be able to establish a connection to the NFS server. To enable this, install the following NFS client library on each node:

sudo apt install nfs-common

Install Dynamic Provisioner

Now you can create NFS PVs to enable each Pod to write its own data in a common place from any node.

In order to successfully complete the Kubeflow installation, your cluster must have an NFS PV for each PVC to bind to.

Since creating NFS PVs can be tedious, you can set up Dynamic provisioning to automatically create PVs based on existing PVCs.

Follow the instructions in this section to install a Dynamic Provisioner for NFS volumes in your cluster.

Install Helm

To install a Dynamic Provisioner, you must first install Helm, the Kubernetes package manager. You can follow this guide to install it in a couple of steps.

Install NFS Provisioner

Next, install NFS client provisioner, which is a Dynamic Provisioner that can create PVs for any PVC for a given storage class.

A storage class is a label associated to volumes to specify a class of storage: storage class definitions make it possible to query and provision volumes with different performances or capabilities (For example, SSD or slower disks).

A special storage class is default: any PV or PVC that doesn’t specify one is associated to it.

You can install NFS Client Provisioner with Helm:

helm install \
  --name nfs-client-provisioner \
  --set nfs.server=<NFS Server IP> \
  --set nfs.path=/exported/path \
  --set storageClass.name=nfs \
  --set storageClass.defaultClass=true \
  stable/nfs-client-provisioner

The component in the system will add an nfs Storage Class that you can see with the following command:

kubectl get storageclass -n kubeflow

NAME                   PROVISIONER                            AGE
nfs (default)          cluster.local/nfs-client-provisioner   6h13m
...

Notice that the installation command set the storageClass.defaultClass parameter to true. This sets the nfs as the default storage class. Therefore when you install Kubeflow, all PVCs will be labelled with the nfs storage class.

Finally: install Kubeflow

Now that you have prepared your on-prem, multi-node Kubernetes cluster to manage volumes using an NFS server, you can install Kubeflow by following the guide to installing Kubeflow on existing clusters.

After installing Kubeflow, notice that that Kubernetes binds each PVC created by Kubeflow to an automatically created PV with the nfs storage class. Notice also that the provisioner has created a directory for each PVC inside the root NFS directory.

In case of existing Kubeflow installation

If you set up Dynamic Provisioning after installing Kubeflow, you must change the storage class on your existing unbound PVCs.

To perform this task you need to:

Download existing PVCs.
Change their storage class.
Delete and recreate them in the cluster.

For downloading the existing PVCs, first get list of all PVCs:

kubectl get pvc --all-namespaces

This will show the list of PVCs with details including NAMESPACE, STATUS, STORAGECLASS. (For example, mysql-pv-claim in kubeflow namespace and authservice-pvc in istio-systems namespace).

Download the PVCs:

kubectl get pvc/<PVC-NAME> -n <NAMESPACE> -o yaml > <PVC-NAME>.yaml

And then modify the YAML files to add the right storageClassName under the spec section. For example:

# mysql-pv-claim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
  namespace: kubeflow
  ...
spec:
  storageClassName: nfs
  ...

After modifying all downloaded PVCs, remove the old PVCs. Note that the field <PVC-NAME> below stands for the downloaded YAML file name.

kubectl delete -f <PVC-NAME>.yaml

Finally, add the modified PVCs:

kubectl apply -f <PVC-NAME>.yaml

The PVCs are now bound to your NFS storage.

Limitations

NFS is a remote filesystem that is high performant in reading but slower in writing. If you have to write a huge amount of data in your workflow, NFS might not be the right choice.

Ensure that you are using version 4 of NFS (instead of version 3): NFS 3 can have some problems like partial writing of documents and it does not support authentication.

In this document you used one of the nodes as an NFS server. This is not a good idea for a production environment because you will have a single point of failure in your cluster.

Resources

This short guide is not intended to be complete. For more information about the following Kubernetes resources, please refer to the official documentation:

Docs: Usage Reporting

Mon, 01 Jan 0001 00:00:00 +0000

When enabled, Kubeflow will report anonymous usage data using Spartakus, Kubernetes' reporting tool. Spartakus does not report any personal information. See the Spartakus docs for more detail.

Allowing usage reporting is entirely voluntary.

Reporting usage data is one of the most significant contributions you can make to Kubeflow

Please consider allowing the reporting of usage data. The data helps the Kubeflow community to improve the project and helps the many companies working on Kubeflow justify continued investment.

Disable usage reporting on an existing Kubeflow deployment

If you’ve already deployed Kubeflow, run the following command to disable usage reporting on your existing deployment. The command removes the spartakus-volunteer application:

export NAMESPACE=kubeflow
kubectl delete -n ${NAMESPACE} deploy spartakus-volunteer

You can run the following command to check for existence of the application:

kubectl get -n ${NAMESPACE} deploy spartakus-volunteer

Remove usage reporting before deploying Kubeflow

The following instructions assume that you plan to use the kfctl command-line interface to deploy Kubeflow.

This guide refers to the ${CONFIG_FILE}, which is the Kubeflow configuration file in your Kubeflow deployment directory. For example, ${KF_DIR}/kfctl_k8s_istio.v1.0.2.yaml or ${KF_DIR}/kfctl_gcp_iap.v1.0.2.yaml.

To prevent Spartakus from being deployed:

Follow your chosen guide to deploying Kubeflow, but stop before you deploy Kubeflow. For example, see the guide to deploying Kubeflow with kfctl_k8s_istio.
When you reach the setup and deploy step, skip the kfctl apply command and run the kfctl build command instead, as described in the above guide. Now you can edit the configuration files before deploying Kubeflow.
Edit your ${CONFIG_FILE} file as described below.
Run the kfctl apply command to deploy Kubeflow:

cd ${KF_DIR}
kfctl apply -V -f ${CONFIG_FILE}

Removing Spartakus from your configuration

You need to remove the Spartakus entry from KfDef.Spec.Applications in the ${CONFIG_FILE} file. Find the applications section of the YAML file and delete the following lines:

- kustomizeConfig:
    parameters:
    - initRequired: true
      name: usageId
      value: <randomly-generated-id>
    - initRequired: true
      name: reportUsage
      value: "true"
    repoRef:
      name: manifests
      path: common/spartakus
  name: spartakus

Alternatively, some YAML configuration files may include entries for KfDef.Spec.Components and KfDef.Spec.ComponentParams instead of KfDef.Spec.Applications. In this case:

Find the componentParams section of the YAML file and delete the following lines:

  spartakus:
  - initRequired: true
    name: usageId
    value: "<randomly-generated-id>"
  - initRequired: true
    name: reportUsage
    value: "true"

Find the components section of the YAML file and delete the following line:
```
  - spartakus
```

Docs: Istio Usage in Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

Kubeflow v0.6 onwards deploys Istio along with configuration to enable end-to-end authentication and access control. This setup is the foundation of multi-tenancy support in Kubeflow. A Kubeflow deployment without Istio is not possible.

A gentle introduction to Istio

Most modern applications are built using a distributed microservices architecture. This ensures that each individual service is simple and has a well defined responsibility. Complex systems and platforms are generally built by combining many such microservices. Each microservice defines its own APIs and the services interact with each other using these APIs in order to serve end-user requests.

The term service mesh is used to describe the network of microservices that make up such applications and the interactions between them. As a service mesh grows in size and complexity, it can become harder to understand and manage. Its requirements can include discovery, load balancing, failure recovery, metrics, and monitoring. A service mesh also often has more complex operational requirements, like A/B testing, canary rollouts, rate limiting, access control, and end-to-end authentication.

Istio is a pioneering and highly performant open source implementation of service mesh by Google. For further details, you can read the conceptual overview of Istio.

Why Kubeflow needs Istio

Kubeflow is a collection of tools, frameworks and services that are deployed together into a single Kubernetes cluster to enable end-to-end ML workflows. Most of these components or services are developed independently and help with different parts of the workflow. Developing a complete ML workflow or an ML development environment requires combining multiple services and components. Kubeflow provides the underlying infrastructure that makes it possible to put such disparate components together.

Kubeflow uses Istio as a uniform way to secure, connect, and monitor microservices. Specifically:

Securing service-to-service communication in a Kubeflow deployment with strong identity-based authentication and authorization.
A policy layer for supporting access controls and quotas.
Automatic metrics, logs, and traces for traffic within the deployment including cluster ingress and egress.

Istio in Kubeflow

The following diagram illustrates how user requests interact with services in Kubeflow. It walks through the process when a user requests to create a new notebook server via the Notebooks Servers UI accessible through the Kubeflow Central Dashboard.

The user request is intercepted by an identification proxy which talks to a SSO service provider such as IAM on Cloud Services Provider or Active Directory/LDAP on-premises.
When the user is authenticated, the request is modified by the Istio Gateway to include a JWT Header token containing the identity of the user. All requests throughout the service mesh carry this token along.
The Istio RBAC policies are applied on the incoming request to validate the access to the service and the requested namespace. If either of those are inaccessible to the user, an error response is sent back.
If the request is validated, it is forwarded to the appropriate controller (Notebooks Controller in this case).
Notebooks Controller validates authorization with Kubernetes RBAC and creates the notebook pod in the namespace that the user requested.

Further actions by the user with the notebook to create training jobs or other resources in the namespace go through a similar process. Profiles Controller manages the creation of profiles, and creates and applies appropriate Istio policies. For more details, please see multi-user isolation.

Deploying Kubeflow without Istio

Currently it is not possible to deploy Kubeflow without Istio. Kubeflow needs the Istio Custom Resource Definitions (CRDs) to express the new route to access the created Notebook from the Gateway.

Docs: Job Scheduling

Mon, 01 Jan 0001 00:00:00 +0000

Alpha

This Kubeflow component has alpha status with limited support. See the Kubeflow versioning policies. The Kubeflow team is interested in your feedback about the usability of the feature.

This guide describes how to use volcano scheduler to support gang-scheduling in Kubeflow, to allow jobs to run multiple pods at the same time.

Running jobs with gang-scheduling

To use gang-scheduling, you have to install volcano scheduler in your cluster first as a secondary scheduler of Kubernetes and configure operator to enable gang-scheduling.

Follow the instructions in the volcano repository to install Volcano.
Take tf-operator for example, enable gang-scheduling in tf-operator by setting true to --enable-gang-scheduling flag.

Note: Volcano scheduler and operator in Kubeflow achieve gang-scheduling by using PodGroup. operator will create the PodGroup of the job automatically.

The yaml to use volcano scheduler to schedule your job as a gang is the same as non-gang-scheduler, for example.

apiVersion: "kubeflow.org/v1beta1"
kind: "TFJob"
metadata:
  name: "tfjob-gang-scheduling"
spec:
  tfReplicaSpecs:
    Worker:
      replicas: 1
      template:
        spec:
          containers:
          - args:
            - python
            - tf_cnn_benchmarks.py
            - --batch_size=32
            - --model=resnet50
            - --variable_update=parameter_server
            - --flush_stdout=true
            - --num_gpus=1
            - --local_parameter_device=cpu
            - --device=gpu
            - --data_format=NHWC
            image: gcr.io/kubeflow/tf-benchmarks-gpu:v20171202-bdab599-dirty-284af3
            name: tensorflow
            resources:
              limits:
                nvidia.com/gpu: 1
            workingDir: /opt/tf-benchmarks/scripts/tf_cnn_benchmarks
          restartPolicy: OnFailure
    PS:
      replicas: 1
      template:
        spec:
          containers:
          - args:
            - python
            - tf_cnn_benchmarks.py
            - --batch_size=32
            - --model=resnet50
            - --variable_update=parameter_server
            - --flush_stdout=true
            - --num_gpus=1
            - --local_parameter_device=cpu
            - --device=cpu
            - --data_format=NHWC
            image: gcr.io/kubeflow/tf-benchmarks-cpu:v20171202-bdab599-dirty-284af3
            name: tensorflow
            resources:
              limits:
                cpu: '1'
            workingDir: /opt/tf-benchmarks/scripts/tf_cnn_benchmarks
          restartPolicy: OnFailure

About volcano scheduler and gang-scheduling

With using volcano scheduler to apply gang-scheduling, a job can run only if there are enough resources for all the pods of the job. Otherwise, all the pods will be in pending state waiting for enough resources. For example, if a job requiring N pods is created and there are only enough resources to schedule N-2 pods, then N pods of the job will stay pending.

Note: when in a high workload, if a pod of the job dies when the job is still running, it might give other pods chance to occupied the resources and cause deadlock.

Troubleshooting

If you keep getting problems related to RBAC in your volcano scheduler.

You can try to add the following rules into your clusterrole of scheduler used by volcano scheduler.

- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'

Docs: Troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

This page presents some hints for troubleshooting specific problems that you may encounter.

macOS: kfctl cannot be opened because the developer cannot be verified

The kfctl binary is currently not signed. That is, kfctl is not registered with Apple. When you run kfctl from the command line on the latest versions of macOS, you may see a message like this:

“kfctl” cannot be opened because the developer cannot be verified. macOS cannot verify that this app is free from malware.

To run kfctl, go to the kfctl binary file in Finder, right-click, then select Open. Then click Open again to confirm that you want to open the app.

For more information, see the macOS user guide.

TensorFlow and AVX

There are some instances where you may encounter a TensorFlow-related Python installation or a pod launch issue that results in a SIGILL (illegal instruction core dump). Kubeflow uses the pre-built binaries from the TensorFlow project which, beginning with version 1.6, are compiled to make use of the AVX CPU instruction. This is a recent feature and your CPU might not support it. Check the host environment for your node to determine whether it has this support.

Linux:

grep -ci avx /proc/cpuinfo

AVX2

Some components requirement AVX2 for better performance, e.g. TF Serving. To ensure the nodes support AVX2, we added minCpuPlatform arg in our deployment config.

On GCP this will fail in regions (e.g. us-central1-a) that do not explicitly have Intel Haswell (even when there are other newer platforms in the region). In that case, please choose another region, or change the config to other platform newer than Haswell.

RBAC clusters

If you are running on a Kubernetes cluster with RBAC enabled, you may get an error like the following when deploying Kubeflow:

ERROR Error updating roles kubeflow-test-infra.jupyter-role: roles.rbac.authorization.k8s.io "jupyter-role" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["*"], APIGroups:["*"], Verbs:["*"]}] user=&{your-user@acme.com  [system:authenticated] map[]} ownerrules=[PolicyRule{Resources:["selfsubjectaccessreviews"], APIGroups:["authorization.k8s.io"], Verbs:["create"]} PolicyRule{NonResourceURLs:["/api" "/api/*" "/apis" "/apis/*" "/healthz" "/swagger-2.0.0.pb-v1" "/swagger.json" "/swaggerapi" "/swaggerapi/*" "/version"], Verbs:["get"]}] ruleResolutionErrors=[]

This error indicates you do not have sufficient permissions. In many cases you can resolve this just by creating an appropriate clusterrole binding like so and then redeploying kubeflow:

kubectl create clusterrolebinding default-admin --clusterrole=cluster-admin --user=your-user@acme.com

Replace your-user@acme.com with the user listed in the error message.

If you’re using GKE, you may want to refer to GKE’s RBAC docs to understand how RBAC interacts with IAM on GCP.

Problems spawning Jupyter pods

This section has been moved to Jupyter Notebooks Troubleshooting Guide.

Pods stuck in Pending state

There are three pods that have Persistent Volume Claims (PVCs) that will get stuck in pending state if they are unable to bind their PVC. The three pods are minio, mysql, and katib-mysql. Check the status of the PVC requests:

kubectl -n ${NAMESPACE} get pvc

Look for the status of “Bound”
PVC requests in “Pending” state indicate that the scheduler was unable to bind the required PVC.

If you have not configured dynamic provisioning for your cluster, including a default storage class, then you must create a persistent volume for each of the PVCs.

You can use the example below to create local persistent volumes:

sudo mkdir /mnt/pv{1..3}

kubectl create -f - <<EOF
kind: PersistentVolume
apiVersion: v1
metadata:
  name: pv-volume1
spec:
  storageClassName:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/pv1"
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: pv-volume2
spec:
  storageClassName:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/pv2"
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: pv-volume3
spec:
  storageClassName:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/pv3"
EOF

Once created the scheduler will successfully start the remaining three pods. The PVs may also be created prior to running any of the kfctl commands.

OpenShift

If you are deploying Kubeflow in an OpenShift environment which encapsulates Kubernetes, you will need to adjust the security contexts for the ambassador and Jupyter-hub deployments in order to get the pods to run:

oc adm policy add-scc-to-user anyuid -z ambassador
oc adm policy add-scc-to-user anyuid -z jupyter-hub

Once the anyuid policy has been set, you must delete the failing pods and allow them to be recreated in the project deployment.

You will also need to adjust the privileges of the tf-job-operator service account for TFJobs to run. Do this in the project where you are running TFJobs:

oc adm policy add-role-to-user cluster-admin -z tf-job-operator

403 API rate limit exceeded error

Because kubectl uses GitHub to pull kubeflow, unless user specifies GitHub API token, it will quickly consume maximum API call quota for anonymous. To fix this issue first create GitHub API token using this guide, and assign this token to GITHUB_TOKEN environment variable:

export GITHUB_TOKEN=<< token >>

Next steps

Visit the Kubeflow support page to find resources and community forums where you can ask for help.

Docs: Frequently Asked Questions

Mon, 01 Jan 0001 00:00:00 +0000

Is deploying Kubeflow with Helm supported?

No, Kubeflow only provides Kustomize packages for Kubeflow applications. So deploying Kubeflow isn’t supported except via Kustomize.

Kubeflow makes use of Kustomize to help customize Kubeflow for different platforms and configurations.

However, once you deploy Kubeflow you can express Kubernetes resources using YAML, Kustomize, Helm or whatever tool you like.

Docs: Support

Mon, 01 Jan 0001 00:00:00 +0000

This page describes the Kubeflow resources and support options that you can explore when you encounter a problem, have a question, or want to make a suggestion about Kubeflow.

Levels of support

Kubeflow applications offer various levels of support, based on the application status. To see the status of each application, refer to the Kubeflow application matrix on the version policies page.

The following table describes the level of support that you can expect based on the status of an application:

Application status	Level of support
Stable	The Kubeflow community offers best-effort support for stable applications. See the section on community support below for a definition of best-effort support and the community channels where you can report and discuss the problem. You can also consider requesting support from a Kubeflow community provider or from your cloud provider.
Beta	The Kubeflow community offers best-effort support for beta applications. See the section on community support below for a definition of best-effort support and the community channels where you can report and discuss the problem.
Alpha	The response differs per application in alpha status, depending on the size of the community for that application and the current level of active development of the application.

Support from the Kubeflow community

Kubeflow has an active and helpful community of users and contributors.

The Kubeflow community offers support on a best-effort basis for stable and beta applications. Best-effort support means that there’s no formal agreement or commitment to solve a problem but the community appreciates the importance of addressing the problem as soon as possible. The community commits to helping you diagnose and address the problem if all the following are true:

The cause falls within the technical framework that Kubeflow controls. For example, the Kubeflow community may not be able to help if the problem is caused by a specific network configuration within your organization.
Community members can reproduce the problem.
The reporter of the problem can help with further diagnosis and troubleshooting.

You can ask questions and make suggestions in the following places:

Slack for online chat and messaging. See details of Kubeflow’s Slack workspace and channels.
Kubeflow discuss for email-based group discussion. Join the kubeflow-discuss group.
Kubeflow documentation for overviews and how-to guides. In particular, refer to the following documents when troubleshooting a problem:
Kubeflow issue trackers for known issues, questions, and feature requests. Search the open issues to see if someone else has already logged the problem that you’re encountering and learn about any workarounds to date. If no one has logged your problem, create a new issue to describe the problem.

Each Kubeflow application has its own issue tracker within the Kubeflow organization on GitHub. To get you started, here are the primary issue trackers:

Support from providers in the Kubeflow ecosystem

The following organizations offer advice and support for Kubeflow deployments:

Organization	Points of contact
Arrikto	sales@arrikto.com
Seldon	Issue tracker

Support from a cloud or platform provider

If you’re using the services of a cloud provider to host Kubeflow, the cloud provider may be able to help you diagnose and solve a problem.

Consult the support page for the cloud service or platform that you’re using:

Getting involved in the Kubeflow community

You can get involved with Kubeflow in many ways. For example, you can contribute to the Kubeflow code or documentation. You can join the community meetings to talk to maintainers about a specific topic. See the Kubeflow community page for further information.

Following the news

Keep up with Kubeflow news:

The Kubeflow blog is the primary channel for announcement of new releases, events, and technical walkthroughs.
Follow Kubeflow on Twitter for shared technical tips.
The release notes give details of the latest updates for each Kubeflow application.

Each Kubeflow application has its own repository within the Kubeflow organization on GitHub. Some of the applications publish release notes. To get you started, here are the release notes for the primary applications:

Docs: Integrations

Mon, 01 Jan 0001 00:00:00 +0000

Kubeflow – Further Setup and Troubleshooting

Docs: Configuring Kubeflow with kfctl and kustomize

Overview of kfctl and kustomize

The Kubeflow deployment process

Specifying a configuration file when initializing your deployment

Applying the configuration to your Kubeflow cluster

Your Kubeflow directory layout

How your configuration is generated

Installing kustomize

Modifying configuration before deployment

Modifying the configuration of an existing deployment

More examples

More about kustomize

Docs: Kubeflow On-prem in a Multi-node Kubernetes Cluster

Vanilla on-prem Kubeflow installation

Background on Kubernetes storage

NFS Persistent Volumes

NFS Client

Install Dynamic Provisioner

Install Helm

Install NFS Provisioner

Finally: install Kubeflow

In case of existing Kubeflow installation

Limitations

Resources

Docs: Usage Reporting

Reporting usage data is one of the most significant contributions you can make to Kubeflow

Disable usage reporting on an existing Kubeflow deployment

Remove usage reporting before deploying Kubeflow

Removing Spartakus from your configuration

Docs: Istio Usage in Kubeflow

A gentle introduction to Istio

Why Kubeflow needs Istio

Istio in Kubeflow

Deploying Kubeflow without Istio

Docs: Job Scheduling

Alpha

Running jobs with gang-scheduling

About volcano scheduler and gang-scheduling

Troubleshooting

Docs: Troubleshooting

macOS: kfctl cannot be opened because the developer cannot be verified

TensorFlow and AVX

AVX2

RBAC clusters

Problems spawning Jupyter pods

Pods stuck in Pending state

OpenShift

403 API rate limit exceeded error

Next steps

Docs: Frequently Asked Questions

Is deploying Kubeflow with Helm supported?

Docs: Support

Levels of support

Support from the Kubeflow community

Support from providers in the Kubeflow ecosystem

Support from a cloud or platform provider

Other places to ask questions

Getting involved in the Kubeflow community

Following the news

Docs: Integrations