Kubeflow – Kubeflow on GCP

Docs: Deploying Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

Docs: Kubeflow On-premises on Anthos GKE

Mon, 01 Jan 0001 00:00:00 +0000

Anthos is a hybrid and multi-cloud application platform developed and supported by Google. Anthos is built on open source technologies, including Kubernetes, Istio, and Knative.

Using Anthos, you can create a consistent setup across your on-premises and cloud environments, helping you to automate policy and security at scale.

If you’re interested in running Kubeflow on Anthos GKE, email the Kubeflow team at google-kubeflow-support@google.com.

Next steps

While waiting for a response from the support team, you may like to deploy Kubeflow on GKE.

Docs: Pipelines on GCP

Mon, 01 Jan 0001 00:00:00 +0000

Docs: Customizing Kubeflow on GKE

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to customize your deployment of Kubeflow on Google Kubernetes Engine (GKE) in Google Cloud Platform (GCP).

Customizing Kubeflow before deployment

The Kubeflow deployment process is divided into two steps, build and apply, so that you can modify your configuration before deploying your Kubeflow cluster.

Follow the guide to deploying Kubeflow on GCP. When you reach the setup and deploy step, skip the kfctl apply command and run the kfctl build command instead, as described in that step. Now you can edit the configuration files before deploying Kubeflow.

Customizing an existing deployment

You can also customize an existing Kubeflow deployment. In that case, this guide assumes that you have already followed the guide to deploying Kubeflow on GCP and have deployed Kubeflow to a GKE cluster.

Before you start

This guide assumes the following settings:

The ${KF_DIR} environment variable contains the path to your Kubeflow application directory, which holds your Kubeflow configuration files. For example, /opt/my-kubeflow/.
```
export KF_DIR=<path to your Kubeflow application directory>
```
The ${CONFIG_FILE} environment variable contains the path to your Kubeflow configuration file.
```
export CONFIG_FILE=${KF_DIR}/kfctl_gcp_iap.v1.0.2.yaml
```
The ${KF_NAME} environment variable contains the name of your Kubeflow deployment. You can find the name in your ${CONFIG_FILE} configuration file, as the value for the metadata.name key.
```
export KF_NAME=<the name of your Kubeflow deployment>
```
The ${PROJECT} environment variable contains the ID of your GCP project. You can find the project ID in your ${CONFIG_FILE} configuration file, as the value for the project key.
```
export PROJECT=<your GCP project ID>
```
For further background about the above settings, see the guide to deploying Kubeflow with the CLI.

Customizing GCP resources

To customize GCP resources, such as your Kubernetes Engine cluster, you can modify the Deployment Manager configuration settings in ${KF_DIR}/gcp_config.

After modifying your existing configuration, run the following command to apply the changes:

cd ${KF_DIR}
kfctl apply -V -f ${CONFIG_FILE}

Alternatively, you can use Deployment Manager directly:

cd ${KF_DIR}/gcp_config
gcloud deployment-manager --project=${PROJECT} deployments update ${KF_NAME} --config=cluster-kubeflow.yaml

Some changes (such as the VM service account for Kubernetes Engine) can only be set at creation time; in this case you need to tear down your deployment before recreating it:

cd ${KF_DIR}
kfctl delete -f ${CONFIG_FILE}
kfctl apply -V -f ${CONFIG_FILE}

Customizing Kubernetes resources

You can use kustomize to customize Kubeflow. Make sure that you have the minimum required version of kustomize: 2.0.3 or later. For more information about kustomize in Kubeflow, see how Kubeflow uses kustomize.

To customize the Kubernetes resources running within the cluster, you can modify the kustomize manifests in ${KF_DIR}/kustomize.

For example, to modify settings for the Jupyter web app:

Open ${KF_DIR}/kustomize/jupyter-web-app.yaml in a text editor.

Find and replace the parameter values:

apiVersion: v1
data:
  ROK_SECRET_NAME: secret-rok-{username}
  UI: default
  clusterDomain: cluster.local
  policy: Always
  prefix: jupyter
kind: ConfigMap
metadata:
  labels:
    app: jupyter-web-app
    kustomize.component: jupyter-web-app
  name: jupyter-web-app-parameters
  namespace: kubeflow

Redeploy Kubeflow using kfctl:

cd ${KF_DIR}
kfctl apply -V -f ${CONFIG_FILE}

Or use kubectl directly:

cd ${KF_DIR}/kustomize
kubectl apply -f jupyter-web-app.yaml

Common customizations

Add GPU nodes to your cluster

To add GPU accelerators to your Kubeflow cluster, you have the following options:

Pick a GCP zone that provides NVIDIA Tesla K80 Accelerators (nvidia-tesla-k80).
Or disable node-autoprovisioning in your Kubeflow cluster.
Or change your node-autoprovisioning configuration.

To see which accelerators are available in each zone, run the following command:

gcloud compute accelerator-types list

To disable node-autoprovisioning, run kfctl build as described above. Then edit ${KF_DIR}/gcp_config/cluster-kubeflow.yaml and set enabled to false:

    ...
    gpu-type: nvidia-tesla-k80
    autoprovisioning-config:
      enabled: false
    ...

You must also set gpu-pool-initialNodeCount.

Add GPU node pool to an existing kubeflow cluster

You can add a GPU node pool to your kubeflow cluster using the following command

export GPU_POOL_NAME=<name of the new gpu pool>

gcloud container node-pools create ${GPU_POOL_NAME} \
--accelerator type=nvidia-tesla-k80,count=1 \
--zone us-central1-a --cluster ${KF_NAME} \
--num-nodes=1 --machine-type=n1-standard-4 --min-nodes=0 --max-nodes=5 --enable-autoscaling

After adding GPU nodes to your cluster, you need to install NVIDIA’s device drivers to the nodes. Google provides a DaemonSet that automatically installs the drivers for you.

To deploy the installation DaemonSet, run the following command:

kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml

Add Cloud TPUs to your cluster

Set enable_tpu:true in ${KF_DIR}/gcp_config/cluster-kubeflow.yaml.

Specify a minimum CPU

Certain instruction sets or hardware features are only available on specific CPUs, so to ensure your cluster utilizes the appropriate hardware you need to set a minimum CPU value.

In brief, inside gcp_config/cluster.jinja change the minCpuPlatform property for the CPU node pool. For example, Intel Broadwell becomes Intel Skylake. Setting a minimum CPU needs to occur during cluster/node creation; it cannot be applied to an existing cluster/node.

More detailed instructions follow.

Choose a zone you want to deploy in that has your required CPU. Zones are listed in the Regions and Zones documentation.
Deploy Kubeflow normally as specified in the “Deploy using CLI” documentation, but stop at section “Set up and deploy Kubeflow”. Instead, navigate to section “Alternatively, set up your configuration for later deployment”. Then follow the steps until you are instructed to edit configuration files.
Navigate to the gcp_config directory and open the cluster.jinja file. Change the cluster property minCpuPlatform. For example, from Intel Broadwell to Intel Skylake. Note: you may notice there are two minCpuPlatform properties in the file. One of them is for GPU node pools. Not all CPU/GPU combinations are compatible, so leave the GPU minCpuPlatform property untouched.
Follow the remaining steps of “Alternatively, set up your configuration for later deployment”.

Add VMs with more CPUs or RAM

Change the machineType.
There are two node pools defined in the GCP Deployment Manager:
- one for CPU only machines, in cluster.jinja.
- one for GPU machines, in cluster.jinja.
When making changes to the node pools you also need to bump the pool-version in cluster-kubeflow.yaml before you update the deployment.

Add users to Kubeflow

To grant users access to Kubeflow, add the “IAP-secured Web App User” role on the IAM page in the GCP console. Make sure you are in the same project as your Kubeflow deployment.
You can confirm the update by inspecting the IAM policy for your project:

gcloud projects get-iam-policy ${PROJECT}

In the output from the above command, users able to access Kubeflow have the following role: roles/iap.httpsResourceAccessor.

More customizations

Refer to the navigation panel on the left of these docs for more customizations, including using your own domain, setting up Cloud Filestore, and more.

Docs: Using Your Own Domain

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to use a custom domain with Kubeflow on Google Cloud Platform (GCP).

Before you start

This guide assumes you have already set up Kubeflow on GCP. If you haven’t done so, follow the guide to getting started with Kubeflow on GCP.

Using your own domain

If you want to use your own domain instead of ${KF_NAME}.endpoints.${PROJECT}.cloud.goog, follow these instructions after running kfctl build:

Replace hostname in kustomize/iap-ingress/base/params.env with your own domain.

...
hostname=<enter your domain here>
ingressName=envoy-ingress
ipName=kf-test-ip
oauthSecretName=kubeflow-oauth
...

Apply the changes.

Using kfctl.

kfctl apply -V -f ${CONFIG_FILE}

Or using kubectl.

kubectl apply -k kustomize/iap-ingress

Remove the cloud-endpoints component.

kubectl delete -k kustomize/cloud-endpoints

Delete the endpoint created by the cloud-endpoints-controller, if any.

gcloud endpoints services delete ${KF_NAME}.endpoints.${PROJECT}.cloud.goog

Check Ingress to verify that your domain was properly configured.
```
kubectl -n istio-system describe ingresses
```

Get the address of the static IP address created.

IPNAME=${KF_NAME}-ip
gcloud compute addresses describe ${IPNAME} --global --format="value(address)"

Use your DNS provider to map the fully qualified domain specified in the first step to the above IP address.

Docs: Authenticating Kubeflow to GCP

Mon, 01 Jan 0001 00:00:00 +0000

This page describes in-cluster and local authentication for Kubeflow GCP deployments.

In-cluster authentication

Starting from Kubeflow v0.6, you consume Kubeflow from custom namespaces (that is, namespaces other than kubeflow). The kubeflow namespace is only for running Kubeflow system components. Individual jobs and model deployments run in separate namespaces. To do this, install GCP credentials into the new namespace.

Starting in Kubeflow v0.7: Google Kubernetes Engine (GKE) workload identity

Starting in v0.7, Kubeflow uses the new GKE feature: workload identity. This is the recommended way to access GCP APIs from your GKE cluster. You no longer have to download GCP service account key. Instead, you can configure a Kubernetes service account (KSA) to act as a GCP service account (GSA).

If you deployed Kubeflow following the GCP instructions, then the profiler controller automatically binds the “default-editor” service account for every profile namespace to a default GCP service account created during kubeflow deployment. The Kubeflow deployment process also creates a default profile for the cluster admin.

For more info about profiles see the Multi-user isolation page.

Here is an example profile spec:

apiVersion: kubeflow.org/v1beta1
kind: Profile
spec:
  plugins:
  - kind: WorkloadIdentity
    spec:
      gcpServiceAccount: ${SANAME}@${PROJECT}.iam.gserviceaccount.com
...

You can verify that there is a KSA called default-editor and that it has an annotation of the corresponding GSA:

kubectl -n ${PROFILE_NAME} describe serviceaccount default-editor

...
Name:        default-editor
Annotations: iam.gke.io/gcp-service-account: ${KFNAME}-user@${PROJECT}.iam.gserviceaccount.com
...

You can double check that GSA is also properly set up:

gcloud --project=${PROJECT} iam service-accounts get-iam-policy ${KFNAME}-user@${PROJECT}.iam.gserviceaccount.com

When a pod uses KSA default-editor, it can access GCP APIs with the role granted to the GSA.

Provisioning custom Google service accounts in namespaces: When creating a profile, you can specify a custom GCP service account for the namespace to control which GCP resources are accessible.

Prerequisite: you must have permission to edit your GCP project’s IAM policy and to create a profile custom resource (CR) in your Kubeflow cluster.

if you don’t already have a GCP service account you want to use, create a new one. For example: user1-gcp@<project-id>.iam.gserviceaccount.com:

gcloud iam service-accounts create user1-gcp@<project-id>.iam.gserviceaccount.com

You can bind roles to the GCP service account to allow access to the desired GCP resources. For example to run BigQuery job, you can grant access like so:

gcloud projects add-iam-policy-binding <project-id> \
      --member='serviceAccount:user1-gcp@<project-id>.iam.gserviceaccount.com' \
      --role='roles/bigquery.jobUser'

Grant owner permission of service account user1-gcp@<project-id>.iam.gserviceaccount.com to cluster account <cluster-name>-admin@<project-id>.iam.gserviceaccount.com:

gcloud iam service-accounts add-iam-policy-binding \
      user1-gcp@<project-id>.iam.gserviceaccount.com \
      --member='serviceAccount:<cluster-name>-admin@<project-id>.iam.gserviceaccount.com' --role='roles/owner'

Manually create a profile for user1 and specify the GCP service account to bind in plugins field:

apiVersion: kubeflow.org/v1beta1
kind: Profile
metadata:
  name: profileName   # replace with the name of the profile (the user's namespace name)
spec:
  owner:
    kind: User
    name: user1@email.com   # replace with the email of the user
  plugins:
  - kind: WorkloadIdentity
    spec:
      gcpServiceAccount: user1-gcp@project-id.iam.gserviceaccount.com

Note: The profile controller currently doesn’t perform any access control checks to see whether the user creating the profile should be able to use the GCP service account. As a result, any user who can create a profile can get access to any service account for which the admin controller has owner permissions. We will improve this in subsequent releases.

You can find more details on workload identity in the GKE documentation.

Kubeflow v0.6 and before: GCP service account key as secret

When you set up Kubeflow for GCP, it automatically provisions three service accounts with different privileges in the kubeflow namespace. In particular, the ${KF_NAME}-user service account is meant to grant your user services access to GCP. The credentials to this service account can be accessed within the cluster as a Kubernetes secret called user-gcp-sa.

The secret has basic access to a limited set of GCP services by default, but more roles can be granted through the GCP IAM console.

You can create a PodDefault object to attach the credentials to certain pods.

Credentials

You can add credentials to the new namespace by either copying them from an existing Kubeflow namespace or by creating a new service account.

To copy credentials from one namespace to another namespace use the following CLI commands (Note: there is an issue filed to automate these commands):

NAMESPACE=<new kubeflow namespace>
SOURCE=kubeflow
NAME=user-gcp-sa
SECRET=$(kubectl -n ${SOURCE} get secrets ${NAME} -o jsonpath="{.data.${NAME}\.json}" | base64 -d)
kubectl create -n ${NAMESPACE} secret generic ${NAME} --from-literal="${NAME}.json=${SECRET}"

To create a new service account instead of copying credentials, use the following steps:

Create a service account with the desired roles:

export PROJECT_ID=<GCP project id>
export NAMESPACE=<new kubeflow namespace>
export SA_NAME=<service account name>
export GCPROLES=roles/editor
gcloud --project=${PROJECT_ID} iam service-accounts create $SA_NAME
gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member serviceAccount:$SA_NAME@$PROJECT_ID.iam.gserviceaccount.com \
    --role $GCPROLES

Download the JSON service account key, set KEYPATH to the correct path, and create the key:

export KEYPATH=some/path/${SA_NAME}.gcp.json
gcloud --project=${PROJECT_ID} iam service-accounts keys create ${KEYPATH} \
   --iam-account $SA_NAME@$PROJECT_ID.iam.gserviceaccount.com

Upload the JSON service account key to cluster as a secret:

kubectl create secret generic user-gcp-sa -n $NAMESPACE \ --from-file=user-gcp-sa.json=${KEYPATH}

PodDefault object

The PodDefault object is a way to centrally manage configurations that should be added to all pods.

The PodDefault will match all pods with the specified selector and modify the pods to inject the volumes, secrets, and environment variables listed in the pod manifest.

Create a pod default in a file called add-gcp-secret.yaml and apply it using kubectl apply -f add-gcp-secret.yaml -n $NAMESPACE:

apiVersion: "kubeflow.org/v1alpha1"
kind: PodDefault
metadata:
  name: add-gcp-secret
spec:
 selector:
  matchLabels:
    addgcpsecret: "true"
 desc: "add gcp credential"
 env:
 - name: GOOGLE_APPLICATION_CREDENTIALS
   value: /secret/gcp/user-gcp-sa.json
 volumeMounts:
 - name: secret-volume
   mountPath: /secret/gcp
 volumes:
 - name: secret-volume
   secret:
    secretName: user-gcp-sa

Authentication from a Pod

You must do two things to access a GCP service account from a Pod:

Mount the secret as a file. This gives your Pod access to your GCP account, so be careful which Pods you grant access to.
Set the GOOGLE_APPLICATION_CREDENTIALS environment variable to point to the service account. GCP libraries use this environment variable to find the service account and authenticate with GCP.

The following YAML describes a Pod that has access to the ${KF_NAME}-user service account:

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mypod
    image: myimage
    env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: "/var/secrets/user-sa.json"
    volumeMounts:
    - name: gcp-secret
      mountPath: "/var/secrets/user-sa.json"
      readOnly: true
  volumes:
  - name: gcp-secret
    secret:
      secretName: myappname-user

Authentication from Kubeflow Pipelines

Refer to Authenticating Pipelines to GCP.

Local authentication

gcloud

Use the gcloud tool to interact with Google Cloud Platform (GCP) on the command line. You can use the gcloud command to set up Google Kubernetes Engine (GKE) clusters, and interact with other Google services.

Logging in

You have two options for authenticating the gcloud command:

You can use a user account to authenticate using a Google account (typically Gmail). You can register a user account using gcloud auth login, which brings up a browser window to start the familiar Google authentication flow.
You can create a service account within your GCP project. You can then download a .json key file associated with the account, and run the gcloud auth activate-service-account command to authenticate your gcloud session.

You can find more information in the GCP docs.

Listing active accounts

You can run the following command to verify you are authenticating with the expected account:

gcloud auth list

In the output of the command, an asterisk denotes your active account.

Viewing IAM roles

Permissions are handled in GCP using IAM Roles. These roles define which resources your account can read or write to. Provided you have the necessary permissions, you can check which roles were assigned to your account using the following gcloud command:

PROJECT_ID=your-gcp-project-id-here

gcloud projects get-iam-policy $PROJECT_ID --flatten="bindings[].members" \
    --format='table(bindings.role)' \
    --filter="bindings.members:$(gcloud config list account --format 'value(core.account)')"

You can view and modify roles through the GCP IAM console.

You can find more information about IAM in the GCP docs.

kubectl

The kubectl tool is used for interacting with a Kubernetes cluster through the command line.

Connecting to a cluster using a GCP account

If you set up your Kubernetes cluster using GKE, you can authenticate with the cluster using a GCP account. The following commands fetch the credentials for your cluster and save them to your local kubeconfig file:

CLUSTER_NAME=your-gke-cluster
ZONE=your-gcp-zone

gcloud container clusters get-credentials $CLUSTER_NAME --zone $ZONE

You can find more information in the GCP docs.

Changing active clusters

If you work with multiple Kubernetes clusters, you may have multiple contexts saved in your local kubeconfig file. You can view the clusters you have saved by run the following command:

kubectl config get-contexts

You can view which cluster is currently being controlled by kubectl with the following command:

CONTEXT_NAME=your-new-context

kubectl config set-context $CONTEXT_NAME

You can find more information in the Kubernetes docs.

Checking RBAC permissions

Like GKE IAM, Kubernetes permissions are typically handled with a “role-based authorization control” (RBAC) system. Each Kubernetes service account has a set of authorized roles associated with it. If your account doesn’t have the right roles assigned to it, certain tasks fail.

You can check if an account has the proper permissions to run a command by building a query structured as kubectl auth can-i [VERB] [RESOURCE] --namespace [NAMESPACE]. For example, the following command verifies that your account has permissions to create deployments in the kubeflow namespace:

kubectl auth can-i create deployments --namespace kubeflow

You can find more information in the Kubernetes docs.

Adding RBAC permissions

If you find you are missing a permission you need, you can grant the missing roles to your service account using Kubernetes resources.

Roles describe the permissions you want to assign. For example, verbs: ["create"], resources:["deployments"]
RoleBindings define a mapping between the Role, and a specific service account

By default, Roles and RoleBindings apply only to resources in a specific namespace, but there are also ClusterRoles and ClusterRoleBindings that can grant access to resources cluster-wide

You can find more information in the Kubernetes docs.

Next steps

See the troubleshooting guide for help with diagnosing and fixing issues you may encounter with Kubeflow on GCP

Docs: Using Cloud Filestore

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to set up and use Cloud Filestore with Kubeflow on Google Cloud Platform (GCP).

About Cloud Filestore

Cloud File Store is a fully managed NFS offering. Cloud Filestore is very useful for creating a shared filesystem that can be mounted into pods such as Jupyter.

Before you start

This guide assumes you have already set up Kubeflow on GCP. If you haven’t done so, follow the guide to deploying Kubeflow on GCP.

This guide assumes the following settings:

The ${KF_DIR} environment variable contains the path to your Kubeflow application directory, which holds your Kubeflow configuration files. For example, /opt/my-kubeflow/.
```
export KF_DIR=<path to your Kubeflow application directory>
```
The ${CONFIG_FILE} environment variable contains the path to your Kubeflow configuration file.
```
export CONFIG_FILE=${KF_DIR}/kfctl_gcp_iap.v1.0.2.yaml
```
The ${KF_NAME} environment variable contains the name of your Kubeflow deployment. You can find the name in your ${CONFIG_FILE} configuration file, as the value for the metadata.name key.
```
export KF_NAME=<the name of your Kubeflow deployment>
```
The ${PROJECT} environment variable contains the ID of your GCP project. You can find the project ID in your ${CONFIG_FILE} configuraiton file, as the value for the project key.
```
export PROJECT=<your GCP project ID>
```
The ${ZONE} environment variable contains the GCP zone where your Kubeflow resources are deployed.
```
export ZONE=<your GCP zone>
```
For further background about the above settings, see the guide to deploying Kubeflow with the CLI.

Create a Cloud Filestore instance

Follow these instructions to create a Cloud Filestore instance; if you already have a Cloud Filestore instance you want to use you can skip this section.

Copy the Cloud Filestore deployment manager configs to the gcp_config directory:

cd ${KF_DIR}
cp .cache/manifests/manifests-${VERSION}/deployment/gke/deployment_manager_configs/gcfs.yaml \
   ./gcp_config/

Edit gcfs.yaml to match your desired configuration:

Set zone
Set name
Set the value of parent to include your project e.g.
```
projects/${PROJECT}/locations/${ZONE}
```

Using yq:

cd ${KF_DIR}
. env.sh
yq -r ".resources[0].properties.instanceId=\"${KF_NAME}\"" gcp_config/gcfs.yaml > gcp_config/gcfs.yaml.new
mv gcp_config/gcfs.yaml.new gcp_config/gcfs.yaml

Apply the changes:

cd ${KF_DIR}/gcp_config
gcloud --project=${PROJECT} deployment-manager deployments create ${KF_NAME}-nfs --config=gcfs.yaml

If you get an error legacy networks are not supported follow the instructions in the troubleshooting guide.

Create a PV and PVC to mount the filestore.

Create a PVC for Cloud Filestore instance.

Run the following command to get the ip address of the cloud file store instance

gcloud --project=${PROJECT} filestore instances list

The output will be something like the following and give you the IP address of your instance.

INSTANCE_NAME          ZONE        TIER      CAPACITY_GB  FILE_SHARE_NAME  IP_ADDRESS     STATE  CREATE_TIME
mykubeflow-nfs  us-east1-d  STANDARD  1024         kubeflow         10.20.148.194  READY  2019-05-15T01:23:53

Now follow the instructions Accessing Fileshares from Google Kubernetes to create a PV and PVC.

Using the PVC

With Jupyter

In the UI to create a jupyter notebook you can specify the PVC as an extra data volume.

Docs: Securing Your Clusters

Mon, 01 Jan 0001 00:00:00 +0000

Alpha

This feature is currently in alpha release status with limited support. The Kubeflow team is interested in any feedback you may have, in particular with regards to usability of the feature. Note the following issues already reported:

This guide describes how to secure Kubeflow using VPC Service Controls and private GKE.

Together these two features signficantly increase security and mitigate the risk of data exfiltration.

VPC Service Controls allow you to define a perimeter around Google Cloud Platform (GCP) services.

Kubeflow uses VPC Service Controls to prevent applications running on GKE from writing data to GCP resources outside the perimeter.
Private GKE removes public IP addresses from GKE nodes making them inaccessible from the public internet.

Kubeflow uses IAP to make Kubeflow web apps accessible from your browser.

VPC Service Controls allow you to restrict which Google services are accessible from your GKE/Kubeflow clusters. This is an important part of security and in particular mitigating the risks of data exfiltration.

For more information refer to the VPC Service Control Docs.

Creating a private Kubernetes Engine cluster means the Kubernetes Engine nodes won’t have public IP addresses. This can improve security by blocking unwanted outbound/inbound access to nodes. Removing IP addresses means external services (such as GitHub, PyPi, and DockerHub) won’t be accessible from the nodes. Google services (such as BigQuery and Cloud Storage) are still accessible.

Importantly this means you can continue to use your Google Container Registry (GCR) to host your Docker images. Other Docker registries (for example, DockerHub) will not be accessible. If you need to use Docker images hosted outside GCR you can use the scripts provided by Kubeflow to mirror them to your GCR registry.

Before you start

Before installing Kubeflow ensure you have installed the following tools:

You will need to know your gcloud organization ID and project number; you can get them via gcloud.

export PROJECT=<your GCP project id>
export ORGANIZATION_NAME=<name of your organization>
export ORGANIZATION=$(gcloud organizations list --filter=DISPLAY_NAME=${ORGANIZATION_NAME} --format='value(name)')
export PROJECT_NUMBER=$(gcloud projects describe ${PROJECT} --format='value(projectNumber)')

Projects are identified by names, IDs, and numbers. For more info, see Identifying projects.

Enable VPC Service Controls In Your Project

Enable VPC service controls:

export PROJECT=<Your project>
gcloud services enable accesscontextmanager.googleapis.com \
                       cloudresourcemanager.googleapis.com \
                       dns.googleapis.com  --project=${PROJECT}

Check if you have an access policy object already created:
```
gcloud beta access-context-manager policies list \
    --organization=${ORGANIZATION}
```
- An access policy is a GCP resource object that defines service perimeters. There can be only one access policy object in an organization, and it is a child of the Organization resource.

If you don’t have an access policy object, create one:

gcloud beta access-context-manager policies create \
--title "default" --organization=${ORGANIZATION}

Save the Access Policy Object ID as an environment variable so that it can be used in subsequent commands:

export POLICYID=$(gcloud beta access-context-manager policies list --organization=${ORGANIZATION} --limit=1 --format='value(name)')

Create a service perimeter:
```
gcloud beta access-context-manager perimeters create KubeflowZone \
    --title="Kubeflow Zone" --resources=projects/${PROJECT_NUMBER} \
    --restricted-services=bigquery.googleapis.com,containerregistry.googleapis.com,storage.googleapis.com \
    --project=${PROJECT} --policy=${POLICYID}
```
- Here we have created a service perimeter with the name KubeflowZone.
- The perimeter is created in PROJECT_NUMBER and restricts access to GCS (storage.googleapis.com), BigQuery (bigquery.googleapis.com), and GCR (containerregistry.googleapis.com).
- Placing GCS (Google Cloud Storage) and BigQuery in the perimeter means that access to GCS and BigQuery resources owned by this project is now restricted. By default, access from outside the perimeter will be blocked
- More than one project can be added to the same perimeter
Create an access level to allow Google Container Builder to access resources inside the perimiter:
- Create a members.yaml file with the following contents
```
- members:      
  - serviceAccount:${PROJECT_NUMBER}@cloudbuild.gserviceaccount.com
  - user:<your email>
```
- Google Container Builder is used to mirror Kubeflow images into the perimeter
- Adding your email allows you to access the GCP services inside the perimeter from outside the cluster
  - This is convenient for building and pushing images and data from your local machine.
- For more information refer to the docs.

Create the access level:

gcloud beta access-context-manager levels create kubeflow \
   --basic-level-spec=members.yaml \
   --policy=${POLICYID} \
   --title="Kubeflow ${PROJECT}"

The name for the level can’t have any hyphens

Bind Access Level to a Service Perimeter:

gcloud beta access-context-manager perimeters update KubeflowZone \
 --add-access-levels=kubeflow \
 --policy=${POLICYID}

Set up container registry for GKE private clusters:

Follow the step belows to configure your GCR registry to be accessible from your secured clusters. For more info see instructions.

Create a managed private zone

export ZONE_NAME=kubeflow
export NETWORK=<Network you are using for your cluster>
gcloud beta dns managed-zones create ${ZONE_NAME} \
 --visibility=private \
 --networks=https://www.googleapis.com/compute/v1/projects/${PROJECT}/global/networks/${NETWORK} \
 --description="Kubeflow DNS" \
 --dns-name=gcr.io \
 --project=${PROJECT}

Start a transaction

gcloud dns record-sets transaction start \
 --zone=${ZONE_NAME} \
 --project=${PROJECT}

Add a CNAME record for *.gcr.io

gcloud dns record-sets transaction add \
 --name=*.gcr.io. \
 --type=CNAME gcr.io. \
 --zone=${ZONE_NAME} \
 --ttl=300 \
 --project=${PROJECT}

Add an A record for the restricted VIP

 gcloud dns record-sets transaction add \
   --name=gcr.io. \
   --type=A 199.36.153.4 199.36.153.5 199.36.153.6 199.36.153.7 \
   --zone=${ZONE_NAME} \
   --ttl=300 \
   --project=${PROJECT}

Commit the transaction

 gcloud dns record-sets transaction execute \
  --zone=${ZONE_NAME} \
  --project=${PROJECT}

Mirror Kubeflow Application Images

Since private GKE can only access gcr.io, we need to mirror all images outside gcr.io for Kubeflow applications. We will use the kfctl tool to accomplish this.

Set your user credentials. You only need to run this command once:
```
gcloud auth application-default login
```
Inside your ${KFAPP} directory create a local configuration file mirror.yaml based on this template
1. Change destination to your project gcr registry.
Generate pipeline files to mirror images by running
```
cd ${KFAPP}
./kfctl alpha mirror build mirror.yaml -V -o pipeline.yaml --gcb
```
- If you want to use Tekton rather than Google Cloud Build(GCB) drop --gcb to emit a Tekton pipeline
- The instructions below assume you are using GCB
Edit the couldbuild.yaml file
1. In the images section add
```
 - <registry domain>/<project_id>/docker.io/istio/proxy_init:1.1.6
```
  - Replace <registry domain>/<project_id> with your registry
2. Under steps section add
```
  - args:
  - build
  - -t
  - <registry domain>/<project id>/docker.io/istio/proxy_init:1.1.6
  - --build-arg=INPUT_IMAGE=docker.io/istio/proxy_init:1.1.6
  - .
  name: gcr.io/cloud-builders/docker
  waitFor:
  - '-'  
```
3. Remove the mirroring of cos-nvidia-installer:fixed image. You don’t need it to be replicated because this image is privately available through GKE internal repo.
  1. Remove the images from the images section
  2. Remove it from the steps section

Create a cloud build job to do the mirroring

 gcloud builds submit --async gs://kubeflow-examples/image-replicate/replicate-context.tar.gz --project <project_id> --config cloudbuild.yaml

Update your manifests to use the mirror’d images
```
kfctl alpha mirror overwrite -i pipeline.yaml
```
Edit file “kustomize/istio-install/base/istio-noauth.yaml”:
1. Replace docker.io/istio/proxy_init:1.16 to gcr.io/<project_id>/docker.io/istio/proxy_init:1.16
2. Replace docker.io/istio/proxyv2:1.1.6 to gcr.io/<project_id>/docker.io/istio/proxyv2:1.1.6

Deploy Kubeflow with Private GKE

Coming Soon

You can follow the issue: Documentation on how to use Kubeflow with private GKE and VPC service controls

Next steps

Use GKE Authorized Networks to restrict access to your GKE master
Learn more about VPC Service Controls
See how to delete your Kubeflow deployment using the CLI.
Troubleshoot any issues you may find.

Docs: Troubleshooting Deployments on GKE

Mon, 01 Jan 0001 00:00:00 +0000

This guide helps diagnose and fix issues you may encounter with Kubeflow on Google Kubernetes Engine (GKE) and Google Cloud Platform (GCP).

Before you start

This guide covers troubleshooting specifically for Kubeflow deployments on GCP.

For more help, try the general Kubeflow troubleshooting guide.

This guide assumes the following settings:

The ${KF_DIR} environment variable contains the path to your Kubeflow application directory, which holds your Kubeflow configuration files. For example, /opt/my-kubeflow/.
```
export KF_DIR=<path to your Kubeflow application directory>
```

The ${CONFIG_FILE} environment variable contains the path to your Kubeflow configuration file.

export CONFIG_FILE=${KF_DIR}/kfctl_gcp_iap.v1.0.2.yaml

Or:

export CONFIG_FILE=${KF_DIR}/kfctl_gcp_basic_auth.v1.0.2.yaml

The ${KF_NAME} environment variable contains the name of your Kubeflow deployment. You can find the name in your ${CONFIG_FILE} configuration file, as the value for the metadata.name key.
```
export KF_NAME=<the name of your Kubeflow deployment>
```
The ${PROJECT} environment variable contains the ID of your GCP project. You can find the project ID in your ${CONFIG_FILE} configuration file, as the value for the project key.
```
export PROJECT=<your GCP project ID>
```
The ${ZONE} environment variable contains the GCP zone where your Kubeflow resources are deployed.
```
export ZONE=<your GCP zone>
```
For further background about the above settings, see the guide to deploying Kubeflow with the CLI.

Troubleshooting Kubeflow deployment on GCP

Here are some tips for troubleshooting GCP.

Make sure you are a GCP project owner.
Make sure you are using HTTPS.
Check project quota page to see if any service’s current usage reached quota limit, increase them as needed.
Check deployment manager page and see if there’s a failed deployment.
Check if endpoint is up: do DNS lookup against your Cloud Identity-Aware Proxy (Cloud IAP) URL and see if it resolves to the correct IP address.
Check if certificate succeeded: kubectl describe certificates -n istio-system should give you certificate status.
Check ingress status: kubectl describe ingress -n istio-system
Check if endpoint entry is created. There should be one entry with name <deployment>.endpoints.<project>.cloud.goog
- If endpoint entry doesn’t exist, check kubectl describe cloudendpoint -n istio-system
If using IAP: make sure you added https://<deployment>.endpoints.<project>.cloud.goog/_gcp_gatekeeper/authenticate as an authorized redirect URI for the OAUTH credentials used to create the deployment.
If using IAP: see the guide to monitoring your Cloud IAP setup.
See the sections below for troubleshooting specific problems.
Please report a bug if you can’t resolve the problem by following the above steps.

DNS name not registered

This section provides troubleshooting information for problems creating a DNS entry for your ingress. The ingress is a K8s resource that creates a GCP loadbalancer to enable http(s) access to Kubeflow web services from outside the cluster. This section assumes you are using Cloud Endpoints and a DNS name of the following pattern

https://${KF_NAME}.endpoints.${PROJECT}.cloud.goog

Symptoms:

When you access the the URL in Chrome you get the error: server IP address could not be found

nslookup for the domain name doesn’t return the IP address associated with the ingress

nslookup ${KF_NAME}.endpoints.${PROJECT}.cloud.goog
Server:   127.0.0.1
Address:  127.0.0.1#53

** server can't find ${KF_NAME}.endpoints.${PROJECT}.cloud.goog: NXDOMAIN

Troubleshooting

Check the cloudendpoints resource
```
kubectl get cloudendpoints -o yaml ${KF_NAME}
kubectl describe cloudendpoints ${KF_NAME}
```
- Check if there are errors indicating problems creating the endpoint

The status of the cloudendpoints object will contain the cloud operation used to register the operation

For example

 status:
   config: ""
   configMapHash: ""
   configSubmit: operations/serviceConfigs.jlewi-1218-001.endpoints.cloud-ml-dev.cloud.goog:43fe6c6f-eb9c-41d0-ac85-b547fc3e6e38
   endpoint: jlewi-1218-001.endpoints.cloud-ml-dev.cloud.goog
   ingressIP: 35.227.243.83
   jwtAudiences: null
   lastAppliedSig: 4f3b903a06a683b380bf1aac1deca72792472429
   observedGeneration: 1
   stateCurrent: ENDPOINT_SUBMIT_PENDING

You can check the status of the operation by running:
```
gcloud --project=${PROJECT} endpoints operations describe ${OPERATION}
```
- Operation is everything after operations/ in the configSubmit field

404 Page Not Found When Accessing Central Dashboard

This section provides troubleshooting information for 404s, page not found, being return by the central dashboard which is served at

https://${KUBEFLOW_FQDN}/

KUBEFLOW_FQDN is your project’s OAuth web app URI domain name <name>.endpoints.<project>.cloud.goog
Since we were able to sign in this indicates the Ambassador reverse proxy is up and healthy we can confirm this is the case by running the following command

kubectl -n ${NAMESPACE} get pods -l service=envoy

NAME                     READY     STATUS    RESTARTS   AGE
envoy-76774f8d5c-lx9bd   2/2       Running   2          4m
envoy-76774f8d5c-ngjnr   2/2       Running   2          4m
envoy-76774f8d5c-sg555   2/2       Running   2          4m

Try other services to see if they’re accessible for example

https://${KUBEFLOW_FQDN}/whoami
https://${KUBEFLOW_FQDN}/tfjobs/ui
https://${KUBEFLOW_FQDN}/hub

If other services are accessible then we know its a problem specific to the central dashboard and not ingress

Check that the centraldashboard is running

kubectl get pods -l app=centraldashboard
NAME                                READY     STATUS    RESTARTS   AGE
centraldashboard-6665fc46cb-592br   1/1       Running   0          7h

Check a service for the central dashboard exists
```
kubectl get service -o yaml centraldashboard
```

Check that an Ambassador route is properly defined

kubectl get service centraldashboard -o jsonpath='{.metadata.annotations.getambassador\.io/config}'

apiVersion: ambassador/v0
  kind:  Mapping
  name: centralui-mapping
  prefix: /
  rewrite: /
  service: centraldashboard.kubeflow,

Check the logs of Ambassador for errors. See if there are errors like the following indicating an error parsing the route.If you are using the new Stackdriver Kubernetes monitoring you can use the following filter in the stackdriver console
```
 resource.type="k8s_container"
 resource.labels.location=${ZONE}
 resource.labels.cluster_name=${CLUSTER}
 metadata.userLabels.service="ambassador"
"could not parse YAML"
```

502 Server Error

A 502 usually means traffic isn’t even making it to the envoy reverse proxy. And it usually indicates the loadbalancer doesn’t think any backends are healthy.

In Cloud Console select Network Services -> Load Balancing
- Click on the load balancer (the name should contain the name of the ingress)
- The exact name can be found by looking at the ingress.kubernetes.io/url-map annotation on your ingress object
```
URLMAP=$(kubectl --namespace=${NAMESPACE} get ingress envoy-ingress -o jsonpath='{.metadata.annotations.ingress\.kubernetes\.io/url-map}')
echo ${URLMAP}
```
- Click on your loadbalancer
- This will show you the backend services associated with the load balancer
  - There is 1 backend service for each K8s service the ingress rule routes traffic too
  - The named port will correspond to the NodePort a service is using
```
NODE_PORT=$(kubectl --namespace=${NAMESPACE} get svc envoy -o jsonpath='{.spec.ports[0].nodePort}')
BACKEND_NAME=$(gcloud compute --project=${PROJECT} backend-services list --filter=name~k8s-be-${NODE_PORT}- --format='value(name)')
gcloud compute --project=${PROJECT} backend-services get-health --global ${BACKEND_NAME}
```
- Make sure the load balancer reports the backends as healthy
  - If the backends aren’t reported as healthy check that the pods associated with the K8s service are up and running
  - Check that health checks are properly configured
    - Click on the health check associated with the backend service for envoy
    - Check that the path is /healthz and corresponds to the path of the readiness probe on the envoy pods
    - See K8s docs for important information about how health checks are determined from readiness probes.
  - Check firewall rules to ensure traffic isn’t blocked from the GCP loadbalancer
    - The firewall rule should be added automatically by the ingress but its possible it got deleted if you have some automatic firewall policy enforcement. You can recreate the firewall rule if needed with a rule like this
```
gcloud compute firewall-rules create $NAME \
--project $PROJECT \
--allow tcp:$PORT \
--target-tags $NODE_TAG \
--source-ranges 130.211.0.0/22,35.191.0.0/16
```
    - To get the node tag
```
# From the Kubernetes Engine cluster get the name of the managed instance group
gcloud --project=$PROJECT container clusters --zone=$ZONE describe $CLUSTER
# Get the template associated with the MIG
gcloud --project=kubeflow-rl compute instance-groups managed describe --zone=${ZONE} ${MIG_NAME}
# Get the instance tags from the template
gcloud --project=kubeflow-rl compute instance-templates describe ${TEMPLATE_NAME}
```
      For more info see GCP HTTP health check docs
- In Stackdriver Logging look at the Cloud Http Load Balancer logs
  - Logs are labeled with the forwarding rule
  - The forwarding rules are available via the annotations on the ingress
```
ingress.kubernetes.io/forwarding-rule
ingress.kubernetes.io/https-forwarding-rule
```
- Verify that requests are being properly routed within the cluster
- Connect to one of the envoy proxies
```
kubectl exec -ti `kubectl get pods --selector=service=envoy -o jsonpath='{.items[0].metadata.name}'` /bin/bash
```
- Install curl in the pod
```
apt-get update && apt-get install -y curl
```
- Verify access to the whoami app
```
curl -L -s -i http://envoy:8080/noiap/whoami
```
- If this doesn’t return a 200 OK response; then there is a problem with the K8s resources
  - Check the pods are running
  - Check services are pointing at the points (look at the endpoints for the various services)

GKE Certificate Fails To Be Provisioned

A common symptom of your certificate failing to be provisioned is SSL errors like ERR_SSL_VERSION_OR_CIPHER_MISMATCH when you try to access the Kubeflow https endpoint.

To troubleshoot check the status of your GKE managed certificate

kubectl -n istio-system describe managedcertificate

If the certificate is in status FailedNotVisible then it means GCP failed to provision the certificate because it could not verify that you owned the domain by doing an ACME challenge. In order for GCP to provision your certificate

Your ingress must be created in order to associated a Google Cloud Load Balancer(GCLB) with the IP address for your endpoint
There must be a DNS entry mapping your domain name to the IP.

If there is a problem preventing either of the above then GCP will be unable to provision your certificate and eventually enter the permanent failure state FailedNotVisible indicating your endpoint isn’t accessible. The most common cause is the ingress can’t be created becuase the K8s secret containing OAuth credentials doesn’t exist.

To fix this you must first resolve the underlying problems preventing your ingress or DNS entry from being created. Once the underlying problem has been fixed you can follow the steps below to force a new certificate to be generated.

You can fix the certificate by performing the following steps to delete the existing certificate and create a new one.

Get the name of the GCP certificate
```
kubectl -n istio-system describe managedcertificate gke-certificate
```
- The status will contain Certificate Name which will start with mcrt make a note of this.

Delete the ingress

kubectl -n istio-system delete ingress envoy-ingress

Ensure the certificate was deleted
```
gcloud --project=${PROJECT} compute ssl-certificates list
```
- Make sure the certificate obtained in the first step no longer exists
Reapply kubeflow in order to recreate the ingress and certificate
- If you deployed with kfctl rerun kfctl apply
- If you deployed using the GCP blueprint rerun make apply-kubeflow

Monitor the certificate to make sure it can be provisioned

kubectl --context=gcp-private-0527 -n istio-system describe managedcertificate gke-certificate

Since the ingress has been recreated we need to restart the pods that configure it

kubectl -n istio-system delete pods -l service=backend-updater
kubectl -n istio-system delete pods -l service=iap-enabler

Problems with SSL certificate from Let’s Encrypt

As of Kubeflow 1.0, Kubeflow should be using GKE Managed Certificates and no longer using Let’s Encrypt.

See the guide to monitoring your Cloud IAP setup.

Envoy pods crash-looping: root cause is backend quota exceeded

If your logs show the Envoy pods crash-looping, the root cause may be that you have exceeded your quota for some backend services such as loadbalancers. This is particularly likely if you have multiple, differently named deployments in the same GCP project using Cloud IAP.

The error

The error looks like this for the pod’s Envoy container:

kubectl logs -n kubeflow envoy-79ff8d86b-z2snp envoy
[2019-01-22 00:19:44.400][1][info][main] external/envoy/source/server/server.cc:184] initializing epoch 0 (hot restart version=9.200.16384.127.options=capacity=16384, num_slots=8209 hash=228984379728933363)
[2019-01-22 00:19:44.400][1][critical][main] external/envoy/source/server/server.cc:71] error initializing configuration '/etc/envoy/envoy-config.json': unable to read file: /etc/envoy/envoy-config.json

And the Cloud IAP container shows a message like this:

Waiting for backend id PROJECT=<your-project> NAMESPACE=kubeflow SERVICE=envoy filter=name~k8s-be-30352-...

Diagnosing the cause

You can verify the cause of the problem by entering the following command:

kubectl -n istio-system describe ingress

Look for something like this in the output:

Events:
  Type     Reason  Age                  From                     Message
  ----     ------  ----                 ----                     -------
  Warning  Sync    14m (x193 over 19h)  loadbalancer-controller  Error during sync: googleapi: Error 403: Quota 'BACKEND_SERVICES' exceeded. Limit: 5.0 globally., quotaExceeded

Fixing the problem

If you have any redundant Kubeflow deployments, you can delete them using the Deployment Manager.

Alternatively, you can request more backend services quota on the GCP Console.

Go to the quota settings for backend services on the GCP Console.
Click EDIT QUOTAS. A quota editing form opens on the right of the screen.
Follow the form instructions to apply for more quota.

Legacy networks are not supported

Cloud Filestore and GKE try to use the network named default by default. For older projects, this will be a legacy network which is incompatible with Cloud Filestore and newer GKE features like private clusters. This will manifest as the error “default is invalid; legacy networks are not supported” when deploying Kubeflow.

Here’s an example error when deploying Cloud Filestore:

ERROR: (gcloud.deployment-manager.deployments.update) Error in Operation [operation-1533189457517-5726d7cfd19c9-e1b0b0b5-58ca11b8]: errors:
- code: RESOURCE_ERROR
  location: /deployments/jl-0801-b-gcfs/resources/filestore
  message: '{"ResourceType":"gcp-types/file-v1beta1:projects.locations.instances","ResourceErrorCode":"400","ResourceErrorMessage":{"code":400,"message":"network
    default is invalid; legacy networks are not supported.","status":"INVALID_ARGUMENT","statusMessage":"Bad
    Request","requestPath":"https://file.googleapis.com/v1beta1/projects/cloud-ml-dev/locations/us-central1-a/instances","httpMethod":"POST"}}'

To fix this we can create a new network:

cd ${KF_DIR}
cp .cache/master/deployment/gke/deployment_manager_configs/network.* \
   ./gcp_config/

Edit network.yaml to set the name for the network.

Edit gcfs.yaml to use the name of the newly created network.

Apply the changes.

cd ${KF_DIR}
kfctl apply -V -f ${CONFIG}

CPU platform unavailable in requested zone

By default, we set minCpuPlatform to Intel Haswell to make sure AVX2 is supported. See troubleshooting for more details.

If you encounter this CPU platform unavailable error (might manifest as Cluster is currently being created, deleted, updated or repaired and cannot be updated.), you can change the zone or change the minCpuPlatform. See here for available zones and cpu platforms.

Changing the OAuth client used by IAP

If you need to change the OAuth client used by IAP, you can run the following commands to replace the Kubernetes secret containing the ID and secret.

kubectl -n kubeflow delete secret kubeflow-oauth
kubectl -n kubeflow create secret generic kubeflow-oauth \
       --from-literal=client_id=${CLIENT_ID} \
       --from-literal=client_secret=${CLIENT_SECRET}

Troubleshooting SSL certificate errors

This section describes how to enable service management API to avoid managed certificates failure.

To check your certificate:

Run the following command:
```
kubectl -n istio-system describe managedcertificate gke-certificate
```
Make sure the certificate status is either Active or Provisioning which means it is not ready. For more details on certificate status, refer to the certificate statuses descriptions section. Also, make sure the domain name is correct.
Run the following command to look for the errors using the certificate name from the previous step:
```
gcloud beta --project=${PROJECT} compute ssl-certificates describe --global ${CERTIFICATE_NAME}
```
Run the following command:
```
kubectl -n istio-system get ingress envoy-ingress -o yaml
```
Make sure of the following:
- networking.gke.io/managed-certificates annotation value points to the name of the Kubernetes managed certificate resource and is gke-certificate;
- public IP address that is displayed in the status is assigned. See the example of IP address below:
```
status:
  loadBalancer:
    ingress:
     - ip: 35.186.212.202
```
- DNS entry for the domain has propogated. To verify this, use the following nslookup command example:
```
`nslookup ${DOMAIN}`
```
- domain name is the fully qualified domain name which be the host value in the ingress. See the example below:
```
${KF_APP_NAME}.endpoints.${PROJECT}.cloud.goog
```
Note that managed certificates cannot provision the certificate if the DNS lookup does not work properly.

Docs: Tutorial: End-to-end Kubeflow on GCP

Mon, 01 Jan 0001 00:00:00 +0000

This guide walks you through an end-to-end example of Kubeflow on Google Cloud Platform (GCP) using a Jupyter notebook, mnist_gcp.ipynb. By working through the notebook, you learn how to deploy Kubeflow on Kubernetes Engine (GKE), train an MNIST machine learning model for image classification, and use the model for online inference (also known as online prediction).

Introductions

Overview of GCP and GKE

Google Cloud Platform (GCP) is a suite of cloud computing services running on Google infrastructure. The services include compute power, data storage, data analytics, and machine learning.

The Cloud SDK is a set of tools that you can use to interact with GCP from the command line, including the gcloud command and others.

Kubernetes Engine (GKE) is a managed service on GCP where you can deploy containerized applications. You describe the resources that your application needs, and GKE provisions and manages the underlying cloud resources.

Here’s a list of the primary GCP services that you use when following this guide:

The model and the data

This tutorial trains a TensorFlow model on the MNIST dataset, which is the hello world for machine learning.

The MNIST dataset contains a large number of images of hand-written digits in the range 0 to 9, as well as the labels identifying the digit in each image.

After training, the model can classify incoming images into 10 categories (0 to 9) based on what it’s learned about handwritten images. In other words, you send an image to the model, and the model does its best to identify the digit shown in the image.

In the above screenshot, the image shows a hand-written 7. This image was the input to the model. The table below the image shows a bar graph for each classification label from 0 to 9, as output by the model. Each bar represents the probability that the image matches the respective label. Judging by this screenshot, the model seems pretty confident that this image is a 7.

The overall workflow

The following diagram shows what you accomplish by following this guide:

In summary:

Setting up Kubeflow in a GKE cluster.
Training the model:
- Packaging a TensorFlow program in a Kubernetes container.
- Uploading the container to Container Registry.
- Submitting a TensorFlow training (tf.train) job.
Using the model for prediction (inference):
- Saving the trained model to Cloud Storage.
- Using TensorFlow Serving to serve the model.
- Running a simple web app to send a prediction request to the model and display the result.

It’s time to get started!

Set up and run the MNIST tutorial on GCP

Follow the GCP instructions to deploy Kubeflow with Cloud Identity-Aware Proxy (IAP).
Launch a Jupyter notebook in your Kubeflow cluster. See the guide to setting up your notebooks. Note: This tutorial has been tested with the Tensorflow 1.15 CPU image as the baseline image for the notebook.
Launch a terminal in Jupyter and clone the Kubeflow examples repository:
```
git clone https://github.com/kubeflow/examples.git git_kubeflow-examples
```
- Tip: When you start a terminal in Jupyter, run the command bash to start a bash terminal which is much more friendly than the default shell.
- Tip: You can change the URL for your notebook from ‘/tree’ to ‘/lab’ to switch to using JupyterLab.
Open the notebook mnist/mnist_gcp.ipynb.
Follow the instructions in the notebook to train and deploy MNIST on Kubeflow.

Docs: Logging and monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Prometheus is a monitoring tool often used with Kubernetes. If you configure Kubernetes Engine Monitoring and include Prometheus support, then the metrics that are generated by services using the Prometheus exposition format can be exported from the cluster and made visible as external metrics in Cloud Monitoring.

To configure and use Prometheus with Kubernetes Engine Monitoring, see the GCP documentation.