Kubeflow – Multi-Tenancy in Kubeflow

Docs: Introduction to Multi-user Isolation

Mon, 01 Jan 0001 00:00:00 +0000

This Kubeflow component has stable status. See the Kubeflow versioning policies.

In a production environment, it is often necessary to share the same pool of resources across different teams and users. These different users need a reliable way to isolate and protect their own resources, without accidentally viewing or changing each other’s resources.

Kubeflow v1.0.2 supports multi-user isolation, which applies access control over namespaces and user-created resources in a deployment. This feature provides the users with the convenience of clutter-free browsing of notebooks, training jobs, serving deployments and other resources. The isolation mechanisms also prevent accidental deletion/modification of resources of other users in the deployment.

Note that the isolation support in Kubeflow doesn’t provide any hard security guarantees against malicious attempts by users to infiltrate other user’s profiles.

Key concepts

Administrator: An administrator is someone who creates and maintains the Kubeflow cluster. This person has the permission to grant access permissions to others.

User: A user is someone who has access to some set of resources in the cluster. A user needs to be granted access permissions by the administrator.

Profile: A profile is a grouping of all Kubernetes clusters owned by a user.

Current integration and limitations

The Jupyter notebooks service is the first application to be fully integrated with multi-user isolation. Access to the notebooks and the creation of notebooks is controlled by the profile access policies set by the administrator or the owners of the profiles. Resources created by the notebooks (for example, training jobs and deployments) also inherit the same access.

Metadata and Pipelines or any other applications currently don’t have full fledged integration with isolation, though they do have access to the user identity through the headers of the incoming requests. It’s up to the individual applications to use the available identity and isolation features in a way that makes sense for each application.

On Google Cloud Platform (GCP), the authentication and identify token is generated by GCP IAM and carried through the requests as a JWT Token in the request header. Other cloud providers can have a similar header to provide identity information.

For on-premises deployments, Kubeflow uses Dex as a federated OpenID connection provider and can be integrated with LDAP or Active Directory to provide authentication and identity services.

Next steps

Understand the detailed design of Kubeflow’s multi-user isolation feature.
Learn how to use multi-user isolation and profiles.

Docs: Design for Multi-user Isolation

Mon, 01 Jan 0001 00:00:00 +0000

This Kubeflow component has stable status. See the Kubeflow versioning policies.

Design overview

Kubeflow multi-tenancy is currently built around user namespaces. Specifically, Kubeflow defines user-specific namespaces and uses Kubernetes role-based access control (RBAC) policies to manage user access.

This feature enables users to share access to their workspaces. Workspace owners can share/revoke workspace access with other users through the Kubeflow UI. After being invited, users have permissions to edit the workspace and operate Kubeflow custom resources.

Kubeflow multi-tenancy is self-served - a new user can self-register to create and own their workspace through the UI.

Kubeflow uses Istio to control in-cluster traffic. By default, requests to user workspaces are denied unless allowed by Istio RBAC. In-bound user requests are identified using an identity provider (for example, Identity Aware Proxy (IAP) on Google Cloud or Dex for on-premises deployments), and then validated by Istio RBAC rules.

Internally, Kubeflow uses the Profile custom resource to control all policies, roles, and bindings involved, and to guarantee consistency. Kubeflow also offers a plugin interface to manage external resource/policy outside Kubernetes, for example interfacing with Amazon Web Services APIs for identity management.

The following diagram illustrates a Kubeflow multi-tenancy cluster with two user-access routes: via the Kubeflow central dashboard and via the kubectl command-line interface (CLI).

Feature requirements

Kubeflow uses Istio to apply access control over in-cluster traffic.
Kubeflow profile controller needs cluster admin permission.
Kubeflow UI needs to be served behind an identity aware proxy. The identity aware proxy and Kubernetes master should share the same identity management.
The Kubeflow installation on Google Cloud uses GKE and IAP.
On-premises installations of Kubeflow make use of Dex, a flexible OpenID Connect (OIDC) provider.

Supported platforms

Kubeflow multi-tenancy is enabled by default if you deploy Kubeflow on GCP with IAP.
If you are not on GCP, you can deploy multi-tenancy to your existing cluster.

Next steps

Learn how to use multi-user isolation and profiles.
Read more about Istio in Kubeflow.

Docs: Getting Started with Multi-user Isolation

Mon, 01 Jan 0001 00:00:00 +0000

This Kubeflow component has stable status. See the Kubeflow versioning policies.

Usage overview

After Kubeflow is installed and configured, you will by default be accessing your primary profile. A profile owns a Kubernetes namespace of the same name along with a collection of Kubernetes resources. Users have view and modify access to their primary profiles. You can share access to your profile with another user in the system. When sharing the access to a profile with another user, you can choose to whether to provide only read access or read/modify access. For all practical purposes when working through the Kubeflow central dashboard, the active namespace is directly tied with the active profile.

Example of usage

You can select your active profile from the top bar on the Kubeflow central dashboard. Note that you can only see the profiles to which you have view or modify access.

This guide illustrates the user isolation functionality using the Jupyter notebooks service which is the first service in the system to have full integration with the multi-user isolation functionality.

After you select an active profile, the Notebooks Servers UI displays only the active notebook servers in the currently selected profile. All other notebook servers remain hidden from you. If you switch the active profile, the view switches the list of active notebooks appropriately. You can connect to any of the listed notebook servers and view and modify the existing Jupyter notebooks available in the server.

For example, the following image shows the list of notebook servers available in a user’s primary profile:

When an unauthorized user accesses the notebooks in this profile, they see an error:

When you create Jupyter notebook servers from the Notebooks Servers UI, the notebook pods are created in your active profile. If you don’t have modify access to the active profile, you can only browse currently active notebook servers and access the existing notebooks but cannot create new notebook servers in that profile. You can create notebook servers in your primary profile which you have view and modify access to.

Onboarding a new user

Kubeflow v1.0.2 provides automatic profile creation for authenticated users on first login. Additionally, an administrator can create a profile for any user in the Kubeflow cluster. Here an administrator is a person who has cluster-admin role binding in the Kubernetes cluster. This person has permissions to create and modify Kubernetes resources in the cluster. For example, the person who deployed Kubeflow will have administration privileges in the cluster.

Pre-requisites: grant user minimal Kubernetes cluster access

You must grant each user the minimal permission scope that allows them to connect to the Kubernetes cluster.

For example, for Google Cloud Platform (GCP) users, you should grant the following Cloud Identity and Access Management (IAM) roles. In the following commands, replace [PROJECT] with your GCP project and replace [EMAIL] with the user’s email address:

To access the Kubernetes cluster, the user needs the Kubernetes Engine Cluster Viewer role:

gcloud projects add-iam-policy-binding [PROJECT] --member=user:[EMAIL] --role=roles/container.clusterViewer

To access the Kubeflow UI through IAP, the user needs the IAP-secured Web App User role:

gcloud projects add-iam-policy-binding [PROJECT] --member=user:[EMAIL] --role=roles/iap.httpsResourceAccessor

To be able to run gcloud get credentials and see logs in Cloud Logging (formerly Stackdriver), the user needs viewer access on the project:
```
gcloud projects add-iam-policy-binding [PROJECT] --member=user:[EMAIL] --role=roles/viewer
```

Automatic creation of profiles

Kubeflow v1.0.2 provides automatic profile creation:

The Kubeflow deployment process automatically creates a profile for the user performing the deployment. When the user access the Kubeflow central dashboard they see their profile in the dropdown list.
When an authenticated user logs into the system and visits the central dashboard for the first time, they trigger a profile creation automatically.
- A brief message introduces profiles:
- The user can name their profile and click Finish:
- This redirects the user to the dashboard where they can view and select their profile in the dropdown list.

Manual profile creation

An administrator can manually create profiles for users as described below.

Create a profile.yaml file with the following content on your local machine:

apiVersion: kubeflow.org/v1beta1
kind: Profile
metadata:
  name: profileName   # replace with the name of profile you want, this will be user's namespace name
spec:
  owner:
    kind: User
    name: userid@email.com   # replace with the email of the user

Run the following command to create the corresponding profile resource:

kubectl create -f profile.yaml

The above command creates a profile named profileName. The profile owner is userid@email.com and has view and modify access to that profile. The following resources are created as part of the profile creation:

A Kubernetes namespace that shares the same name with the corresponding profile.
Kubernetes RBAC role binding for the namespace: Admin. This makes the profile owner the namespace administrator, thus giving them access to the namespace using kubectl (via the Kubernetes API).
Istio namespace-scoped ServiceRole: ns-access-istio. This allows access to all services in the target namespace via Istio routing.
Istio namespace-scoped ServiceRoleBinding: owner-binding-istio. This binds the ServiceRole ns-access-istio to the profile owner. The profile owner can therefore access services in the namespace.
Namespace-scoped service-accounts editor and viewer to be used by user-created pods in the namespace.

Note: Due to a one-to-one correspondence of profiles with Kubernetes namespaces, the terms profile and namespace are sometimes used interchangably in the documentation.

Batch creation of user profiles

Administrators might want to create profiles for multiple users as a batch. You can do this by creating a profile.yaml on the local machine with multiple sections of profile descriptions as shown below:

apiVersion: kubeflow.org/v1beta1
kind: Profile
metadata:
  name: profileName1   # replace with the name of profile you want
spec:
  owner:
    kind: User
    name: userid1@email.com   # replace with the email of the user
---
apiVersion: kubeflow.org/v1beta1
kind: Profile
metadata:
  name: profileName2   # replace with the name of profile you want
spec:
  owner:
    kind: User
    name: userid2@email.com   # replace with the email of the user

Run the following command to apply the namespaces to the Kubernetes cluster:

kubectl create -f profile.yaml

This will create multiple profiles, one for each individual listed in the sections in profile.yaml.

Listing and describing profiles

An administrator can list the existing profiles in the system:

$ kubectl get profiles

and describe a specific profile using:

$ kubectl describe profile profileName

Deleting an existing profile

An administrator can delete an existing profile using:

$ kubectl delete profile profileName

This will delete the profile, the corresponding namespace and any Kubernetes resources associated with the profile. The profile’s owner or other users with access to the profile will no longer have access to the profile and will not see it in the dropdown list on the central dashboard.

Managing contributors through the Kubeflow UI

Kubeflow v1.0.2 allows sharing of profiles with other users in the system. An owner of a profile can share access to their profile using the Manage Contributors tab available through the dashboard.

Here is an example of the Manage Contributors tab view:

Notice that in the above view the account associated with the profile is a cluster administrator (Cluster Admin) as this account was used to deploy Kubeflow. The view lists the profiles accessible to the user along with the role associated with that profile.

To add or remove a contributor, add/remove the email address or the user identifier in the Contributors to your namespace field.

The Manage Contributors tab shows the contributors that the namespace owner has added. Note that the cluster administrator can view all the profiles in the system along with their contributors.

The contributors have access to all the Kubernetes resources in the namespace and can create notebook servers as well as access existing notebooks.