Kubeflow – Tools for Serving

Docs: Overview

Mon, 01 Jan 0001 00:00:00 +0000

Kubeflow supports two model serving systems that allow multi-framework model serving: KFServing and Seldon Core. Alternatively, you can use a standalone model serving system. This page gives an overview of the options, so that you can choose the framework that best supports your model serving requirements.

Multi-framework serving with KFServing or Seldon Core

KFServing and Seldon Core are both open source systems that allow multi-framework model serving. The following table compares KFServing and Seldon Core. A check mark (✓) indicates that the system (KFServing or Seldon Core) supports the feature specified in that row.

Feature	Sub-feature	KFServing	Seldon Core
Framework	TensorFlow	✓ sample	✓ docs
	XGBoost	✓ sample	✓ docs
	scikit-learn	✓ sample	✓ docs
	NVIDIA Triton Inference Server	✓ sample	✓ docs
	ONNX	✓ sample
	PyTorch	✓ sample	✓
Graph	Transformers	✓ sample	✓ docs
	Combiners	Roadmap	✓ sample
	Routers including MAB	Roadmap	✓ docs
Analytics	Explanations	✓ sample	✓ docs
Scaling	Knative	✓ sample
	GPU AutoScaling	✓ sample
	HPA	✓	✓ docs
Custom	Container	✓ sample	✓ docs
	Language Wrappers		✓ Python, Java, R
	Multi-Container		✓ docs
Rollout	Canary	✓ sample	✓ docs
	Shadow		✓
Istio		✓	✓

Notes:

KFServing and Seldon Core share some technical features, including explainability (using Seldon Alibi Explain) and payload logging, as well as other areas.
A commercial product, Seldon Deploy, supports both KFServing and Seldon in production.
KFServing is part of the Kubeflow project ecosystem. Seldon Core is an external project supported within Kubeflow.

Further information:

KFServing:
Seldon Core

TensorFlow Serving

For TensorFlow models you can use TensorFlow Serving for real-time prediction. However, if you plan to use multiple frameworks, you should consider KFServing or Seldon Core as described above.

NVIDIA Triton Inference Server

NVIDIA Triton Inference Server is a REST and GRPC service for deep-learning inferencing of TensorRT, TensorFlow, Pytorch, ONNX and Caffe2 models. The server is optimized to deploy machine learning algorithms on both GPUs and CPUs at scale. Triton Inference Server was previously known as TensorRT Inference Server.

You can use NVIDIA Triton Inference Server as a standalone system, but you should consider KFServing as described above. KFServing includes support for NVIDIA Triton Inference Server.

BentoML

BentoML is an open-source platform for high-performance ML model serving. It makes building production API endpoint for your ML model easy and supports all major machine learning training frameworks, including Tensorflow, Keras, PyTorch, XGBoost, scikit-learn and etc.

BentoML comes with a high-performance API model server with adaptive micro-batching support, which achieves the advantage of batch processing in online serving. It also provides model management and model deployment functionality, giving ML teams an end-to-end model serving workflow, with DevOps best practices baked in.

Docs: KFServing

Mon, 01 Jan 0001 00:00:00 +0000

Beta

This Kubeflow component has beta status. See the Kubeflow versioning policies. The Kubeflow team is interested in your feedback about the usability of the feature.

KFServing enables serverless inferencing on Kubernetes and provides performant, high abstraction interfaces for common machine learning (ML) frameworks like TensorFlow, XGBoost, scikit-learn, PyTorch, and ONNX to solve production model serving use cases.

You can use KFServing to do the following:

Provide a Kubernetes Custom Resource Definition for serving ML models on arbitrary frameworks.
Encapsulate the complexity of autoscaling, networking, health checking, and server configuration to bring cutting edge serving features like GPU autoscaling, scale to zero, and canary rollouts to your ML deployments.
Enable a simple, pluggable, and complete story for your production ML inference server by providing prediction, pre-processing, post-processing and explainability out of the box.

Our strong community contributions help KFServing to grow. We have a Technical Steering Committee driven by Google, IBM, Microsoft, Seldon, and Bloomberg. Browse the KFServing GitHub repo to give us feedback!

Install with Kubeflow

KFServing works with Kubeflow 0.7. Kustomize installation files are located in the manifests repo.

Examples

Sample notebooks

We frequently add examples to our GitHub repo.

Learn more

Join our working group for meeting invitations and discussion.
Read the docs.
API docs.
Roadmap.
KFServing 101 slides.

Prerequisites

Knative Serving (v0.8.0 +) and Istio (v1.1.7+) should be available on your Kubernetes cluster.

KFServing installation using kubectl

The following commands install KFServing 0.2.2, using a yaml file in GitHub repo. See here for other available releases. Alternatively, you can clone the GitHub repo and run kubectl on top of it.

TAG=0.2.2
CONFIG_URI=https://raw.githubusercontent.com/kubeflow/kfserving/master/install/$TAG/kfserving.yaml
kubectl apply -f ${CONFIG_URI}

Use

Install the SDK.
```
pip install kfserving
```
Follow the example to use the KFServing SDK to create, patch, roll out, and delete a KFServing instance.

Contribute

Developer guide.

Docs: Seldon Core Serving

Mon, 01 Jan 0001 00:00:00 +0000

This Kubeflow component has stable status. See the Kubeflow versioning policies.

Seldon Core comes installed with Kubeflow. The Seldon Core documentation site provides full documentation for running Seldon Core inference.

Seldon presently requires a Kubernetes cluster version >= 1.12 and <= 1.17.

If you have a saved model in a PersistentVolume (PV), Google Cloud Storage bucket or Amazon S3 Storage you can use one of the prepackaged model servers provided by Seldon Core.

Seldon Core also provides language specific model wrappers to wrap your inference code for it to run in Seldon Core.

Kubeflow specifics

You need to ensure the namespace where your models will be served has:

An Istio gateway named kubeflow-gateway
A label set as serving.kubeflow.org/inferenceservice=enabled

The following example applies the label my-namespace to the namespace for serving:

kubectl label namespace my-namespace serving.kubeflow.org/inferenceservice=enabled

Create a gateway called kubeflow-gateway in namespace my-namespace:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: kubeflow-gateway
  namespace: my-namespace
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP

Save the above resource and apply it with kubectl.

Simple example

Create a new namespace:

kubectl create ns testseldon

Label that namespace so you can run inference tasks in it:

kubectl label namespace testseldon serving.kubeflow.org/inferenceservice=enabled

Create an Istio gateway in that namespace named kubeflow-gateway:

cat <<EOF | kubectl create -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: kubeflow-gateway
  namespace: testseldon
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP
EOF

Create an example SeldonDeployment with a dummy model:

cat <<EOF | kubectl create -n testseldon -f -
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: seldon-model
spec:
  name: test-deployment
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: seldonio/mock_classifier_rest:1.3
          name: classifier
    graph:
      children: []
      endpoint:
        type: REST
      name: classifier
      type: MODEL
    name: example
    replicas: 1
EOF

Wait for state to become available:

kubectl get sdep seldon-model -n testseldon -o jsonpath='{.status.state}\n'

Port forward to the Istio gateway:

kubectl port-forward $(kubectl get pods -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}') -n istio-system 8004:80

Send a prediction request:

curl -s -d '{"data": {"ndarray":[[1.0, 2.0, 5.0]]}}'    -X POST http://localhost:8004/seldon/testseldon/seldon-model/api/v1.0/predictions    -H "Content-Type: application/json"

You should see a response:

{
  "meta": {
    "puid": "i2e1i8nq3lnttadd5i14gtu11j",
    "tags": {
    },
    "routing": {
    },
    "requestPath": {
      "classifier": "seldonio/mock_classifier_rest:1.3"
    },
    "metrics": []
  },
  "data": {
    "names": ["proba"],
    "ndarray": [[0.43782349911420193]]
  }
}

Further documentation

Docs: BentoML

Mon, 01 Jan 0001 00:00:00 +0000

This guide demonstrates how to serve a scikit-learn based iris classifier model with BentoML on a Kubernetes cluster. The same deployment steps are also applicable for models trained with other machine learning frameworks, see more BentoML examples here.

Prerequisites

Before starting this tutorial, make sure you have the following:

a Kubernetes cluster and kubectl installed on your local machine.
- kubectl install instruction: https://kubernetes.io/docs/tasks/tools/install-kubectl/
Docker and Docker Hub installed and configured in your local machine.
- Docker install instruction: https://docs.docker.com/get-docker/
Python 3.6 or above and required PyPi packages: bentoml, scikit-learn
- pip install bentoml scikit-learn

Build an iris classifier model server with BentoML

The following code defines a BentoML prediction service that requires a scikit-learn model, and asks BentoML to figure out the required PyPI packages automatically. It also defines an API, which is the entry point for accessing this prediction service. And the API is expecting a pandas.DataFrame object as its input data.

# iris_classifier.py
from bentoml import env, artifacts, api, BentoService
from bentoml.handlers import DataframeHandler
from bentoml.artifact import SklearnModelArtifact

@env(auto_pip_dependencies=True)
@artifacts([SklearnModelArtifact('model')])
class IrisClassifier(BentoService):

    @api(DataframeHandler)
    def predict(self, df):
        return self.artifacts.model.predict(df)

The following code trains a classifier model and serves it with the IrisClassifier defined above:

# main.py
from sklearn import svm
from sklearn import datasets

from iris_classifier import IrisClassifier

if __name__ == "__main__":
    # Load training data
    iris = datasets.load_iris()
    X, y = iris.data, iris.target

    # Model Training
    clf = svm.SVC(gamma='scale')
    clf.fit(X, y)

    # Create a iris classifier service instance
    iris_classifier_service = IrisClassifier()

    # Pack the newly trained model artifact
    iris_classifier_service.pack('model', clf)

    # Save the prediction service to disk for model serving
    saved_path = iris_classifier_service.save()

The sample code above can be found in the BentoML repository, run them directly with the following command:

git clone git@github.com:bentoml/BentoML.git
python ./bentoml/guides/quick-start/main.py

After saving the BentoService instance, you can now start a REST API server with the model trained and test the API server locally:

# Start BentoML API server:
bentoml serve IrisClassifier:latest

# Send test request
curl -i \
  --header "Content-Type: application/json" \
  --request POST \
  --data '[[5.1, 3.5, 1.4, 0.2]]' \
  localhost:5000/predict

BentoML provides a convenient way of containerizing the model API server with Docker. To create a docker container image for the sample model above:

Find the file directory of the SavedBundle with bentoml get command, which is directory structured as a docker build context.
Running docker build with this directory produces a docker image containing the model API server

saved_path=$(bentoml get IrisClassifier:latest -q | jq -r ".uri.uri")

# Replace `{docker_username} with your Docker Hub username
docker build -t {docker_username}/iris-classifier $saved_path
docker push {docker_username}/iris-classifier

Deploy model server to Kubernetes

The following is an example YAML file for specifying the resources required to run and expose a BentoML model server in a Kubernetes cluster. Replace {docker_username} with your Docker Hub username and save it to iris-classifier.yaml:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: iris-classifier
  name: iris-classifier
  namespace: kubeflow
spec:
  ports:
  - name: predict
    port: 5000
    targetPort: 5000
  selector:
    app: iris-classifier
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: iris-classifier
  name: iris-classifier
  namespace: kubeflow
spec:
  selector:
    matchLabels:
      app: iris-classifier
  template:
    metadata:
      labels:
        app: iris-classifier
    spec:
      containers:
      - image: {docker_username}/iris-classifier
        imagePullPolicy: IfNotPresent
        name: iris-classifier
        ports:
        - containerPort: 5000

Use kubectl CLI to deploy the model API server to the Kubernetes cluster

kubectl apply -f iris-classifier.yaml

Send prediction request

Use kubectl describe command to get the NODE_PORT

kubectl describe svc iris-classifier --namespace kubeflow

And then send the request:

curl -i \
  --header "Content-Type: application/json" \
  --request POST \
  --data '[[5.1, 3.5, 1.4, 0.2]]' \
  http://EXTERNAL_IP:NODE_PORT/predict

Monitor metrics with Prometheus

Prerequisites

Before starting this section, make sure you have the following:

Prometheus installed in the cluster
- Prometheus documentation
- Installation instruction with Helm chart

BentoML API server provides Prometheus support out of the box. It comes with a “/metrics” endpoint which includes the essential metrics for model serving and the ability to create and customize new metrics base on needs.

To enable Prometheus monitoring on the deployed model API server, update the YAML file with Prometheus related annotations. Change the deployment spec as the following, and replace {docker_username} with your Docker Hub username:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: iris-classifier
  name: iris-classifier
  namespace: kubeflow
spec:
  selector:
    matchLabels:
      app: iris-classifier
  template:
    metadata:
      labels:
        app: iris-classifier
      annotations:
        prometheus.io/scrape: true
        prometheus.io/port: 5000
    spec:
      containers:
      - image: {docker_username}/iris-classifier
        imagePullPolicy: IfNotPresent
        name: iris-classifier
        ports:
        - containerPort: 5000

Apply the change with kubectl CLI.

kubectl apply -f iris-classifier.yaml

Remove deployment

kubectl delete -f iris-classifier.yaml

Additional resources

Docs: NVIDIA Triton Inference Server

Mon, 01 Jan 0001 00:00:00 +0000

Kubeflow currently doesn’t have a specific guide for NVIDIA Triton Inference Server. Note that Triton was previously known as the TensorRT Inference Server. See the NVIDIA documentation for instructions on running NVIDIA inference server on Kubernetes.

Docs: TensorFlow Serving

Mon, 01 Jan 0001 00:00:00 +0000

This Kubeflow component has stable status. See the Kubeflow versioning policies.

Serving a model

To deploy a model we create following resources as illustrated below

A deployment to deploy the model using TFServing
A K8s service to create an endpoint a service
An Istio virtual service to route traffic to the model and expose it through the Istio gateway
An Istio DestinationRule is for doing traffic splitting.

apiVersion: v1
kind: Service
metadata:
  labels:
    app: mnist
  name: mnist-service
  namespace: kubeflow
spec:
  ports:
  - name: grpc-tf-serving
    port: 9000
    targetPort: 9000
  - name: http-tf-serving
    port: 8500
    targetPort: 8500
  selector:
    app: mnist
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: mnist
  name: mnist-v1
  namespace: kubeflow
spec:
  selector:
    matchLabels:
      app: mnist
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "true"
      labels:
        app: mnist
        version: v1
    spec:
      containers:
      - args:
        - --port=9000
        - --rest_api_port=8500
        - --model_name=mnist
        - --model_base_path=YOUR_MODEL
        command:
        - /usr/bin/tensorflow_model_server
        image: tensorflow/serving:1.11.1
        imagePullPolicy: IfNotPresent
        livenessProbe:
          initialDelaySeconds: 30
          periodSeconds: 30
          tcpSocket:
            port: 9000
        name: mnist
        ports:
        - containerPort: 9000
        - containerPort: 8500
        resources:
          limits:
            cpu: "4"
            memory: 4Gi
          requests:
            cpu: "1"
            memory: 1Gi
        volumeMounts:
        - mountPath: /var/config/
          name: config-volume
      volumes:
      - configMap:
          name: mnist-v1-config
        name: config-volume
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  labels:
  name: mnist-service
  namespace: kubeflow
spec:
  host: mnist-service
  subsets:
  - labels:
      version: v1
    name: v1
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  labels:
  name: mnist-service
  namespace: kubeflow
spec:
  gateways:
  - kubeflow-gateway
  hosts:
  - '*'
  http:
  - match:
    - method:
        exact: POST
      uri:
        prefix: /tfserving/models/mnist
    rewrite:
      uri: /v1/models/mnist:predict
    route:
    - destination:
        host: mnist-service
        port:
          number: 8500
        subset: v1
      weight: 100

Referring to the above example, you can customize your deployment by changing the following configurations in the YAML file:

In the deployment resource, the model_base_path argument points to the model. Change the value to your own model.
The example contains three configurations for Google Cloud Storage (GCS) access: volumes (secret user-gcp-sa), volumeMounts, and env (GOOGLE_APPLICATION_CREDENTIALS). If your model is not at GCS (e.g. using S3 from AWS), See the section below on how to setup access.
GPU. If you want to use GPU, add nvidia.com/gpu: 1 in container resources, and use a GPU image, for example: tensorflow/serving:1.11.1-gpu.
```
resources:
  limits:
    cpu: "4"
    memory: 4Gi
    nvidia.com/gpu: 1
```
The resource VirtualService and DestinationRule are for routing. With the example above, the model is accessible at HOSTNAME/tfserving/models/mnist (HOSTNAME is your Kubeflow deployment hostname). To change the path, edit the http.match.uri of VirtualService.

Pointing to the model

Depending where model file is located, set correct parameters

Google cloud

Change the deployment spec as follows:

spec:
  selector:
    matchLabels:
      app: mnist
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "true"
      labels:
        app: mnist
        version: v1
    spec:
      containers:
      - args:
        - --port=9000
        - --rest_api_port=8500
        - --model_name=mnist
        - --model_base_path=gs://kubeflow-examples-data/mnist
        command:
        - /usr/bin/tensorflow_model_server
        env:
        - name: GOOGLE_APPLICATION_CREDENTIALS
          value: /secret/gcp-credentials/user-gcp-sa.json
        image: tensorflow/serving:1.11.1-gpu
        imagePullPolicy: IfNotPresent
        livenessProbe:
          initialDelaySeconds: 30
          periodSeconds: 30
          tcpSocket:
            port: 9000
        name: mnist
        ports:
        - containerPort: 9000
        - containerPort: 8500
        resources:
          limits:
            cpu: "4"
            memory: 4Gi
            nvidia.com/gpu: 1
          requests:
            cpu: "1"
            memory: 1Gi
        volumeMounts:
        - mountPath: /var/config/
          name: config-volume
        - mountPath: /secret/gcp-credentials
          name: gcp-credentials
      volumes:
      - configMap:
          name: mnist-v1-config
        name: config-volume
      - name: gcp-credentials
        secret:
          secretName: user-gcp-sa

The changes are:

environment variable GOOGLE_APPLICATION_CREDENTIALS
volume gcp-credentials
volumeMount gcp-credentials

We need a service account that can access the model. If you are using Kubeflow’s click-to-deploy app, there should be already a secret, user-gcp-sa, in the cluster.

The model at gs://kubeflow-examples-data/mnist is publicly accessible. However, if your environment doesn’t have google cloud credential setup, TF serving will not be able to read the model. See this issue for example. To setup the google cloud credential, you should either have the environment variable GOOGLE_APPLICATION_CREDENTIALS pointing to the credential file, or run gcloud auth login. See doc for more detail.

To use S3, first you need to create secret that will contain access credentials. Use base64 to encode your credentials and check details in the Kubernetes guide to creating a secret manually

apiVersion: v1
metadata:
  name: secretname
data:
  AWS_ACCESS_KEY_ID: bmljZSB0cnk6KQ==
  AWS_SECRET_ACCESS_KEY: YnV0IHlvdSBkaWRuJ3QgZ2V0IG15IHNlY3JldCE=
kind: Secret

Then use the following manifest as an example:


apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: s3
  name: s3
  namespace: kubeflow
spec:
  selector:
    matchLabels:
      app: mnist
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: null
      labels:
        app: s3
        version: v1
    spec:
      containers:
      - args:
        - --port=9000
        - --rest_api_port=8500
        - --model_name=s3
        - --model_base_path=s3://abc
        - --monitoring_config_file=/var/config/monitoring_config.txt
        command:
        - /usr/bin/tensorflow_model_server
        env:
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              key: AWS_ACCESS_KEY_ID
              name: secretname
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              key: AWS_SECRET_ACCESS_KEY
              name: secretname
        - name: AWS_REGION
          value: us-west-1
        - name: S3_USE_HTTPS
          value: "true"
        - name: S3_VERIFY_SSL
          value: "true"
        - name: S3_ENDPOINT
          value: s3.us-west-1.amazonaws.com
        image: tensorflow/serving:1.11.1
        imagePullPolicy: IfNotPresent
        livenessProbe:
          initialDelaySeconds: 30
          periodSeconds: 30
          tcpSocket:
            port: 9000
        name: s3
        ports:
        - containerPort: 9000
        - containerPort: 8500
        resources:
          limits:
            cpu: "4"
            memory: 4Gi
          requests:
            cpu: "1"
            memory: 1Gi
        volumeMounts:
        - mountPath: /var/config/
          name: config-volume
      volumes:
      - configMap:
          name: s3-config
        name: config-volume

Sending prediction request directly

If the service type is LoadBalancer, it will have its own accessible external ip. Get the external ip by:

kubectl get svc mnist-service

And then send the request

curl -X POST -d @input.json http://EXTERNAL_IP:8500/v1/models/mnist:predict

Sending prediction request through ingress and IAP

If the service type is ClusterIP, you can access through ingress. It’s protected and only one with right credentials can access the endpoint. Below shows how to programmatically authenticate a service account to access IAP.

Save the client ID that you used to deploy Kubeflow as IAP_CLIENT_ID.

Create a service account

gcloud iam service-accounts create --project=$PROJECT $SERVICE_ACCOUNT

Grant the service account access to IAP enabled resources:

gcloud projects add-iam-policy-binding $PROJECT \
 --role roles/iap.httpsResourceAccessor \
 --member serviceAccount:$SERVICE_ACCOUNT

Download the service account key:

gcloud iam service-accounts keys create ${KEY_FILE} \
   --iam-account ${SERVICE_ACCOUNT}@${PROJECT}.iam.gserviceaccount.com

Export the environment variable GOOGLE_APPLICATION_CREDENTIALS to point to the key file of the service account.

Finally, you can send the request with an input file with this python script

python iap_request.py https://YOUR_HOST/tfserving/models/mnist IAP_CLIENT_ID --input=YOUR_INPUT_FILE

To send a GET request:

python iap_request.py https://YOUR_HOST/models/MODEL_NAME/ IAP_CLIENT_ID

Telemetry and Rolling out model using Istio

Please look at the Istio guide.

Logs and metrics with Stackdriver

See the guide to logging and monitoring for instructions on getting logs and metrics using Stackdriver.

Docs: TensorFlow Batch Prediction

Mon, 01 Jan 0001 00:00:00 +0000

Alpha

This Kubeflow component has alpha status with limited support. See the Kubeflow versioning policies. The Kubeflow team is interested in your feedback about the usability of the feature.

TensorFlow batch prediction is not supported in Kubeflow versions greater than v0.6. See the Kubeflow v0.6 documentation for earlier support for batch prediction with TensorFlow models.