Kubeflow – Hyperparameter Tuning

Docs: Introduction to Katib

Mon, 01 Jan 0001 00:00:00 +0000

Beta

This Kubeflow component has beta status. See the Kubeflow versioning policies. The Kubeflow team is interested in your feedback about the usability of the feature.

Use Katib for automated tuning of your machine learning (ML) model’s hyperparameters and architecture.

This page introduces the concepts of hyperparameter tuning, neural architecture search, and the Katib system as a component of Kubeflow.

Hyperparameters and hyperparameter tuning

Hyperparameters are the variables that control the model training process. For example:

Learning rate.
Number of layers in a neural network.
Number of nodes in each layer.

Hyperparameter values are not learned. In other words, in contrast to the node weights and other training parameters, the model training process does not adjust the hyperparameter values.

Hyperparameter tuning is the process of optimizing the hyperparameter values to maximize the predictive accuracy of the model. If you don’t use Katib or a similar system for hyperparameter tuning, you need run many training jobs yourself, manually adjusting the hyperparameters to find the optimal values.

Automated hyperparameter tuning works by optimizing a target variable, also called the objective metric, that you specify in the configuration for the hyperparameter tuning job. A common metric is the model’s accuracy in the validation pass of the training job (validation-accuracy). You also specify whether you want the hyperparameter tuning job to maximize or minimize the metric.

For example, the following graph from Katib shows the level of accuracy for various combinations of hyperparameter values (learning rate, number of layers, and optimizer):

(To run the example that produced this graph, follow the getting-started guide.)

Katib runs several training jobs (known as trials) within each hyperparameter tuning job (experiment). Each trial tests a different set of hyperparameter configurations. At the end of the experiment, Katib outputs the optimized values for the hyperparameters.

Neural architecture search

Alpha version

Neural architecture search is currently in alpha with limited support. The Kubeflow team is interested in any feedback you may have, in particular with regards to usability of the feature. You can log issues and comments in the Katib issue tracker.

In addition to hyperparameter tuning, Katib offers a neural architecture search (NAS) feature. You can use the NAS to design your artificial neural network, with a goal of maximizing the predictive accuracy and performance of your model.

NAS is closely related to hyperparameter tuning. Both are subsets of automated machine learning (AutoML). While hyperparameter tuning optimizes the model’s hyperparameters, a NAS system optimizes the model’s structure, node weights, and hyperparameters.

NAS technology in general uses various techniques to find the optimal neural network design. The NAS in Katib uses the reinforcement learning technique.

You can submit Katib jobs from the command line or from the UI. (Read more about the Katib interfaces later on this page.) The following screenshot shows part of the form for submitting a NAS job from the Katib UI:

The Katib project

Katib is a Kubernetes-based system for hyperparameter tuning and neural architecture search. Katib supports a number of ML frameworks, including TensorFlow, MXNet, PyTorch, XGBoost, and others.

The Katib project is open source. The developer guide is a good starting point for developers who want to contribute to the project.

Katib interfaces

You can use the following interfaces to interact with Katib:

A web UI that you can use to submit experiments and to monitor your results. See the getting-started guide for information on how to access the UI. The Katib home page within Kubeflow looks like this:
A REST API. See the API reference on GitHub.
Command-line interfaces (CLIs):
- Kfctl is the Kubeflow CLI that you can use to install and configure Kubeflow. Read about kfctl in the guide to configuring Kubeflow.
- The Kubernetes CLI, kubectl, is useful for running commands against your Kubeflow cluster. Read about kubectl in the Kubernetes documentation.

Katib concepts

This section describes the terms used in Katib.

Experiment

An experiment is a single tuning run, also called an optimization run.

You specify configuration settings to define the experiment. The following are the main configurations:

Objective: What you want to optimize. This is the objective metric, also called the target variable. A common metric is the model’s accuracy in the validation pass of the training job (validation-accuracy). You also specify whether you want the hyperparameter tuning job to maximize or minimize the metric.
Search space: The set of all possible hyperparameter values that the hyperparameter tuning job should consider for optimization, and the constraints for each hyperparameter. Other names for search space include feasible set and solution space. For example, you may provide the names of the hyperparameters that you want to optimize. For each hyperparameter, you may provide a minimum and maximum value or a list of allowable values.
Search algorithm: The algorithm to use when searching for the optimal hyperparameter values.

For details of how to define your experiment, see the guide to running an experiment.

Suggestion

A suggestion is a set of hyperparameter values that the hyperparameter tuning process has proposed. Katib creates a trial to evaluate the suggested set of values.

Trial

A trial is one iteration of the hyperparameter tuning process. A trial corresponds to one worker job instance with a list of parameter assignments. The list of parameter assignments corresponds to a suggestion.

Each experiment runs several trials. The experiment runs the trials until it reaches either the objective or the configured maximum number of trials.

Worker job

The worker job is the process that runs to evaluate a trial and calculate its objective value.

The worker job can be one of the following types:

Kubernetes Job (does not support distributed execution).
Kubeflow TFJob (supports distributed execution).
Kubeflow PyTorchJob (supports distributed execution).

By offering the above worker job types, Katib supports multiple ML frameworks.

Next steps

Follow the getting-started guide to set up Katib and run some hyperparameter tuning examples.

Docs: Getting started with Katib

Mon, 01 Jan 0001 00:00:00 +0000

This page gets you started with Katib. Follow this guide to perform any additional setup you may need, depending on your environment, and to run a few examples using the command line and the Katib user interface (UI).

For an overview of the concepts around Katib and hyperparameter tuning, read the introduction to Katib.

Katib setup

This section describes some configurations that you may need to add to your Kubernetes cluster, depending on the way you’re using Kubeflow and Katib.

Installing Katib

You can skip this step if you have already installed Kubeflow. Your Kubeflow deployment includes Katib.

To install Katib as part of Kubeflow, follow the Kubeflow installation guide.

If you want to install Katib separately from Kubeflow, or to get a later version of Katib, run the following commands to install Katib directly from its repository on GitHub and deploy Katib to your cluster:

git clone https://github.com/kubeflow/katib
bash ./katib/scripts/v1alpha3/deploy.sh

Setting up persistent volumes

If you used above script to deploy Katib, you can skip this step. This script deploys PVC and PV on your cluster.

You can skip this step if you’re using Kubeflow on Google Kubernetes Engine (GKE) or if your Kubernetes cluster includes a StorageClass for dynamic volume provisioning. For more information, see the Kubernetes documentation on dynamic provisioning and persistent volumes.

If you’re using Katib outside GKE and your cluster doesn’t include a StorageClass for dynamic volume provisioning, you must create a persistent volume (PV) to bind to the persistent volume claim (PVC) required by Katib.

After deploying Katib to your cluster, run the following command to create the PV:

kubectl apply -f https://raw.githubusercontent.com/kubeflow/katib/master/manifests/v1alpha3/pv/pv.yaml

The above kubectl apply command uses a YAML file (pv.yaml) that defines the properties of the PV.

Accessing the Katib UI

You can use the Katib user interface (UI) to submit experiments and to monitor your results. The Katib home page within Kubeflow looks like this:

If you installed Katib as part of Kubeflow, you can access the Katib UI from the Kubeflow UI:

Open the Kubeflow UI. See the guide to accessing the central dashboard.
Click Katib in the left-hand menu.

Alternatively, you can set port-forwarding for the Katib UI service:

kubectl port-forward svc/katib-ui -n kubeflow 8080:80

Then you can access the Katib UI at this URL:

http://localhost:8080/katib/

Examples

This section introduces some examples that you can run to try Katib.

Example using random algorithm

You can create an experiment for Katib by defining the experiment in a YAML configuration file. The YAML file defines the configurations for the experiment, including the hyperparameter feasible space, optimization parameter, optimization goal, suggestion algorithm, and so on.

This example uses the YAML file for the random algorithm example.

The random algorithm example uses an MXNet neural network to train an image classification model using the MNIST dataset. You can check training container source code here. The experiment runs three training jobs with various hyperparameters and saves the results.

Run the following commands to launch an experiment using the random algorithm example:

Download the example:

curl https://raw.githubusercontent.com/kubeflow/katib/master/examples/v1alpha3/random-example.yaml --output random-example.yaml

Edit random-example.yaml and change the following line to use your Kubeflow user profile namespace:
```
Namespace: kubeflow
```
Deploy the example:
```
kubectl apply -f random-example.yaml
```

This example embeds the hyperparameters as arguments. You can embed hyperparameters in another way (for example, using environment variables) by using the template defined in the TrialTemplate.GoTemplate.RawTemplate section of the YAML file. The template uses the Go template format.

This example randomly generates the following hyperparameters:

--lr: Learning rate. Type: double.
--num-layers: Number of layers in the neural network. Type: integer.
--optimizer: Optimizer. Type: categorical.

Check the experiment status:

kubectl -n <your user profile namespace> describe experiment random-example

The output of the above command should look similar to this:

Name:         random-example
Namespace:    <your user namespace> 
Labels:       controller-tools.k8s.io=1.0
Annotations:  <none>
API Version:  kubeflow.org/v1alpha3
Kind:         Experiment
Metadata:
  Creation Timestamp:  2019-12-22T22:53:25Z
  Finalizers:
    update-prometheus-metrics
  Generation:        2
  Resource Version:  720692
  Self Link:         /apis/kubeflow.org/v1alpha3/namespaces/kubeflow/experiments/random-example
  UID:               dc6bc15a-250d-11ea-8cae-42010a80010f
Spec:
  Algorithm:
    Algorithm Name:        random
    Algorithm Settings:    <nil>
  Max Failed Trial Count:  3
  Max Trial Count:         12
  Metrics Collector Spec:
    Collector:
      Kind:  StdOut
  Objective:
    Additional Metric Names:
      accuracy
    Goal:                   0.99
    Objective Metric Name:  Validation-accuracy
    Type:                   maximize
  Parallel Trial Count:     3
  Parameters:
    Feasible Space:
      Max:           0.03
      Min:           0.01
    Name:            --lr
    Parameter Type:  double
    Feasible Space:
      Max:           5
      Min:           2
    Name:            --num-layers
    Parameter Type:  int
    Feasible Space:
      List:
        sgd
        adam
        ftrl
    Name:            --optimizer
    Parameter Type:  categorical
  Trial Template:
    Go Template:
      Raw Template:  apiVersion: batch/v1
kind: Job
metadata:
  name: {{.Trial}}
  namespace: {{.NameSpace}}
spec:
  template:
    spec:
      containers:
      - name: {{.Trial}}
        image: docker.io/kubeflowkatib/mxnet-mnist-example
        command:
        - "python"
        - "/mxnet/example/image-classification/train_mnist.py"
        - "--batch-size=64"
        {{- with .HyperParameters}}
        {{- range .}}
        - "{{.Name}}={{.Value}}"
        {{- end}}
        {{- end}}
      restartPolicy: Never
Status:
  Conditions:
    Last Transition Time:  2019-12-22T22:53:25Z
    Last Update Time:      2019-12-22T22:53:25Z
    Message:               Experiment is created
    Reason:                ExperimentCreated
    Status:                True
    Type:                  Created
    Last Transition Time:  2019-12-22T22:55:10Z
    Last Update Time:      2019-12-22T22:55:10Z
    Message:               Experiment is running
    Reason:                ExperimentRunning
    Status:                True
    Type:                  Running
  Current Optimal Trial:
    Observation:
      Metrics:
        Name:   Validation-accuracy
        Value:  0.981091
    Parameter Assignments:
      Name:          --lr
      Value:         0.025139701133432946
      Name:          --num-layers
      Value:         4
      Name:          --optimizer
      Value:         sgd
  Start Time:        2019-12-22T22:53:25Z
  Trials:            12
  Trials Running:    2
  Trials Succeeded:  10
Events:              <none>

When the last value in Status.Conditions.Type is Succeeded, the experiment is complete.

View the results of the experiment in the Katib UI:

Open the Katib UI as described above.
Click Hyperparameter Tuning on the Katib home page.
Open the Katib menu panel on the left, then open the HP section and click Monitor:
Click on the right-hand panel to close the menu panel. You should see the list of experiments:
Click the name of the experiment, random-example.
You should see a graph showing the level of accuracy for various combinations of the hyperparameter values (learning rate, number of layers, and optimizer):
Below the graph is a list of trials that ran within the experiment:

TensorFlow example

Run the following commands to launch an experiment using the Kubeflow’s TensorFlow training job operator, TFJob:

Download the tfjob-example.yaml file

curl https://raw.githubusercontent.com/kubeflow/katib/master/examples/v1alpha3/tfjob-example.yaml --output tfjob-example.yaml

Edit tfjob-example.yaml and change the following line to use your Kubeflow user profile namespace:
```
Namespace: kubeflow
```
Deploy the example:
```
kubectl apply -f tfjob-example.yaml
```

You can check the status of the experiment:

kubectl -n <your user profile namespace> describe experiment tfjob-example

Follow the steps as described for the random algorithm example above, to see the results of the experiment in the Katib UI.

PyTorch example

Run the following commands to launch an experiment using Kubeflow’s PyTorch training job operator, PyTorchJob:

Download the pytorchjob-example.yaml file

curl https://raw.githubusercontent.com/kubeflow/katib/master/examples/v1alpha3/pytorchjob-example.yaml --output pytorchjob-example.yaml

Edit pytorchjob-example.yaml and change the following line to use your Kubeflow user profile namespace:
```
Namespace: kubeflow
```

Deploy the example:

kubectl apply -f pytorchjob-example.yaml

You can check the status of the experiment:

kubectl -n <your user profile namespace> describe experiment pytorchjob-example

Follow the steps as described for the random algorithm example above, to see the results of the experiment in the Katib UI.

Cleanup

Delete the installed components:

bash ./scripts/v1alpha3/undeploy.sh

Next steps

For details of how to configure and run your experiment, see the guide to running an experiment.
For a detailed instruction of the Katib Configuration file, read the Katib config page.
See how you can change installation of Katib component in the environment variables guide.

Docs: Running an experiment

Mon, 01 Jan 0001 00:00:00 +0000

This page describes in detail how to configure and run a Katib experiment. The experiment can perform hyperparameter tuning or a neural architecture search (NAS) (alpha), depending on the configuration settings.

For an overview of the concepts involved, read the introduction to Katib.

Packaging your training code in a container image

Katib and Kubeflow are Kubernetes-based systems. To use Katib, you must package your training code in a Docker container image and make the image available in a registry. See the Docker documentation and the Kubernetes documentation.

Configuring the experiment

To create a hyperparameter tuning or NAS experiment in Katib, you define the experiment in a YAML configuration file. The YAML file defines the range of potential values (the search space) for the parameters that you want to optimize, the objective metric to use when determining optimal values, the search algorithm to use during optimization, and other configurations.

See the YAML file for the random algorithm example.

The list below describes the fields in the YAML file for an experiment. The Katib UI offers the corresponding fields. You can choose to configure and run the experiment from the UI or from the command line.

Configuration spec

These are the fields in the experiment configuration spec:

parameters: The range of the hyperparameters or other parameters that you want to tune for your ML model. The parameters define the search space, also known as the feasible set or the solution space. In this section of the spec, you define the name and the distribution (discrete or continuous) of every hyperparameter that you need to search. For example, you may provide a minimum and maximum value or a list of allowed values for each hyperparameter. Katib generates hyperparameter combinations in the range based on the hyperparameter tuning algorithm that you specify. See the ParameterSpec type.
objective: The metric that you want to optimize. The objective metric is also called the target variable. A common metric is the model’s accuracy in the validation pass of the training job (validation-accuracy). You also specify whether you want Katib to maximize or minimize the metric. Katib uses the objectiveMetricName and additionalMetricNames to monitor how the hyperparameters work with the model. Katib records the value of the best objectiveMetricName metric (maximized or minimized based on type) and the corresponding hyperparameter set in Experiment.status. If the objectiveMetricName metric for a set of hyperparameters reaches the goal, Katib stops trying more hyperparameter combinations. See the ObjectiveSpec type.
algorithm: The search algorithm that you want Katib to use to find the best hyperparameters or neural architecture configuration. Examples include random search, grid search, Bayesian optimization, and more. See the search algorithm details below.
trialTemplate: The template that defines the trial. You must package your ML training code into a Docker image, as described above. You must configure the model’s hyperparameters either as command-line arguments or as environment variables, so that Katib can automatically set the values in each trial.

You can use one of the following job types to train your model:
- Kubernetes Job (does not support distributed execution).
- Kubeflow TFJob (supports distributed execution).
- Kubeflow PyTorchJob (supports distributed execution).
See the TrialTemplate type. The template uses the Go template format.

You can define the job in raw string format or you can use a ConfigMap. Here is an example how to create ConfigMap with trial templates.
parallelTrialCount: The maximum number of hyperparameter sets that Katib should train in parallel.
maxTrialCount: The maximum number of trials to run. This is equivalent to the number of hyperparameter sets that Katib should generate to test the model.
maxFailedTrialCount: The maximum number of failed trials before Katib should stop the experiment. This is equivalent to the number of failed hyperparameter sets that Katib should test. If the number of failed trials exceeds maxFailedTrialCount, Katib stops the experiment with a status of Failed.
metricsCollectorSpec: A specification of how to collect the metrics from each trial, such as the accuracy and loss metrics. See the details of the metrics collector below.
nasConfig: The configuration for a neural architecture search (NAS). Note: NAS is currently in alpha with limited support. You can specify the configurations of the neural network design that you want to optimize, including the number of layers in the network, the types of operations, and more. See the NasConfig type.
operations: The range of operations that you want to tune for your ML model. For each neural network layer NAS algorithm selects one of the operation to build neural network. Each operation has sets of parameters which described above. See the Operation type.

As a NAS example, see the YAML file for the nasjob-example-RL-gpu. The example aims to show all the possible operations. Due to the large search space, the example is not likely to generate a good result.

Background information about Katib’s Experiment type: In Kubernetes terminology, Katib’s Experiment type is a custom resource (CR). The YAML file that you create for your experiment is the CR specification.

Search algorithms in detail

Katib currently supports several search algorithms. See the AlgorithmSpec type.

Here’s a list of the search algorithms available in Katib. The links lead to descriptions on this page:

More algorithms are under development. You can add an algorithm to Katib yourself. See the guide to adding a new algorithm and the developer guide.

Grid search

The algorithm name in Katib is grid.

Grid sampling is useful when all variables are discrete (as opposed to continuous) and the number of possibilities is low. A grid search performs an exhaustive combinatorial search over all possibilities, making the search process extremely long even for medium sized problems.

Katib uses the Chocolate optimization framework for its grid search.

Random search

The algorithm name in Katib is random.

Random sampling is an alternative to grid search, useful when the number of discrete variables to optimize is large and the time required for each evaluation is long. When all parameters are discrete, random search performs sampling without replacement. Random search is therefore the best algorithm to use when combinatorial exploration is not possible. If the number of continuous variables is high, you should use quasi random sampling instead.

Katib uses the hyperopt optimization framework for its random search.

Katib supports the following algorithm settings:

Setting name	Description	Example
random_state	[int]: Set `random_state` to something other than None for reproducible results.	10

Bayesian optimization

The algorithm name in Katib is bayesianoptimization.

The Bayesian optimization method uses gaussian process regression to model the search space. This technique calculates an estimate of the loss function and the uncertainty of that estimate at every point in the search space. The method is suitable when the number of dimensions in the search space is low. Since the method models both the expected loss and the uncertainty, the search algorithm converges in a few steps, making it a good choice when the time to complete the evaluation of a parameter configuration is long.

Katib uses the Scikit-Optimize library for its Bayesian search. Scikit-Optimize is also known as skopt.

Katib supports the following algorithm settings:

Setting Name	Description	Example
base_estimator	[“GP”, “RF”, “ET”, “GBRT” or sklearn regressor, default=“GP”]: Should inherit from `sklearn.base.RegressorMixin`. The `predict` method should have an optional `return_std` argument, which returns `std(Y \| x)` along with `E[Y \| x]`. If `base_estimator` is one of [“GP”, “RF”, “ET”, “GBRT”], the system uses a default surrogate model of the corresponding type. See more information in the skopt documentation.	GP
n_initial_points	[int, default=10]: Number of evaluations of `func` with initialization points before approximating it with `base_estimator`. Points provided as `x0` count as initialization points. If `len(x0) < n_initial_points`, the system samples additional points at random. See more information in the skopt documentation.	10
acq_func	[string, default=`"gp_hedge"`]: The function to minimize over the posterior distribution. See more information in the skopt documentation.	gp_hedge
acq_optimizer	[string, “sampling” or “lbfgs”, default=“auto”]: The method to minimize the acquisition function. The system updates the fit model with the optimal value obtained by optimizing `acq_func` with `acq_optimizer`. See more information in the skopt documentation.	auto
random_state	[int]: Set `random_state` to something other than None for reproducible results.	10

HYPERBAND

The algorithm name in Katib is hyperband.

Katib supports the HYPERBAND optimization framework. Instead of using Bayesian optimization to select configurations, HYPERBAND focuses on early stopping as a strategy for optimizing resource allocation and thus for maximizing the number of configurations that it can evaluate. HYPERBAND also focuses on the speed of the search.

Hyperopt TPE

The algorithm name in Katib is tpe.

Katib uses the Tree of Parzen Estimators (TPE) algorithm in hyperopt. This method provides a forward and reverse gradient-based search.

NAS using reinforcement learning

Alpha version

The algorithm name in Katib is nasrl.

For more information, see:

Information in the Katib repository on NAS with reinforcement learning.
The description of the nasConfig field in the configuration file earlier on this page.

Metrics collector

In the metricsCollectorSpec section of the YAML configuration file, you can define how Katib should collect the metrics from each trial, such as the accuracy and loss metrics. See the MetricsCollectorSpec type

Your training code can record the metrics into stdout or into arbitrary output files. Katib collects the metrics using a sidecar container. A sidecar is a utility container that supports the main container in the Kubernetes Pod.

To define the metrics collector for your experiment:

Specify the collector type in the collector field. Katib’s metrics collector supports the following collector types:
- StdOut: Katib collects the metrics from the operating system’s default output location (standard output).
- File: Katib collects the metrics from an arbitrary file, which you specify in the source field.
- TensorFlowEvent: Katib collects the metrics from a directory path containing a tf.Event. You should specify the path in the source field.
- Custom: Specify this value if you need to use custom way to collect metrics. You must define your custom metrics collector container in the collector.customCollector field.
- None: Specify this value if you don’t need to use Katib’s metrics collector. For example, your training code may handle the persistent storage of its own metrics.
Specify the metrics output location in the source field. See const for default values.
Write code in your training container to print metrics in the format specified in the metricsCollectorSpec.source.filter.metricsFormat field. The default format is ([\w|-]+)\s*=\s*((-?\d+)(\.\d+)?). Each element is a regular expression with two subexpressions. The first matched expression is taken as the metric name. The second matched expression is taken as the metric value.

For example, using the default metrics format, if the name of your objective metric is loss and the metrics are recall and precision, your training code should print the following output:
```
epoch 1:
loss=0.3
recall=0.5
precision=0.4

epoch 2:
loss=0.2
recall=0.55
precision=0.5
```

Running the experiment

You can run a Katib experiment from the command line or from the Katib UI.

Running the experiment from the command line

You can use kubectl to launch an experiment from the command line:

kubectl apply -f <your-path/your-experiment-config.yaml>

For example, run the following command to launch an experiment using the random algorithm example:

kubectl apply -f https://raw.githubusercontent.com/kubeflow/katib/master/examples/v1alpha3/random-example.yaml

Check the experiment status:

kubectl -n kubeflow describe experiment <your-experiment-name>

For example, to check the status of the random algorithm example:

kubectl -n kubeflow describe experiment random-example

Running the experiment from the Katib UI

Instead of using the command line, you can submit an experiment from the Katib UI. The following steps assume you want to run a hyperparameter tuning experiment. If you want to run a neural architecture search, access the NAS section of the UI (instead of the HP section) and then follow a similar sequence of steps.

To run a hyperparameter tuning experiment from the Katib UI:

Follow the getting-started guide to access the Katib UI.
Click Hyperparameter Tuning on the Katib home page.
Open the Katib menu panel on the left, then open the HP section and click Submit:
Click on the right-hand panel to close the menu panel. You should see tabs offering you the following options:

YAML file: Choose this option to supply an entire YAML file containing the configuration for the experiment.

<img src="/docs/images/katib-deploy-yaml.png" 
  alt="UI tab to paste a YAML configuration file"
  class="mt-3 mb-3 border border-info rounded">

Parameters: Choose this option to enter the configuration values into a form.

<img src="/docs/images/katib-deploy-form.png" 
  alt="UI form to deploy a Katib experiment"
  class="mt-3 mb-3 border border-info rounded">

View the results of the experiment in the Katib UI:

Open the Katib menu panel on the left, then open the HP section and click Monitor:
Click on the right-hand panel to close the menu panel. You should see the list of experiments:
Click the name of your experiment. For example, click random-example.
You should see a graph showing the level of accuracy for various combinations of the hyperparameter values. For example, the graph below shows learning rate, number of layers, and optimizer:
Below the graph is a list of trials that ran within the experiment. Click a trial name to see the trial data.

Next steps

See how to run the random algorithm and other Katib examples in the getting-started guide.
For an overview of the concepts involved in hyperparameter tuning and neural architecture search, read the introduction to Katib.
For a detailed instruction of the Katib Configuration file, read the Katib config page.
See how you can change installation of Katib component in the environment variables guide.

Docs: Katib Configuration Overview

Mon, 01 Jan 0001 00:00:00 +0000

This page describes information about Katib config.

Katib config is the Kubernetes Config Map that contains information about:

Current Metrics Collectors (key = metrics-collector-sidecar)
Current Algorithms (Suggestions) (key = suggestion).

Katib Config Map must be deployed in KATIB_CORE_NAMESPACE namespace with katib-config name. Katib controller parses Katib config when you submit Experiment.

You can edit this Config Map even after deploying Katib.

If you deploy Katib in Kubeflow namespace, to edit Katib config run this:

kubectl edit configMap katib-config -n kubeflow

Metrics Collector Sidecar settings

These settings are related to Katib Metrics Collectors, where:

key = metrics-collector-sidecar
value = corresponding settings JSON for each Metrics Collector.

Example for the File Metrics Collector with all settings:

metrics-collector-sidecar: |-
{
  "File": {
    "image": "gcr.io/kubeflow-images-public/katib/v1alpha3/file-metrics-collector",
    "imagePullPolicy": "Always",
    "resources": {
      "requests": {
        "memory": "200Mi",
        "cpu": "250m",
        "ephemeral-storage": "200Mi"
      },
      "limits": {
        "memory": "1Gi",
        "cpu": "500m",
        "ephemeral-storage": "2Gi"
      }
    }
  },
  ...
}

All of these settings except image can be omitted. If you don’t specify any other settings, default value is be set.

image - Docker image name for the File Metrics Collector.

Must be specified.
imagePullPolicy - File Metrics Collector container image pull policy.

Default value is IfNotPresent.
resources - File Metrics Collector container resources. In the above example you can see how to specify limits and requests. Currently, you can specify only memory, cpu and ephemeral-storage resource.

Default values for the requests are:
- memory = 10Mi.
- cpu = 50m.
- ephemeral-storage = 500Mi.
Default values for the limits are:
- memory = 100Mi.
- cpu = 500m.
- ephemeral-storage = 5Gi.

Suggestion settings

These settings are related to Katib suggestions, where:

key = suggestion
value = corresponding settings JSON for each algorithm.

If you want to use new algorithm, you must update Katib config with the new suggestion.

Example for the random suggestion with all settings:

suggestion: |-
{
  "random": {
    "image": "gcr.io/kubeflow-images-public/katib/v1alpha3/suggestion-hyperopt",
    "imagePullPolicy": "Always",
    "resources": {
      "requests": {
        "memory": "100Mi",
        "cpu": "100m",
        "ephemeral-storage": "100Mi"
      },
      "limits": {
        "memory": "500Mi",
        "cpu": "500m",
        "ephemeral-storage": "3Gi"
      }
    },
    "serviceAccountName": "suggestion-serviceaccount"
  },
  ...
}

All of these settings except image can be omitted. If you don’t specify any other settings, default value is be set.

image - Docker image name for the random suggestion.

Must be specified.
imagePullPolicy - Random suggestion container image pull policy.

Default value is IfNotPresent.
resources - Random suggestion container resources. In the above example you can see how to specify limits and requests. Currently, you can specify only memory, cpu and ephemeral-storage resource.

Default values for the requests are:
- memory = 10Mi.
- cpu = 50m.
- ephemeral-storage = 500Mi.
Default values for the limits are:
- memory = 100Mi.
- cpu = 500m.
- ephemeral-storage = 5Gi.
serviceAccountName - Random suggestion container service account. In the above example, suggestion-serviceaccount service account is used for each Experiment with random algorithm, until you change or delete this service account from the Katib config.

By default suggestion pod doesn’t have specific service account. In that case, Suggestion pod uses default service account.

Docs: Environment Variables for Katib Components

Mon, 01 Jan 0001 00:00:00 +0000

This page describes information about environment variables for each Katib component. If you want to change Katib installation, you can modify some of these variables.

In the tables below you can see description, default values and mandatory property for all environment variables in each Katib component. If variable has mandatory property, appropriate Katib component’s manifest must have this environment variable.

Katib Controller

This is the table of environment variables for the Katib Controller deployment.

Variable	Description	Default Value	Mandatory
`KATIB_CORE_NAMESPACE`	Base Namespace for all Katib components and default Experiment	`metadata.namespace`	Yes
`KATIB_SUGGESTION_COMPOSER`	Composer for the Katib Suggestions. You can use your own Composer	general	No
`KATIB_DB_MANAGER_SERVICE_NAMESPACE`	Katib DB Manager Namespace	`KATIB_CORE_NAMESPACE` env variable	No
`KATIB_DB_MANAGER_SERVICE_IP`	Katib DB Manager IP	katib-db-manager	No
`KATIB_DB_MANAGER_SERVICE_PORT`	Katib DB Manager Port	6789	No

Katib Controller calls Katib DB Manager with this address expression:

KATIB_DB_MANAGER_SERVICE_IP.KATIB_DB_MANAGER_SERVICE_NAMESPACE:KATIB_DB_MANAGER_SERVICE_PORT

If you set KATIB_DB_MANAGER_SERVICE_NAMESPACE="", Katib Controller calls Katib DB Manager with this address:

KATIB_DB_MANAGER_SERVICE_IP:KATIB_DB_MANAGER_SERVICE_PORT

If you want to use your own DB Manager to report Katib metrics, you can change KATIB_DB_MANAGER_SERVICE_NAMESPACE, KATIB_DB_MANAGER_SERVICE_IP and KATIB_DB_MANAGER_SERVICE_PORT variables.

Katib UI

This is the table of environment variables for the Katib UI deployment.

Variable	Description	Default Value	Mandatory
`KATIB_CORE_NAMESPACE`	Base Namespace for all Katib components and default Experiment	`metadata.namespace`	Yes
`KATIB_DB_MANAGER_SERVICE_NAMESPACE`	Katib DB Manager Namespace	`KATIB_CORE_NAMESPACE` env variable	No
`KATIB_DB_MANAGER_SERVICE_IP`	Katib DB Manager IP	katib-db-manager	No
`KATIB_DB_MANAGER_SERVICE_PORT`	Katib DB Manager Port	6789	No

Katib UI calls Katib DB Manager with the same address expression as Katib Controller.

Katib DB Manager

This is the table of environment variables for the Katib DB Manager deployment.

Variable	Description	Default Value	Mandatory
`DB_NAME`	Katib DB Name	mysql	Yes
`DB_PASSWORD`	Katib DB Password	test	Yes
`DB_USER`	Katib DB User	root	No
`KATIB_MYSQL_DB_HOST`	Katib MySQL Host	katib-mysql	No
`KATIB_MYSQL_DB_PORT`	Katib MySQL Port	3306	No
`KATIB_MYSQL_DB_DATABASE`	Katib MySQL Database name	katib	No

Currently, Katib DB Manager supports only MySQL database. You can use your own DB Manager and Database to report metrics.

For the Katib DB Manager you can change DB_PASSWORD to your own MySQL DB password.

Katib DB Manager creates connection to the DB, using mysql driver and this data source name:

DB_USER:DB_PASSWORD@tcp(KATIB_MYSQL_DB_HOST:KATIB_MYSQL_DB_PORT)/KATIB_MYSQL_DB_DATABASE?timeout=5s

Katib MySQL DB

For the Katib MySQL we set these environment variables:

MYSQL_ROOT_PASSWORD to a value from katib-mysql-secrets, which is equal to “test”.
MYSQL_ALLOW_EMPTY_PASSWORD as true
MYSQL_DATABASE as katib.

Check here about all environment variables for the MySQL Docker image.

Katib MySQL environment variables must be matched with the Katib DB Manager environment variables, it means:

MYSQL_ROOT_PASSWORD = DB_PASSWORD
MYSQL_DATABASE = KATIB_MYSQL_DB_DATABASE