Deploy MLflow models#

OCI Data Science supports two forms of runtime environment for running inference -

  • Conda Environment packaged using conda pack

  • Container image

MLflow CLI and SDK can be used for deploying models on OCI Data Science. The CLI and the SDK API accepts target parameter to specify the deployment target. To deploy on OCI, the specify oci-datascience as the target.

Prerequisites

  • pip install oci-mlflow

  • pip install oracle-ads[opctl]

  • A model is registered in MLflow Tracking server

  • The conda pack used for model deployment must have mlflow

CLI help#

Check CLI options for mlfow deploments supported with oci-datascience target by running -

mlflow deployments help -t oci-datascience

Create Inference Endpoint Using Conda Environments#

In conda based deployment, the dependencies required to run the model is packaged using conda pack. OCI Data Science provides curated conda environments to support wide variety of popular machine learning frameworks. To use conda runtime, you would choose from one of the following options: -

  1. The service provided conda pack

  2. Build your own conda environment and package it using conda pack and upload to object storage bucket . More details on how to manage your own conda packs can be found here

  3. MLflow CLI with help of oci-mlflow plugin, can build conda environment from conda.yaml available in the model artifact and push it to the object storage. The conda.yaml file is auto generated by mlflow when you log/register a model. Before using the autogenerated conday.yaml, verify that it has all the required dependencies. You could create the right conda.yaml while logging the model by providing the conda dependencies as dictionary input. More information is available in the API docs

Deployment Specification#

Create a model deployment specification in YAML file. Refer [schema] for YAML specification. Copy one of the templates below and customize it as per your requirement -

  1. Template to build conda pack on the fly using conda.yaml stored in the model artifact.

    Prerequisites

  • yaml
kind: deployment
spec:
  infrastructure:
    kind: infrastructure
    type: modelDeployment
    spec:
      logGroupId: ocid1.loggroup.oc1.iad..<unique_ID>
      logId: ocid1.log.oc1.iad..<unique_ID>
      projectId: ocid1.datascienceproject.oc1.iad..<unique_ID>
      compartmentId: ocid1.compartment.oc1..<unique_ID>
      shapeName: VM.Standard.E3.Flex
      shapeConfigDetails:
        memoryInGBs: 32
        ocpus: 4
      blockStorageSize: 50
  runtime:
    kind: runtime
    type: conda
    spec:
      uri:
        name: bc_sklearn_conda
        destination: oci://mayoor-dev@ociodscdev/mlflow-conda-envs/
        gpu: false
        overwrite: false
        keepLocal: true
        localCondaDir: ./conda
        #scoreCode: path/to/score.py [optional: This is required if you want to customize score.py]

  1. Template to deploy using previously published conda pack.

  • yaml
kind: deployment
spec:
  infrastructure:
    kind: infrastructure
    type: modelDeployment
    spec:
      logGroupId: ocid1.loggroup.oc1.iad..<unique_ID>
      logId: ocid1.log.oc1.iad..<unique_ID>
      projectId: ocid1.datascienceproject.oc1.iad..<unique_ID>
      compartmentId: ocid1.compartment.oc1..<unique_ID>
      shapeName: VM.Standard.E3.Flex
      shapeConfigDetails:
        memoryInGBs: 32
        ocpus: 4
      blockStorageSize: 50
  runtime:
    kind: runtime
    type: conda
    spec:
      uri: oci://bucket@namespace/path/to/conda-env
      pythonVersion: 3.9.15
      #scoreCode: path/to/score.py [optional: This is required if you want to customize score.py]

Create Deployment#

Use MLflow CLI/SDK to create a deployment. Once the deployment specification is created, pass it as input to the mlflow deployments command using --config deploy-config-file=<deployment specification yaml> and oci-datascience as the target.

export MLFLOW_TRACKING_URI=<tracking server url>

mlflow deployments create --name <model deployment name> -m models:/<registered model name>/<model version> -t oci-datascience --config deploy-config-file=deployment_specification.yaml

Invoke Inference Endpoint#

Invoke the endpoint through code or CLI. Here is an example of invoking an endpoint using oci raw-request CLI command -

data='{"columns":["mean radius","mean texture","mean perimeter","mean area","mean smoothness","mean compactness","mean concavity","mean concave points"],"index":[0],"data":[[17.99,10.38,122.8,1001.0,0.1184,0.2776,0.3001,0.1471]]}'

oci raw-request --http-method POST --target-uri https://modeldeployment.us-ashburn-1.oci.customer-oci.com/ocid1.datasciencemodeldeployment.oc1.iad..<unique_ID>/predict --request-body "$data"

{
  "data": {
    "predictions": [
      0
    ]
  },
  "headers": {
    "Connection": "keep-alive",
    "Content-Length": "20",
    "Content-Type": "application/json",
    "Date": "Wed, 15 Feb 2023 04:26:18 GMT",
    "Server": "nginx/1.14.1",
    "X-Content-Type-Options": "nosniff",
    "opc-request-id": "72BD2656826241C586FD29D9F03EA2E1/D95ADB6267CD5390F9E6D26108E60AF9/907E1377442682A9A72AB1D797056240"
  },
  "status": "200 OK"
}

Create Inference endpoint Using Container Images#

Container image allows you to not just bundle the runtime dependencies, but also allows you to use the inference serving framework of your choice. The container has to adhere to following requirements -

  1. Provides /predict endpoint for prediction

  2. Provides /health endpoint for health check

  3. Is published to ocir registry in your tenancy and the policies are setup such that OCI Data Science service can pull the image from your registry. More infomration here

In order to adhere to these requirements, you will have to add a reverse proxy on your container which will map the default endpoint offered by your model serving framework to /predict and health endpoint to /health

Refer how to produce a container image that uses mlflow models serve framework for model

Prerequisites

  • pip install oci-mlflow

  • pip install oracle-ads[opctl]

  • A model is registered in MLflow Tracking server

  • Container image is published to ocir

Deployment Specification#

Create a model deployment specification in YAML file. Refer [schema] for YAML specification. Copy the template below and customize it as per your requirement -

  • yaml
kind: deployment
spec:
  infrastructure:
    kind: infrastructure
    type: modelDeployment
    spec:
      logGroupId: ocid1.loggroup.oc1.iad..<unique_ID>
      logId: ocid1.log.oc1.iad..<unique_ID>
      projectId: ocid1.datascienceproject.oc1.iad..<unique_ID>
      compartmentId: ocid1.compartment.oc1..<unique_ID>
      shapeName: VM.Standard.E3.Flex
      shapeConfigDetails:
        memoryInGBs: 32
        ocpus: 4
      blockStorageSize: 50
      replica: 1
  runtime:
    kind: runtime
    type: container
    spec:
      image: {region}.ocir.io/<your tenancy>/<imagename>
      serverPort: 5001
      healthCheckPort: 5001

Create Deployment#

Use MLflow CLI/SDK to create a deployment. Once the deployment specification is created, pass it as input to the mlflow deployments command using --config deploy-config-file=<deployment specification yaml> and oci-datascience as the target.

  • shell
export MLFLOW_TRACKING_URI=<tracking server url>

mlflow deployments create --name <model deployment name> -m models:/<registered model name>/<model version> -t oci-datascience --config deploy-config-file=deployment_specification.yaml

Invoke Inference Endpoint#

Invoke the endpoint through code or CLI. Here is an example of invoking an endpoint using oci raw-request CLI command -

data='{"dataframe_split": {"columns":["mean radius","mean texture","mean perimeter","mean area","mean smoothness","mean compactness","mean concavity","mean concave points"],"index":[0],"data":[[17.99,10.38,122.8,1001.0,0.1184,0.2776,0.3001,0.1471]]}}'

oci raw-request --http-method POST --target-uri https://modeldeployment.us-ashburn-1.oci.customer-oci.com/ocid1.datasciencemodeldeployment.oc1.iad..<unique_ID>/predict --request-body "$data"

{
  "data": {
    "predictions": [
      0
    ]
  },
  "headers": {
    "Connection": "keep-alive",
    "Content-Length": "20",
    "Content-Type": "application/json",
    "Date": "Wed, 15 Feb 2023 04:26:18 GMT",
    "Server": "nginx/1.14.1",
    "X-Content-Type-Options": "nosniff",
    "opc-request-id": "72BD2656826241C586FD29D9F03EA2E1/D95ADB6267CD5390F9E6D26108E60AF9/907E1377442682A9A72AB1D797056240"
  },
  "status": "200 OK"
}

Update Model Deployment Details#

To update model deployment configuration use the YAML specification file that was created for deployment and then make changes to the attributes that you want to change. Use mlflow CLI/SDK and provide the OCID of the model deployment for name parameter and use --config deploy-config-file=<deployment specification file> option.

mlflow deployments update --name ocid1.datasciencemodeldeployment.oc1..<unique_ID> -t oci-datascience --config deploy-config-file=./deployment_update_config.yaml

Note: You may not be able to change all the configuration in one pass. Check Editing Model Deployments for more details.

Get Model Deployment Information#

Fetch Model deployment information for any OCI by providing OCID of the model deployment for name parameter.

mlflow deployments get -t oci-datascience --name ocid1.datasciencemodeldeployment.oc1..<unique_ID>