Dockerize and save money using your own ML algorithm

Hi,

In this blog post we’re going to learn how to dockerize your own ML algorithms into a Docker image which will be then published into Amazon ECR.

We based our approach on the advanced examples on the AWS Labs Github repository (https://github.com/awslabs/amazon-sagemaker-examples/tree/master/advanced_functionality) and the main reason was to save costs.

As you might know, AWS Sagemaker starts costing you money from the second you open up a Jupyter notebook instance.

Not only that, but also you’ll end up receiving charges for the time spent while you were training an ML model and also for the time and endpoint is alive to predict and make inferences on features the end-user gives.

Intro

Here’s the US East (N. Virginia) region pricing details from the less to the most expensive on each of the Sagemaker subcategories we’re gonna receive charges.

Building – On-Demand ML Notebook Instances

Lowest Standard Instances

$0⁰⁴⁶⁴per hour

ml.t2.medium

Highest Standard Instances

$6⁴⁵¹per hour

ml.m5.24xlarge

Lowest Compute Optimized

$0²⁷⁹per hour

ml.c4.xlarge

Highest Compute Optimized

$4⁸³⁸per hour

ml.c5d.18xlarge

Lowest GPU Instance

$1²⁶per hour

ml.p2.xlarge

Highest GPU Instance

$34²⁷²per hour

ml.p3.16xlarge

Model Training On-Demand ML Training Instances

Lowest Standard Instances

$0¹³⁴per hour

ml.m5.large

Highest Standard Instances

$4⁴⁸per hour

ml.m4.16xlarge

Lowest Compute Optimized

$0²³⁸per hour

ml.c5.xlarge

Highest Compute Optimized

$2²²⁷per hour

ml.c4.8xlarge

Lowest GPU Instance

$1²⁶per hour

ml.p2.xlarge

Highest GPU Instance

$34²⁷²per hour

ml.p3.16xlarge

Model Deployment – On-Demand ML Hosting Instances for Real-Time Inference

Lowest Standard Instances

$0⁰⁶⁵per hour

ml.t2.medium

Highest Standard Instances

$4⁴⁸per hour

ml.t2.medium

Lowest Compute Optimized

$0¹¹⁹per hour

ml.c5.large

Highest Compute Optimized

$2²²⁷per hour

ml.c4.8xlarge

Lowest GPU Instance

$1²⁶per hour

ml.p2.xlarge

Highest GPU Instance

$34²⁷²per hour

ml.p3.16xlarge

Cost Calculation

So, let’s say we spend an hour training (this will involve having the Jupyter notebook instance too for the duration of an hour) and then leaving the endpoint alive for real time inference 24-hours a day.

Avg cost = 1 x $0.0464 + 1 x $0.134 + 24 x $0.065 = $1.7404, which rounds up to more than $50 dollars a month.

And this is just using the minimums!

That’s the reason why we switched to use Docker and train and deploy our ML models locally.

After you are satisfied with the results and ready to use your model on production you just have to publish the Docker image to AWS ECR.

In our case, we wanted to use a premade Tensorflow DNNClassifier estimator.

Required Steps

Write a script called train which will contain the estimator, but it could have any other training spec that you want to use, not necessarily using Tensorflow, but MXNet or another ML framework.

Create this folder structure:

Define a script file that is responsible for packing the scripts. Also for building, tagging and pushing the image on ECR.

build_and_push.sh

#!/usr/bin/env bash

# This script shows how to build the Docker image and push it to ECR to be ready for use
# by SageMaker.

# Image name. This will be used as the image on the local
# machine and combined with the account and region to form the repository name for ECR.

if [ "$image" == "" ]
then
    echo "Usage: $0 <image-name>"
    exit 1
fi

chmod +x invoiceit/train
chmod +x invoiceit/serve

# Get the account number associated with the current IAM credentials
account=$(aws sts get-caller-identity --query Account --output text)

if [ $? -ne 0 ]
then
    exit 255
fi

# Get the region defined in the current configuration (default to us-west-1 if none defined)
region=$(aws configure get region)
region=${region:-us-east-1}

fullname="${account}.dkr.ecr.${region}.amazonaws.com/${image}:latest"

# If the repository doesn't exist in ECR, create it.

aws ecr describe-repositories --repository-names "${image}" > /dev/null 2>&1

if [ $? -ne 0 ]
then
    aws ecr create-repository --repository-name "${image}" > /dev/null
fi

# Get the login command from ECR and execute it directly
$(aws ecr get-login --region ${region} --no-include-email)

# Build the docker image locally with the image name and then push it to ECR
# with the full name.

docker build  -t ${image} .
docker tag ${image} ${fullname}
docker push ${fullname}docker push ${fullname}

We’ll start by defining a Dockerfile to get the required contents available on our image

Firstly, we are using as a base image “tensorflow/tensorflow:1.8.0-py3”, secondly installing nginx and curl and in addition the package tensorflow model serving that handle and run the predictions against our model.

After that, we mount the current path where our code resides from the local directory to the virtual dir.

Define also 2 files:

Serve

Copy to Clipboard

# This file implements the hosting solution, which just starts TensorFlow Model Serving.
import subprocess

def start_server():
    print('Starting TensorFlow Serving.')

# link the log streams to stdout/err so they will be logged to the container logs
    subprocess.check_call(['ln', '-sf', '/dev/stdout', '/var/log/nginx/access.log'])
    subprocess.check_call(['ln', '-sf', '/dev/stderr', '/var/log/nginx/error.log'])

# start nginx server
    nginx = subprocess.Popen(['nginx', '-c', '/opt/ml/code/nginx.conf'])

# start TensorFlow Serving
    # https://www.tensorflow.org/serving/api_rest#start_modelserver_with_the_rest_api_endpoint
    # SageMaker copies our model artifact from our Training Job into the /opt/ml/model.
    # https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-inference-code.html#your-algorithms-inference-code-load-artifacts
    tf_model_server = subprocess.call(['tensorflow_model_server',
                                       '--rest_api_port=8501',
                                       '--model_name=invoiceit_model',
                                       '--model_base_path=/opt/ml/model/export/Servo'])

# The main routine just invokes the start function.
if __name__ == '__main__':
    start_server()

Train

from __future__ import print_function

import os
import json
import sys
import subprocess
import traceback

# These are the paths to where SageMaker mounts interesting things in your container.
prefix = '/opt/ml/'
input_path = os.path.join(prefix,'input/data')
output_path = os.path.join(prefix, 'output')
model_path = os.path.join(prefix, 'model')
param_path = os.path.join(prefix, 'input/config/hyperparameters.json')

# This algorithm has a single channel of input data called 'training'. Since we run in
# File mode, the input files are copied to the directory specified here.
channel_name = 'training'
training_path = os.path.join(input_path, channel_name)

# default params
training_script = 'invoiceit.py'
default_params = ['--model-dir', str(model_path)]

# Execute your training algorithm.
def _run(cmd):
    """Invokes your training algorithm."""
    process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=os.environ)
    stdout, stderr = process.communicate()

return_code = process.poll()
    if return_code:
        error_msg = 'Return Code: {}, CMD: {}, Err: {}'.format(return_code, cmd, stderr)
        raise Exception(error_msg)

def _hyperparameters_to_cmd_args(hyperparameters):
    """
    Converts our hyperparameters, in json format, into key-value pair suitable for passing to our training
    algorithm.
    """
    cmd_args_list = []

for key, value in hyperparameters.items():
        cmd_args_list.append('--{}'.format(key))
        cmd_args_list.append(value)

return cmd_args_list

if __name__ == '__main__':
    try:
        # Amazon SageMaker makes our specified hyperparameters available within the
        # /opt/ml/input/config/hyperparameters.json.
        # https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-training-algo.html#your-algorithms-training-algo-running-container
        #with open(param_path, 'r') as tc:
        #    training_params = json.load(tc)

python_executable = sys.executable
        #cmd_args = _hyperparameters_to_cmd_args(training_params)

train_cmd = [python_executable, training_script] + default_params

_run(train_cmd)
        print('Training complete.')

# A zero exit code causes the job to be marked a Succeeded.
        sys.exit(0)
    except Exception as e:
        # Write out an error file. This will be returned as the failureReason in the
        # DescribeTrainingJob result.
        trc = traceback.format_exc()
        with open(os.path.join(output_path, 'failure'), 'w') as s:
            s.write('Exception during training: ' + str(e) + '\n' + trc)
        # Printing this causes the exception to be in the training job logs, as well.
        print('Exception during training: ' + str(e) + '\n' + trc, file=sys.stderr)
        # A non-zero exit code causes the training job to be marked as Failed.
        sys.exit(255)

Pack your custom tensorflow (or any other ML framework code) script into a .py file with the custom training code

Eventually, the file will be called by the train script when launching the training job.

import argparse
import functools
import numpy as np
import os

import tensorflow as tf

from tensorflow.python.tools import inspect_checkpoint as chkp

INPUT_TENSOR_NAME = 'inputs'

# Disable MKL to get a better perfomance for this model.
os.environ['TF_DISABLE_MKL'] = '1'
os.environ['TF_DISABLE_POOL_ALLOCATOR'] = '1'

def serving_input_fn():
   receiver_tensors = {
       'feature_1': tf.placeholder(tf.int64, [None, 1]),
       'feature_2': tf.placeholder(tf.int64, [None, 1]),
       'feature_3': tf.placeholder(tf.int64, [None, 1]),
       'feature_4': tf.placeholder(tf.int64, [None, 1]),
       'feature_5': tf.placeholder(tf.int64, [None, 1]),
       'feature_6': tf.placeholder(tf.int64, [None, 1]),
       'feature_7': tf.placeholder(tf.int64, [None, 1]),
       'feature_8': tf.placeholder(tf.int64, [None, 1])
   }

# Convert give inputs to adjust to the model.
   features = {
       INPUT_TENSOR_NAME: tf.concat([
           receiver_tensors['feature_1'],
           receiver_tensors['feature_2'],
           receiver_tensors['feature_3'],
           receiver_tensors['feature_4'],
           receiver_tensors['feature_5'],
           receiver_tensors['feature_6'],
           receiver_tensors['feature_7'],
           receiver_tensors['feature_8']
       ], axis=1)
   }
   return tf.estimator.export.ServingInputReceiver(receiver_tensors=receiver_tensors,
                                                   features=features)
  
def train_input_fn(data_dir):
   """Returns input function that would feed the model during training"""
   return _generate_input_fn(data_dir, 'train_data_final.csv')
  
def eval_input_fn(data_dir, params):
   """Returns input function that would feed the model during evaluation"""
   return _generate_input_fn(data_dir, 'test_data_final.csv')

def _generate_input_fn(data_dir, training_filename):
   print(os.path.join(data_dir, training_filename))
   training_set = tf.contrib.learn.datasets.base.load_csv_with_header(
       filename=os.path.join(data_dir, training_filename),
       target_dtype=np.int,
       features_dtype=np.int)
      
   return tf.estimator.inputs.numpy_input_fn(
       x={INPUT_TENSOR_NAME: np.array(training_set.data)},
       y=np.array(training_set.target),
       num_epochs=None,
       shuffle=True)()

def train(model_dir, data_dir, train_steps):
   feature_columns = [tf.feature_column.numeric_column(INPUT_TENSOR_NAME, shape=[8])]
   estimator = tf.estimator.DNNClassifier(model_dir=model_dir, feature_columns=feature_columns, hidden_units=[10, 20, 10], n_classes=3)
  
   temp_input_fn = functools.partial(train_input_fn, data_dir)

train_spec = tf.estimator.TrainSpec(temp_input_fn, max_steps=train_steps)

exporter = tf.estimator.LatestExporter('Servo', serving_input_receiver_fn=serving_input_fn)
   temp_eval_fn = functools.partial(eval_input_fn, data_dir)
   eval_spec = tf.estimator.EvalSpec(temp_eval_fn, steps=100, exporters=exporter)

tf.estimator.train_and_evaluate(estimator=estimator, train_spec=train_spec, eval_spec=eval_spec)
  
def main(model_dir, data_dir, train_steps):
   train(model_dir, data_dir, train_steps)

if __name__ == '__main__':
   args_parser = argparse.ArgumentParser()
   # For more information:
   # https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-training-algo.html
   args_parser.add_argument(
       '--data-dir',
       default='/opt/ml/input/data/training',
       type=str,
       help='The directory where the INVOICEIT input data is stored. Default: /opt/ml/input/data/training. This '
            'directory corresponds to the SageMaker channel named \'training\', which was specified when creating '
            'our training job on SageMaker')

# For more information:
   # https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-inference-code.html
   args_parser.add_argument(
       '--model-dir',
       default='/opt/ml/model',
       type=str,
       help='The directory where the model will be stored. Default: /opt/ml/model. This directory should contain all '
            'final model artifacts as Amazon SageMaker copies all data within this directory as a single object in '
            'compressed tar format.')

args_parser.add_argument(
       '--train-steps',
       type=int,
       default=100,
       help='The number of steps to use for training.')
   args = args_parser.parse_args()
   main(**vars(args))

These files will be invoked when using docker-compose automatically when calling the 2 main high-level methods when using the python sagemaker api.

After that, launch a Jupyter Notebook instance on our local environment (https://anaconda.org/anaconda/anaconda-navigator).

Remember that is convenient that you previously had installed Anaconda Navigator and created an environment.

Here is the code we have on the notebook.

The basics are:

Define a generic Estimator
Point to an existing role that has access to execute tasks on Sagemaker
Use the URI of the image previously published on ECR using the build_and_push.sh script.

When we make the call to the .fit() method a Docker container with the train arguments will be orchestrated. As a result, the train script will be run.

In addition, a script with our custom tensorflow python file will be called within a subprocess.

As instructed previously, the training job is created on Sagemaker and the compressed model file is uploaded to an S3 bucket.

On the other hand, when we make the call to the .deploy() method a Docker container with the serve argument and our serv script will be called.

This script could be taken as is.

For most common cases this could be reused since the only thing it does is use the library tensorflow-serving (as you might have guess: installation step is defined on the Docker image).

The instance will pick the uploaded model and create an endpoint configuration and the actual endpoint which will be used for making predictions.

In conclusion, when you are happy with the results on your model and want to train and host your algorithm in Amazon SageMaker, you just have to switch the instance_type from ‘local’ to one of the available instances on AWS.

That’s all folks! Hope you like it.

Intro

Building – On-Demand ML Notebook Instances

Lowest Standard Instances

Highest Standard Instances

Lowest Compute Optimized

Highest Compute Optimized

Lowest GPU Instance

Highest GPU Instance

Model Training On-Demand ML Training Instances

Lowest Standard Instances

Highest Standard Instances

Lowest Compute Optimized

Highest Compute Optimized

Lowest GPU Instance

Highest GPU Instance

Model Deployment – On-Demand ML Hosting Instances for Real-Time Inference

Lowest Standard Instances

Highest Standard Instances

Lowest Compute Optimized

Highest Compute Optimized

Lowest GPU Instance

Highest GPU Instance

Cost Calculation

Required Steps

build_and_push.sh

Serve

Train

Scaling basic back-end architectures with Golang, Nodejs, Redis, NGINX, and AWS

2020 US Presidential Election pulse using NLP Transfer Learning

How to calculate the size of the WHOLE Internet with AWS EMR and Apache Spark

Top Tech Trends 2019

Custom Data for TF Object Detection API – Part 2

Dockerize and save money using your own ML algorithm

Intro

Building – On-Demand ML Notebook Instances

Lowest Standard Instances

Highest Standard Instances

Lowest Compute Optimized

Highest Compute Optimized

Lowest GPU Instance

Highest GPU Instance

Model Training On-Demand ML Training Instances

Lowest Standard Instances

Highest Standard Instances

Lowest Compute Optimized

Highest Compute Optimized

Lowest GPU Instance

Highest GPU Instance

Model Deployment – On-Demand ML Hosting Instances for Real-Time Inference

Lowest Standard Instances

Highest Standard Instances

Lowest Compute Optimized

Highest Compute Optimized

Lowest GPU Instance

Highest GPU Instance

Cost Calculation

Required Steps

build_and_push.sh

Serve

Train

SHARE THIS POST

Related Posts

Scaling basic back-end architectures with Golang, Nodejs, Redis, NGINX, and AWS

2020 US Presidential Election pulse using NLP Transfer Learning

How to calculate the size of the WHOLE Internet with AWS EMR and Apache Spark

Top Tech Trends 2019

Custom Data for TF Object Detection API – Part 2