Serving a Keras model using Tensorflow serving and Docker

A Keras model can be created in various ways, for example using the sequential model:

from keras.models import Sequential
from keras.layers import Dense, Activation

model = Sequential([
    Dense(32, input_shape=(784,)),
    Activation('relu'),
    Dense(10),
    Activation('softmax'),
])

Such model can be saved using the save() function:

model.save('./my_model/1')

Here, the /1 at the end of the path is important for later.

The model will be run inside a docker container. To do so, first download and run an "empty" serving container which will serve as a base:

docker run -d --name serving_base tensorflow/serving

Here, the name serving_base is given to the container using the --name flag to easily manipulate it later

Then copy the model into it. Here it is assumed that the user is in the working directory of the python script

docker cp ./my_model serving_base:/models/my_model 

The container thus obtained can be saved into a new image using the commit command:

docker commit --change "ENV MODEL_NAME my_model" serving_base my-model-serving

With this done, the base container is no longer needed and can be deleted:

docker kill serving_base
docker rm serving_base

Containers can now be started from the generated image:

docker run -d -p 8501:8501 my_model_serving

To test if this is working, curl can be used:

curl -d '{"instances": [1.0, 2.0, 5.0]}' http://localhost:8501/v1/models/my_model:predict