tensorflow serving slim配置流程

发现国内tensorflow serving方面的博文非常少，包括国外也不多，主要依靠官方文档，但官方文档有些碎片化并包含一些冗余，因此整合为一篇文章方便国人。

之前在我在生产环境下用tensorflow的方式是启动一个thrift服务，load进模型之后不断处理请求，但是在面对大批量数据时有些力不从心，对机器的压力很大，因此考虑使用tensorflow serving的方式部署服务。

服务调用的模型是slim下的inception v3 图像分类模型，对模型进行了迁移学习，得到了新的分类模型，运用该模型对单张图片或者批量图片做预测或者提取特征。

本文假设你已成功安装tensorflow，并利用slim的代码，迁移学习获得了自己的inception v3模型，本文主要阐述如何将该模型部署到tensorflow serving 上。

tensorflow 版本：r.1.3 ， python版本：2.7.8 ，无GPU

Step1：安装tensorflow Serving r1.3

参考官方文档：https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/setup.md

前期准备：

安装bazel

pip install tensorflow-serving-api

pip install grpcio

安装过程：

  1.git clone --recurse-submodules https://github.com/tensorflow/serving

     cd serving

  2. cd tensorflow

   ./configure

---

 

You have bazel 0.7.0- (@non-git) installed.
Please specify the location of python. [Default is /data0/home/wang16/local/python/bin/python]:

Found possible Python library paths:
/data0/home/wang16/local/python/lib/python2.7/site-packages/
/data0/install/caffe/python/
/data0/home/wang16/simba/trunk/src/content_analysis/
/data0/home/wang16/local/python/lib/python2.7/site-packages
Please input the desired Python library path to use. Default is [/data0/home/wang16/local/python/lib/python2.7/site-packages/]

Do you wish to build TensorFlow with jemalloc as malloc support? [Y/n]: 
jemalloc as malloc support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Google Cloud Platform support? [Y/n]: n
No Google Cloud Platform support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Hadoop File System support? [Y/n]: n
No Hadoop File System support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Amazon S3 File System support? [Y/n]: n
No Amazon S3 File System support will be enabled for TensorFlow.

Do you wish to build TensorFlow with XLA JIT support? [y/N]: 
No XLA JIT support will be enabled for TensorFlow.

Do you wish to build TensorFlow with GDR support? [y/N]: 
No GDR support will be enabled for TensorFlow.

Do you wish to build TensorFlow with VERBS support? [y/N]: 
No VERBS support will be enabled for TensorFlow.

Do you wish to build TensorFlow with OpenCL support? [y/N]: 
No OpenCL support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]: 
No CUDA support will be enabled for TensorFlow.

Do you wish to build TensorFlow with MPI support? [y/N]: 
No MPI support will be enabled for TensorFlow.

Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]:

Add "--config=mkl" to your bazel command to build with MKL support.
Please note that MKL on MacOS or windows is still not supported.
If you would like to use a local MKL instead of downloading, please set the environment variable "TF_MKL_ROOT" every time before build.
Configuration finished

---

 

 cd ..

  3. bazel build -c opt --copt=-msse4.1 --copt=-msse4.2 --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-O3 tensorflow_serving/...(耗时较长)

这里如果是centos可能会报错：fatal error: stropts.h: No such file or directory

在/usr/include建一个空的文件。命名stropts.h就可以了。

  4. bazel test -c opt tensorflow_serving/...

 

Step2：修改代码以支持新的模型

参考文档：https://gyang274.github.io/docker-tensorflow-serving-slim/0x02b00.slim.inception.v4.html，https://github.com/gyang274/docker-tensorflow-serving-slim

参考文档：https://www.tensorflow.org/serving/serving_inception

这部分是最难的，官方提供的示例代码很少，幸亏找到了一篇相关文档，照猫画虎，才算完成。

  1. 回到 serving 主目录，目录下有如下内容

 mkdir -p tf_checkpoints/slim/my_inception_v3

 拷贝迁移学习训练好的模型到该目录中，拷贝完目录内容如下：

  2.参照示例写模型导出代码：tensorflow_serving/example/my_inception_v3_saved_model.py

# Copyright 2016 Google Inc. All Rights Reserved.

#

# Licensed under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.

# You may obtain a copy of the License at

#

#http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

# ==============================================================================

#!/usr/bin/env python2.7

"""Export inception v3 model given existing training checkpoints.

The model is exported as SavedModel with proper signatures that can be loaded by

standard tensorflow_model_server.

"""

importos.path

# This is a placeholder for a Google-internal import.

importtensorflow as tf

slim=tf.contrib.slim

# from preprocessing import inception_preprocessing

fromnets importinception

tf.app.flags.DEFINE_string('checkpoint_dir','/tmp/slim_inception_v3_train',

 """Directory where to read training checkpoints.""")

tf.app.flags.DEFINE_string('output_dir','/tmp/slim_inception_v3_output',

 """Directory where to export inference model.""")

tf.app.flags.DEFINE_integer('model_version',1,

 """Version number of the model.""")

tf.app.flags.DEFINE_integer('image_size',299,

 """Needs to provide same value as in training.""")

FLAGS=tf.app.flags.FLAGS

NUM_CLASSES=76

NUM_TOP_CLASSES=5

WORKING_DIR=os.path.dirname(os.path.realpath(__file__))

 

defpreprocess_image(image_buffer):

 """Preprocess JPEG encoded bytes to 3D float Tensor."""

 # Decode the string as an RGB JPEG.

 # Note that the resulting image contains an unknown height and width

 # that is set dynamically by decode_jpeg. In other words, the height

 # and width of image is unknown at compile-time.

 image=tf.image.decode_jpeg(image_buffer, channels=3)

 # After this point, all image pixels reside in [0,1)

 # until the very end, when they're rescaled to (-1, 1). The various

 # adjust_* ops all require this range for dtype float.

 image=tf.image.convert_image_dtype(image, dtype=tf.float32)

 # Crop the central region of the image with an area containing 87.5% of

 # the original image.

 image=tf.image.central_crop(image, central_fraction=0.875)

 # Resize the image to the original height and width.

 image=tf.expand_dims(image, 0)

 image=tf.image.resize_bilinear(

 image, [FLAGS.image_size, FLAGS.image_size], align_corners=False)

 image=tf.squeeze(image, [0])

 # Finally, rescale to [-1,1] instead of [0, 1)

 image=tf.subtract(image, 0.5)

 image=tf.multiply(image, 2.0)

 returnimage

 

defexport():

 with tf.Graph().as_default():

 # build inference model

 # labels

 f=open('/data0/home/wang16/package/serving/tf_checkpoints/slim/my_inception_v3/labels.txt')

 lines=f.readlines()

 names={0:'background'}

 forline inlines:

 i,label=line.split(':')

 names[int(i)]=label

 

 names_tensor=tf.constant(

 names.values()

 )

 names_lookup_table=tf.contrib.lookup.index_to_string_table_from_tensor(

 names_tensor

 )

 # input transformation

 serialized_tf_example=tf.placeholder(

 tf.string, name='tf_example'

 )

 feature_configs={

 'image/encoded': tf.FixedLenFeature(

 shape=[], dtype=tf.string

 ),

 }

 tf_example=tf.parse_example(

 serialized_tf_example, feature_configs

 )

 jpegs=tf_example['image/encoded']

 images=tf.map_fn(

 preprocess_image, jpegs, dtype=tf.float32

 )

 # run inference

 with slim.arg_scope(

 inception.inception_v3_arg_scope()

 ):

 # inception v3 models

 logits, end_points =inception.inception_v3(

 images, num_classes=NUM_CLASSES, is_training=False

 )

 # logits = tf.Print(logits, [logits])

 probs=tf.nn.softmax(logits)

 # transform output to topk result

 topk_probs, topk_indices =tf.nn.top_k(

 probs, NUM_TOP_CLASSES

 )

 topk_names=names_lookup_table.lookup(

 tf.to_int64(topk_indices)

 )

 init_fn=slim.assign_from_checkpoint_fn(

 os.path.join(FLAGS.checkpoint_dir,'model.ckpt-100000'),

 slim.get_model_variables('InceptionV3')

 )

 # sess config

 config=tf.ConfigProto(

 # device_count = {

 # 'GPU': 0

 # },

 gpu_options={

 'allow_growth':1,

 # 'per_process_gpu_memory_fraction': 0.01

 },

 allow_soft_placement=True,

 log_device_placement=False,

 )

 with tf.Session(config=config) as sess:

 init_fn(sess)

 # init on 2017.08.05

 # prelogits = sess.graph.get_tensor_by_name(

 # 'InceptionV3/Logits/PreLogitsFlatten/Reshape:0'

 # )

 # update on 2017.10.22

 # note: looks like the graphdef is updated for slim-inception-v3

 print('Graph Node Tensor Name:')

 fornode_tensor intf.get_default_graph().as_graph_def().node:

 ifstr(node_tensor.name).startswith('InceptionV3/Logits'):

 printstr(node_tensor.name)

 prelogits=sess.graph.get_tensor_by_name(

 'InceptionV3/Logits/Dropout_1b/Identity:0'

 )

 # an optional alternative

 # prelogits = end_points['PreLogitsFlatten']

 # export inference model.

 output_path=os.path.join(

 tf.compat.as_bytes(FLAGS.output_dir),

 tf.compat.as_bytes(str(FLAGS.model_version))

 )

 print'Exporting trained model to', output_path

 builder=tf.saved_model.builder.SavedModelBuilder(output_path)

 # build the signature_def_map.

 predict_inputs_tensor_info=tf.saved_model.utils.build_tensor_info(

 jpegs

 )

 classes_output_tensor_info=tf.saved_model.utils.build_tensor_info(

 topk_names

 )

 scores_output_tensor_info=tf.saved_model.utils.build_tensor_info(

 topk_probs

 )

 prelogits_output_tensor_info=tf.saved_model.utils.build_tensor_info(

 prelogits

 )

 prediction_signature=(

 tf.saved_model.signature_def_utils.build_signature_def(

 inputs={

 'images': predict_inputs_tensor_info

 },

 outputs={

 'classes': classes_output_tensor_info,

 'scores': scores_output_tensor_info,

 'prelogits': prelogits_output_tensor_info

 },

 method_name=tf.saved_model.signature_constants.PREDICT_METHOD_NAME

 )

 )

 legacy_init_op=tf.group(

 tf.tables_initializer(), name='legacy_init_op'

 )

 builder.add_meta_graph_and_variables(

 sess, [tf.saved_model.tag_constants.SERVING],

 signature_def_map={

 'predict_images':

 prediction_signature,

 },

 legacy_init_op=legacy_init_op

 )

 builder.save()

 print'Successfully exported model to %s' % FLAGS.output_dir

 

defmain(unused_argv=None):

 export()

 

if__name__ =='__main__':

 tf.app.run()

     参照示例写模型服务代码：tensorflow_serving/example/my_inception_v3_client.py

# Copyright 2016 Google Inc. All Rights Reserved.

#

# Licensed under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.

# You may obtain a copy of the License at

#

#http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

# ==============================================================================

#!/usr/bin/env python2.7

"""Send JPEG image to tensorflow_model_server loaded with inception model.

"""

from__future__ importprint_function

importurllib2

# This is a placeholder for a Google-internal import.

fromgrpc.beta importimplementations

importtensorflow as tf

fromtensorflow_serving.apis importpredict_pb2

fromtensorflow_serving.apis importprediction_service_pb2

 

tf.app.flags.DEFINE_string(

 'server','localhost:9000','PredictionService host:port'

)

tf.app.flags.DEFINE_string(

 'image_url', '', 'URL to image inJPEG format'

)

FLAGS=tf.app.flags.FLAGS

 

defmain(_):

 host, port =FLAGS.server.split(':')

 channel=implementations.insecure_channel(host, int(port))

 stub=prediction_service_pb2.beta_create_PredictionService_stub(channel)

 # Send request

 # See prediction_service.proto for gRPC request/response details.

 image_bytes=urllib2.urlopen(FLAGS.image_url).read()

 request=predict_pb2.PredictRequest()

 request.model_spec.name='my_inception_v3'

 request.model_spec.signature_name='predict_images'

 request.inputs['images'].CopyFrom(

 tf.contrib.util.make_tensor_proto(

 image_bytes, shape=[1]

 )

 )

 result=stub.Predict(request, 60.0)# 60 secs timeout

 print(result)

 

if__name__ =='__main__':

 tf.app.run()

     参照参考文献修改：tensorflow_serving/example/BUILD

...

 py_binary(

 name="my_inception_v3_saved_model",

 srcs=[

 "my_inception_v3_saved_model.py",

 ],

 srcs_version="PY2AND3",

 deps=[

 "@slim_model//:dataset_factory",

 "@slim_model//:preprocessing_factory",

 "@slim_model//:nets",

 "@org_tensorflow//tensorflow:tensorflow_py",

 ],

)

py_binary(

 name="my_inception_v3_client",

 srcs=[

 "my_inception_v3_client.py",

 ],

 srcs_version="PY2AND3",

 deps=[

 "//tensorflow_serving/apis:predict_proto_py_pb2",

 "//tensorflow_serving/apis:prediction_service_proto_py_pb2",

 "@org_tensorflow//tensorflow:tensorflow_py",

 ],

)

...

    参照参考文献修改：/tensorflow_serving/workspace.bzl

...

 native.new_local_repository(

 name="slim_model",

 path="tf_models/research/slim",

 build_file="tf_models/research/slim/BUILD",

 )

...

      修改代码： /tf_models/research/slim/BUILD，注释掉所有  "//tensorflow",

      编译代码:

      bazel build -c opt --copt=-msse4.1 --copt=-msse4.2 --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-O3 tensorflow_serving/...

      提取模型：      

    bazel-bin/tensorflow_serving/example/my_inception_v3_saved_model \
      --checkpoint_dir=tf_checkpoints/slim/my_inception_v3 \
      --output_dir=tf_servables/slim/my_inception_v3 \
      --model_version=1 \
      --image_size=299

      提取后的模型目录：

 

 

Step3：部署服务

   1、部署服务

    bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server --model_name=my_inception_v3 --model_base_path=$PWD/tf_servables/slim/my_inception_v3 --port=9000

   2、测试服务

    python my_inception_v3_client.py

  3、返回结果

outputs {
key: "classes"
value {
dtype: DT_STRING
tensor_shape {
dim {
size: 1
}
dim {
size: 5
}
}
string_val: "BeautifulSignt\n"
string_val: "Building\n"
string_val: "Cartoon\n"
string_val: "Society\n"
string_val: "Trip\n"
}
}
outputs {
key: "prelogits"
value {
dtype: DT_FLOAT
tensor_shape {
dim {
size: 1
}
dim {
size: 1
}
dim {
size: 1
}
dim {
size: 2048
}
}
float_val: 0.29251652956
float_val: 0.283589184284
float_val: 0.0797990858555
float_val: 0.427444338799
float_val: 0.147782608867
...
float_val: 0.784989655018
float_val: 0.0607083588839
float_val: 0.256205767393
float_val: 0.177307203412
}
}
outputs {
key: "scores"
value {
dtype: DT_FLOAT
tensor_shape {
dim {
size: 1
}
dim {
size: 5
}
}
float_val: 0.995890200138
float_val: 0.00202985620126
float_val: 0.000434114364907
float_val: 0.000334182434017
float_val: 0.00018374362844
}
}