Loading a TensorFlow graph with the C++ API

From https://medium.com/jim-fleming/loading-a-tensorflow-graph-with-the-c-api-4caaff88463f#.j1dwfsrbj

Check out the related post:Loading TensorFlow graphs via host languages (using the C API).

The current documentation around loading a graph with C++ isprettysparse so I spent some time setting up a barebones example. In the TensorFlow repo there are more involved examples, such asbuilding a graph in C++. However, the C++ API for constructing graphs is not as complete as the Python API. Many features (including automatic gradient computation) are not available from C++ yet. Another example in the repo demonstratesdefining your own operations but most users will never need this. I imagine the most common use case for the C++ API is for loading pre-trained graphs to be standalone or embedded in other applications.

Be aware, there are some caveats to this approach that I’ll cover at the end.

Requirements

• Install Bazel: Google’s build tool used to compile things for TensorFlow.
• Clone the TensorFlow repo. Be sure to include submodules using the recursive flag (thanks to@kristophergiesing for catching this):

git clone --recursive https://github.com/tensorflow/tensorflow

Creating the graph

Let’s start by creating a minimal TensorFlow graph and write it out as a protobuf file. Make sure to assign names to your inputs and operations so they’re easier to assign when we execute the graph later. The node’s do have default names but they aren’t very useful: Variable_1 or Mul_3. Here’s an example created with Jupyter:

In [4]:

import tensorflow as tfimport numpy as npwith tf.Session() as sess:    a = tf.Variable(5.0, name='a')    b = tf.Variable(6.0, name='b')    c = tf.mul(a, b, name="c")    sess.run(tf.initialize_all_variables())    print a.eval() # 5.0    print b.eval() # 6.0    print c.eval() # 30.0        tf.train.write_graph(sess.graph_def, 'models/', 'graph.pb', as_text=False)

5.06.030.0

In [ ]:

Creating a simple binary or shared library

Let’s create a new folder liketensorflow/tensorflow/<my project name> for your binary or library to live. I’m going to call the projectloader since it will be loading a graph.

Inside this project folder we’ll create a new file called<my project name>.cc (e.g.loader.cc). If you’re curious, the .cc extension is essentially the same as .cpp but is preferred by Google’s code guidelines.

Inside loader.cc we’re going to do a few things:

1. Initialize a TensorFlow session.
2. Read in the graph we exported above.
3. Add the graph to the session.
4. Setup our inputs and outputs.
5. Run the graph, populating the outputs.
6. Read values from the outputs.
7. Close the session to release resources.

#include "tensorflow/core/public/session.h"
#include "tensorflow/core/platform/env.h"

using namespace tensorflow;

int main(int argc, char* argv[]) {
  // Initialize a tensorflow session
  Session* session;
  Status status = NewSession(SessionOptions(), &session);
  if (!status.ok()) {
    std::cout << status.ToString() << "\n";
    return 1;
  }

  // Read in the protobuf graph we exported
  // (The path seems to be relative to the cwd. Keep this in mind
  // when using `bazel run` since the cwd isn't where you call
  // `bazel run` but from inside a temp folder.)
  GraphDef graph_def;
  status = ReadBinaryProto(Env::Default(), "models/train.pb", &graph_def);
  if (!status.ok()) {
    std::cout << status.ToString() << "\n";
    return 1;
  }

  // Add the graph to the session
  status = session->Create(graph_def);
  if (!status.ok()) {
    std::cout << status.ToString() << "\n";
    return 1;
  }

  // Setup inputs and outputs:

  // Our graph doesn't require any inputs, since it specifies default values,
  // but we'll change an input to demonstrate.
  Tensor a(DT_FLOAT, TensorShape());
  a.scalar<float>()() = 3.0;

  Tensor b(DT_FLOAT, TensorShape());
  b.scalar<float>()() = 2.0;

  std::vector<std::pair<string, tensorflow::Tensor>> inputs = {
    { "a", a },
    { "b", b },
  };

  // The session will initialize the outputs
  std::vector<tensorflow::Tensor> outputs;

  // Run the session, evaluating our "c" operation from the graph
  status = session->Run(inputs, {"c"}, {}, &outputs);
  if (!status.ok()) {
    std::cout << status.ToString() << "\n";
    return 1;
  }

  // Grab the first output (we only evaluated one graph node: "c")
  // and convert the node to a scalar representation.
  auto output_c = outputs[0].scalar<float>();

  // (There are similar methods for vectors and matrices here:
  // https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/public/tensor.h)

  // Print the results
  std::cout << outputs[0].DebugString() << "\n"; // Tensor<type: float shape: [] values: 30>
  std::cout << output_c() << "\n"; // 30

  // Free any resources used by the session
  session->Close();
  return 0;
}

Now we create a BUILD file for our project. This tells Bazel what to compile. Inside we want to define acc_binary for our program. You can also use thelinkshared option on the binary to produce a shared library or thecc_library rule if you’re going to link it using Bazel.

cc_binary(name="loader",srcs= ["loader.cc"],deps= ["//tensorflow/core:tensorflow",])

Here’s the final directory structure:

• tensorflow/tensorflow/loader/
• tensorflow/tensorflow/loader/loader.cc
• tensorflow/tensorflow/loader/BUILD

Compile & Run

• From inside the project folder callbazel build :loader.
• From the repository root, go intobazel-bin/tensorflow/loader.
• Copy the graph protobuf tomodels/train.pb.
• Then run ./loader and check the output!

You could also call bazel run :loader to run the executable directly, however the working directory forbazel run is buried in a temporary folder andReadBinaryProto looks in the current working directory for relative paths.

And that should be all we need to do to compile and run C++ code for TensorFlow.

The last thing to cover are the caveats I mentioned:

1. The build is huge, coming in at 103MB, even for this simple example. Much of this is for TensorFlow, CUDA support and numerous dependencies we never use. This is especially true since the C++ API doesn’t support much functionality right now, as a large portion of the TensorFlow API is Python-only. There is probably a better way of linking to TensorFlow (e.g. shared library) but I haven’t gotten it working yet.
2. There doesn’t seem to be a straightforward way of building thisoutside of the TensorFlow repo because of Bazel (many of the modules needed to link to are marked as internal). Again, there is probably a solution to this, it’s just non-obvious.

Hopefully someone can shed some light on these last points so we can begin to embed TensorFlow graphs in applications. If you are that person, message me onTwitter or email. I’ll update this post when that happens.