We tried to use Tensorflow's native C++ library to run some predictive model in real time on CPU-only. The performance sucked, even though the model was not particularly complex.

My advice would be to only use Tensorflow with the python wrappers. It's just too complex.