Estimator, Experiment, and Dataset的关系
The Estimator class represents a model, as well as how this model should be trained and evaluated. We can create an estimator as follows:
To create the Estimator we need to pass in a model function, a collection of parameters and some configuration.
- The parameters should be a collection of the model’s hyperparameters. This can be a dictionary, but we will represent it in this example as an HParams object, which acts as a namedtuple.
- The configuration specifies how the training and evaluation are run, and where to store the results. This configuration will be represented by a RunConfig object, which communicates everything the Estimator needs to know about the environment in which the model will be run.
- The model function is a Python function, which builds the model given the input. (More on this later)
The model function is a Python function which is passed as a first-class function to the Estimator. We’ll see later that TensorFlow uses first-class functions in other places. The benefit of representing the model as a function is that the model can be recreated over and over by instantiating the function. The model can be recreated during the training with different input, for example, to run validation tests during training.
The model function takes the input features as parameters and the corresponding labels as tensors. It also takes a mode that signals if the model is training, evaluating or performing inference. The last parameter to the model function should be a collection of hyperparameters, which are the same as those passed to the Estimator. This model function should return an EstimatorSpec object which will define the complete model.
The EstimatorSpec takes in the prediction, loss, training and evaluation Operations so it defines the full model graph used for training, evaluation, and inference. Because the EstimatorSpec just takes in regular TensorFlow Operations, we can use frameworks like TF-Slim to define our model.
- 预创建的 Estimator
- 自定义 Estimator
- 从 Keras 模型创建 Estimator
# Instantiate a Keras inception v3 model.
Keras 模型本身容易建立，导出到Estimator后又可以利用 Estimator 的优势，例如分布式训练。
- 环境适配性好。可以在本地主机、分布式多服务器环境中运行基于 Estimator 的模型，而无需更改模型。此外，可以在 CPU、GPU 或 TPU 上运行基于 Estimator 的模型，而无需重新编码模型。
- 采用 Estimator 创建模型通常比采用低阶 TensorFlow API 更简单。
- Estimator 会自动构建图。
- Estimator 提供安全的分布式训练循环，可以控制如何以及何时：
- 保存 TensorBoard 的摘要
- 学习流程：Estimator 封装了对机器学习不同阶段的控制，用户无需不断的为新机器学习任务重复编写训练、评估、预测的代码。可以专注于对网络结构的控制。
- 网络结构：Estimator 的网络结构是在 model_fn 中独立定义的，用户创建的任何网络结构都可以在 Estimator 的控制下进行机器学习。这可以允许用户很方便的使用别人定义好的 model_fn。
- 数据导入：Estimator 的数据导入也是由 input_fn 独立定义的。例如，用户可以非常方便的只通过改变 input_fn 的定义，来使用相同的网络结构学习不同的数据。
使用 Estimator 编写应用时，您必须将数据输入管道从模型中分离出来。这种分离简化了不同数据集的实验流程。
依赖预创建的 Estimator 的 TensorFlow 程序通常包含下列四个步骤：
- 一个字典，其中键是特征名称，值是包含相应特征数据的张量（或 SparseTensor）
- 定义特征列。 每个
tf.feature_column都标识了特征名称、特征类型和任何输入预处理操作。例如，以下代码段创建了三个存储整数或浮点数据的特征列。前两个特征列仅标识了特征的名称和类型。第三个特征列还指定了一个 lambda，该程序将调用此 lambda 来调节原始数据：
# Define three numeric feature columns.
- 实例化相关的预创建的 Estimator。 例如，下面是对名为
LinearClassifier的预创建 Estimator 进行实例化的示例代码：
# Instantiate an estimator, passing the feature columns.
- 调用训练、评估或推理方法。例如，所有 Estimator 都提供训练模型的
# my_training_set is the function created in Step 1 estimator.train(input_fn=my_training_set, steps=2000)
The Experiment class defines how to train a model and integrates nicely with the Estimator. We can create an experiment as follows:
experiment = tf.contrib.learn.Experiment(
The Experiment takes as input:
- An estimator (for example the one we defined above).
- Train and evaluation data as a first-class function. The same concept as the model function explained earlier is used here. By passing in a function instead of operation, the input graph can be recreated if needed. We’ll talk more about this later.
- Training and Evaluating hooks. These hooks can be used to save or monitor specific things, or to set up certain operations in the Graph or Session. For example, we will be passing in operations to help initialize the data loaders (again, more later).
- Various parameters describing how long to train for and when to evaluate.
Once we have defined the experiment, we can run it to train and evaluate the model with learn_runner.run as follows:
Like the model function and the data functions, the learn runner takes in the function that creates the experiment as a parameter.
We’ll be using the Dataset class and the corresponding Iterator to represent our training and evaluation data, and to create data feeders that iterate over the data during training. In this example, we will use the MNIST data that’s available in Tensorflow, and build a Dataset wrapper around it. For example, we will represent the training input data as:
# Define the training inputs
Calling this get_train_inputs will return a first-class function that creates the data loading operations in a TensorFlow graph, together with a Hook to initialize the iterator.
The MNIST data used in this example is initially represented as a Numpy array. We create a placeholder tensor that gets the data fed in; we use a placeholder in order to avoid copying the data. Next, we create a sliced dataset with the help of from_tensor_slices. We will make sure that this dataset runs for an infinite amount of epochs (the experiment can take care of limiting the number of epochs), and that the data gets shuffled and put into batches of the required size.
To iterate over the data we need to create an iterator from the dataset. Because we are using a placeholder we need to initialize the placeholder in the relevant session with the NumPy data. We can do this by creating an initializable iterator. We will create a custom defined _IteratorInitializerHook _object to initialize the iterator when the graph is created:
The IteratorInitializerHook inherits from SessionRunHook. This hook will call after_create_session as soon as the relevant session is created, and initialize the placeholder with the right data. This hook is returned by our get_train_inputs function and will be passed to the Experiment object upon creation.
The data loading operations returned by the train_inputs function are TensorFlow operations that will return a new batch every time they are evaluated.
- Higher-Level APIs in TensorFlow
- TensorFlow Estimator
- TensorFlow高层API：Custom Estimator建立CNN+RNN
- There is a paper called “TensorFlow Estimators: Managing Simplicity vs. Flexibility in High-Level Machine Learning Frameworks” describing the high level-design of the Estimator framework.
- TensorFlow has more documentation on using the Dataset API.
- There are 2 versions of the Estimator class. We are using the one at tf.estimator.Estimator in this example, but there is also an older unstable version at tf.contrib.learn.Estimator.
- There are also 2 versions of the RunConfig class. While we are using the one at tf.contrib.learn.RunConfig there is also a version at tf.estimator.RunConfig. I couldn’t get the latter one to work with the Experiment framework so I stuck with the tf.contrib version.
- While we didn’t use them in this example, the Estimator framework defines predefined estimators for typical models such as classifiers and regressors. These predefined estimators are easy to use and come with a detailed tutorial.
- TensorFlow also defines an abstraction for the “head” of a model, the part that sits on top of the architecture and defines the loss, evaluation and training operations. This head will take care of things like defining the model function, and all the required Operations. You can find a version at tf.contrib.learn.Head. There is also a prototype version in the newer estimator framework. We decided not to use it in this example due to its development being quite unstable.
- This blog uses the TensorFlow slim framework to define the architecture of the model. Slim is a lightweight library for defining complex models in tensorflow. They also define pre-defined architectures and pre-trained models.