chainer.training.extensions.Evaluator¶
-
class
chainer.training.extensions.
Evaluator
(iterator, target, converter=<function concat_examples>, device=None, eval_hook=None, eval_func=None)[source]¶ Trainer extension to evaluate models on a validation set.
This extension evaluates the current models by a given evaluation function. It creates a
Reporter
object to store values observed in the evaluation function on each iteration. The report for all iterations are aggregated toDictSummary
. The collected mean values are further reported to the reporter object of the trainer, where the name of each observation is prefixed by the evaluator name. SeeReporter
for details in naming rules of the reports.Evaluator has a structure to customize similar to that of
StandardUpdater
. The main differences are:There are no optimizers in an evaluator. Instead, it holds links to evaluate.
An evaluation loop function is used instead of an update function.
Preparation routine can be customized, which is called before each evaluation. It can be used, e.g., to initialize the state of stateful recurrent networks.
There are two ways to modify the evaluation behavior besides setting a custom evaluation function. One is by setting a custom evaluation loop via the
eval_func
argument. The other is by inheriting this class and overriding theevaluate()
method. In latter case, users have to create and handle a reporter object manually. Users also have to copy the iterators before using them, in order to reuse them at the next time of evaluation. In both cases, the functions are called in testing mode (i.e.,chainer.config.train
is set toFalse
).This extension is called at the end of each epoch by default.
- Parameters
iterator – Dataset iterator for the validation dataset. It can also be a dictionary of iterators. If this is just an iterator, the iterator is registered by the name
'main'
.target – Link object or a dictionary of links to evaluate. If this is just a link object, the link is registered by the name
'main'
.converter – Converter function to build input arrays.
concat_examples()
is used by default.device – Device to which the validation data is sent. Negative value indicates the host memory (CPU).
eval_hook – Function to prepare for each evaluation process. It is called at the beginning of the evaluation. The evaluator extension object is passed at each call.
eval_func – Evaluation function called at each iteration. The target link to evaluate as a callable is used by default.
- Variables
converter – Converter function.
device – Device to which the validation data is sent.
eval_hook – Function to prepare for each evaluation process.
eval_func – Evaluation function called at each iteration.
Methods
-
__call__
(trainer=None)[source]¶ Executes the evaluator extension.
Unlike usual extensions, this extension can be executed without passing a trainer object. This extension reports the performance on validation dataset using the
report()
function. Thus, users can use this extension independently from any trainer by manually configuring aReporter
object.
-
evaluate
()[source]¶ Evaluates the model and returns a result dictionary.
This method runs the evaluation loop over the validation dataset. It accumulates the reported values to
DictSummary
and returns a dictionary whose values are means computed by the summary.Note that this function assumes that the main iterator raises
StopIteration
or code in the evaluation loop raises an exception. So, if this assumption is not held, the function could be caught in an infinite loop.Users can override this method to customize the evaluation routine.
Note
This method encloses
eval_func
calls withfunction.no_backprop_mode()
context, so all calculations usingFunctionNode
s insideeval_func
do not make computational graphs. It is for reducing the memory consumption.
-
finalize
()[source]¶ Finalizes the evaluator object.
This method calls the finalize method of each iterator that this evaluator has. It is called at the end of training loops.
-
initialize
(trainer)[source]¶ Initializes up the trainer state.
This method is called before entering the training loop. An extension that modifies the state of
Trainer
can override this method to initialize it.When the trainer has been restored from a snapshot, this method has to recover an appropriate part of the state of the trainer.
For example,
ExponentialShift
extension changes the optimizer’s hyperparameter at each invocation. Note that the hyperparameter is not saved to the snapshot; it is the responsibility of the extension to recover the hyperparameter. TheExponentialShift
extension recovers it in itsinitialize
method if it has been loaded from a snapshot, or just setting the initial value otherwise.- Parameters
trainer (Trainer) – Trainer object that runs the training loop.
-
on_error
(trainer, exc, tb)[source]¶ Handles the error raised during training before finalization.
This method is called when an exception is thrown during the training loop, before finalize. An extension that needs different error handling from finalize, can override this method to handle errors.
-
serialize
(serializer)[source]¶ Serializes the extension state.
It is called when a trainer that owns this extension is serialized. It serializes nothing by default.
-
__eq__
()¶ Return self==value.
-
__ne__
()¶ Return self!=value.
-
__lt__
()¶ Return self<value.
-
__le__
()¶ Return self<=value.
-
__gt__
()¶ Return self>value.
-
__ge__
()¶ Return self>=value.
Attributes
-
default_name
= 'validation'¶
-
name
= None¶
-
priority
= 300¶
-
trigger
= (1, 'epoch')¶