tensorpack.models package¶

tensorpack.models.
BatchNorm
(scope_name, x, use_local_stat=None, decay=0.9, epsilon=1e05, use_scale=True, use_bias=True, gamma_init=<tf.python.ops.init_ops.Constant object>, data_format='NHWC')[source]¶ Batch Normalization layer, as described in the paper: Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariance Shift.
Parameters: x (tf.Tensor) – a 4D or 2D tensor. When 4D, the layout should match data_format.
use_local_stat (bool) – whether to use mean/var of the current batch or the moving average. Defaults to True in training and False in inference.
decay (float) – decay rate of moving average.
epsilon (float) – epsilon to avoid dividebyzero.
use_bias (use_scale,) – whether to use the extra affine transformation or not.
gamma_init – initializer for gamma (the scale).
Returns: tf.Tensor – a tensor named
output
with the same shape of x.Variable Names:
beta
: the bias term. Will be zeroinited by default.gamma
: the scale term. Will be oneinited by default. Input will be transformed byx * gamma + beta
.mean/EMA
: the moving average of mean.variance/EMA
: the moving average of variance.
Note
About multiGPU training: moving averages across GPUs are not aggregated. Batch statistics are computed independently. This is consistent with most frameworks.
 Combinations of
use_local_stat
andctx.is_training
: use_local_stat == is_training
: standard BN, EMA aremaintained during training and used during inference.
use_local_stat and not is_training
: still use local (batch)statistics in inference.
not use_local_stat and is_training
: use EMA to normalize intraining. This is useful when you load a pretrained BN and don’t want to fine tune the EMA. EMA will not be updated in this case.
 Combinations of

tensorpack.models.
BatchRenorm
(scope_name, x, rmax, dmax, decay=0.9, epsilon=1e05, use_scale=True, use_bias=True, data_format='NHWC')[source]¶ Batch Renormalization layer, as described in the paper: Batch Renormalization: Towards Reducing Minibatch Dependence in BatchNormalized Models. This implementation is a wrapper around tf.layers.batch_normalization.
Parameters: Returns: tf.Tensor – a tensor named
output
with the same shape of x.Variable Names:
beta
: the bias term.gamma
: the scale term. Input will be transformed byx * gamma + beta
.moving_mean, renorm_mean, renorm_mean_weight
: See TF documentation.moving_variance, renorm_stddev, renorm_stddev_weight
: See TF documentation.

tensorpack.models.
layer_register
(log_shape=False, use_scope=True)[source]¶ Parameters: log_shape (bool) – log input/output shape of this layer
use_scope (bool or None) – Whether to call this layer with an extra first argument as scope. When set to None, it can be called either with or without the scope name argument. It will try to figure out by checking if the first argument is string or not.
Returns: A decorator used to register a layer.
Examples:
@layer_register(use_scope=True) def add10(x): return x + tf.get_variable('W', shape=[10])

class
tensorpack.models.
VariableHolder
(**kwargs)[source]¶ Bases:
object
A proxy to access variables defined in a layer.

tensorpack.models.
Conv2D
(scope_name, x, out_channel, kernel_shape, padding='SAME', stride=1, W_init=None, b_init=None, nl=<function identity>, split=1, use_bias=True, data_format='NHWC')[source]¶ 2D convolution on 4D inputs.
Parameters: x (tf.Tensor) – a 4D tensor. Must have known number of channels, but can have other unknown dimensions.
out_channel (int) – number of output channel.
kernel_shape – (h, w) tuple or a int.
stride – (h, w) tuple or a int.
padding (str) – ‘valid’ or ‘same’. Case insensitive.
split (int) – Split channels as used in Alexnet. Defaults to 1 (no split).
W_init – initializer for W. Defaults to variance_scaling_initializer.
b_init – initializer for b. Defaults to zero.
nl – a nonlinearity function.
use_bias (bool) – whether to use bias.
Returns: tf.Tensor named
output
with attribute variables.Variable Names:
W
: weightsb
: bias

tensorpack.models.
Deconv2D
(scope_name, x, out_channel, kernel_shape, stride, padding='SAME', W_init=None, b_init=None, nl=<function identity>, use_bias=True, data_format='NHWC')[source]¶ 2D deconvolution on 4D inputs.
Parameters: x (tf.Tensor) – a tensor of shape NHWC. Must have known number of channels, but can have other unknown dimensions.
out_channel – the output number of channel.
kernel_shape – (h, w) tuple or a int.
stride – (h, w) tuple or a int.
padding (str) – ‘valid’ or ‘same’. Case insensitive.
W_init – initializer for W. Defaults to variance_scaling_initializer.
b_init – initializer for b. Defaults to zero.
nl – a nonlinearity function.
use_bias (bool) – whether to use bias.
Returns: tf.Tensor – a NHWC tensor named
output
with attribute variables.Variable Names:
W
: weightsb
: bias

tensorpack.models.
FullyConnected
(scope_name, x, out_dim, W_init=None, b_init=None, nl=<function identity>, use_bias=True)[source]¶ FullyConnected layer, takes a N>1D tensor and returns a 2D tensor. It is an equivalent of tf.layers.dense except for naming conventions.
Parameters: Returns: tf.Tensor – a NC tensor named
output
with attribute variables.Variable Names:
W
: weights of shape [in_dim, out_dim]b
: bias

tensorpack.models.
ImageSample
(scope_name, inputs, borderMode='repeat')[source]¶ Sample the images using the given coordinates, by bilinear interpolation. This was described in the paper: Spatial Transformer Networks.
Parameters: inputs (list) – [images, coords]. images has shape NHWC. coords has shape (N, H’, W’, 2), where each pair of the last dimension is a (y, x) realvalue coordinate.
borderMode – either “repeat” or “constant” (zerofilled)
Returns: tf.Tensor – a tensor named
output
of shape (N, H’, W’, C).

tensorpack.models.
LayerNorm
(scope_name, x, epsilon=1e05, use_bias=True, use_scale=True, data_format='NHWC')[source]¶ Layer Normalization layer, as described in the paper: Layer Normalization.
Parameters: x (tf.Tensor) – a 4D or 2D tensor. When 4D, the layout should match data_format.
epsilon (float) – epsilon to avoid dividebyzero.
use_bias (use_scale,) – whether to use the extra affine transformation or not.

tensorpack.models.
InstanceNorm
(scope_name, x, epsilon=1e05, data_format='NHWC', use_affine=True)[source]¶ Instance Normalization, as in the paper: Instance Normalization: The Missing Ingredient for Fast Stylization.
Parameters:

class
tensorpack.models.
LinearWrap
(tensor)[source]¶ Bases:
object
A simple wrapper to easily create “linear” graph, consisting of layers / symbolic functions with only one input & output.

apply
(func, *args, **kwargs)[source]¶ Apply a function on the wrapped tensor.
Returns: LinearWrap – LinearWrap(func(self.tensor(), *args, **kwargs))
.

apply2
(func, *args, **kwargs)[source]¶ Apply a function on the wrapped tensor. The tensor will be the second argument of func.
Returns: LinearWrap – LinearWrap(func(args[0], self.tensor(), *args[1:], **kwargs))
.

print_tensor
()[source]¶ Print the underlying tensor and return self. Can be useful to get the name of tensors inside
LinearWrap
.Returns: self


tensorpack.models.
Maxout
([scope_name, ]x, num_unit)[source]¶ Maxout as in the paper Maxout Networks.
Parameters: x (tf.Tensor) – a NHWC or NC tensor. Channel has to be known.
num_unit (int) – a int. Must be divisible by C.
Returns: tf.Tensor – of shape NHW(C/num_unit) named
output
.

tensorpack.models.
PReLU
(scope_name, x, init=0.001, name='output')[source]¶ Parameterized ReLU as in the paper Delving Deep into Rectifiers: Surpassing HumanLevel Performance on ImageNet Classification.
Parameters: Variable Names:
alpha
: learnable slope.

tensorpack.models.
LeakyReLU
([scope_name, ]x, alpha, name='output')[source]¶ Leaky ReLU as in paper Rectifier Nonlinearities Improve Neural Network Acoustic Models.
Parameters: x (tf.Tensor) – input
alpha (float) – the slope.

tensorpack.models.
BNReLU
([scope_name, ]x, name=None)[source]¶ A shorthand of BatchNormalization + ReLU.

tensorpack.models.
MaxPooling
(scope_name, x, shape, stride=None, padding='VALID', data_format='NHWC')[source]¶ Max Pooling on 4D tensors.
Parameters: x (tf.Tensor) – a 4D tensor.
shape – int or (h, w) tuple
stride – int or (h, w) tuple. Defaults to be the same as shape.
padding (str) – ‘valid’ or ‘same’.
Returns: tf.Tensor named
output
.

tensorpack.models.
FixedUnPooling
(scope_name, x, shape, unpool_mat=None, data_format='NHWC')[source]¶ Unpool the input with a fixed matrix to perform kronecker product with.
Parameters: x (tf.Tensor) – a 4D image tensor
shape – int or (h, w) tuple
unpool_mat – a tf.Tensor or np.ndarray 2D matrix with size=shape. If is None, will use a matrix with 1 at topleft corner.
Returns: tf.Tensor – a 4D image tensor.

tensorpack.models.
AvgPooling
(scope_name, x, shape, stride=None, padding='VALID', data_format='NHWC')[source]¶ Average Pooling on 4D tensors.
Parameters: x (tf.Tensor) – a 4D tensor.
shape – int or (h, w) tuple
stride – int or (h, w) tuple. Defaults to be the same as shape.
padding (str) – ‘valid’ or ‘same’.
Returns: tf.Tensor named
output
.

tensorpack.models.
GlobalAvgPooling
(scope_name, x, data_format='NHWC')[source]¶ Global average pooling as in the paper Network In Network.
Parameters: x (tf.Tensor) – a NHWC tensor. Returns: tf.Tensor – a NC tensor named output
.

tensorpack.models.
BilinearUpSample
(scope_name, x, shape)[source]¶ Deterministic bilinearlyupsample the input images.
Parameters: x (tf.Tensor) – a NHWC tensor
shape (int) – the upsample factor
Returns: tf.Tensor – a NHWC tensor.

tensorpack.models.
regularize_cost
(regex, func, name='regularize_cost')[source]¶ Apply a regularizer on trainable variables matching the regex, and print the matched variables (only print once in multitower training). In replicated mode, it will only regularize variables within the current tower.
Parameters: regex (str) – a regex to match variable names, e.g. “conv.*/W”
func – the regularization function, which takes a tensor and returns a scalar tensor. E.g.,
tf.contrib.layers.l2_regularizer
.
Returns: tf.Tensor – the total regularization cost.
Example
cost = cost + regularize_cost("fc.*/W", l2_regularizer(1e5))

tensorpack.models.
l2_regularizer
(scale, scope=None)[source]¶ Returns a function that can be used to apply L2 regularization to weights.
Small values of L2 can help prevent overfitting the training data.
Parameters: scale – A scalar multiplier Tensor. 0.0 disables the regularizer.
scope – An optional scope name.
Returns: A function with signature l2(weights) that applies L2 regularization.
Raises: ValueError
– If scale is negative or if scale is not a float.

tensorpack.models.
l1_regularizer
(scale, scope=None)[source]¶ Returns a function that can be used to apply L1 regularization to weights.
L1 regularization encourages sparsity.
Parameters: scale – A scalar multiplier Tensor. 0.0 disables the regularizer.
scope – An optional scope name.
Returns: A function with signature l1(weights) that apply L1 regularization.
Raises: ValueError
– If scale is negative or if scale is not a float.

tensorpack.models.
Dropout
([scope_name, ]x, keep_prob=0.5, is_training=None, noise_shape=None)[source]¶ Dropout layer as in the paper Dropout: a Simple Way to Prevent Neural Networks from Overfitting.
Parameters:

tensorpack.models.
ConcatWith
([scope_name, ]x, tensor, dim)[source]¶ A wrapper around
tf.concat
to cooperate withLinearWrap
.Parameters: Returns: tf.Tensor –
tf.concat([x] + tensor, dim)

tensorpack.models.
SoftMax
([scope_name, ]x, use_temperature=False, temperature_init=1.0)[source]¶ A SoftMax layer (w/o linear projection) with optional temperature, as defined in the paper Distilling the Knowledge in a Neural Network.
Parameters: Returns: tf.Tensor – a tensor of the same shape named
output
.Variable Names:
invtemp
: 1.0/temperature.