tensorpack.utils package

Common utils. These utils should be irrelevant to tensorflow.

tensorpack.utils.change_env(name, val)[source]
Parameters
  • name (str) – name of the env var

  • val (str or None) – the value, or set to None to clear the env var.

Returns

a context where the environment variable name being set to val. It will be set back after the context exits.

tensorpack.utils.get_rng(obj=None)[source]

Get a good RNG seeded with time, pid and the object.

Parameters

obj – some object to use to generate random seed.

Returns

np.random.RandomState – the RNG.

tensorpack.utils.fix_rng_seed(seed)[source]

Call this function at the beginning of program to fix rng seed within tensorpack.

Parameters

seed (int) –

Example

Fix random seed in both tensorpack and tensorflow.

seed = 42
utils.fix_rng_seed(seed)
tesnorflow.set_random_seed(seed)
# run trainer
tensorpack.utils.get_tqdm(*args, **kwargs)[source]

Similar to tqdm.tqdm(), but use tensorpack’s default options to have consistent style.

tensorpack.utils.argtools module

tensorpack.utils.argtools.map_arg(**maps)[source]

Apply a mapping on certain argument before calling the original function.

Parameters

maps (dict) – {argument_name: map_func}

tensorpack.utils.argtools.memoized(user_function)

Alias to functools.lru_cache() WARNING: memoization will keep keys and values alive!

tensorpack.utils.argtools.memoized_method(func)[source]

A decorator that performs memoization on methods. It stores the cache on the object instance itself.

tensorpack.utils.argtools.graph_memoized(func)[source]

Like memoized, but keep one cache per default graph.

tensorpack.utils.argtools.shape2d(a)[source]

Ensure a 2D shape.

Parameters

a – a int or tuple/list of length 2

Returns

list – of length 2. if a is a int, return [a, a].

tensorpack.utils.argtools.shape4d(a, data_format='NHWC')[source]

Ensuer a 4D shape, to use with 4D symbolic functions.

Parameters

a – a int or tuple/list of length 2

Returns

list

of length 4. if a is a int, return [1, a, a, 1]

or [1, 1, a, a] depending on data_format.

tensorpack.utils.argtools.memoized_ignoreargs(func)[source]

A decorator. It performs memoization ignoring the arguments used to call the function.

tensorpack.utils.argtools.log_once(message, func='info')[source]

Log certain message only once. Call this function more than one times with the same message will result in no-op.

Parameters
  • message (str) – message to log

  • func (str) – the name of the logger method. e.g. “info”, “warn”, “error”.

tensorpack.utils.concurrency module

class tensorpack.utils.concurrency.StoppableThread(evt=None)[source]

Bases: threading.Thread

A thread that has a ‘stop’ event.

__init__(evt=None)[source]
Parameters

evt (threading.Event) – if None, will create one.

queue_get_stoppable(q)[source]

Take obj from queue, but will give up when the thread is stopped

queue_put_stoppable(q, obj)[source]

Put obj to queue, but will give up when the thread is stopped

stop()[source]

Stop the thread

stopped()[source]
Returns

bool – whether the thread is stopped or not

class tensorpack.utils.concurrency.LoopThread(func, pausable=True)[source]

Bases: tensorpack.utils.concurrency.StoppableThread

A pausable thread that simply runs a loop

__init__(func, pausable=True)[source]
Parameters

func – the function to run

pause()[source]

Pause the loop

resume()[source]

Resume the loop

run()[source]

Method representing the thread’s activity.

You may override this method in a subclass. The standard run() method invokes the callable object passed to the object’s constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively.

class tensorpack.utils.concurrency.ShareSessionThread(th=None)[source]

Bases: threading.Thread

A wrapper around thread so that the thread uses the default session at “start()” time.

__init__(th=None)[source]
Parameters

th (threading.Thread or None) –

default_sess()[source]
run()[source]

Method representing the thread’s activity.

You may override this method in a subclass. The standard run() method invokes the callable object passed to the object’s constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively.

start()[source]

Start the thread’s activity.

It must be called at most once per thread object. It arranges for the object’s run() method to be invoked in a separate thread of control.

This method will raise a RuntimeError if called more than once on the same thread object.

tensorpack.utils.concurrency.ensure_proc_terminate(proc)[source]

Make sure processes terminate when main process exit.

Parameters

proc (multiprocessing.Process or list) –

tensorpack.utils.concurrency.start_proc_mask_signal(proc)[source]

Start process(es) with SIGINT ignored.

Parameters

proc – (mp.Process or list)

Note

The signal mask is only applied when called from main thread.

tensorpack.utils.fs module

tensorpack.utils.fs.mkdir_p(dirname)[source]

Like “mkdir -p”, make a dir recursively, but do nothing if the dir exists

Parameters

dirname (str) –

tensorpack.utils.fs.download(url, dir, filename=None, expect_size=None)[source]

Download URL to a directory. Will figure out the filename automatically from URL, if not given.

tensorpack.utils.fs.recursive_walk(rootdir)[source]
Yields

str – All files in rootdir, recursively.

tensorpack.utils.fs.get_dataset_path(*args)[source]

Get the path to some dataset under $TENSORPACK_DATASET.

Parameters

args – strings to be joined to form path.

Returns

str – path to the dataset.

tensorpack.utils.fs.normpath(path)[source]

Normalizes a path to a folder by taking into consideration remote storages like Cloud storaged referenced by ‘://’ at the beginning of the path.

Parameters

args – path to be normalized.

Returns

str – normalized path.

tensorpack.utils.loadcaffe module

tensorpack.utils.loadcaffe.load_caffe(model_desc, model_file)[source]

Load a caffe model. You must be able to import caffe to use this function.

Parameters
  • model_desc (str) – path to caffe model description file (.prototxt).

  • model_file (str) – path to caffe model parameter file (.caffemodel).

Returns

dict – the parameters.

tensorpack.utils.loadcaffe.get_caffe_pb()[source]

Get caffe protobuf.

Returns

The imported caffe protobuf module.

tensorpack.utils.logger module

The logger module itself has the common logging functions of Python’s logging.Logger. For example:

from tensorpack.utils import logger
logger.set_logger_dir('train_log/test')
logger.info("Test")
logger.error("Error happened!")
tensorpack.utils.logger.set_logger_dir(dirname, action=None)[source]

Set the directory for global logging.

Parameters
  • dirname (str) – log directory

  • action (str) –

    an action of [“k”,”d”,”q”] to be performed when the directory exists. Will ask user by default.

    ”d”: delete the directory. Note that the deletion may fail when the directory is used by tensorboard.

    ”k”: keep the directory. This is useful when you resume from a previous training and want the directory to look as if the training was not interrupted. Note that this option does not load old models or any other old states for you. It simply does nothing.

tensorpack.utils.logger.auto_set_dir(action=None, name=None)[source]

Use logger.set_logger_dir() to set log directory to “./train_log/{scriptname}:{name}”. “scriptname” is the name of the main python file currently running

tensorpack.utils.logger.get_logger_dir()[source]
Returns

The logger directory, or None if not set. The directory is used for general logging, tensorboard events, checkpoints, etc.

tensorpack.utils.logger.info(msg, *args, **kwargs)[source]

Log ‘msg % args’ with severity ‘INFO’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.info(“Houston, we have a %s”, “interesting problem”, exc_info=1)

tensorpack.utils.logger.warning(msg, *args, **kwargs)[source]

Log ‘msg % args’ with severity ‘WARNING’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.warning(“Houston, we have a %s”, “bit of a problem”, exc_info=1)

tensorpack.utils.logger.error(msg, *args, **kwargs)[source]

Log ‘msg % args’ with severity ‘ERROR’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.error(“Houston, we have a %s”, “major problem”, exc_info=1)

tensorpack.utils.logger.critical(msg, *args, **kwargs)[source]

Log ‘msg % args’ with severity ‘CRITICAL’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.critical(“Houston, we have a %s”, “major disaster”, exc_info=1)

tensorpack.utils.logger.exception(msg, *args, exc_info=True, **kwargs)[source]

Convenience method for logging an ERROR with exception information.

tensorpack.utils.logger.debug(msg, *args, **kwargs)[source]

Log ‘msg % args’ with severity ‘DEBUG’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.debug(“Houston, we have a %s”, “thorny problem”, exc_info=1)

tensorpack.utils.logger.setLevel(level)

Set the logging level of this logger. level must be an int or a str.

tensorpack.utils.logger.addFilter(filter)

Add the specified filter to this handler.

tensorpack.utils.logger.warn(msg, *args, **kwargs)[source]

Log ‘msg % args’ with severity ‘WARNING’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.warning(“Houston, we have a %s”, “bit of a problem”, exc_info=1)

tensorpack.utils.serialize module

tensorpack.utils.serialize.loads(buf)
Parameters

bytes

tensorpack.utils.serialize.dumps(obj)
Returns

bytes

tensorpack.utils.stats module

class tensorpack.utils.stats.StatCounter[source]

Bases: object

A simple counter

property average
property count
feed(v)[source]
Parameters

v (float or np.ndarray) – has to be the same shape between calls.

property max
property min
reset()[source]
samples()[source]

Returns all samples.

property sum
class tensorpack.utils.stats.BinaryStatistics[source]

Bases: object

Statistics for binary decision, including precision, recall, false positive, false negative

property false_negative
property false_positive
feed(pred, label)[source]
Parameters
  • pred (np.ndarray) – binary array.

  • label (np.ndarray) – binary array of the same size.

property precision
property recall
reset()[source]
class tensorpack.utils.stats.RatioCounter[source]

Bases: object

A counter to count ratio of something.

property count

Returns: int: the total

feed(count, total=1)[source]
Parameters
  • cnt (int) – the count of some event of interest.

  • tot (int) – the total number of events.

property ratio
reset()[source]
property total

Returns: int: the total

class tensorpack.utils.stats.Accuracy[source]

Bases: tensorpack.utils.stats.RatioCounter

A RatioCounter with a fancy name

property accuracy
class tensorpack.utils.stats.OnlineMoments[source]

Bases: object

Compute 1st and 2nd moments online (to avoid storing all elements).

See algorithm at: https://www.wikiwand.com/en/Algorithms_for_calculating_variance#/Online_algorithm

feed(x)[source]
Parameters

x (float or np.ndarray) – must have the same shape.

property mean
property std
property variance

tensorpack.utils.timer module

tensorpack.utils.timer.timed_operation(msg, log_start=False)[source]

Surround a context with a timer.

Parameters
  • msg (str) – the log to print.

  • log_start (bool) – whether to print also at the beginning.

Example

with timed_operation('Good Stuff'):
    time.sleep(1)

Will print:

Good stuff finished, time:1sec.
class tensorpack.utils.timer.Timer[source]

Bases: object

A timer class which computes the time elapsed since the start/reset of the timer.

is_paused()[source]
pause()[source]

Pause the timer.

reset()[source]

Reset the timer.

resume()[source]

Resume the timer.

seconds()[source]
Returns

float

the total number of seconds since the start/reset of the timer, excluding the

time in between when the timer is paused.

tensorpack.utils.viz module

tensorpack.utils.viz.interactive_imshow(img, lclick_cb=None, rclick_cb=None, **kwargs)[source]
Parameters
  • img (np.ndarray) – an image (expect BGR) to show.

  • rclick_cb (lclick_cb,) – a callback func(img, x, y) for left/right click event.

  • kwargs – can be {key_cb_a: callback_img, key_cb_b: callback_img}, to specify a callback func(img) for keypress.

Some existing keypress event handler:

  • q: destroy the current window

  • x: execute sys.exit()

  • s: save image to “out.png”

tensorpack.utils.viz.stack_patches(patch_list, nr_row, nr_col, border=None, pad=False, bgcolor=255, viz=False, lclick_cb=None)[source]

Stacked patches into grid, to produce visualizations like the following:

https://github.com/tensorpack/tensorpack/raw/master/examples/GAN/demo/BEGAN-CelebA-samples.jpg
Parameters
  • patch_list (list[ndarray] or ndarray) – NHW or NHWC images in [0,255].

  • nr_row (int), nr_col(int) – rows and cols of the grid. nr_col * nr_row must be no less than len(patch_list).

  • border (int) – border length between images. Defaults to 0.05 * min(patch_width, patch_height).

  • pad (boolean) – when patch_list is a list, pad all patches to the maximum height and width. This option allows stacking patches of different shapes together.

  • bgcolor (int or 3-tuple) – background color in [0, 255]. Either an int or a BGR tuple.

  • viz (bool) – whether to use interactive_imshow() to visualize the results.

  • lclick_cb – A callback function f(patch, patch index in patch_list) to get called when a patch get clicked in imshow.

Returns

np.ndarray – the stacked image.

tensorpack.utils.viz.gen_stack_patches(patch_list, nr_row=None, nr_col=None, border=None, max_width=1000, max_height=1000, bgcolor=255, viz=False, lclick_cb=None)[source]

Similar to stack_patches() but with a generator interface. It takes a much-longer list and yields stacked results one by one. For example, if patch_list contains 1000 images and nr_row==nr_col==10, this generator yields 10 stacked images.

Parameters
  • nr_row (int), nr_col(int) – rows and cols of each result.

  • max_width (int), max_height(int) – Maximum allowed size of the stacked image. If nr_row/nr_col are None, this number will be used to infer the rows and cols. Otherwise the option is ignored.

  • border, viz, lclick_cb (patch_list,) – same as in stack_patches().

Yields

np.ndarray – the stacked image.

tensorpack.utils.viz.dump_dataflow_images(df, index=0, batched=True, number=1000, output_dir=None, scale=1, resize=None, viz=None, flipRGB=False)[source]

Dump or visualize images of a DataFlow.

Parameters
  • df (DataFlow) – the DataFlow.

  • index (int) – the index of the image component.

  • batched (bool) – whether the component contains batched images (NHW or NHWC) or not (HW or HWC).

  • number (int) – how many datapoint to take from the DataFlow.

  • output_dir (str) – output directory to save images, default to not save.

  • scale (float) – scale the value, usually either 1 or 255.

  • resize (tuple or None) – tuple of (h, w) to resize the images to.

  • viz (tuple or None) – tuple of (h, w) determining the grid size to use with gen_stack_patches() for visualization. No visualization will happen by default.

  • flipRGB (bool) – apply a RGB<->BGR conversion or not.

tensorpack.utils.viz.intensity_to_rgb(intensity, cmap='cubehelix', normalize=False)[source]

Convert a 1-channel matrix of intensities to an RGB image employing a colormap. This function requires matplotlib. See matplotlib colormaps for a list of available colormap.

Parameters
  • intensity (np.ndarray) – array of intensities such as saliency.

  • cmap (str) – name of the colormap to use.

  • normalize (bool) – if True, will normalize the intensity so that it has minimum 0 and maximum 1.

Returns

np.ndarray – an RGB float32 image in range [0, 255], a colored heatmap.

tensorpack.utils.viz.draw_boxes(im, boxes, labels=None, color=None)[source]
Parameters
  • im (np.ndarray) – a BGR image in range [0,255]. It will not be modified.

  • boxes (np.ndarray) – a numpy array of shape Nx4 where each row is [x1, y1, x2, y2].

  • labels – (list[str] or None)

  • color – a 3-tuple BGR color (in range [0, 255])

Returns

np.ndarray – a new image.

tensorpack.utils.gpu module

tensorpack.utils.gpu.change_gpu(val)[source]
Parameters

val – an integer, the index of the GPU or -1 to disable GPU.

Returns

a context where CUDA_VISIBLE_DEVICES=val.

tensorpack.utils.gpu.get_num_gpu()[source]
Returns

int – #available GPUs in CUDA_VISIBLE_DEVICES, or in the system.