tensorpack.utils package

tensorpack.utils.argtools module

tensorpack.utils.argtools.map_arg(**maps)[source]

Apply a mapping on certains argument before calling the original function.

Parameters:maps (dict) – {key: map_func}
tensorpack.utils.argtools.memoized = <Mock name='functools32.lru_cache()' id='140624094749904'>

Alias to functools.lru_cache()

tensorpack.utils.argtools.shape2d(a)[source]

Ensure a 2D shape.

Parameters:a – a int or tuple/list of length 2
Returns:list – of length 2. if a is a int, return [a, a].
tensorpack.utils.argtools.shape4d(a, data_format='NHWC')[source]

Ensuer a 4D shape, to use with 4D symbolic functions.

Parameters:a – a int or tuple/list of length 2
Returns:list
of length 4. if a is a int, return [1, a, a, 1]
or [1, 1, a, a] depending on data_format.
tensorpack.utils.argtools.memoized_ignoreargs(func)[source]

A decorator. It performs memoization ignoring the arguments used to call the function.

tensorpack.utils.concurrency module

class tensorpack.utils.concurrency.StoppableThread[source]

Bases: threading.Thread

A thread that has a ‘stop’ event.

queue_get_stoppable(q)[source]

Take obj from queue, but will give up when the thread is stopped

queue_put_stoppable(q, obj)[source]

Put obj to queue, but will give up when the thread is stopped

stop()[source]

Stop the thread

stopped()[source]
Returns:bool – whether the thread is stopped or not
class tensorpack.utils.concurrency.LoopThread(func, pausable=True)[source]

Bases: tensorpack.utils.concurrency.StoppableThread

A pausable thread that simply runs a loop

__init__(func, pausable=True)[source]
Parameters:func – the function to run
pause()[source]

Pause the loop

resume()[source]

Resume the loop

run()[source]
class tensorpack.utils.concurrency.ShareSessionThread(th=None)[source]

Bases: threading.Thread

A wrapper around thread so that the thread uses the default session at “start()” time.

__init__(th=None)[source]
Parameters:th (threading.Thread or None) –
default_sess(*args, **kwds)[source]
run()[source]
start()[source]
tensorpack.utils.concurrency.ensure_proc_terminate(proc)[source]

Make sure processes terminate when main process exit.

Parameters:proc (multiprocessing.Process or list) –
class tensorpack.utils.concurrency.OrderedResultGatherProc(data_queue, nr_producer, start=0)[source]

Bases: multiprocessing.process.Process

Gather indexed data from a data queue, and produce results with the original index-based order.

__init__(data_queue, nr_producer, start=0)[source]
Parameters:
  • data_queue (multiprocessing.Queue) – a queue which contains datapoints.

  • nr_producer (int) – number of producer processes. This process will terminate after receiving this many of DIE sentinel.

  • start (int) – the rank of the first object

get()[source]
run()[source]
class tensorpack.utils.concurrency.OrderedContainer(start=0)[source]

Bases: object

Like a queue, but will always wait to receive item with rank (x+1) and produce (x+1) before producing (x+2).

Warning

It is not thread-safe.

__init__(start=0)[source]
Parameters:start (int) – the starting rank.
get()[source]
has_next()[source]
put(rank, val)[source]
Parameters:
  • rank (int) – rank of th element. All elements must have different ranks.

  • val – an object

class tensorpack.utils.concurrency.DIE[source]

Bases: object

A placeholder class indicating end of queue

tensorpack.utils.concurrency.mask_sigint(*args, **kwds)[source]
Returns:a context where SIGINT is ignored.
tensorpack.utils.concurrency.start_proc_mask_signal(proc)[source]

Start process(es) with SIGINT ignored.

Parameters:proc – (multiprocessing.Process or list)

tensorpack.utils.fs module

tensorpack.utils.fs.mkdir_p(dirname)[source]

Make a dir recursively, but do nothing if the dir exists

Parameters:dirname (str) –
tensorpack.utils.fs.download(url, dir, filename=None)[source]

Download URL to a directory. Will figure out the filename automatically from URL, if not given.

tensorpack.utils.fs.recursive_walk(rootdir)[source]
Yields:str – All files in rootdir, recursively.
tensorpack.utils.fs.get_dataset_path(*args)[source]

Get the path to some dataset under $TENSORPACK_DATASET.

Parameters:args – strings to be joined to form path.
Returns:str – path to the dataset.

tensorpack.utils.globvars module

tensorpack.utils.globvars.use_global_argument(args)[source]

Add the content of argparse.Namespace to globalns.

Parameters:args (argparse.Namespace) – arguments

tensorpack.utils.loadcaffe module

tensorpack.utils.loadcaffe.load_caffe(model_desc, model_file)[source]

Load a caffe model. You must be able to import caffe to use this function. :param model_desc: path to caffe model description file (.prototxt). :type model_desc: str :param model_file: path to caffe model parameter file (.caffemodel). :type model_file: str

Returns:dict – the parameters.
tensorpack.utils.loadcaffe.get_caffe_pb()[source]

Get caffe protobuf. :returns: The imported caffe protobuf module.

tensorpack.utils.logger module

tensorpack.utils.logger.set_logger_dir(dirname, action=None)[source]

Set the directory for global logging.

Parameters:
  • dirname (str) – log directory

  • action (str) – an action of (“k”,”b”,”d”,”n”,”q”) to be performed. Will ask user by default.

tensorpack.utils.logger.disable_logger()[source]

Disable all logging ability from this moment

tensorpack.utils.logger.auto_set_dir(action=None, name=None)[source]

Use logger.set_logger_dir() to set log directory to ”./train_log/{scriptname}:{name}”. “scriptname” is the name of the main python file currently running

tensorpack.utils.rect module

class tensorpack.utils.rect.Rect(x=0, y=0, w=0, h=0, allow_neg=False)[source]

Bases: object

A rectangle class.

Note that x1 = x + w, not x+w-1 or something else.

area()[source]
copy()[source]
expand(frac)[source]
h
roi(img)[source]
roi_zeropad(img)[source]
validate(shape=None)[source]

Check that this rect is a valid bounding box within this shape. :param shape: [h, w]

Returns:bool
w
x
x0
x1
y
y0
y1

tensorpack.utils.serialize module

tensorpack.utils.serialize.loads(buf)[source]
Parameters:buf (str) – serialized object.
tensorpack.utils.serialize.dumps(obj)[source]

Serialize an object.

Returns:str
tensorpack.utils.serialize.dumps_for_tfop(dp)[source]
tensorpack.utils.serialize.dump_tensor_protos(protos)[source]

Serialize a list of TensorProto, for communication between custom TensorFlow ops.

Parameters:protos (list) – list of TensorProto instance

Notes

The format is:

[#tensors(int32)] (tensor1)[size of meta proto][serialized meta proto][size of buffer][buffer] (tensor2)...

tensorpack.utils.serialize.to_tensor_proto(arr)[source]

Convert a numpy array to TensorProto

Parameters:arr – numpy.ndarray. only supports common numerical types

tensorpack.utils.stats module

class tensorpack.utils.stats.StatCounter[source]

Bases: object

A simple counter

average
count
feed(v)[source]
Parameters:v (float or np.ndarray) – has to be the same shape between calls.
max
min
reset()[source]
sum
class tensorpack.utils.stats.BinaryStatistics[source]

Bases: object

Statistics for binary decision, including precision, recall, false positive, false negative

false_negative
false_positive
feed(pred, label)[source]
Parameters:
  • pred (np.ndarray) – binary array.

  • label (np.ndarray) – binary array of the same size.

precision
recall
reset()[source]
class tensorpack.utils.stats.RatioCounter[source]

Bases: object

A counter to count ratio of something.

count

Returns – int: the total

feed(cnt, tot=1)[source]
Parameters:
  • cnt (int) – the count of some event of interest.

  • tot (int) – the total number of events.

ratio
reset()[source]
class tensorpack.utils.stats.Accuracy[source]

Bases: tensorpack.utils.stats.RatioCounter

A RatioCounter with a fancy name

accuracy
class tensorpack.utils.stats.OnlineMoments[source]

Bases: object

Compute 1st and 2nd moments online (to avoid storing all elements).

See algorithm at: https://www.wikiwand.com/en/Algorithms_for_calculating_variance#/Online_algorithm

feed(x)[source]
Parameters:x (float or np.ndarray) – must have the same shape.
mean
std
variance

tensorpack.utils.timer module

tensorpack.utils.timer.total_timer(*args, **kwds)[source]

A context which add the time spent inside to TotalTimer.

tensorpack.utils.timer.timed_operation(*args, **kwds)[source]

Surround a context with a timer.

Parameters:
  • msg (str) – the log to print.

  • log_start (bool) – whether to print also at the beginning.

Example

with timed_operation('Good Stuff'):
    time.sleep(1)

Will print:

Good stuff finished, time:1sec.
tensorpack.utils.timer.print_total_timer()[source]

Print the content of the TotalTimer, if it’s not empty. This function will automatically get called when program exits.

class tensorpack.utils.timer.IterSpeedCounter(print_every, name=None)[source]

Bases: object

Test how often some code gets reached.

Example

Print the speed of the iteration every 100 times.

speed = IterSpeedCounter(100)
for k in range(1000):
    # do something
    speed()
__init__(print_every, name=None)[source]
Parameters:
  • print_every (int) – interval to print.

  • name (str) – name to used when print.

reset()[source]

tensorpack.utils.viz module

tensorpack.utils.viz.pyplot2img(*args, **kwargs)
tensorpack.utils.viz.interactive_imshow(img, lclick_cb=None, rclick_cb=None, **kwargs)[source]
Parameters:
  • img (np.ndarray) – an image (expect BGR) to show.

  • rclick_cb (lclick_cb,) – a callback func(img, x, y) for left/right click event.

  • kwargs – can be {key_cb_a: callback_img, key_cb_b: callback_img}, to specify a callback func(img) for keypress.

Some existing keypress event handler:

  • q: destroy the current window

  • x: execute sys.exit()

  • s: save image to “out.png”

tensorpack.utils.viz.stack_patches(patch_list, nr_row, nr_col, border=None, pad=False, bgcolor=255, viz=False, lclick_cb=None)[source]

Stacked patches into grid, to produce visualizations like the following:

https://github.com/ppwwyyxx/tensorpack/raw/master/examples/GAN/demo/CelebA-samples.jpg
Parameters:
  • patch_list (list[ndarray] or ndarray) – NHW or NHWC images in [0,255].

  • nr_row (int), nr_col(int) – rows and cols of the grid. nr_col * nr_row must be equal to len(patch_list).

  • border (int) – border length between images. Defaults to 0.1 * min(patch_width, patch_height).

  • pad (boolean) – when patch_list is a list, pad all patches to the maximum height and width. This option allows stacking patches of different shapes together.

  • bgcolor (int or 3-tuple) – background color in [0, 255]. Either an int or a BGR tuple.

  • viz (bool) – whether to use interactive_imshow() to visualize the results.

  • lclick_cb – A callback function f(patch, patch index in patch_list) to get called when a patch get clicked in imshow.

Returns:

np.ndarray – the stacked image.

tensorpack.utils.viz.gen_stack_patches(patch_list, nr_row=None, nr_col=None, border=None, max_width=1000, max_height=1000, bgcolor=255, viz=False, lclick_cb=None)[source]

Similar to stack_patches() but with a generator interface. It takes a much-longer list and yields stacked results one by one. For example, if patch_list contains 1000 images and nr_row==nr_col==10, this generator yields 10 stacked images.

Parameters:
  • nr_row (int), nr_col(int) – rows and cols of each result.

  • max_width (int), max_height(int) – Maximum allowed size of the stacked image. If nr_row/nr_col are None, this number will be used to infer the rows and cols. Otherwise the option is ignored.

  • border, viz, lclick_cb (patch_list,) – same as in stack_patches().

Yields:

np.ndarray – the stacked image.

tensorpack.utils.viz.dump_dataflow_images(df, index=0, batched=True, number=1000, output_dir=None, scale=1, resize=None, viz=None, flipRGB=False)[source]

Dump or visualize images of a DataFlow.

Parameters:
  • df (DataFlow) – the DataFlow.

  • index (int) – the index of the image component.

  • batched (bool) – whether the component contains batched images (NHW or NHWC) or not (HW or HWC).

  • number (int) – how many datapoint to take from the DataFlow.

  • output_dir (str) – output directory to save images, default to not save.

  • scale (float) – scale the value, usually either 1 or 255.

  • resize (tuple or None) – tuple of (h, w) to resize the images to.

  • viz (tuple or None) – tuple of (h, w) determining the grid size to use with gen_stack_patches() for visualization. No visualization will happen by default.

  • flipRGB (bool) – apply a RGB<->BGR conversion or not.

tensorpack.utils.viz.intensity_to_rgb(*args, **kwargs)

Module contents

tensorpack.utils.change_gpu(val)[source]
Returns:a context where CUDA_VISIBLE_DEVICES=val.
tensorpack.utils.get_nr_gpu()[source]
Returns:int – #available GPUs in CUDA_VISIBLE_DEVICES, or in the system.
tensorpack.utils.change_env(*args, **kwds)[source]
Parameters:name (str), val(str) –
Returns:a context where the environment variable name being set to val. It will be set back after the context exits.
tensorpack.utils.get_rng(obj=None)[source]

Get a good RNG seeded with time, pid and the object.

Parameters:obj – some object to use to generate random seed.
Returns:np.random.RandomState – the RNG.
tensorpack.utils.fix_rng_seed(seed)[source]

Call this function at the beginning of program to fix rng seed within tensorpack.

Parameters:seed (int) –
tensorpack.utils.get_tqdm_kwargs(**kwargs)[source]

Return default arguments to be used with tqdm.

Parameters:kwargs – extra arguments to be used.
Returns:dict
tensorpack.utils.get_tqdm(**kwargs)[source]

Similar to get_tqdm_kwargs(), but returns the tqdm object directly.

tensorpack.utils.execute_only_once()[source]

Each called in the code to this function is guranteed to return True the first time and False afterwards.

Returns:bool – whether this is the first time this function gets called from this line of code.

Example

if execute_only_once():
    # do something only once