tensorpack.utils package¶

Common utils. These utils should be irrelevant to tensorflow.

tensorpack.utils.change_env(name, val)[source]¶

Parameters

name (str) – name of the env var
val (str or None) – the value, or set to None to clear the env var.

Returns

a context where the environment variable name being set to val. It will be set back after the context exits.

tensorpack.utils.get_rng(obj=None)[source]¶

Get a good RNG seeded with time, pid and the object.

Parameters: obj – some object to use to generate random seed.
Returns: np.random.RandomState – the RNG.

tensorpack.utils.fix_rng_seed(seed)[source]¶

Call this function at the beginning of program to fix rng seed within tensorpack.

Parameters: seed (int) –

Note

See https://github.com/tensorpack/tensorpack/issues/196.

Example

Fix random seed in both tensorpack and tensorflow.

seed = 42
utils.fix_rng_seed(seed)
tesnorflow.set_random_seed(seed)
# run trainer

tensorpack.utils.get_tqdm(*args, **kwargs)[source]¶: Similar to tqdm.tqdm(), but use tensorpack’s default options to have consistent style.

tensorpack.utils.argtools module¶

tensorpack.utils.argtools.map_arg(**maps)[source]¶

Apply a mapping on certain argument before calling the original function.

Parameters: maps (dict) – {argument_name: map_func}

tensorpack.utils.argtools.memoized(user_function)¶: Alias to functools.lru_cache() WARNING: memoization will keep keys and values alive!

tensorpack.utils.argtools.memoized_method(func)[source]¶: A decorator that performs memoization on methods. It stores the cache on the object instance itself.

tensorpack.utils.argtools.graph_memoized(func)[source]¶: Like memoized, but keep one cache per default graph.

tensorpack.utils.argtools.shape2d(a)[source]¶

Ensure a 2D shape.

Parameters: a – a int or tuple/list of length 2
Returns: list – of length 2. if a is a int, return [a, a].

tensorpack.utils.argtools.shape4d(a, data_format='NHWC')[source]¶

Ensuer a 4D shape, to use with 4D symbolic functions.

Parameters

a – a int or tuple/list of length 2

Returns

list –

of length 4. if a is a int, return [1, a, a, 1]: or [1, 1, a, a] depending on data_format.

tensorpack.utils.argtools.memoized_ignoreargs(func)[source]¶: A decorator. It performs memoization ignoring the arguments used to call the function.

tensorpack.utils.argtools.log_once(message, func='info')[source]¶

Log certain message only once. Call this function more than one times with the same message will result in no-op.

Parameters

message (str) – message to log
func (str) – the name of the logger method. e.g. “info”, “warn”, “error”.

tensorpack.utils.concurrency module¶

class tensorpack.utils.concurrency.StoppableThread(evt=None)[source]¶

Bases: threading.Thread

A thread that has a ‘stop’ event.

__init__(evt=None)[source]¶

Parameters: evt (threading.Event) – if None, will create one.

queue_get_stoppable(q)[source]¶: Take obj from queue, but will give up when the thread is stopped

queue_put_stoppable(q, obj)[source]¶: Put obj to queue, but will give up when the thread is stopped

stop()[source]¶: Stop the thread

stopped()[source]¶

Returns: bool – whether the thread is stopped or not

class tensorpack.utils.concurrency.LoopThread(func, pausable=True)[source]¶

Bases: tensorpack.utils.concurrency.StoppableThread

A pausable thread that simply runs a loop

__init__(func, pausable=True)[source]¶

Parameters: func – the function to run

pause()[source]¶: Pause the loop

resume()[source]¶: Resume the loop

run()[source]¶

Method representing the thread’s activity.

You may override this method in a subclass. The standard run() method invokes the callable object passed to the object’s constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively.

class tensorpack.utils.concurrency.ShareSessionThread(th=None)[source]¶

Bases: threading.Thread

A wrapper around thread so that the thread uses the default session at “start()” time.

__init__(th=None)[source]¶

Parameters: th (threading.Thread or None) –

default_sess()[source]¶

run()[source]¶

Method representing the thread’s activity.

You may override this method in a subclass. The standard run() method invokes the callable object passed to the object’s constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively.

start()[source]¶

Start the thread’s activity.

It must be called at most once per thread object. It arranges for the object’s run() method to be invoked in a separate thread of control.

This method will raise a RuntimeError if called more than once on the same thread object.

tensorpack.utils.concurrency.ensure_proc_terminate(proc)[source]¶

Make sure processes terminate when main process exit.

Parameters: proc (multiprocessing.Process or list) –

tensorpack.utils.concurrency.start_proc_mask_signal(proc)[source]¶

Start process(es) with SIGINT ignored.

Parameters: proc – (mp.Process or list)

Note

The signal mask is only applied when called from main thread.

tensorpack.utils.fs module¶

tensorpack.utils.fs.mkdir_p(dirname)[source]¶

Like “mkdir -p”, make a dir recursively, but do nothing if the dir exists

Parameters: dirname (str) –

tensorpack.utils.fs.download(url, dir, filename=None, expect_size=None)[source]¶: Download URL to a directory. Will figure out the filename automatically from URL, if not given.

tensorpack.utils.fs.recursive_walk(rootdir)[source]¶

Yields: str – All files in rootdir, recursively.

tensorpack.utils.fs.get_dataset_path(*args)[source]¶

Get the path to some dataset under $TENSORPACK_DATASET.

Parameters: args – strings to be joined to form path.
Returns: str – path to the dataset.

tensorpack.utils.fs.normpath(path)[source]¶

Normalizes a path to a folder by taking into consideration remote storages like Cloud storaged referenced by ‘://’ at the beginning of the path.

Parameters: args – path to be normalized.
Returns: str – normalized path.

tensorpack.utils.loadcaffe module¶

tensorpack.utils.loadcaffe.load_caffe(model_desc, model_file)[source]¶

Load a caffe model. You must be able to import caffe to use this function.

Parameters

model_desc (str) – path to caffe model description file (.prototxt).
model_file (str) – path to caffe model parameter file (.caffemodel).

Returns

dict – the parameters.

tensorpack.utils.loadcaffe.get_caffe_pb()[source]¶

Get caffe protobuf.

Returns: The imported caffe protobuf module.

tensorpack.utils.logger module¶

The logger module itself has the common logging functions of Python’s logging.Logger. For example:

from tensorpack.utils import logger
logger.set_logger_dir('train_log/test')
logger.info("Test")
logger.error("Error happened!")

tensorpack.utils.logger.set_logger_dir(dirname, action=None)[source]¶

Set the directory for global logging.

Parameters

dirname (str) – log directory
action (str) –
an action of [“k”,”d”,”q”] to be performed when the directory exists. Will ask user by default.

”d”: delete the directory. Note that the deletion may fail when the directory is used by tensorboard.

”k”: keep the directory. This is useful when you resume from a previous training and want the directory to look as if the training was not interrupted. Note that this option does not load old models or any other old states for you. It simply does nothing.

tensorpack.utils.logger.auto_set_dir(action=None, name=None)[source]¶: Use logger.set_logger_dir() to set log directory to “./train_log/{scriptname}:{name}”. “scriptname” is the name of the main python file currently running

tensorpack.utils.logger.get_logger_dir()[source]¶

Returns: The logger directory, or None if not set. The directory is used for general logging, tensorboard events, checkpoints, etc.

tensorpack.utils.logger.info(msg, *args, **kwargs)[source]¶

Log ‘msg % args’ with severity ‘INFO’.