tensorpack.dataflow.dataset package

class tensorpack.dataflow.dataset.BSDS500(name, data_dir=None, shuffle=True)[source]

Bases: tensorpack.dataflow.base.RNGDataFlow

Berkeley Segmentation Data Set and Benchmarks 500 dataset.

Produce (image, label) pair, where image has shape (321, 481, 3(BGR)) and ranges in [0,255]. Label is a floating point image of shape (321, 481) in range [0, 1]. The value of each pixel is number of times it is annotated as edge / total number of annotators for this image.

__init__(name, data_dir=None, shuffle=True)[source]
Parameters:
  • name (str) – ‘train’, ‘test’, ‘val’

  • data_dir (str) – a directory containing the original ‘BSR’ directory.

class tensorpack.dataflow.dataset.Cifar10(train_or_test, shuffle=None, dir=None)[source]

Bases: tensorpack.dataflow.dataset.cifar.CifarBase

Produces [image, label] in Cifar10 dataset, image is 32x32x3 in the range [0,255]. label is an int.

__init__(train_or_test, shuffle=None, dir=None)[source]
Parameters:
  • train_or_test (str) – either ‘train’ or ‘test’.

  • shuffle (bool) – shuffle the dataset, default to shuffle in training

class tensorpack.dataflow.dataset.Cifar100(train_or_test, shuffle=None, dir=None)[source]

Bases: tensorpack.dataflow.dataset.cifar.CifarBase

Similar to Cifar10

class tensorpack.dataflow.dataset.ILSVRCMeta(dir=None)[source]

Bases: object

Provide methods to access metadata for ILSVRC dataset.

get_image_list(name, dir_structure='original')[source]
Parameters:
Returns:

list – list of (image filename, label)

get_per_pixel_mean(size=None)[source]
Parameters:size (tuple) – image size in (h, w). Defaults to (256, 256).
Returns:np.ndarray – per-pixel mean of shape (h, w, 3 (BGR)) in range [0, 255].
get_synset_1000()[source]
Returns:dict – {cls_number: synset_id}
get_synset_words_1000()[source]
Returns:dict – {cls_number: cls_name}
static guess_dir_structure(dir)[source]

Return the directory structure of “dir”.

Parameters:dir (str) – something like ‘/path/to/imagenet/val’
Returns:either ‘train’ or ‘original’
class tensorpack.dataflow.dataset.ILSVRC12(dir, name, meta_dir=None, shuffle=None, dir_structure=None)[source]

Bases: tensorpack.dataflow.dataset.ilsvrc.ILSVRC12Files

Produces uint8 ILSVRC12 images of shape [h, w, 3(BGR)], and a label between [0, 999].

__init__(dir, name, meta_dir=None, shuffle=None, dir_structure=None)[source]
Parameters:
  • dir (str) – A directory containing a subdir named name, containing the images in a structure described below.

  • name (str) – One of ‘train’ or ‘val’ or ‘test’.

  • shuffle (bool) – shuffle the dataset. Defaults to True if name==’train’.

  • dir_structure (str) – One of ‘original’ or ‘train’. The directory structure for the ‘val’ directory. ‘original’ means the original decompressed directory, which only has list of image files (as below). If set to ‘train’, it expects the same two-level directory structure similar to ‘dir/train/’. By default, it tries to automatically detect the structure. You probably do not need to care about this option because ‘original’ is what people usually have.

Example:

When dir_structure==’original’, dir should have the following structure:

dir/
  train/
    n02134418/
      n02134418_198.JPEG
      ...
    ...
  val/
    ILSVRC2012_val_00000001.JPEG
    ...
  test/
    ILSVRC2012_test_00000001.JPEG
    ...

With the downloaded ILSVRC12_img_*.tar, you can use the following command to build the above structure:

mkdir val && tar xvf ILSVRC12_img_val.tar -C val
mkdir test && tar xvf ILSVRC12_img_test.tar -C test
mkdir train && tar xvf ILSVRC12_img_train.tar -C train && cd train
find -type f -name '*.tar' | parallel -P 10 'echo {} && mkdir -p {/.} && tar xf {} -C {/.}'

When dir_structure==’train’, dir should have the following structure:

dir/
  train/
    n02134418/
      n02134418_198.JPEG
      ...
    ...
  val/
    n01440764/
      ILSVRC2012_val_00000293.JPEG
      ...
    ...
  test/
    ILSVRC2012_test_00000001.JPEG
    ...
static get_training_bbox(bbox_dir, imglist)[source]
class tensorpack.dataflow.dataset.ILSVRC12Files(dir, name, meta_dir=None, shuffle=None, dir_structure=None)[source]

Bases: tensorpack.dataflow.base.RNGDataFlow

Same as ILSVRC12, but produces filenames of the images instead of nparrays. This could be useful when cv2.imread is a bottleneck and you want to decode it in smarter ways (e.g. in parallel).

__init__(dir, name, meta_dir=None, shuffle=None, dir_structure=None)[source]

Same as in ILSVRC12.

class tensorpack.dataflow.dataset.Mnist(train_or_test, shuffle=True, dir=None)[source]

Bases: tensorpack.dataflow.base.RNGDataFlow

Produces [image, label] in MNIST dataset, image is 28x28 in the range [0,1], label is an int.

__init__(train_or_test, shuffle=True, dir=None)[source]
Parameters:
  • train_or_test (str) – either ‘train’ or ‘test’

  • shuffle (bool) – shuffle the dataset

class tensorpack.dataflow.dataset.FashionMnist(train_or_test, shuffle=True, dir=None)[source]

Bases: tensorpack.dataflow.dataset.mnist.Mnist

Same API as Mnist, but more fashion.

get_label_names()[source]
Returns:[str] – the name of each class
class tensorpack.dataflow.dataset.SVHNDigit(name, data_dir=None, shuffle=True)[source]

Bases: tensorpack.dataflow.base.RNGDataFlow

SVHN Cropped Digit Dataset. Produces [img, label], img of 32x32x3 in range [0,255], label of 0-9

__init__(name, data_dir=None, shuffle=True)[source]
Parameters:
  • name (str) – ‘train’, ‘test’, or ‘extra’.

  • data_dir (str) – a directory containing the original {train,test,extra}_32x32.mat.

  • shuffle (bool) – shuffle the dataset.

static get_per_pixel_mean()[source]
Returns:a 32x32x3 image