tensorflow data reading (2)

A typical pipeline for reading records from files has the following stages:

1. The list of filenames
2. Optional filename shuffling
3. Optional epoch limit
4. Filename queue
5. A Reader for the file format
6. A decoder for a record read by the reader
7. Optional preprocessing
8. Example queue

Filenames, shuffling, and epoch limits

For the list of filenames, use either a constant string Tensor (like["file0", "file1"] or[("file%d" % i) for i in range(2)]) or thetf.train.match_filenames_once    function.

Pass the list of filenames to the tf.train.string_input_producerfunction.string_input_producer creates a FIFO queue for holding the filenames untilthe reader needs them.

string_input_producer has options for shuffling and setting a maximum numberof epochs. A queue runner adds the whole list of filenames to the queue oncefor each epoch, shuffling the filenames within an epoch ifshuffle=True.This procedure provides a uniform sampling of files, so that examples are notunder- or over- sampled relative to each other.

The queue runner works in a thread separate from the reader that pullsfilenames from the queue, so the shuffling and enqueuing process does notblock the reader.