1. Debugging Dataflow Graphs using TensorFlow Debugger

2. TensorFlow programs are usually structured into  a construction phase, that assembles a graph, and  an execution phase that uses a session to execute ops in the graph.  The actual computation is done inside session.run();

3. Architecture of the TensorFlow Debugger  Tfdbg consists of three main components, namely the Analyzer, the NodeStepper and the RunStepper.  The Analyzer adds observability to the graph execution process.  while the NodeStepper and RunStepper focus on the controllability of TensorFlow code.

4. Debugging in TensorFlow: Overview Basic ways: Explicitly fetch, and print (or do whatever you want)! Session.run() Tensorboard: Histogram and Image Summary the tf.Print() operation Advanced ways: Interpose any python codelet in the computation graph A step-by-step debugger tfdbg: The TensorFlow debugger

5. Basic ways: through command line

6. Command line interface tfdbg:  Prints tensors list in the network,list of inputs and outputs of the selected tensors.  We can get node info,ops of tensor(type of tensor like float,constant etc)  The most basic method to get debugging information.  We can fetch any evaluation result in numpy arrays.

7. tfdbg: Tensor Fetching Commands: list_tensors (lt) : Show the list of dumped tensor(s). print_tensor (pt) : Print the value of a dumped tensor. node_info (ni) : Show information about a node ni -t : Shows the traceback of tensor creation list_inputs (li) : Show inputs to a node list_outputs (lo) : Show outputs to a node run_info (ri) : Show the information of current run invoke_stepper (s) : Invoke the stepper! run (r) : Move to the next run

8. Advanced ways: through tensor board debbuger

9. Debugging Scenarios: inspect intra-layer activations (during training and inference ) e.g. See the output of conv3 in CNNs inspect parameter weights (during training) under some conditions, pause the execution (i.e. breakpoint) and evaluate some expressions for debugging during training, NaN occurs in loss and variables

10. Debugging Scenarios: inspect intra-layer activations for layer 0 both features in image and curve(high dimensional data reduced into graph)

11. Debugging Scenarios: inspect intra-layer activations for layer 1 both features in image and curve(high dimensional data reduced into graph)

12. Debugging Scenarios: inspect intra-layer activations for layer 2 both features in image and curve(high dimensional data reduced into graph)

13. Debugging Scenarios: inspect intra-layer activations for layer 3 both features in image and curve(high dimensional data reduced into graph)

14. Debugging Scenarios: inspect intra-layer activations for layer 4 both features in image and curve(high dimensional data reduced into graph)

15. Debugging Scenarios: inspect intra-layer activations for layer 5 both features in image and curve(high dimensional data reduced into graph)

16. Debugging Solution: If we inspect layer 4 and layer 5 the activations of the layers, number of steps performed in the graph are reduced, (not sure whether the layers are frozen while transfer learning or we have to optimize ) the first couple convolutional layers will learn filters for finding lines, dots, curves etc. while the later layers will learn to recognize common objects and shapes.(have to go through all 28 layers visualization for specific feature extractions) the first couple convolutional layers will learn filters for finding lines, dots, curves etc. while the later layers will learn to recognize common objects and shapes.

17. Debugging Scenarios: inspect intra-layer activations (output FC layer both features in image and curve(high dimensional data reduced into graph) )

18. Debugging: Summary Session.run(): Explicitly fetch, and print Tensorboard:generates Histogram and Image Summary Interpose your debugging python code in the graph We can see flow of the tensors in real time We can pause and perform the diagnostics to the net. Tensor board is flashlight to neural net Tensor flow debugger is x-ray to neural net