1. Tensorflow Tutorial Presented By :- Ankur Mali
The Pennsylvania State University
IST 597: Foundations of Deep Learning
Fall 2018

2. Widely used Deep Learning frameworks
Low Level frameworks
Tensorflow
Pytorch
Lua
Theano
Caffee
MxNet
High Level Frameworks
Keras
TFlearn

3. What is Tensorflow?
Open source software library for numerical computation using data flow graphs.
Good for training and deploying deep learning models.
Widely popular due to its flexibility and scalability.

4. Graphs and Session Data Flow Graphs
TensorFlow separates definition of computations from their execution.
Step 1 :- Build the graph
Step 2 :- use session to execute operations in the created graph(Nodes: operators, variables, and constants , Edges: tensors)

5. Tensor? Tensor can be anything from scalar to n - dimensional array
0-d (scalar or number)
1-d (vector)
2-d (matrix) ...

6. Basic Operations Import tensorflow as tf # var1 = tf.constant(2.0)
Var1 = tf.constant(2.0,name=’var1’)
Var2 = tf.constant([3.0,4.5,6.0],name=’var2’)
Result = tf.multiply(var1,var2)
#broadcasting is similar to numpy.broadcast
tf.constant(
value,
dtype=None,
shape=None,
name='Const_var',
verify_shape=False )
tf.zeros is similar to np.zeros
tf.zeros_like is similar to np.zeros_like
tf.ones is similar to np.ones
tf.ones_like is similar to np.ones_like
tf.fill is similar to tf.full

7. Can we use Constants as Sequences?
Tensor objects are not iterable , therefore are not similar to numpy sequences
If we do
For val in tf.range(4):
# we would get TypeError
Other types of randomly generated of constants
tf.random_normal
tf.truncated_normal
tf.random_uniform
tf.random_shuffle
tf.random_crop
tf.multinomial
tf.random_gamma

8. Why don’t we use numpy datatype instead of Tf?
Tensorflow does integrates seamlessly with numpy
tf.float32 ⇒ np.float32
var1 = tf.ones([10,2],np.float32) #requested fetch type is tensor and output is n-dimensional numpy array
What’s wrong in getting output as numpy ndarray?

9. When to use constants?
Only use for primitive types.
Why?

10. How to create variables in tensorflow
With tf.Variable
Var1 = tf.Variable(tf.zeros(784,10),name=’matrix’)
#class with many Ops
With tf.get_variable
Var2 = tf.get_variable(“matrix”,shape=(784,10),initializer=tf.zeros_initializer())
Initialize your variables within the context of the session
Use tf.global_variables_initializer() #all variables Or
Initialize specific variables using tf.variables_initializer([var1,var2,..])

11. First Tensorflow Program
>>> import tensorflow as tf
>>> a = tf.constant(3)
>>> b = tf.constant(5)
>>> c = tf.multiply(a,b)
>>> print(c)
Tensor("Mul:0", shape=(), dtype=int32)

12. How to get value for c? Create a session Sess = tf.Session()
Within the session , evaluate the graph to fetch the value of c.
print(sess.run(c))
Close the session.
sess.close()
Alternate approach
with tf.Session() as sess:
print(sess.run(c))

13. How to visualize Data Flow Graph in Tensorflow?
Tensorboard
In above example add tf.summary()
writer = tf.summary.FileWriter("output_folder", sess.graph)
With tf.Session() as sess:
print(sess.run(c))
writer.close()
with tf.Session() as sess:
writer = tf.summary.FileWriter("output_folder", sess.graph)
print(sess.run(c))
writer.close()
Open another terminal
tensorboard --logdir=current_dir/output_folder/

14. Placeholders(user input?)
tf.placeholder(dtype, shape = None , name = None)
var1 = tf.placeholder(tf.float32 , shape=[3])
var2 = tf.placeholder([2.3,4.7],tf.float32)
Result = tf.add(var1,var2)
with tf.Session() as sess:
print(sess.run(Result)) ⇒ Output?

15. Placeholders tf.placeholder(dtype, shape = None , name = None)
var1 = tf.placeholder(tf.float32 , shape=[3])
var2 = tf.placeholder([2.3,4.7],tf.float32)
Result = tf.add(var1,var2)
with tf.Session() as sess:
print(sess.run(Result,feed_dict={var1:[0.7,1.3]} ) )
Avoid lazy Loading

16. Control Dependencies
Important when you need to execute operations sequentially or data is dependent on previous input
tf.Graph.control_dependencies(inputs)
Control_graph = tf.get_default_graph()
With g.control_dependencies([var1,var2,..]):
execute(var4,var5)

17. Subgraphs (save Computation?)
import tensorflow as tf
x = 5
y = 17
add_op = tf.add(x,y)
multiply_op = tf.multiply(x,y)
kid_programmer = tf.multiply(x,add_op)
naive_programmer = tf.multiply(x,y)
power_op = tf.pow(add_op , multiply_op)
with tf.Session() as sess:
writer = tf.Summary.FileWriter("ist597demo",sess.graph)
z = sess.run(power_op)
writer.close()

18. Subgraphs(Distributed Computing ?)
We can break graph into several chunks or several sub graphs .
Each subgraph can run parallely across CPUs , GPUs or even across powerful TPUs cores.
Example Vggnet , alexnet , inceptionnet

19. Simple approach for distributed computing
Steps to follow
Create a graph and set log_device_placement TRUE in your session
with tf.device('/gpu:0'):
a = tf.constant([6.0,60.0], name='a')
b = tf.constant([7.0,56.5], name='b')
c = tf.multiply(a, b)
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
Z = sess.run(c)

20. Big No with tensorflow?
If you need more than one graph?(seriously ? , that’s poor development practice)
Let’s assume we leave behind development principles and if you still need multiple graphs for your logic to work.
Multiple graphs require multiple sessions, each will try to use all available resources by default(sometimes causes operating system to freeze/crash(Windows users rest you all know :P))
Can't pass data between them without passing them through python/numpy, which doesn't work in distributed
Alternate Approach
It’s better to have disconnected subgraphs within one graph

21. But still if you need #Use tf.Graph to build User graph
graph1 = tf.get_default_graph()
graph2 = tf.Graph()
with graph1.as_default():
var1 = tf.Constant(3)
with graph2.as_default():
var2 = tf.Constant(5)
Important Note :- Don’t mix default and user graph.
And also don’t do any numpy matrix computation (such as load or save)

22. Useful Tricks
Warning
“The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
Solution
Import os
os.environ[‘TF_CPP_MIN_LOG_LEVEL’] = ‘2’
CUDA_VISIBLE_DEVICES=0,1,2,3 Or
CUDA_VISIBLE_DEVICES = 0
Print always help

23. Questions? Deep Questions?