cs638/838 - Spring 2017 (Shavlik©), Week 11 CS 540 Fall 2015 (Shavlik) 4/18/2018 Today’s Topics Talk by Sidharth Mudgal Sunil Kumar on text and Deep Nets Keras (on top of TensorFlow) for Solving Lab 3 I have uploaded to class web page my latest Python - 10 folds, perturbed examples, Condor, in multiple files, etc An ImageNet dataset (same six categories as Lab 3) Some Python/Keras questions I have April 11: GAN Intro, plus some cs838 progress reports April 18: Google-Madison talk on TPUs, plus some cs638 progress reports 4/3/17 cs638/838 - Spring 2017 (Shavlik©), Week 11

cs638/838 - Spring 2017 (Shavlik©), Week 11 Keras – CONV Layer 1 model = Sequential() # A ‘left-to-right’ model with no recurrent links. # The following needs to be a layer, not an argument (‘relu’ can be an arg). leakyReLUtoUse = LeakyReLU(alpha = 0.1) model.add(Conv2D(platesConv1, kernel_size = kernelSizeConv1, # Give two numbers if non-square. input_shape = [imageDimension, imageDimension, numberOfColors], data_format = "channels_last", # Says that the color channels are LAST. strides = strideConv1, padding = "valid", # ???? use_bias = True)) 4//17 cs638/838 - Spring 2017 (Shavlik©), Week 11

cs638/838 - Spring 2017 (Shavlik©), Week 11 Keras – CONV Layer 1 (pg 2) model.add(leakyReLUtoUse); # Had been model.add(Activation('relu')) model.add(ZeroPadding2D(padding = zeroPaddingConv1, data_format = "channels_last")) # Not needed. model.add(Dropout(conv1_dropoutProb)) - Seems one CANNOT do dropout on INPUT UNITS (but that’s ok) 4//17 cs638/838 - Spring 2017 (Shavlik©), Week 11

cs638/838 - Spring 2017 (Shavlik©), Week 11 Keras – POOL Layer 1 model.add(MaxPooling2D(pool_size = kernelSizePool1, strides = stridePool1, padding = 'valid')) model.add(Dropout(pool1_dropoutProb)) # I set prob to 0. Should use an IF? model.add(ZeroPadding2D(padding = zeroPaddingPool1)) 4//17 cs638/838 - Spring 2017 (Shavlik©), Week 11

cs638/838 - Spring 2017 (Shavlik©), Week 11 Keras – CONV Layer 2 model.add(Conv2D(platesConv2, // No need for input_shape. kernel_size = kernelSizeConv2, strides = strideConv, padding = "valid", use_bias = True)) model.add(leakyReLUtoUse); model.add(ZeroPadding2D(padding = zeroPaddingConv2)) model.add(Dropout(conv2_dropoutProb)) 4//17 cs638/838 - Spring 2017 (Shavlik©), Week 11

Keras – POOL Layer 2 (same as 1) model.add(MaxPooling2D(pool_size = kernelSizePool2, strides = stridePool2, padding = 'valid')) model.add(Dropout(pool2_dropoutProb)) # I set prob to 0. Should use an IF. model.add(ZeroPadding2D(padding = zeroPaddingPool2)) # Later I added a FOR LOOP that allowed N CONV+POOL layers # (all used the parameters of Conv1 and Pool2). 4//17 cs638/838 - Spring 2017 (Shavlik©), Week 11

cs638/838 - Spring 2017 (Shavlik©), Week 11 Wrapping Up the Model model.add(Flatten()) # Flattens the last MAX POOL layer so can fully # connect to the final HU layer. model.add(Dense(units = numberOfFinalHUs)) # Fully connected ‘flat’ HUs. model.add(leakyReLUtoUse) model.add(Dropout(final_dropoutProb)) model.add(Dense(units = numberOfClasses)) # The OUTPUT units. model.add(Activation("softmax")) # Other choices exist of course. 4/3/17 cs638/838 - Spring 2017 (Shavlik©), Week 11

Training the Model (setup) OLD model.compile(loss = 'categorical_crossentropy', optimizer = 'rmsprop', metrics = ['accuracy']) NEW optimizerToUse = Adam() # Had been 'rmsprop' (good for recurrent nets). model.compile(loss = 'categorical_crossentropy', optimizer = optimizerToUse, metrics = ['accuracy 4/3/17 cs638/838 - Spring 2017 (Shavlik©), Week 11

Training the Model (epochs) for i in range(epochsToRun): model.fit(X_train, y_onehot_train, epochs = 1, # We manage training ourselves. batch_size = batchSize, verbose = 0, # 0 for no logging to stdout, 1 for progress bar logging, # 2 for one log line per epoc # validation_data=(X_tune, y_onehot_tune), shuffle = True) # Permute examples each epoch! acc_train = accuracy(model, X_train, y_train) acc_tune = accuracy(model, X_tune, y_tune) # Only need this one every epoch. acc_test = accuracy(model, X_test, y_test) 4/3/17 cs638/838 - Spring 2017 (Shavlik©), Week 11

Creating the Confusion Matrix If new best TUNE set accuracy: y_pred_test = model.predict_classes(X_test, verbose = 0) confusionTestsetAtBestTuneset.fill(0) # Initialization earlier via: np.zeros((numberOfClasses, numberOfClasses)) # Then whenever new best TUNE result, refill it (no need to keep model) for y_pred, y in zip(y_pred_test, y_test): confusionTestsetAtBestTuneset[ y, y_pred ] += 1 # EXAMPLE: zip((a, b, c), (x, y, z)) produces ( (a, x), (b, y), (c, z) ). 4/3/17 cs638/838 - Spring 2017 (Shavlik©), Week 11

Keras (https://en.wikipedia.org/wiki/Keras) Lots of possible network topologies and methods for Deep ANNs Seems well designed, easy to use Runs fast Written (mainly) by ONE person! I’m working on getting Kera on CS Dept computers I have a Condor-ready version, but working with BMI Dept lab staff on Kera+Python 4/3/17 cs638/838 - Spring 2017 (Shavlik©), Week 11

Results on New Dataset (from ImageNet, down-sampled to 32x32 images)                |  airplanes  butterfly   flower    grand   starfish      watch     TRUE SUMs ------------------------------------------------------------------------------------------------------------------     airplanes  |        111           3          0          5          4          2 |  Row Sum =  125     butterfly  |          9         77          4          8         21          6 |  Row Sum =  125     flower     |         11          8         90         3         11          2 |  Row Sum =  125     grand      |         13          0          3        95          6          2 |  Row Sum =  119     starfish   |         34          1          4         13         70          3 | Row Sum =  125     watch      |         26          3          3        24         17         49 |  Row Sum =  122 ------------------------------------------------------------------------------------------------------------------ PREDICTED |        204         92        104        148        129         64     TOTAL  =  741 SUMS This dataset (in 10 folds, zipped) is in the same course directory as the Lab 3 data. 4/3/17 cs638/838 - Spring 2017 (Shavlik©), Week 11

“I thought this was a starfish” 4/3/17 cs638/838 - Spring 2017 (Shavlik©), Week 11

“I thought this was a watch” 4/3/17 cs638/838 - Spring 2017 (Shavlik©), Week 11

“I thought this was a flower” 4/3/17 cs638/838 - Spring 2017 (Shavlik©), Week 11

Some Python/Keras Questions I Have Saving models? Repeatable random-number sequences Took 27 mins to do early Confusion Matrix (100 epochs, 6000 training examples, 416k weights) – how much would this cost in the cloud? How to implement waitForEnter? 4/3/17 cs638/838 - Spring 2017 (Shavlik©), Week 11

In-Class (Ungraded) Lab Get Lab3 to work in Keras (might want to use my newer code; run main.py) Add ability to learn an ENSEMBLE Collect all the REAL-VALUED, TESTSET predictions in a 2D-array (ie, for each ensemble’s ‘best tune epoch”) testExamples x outputCategories When done with N models for the ensemble, collect accuracy of choosing the category with the most ‘points’ per training example Feel free to do a different extension (suggestions?) 4/3/17 cs638/838 - Spring 2017 (Shavlik©), Week 11

Version 2 of my Lab3 Keras Code main.py you should run this Python script parameters.py create an object that holds all the params, collect images, create unique ‘marker string’ per experiment dribbler.py prints both to screen and a file processCondorID.py convert the single int that condor provides into param settings utilsJWS.py read and perturb images, other misc utility/supporting functions JoeAndJudeCode.py the Keras code (presented in this lecture) is isolated here 4/3/17 cs638/838 - Spring 2017 (Shavlik©), Week 11