Automatic Handwriting Generation Presented by Ms.S.T.shenbagavalli, AP/CSE

Deep Learning Deep Learning is changing the way we look at technologies. There is a lot of excitement around Artificial Intelligence (AI) along with its branches namely Machine Learning (ML) and Deep Learning at the moment.

Traditional handwriting Generation system Handwriting recognition - classifying each handwritten document by its writer is a challenging problem due to huge variation in individual writing styles. The traditional approach to solving this would be to extract language dependent features like curvature of different letters, spacing b/w letters etc. and then use a classifier like SVM to distinguish between writers. 

Hand writing recognition using deep learning is a very powerful technique for several reasons: It automatically identifies deep powerful features Our approach of feeding in random patches makes the model text independent High prediction accuracy makes it possible to use this in practical applications

Handwritten Text Recognition (HTR)

INTRODUCTION This is a task where given a corpus of handwriting examples, generate new handwriting for a given word or phrase. The handwriting is provided as a sequence of coordinates used by a pen when the handwriting samples were created. From this corpus, the relationship between the pen movement and the letters is learned and new examples can be generated ad hoc.

Step 1:Converting into digital text Offline Handwritten Text Recognition (HTR) systems transcribe text contained in scanned images into digital text

Step-2:Building the neural network Layer Build a Neural Network (NN) which is trained on word-images from the IAM dataset. As the input layer (and therefore also all the other layers) can be kept small for word-images, NN-training is feasible on the CPU (of course, a GPU would be better). 

Model Overview- HTR system

It consists of convolutional NN (CNN) layers, Recurrent NN (RNN) layers and a final Connectionist Temporal Classification (CTC) layer.  We can also view the NN in a more formal way as a function which maps an image (or matrix) M of size W×H to a character sequence (c1, c2, …) with a length between 0 and L.

Operations CNN The input image is fed into the CNN layers. These layers are trained to extract relevant features from the image. Each layer consists of three operation. First, the convolution operation, which applies a filter kernel of size 5×5 in the first two layers and 3×3 in the last three layers to the input. The non-linear RELU function is applied. Finally, a pooling layer summarizes image regions and outputs a downsized version of the input. 

Operations RNN The feature sequence contains 256 features per time-step, the RNN propagates relevant information through this sequence. The popular Long Short-Term Memory (LSTM) implementation of RNNs is used, as it is able to propagate information through longer distances and provides more robust training-characteristics than vanilla RNN.

Operations CTC While training the NN, the CTC is given the RNN output matrix and the ground truth text and it computes the loss value. While inferring, the CTC is only given the matrix and it decodes it into the final text. 

Implementation using TF The implementation consists of 4 modules: SamplePreprocessor.py: prepares the images from the IAM dataset for the NN DataLoader.py: reads samples, puts them into batches and provides an iterator-interface to go through the data Model.py: creates the model as described above, loads and saves models, manages the TF sessions and provides an interface for training and inference main.py: puts all previously mentioned modules together

Conclusion The NN consists of 5 CNN and 2 RNN layers and outputs a character-probability matrix. This matrix is either used for CTC loss calculation or for CTC decoding. An implementation using TF is provided and some important parts of the code were presented. Finally, hints to improve the recognition accuracy were given.