The Big Health Data–Intelligent Machine Paradox D. Douglas Miller, MD, CM, MBA  The American Journal of Medicine  Volume 131, Issue 11, Pages 1272-1275 (November 2018) DOI: 10.1016/j.amjmed.2018.05.038 Copyright © 2018 Terms and Conditions

Figure 1 (A) Recurrent neural network (RNN) architecture makes use of sequential information. RNNs are called recurrent because they perform the same task for each element of a sequence, with the output being dependent on the prior computations. This creates a short-term ‘memory’ functionality that captures information about the prior calculations. This simple RNN is unrolled into a neural network of 3 layers designed to decode a 3-word phrase; the input at the time step (t) is a vectorial representation of word 2 in the phrase. The main feature of an RNN is the so-called hidden state, which comprises the interconnected memories at each time step (the blue arrows from and to the gray boxes). This memory is actually a mathematical function calculated based on the previous hidden state at time t – 1 and the current input at time t. The final output is a vector of probabilities of word 3 in the phrase from a vocabulary of choices available at time t + 1. (B) For an RNN to predict the next word in a sentence (ie, language modeling), it is helpful to know which words came before it. In these 2 sentences, a multilayer neural network is used to sequentially predict the next word from the unrolled RNN's hidden state memory of prior layers’ outputs and the current input (ie, “I have a pen. I have an ???”). Performing the same tasks at each step in the sequence with different inputs generates a vector of mathematical probabilities (ie, a generative model) that the final word in the second sentence is apple and not pen, red, or hello. High-probability sentences are typically correct (ie, “I have an apple”). This explains (in part) how RNNs (and more sophisticated long short-term memory units) can successfully carry out natural language processing tasks like reading a medical record. The American Journal of Medicine 2018 131, 1272-1275DOI: (10.1016/j.amjmed.2018.05.038) Copyright © 2018 Terms and Conditions

Figure 2 Researchers registered with CLEF dev set can access open-source captioned image databases (ie, plant specimen photos, digital medical images) and submit requests to use large training data sets to run artificial intelligence analytics.10 The upper run illustrates training a sentence-generating model on an ImageCLEF dev set. The lower run first trains the model on a Microsoft MS COCO image data set, then tests it on the ImageCLEF dev set. An algorithm scoring system (ie, METEOR) is used to assess the performance of different image-captioning software for concept detection (ie, a higher METEOR score equals better concept detection). Concept-based sentence re-ranking can then be applied on sentences generated by these LSTM-RNN models. The outcome sentence, “A plant with pink flower and brown stem…,” reflects the transformed hidden state description of the original image as generated by the neural network system. CLEF = Cross Language Evaluation Forum; CNN = convolutional neural network; LSTM = long short-term memory unit; METEOR = Metric for Evaluation of Translation with Explicit Ordering; MS COCO = Microsoft Coco framework; RNN = recurrent neural network. The American Journal of Medicine 2018 131, 1272-1275DOI: (10.1016/j.amjmed.2018.05.038) Copyright © 2018 Terms and Conditions