Xin Li LDCSEE WVU Spring From Gene to Meme Example of Memes: -Cultural ideas, symbols -Religions -Languages -Mathematics (e.g., Gaussian distribution) -Scientific understanding (e.g., Quantum mechanics) -Technological achievements (e.g., Atomic force microscopy) -Engineering designs (e.g., ipod, Wii, iphone)

Xin Li LDCSEE WVU Spring An Objective Measurement of Meme’s Fitness Industry Academia

Xin Li LDCSEE WVU Spring What is in Common? Steve JobsHerbert Simon

Xin Li LDCSEE WVU Spring

Xin Li LDCSEE WVU Spring 2009 Theme of This Talk “Life is about connecting dots” – –in “Staying Hungry, Staying Foolish” Stanford Commencement Address by Steve Jobs in 2005 Scientific research is also about connecting dots – –Search is an important component part of scientific research (where are the dots? how are they related?) – –Re-search often reveals hidden relationship among isolated dots that is not known before

Xin Li LDCSEE WVU Spring Image processing Statistical physics Communication Networking Cognitive Science geometry PRAM/Microarray STM/AFM MRI/PET Control analysis Chemical oscillation algebra statistics Image Processing as the Showcase Science Technology Engineering Mathematics

Xin Li LDCSEE WVU Spring Image Processing: at the Intersection of Science, Technology, Engineering and Mathematics (STEM) + our starting point

Xin Li LDCSEE WVU Spring 2009 So how should you choose your technical field? Outside environment plays some role –Emerging areas tend to attract more resources than traditional fields –Every dept. has its focused areas Learn yourself better –Good at theory or experiment/application? –Good at algebraic or geometric thinking? –Good at depth-first or width-first reasoning? Find a good match

Xin Li LDCSEE WVU Spring 2009 Taking Myself as an Example Entered Princeton ISS group in 1996 Very little research experience in my undergraduate study (BS thesis is on speech coding) Information theory or signal processing? – –Princeton EE is really strong in theory (not to mention Math and Physics) – –Majority of ISS students will take the theory path even by doing TAs due to limited research funding in the area of information theory – –Graduate students in Princeton EE from India are also really good at theory

Xin Li LDCSEE WVU Spring 2009 Information Theory vs. Image Processing IEEE TIP – –H-index=148 – –1992-present – –Most influential papers: image watermarking, image coding, image segmentation IEEE TIT – –H-index=214 – –1963-present – –Most influential papers: cryptography, space-time codes, error-correcting codes, wavelets,...

Xin Li LDCSEE WVU Spring 2009 Highly-cited Papers related to Image Processing Markov Random Field (Geman and Geman 1984) >8000 citations Wavelet theory (Daubechies, Mallat, Vetterli …) >180,000 citations Why do they last? “the most fruitful areas for the growth of sciences were those which had been neglected as a no-man’s land between the various established fields.” –Norbert Wiener

Xin Li LDCSEE WVU Spring 2009 Connection 1: MRF Pixels vs. Particles Pixel value = 0 or 1Spin direction = up or down

Xin Li LDCSEE WVU Spring A Little Bit History of Ising Model Proposed by Ising in his PhD thesis in D Ising model was analytically solved by L. Onsager in 1944 (who won the Nobel Prize in 1968) Phase transition behavior investigated by Yang and Lee in 1950s related to renormalization theory pioneered by RG Wilson (who won the Nobel Prize in 1982)

Xin Li LDCSEE WVU Spring Apply Ising Model to Images Applied to image restoration by Geman and Geman in 1984 (bring statistical mechanics to engineering) Stirred up lots of interest –More powerful image models (line process, higher-order MRF) –More efficient optimization algorithms (Gibbs sampling, Swendsen-Wang, Wolff algorithm) –New applications

Xin Li LDCSEE WVU Spring 2009 Image Example original noisy restored Monte-Carlo Optimization (minimize E)

Xin Li LDCSEE WVU Spring Connection to Hopfield Networks Why is this model so influential? The first-order approximation of associative memory in brain theory Prof. Hopfield gave a talk at WVU on Mar. 13, 2007 titled “How Do We Think So Fast? From Neurons to Brain Computations,”

Xin Li LDCSEE WVU Spring 2009 Statistical Mechanics and IT Shannon was the first to recognize the connection between statistical mechanics and communication theory Connection with statistical mechanics also exists for Turbo codes (belief propagation is related to Bethe free energy) “Multiuser detection and statistical mechanics” (Guo and Verdu’ 2003) “Evolution and structure of the Internet: A statistical physics approach” (R Pastor-Satorras and A Vespignani‘ 2004) “Statistical mechanics of complex networks” (R. Albert’ PhD thesis in 2001)

Xin Li LDCSEE WVU Spring 2009 If you think you have understood entropy “My greatest concern was what to call it. I thought of calling it ‘information’, but the word was overly used, so I decided to call it ‘uncertainty’. When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, ‘You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, nobody knows what entropy really is, so in a debate you will always have the advantage. ” -Conversation between Claude Shannon and John von Neumann regarding what name to give to the “measure of uncertainty” or attenuation in phone-line signals (1949)

Xin Li LDCSEE WVU Spring 2009 Connection II: Wavelet Theory and Image Processing Wavelet theory was established in late 1980s by mathematicians, computer scientists and electrical engineers together The most successful application of wavelets is likely to be lossy image compression (e.g., JPEG2000) –Also popular in other processing tasks such as segmentation, denoising and retrieval The question is: Why?

Xin Li LDCSEE WVU Spring 2009 Where do Wavelets Come from? Before wavelet, people used Short-Time FT to analyze transient signals J. Morlet – a geophysical engineer at a French oil company came up with an alternative approach which was recognized by Grossmann – Daubechies’ advisor S. Mallat – a graduate student at Penn met Y. Meyer’s student and recognized its connection to multi-resolution analysis

Xin Li LDCSEE WVU Spring 2009 Laplacian Pyramids invented by RCA Engineers

Xin Li LDCSEE WVU Spring 2009 At the Intersection of Math, CS and EE Math: construction of basis functions with good localization property in both time and frequency CS: decomposes images under a multi- resolution analysis framework in analogy to HVS EE: analysis-and-synthesis filter banks used by TV engineers Merge of roots: a new tool for data/signal analysis Different perspectives: deterministic (Besov- space functions) vs. statistical (heavy-tail distributions)

Xin Li LDCSEE WVU Spring 2009 Why Wavelets for Images? Math: Besov-space function, statistics: sparse component analysis, neuroscience: Independent components of natural scenes

Xin Li LDCSEE WVU Spring 2009 Beyond Image Processing Statistics: nonparametric regression Graphics: progressive mesh compression Turbulence: one of the most complicated phenomenon in nature Astronomy: hierarchical clustering theory of galaxy formation Biomedical: MRI, EEG, PET, mammography Acoustic: computer music analysis

Xin Li LDCSEE WVU Spring 2009 What is Missing in Wavelet Models? DWT sign flip IWT

Xin Li LDCSEE WVU Spring 2009 Connection III: Complex Networks and Image processing Common assumption made by MRF and wavelet models: locality or Markovian Most existing physical laws are defined locally; but what about nonlocality? A great mystery in brain science is how it collectively processes local information –Speed of nerve impulse transmission is much slower than that of logic gates –The power consumption of neural system is also much more efficient

Xin Li LDCSEE WVU Spring 2009 Networks of Neurons

Xin Li LDCSEE WVU Spring 2009 Complex Networks Internet World Wide Web Movie actor collaboration network Science collaboration network Citation networks Cellular networks Ecological networks Power networks

Xin Li LDCSEE WVU Spring 2009 How is it related to Image Processing? B 11 B 22 B 14 B 13 B 12 B 41 B 31 B 21 B 33 B 32 B 23 B 24 B 34 B 44 B 43 B 42 f3f3 f2f2 f1f1 B ij i j Parallel and Distributed Processing (PDP) or connectionism was at the foundation of neural networks

Xin Li LDCSEE WVU Spring 2009 How is it Different from NN? What role does time play? –Temporal binding hypothesis in neuroscience –Synchronization of nonlinear oscillators in chemical, biological and physical systems What role does feedback play? –As important as feedforward –Mountcastle’s uniformity principle in psychology Why does the network have to be hierarchical? –Natural world is organized in a hierarchical fashion –Our perception of natural world is the consequence of mapping from outside (physical stimuli) to inside (synaptic connections)

Xin Li LDCSEE WVU Spring dB 31.56dB 34.96dB xy DT KRFG dB 31.16dB 36.51dB 17.90dB 18.49dB 29.25dB 26.04dB 24.63dB 29.91dB Experiment 1: Compressed Sensing DT- Delauney Triangle-based (griddata under MATLAB) KR- Kernal Regression-based (Takeda et al. IEEE TIP 2007 w/o parameter optimization) 1 X. Li, “ Patch-based image interpolation: algorithms and applications, ” Inter. Workshop on Local and Non-Local Approximation (LNLA) ’ % kept

Xin Li LDCSEE WVU Spring 2009 Experiment 2: Image Coding JPEG-decoded at rate of 0.32bpp (PSNR=32.07dB) SFG-enhanced at rate of 0.32bpp (PSNR=33.22dB) SPIHT-decoded at rate of 0.20bpp (PSNR=26.18dB) SFG-enhanced at rate of 0.20bpp (PSNR=27.33dB) Maximum-Likelihood (ML) Decoding Maximum a Posterior (MAP) Decoding

Xin Li LDCSEE WVU Spring 2009 Unfulfilled Connections (I) Bayer Pattern (US3,971,065) Cone distribution in human retina CCD sensor design (engineering) could benefit from the organizational principle of cones in human retina (biology)

Xin Li LDCSEE WVU Spring Unfulfilled Connection (II) Q: Can we generate a HDR image (16bpp) by a standard camera? A: Yes, adjust the exposure and fuse multiple LDR images together

Xin Li LDCSEE WVU Spring High Dynamic Range Imaging Note that any commercial display devices we see these days are NOT HDR

Xin Li LDCSEE WVU Spring 2009 Unfulfilled Connection (III) Visual perception might be the first small step towards human intelligence but it will be a huge leap in human intelligence (can brain understand brain?)

Xin Li LDCSEE WVU Spring Summary and Conclusions Every theory, technology or system has its own evolution path –Understand its connection is the most difficult yet important task –My learning about image processing is still evolving, but I am hoping its principle can be also applied to other technical field such as communication “If you truly believe that God creates this world in a unified fashion, when you get stuck with a problem, seek your inspiration from around: nature, art and other sciences. Essentially, the principles are the same. ”

Xin Li LDCSEE WVU Spring 2009 Research vs. Development Good development/programming skills are a plus but secondary to good analytical/logical reasoning skills in my own assessment Implementation skills should be viewed at the same level as mathematical skills; they are both technical tools but cannot replace scientific vision/understanding ``Knowledge and productivity are like compound interest.'' –Richard Hamming

Xin Li LDCSEE WVU Spring 2009 How Good do You Need to at Mathematics? “Never be overwhelmed by the mathematics other people are boasting in their papers if you are in engineering: the more equations, the fewer ideas” Mathematics is a language – you cannot communicate well if you don’t master it; but you cannot advance science by simply playing with mathematics good-do-you-need-to-be-at-mathematics/

Xin Li LDCSEE WVU Spring 2009 Why does engineering/math/science education in the US suck?