Neural Networks and Their Applications John Paxton Montana State University August 14, 2003

Yellowstone National Park

Problem Domains Storing and recalling patterns Classifying patterns Mapping inputs onto outputs Grouping similar patterns Finding solutions to constrained optimization problems

Human Brain 10 billion neurons 60 trillion connections (synapses) soma axon dendrite

Human Brain Plastic Nonlinear Parallel Distributed Memory Distributed Processing

Artificial Neural Network input layer middle layer output layer

Comparison Biological Neural NetArtificial Neural Net SomaNeuron DendriteInput AxonOutput SynapseWeight

Neuron x1x2x3x1x2x3 y w1w2w3w1w2w3

Neuron Input -1 (absent) 0 (unknown) 1 (present)

Neuron Output X = ( ∑ x i w i ) y = fn ( X ) Common functions (fn) –sign function (-1 if X < 0, else 1) –sigmoid function (1 / (1 + e -x ) )

Perceptron OR concept sign function x 0 (1) x 1 x 2 2

Perceptron Can automatically learn the weights in a provable fashion! But can only learn linearly separable concepts. no yes no

Multilayer Neural Network Can include zero or more hidden layers Has a learning algorithm (backpropagation) that works in practice! XOR

Backpropagation Determine network topology Initialize weights Present a training example Apply inputs, calculate activations in middle layer, then calculate activations in output layer Calculate errors in output layer Calculate errors in hidden layer

Backpropagation Update weights Repeat process until some stopping condition is met Possible stopping conditions –largest error is below some threshold –total error is no longer decreasing –a time limit is exceeded

Clustering (Unsupervised) Traveling Salesperson Problem (6 cities) Kohonen Nets

Strengths Very versatile. They can predict, they can classify, they can cluster. They produce good results in complicated domains. Can handle categorical and continuous data. Available in many off-the-shelf products

Weaknesses All inputs must be massaged onto the range [ ]. Can not explain results. May converge on an inferior solution. Determining the topology is as much an art as it is a science. Might take a long time to converge.

Commercial Applications Neural networks were involved in more than 1 billion U.S. dollars in 1997!

Business Marketing –Microsoft. Direct mail marketing. –Albertsons. Determine the connection between buying diapers and buying beer. –BehavHeuristics Inc. Forecasts demand of airline flights. Real Estate –HNC. Automated Real Estate Appraisal.

Document and Form Processing Machine Printed Form Processing –Caere Corporation. Optical character recognition (FoxMaster). –Synaptics. Check reader. Hand Written Character Recognition. –Eastman Kodak. Forms processing for UK motor vehicle registration. –Fujitsu. Input to pen based computers.

Document and Form Processing Cursive Handwriting Character Recognition –Apple Newton 120. Input to PDA. Graphic Recognition –Fein-Marquet Associates, Inc. Converts a hand drawn chemistry picture into a table.

Food Industry Odor/Aroma Analysis –Sharp. Cooking control via an electronic nose in a microwave oven. Produce Development –M&M/Mars. Improved chemical formulations of products.

Food Industry Quality Assurance –Anheuser-Busch. Beer testing. –Florida Department of Citrus. Pulp wash detection. –Frito-Lay. Potato chip testing.

Financial Industry Market Trading –Gerber Baby Foods. Cattle futures trading. –John Deere. Pension management. –Walkrich Investment Advisors. Stock valuation. Credit Rating –Chase Financial. Forecast credit worthiness.

Financial Industry Fraud Detection –Dunn and Bradstreet. Check approval. –HNC. Credit card fraud detection (Falcon). –Mastercard. Deviations in spending habits.

Energy Electrical Load Forecasting –Bayernwerk AG. Hydroelectric Dam Operation –Tauernkraftwerke. Dam displacement prediction. Natural Gas –Northern Natural Gas. Predict gas index prices.

Manufacturing Process Controllers –Nippon Steel. Continuous casting. –Siemans. Rolling mill. Quality Control Systems –Dunlop. Tires. –Intel. Computer chips. –Volvo. Diesel knock testing. Paint inspection.

Medical and Health Care Image Analysis –NeuroMedical Systems, Inc. Pap smears. Drug Development –Vysis Inc. Protein analysis Resource Allocation –Anderson Memorial Hospital. Predict use of hospital resources.

Science and Engineering Chemical Engineering –StellarNet Inc. Spectroscopy. Electrical Engineering –NeuroCad Inc. Optimized circuit routing. Weather –National Weather Service.

Transportation and Communication Transportation –London Underground. Fault detection. –Rolls Royce. Fault detection. Communication –AT&T/Lucent. Echo cancellation systems (more than 20 years).

Questions?