Interactive Evolutionary Computation Hideyuki TAKAGI Kyushu University takagi@design.kyushu-u.ac.jp http:// www.design.kyushu-u.ac.jp/~takagi

Part 1: Humanized Computational Intelligence 2. What is IEC? 3. IEC Applications 4. IEC research for Reducing IEC User Fatigue 5. IEC for Human Science 6. New IEC framework

Computational Intelligence in the 20th Century 3

Cooperation of Computational Intelligence K F S N Powerful cooperative technologies were developed in the end of 20th Century.

What Comes Next?

Two Different Human Capabilities

Importance of Human Factors with Computational Intelligence preference 1st Generation: 2nd Generation: 3rd Generation: experience auto-design computational intelligence implicit knowledge explicit knowledge optimization

Humanized technology ROBOTICS, CONTROL DATA MINING SIGNAL PROCESSING natural environment human environment ROBOTICS, CONTROL location, speed, obstacles, … preference, subjective evaluation emotion … acquired knowledge qualifying the knowledge DATA MINING database mining IF … THEN … physical measurement human perception SIGNAL PROCESSING filter design S/N ratio

Direction of Computational Intelligence Interactive Evolutionary Computation 1 9 8 2 N F S E C + G A P : K o z a ( s ) - d r i v e n u y R g t . l I b c m p f h Humanized Computational Intelligence

Part 2: What is IEC? 1. Humanized Computational Intelligence 3. IEC Applications 4. IEC research for Reducing IEC User Fatigue 5. IEC for Human Science 6. New IEC framework

evolutionary computation What is IEC ? Target System evolutionary computation subjective evaluation Features (1) fewer population size and fewer generation numbers, (2) nevertheless, satisfactory solutions can be obtained, i.e. simple landscape, (3) discrete fitness in n-evaluation levels, and (4) relative fitness in general.

Searching spaces of IEC tasks are generally simple, because ...... Any searching points that human operators cannot distinguish are same for human.

Statistics of IEC Papers

IEC Research @ Takagi Lab application-oriented interface research (1) 3-D CG lighting design support (2) montage image system (3) speech processing (4) image processing (5) hearing-aid/cochlear implant fitting (6) virtual reality in robot control (7) media database retrieval (joint1) virtual aquarium (joint2) geoscientific simulation (joint3) 3-D CG modeling education (joint3) fireworks animation design (joint4) mental disease diagnosis (1) input interface 1.1 discrete fitness value input method (2) display interface 2.1 prediction of user's evaluation char's 2.2 display for time-sequential tasks (3) acceleration of EC convergence 3.1 approximation of EC landscape 3.2 IDE/gravity, IDE/moving vector (4) active user intervention to EC search 4.1 on-line knowledge embedding 4.2 Visualized IEC (5) IEC user model (6) New IEC frameworks IEC for Human Science (1) measuring human mind (2) finding unknown knowledge (3) IEC with physiological responces

IEC Research Categories graphic art & CG animation 3-D CG lighting design music editorial design industrial design face image generation speech processing hearing aids fitting virtual reality database retrieval data mining image processing control & robotics internet food industry geophysics Hideyuki Takagi, "Interactive Evolutionary Computation: Fusion of the Capacities of EC Optimization and Human Evaluation,” Proceedings of the IEEE, vol.89, no.9, pp.1275-1296 (Sept., 2001). discrete fitness value input method prediction of fitness values interface for dynamic tasks acceleration of EC convergence combination of IEC and non-IEC active intervention Visualized IEC art education writing education games and therapy social system

Part 3: IEC Applications Hideyuki Takagi, "Interactive Evolutionary Computation: Fusion of the Capacities of EC Optimization and Human Evaluation,” Proceedings of the IEEE vol.89, no.9, pp.1275-1296 (Sept., 2001). 1. Humanized Computational Intelligence 2. What is IEC? 3. IEC Applications 4. IEC research for Reducing IEC User Fatigue 5. IEC for Human Science 6. New IEC framework

Part 5: IEC for Human Science CONTENTS 1. Artistic Applications CG, lighting design, CG animation, montage, music, industrial design 2. Engineering Applications signal processing, robotics, VR, media DB retrieval, data mining, intrusion detection 3. Others geo-science, coffee blending, game & education

Interactive GP for Graphic Art S y s t e m d i g n b T a u o U N E M I

Breeding Forms on the Infinite Plane by W. Latham (1992)

IGA for 3D CG Lighting Design By K. Aoki and H. Takagi Gloomy impression Cheerful impression Heroine movie star Wicked movie star by Interactive GA by HAND

Result of Subjective and Statistic Tests 1% level of significance 5% level of significance gloomy impression heroine impression cheerful impression villain impression IGA-based lighting design looks effective for beginners.

Lighting environments expands thanks to Increasing LED performance and decreasing LED costs. Increasing the # of LED lights makes lighting design difficult. Expanding Lighting Environments Thanks to Increasing LED Performance and Decreasing LED Costs

IEC-based Lighting Design

Education of Imagination and Creativity for 3D Shape It is difficult and time consuming for computer beginners (artist, non-technical student etc.) to acquire 3D modeling skill. Apply IEC to support CG skill and focus on educating imagination and creativity.

IEC changes shape parameters based on user's imagination 3D Model Example Nishino et al. (SMC1998, SMC1999) blend deform P2 P3 P1 P5 P4 IEC changes shape parameters based on user's imagination P5 P4 P3 P6 P1 P2

IEC-base 3D Modeling Concept Nishino, Takagi, and Utsumiya (2000) Genetic Algorithm create new 3D geometries by inheriting highly rated shapes and simulating natural evolutionary processes like crossover and mutation display a set of new shapes designer 3D shape image to create rate each shape :very good :good :not good

Subjective Test Manual >> IEC Manual > IEC Manual = IEC Only 48 (8 parameters ×6 primitives) parameters are modified by GA Realistic Examination Creative Examination ferocious green pepper Quality Operability Manual >> IEC Manual > IEC Manual = IEC Manual < IEC

Internet version of IEC 3-D Modeling Education System visualize 3D models by explored parameters CG System for Education CG System for Artwork Creation CG System for Car Body Design explore CG parameters by evolutionary computation display generated models IEC-based Modeling Interface specify rates based on subjective preference network students artists car designers

Evolution of Funny Animated Figures by Jeffrey Ventrella

CG Fireworks Animation Made by IEC

three days later, 10 GA generations Montage Image by C. Caldwell and V.S. Johnston (1991) target face montage face three days later, 10 GA generations

Montage Image by H. Takagi and K. Kishi (KES1999)

IGA for Music Sonomorphs: Melody Generator GenJam: Jazz Solo Generator by G. L. Nelson (1993) GenJam: Jazz Solo Generator by J. A. Biles (1994) Rhythm Generator by D. Horowithz (1994)

Industrial Design by H. Furuta

HTML Design EC Looks nice ! font of letters, colors of letters and background, and etc... Looks nice ! by N. Monmarche et al.

Color Poster Design C h a n g i o l r s d F t w I E M k L y u m p B y T . O b a t n d M H g i w r C h a n g i o l r s d F t w I E M k L y u S b 1 . , P e 5 c V m p Colors and fonts are VIGOR, and layout is STABILITY. Colors and fonts are CLEAN, and layout is ELEGANT.

EC Evolutionary Banner Visitors who click the banner Q 1 2 R by R. Gatarsk Q 1 2 R Visitors who click the banner EC

Part 5: IEC for Human Science CONTENTS 1. Artistic Applications CG, lighting design, CG animation, montage, music, industrial design 2. Engineering Applications signal processing, robotics, VR, media DB retrieval, data mining, intrusion detection 3. Others geo-science, coffee blending, game & education

Why IEC-based Signal Processing? There are many cases that SP users are not SP experts but need to design SP filters. ? ? ? ? ? ? ? ? hearing aid image filter Solution is auditory-based SP and visual-based SP without any SP knowledge. IEC realizes this approach.

Recovering Distorted Speech Watanabe and Takagi (SMC2005) Distorted Speech

Experimental Result -- Method of Successive Categories -- distance scale Ideal Situation : GA filter design when ideal speech is given. Sheffe’s method of paired comparison showed that this difference is significant (p<0.01).

Experimental Result Sheffe’s Method of Paired Comparisons difference is significant (p<0.01)

How Distorted Speech was Recovered? Formant area was mainly recovered

Voice Conversion by Interactive EC by Y. Sato et al.

Example of Prosodic Modification by Y. Satoh, et al Original Voice Modified by IEC Text-to-Speech Intelligible voice male-like voice real voice child-like voice

IEC for Agent’s Voice Design B y T . M o r i t a , S . I b a a n d M . I s h i z u k a evaluation Prosody control EC

Hearing Compensation is Difficult

Ideal Goal is Far final hearing target goal preference cognitive level perceptual level sensory level preference final hearing input sounds measurable in part target goal

Tuning System based on How User hears Ohsaki and Takagi (1998-2007) EC hearing aid hearing aid Conventional Proposed

Evaluation One – Two Weeks Later (5 subjects) IEC Fitting vs. audiologist fitting monosyllable articulation sound quality fitting time IEC < audiologist Six Months Later (4 subjects) evaluation IEC Fitting vs. audiologist fitting sound quality APHAB

Visualized IEC Fitting on a PDA Hayashida and Takagi (IECON2000) Visualized EC: parameters in an n-D EC landscape are mapped on a 2-D space for visualization. IEC Fitting: IEC-based hearing aid fitting

Image Enhancement using Interactive GP by R. Poli and S. Cagnoini Image enhancement filter evolves according to how processed images look well.

Echo-Cardiographic Image by R. Poli and S. Cagnoni (1997)

Non-linear Density Transformation by Using of GP g(i,j) ex.invert n output density f(i,j) input o n input density output non-linear density transformation by using GP n output density f(i,j) o n input input density output g(i,j)= G ( f ( i, j ) ) generated by GP

GP-Based Image Filter by Using Neighborhood Pixels Information input pixels output pixels x x y operator G y g(i,j) = G(f) zoom in GP generates G( ) G=max[abs(f(i,j-1)),max{-0.41,0.79+sin(min(f(i+1,j-1),-0.03))-f(i+1,j+1)}]

Experiment of IGP-Based Edge Detection IEC-based filter conventional math-based filter input image 2nd generation Laplacian filter

Experiment of IGP-Based High Pass Filter Design conventional math-based filter IEC-based filter input image high-pass filter 6th generation

IGP-Based X-component Extraction conventional math-based filter IEC-based filter input image Prewitt x-axis pass filter 4th generation

IGP-Based Y-component Extraction conventional math-based filter IEC-based filter input image Prewitt y-axis pass filter 6th generation

IEC-based Color Filter Design input image examples of coloring

Image Enhancement Filter Design by a Medical Doctor original ultrasonic image of a lymph node Enhanced Image after 12 IEC generations

Consumer Robots Business Has Been Launched Home robots will come soon.

Human Friendly Trajectory Control of a Robot Arm by N. Kubota, K. Watanabe and F. Kojim

NN Controller for LEGO Jeep-Robot by H. H. Lund, O. Miglimo, L. Pagliarini, A. Billard, and A. Ijspeert Children want to make a robot avoiding obstacles. Children cannot make a program of its controller but can choose better robot moving. Let's evolve the robot controller according to the children's choice.

Interactive EC for Virtual Reality Kamohara, Takagi, and Takeda (SMC2007) A r m w e s t l i n g : h u a v . o b c f z y V R I E C j d

IEC for Dental Education in VR How to realize the reality of VR force feedback? VR dental education software http://elianealhadeff.blogspot.com/2009/05/dental-serious-games-with-3d-touch.html http://www.1800dentist.com/dentist-love-blog/blog/default.aspx?useridpost=3 http://jks-folks.stanford.edu/bonesim/

Media DB Retrieval & Media Converter by H. Takagi et al. My impression is …

IEC-based Image DB Retrieval by S.-B.- Cho, et al.

Interactive EC for Data Mining by T. Terano, Y. Ishino, et al.

Visual Data-mining Through 2-D Mapping by GP by Venturini for example, X = (x1-u) / (87.3 / x12) Y = (46.7 * x6) * (25.2 + 81.0)

Intrusion Pattern Generations for Intrusion Detection System Ja-Min KOO and Sung-Bae CHO (2004) illegal attack computer virus IDS virus virus virus How to evaluate an intrusion Detection system? How to generate several new types of intrusion patterns for evaluation?

Generating Intrusion Patterns for Intrusion Detection Systems J. Budynek, E. Bonabeau, and B. Shargel (2005) script individuals gene pool

Generating Intrusion Patterns for Intrusion Detection Systems J. Budynek, E. Bonabeau, and B. Shargel (2005) crossover mutation deletion

Intrusion Pattern Generations for Intrusion Detection System Ja-Min KOO and Sung-Bae CHO (2004) Intrusion Detection System generated intrusion patterns GA + code evaluation detection performance 73% of generated executable intrusion patterns were different.

Part 5: IEC for Human Science CONTENTS 1. Artistic Applications CG, lighting design, CG animation, montage, music, industrial design 2. Engineering Applications signal processing, robotics, VR, media DB retrieval, data mining, intrusion detection 3. Others geo-science, coffee blending, game & education

Geological Modeling Based on Interactive EC

Difficulty many variables no numerical target strength, depth, stress, etc. no numerical target Numerical similarity may not mean qualitative similarity, such as wrong fault inclination, wrong depth extent. Help of geologists is necessary. computational optimisation with human judgement Computational optimisation methods to search material parameters are required besides geologist’s judgement.

IEC-based Modelling of Extension of the Earth’s Crust Wijns, Moresi, Boschetti, and Takagi (SMC2001) sketch drawn by a geologist geological forward model EC evaluation based on geological knowledge and experience parameters material

IEC-based Modelling of Subduction of Oceanic Crust geological forward model animations Strength of continental/oceanic crust Convergence rate Depth of sedimentary wedge parameters material evaluation based on geological knowledge and experience EC

Evolving Blends of Coffee by M. Hardy M o c h a K i l m n j r C b ? % T g e t B d f I E F s V u x R

Simulated Breeding for Composition Support System by K. Kuriyama, T. Terano, and M. Numao Two sequences of four scenes are chosen as parents. Two better sequences are chosen as parents. Composition and comparison with similar composition.

Part 4: IEC research for Reducing IEC User Fatigue 1. Humanized Computational Intelligence 2. What is IEC? 3. IEC Applications 4. IEC research for Reducing IEC User Fatigue 5. IEC for Human Science 6. New IEC framework

Approaches for Reducing IEC User Fatigue (1) input interface 1.1 discrete fitness value input method (2) display interface 2.1 prediction of user's evaluation char's 2.2 display for time-sequential tasks (3) acceleration of EC convergence 3.1 approximation of EC landscape 3.2 IDE/gravity, IDE/moving vector (4) active user intervention to EC search 4.1 on-line knowledge embedding 4.2 Visualized IEC (5) IEC user model (6) New IEC frameworks

Part 5: IEC for Human Science 1. Humanized Computational Intelligence 2. What is IEC? 3. IEC Applications 4. IEC research for Reducing IEC User Fatigue 5. IEC for Human Science 6. New IEC framework

Part 5: IEC for Human Science CONTENTS 1. measuring dynamic range of happy-sad scale 2. finding unknown psychophysiological facts 3. extended IEC: IEC with physiological responses

Psychotherapy / Diagnostics Takagi, Takahashi, and Aoki (SMC2004) IEC-based CG Lighting Design Umm, still his range is narrower. mental patient (schizophrenics) sad happy emotion range of the patient sad happy psychotherapist psychiatrist emotion range of mentally healthy persons

PT PM NY NH happy impression NN PK happy scale NS NK

NH NK NS PK sad impression NY sad scale PT PM NN

Discussion obtain the order scale of "happy - sad" expression range from psychological scales of happy and sad obtained through a subjective test sad happy sad happy 1st NS 5th NY 2nd NN 6th PT 3rd NK 7th NH 4th PM 8th PK

Part 5: IEC for Human Science CONTENTS 1. measuring dynamic range of happy-sad scale 2. finding unknown psychophysiological facts 3. extended IEC: IEC with physiological responses

48.5% (practitioner’s result) Cochlear Implants Fitting P. Legrand, C. Bourgeois-Republique, et al. evaluation fitting 45% 91.5% (IEC fitting) 48.5% (practitioner’s result) conventional fitting 20 IEC fitting 15 intensity 10 5 E1 E2 E3 E4 E5 E6 E7 E8 E9 E13 E14 E15 electric channels

IEC Breaked Common Sense? Slide made by Prof. Pierre Collet 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 E 1 E 2 E 3 E 4 E 5 E 6 E 7 E 8 E 9 E 13 E 14 E 15 Only 3 electrodes (E1, E7, E9) have 3 a significant C-T range, but the best 91.5%! Setting all electrodes but E1,E7, E9 under the liminary value gives 82.1% Adding electrodes E4 and E15 gives 58.8% !!! More electrodes  better audition!

Obtained Knowledge from Analysis of IEC Fitting hearing aid M. Ohsaki and H. Takagi (1999) The best hearing characteristics of human seems to depend on sounds. (1) Loudness charact. tuned by noise or pure tones (conventional fitting) ≠ Loudness charact. tuned by speech (IEC fitting) (2) fitting using speech1 w/wo noise fitting using speech2 w/wo noise ≠ fitting using music w/wo noise ・・・・ fitting using speechN w/wo noise differences are small

Part 5: IEC for Human Science CONTENTS 1. measuring dynamic range of happy-sad scale 2. finding unknown psychophysiological facts 3. extended IEC: IEC with physiological responses

IEC with Physiology physical features control physiological responses H. takagi (2004) physical features Emotion Control with 3 Steps control physiological responses 視聴者の感動・情動などは視聴者の生理反応として表れる Emotion Enhancement Filter Example Application

Step 3: Conventional IEC Based on Psychological Feedback image filters EC Conventional subjective evaluation IEC = system optimization technique based on human subjective feedback

－ Step 3: Extended IEC Based on Physiological Feedback Proposal image filters physiological measurement － EC Proposal given target physiological conditions

Part 6: New IEC framework 1. Humanized Computational Intelligence 2. What is IEC? 3. IEC Applications 4. IEC research for Reducing IEC User Fatigue 5. IEC for Human Science 6. New IEC framework

Part 6: New IEC framework CONTENTS 1. IEC + EMO 2. Interactive PSO and interactive DE 3. paired comparison-based IEC

Multi-Objective Optimization with Subjective Object(s) I'm looking for an apartment that is (1) wider, (2) less expensive, and (3) closer to a station. width tenant distance to st. tenant width distance to st. Then, how to find an apartment (4) with great view (5) in a quite area? 101

Multi-Objective Optimization: MEMS Design joint research with UC Berkeley MEMS (Micro Electronic Mechanical Systems) for Sensors, Robotics, Communications, Biotechnology, Energy Generation Multi-objective optimization for given specification: receiving frequency, strength, and others. We want to use domain knowledge for circuit design.

IEC + EMO for MEMS Design Raffi Kamalian et al. (2004) EMO: four objects: Resonant freq = 100kHz K_vertical = 1 N/m K_lateral = 100 N/m Area = minimized IEC: embedding domain knowledge for MEMS circuit design

Subjective Test + Statistic Test for EMO+IEC vs. EMO EMO+IEC > EMO (99% significant by Wilcoxon matched-pairs signed-ranks test .)

IEC + EMO for Architectural Design http://www.mlit.go.jp/kisha/kisha02/07/071220_.html

Algorithm for Generating Layout 106

IEC +EMO for Room Layout -- Algorithm for Generating Layout -- Makoto Inoue et al. (2008) wall wall wall wall wall wall 107

IEC +EMO for Room Layout Layout Algorithm objective 1 objective 2 EMO IEC EC room sizes room shapes moving line to any rooms without passing through other private rooms window sizes requested by architectural laws total water pile length total wall length

Part 6: New IEC framework CONTENTS 1. IEC + EMO 2. Interactive PSO and interactive DE 3. paired comparison-based IEC

Particle Swarm Optimization 3 2 1

GA vs. POS and IGA vs. IPOS F1 ○ F2 F3 F4 F5 F6 ≒ F7 F8 ○ GA PSO IGA Comparison of Performance (○：faster，≒：equivalent ) GA PSO F1 ○ F2 F3 F4 F5 F6 ≒ F7 F8 IGA IPSO ○ low complexity high

Tolerance of PSO for Quantization Noise in Fitness more 5-levels 10-levels 100-levels 1,000-levels 5 4 3 2 1 quantization noise in fitness fewer

Experimental Result for F1 Interactive GA levels levels levels levels F2～F8 has similar tendency Interactive PSO levels levels levels levels

After Implementing Four Methods for Increasing Noise Tolerance generations Nakano and Takagi (CEC2009) fitness IGA IPSO IPSO with proposed methods 114

Part 6: New IEC framework CONTENTS 1. IEC + EMO 2. Interactive PSO and interactive DE 3. paired comparison-based IEC

Pair Comparison: Tournament IEC Ordinary IEC 116

Paired Comparison-based Interactive Differential Evolution Takagi and Mitsuyasu (patent) Takagi and Palles(CEC2009) target vector base vector parameter vector 1 vector 2 F mutant vector individuals trial vector target vector 117

IGA vs. Paired Comparison-based IDE HA 1 HA 2 HA 3 HA 4 1 2 3 4 5 HA 5 HA 6 HA 7 HA 8 GA 1 2 3 4 5 HA 9 HA 10 HA 11 HA 12 1 2 3 4 5 HA 13 HA 14 HA 15 HA 16 1 2 3 4 5 HA 1 HA 2 DE ON ON

CONCLUSION One of IEC research directions is Humanized Computational Intelligence, and IEC is one of tools for this direction. IEC has high applicability in wide variety of areas. IEC research includes research reducing IEC user fatigue and research for human science besides major IEC research, IEC applications. We also introduced recent new IEC research on extending IEC algorithms and frameworks.

Further Information IEC tutorial & survey paper Hideyuki Takagi, "Interactive Evolutionary Computation: Fusion of the Capacities of EC Optimization and Human Evaluation,” Proceedings of the IEEE vol.89, no.9, pp.1275--1296 (Sept., 2001). Its draft paper is downloadable from the below URL. IEC slides are downloadable from the below URL. Personal Contact takagi@design.kyushu-u.ac.jp http://www.design.kyushu-u.ac.jp/~takagi