Demonstration Study for Applying AVED to Still Images from Station M Update, next steps, workflow overview Demonstration Study for Applying AVED to Still Images from Station M MBARI Internal Project Update, next steps, workflow overview Danelle Cline

MBARI May 13rd, Agenda Updates Updates Next steps discussion Next steps discussion Data workflow overview Data workflow overview Action items Action items

MBARI May 13rd, Updates Modifications completed to core AVED software for still frame processing Modifications completed to core AVED software for still frame processing Required changes in segmentation and saliency modelRequired changes in segmentation and saliency model Added customized cmd-line option for processing time-lapse imagesAdded customized cmd-line option for processing time-lapse images --mbari-timelapse-stills --mbari-timelapse-stills Examples StaM4211PsychroSeries/WhiteEchSeries: Examples StaM4211PsychroSeries/WhiteEchSeries: Working on getting more compute resources to process a demonstration data set Working on getting more compute resources to process a demonstration data set

MBARI May 13rd, Next Steps Now that we know the types of possible detections… Now that we know the types of possible detections… Decide use case for demonstration Decide use case for demonstration This will drive what and how much data to process, and what kind of training libraries to create for image classification.This will drive what and how much data to process, and what kind of training libraries to create for image classification. Example use cases: Example use cases: Process a collection looking for temporal changes in fauna and structures on the seafloor, focusing on the sessile fauna polychaete(Paradiopatra) burrows and glass sponge (Hyalinacea) stalks.Process a collection looking for temporal changes in fauna and structures on the seafloor, focusing on the sessile fauna polychaete(Paradiopatra) burrows and glass sponge (Hyalinacea) stalks. Process a collection around major El Niño La Niña events between 1997, searching for a few select animals previously analyzed by hand to ground truth against AVED Other ideas?Other ideas? Example AVED events from Sta4211 image set

MBARI May 13rd, Data Workflow

MBARI May 13rd, AVED Process Image Preprocessing Scale and reformat Histogram equalize Mask equipment, time code overlays, black bars, etc. Post- processing Segmentation and Tracking Detection events.XML Every frame

MBARI May 13rd, AVED Editor Optional, but can useful for removing “false detections”, or combining eventsOptional, but can useful for removing “false detections”, or combining events

MBARI May 13rd, Classification Matlab program developed by Perona student Marc’Aurelio Ranzato at Caltech and Universita’ degli studi di Padova Matlab program developed by Perona student Marc’Aurelio Ranzato at Caltech and Universita’ degli studi di Padova Developed to analyze biological particlesDeveloped to analyze biological particles Based on extracting features usingBased on extracting features using local jets (Schmid et al. 1997) (convolution of the image with a derivative of Gaussian kernel) local jets (Schmid et al. 1997) (convolution of the image with a derivative of Gaussian kernel) image and power spectrum principal components (Torralba et al. 2003) image and power spectrum principal components (Torralba et al. 2003) Model training data with mixture of Gaussians (Choudrey and Roberts 2003)Model training data with mixture of Gaussians (Choudrey and Roberts 2003) Implemented in Matlab Implemented in Matlab processes grayscale square subimages of the segmented scene containing the object to be classifiedprocesses grayscale square subimages of the segmented scene containing the object to be classified

MBARI May 13rd, Classifier Example small benthic image set Example training images Leukothele other Rathbunaster

MBARI May 13rd, Action Items 1. Provide the data set (Ken, Jake, Mike) 2. Engineer the workflow to process the data set (Danelle) When steps 1 and 2 complete, can start on creating a training library (Linda) When steps 1 and 2 complete, can start on creating a training library (Linda)

MBARI May 13rd, Q&A Project # Project # Project wiki: Project wiki: