Department of Computer Science Computer Vision & Pattern Recognition Group IAPR Workshop on Graph-based Representations in Pattern Recognition June 11 th -13 th, 2007 – Alicante (Spain) GbR ’07 Separation of the Retinal Vascular Graph in Arteries and Veins Speaker: Kai Rothaus Co-authors: P. Rhiem, X. Jiang CVPR Group, University of Münster Homepage: cvpr.uni-muenster.de

GbR ’07 Department of Computer Science Computer Vision & Pattern Recognition Group 1 Rothaus, Rhiem and Jiang: Separation of the Vascular Graph in Arteries and Veins Outline Introduction – Medical purpose – Image-processing Method – SAT-problem specification (vessel labelling) – Operations for graph manipulation (edge labelling) – Solving Conflicts Results Conclusions and further work

GbR ’07 Department of Computer Science Computer Vision & Pattern Recognition Group 2 Rothaus, Rhiem and Jiang: Separation of the Vascular Graph in Arteries and Veins Medical Purpose Why retinal vessel are of interest? – Vessels of retina and brain are conjuct – Only on retina vessels are visible directly – Conclusions on diseases are possible Anatomy of the eye – Vessels enter the eyeball at the optic disc – Vessels only branch (no reconnection) – Capillars are invisible Differences of two vessel types on retina: ArteriesVeins oxygenated bloodoxygen-deficient blood thinnerthicker light-reddark-red stronger central reflexpoor central reflex never crossing arteriesnever crossing veins

GbR ’07 Department of Computer Science Computer Vision & Pattern Recognition Group 3 Rothaus, Rhiem and Jiang: Separation of the Vascular Graph in Arteries and Veins Vessel segmentation Input: Retinal Image Output: Binary vessel image Many segmentation algorithms, based on – Matched-filter – Tracking – Intensity riges or (1 st moment deviations) – Curvature (2 nd moment deviations) Special difficulties – Handling of bifurcations and crossings – Central-light reflex – Different vessel width – Wide intensity spectrum – Pathological objects nearby Mainly, we use hand-segmented images

GbR ’07 Department of Computer Science Computer Vision & Pattern Recognition Group 4 Rothaus, Rhiem and Jiang: Separation of the Vascular Graph in Arteries and Veins Graph-based representation of the vasculature Input: Binary vessel image Output: Vasculature graph 1. Compute the skeleton of the vasculature 2. Classify skeleton pixel in – End pixel (form vertices of degree 1) – Connection pixel (form edges) – Branching pixel (form vertices of degree 3) – Crossing pixel (form vertices of degree 4) 3. Construct graph-based representation Arising Problems: – Segmentation errors could lead to small cycles – Discontinuous segmentation leads to an over- fragmented graph representation – Skeleton of a crossing could lead to two branches binary vessel imageskeleton imagevasculature graph

GbR ’07 Department of Computer Science Computer Vision & Pattern Recognition Group 5 Rothaus, Rhiem and Jiang: Separation of the Vascular Graph in Arteries and Veins SAT-Problem Specification (vessel labelling) Problem: Classify each vessel as artery ( a ) or vein ( v ) Mainly recent approaches are based on local features – Colour, cross-profile, thickness, etc. – Work only good for thick vessels nearby the optic disc We propose a structure-based approach (on vasculature graph) – Label each vessel segment v i as artery (L i = a ) or vein (L i = v ) – Formalise anatomical properties of the vasculature: 1. At branches only edges of the same labelling are involved 2. At crossings an artery crossing a vein – Construct logical clauses that describe the properties – Cumulate above rules for all vertices and formulate the SAT-problem – Solve this as a CSP (Constraint Search Problem) with AC-3 a a a v v v v v a a a a v v

GbR ’07 Department of Computer Science Computer Vision & Pattern Recognition Group 6 Rothaus, Rhiem and Jiang: Separation of the Vascular Graph in Arteries and Veins The labelling process (AC-3*) 1. Add the incident vertices of few manually labelled vessel segments in the process queue Q 2. While Q is not empty – Take the first vertex and corresponding logical rule – Reduce set of labels of the incident vessels consistent to the rule – If there is a conflict try to solve it (details later) – Otherwise add the new vertices to Q Order of processing the vertices (rules) is important conflict manual label conflict manual label

GbR ’07 Department of Computer Science Computer Vision & Pattern Recognition Group 7 Rothaus, Rhiem and Jiang: Separation of the Vascular Graph in Arteries and Veins Q={ v 6 }Q={ v 3, v 8 }Q={ v 4, v 8 }Q={ v 8, v 7 }Improvement: Introduce an intelligent initial edge labelling to detect split crossings Q={ v 7 } conflict

GbR ’07 Department of Computer Science Computer Vision & Pattern Recognition Group 8 Rothaus, Rhiem and Jiang: Separation of the Vascular Graph in Arteries and Veins Operations for graph manipulation (edge labelling) Segmentation or skeleton errors lead to unsolvable SAT-problem Graph structure has to be manipulated slightly Allowed operations should handle: 1. Split crossings (instead of 1 deg. 4 vertex 2 adjacent deg. 3 vertices) 2. Missing segments (crossing degenerated to vertex of degree 3) 3. Falsely detected branches 4. Falsely detected segments Instead of manipulating the graph directly we introduce a second order labelling (edge labelling): vessel labelling resolve problemlabel graph manipulation op11 c melt 2 branches to one crossing op22+3 e split a branching op34 f delete an edge –– n nothing (normal edge) vasculature graph edge labelling

GbR ’07 Department of Computer Science Computer Vision & Pattern Recognition Group 9 Rothaus, Rhiem and Jiang: Separation of the Vascular Graph in Arteries and Veins Steering the labelling process (Belief propagation) Plausibility weights [0,1] for each vertex – Assign crossing vertex the plausibility 1 - P 1 (d) – Assign branch vertex the plausibility (with β = max α i ) P 1 (d) + P 2 (β) - P 1 (d) P 2 (β) Plausibility weights [0,1] for each a / v -labelled vessel – Assign hand-labelled vessels plausibility 1 – During AC-3* algorithm use a multiplicative propagation scheme (with weights of corresponding vertex and edge) Use weights as heuristic to order Q as priority-queue Use the average vessel weights to rate labelling results P1(d)P1(d) P2(β)P2(β)

GbR ’07 Department of Computer Science Computer Vision & Pattern Recognition Group 10 Rothaus, Rhiem and Jiang: Separation of the Vascular Graph in Arteries and Veins Initial edge labelling Decide on plausibility measures P 1 (d) and P 2 (β) if a connection edge between to branches is probably a crossing No false c -label should be introduced Label edge with c -label iff [ d<3 ] or [ P 1 (d)<0.75 and P 2 (β)< P 2 (30°) ] CN C GT N GT 0621 Confusion matrix on 10 training images Accuracy of >96 %

GbR ’07 Department of Computer Science Computer Vision & Pattern Recognition Group 11 Rothaus, Rhiem and Jiang: Separation of the Vascular Graph in Arteries and Veins Solving Conflicts Conflicts cannot been avoided (even not with initial labelling) Conflicts are basically introduced by cycles in the vascular graph Topology is responsible for conflicts Solving-strategy: – Search cycle (vertex set V ’), where all vessel labels are defined – Establish edge candidate set E ’={ e | e incident to a v in V ’ } – Choose a “suitable” n -labelled edge of E ’, with minimum weight and change edge label to c (crossing) – Otherwise label the conflict edge with e (end-segment) – Restart the AC-3* algorithm

GbR ’07 Department of Computer Science Computer Vision & Pattern Recognition Group 12 Rothaus, Rhiem and Jiang: Separation of the Vascular Graph in Arteries and Veins Interactive labelling tool Requirement: binary vessel image Physician mark single vessel segments as arteries an veins Propagation of the manual labelling as far as possible Solve logical conflicts automatically If the result is not good enough for the observer, more vessel label could be manually added Presenting results in two different ways: artery (auto.)vene (auto.)artery (man.)vene (man.) Original image Binary image

GbR ’07 Department of Computer Science Computer Vision & Pattern Recognition Group 13 Rothaus, Rhiem and Jiang: Separation of the Vascular Graph in Arteries and Veins Results on manually segmented images STARE data set of A. Hoover et al. image im0082 manuel label init. c -labelfinal c -labelfinal e -label solved confl.avg. weight / manuel label init. c -labelfinal c -labelfinal e -label solved confl.avg. weight /90.21

GbR ’07 Department of Computer Science Computer Vision & Pattern Recognition Group 14 Rothaus, Rhiem and Jiang: Separation of the Vascular Graph in Arteries and Veins Discussion results on manual segmentations imagemanuel label init. c -labelfinal c -labelfinal e -label solved confl.avg. weight / / / / / / / Most conflicts could be solved by introducing c -label Only few conflicts could not been solved Problematic regions are even hard to been labelled by experts Normally few hand-labels are necessary

GbR ’07 Department of Computer Science Computer Vision & Pattern Recognition Group 15 Rothaus, Rhiem and Jiang: Separation of the Vascular Graph in Arteries and Veins Results on automatic segmentations Method of Soares et al. and test database DRIVE of Staal High demands on segmentation algorithm: Different vessel width, no gaps in segmentation, low false positive rate, etc. Some segmentations leads to poorly connected graphs (less rules)

GbR ’07 Department of Computer Science Computer Vision & Pattern Recognition Group 16 Rothaus, Rhiem and Jiang: Separation of the Vascular Graph in Arteries and Veins Summary and Conclusions We have developed a method for propagating vessel classification Requirement is a binary vessel image Problem is formulated as Constraint Search Problem Arising conflicts are solved by manipulating graph structure Interactive environment is developed for physicians Methods works good for tested image databases Quality depends strongly on segmentation result Further works – Statistical foundation of plausibility function – Realise initial labelling with Bayesian classifier – Justify method by comparison with ground-truth data – Enhance conflict solver – Classify strong vessel automatically as artery or vein – Integrate method in a framework for vascular structure analysis

GbR ’07 Department of Computer Science Computer Vision & Pattern Recognition Group 17 Rothaus, Rhiem and Jiang: Separation of the Vascular Graph in Arteries and Veins Final slide Thank you for your attention! Are there any questions?

GbR ’07 Department of Computer Science Computer Vision & Pattern Recognition Group 18 Rothaus, Rhiem and Jiang: Separation of the Vascular Graph in Arteries and Veins e -label: Splitting a branch (op2) Case 1: crossing/ branch: Case 2: branch/ branch: Handles segmentation errors: – Missing vessel segment on one side of a crossing – Vessel ends nearby another vessel (falsely detected branch) Evident interpretation in case 1 (crossing/ branch) – Eliminate logical clauses of the branch (with the e -labelled vessel) Difficult interpretation in case 2 (branch/ branch) – We use a FCFS strategy, choose the 1 st label and ignore 2 nd – Adjust set of logical clauses dynamically Type of segmentation error cannot be stated