M. Sznaier O. Camps Robust Systems Lab Dept. of Electrical and Computer Eng. Northeastern University Compressive Information Extraction TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A A AA A AA A Robust Identification of Hybrid Systems C. Lagoa Dept. of Electrical Eng. Penn State University

Hybrid systems in control: Controller design reasonably well understood Identification/Validation: work in progress u y GσtGσt η

What do these have in common?: Detecting gene activity in a diauxic shift Human tracking and activity analysis Tumor detection in low contrast images In all cases, relevant events comparatively rare and encoded in 1/100 to less than 1/10 6 of the data

What do these have in common?: Detecting gene activity in a diauxic shift Human tracking and activity analysis Tumor detection in low contrast images Claim: A hidden hybrid systems identification problem

Strong prior: – Signal has a sparse representation only a few c i ≠ 0 Signal Recovery: – “sparsify” the coefficients Relax to LP: Compressive Sensing:

Where should we pay attention?: Features (edges, regions, etc.) are important.

Where should we pay attention?: Dynamics are important too!

Strong prior: – Signal has a sparse representation only a few c i ≠ 0 Signal Recovery: – “sparsify” the coefficients Relax to LP: Compressive Sensing:

Strong prior: – Signal has a sparse representation only a few c i ≠ 0 Signal Recovery: – “sparsify” the coefficients Relax to LP: Compressive Sensing: Strong prior: – Actionable information is generated by low complexity dynamical systems. Information extraction: – “sparsify” the dynamics Relax to SDP: Compressive information Extraction

Information extraction as an Id problem: – Model data streams as outputs of piecewise LTI systems – “Interesting” events  Model invariant(s) changes – “Homogeneous” segments  output of a single LTI sub-system u G(  ) y features, pixel values, …

Piecewise Affine (PWA) Systems Id problem Given : – Bounds on noise (|| η|| * · ² ), sub-system order (n o ) – Input/output data (u,y) Find: – A piecewise affine model such that

Piecewise Affine (PWA) Systems Id problem Ill posed, always has a trivial solution Given : – Bounds on noise (|| η|| * · ² ), sub-system order (n o ) – Input/output data (u,y) Find: – A piecewise affine model such that

Given : – Bounds on noise (|| η|| * · ² ), sub-system order (n o ) – Input/output data (u,y) Find: – A piecewise affine model such that with minimum number of switches systems Piecewise Affine (PWA) Systems Id problem

Non-zero g(t) = SWITCH Main idea : PWAS Id problem with min # switches:

Min # switches min||g|| o Main idea : A sparsification problem PWAS Id problem with min # switches:

Formally : PWAS Id problem with min # switches:

Formally : FACT: “exact” solution tktk t k+1 PWAS Id problem with min # switches:

Example: Video segmentation

PWAS Id problem with fixed # subsystems: Activity Analysis Need to tell when we are back to the original system Medical Image Segmentation

Given : – Bounds on noise (|| η|| * · ² ), sub-system order (n o ) – Input/output data (u,y) – Number of sub-models Find: – A piecewise affine model such that: PWAS Id problem with fixed # subsystems: NP-hard, MILP (Bemporad et. Al.)

Given : – Bounds on noise (|| η|| * · ² ), sub-system order (n o ) – Input/output data (u,y) – Number of sub-models Find: – A piecewise affine model such that: PWAS Id problem with fixed # subsystems: Reduces to a rank minimization problem

Given : – Bounds on noise (|| η|| * · ² ), sub-system order (n o ) – Input/output data (u,y) – Number of sub-models Find: – A piecewise affine model such that: PWAS Id problem with fixed # subsystems: Reduces to a SDP

PWAS Id problem in the noise free case: Neither the mode signal nor the parameters, b, are known! Independent of mode signal, linear in parameters, c! GPCA: an algebraic geometric method due to Vidal et al. Main Idea: * = 0

Toy example: 2 first order systems:

Function of the data only System parameters Independent of the data One such equation per data point

GPCA: an algebraic geometric method due to Vidal et al. Main Idea: PWAS Id problem in the noise free case: Embed in a higher dimensional space via Veronese map

GPCA: an algebraic geometric method due to Vidal et al. Main Idea: PWAS Id problem in the noise free case: Solve for c s from the null space of the embedded data matrix. Get b i from c s via polynomial differentiation Details in Vidal et al., 2003

GPCA: an algebraic geometric method due to Vidal et al. Main Idea: What happens with noisy measurements? Solve for c s from the null space of the embedded data matrix. Get b i from c s via polynomial differentiation toto η t η T η η t

GPCA: an algebraic geometric method due to Vidal et al. Main Idea: What happens with noisy measurements? Solve for c s from the null space of the embedded data matrix. Get b i from c s via polynomial differentiation Need to find the null space of a matrix that depends polynomially on the noise t η

GPCA: an algebraic geometric method due to Vidal et al. Main Idea: What happens with noisy measurements? Solve for c s from the null space of the embedded data matrix. Get b i from c s via polynomial differentiation Need to find the null space of a matrix that depends polynomially on the noise. Obvious approach: SVD toto η t η T η η t

Academic Example Noise bound: 0.25

GPCA: an algebraic geometric method due to Vidal et al. Main Idea: What happens with noisy measurements? Solve for c s from the null space of the embedded data matrix. Get b i from c s via polynomial differentiation Need to find the null space of a matrix that depends polynomially on the noise. Minimize rank V s w.r.t η t toto η t η T η η t

Detour: Polynomial Optimization Theorem: (P1) and (P2) are equivalent; that is: From Lasserre 01:

Detour: Polynomial Optimization Theorem: (P1) and (P2) are equivalent; that is: From Lasserre 01:

Detour: Polynomial Optimization From Lasserre 01: Affine in m i

Detour: Polynomial Optimization From Lasserre 01: Affine in m i Hausdorff, Hamburger moments problem. Set of LMIs.

Rank is not a polynomial function. Can we use ideas from polynomial optimization? – YES Optimization Problem 1: What happens with noisy measurements?

Rank is not a polynomial function. Can we use ideas from polynomial optimization? – YES Optimization Problem 1: What happens with noisy measurements? Optimization Problem 2:

Rank is not a polynomial function. Can we use ideas from polynomial optimization? – YES Optimization Problem 1: What happens with noisy measurements? Optimization Problem 2: Convex constraint set!!

Rank is not a polynomial function. Can we use ideas from polynomial optimization? – YES Optimization Problem 1: What happens with noisy measurements? Optimization Problem 2: Fact: – There exists a rank deficient solution for Problem 2 if and only if there exists a rank deficient solution for Problem 1. – If c belongs to the nullspace of the solution of Problem 2, there exists a noise value with such that c belongs to the nullspace of

Rank is not a polynomial function. Can we use ideas from polynomial optimization? – YES Optimization Problem 1: What happens with noisy measurements? Optimization Problem 2: Problem 2 – Matrix rank minimization – Subject to LMI constraints Use a convex relaxation (e.g. log-det heuristic of Fazel et al.) to solve Problem 2 Find a vector c in the nullspace Estimate noise by root finding (V s c = 0 polynomials of one variable) Proceed as in noise-free case

Academic Example

Noise bound: 0.25 Academic Example

Parameter estimation trueMoments-basedGPCA p Submodel 1p p p Submodel 2p p p Submodel 3p p Error ||Δp|| 2 Moments-based: GPCA:

Example: Human Activity Analysis WALKBENDWALK

Example: Recovering 3D Geometry Example: denoising and clustering

Example: Image Segmentation Original image GPCA segmentation “dynamics” based segmentation

Given: – A nominal hybrid model of the form: – A bound on the noise (||η|| ∞ ≤ε) – Experimental Input/Output Data Determine: – whether there exist noise and switching sequences – consistent with a priori information and experimental data Reduces to SPD via moments and duality Model (In)validation of SARX Systems Equivalent to checking emptyness of a semialgebraic set

Semi-algebraic Consistency Set

One of the submodels is active at time t (logical OR) Semi-algebraic Consistency Set * = 0

Semi-algebraic Consistency Set The model is invalid if and only if is empty. Possible to use Positivstellensatz to get invalidation certificates. However, easier to utilize problem structure via moment-based polynomial optimization: Model is invalid iff o*>0

Certificates for (In)Validation The model is invalid if and only if there exist an N such that solution of the moments-based relaxation is positive! Numerically, it is easier to examine the dual problem – Model is invalid if where p* is the maximum of the dual SDP of the N th order relaxation.

Problem has a sparse structure (running intersection property holds) Details in Lasserre 06 Polynomial Optimization pnapna +P n a +1 PTPT + … + No need to consider all cross moments!

Problem has a sparse structure (running intersection property holds) A moments-based relaxation (convergent as N ↑ ): Convex SDP! Polynomial Optimization standard relaxation: O((Tn y ) 2N ) variables exploiting structure: O((n a n y ) 2N ) variables

Example: Activity Monitoring Set of “normal” activities: walking and waiting Estimate center of mass with background subtraction Identified model for walk: Model for wait: Training sequence for WALK

Example: Activity Monitoring A priori hybrid model: walking and waiting, 4% noise Test sequences of hybrid behavior: WALK, WAIT RUN WALK, JUMP Not InvalidatedInvalidated

Identifying Sparse Dynamical Networks Who is in the same team? Who reacts to whom?

Given time series data: What causes what? (Granger causality) Are there hidden inputs? Identifying Sparse Dynamical Networks

Formalization as a graph id problem: Given time series data: What causes what? (Granger causality) Are there hidden inputs?

Each time series becomes a node in the graph Formalization as a graph id problem:

Each time series becomes a node in the graph Formalization as a graph id problem: Each link series becomes a system = a1a1 a2a2 a3a3 ??? ?

Problem Formulation In matrix form: where and for a single node

Problem Formulation Complete network structure can be written as where

A Sparsification Problem: Find block sparse solutions to: Heuristic solution: group-lasso:

A Sparsification Problem: Find block sparse solutions to: A better heuristics: re-weighted group-lasso: link strength 1 st order difference of u u ∂u

Distributed solution: Solve for one node at a time: Complexity of the global solution O(P 8 (PN+T) 8 ) vs Complexity of the distributed solution O(P (PN+T) 8 )

Examples Australian Open Doubles Tennis Final game Network identified using Distributed Method Network identified using Competing Method

Examples Interesting structure is unveiled: A strong relationship was identified between the Brazilian Real and Canadian Dollar, the 8th and 9th largest world economies, respectively. The strongest connection of the Australian Dollar is with the New Zealand Dollar, as it would be expected from geographical proximity. One of the strongest connection of Chinese Yuan is to the United States Dollar. This is expected since the Yuan is indeed pegged to the US Dollar, rather than floating freely.

Examples Our method is capable giving more insight through identified external inputs The two identified jumps at Sep/10/2007(A) Oct/3/2007(B) Google Insight Results with headlines for ‘Brazil Real’ Keyword A)’Brazil real gains beyond 1.9 per dollar on inflows’ by Reuters on Sep/13/2007 B)’Brazil Real Strengthens Beyond 1.80 Per Dollar, a 7-Year High’ by Bloomberg Oct/11/2007

Finding “coordinated”activites: A “maximal clique” type problem

Finding “coordinated”activites: Rank minimization with sparsity constraints

Strong prior: – Signal has a sparse representation only a few c i ≠ 0 Signal Recovery: – “sparsify” the coefficients Relax to LP: Compressive Sensing: Strong prior: – Actionable information is generated by low complexity dynamical systems. Information extraction: – “sparsify” the dynamics Relax to SDP: Compressive information Extraction

– Data as manifestation of hidden, “sparse” dynamic structures – Extracting information from high volume data streams: finding changes in dynamic invariants (often no need to find the models) – Dynamic models as very compact, robust data surrogates – An Interesting connection between several communities: Control, computer vision, systems biology, compressive sensing, machine learning,…. Dynamic models as the key to encapsulate and analyze (extremely) high dimensional data Compressive Information Extraction:

Acknowledgements: Many thanks to: – Workshop organizers – Students Dr. N. Ozay, M. Ayazoglu – Funding agencies (NSF, AFOSR, DHS) More information as