Process Query Systems ENGS 112 Lecture 7

Process Query Systems (PQS) vs Data Base Systems (DBS) Data Base System Process Query System Data Sources Data Base System Field Oriented Data Field Oriented Query (eg SQL) Business Requirement Query Responses Data Sources Field Oriented Data Data Base System Business Requirement Process Oriented Query Process Query Responses Process Query System

When do we use “processes”? Is there a large ground vehicle convoy moving towards our position? (Tactical C4ISR) Is there an unusual pattern of network and system calls on a server? (Cyber-security) Is there a pattern of unusual document accesses within the enterprise document control system? (Insider Threat Detection) Is there a pattern of unusual transactions? (Homeland Security) Is my software operating normally? (Autonomic computing) Is the workflow system working normally? (Business Process Engineering) IMPORTANT – All are “adversarial” processes, not cooperative so the observations are not necessarily labeled for easy identification and association with a process!

How are processes defined or specified? (viz SQL) A “state”State transition An “observation” Non-branching process (A then B then C then D then …) Branching process (A then (B or C or D) then (if B then E or F) or if then …)

How are processes defined or specified? (viz SQL) cont’d A “state”State transition An “observation” Continuous kinematic processes (constrained by physical laws) kinematic state at time t kinematic state at time t +  t F = ma Laws of motion Markovian models (eg, source code generators and channel models) p( s(t) = s i | s(t-1) = s j ) = p ij, p( o(t) = o k | s(t) = s j ) State (observation) at time t is s(t) (o(t), state space = { s i }, observation space = { o j } Petri network models

Process descriptions Using RDF or DAML ontologies Describe states, possible state transitions and observables using either RDF/DAML and/or remote objects and methods Detection of identifiable “allowed” processes leaves a residue of anomalies in the DBS so this can be used for anomaly-based detection as well as signature-based detection

Example – vehicle tracking Time t Time t+1 Time t+2

Example – vehicle tracking Time t Time t+1 Time t+2

Example – vehicle tracking Time t Time t+1 Time t+2

Example – vehicle tracking Time t Time t+1 Time t+2

Example – vehicle tracking Time t Time t+1 Time t+2

Example – vehicle tracking Time t Time t+1 Time t+2 Gates Predicted tracks Hypotheses

Common Logic Track-hypothesis initiation Hypothesis management – pruning Handling missed detections Handling track termination Subscription via gates Publication of hypothesized tracks

Process Query System algorithms Continuous kinematics – Kalman filtering Markov processes – Viterbi-like algorithms Multiple process disambiguation – multi- target multiple hypothesis tracking (MHT) algorithms Model estimation and optimization - Estimation-Maximization (EM) Methods Many-to-one (multispectral sensing) and one-to-many (coincidental evidence) observation-to-process resolution methods

Generic PQS Operation 1.A user defines a “query” in terms of a process description 2.The process query is submitted to the PQS 3.The PQS searches the DBS for evidence of process instances (may be more than 1) 4.The evidence consists of ordinary DBS records 5.The PQS manages the complexity of having multiple instances of the processes 6.The PQS maintains a pool of the most likely hypotheses about which processes exist and what their states are

A PQS Implementation - TRAFEN TRacking And Fusion ENgine Currently in alpha version Handles ground vehicle tracking using acoustic sensors Handles ICMP-T3 event analysis for worm detection Uses RDF, DAML and web services for resource description, discovery and access

Data mining and Expert Systems Data mining –used to discover unknown patterns in data –can be used to define processes (front end) Expert Systems –TRAFEN implements a specialized logic for process discovery –ES’s do not handle statistical inferences as needed in this domain

To do list Graphical programming tools for processes and semantic descriptions Control/activation of processes and/or sensor infrastructure – ie. “feedback loops” Performance models and analysis (sampling rates, hypothesis growth, etc) Hierarchical process capabilities (ie, identified processes are observables of higher level processes, such as convoys are aggregations of vehicles, etc) Truly generic Kalman, HMM, MHT algorithms