A Graphical Monitoring and Notification Flow Language (MNFL) for User- Configurable In-Home Assistive Monitoring Alex Edgcomb Oral Qualification March 2nd, 2011

The Goal: User-configurable notification systems for in-home use Identify person at front door Fall down at homeIn bathroom for 1 hour Napping too long 2 of 25

Not enough configurability 3 of 25 GrandCare BeClose QuietCare

Too much configurability 4 of 25 X10 (Also dangerous) SmartHome

Outside too long at night Not safe to be outside at night Grandma has Alzheimer’s Want to be notified if she is for long 5 of 25

Defining the system Send a text message if someone leaves, without returning, for more than 10 minutes at night. Leaving = motion to the rightReturning = motion to the left Night = 10PM to 5AM 6 of 25

Microsoft VPL 7 of 25

Microsoft VPL If-statements with >1 variable require Join Join operator only takes two values at a time Join waits for both variables to receive data Asynchronous data sources leads to raise condition. If-statements require all variables using in If-statement and Calculations done afterward. Hard (if not impossible) to increment on a raising/falling edge 8 of 25

Lego Mindstorms v2.0 Lacks global variables. Variables cannot be read/written more than once. No sense of actual time. 9 of 25

Scratch New functionality leads to: 1.Redundant code 2.Complex code Reason: Independent behavior is not separated away. For example, add a 2 nd door: 1.Create second code block 2.Embed new timer/variable 10 of 25

RIBS and RIMS CS120B (Introduction to Embedded Systems) students take 5+ weeks to correctly describe exact behavior. inWindow = (time >= 10PM || time <= 5AM) var Time 10minWait = time Period length = 1 second 1) Define synch SM in RIBS. 2) Automatically convert to C code in RIMS. 11 of 25

MNFL SystemProsCons Microsoft VPLFlow language style is intuitive for streaming applications If-statements do not handle asynchronous data Lego MindstormEasy to learn for people of age 10+ Variables too restrictive ScratchGraphical cues indicate purpose/usage of blocks Temporal programming does not handle new functionality well RIBS and RIMSPrecise behavioral description Challenging to learn 12 of 25

MNFL Data Types 13 of 25 Boolean – True / False Tone – A pair of {frequency, amplitude} Sound – Stream of tones Picture – 2D array of pixels Video – Stream of pictures Number – Negative and positive whole numbers

MNFL Modifiers of Data Types 14 of 25 Snapshot Stream Snippet

MNFL Core blocks 15 of 25

Video demo 16 of 25 YouTube version of video:

In bathroom for over 1 hour 17 of 25

Got up 5 or more times at night 18 of 25 “Got up” = was motion, then no motion for 10 minutes

Take 1 pill in the morning 19 of 25

KPN Execution semantics Kahn processing networks (KPN) Processes communicate through channels When all input channels have a token, the process executes and outputs tokens 20 of 25 Kahn, G. The Semantics of a Simple Language for Parallel Programming. Information Processing North-Holland Publishing Company, 1974.

SDF Execution semantics Synchronous data flow Process = KPN process Channel = KPN channel Difference: Processes read and write a constant number of tokens. 21 of 25 Lee, Edward A. and David G. Messerschmitt. Synchronous Data Flow. Proceedings of the IEEE, vol. 75, no. 9, p , September, 1987.

MNFL Execution semantics Monitoring and notification network language (MNFL) Block = KPN process Connection = KPN channel Difference Block’s execute when any input channel has a token 22 of 25

MNFL block SM Inside each block is a state machine. Q = queue of inputs I = current input P = previous input O = output 23 of 25

Implementation figures (as of 02/27/11) 8,000 total lines of ActionScript 700 lines for discrete event simulation engine 400 lines for base block class 80% related to graphical appearance and behavior 50 – 600 lines for each block Largest two blocks: Send message - high configurability Fall extractor - detecting falls from silhouettes 26 fully implemented blocks Block spawn, deletion and connection creation take less than 1 ms on a single core 3.0GHz PC. 24 of 25

Conclusions and Future work EasyNotify Easy to use compared to currently available systems Core blocks contain in-home functionality MNFL Data types tailored to in-home monitoring sensors Non-restrictive block communication 1.Human subject trials to improve usability 2.Develop mechanism for users to build and integrate new blocks 3.Move computations to cloud 25 of 25