TEMPORAL DATA AND REAL- TIME ALGORITHMS CHAPTER 4 – GROUP 3

TEMPORAL DATA Generated Traffic: Social Media Web Traffic (advertisement clicks) Physical: Audio and video recording

DATA ACQUISITION Data from the distributed sources must generally be collected social media, data are often analyzed as they are collected Eventual consistency is often used in large-scale distributed systems.

One does not know when all relevant data have arrived Failures and reconfigurations are common in very-large-scale monitoring systems, One cannot determine whether a data item is missing or merely late

Best strategy is generally to do as much processing as possible with the data that are available, and Perhaps recompute answers as additional data come in.

ACQUISITION TIMING Different data streams typically exhibit different temporal latencies. Temporal consistency : determining when data produce trustworthy answer

SCHEDULING ALGORITHMS Tasks that miss deadlines either break the system (hard real-time), are discarded (firm real-time), or are ignored (soft real-time) Bounded-tardiness scheduling : most appropriate way to model a real-time data analysis system Earliest-deadline first

SERVER BREAKDOWN

RECOVERY During Recovery dependent data products are updated. huge load on a temporal massive data warehouse Creates a tension in such systems between timely serving needs and the synchronization latency,

DATA PROCESSING AND REPRESENTATION Once the data is acquired and aggregated, it must be modeled so that accurate predictions can be made. Coding can be used to analyze the data, and sketching can be used to summarized the data streams. Summarization tool : sketching (native format, derived format) Most inputs can be represented in averaged snapshots over a certain period of time.

LEARNING AND INFERENCE Once the data has been appropriately represented, inferences can be made. Processing real-time temporal data has unique issues. For high accuracy, more computationally demanding and space-consuming approaches are needed.

INFRASTRUCTURE REQUIREMENTS Massive real-time data sets are resource intensive. Analysis systems require large distributed file systems with many acquisition systems directing new data over high speed connections. Need significant capital investment Data integrity solutions also become expensive and difficult to use as the system becomes large.

SYSTEM AND HARDWARE FOR TEMPORAL DATA SETS