Archived Data User Services (ADUS)

Archive Creation Summary Data to be stored –Level of aggregation –All, or some subset of data available Quality control Aggregation steps

Archive Creation Summary (2) User access –Communications mechanism –Privacy –Directions for use Truth in Data –Meta data

Archive Creation First question: What data is to be stored? –Raw data –Summary statistics –Both –For example Volume and lane occupancy, or Estimated speed

Archive Creation How much data gets stored? –All raw data –Only summary statistics –A sample of the data (raw or summary statistics) –All variables, or only some (tag IDs)

Archive Creation At what level of aggregation Lowest level collected? Individual vehicle passages 20 second intervals 5 minute intervals 15 minute intervals Higher More than one level –Does who will use the data change this decision?

Archive Creation Issues that impact decision: –What use is planned for the data? –Who uses it, and how do they access it? –How big is the storage requirement? –Cost/speed of processing raw data to a more useful form –How much additional data is needed to make the “raw” data useful? –Are there privacy concerns?

Archive Creation Example: Tag Observations –Raw data: Tag ID, location, time and date Store all of the above? Store O/D pairs? Travel times? Privacy of tag ID? Speeds? (how far between readers, exactly)

Archive Creation Example: Transit AVL Information –Raw data: Tag ID, location, time, and date –Tag ID does not equal Route, Run, and trip Also need schedule information, operations info. These relationships change every day Routes can change every schedule change, so you need historical information, not current information

Archive Creation How is aggregation performed? –Quality control –Assumptions made

Quality Control Not all collected data is valid Can the archive identify bad or questionable data? How do you indicate these judgments? –How do you inform users of these conditions?

Quality Control How do you identify “bad” data? –Sensor output –Checks against historical data –Checks against expected ranges –Other comparisons

Quality Control What do you do with “questionable” data? –SR-520 during the 1-lane closure after barge accident What resources are needed to investigate “questionable” data? Does lack of these resources change your willingness to share data?

Quality Control How do you handle missing/bad data? Does this change if you are –Storing raw data –Only storing summary data –Storing both

User Access Who gets access to the data? How do users get access to the data? Can they access all data, or only summary data?

User Access How do you communicate –What data (variables) are available –What geographic locations are available –What quality issues exist –How the data can (should) and can not (should not) be used Volume data from stop bar loops

User Access Meta Data –Data about data Truth-in-Data –The principal that says you will be honest with users about What data are real What data are interpolated What data missing and have/have not been replaced, and how those data were replaced/estimated

User Access Do you trust users to use the data correctly? –At what level of summarization? –Site specific data isn’t always representative of reality How easy do you make their retrieval of data? –Cost implications of that task –Political benefits/costs of providing access

User Access Mechanism used to provide access –CD-ROM –Web access –File transfer on request –Real time data transfer Cost to user for access?

Communications How do you communicate with potential users? –Staff time –On-line help –You don’t –Other

Privacy Privacy concerns grow with increased user access and sensitivity of the data being collected –Personal IDs Vehicle tags Driver identification (union issues)

Who Pays? ITS systems are traditionally paid for by those who operate the system Often the greatest use for the archive is from a different group –Control of resources –Ownership –Willingness to cooperate