To Ping or Not to Ping An exercise working with a large volume of electronic data.

Viewing Data Can be intimidating Can be confusing Can be limited Can be time consuming Can be frustrating Can simply be SCARY

Intimidation

Intimidating The next couple of slides show large volumes of data that were submitted for an audit recently completed. The first slide shows three Zip Files. Each file contains data for January, February and March. The second slide shows the January file opened. The third slide is the January 2 nd text file opened.

Intimidating cont. Zip file for Jan. Feb. and March.

Intimidating cont. Data for each day of the month

Intimidating cont. Data from January 2 nd. There are 319,876 lines of data.

Confusing What to do? – Through discussions, along with other items you should determine: Column Headings; What the data means; Is the information used for reporting; and If available, the raw data.

Confusing cont. Looking back at the previous slide: – There are no column headings; – There is no consistency in column length; – Portions just look messy and hard to read or understand.

Limitations The files received are text files. Working with Distance software programs may require conversions into other formats. Such files include: – Excel Files; – Access Files; – CSV Files.

Re-Formatting When working with different formats, you will need to know any limitations you may have. We found that when working in Excel we were limited to the number of lines of data that could be saved. The next slide shows data in a text file from January 1 st. There are 151,483 lines of data.

Re-Formatting cont.

Reformatting cont. The Excel file only saved 65,536 lines of data. As a result we used an access file.

Time Consuming/Frustrating Since we were not able to use Excel, the files were re-formatted using access files. As part of the re-formatting, each of the monthly files were combined into one file. By adding this step, vehicle data for each month was consolidated for audit purposes. This step was time consuming and frustrating due to lack of experience working with access.

Access File There are 10,634,152 lines of data in this file.

INTIMIDATING!!!

Time Consuming/Frustrating With experience – time decreased; – Frustration due to inexperience was also reduced, if not eliminated. Although a considerable amount of time was spent re-formatting the files, time needed to route the sample vehicles was greatly reduced.

Back to limitations After completion of the audit, system updates decreased excel limitations. The next couple of screen shots shows the January 2nd data. In both cases you will see that all 319,876 lines of data have been captured from the original text file.

Limitations Sorted by Date and time.

Limitations Sorted by unit.

What’s Next? Plot the Pings? How? – One point/location at a time? Or – load large volumes at once?

What’s Next? Answer- LOAD DATA AT ONE TIME!!! Many of the providers of distance programs provide for uploading batch data. You may need to contact each to find out what capabilities may be available. You may need to obtain additional products or resources. There may be limitations.

Plotting the Pings Starting with the access file: – Open the file; – Find the unit number. Unit 1274 was selected; – Filter the selection; At this point only the data for unit 1274 remains: You may need to sort by some other criteria. In this case, we sorted by the date/time to sequence the event properly.

Plotting the Pings

Select and copy the pings.

Plotting the Pings Open the distance program and paste the list.

Plotting the Pings Run the Pings.

Plotting the Pings Generate reports.

Summary Working with large volumes of Data can be scary, intimidating and at times frustrating. Using the tools and resources available can lesson the intimidation and shorten the time needed to complete your audit engagement. Communication, experience and understanding your limitations can reduce frustration.

It’s not that Scary

Questions??? Questions Comments If time permits, live demonstration.