Rutgers University - Center for Vector Biology Data Management and Manipulations: The Good, the Bad and the Fuhgeddaboudit ! Lisa Reed Center for Vector Biology Rutgers University 16 November 2011 Biologists Meeting

Rutgers University - Center for Vector Biology What this talk is about Data Management –Why is it important? –How to do it effectively –How to protect your data Data Manipulations –What is it and what can it do for me? Pivot and Graphs –Connectivity and Reports

Rutgers University - Center for Vector Biology Data Management Broad definition –Data Resource Management is the development and execution of architectures, policies, practices and procedures that properly manage the full data lifecycle needs of an enterprise -Wikipedia –How you keep data in a clear, concise and safe manner that anyone can retrieve at a later date without confusion or frustration. Data structure Meta data Safe keeping and retrieval

Rutgers University - Center for Vector Biology Data Structure Data structure is a “way of storing and organizing data [in a computer] so that it can be used efficiently.” (Wikipedia) –Should be easy to understand (for example, by date) How is data stored? –Records: One line of data arranged by “fields” or “variables.” –Arrays: A set of data arranged by position to denote variables How does Excel do it? –Records

Rutgers University - Center for Vector Biology Excel Navigation – The Top Office Button Active Cell Formula Bar Row Number Name Box Ribbon Column Letter Split Pane Multiple Tabs

Rutgers University - Center for Vector Biology Variables into Columns First Row are Variable Names –Use names that make sense –Code –Format (date) A right mouse click on either column or cell > Format cells…

Rutgers University - Center for Vector Biology Data Structure Expandable in both directions to 16,348 columns and 1,048,576 rows

Rutgers University - Center for Vector Biology Things to Consider in Your Dataset Do I use Zero? –Pro = you fill in each potential place setting with a value. –Pro = average calculate correctly! –Con = Can take a long time (but there is a short cut) –Con = Each entry is a possible data error Do you have a way to distinguish when the trap is out of commission? –Use of period or other non-numerical value

Rutgers University - Center for Vector Biology Calculations with and without Zero Blanks are NOT considered zero. Sums are counted correctly (can define average through a pivot table). Can replace blanks with zeroes. Highlight>Find and Replace>Go To Special>Select Blanks>type 0>Control-Enter

Rutgers University - Center for Vector Biology Meta Data Meta = “About” –What are the data elements? –Descriptions about how the data is organized? –Coding and their meanings Everything someone needs to know in order to use the dataset Best place to put this information is in the dataset itself Excel provides an good format for doing so.

Rutgers University - Center for Vector Biology MetaData

Rutgers University - Center for Vector Biology Saving Your Data Save it. Save it Often. Save it with AutoSave. Back it up. –Keep a copy in a safe place. –Keep a copy in a separate place. –Keep a copy in the cloud. “We have recently seen some really nasty malware on several computers that simply was not removable by numerous sophisticated software tools at our disposal. The only safe remedy was to completely rewrite the machines from scratch.”

Rutgers University - Center for Vector Biology Data Manipulations Exploratory Data Analysis vs. Confirmatory Data Analysis –What is your data’s story? –Hand entry – Know your data. –Summarizing data Tables Graphs Pivot Tables Graphs Into a Document

Rutgers University - Center for Vector Biology Pivot Table 1 Highlight Data (hold down shift key and arrow down, then arrow across) INSERT Pivot Table

Rutgers University - Center for Vector Biology Pivot 2 A new sheet is created Can be modified Lists variables in your dataset Has place for column, row, value and filter

Rutgers University - Center for Vector Biology Drag Variables to Label, Value and Filters Drag Species to Row Labels and it will place the species names, one to each row Drag Results to Column Label Drag Pathogen to Filter Drag Moscount to Values

Rutgers University - Center for Vector Biology Manipulation of variables

Rutgers University - Center for Vector Biology Pivot Example Excel

Rutgers University - Center for Vector Biology Graphing Your Data Highlight your data. Apply a graph template. The greatest value of a picture is when it forces us to notice what we never expected to see. –Tukey 1977

Rutgers University - Center for Vector Biology Live Links from One Document to Another With “hot” links, your information will always update. –Saves time –Preserves method You can also paste a static picture that is not linked to data. Click on graph<Copy<Go to Document<Paste Special<Paste Link<Excel Object –Graph will be shown and can be updated with a right click<Update.

Rutgers University - Center for Vector Biology Graphs into Word

Rutgers University - Center for Vector Biology Conclusions Data Management will provide continuity through data structure, meta data and data preservation. Exploring data can provide insight and new questions. You can link these explorations into different documents.

Rutgers University - Center for Vector Biology Data Management and Manipulations: The Good, the Bad and the Fuhgeddaboudit ! Lisa Reed Center for Vector Biology Rutgers University 16 November 2011 Biologists Meeting