Download presentation
Presentation is loading. Please wait.
1
Preparing your Data using Python
Samuel G. Mori, CISA Managing Partner, Analytics & Advisory Services Spyrion LLC April 12, 2018
2
Background Samuel G. Mori, CISA, Six Sigma Green Belt
Managing Partner, Analytics and Advisory Services Software Quality Assurance, Internal/External Audit, Business Intelligence and Reporting, Advisory Services (GRC and Analytics) Subject matter expertise within commercial, manufacturing, healthcare, biomedical and entertainment sectors B.S. Cognitive Science – Human Computer Interaction (UC San Diego) M.S. Accountancy – Accounting Information Systems (San Diego State) M.S. Data Science – Analytics & Modeling (Northwestern)
3
Agenda Learning Objectives Why should I prepare my data?
What types of data might I encounter? How can Python help me?
4
Learning Objectives Understand the importance of preparing your data for analysis Understand different types of data formats you may encounter Understand what Python is and why you should use it Understand strategies and techniques for importing, preparing, and saving your data using Python
5
Why should I prepare my data?
Garbage in, garbage out Reduce errors Remove duplicate records Fix missing values Correct range values Fix formatting (i.e. date, text, number)
6
Experience Check How many people have experience with Python?
What types of data formats do you use in your organizations? CSV, Excel, PDF, JSON, XML, SQL databases, etc What types of tools do you use? Excel, ACL, IDEA, SQL Server, Python, R, SAS, Cognos, etc
7
What types of data formats might I encounter?
Comma Separated Value (CSV) Excel JavaScript Object Notation (JSON) Structured Query Language (SQL) And more! Python can help with these!
8
CSV Example SFO Airport Survey Results
9
Excel Example SFO Airport Survey Results
10
JSON Examples Trip Advisor JSON file Yelp JSON file
11
SQL Example Sample Customer Data
12
What is Python? Definition
Object-oriented, high-level programming language Used as a scripting or glue language to connect existing components together Simple, easy to learn syntax emphasizes readability Supports modules and packages Python interpreter and the extensive standard library are FREE!
13
What is Python? (cont.) Key Python Package: Pandas
Open source library that allows you to work with CSV, Excel, JSON, and SQL database files, pull them into tables (called dataframes), and perform various data analysis techniques.
14
Coding Basics Some basic python syntax to keep in mind:
Declaring a variable (always to the left of equal sign) File names (can use “ “ or ‘ ‘) dataframe = pd.read_excel(‘file_name.xlsx', ‘sheet_name’) Or file_name = ‘file_name.xlsx’ sheet_name = ‘sheet_name’ dataframe = pandas.read_excel(file_name, sheet_name)
15
Coding Basics (cont.) Some basic python syntax to keep in mind:
Using library packages Import pandas as pd #calling pandas library and creating reference ‘pd’ dataframe = pd.read_excel(‘file_name.xlsx', ‘sheet_name’) Or dataframe = pandas.read_excel(‘file_name.xlsx', ‘sheet_name’)
16
SFO Airport Customer Survey Data – Excel & CSV files
Case Study SFO Airport Customer Survey Data – Excel & CSV files
17
Importing the Data How do I import an Excel file?
18
Data Characteristics What columns do we have?
19
Data Characteristics What if I just want a subset of these columns?
20
Data Characteristics What columns do I have and what are their data types?
21
Data Characteristics How many columns and records do I have? Can I do a count of different values within a column?
22
Modifying Data Values Lets look at the data dictionary How do I replace values to make them meaningful?
23
Saving to Excel How do I save this new file? What does my file look like?
24
Importing the Data How do I import a CSV file? What is NaN?
25
Fixing Error Values How do I fix NaN values?
26
Adding Custom Columns What if I want to add the Year in a column?
27
Identifying Value Ranges
How do I look at the data value ranges for multiple columns?
28
Saving to CSV How do I save this new file?
What does my file look like?
29
Appendix
30
Additional Information
Python Development Environments Enthought Canopy Anaconda/Spyder Python Libraries Pandas
31
Questions?
Similar presentations
© 2024 SlidePlayer.com. Inc.
All rights reserved.