SNS COLLEGE OF TECHNOLOGY COIMBATORE – 64035 Intelligent Data Analysis Big Data / Departme22nt of MCA

Intelligent Data Analysis (IDA) It is an interdisciplinary study for effective analysis of data used for extracting useful information, knowledge or interesting pattern from large quantities of online data use of statistical, pattern recognition, machine learning, data abstraction, and visualization tools for analysis of data and discovery of mechanism that created data Big Data / Departme22nt of MCA

Big Data / Departme22nt of MCA IDA – Processing Steps identifying a problem depending on the interest of a data analyst sources of information are identified and a subset of data is generated from the accumulated data the data set is pre-processed by removing noise, handling missing information and transforming to an appropriate forma IDA technique or a combination of techniques appropriate for the type of knowledge to be discovered is then applied to the derived data set. The discovered knowledge is then manipulatated, evaluated and interpreted Big Data / Departme22nt of MCA

Big Data / Departme22nt of MCA IDA -Tools See5 program for analyzing data and generating classifiers in the form of decision trees and/or rule sets Cubist analyzes data and generates rule-based piecewise linear models – collections of rules, each with an associated linear expression for computing a target value ILLM the tool constructs classification models in the form of rules which represent knowledge about relations hidden in data Magnum Opus finds association rules providing competitive advantage by revealing underlying interactions between factors within the data Big Data / Departme22nt of MCA

IDA –Disciplinary area Statistics Classification Prediction Modeling Pattern Regression Clustering Machine Learning study of algorithms that can learn from and make prediction of data ability to learn without being explicitly programmed Types: Supervised learning Types: Unsupervised learning Types: Reinforcement learning Big Data / Departme22nt of MCA

Big Data / Departme22nt of MCA Thank You Big Data / Departme22nt of MCA

SNS COLLEGE OF TECHNOLOGY Analytic processes and tools COIMBATORE – 64035 Analytic processes and tools Big Data / Departme22nt of MCA

Analytic processes and tools a process for obtaining raw data and converting it into information useful for decision-making Data is collected and analyzed to answer questions, test hypotheses Data Requirements Data collection Data processing Data cleaning Data analysis Result Big Data / Departme22nt of MCA

Big Data / Departme22nt of MCA Analytic processes Data Require ments Data are specified based upon the requirements Specific variables regarding a population (age, income) may be numerical or categorical Data collection collected from a variety of sources (sensors, CCTV, satellite, recording devices) It also be obtained through interviews and downloads from online sources Data processing placing data into rows and columns in a table format for further analysis (spreadsheet /statistical software) Data cleaning data may be incomplete, contain duplicates, or contain errors is the process of preventing and correcting these errors Tasks like record matching, deduplication, and column segmentation Analysis variety of techniques referred Mathematical formulas / models called algorithms may be applied to the data to identify relationships among the variables, such as correlation or causation Result Big Data / Departme22nt of MCA

Big Data / Departme22nt of MCA Thank You Big Data / Departme22nt of MCA

Modern data analytical tools Analytical tools in the market from different vendors including Amazon, IBM, Microsoft Analytical tools comprises of Business Analytics monitors the status of any relevant business component or characteristic on-demand, in real-time Data Management It handles large amount of data, different data types including unstructured data Predictive Analytics and Performance Management helps to identify trends and characteristics, both positive of negative Data warehousing it can handle the traditional, processed data, unprocessed, raw data along with live data streams Business intelligence Big Data / Departme22nt of MCA

Modern data analytical tools Hadoop – Java based framework allow for distributed processing of large data set using commodity hardware Open source data management with scale-out storage and distributed processing Big Data / Departme22nt of MCA

Big Data / Departme22nt of MCA Hadoop Ecosystem Big Data / Departme22nt of MCA

Big Data / Departme22nt of MCA Hadoop Architecture It has two main components HDFS – Hadoop Distributed File System Distributed across nodes Namenode and datanode manage storage MapReduce – Programming model for distributed processing Split a task across processors Near the data and assembles result High bandwidth Clustered storage Jobtracker and tasktracker manage process Big Data / Departme22nt of MCA

Big Data / Departme22nt of MCA Hadoop Architecture Other major components are YARN A framework for job scheduling and cluster resource management Hadoop Common Contains Java libraries and utilities required by other Hadoop modules provide file system and OS level abstractions and java files to start Hadoop Big Data / Departme22nt of MCA

Big Data / Departme22nt of MCA Hadoop – File write Big Data / Departme22nt of MCA

Big Data / Departme22nt of MCA Hadoop – File read Big Data / Departme22nt of MCA