Special Topics in Database

DATABASE-STORED CODE

Accessing Databases from Application Programs Options include: Script Files APIs Database-Stored Code 3

Program code stored in database T-SQL CLR Languages (C#, VB.net, …) Good for: sharing code performance Enforcing security, business rules, audits 4 Procedure A SQL… SQL end A Application Program execute A execute B Database

Database-Stored Code, con’t... Stored Procedure T-SQL block Procedural programming features stored in database * invoked by name * accepts arguments Database Trigger T-SQL block Procedural programming features stored in database * invoked by DML (Insert, Update, or Delete) * accepts no arguments 5

Create Stored Procedure: Template 6 CREATE PROCEDURE [parameter_information] AS DECLARE /* program variables defined here */ BEGIN /* program logic goes here */ END

Example 1: Parameters, Variables, Program Logic, Printing 7 /* Retrieve a specific customer record based on customer ID */ /* If the customer lives in the state of FL, then add “-1111” after the zip code & print a message to user */ /* CONVERT is needed because we're concatenating numeric data to text data */ CREATE PROCEDURE numeric(5) = null AS varchar(12); BEGIN = = = state FROM customer_t WHERE customer_id = ‘ FL ' BEGIN + ' ' UPDATE customer_t SET postal_code WHERE customer_id print ‘Customer#: ' + zipcode changed to: ' END; GO

Example 1: Invoke and Verify 8 /* Statement below included ONLY so I can undo changes after demo. */ /* You DO NOT need to do this. */ Set implicit_transactions on GO /* Show customer 9's record before calling procedure */ Select * from customer_t where customer_id = 9; /* Call the procedure and supply the customer number */ exec = 9; --or-- exec alter_zip_and_print 9 GO /* Check to see if procedure updated customer 9's record */ Select * from customer_t where customer_id = 9; GO /* Statements below included ONLY so I can undo changes after demo. */ /* You DO NOT need to do this. */ Rollback; Set implicit_transactions off

Multiple-Record Retrievals 9  Require cursorscursors 1.Declare CURSOR 2.OPEN cursor 3.FETCH data from cursor 4.CLOSE & DEALLOCATE cursor  Use control structures to process each record WHILE … [BREAK can be used to force exit from WHILE loop] END; IF ELSE IF BEGIN END; ELSE BEGIN END;

Example 2: Cursor 10 /* retrieve customer records based on NAME; if customer(s) live in FL, ‘-1111’ is added to zip */ CREATE PROCEDURE varchar(25) AS DECLARE custcursor CURSOR FOR SELECT * from customer_t WHERE customer_name like ' % ' + ' % ' ; numeric numeric(4); BEGIN -- logic OPEN custcursor; --find customers meeting criteria IF = 0 BEGIN RAISERROR ( ' no customer found ', 10,1); RETURN; END; FETCH FROM /* code continued here… */ WHILE = 0 BEGIN -- loop thru each customer record = ' fl ' BEGIN -- processing and displaying old/new zip PRINT ' id: ' + + ', name: ' PRINT ' old zip: ' + ' ' UPDATE customer_t SET postal_code WHERE customer_id PRINT ' new zip: ' PRINT ' ======================= ' ; END; -- processing and displaying of zip FETCH NEXT FROM custcursor INTO END; -- looping through customer records CLOSE custcursor; DEALLOCATE custcursor; END; -- end stored procedure GO

Example 2: Invoke and Verify 11 Set implicit_transactions on GO Select * from customer_t where customer_name like ' %furn% ' ; Exec getcust ' furniture ' GO Select * from customer_t where customer_name like ' %furn% ' ; GO Rollback; Set implicit_transactions off

Summary: Steps to Create / Invoke Procedures  Store code that creates procedure in a script  Run and debug script in SSMS  Creates & stores procedure in database  Invoke procedure  Issue "execute" command  Include parameter values where required 12

Triggers Common Uses: Implementing Referential Integrity Complex defaults Interrelation constraints/updates Updating views Auditing business activities 2 Kinds: After Instead of 13

Create Trigger: Template 14 CREATE TRIGGER ON AS DECLARE /* program variables defined here */ BEGIN /* program logic goes here */ END

Example 1: Maintain Counter 15 /* maintain the orders placed counter in the customer table */ /* add 1 to the counter each time a customer places a new order */ CREATE TRIGGER add_order_placed ON order_t AFTER INSERT AS AS numeric(11,0) BEGIN = customer_ID FROM INSERTED; UPDATE customer_t SET orders_placed = orders_placed+1 WHERE customer_t.customer_ID END; GO

Example 1: Fire/Verify Trigger /* Statement below included ONLY so I can undo changes after demo. */ /* You DO NOT need to do this. */ Set implicit_transactions on GO /* check status of customer’s orders_placed counter before placing new orders */ /* this is ONLY for testing purposes */ select customer_id, orders_placed from customer_t where customer_id=10; /* fire insert trigger by placing orders for customer #10 */ insert into order_t values (9900, '10-OCT-01', 10); insert into order_t values (9901, '10-OCT-01', 10); insert into order_t values (9902, '10-OCT-01', 10); /* issue select statement to verify that trigger updated orders_placed counter */ /* this is ONLY for testing purposes */ select customer_id, orders_placed from customer_t where customer_id=10; GO /* Statements below included ONLY so I can undo changes after demo. */ /* You DO NOT need to do this. */ Rollback; Set implicit_transactions off

Example 2: Setting Default Value /* when a new product is added, if the standard price is NOT given, then set the product’s standard price to either: */ /* the highest standard price of a product in the same product line if one exists, OR to $300 */ IF EXISTS (SELECT name FROM sysobjects WHERE name = 'set_standard_price' AND type = 'TR') DROP TRIGGER set_standard_price; GO CREATE TRIGGER set_standard_price ON product_t AFTER INSERT AS as as as as numeric (11,0) = = standard_price FROM INSERTED; IS NULL BEGIN /* find the highest standard price for existing products in the product line of the new product */ = MAX(standard_price) FROM product_t WHERE product_line_id /* if there was a value resulting from the above query, use that value for the price of the new product */ IS NOT NULL /* if there was no value retrieved from the above query, then set the price of the new product to $300 */ ELSE = 300; UPDATE product_t SET standard_price WHERE product_id END --end IF END; -- end program GO 17

Example 2: Fire/Verify Trigger Set implicit_transactions on GO /* issue select statement to view product table before adding new products */ select * from product_t; /* fire insert trigger adding new products to the product_t table*/ insert into product_t values (999,null,null,null,null,20001); insert into product_t values (998, null, null, 123, 3, 33333); Insert into product_t values (997,null,null,null,null,44444); /* issue select statement to verify that trigger updated standard prices for products */ select product_id, product_line_id, standard_price from product_t where product_id>900; GO Rollback; Set implicit_transactions off 18

WEB-DATABASE CONNECTIVITY

Active Server Pages Server-side technology dynamicinteractivedata-driven Enables dynamic, interactive, data-driven web pages languages Scripting OR compiled languages VBScript, VB, C# Jscript, Java Perl Etc… HTML HTML for display/gathering of data ADO ADO for access to data 20

ASP vs. ASP.net ASPASP.net Language SupportScripting (limited functionality) Full-Featured Run-timeInterpreted upon executionCompiled upon development Platform SupportRequires MS web server (mainly) Can be run on Apache server UIHTML Tags (manual) Server & HTML Controls (drag/drop; click) ArchitecturePresentation and Logic together Presentation separate from logic 21

Basic Steps 22 Create New ASP.net project (Optional) Create or Customize Site.Master template (Optional) Create or Customize Default.aspx page Create New webpages (.aspx files) Labels Data Controls Buttons Create Code-Behind (.vb) files Database access Data manipulation Test/Debug/Deploy

Database Access Steps 23 Define program variables for data access objects ConnectionDim connObject = New SQLClient.SQLConnection(…..) Query / CommandDim cmdObject AS New SQLClient.SQLCommand (….., …..) Data ContainerDim contObject AS SQLClient.SQLDataReader  Data Reader  Data Adapter… Create connection to the database Open connObject.Open() Run query/command and store data in container ExecutecontObject = cmdObject.ExecuteReader Display, Maintain data via Data “Container” Controls Grid View Details View Form View List View… Release objects Close contObject.close() connObject.close()

Create New ASP.net Project 24

Example 1: View Student Info: Textbox, Gridview 25 Prompt user for student search criteria Use text boxes for entering Student search criteria Display student information based on search criteria Use GridView data control for displaying Student data

Example 1: Create Webpage 26

Example 1: Create Webpage, cont… 27

Example 1: Configure the Gridview Control 28

Database Access Code 29 'THIS CODE GOES IN THE VB FILE for the View Student Information button lblStatus.Visible = False '(1) create an ADO connection object to establish connection to the database 'first define a text string that describes database connection information Dim connectionstring As String = "server=hsb-mis-sql; database=mis4340; integrated security=sspi" 'create the ADO connection object that will establish a connection to the database Dim myConnection = New SqlClient.SqlConnection(connectionstring) '(2) make the connection to the database myConnection.Open() '(3) setup queries that get student data from database based on values the user entered Dim querystring1 As String = "select * from student where stud_id='" & txtStudentID.Text & "';" Dim querystring2 As String = "select * from student where firstname like '%" & txtFirstName.Text & "%' and lastname like '%" & txtLastName.Text & "';" Dim querystring 'Decide which query to use based on which values the user entered If (txtLastName.Text <> String.Empty Or txtFirstName.Text <> String.Empty) Then querystring = querystring2 Else querystring = querystring1 End If '(4) create an ADO command object that will contain the query command to be executed by the database Dim myCommand As New SqlClient.SqlCommand(querystring, myConnection) '(5) create an ADO container object that will hold data retrieved from database Dim myDatareader As SqlClient.SqlDataReader '(6) execute the command and store results in the datareader container object myDatareader = myCommand.ExecuteReader 'if datareader container is empty, display error message If Not myDatareader.HasRows Then lblStatus.Text = "No Student Found" lblStatus.Visible = True myDatareader.Close() Exit Sub End If '(7) bind control to data source, then display data to user StudentGrid.DataSource = myDatareader StudentGrid.DataBind() StudentGrid.Visible = True '(8) close objects myDatareader.Close() myDatareader = Nothing myCommand = Nothing myConnection.Close() myConnection = Nothing

Data Bind/Display Code 30 'THIS CODE GOES IN THE SAME VB CODE for View Student Information button '(7) bind control to data source, then display data to user StudentGrid.DataSource = myDatareader StudentGrid.DataBind() StudentGrid.Visible = True

Example 2: View Student Info: add Dropdown 31 Prompt for search criteria Use Dropdown web control for specifying Student ID See for how to implement “cascading dropdowns” for Lastname and Firstname

Wizard: Configure the Database Connection 32

Wizard: Configure the SQL Command 33 Will give us student ID as the key field Will give us a field which is the concatenation of student ID and first/last name for ease of lookup

Wizard: Bind Data to Dropdown 34 Name of the data source (connection and query) we just created The field from the query that we want displayed in the dropdown The field from the query that we want to use as the selected value from the dropdown

Result: ASP.net Generates "Code" to Access Data 35 Will be used by a Command object Will be used as the Data Container object and dropdown control will be set to its values Will be used by Connection object Used DropdownList control & configured its data source via Wizard Ensure DropDownList control has this name

Modify Database Access Code 36 In VB Code for View Student Information button, retrieve the selected student from the new dropdown instead of from the text box  i.e., replace txtStudentID.Text with SelectedStudentID.Text

Example 3: View Student Info: add Data Controls, cont… 37 Add dropdown option for "All Students"

Modify Database Access Code 38 Add BOLD code below to the aspx file: <asp:DropDownList ID="SelectedStudentID" runat="server" DataSourceID="SqlDataSource1" DataTextField="studentinfo" DataValueField="stud_id" AppendDataBoundItems = "true"> All Students In the “View Student Information” button VB code, add third querystring to select all rows and columns from student table Dim querystring3 As String = "select * from student;" Also in the “View Student Information” button VB code, update IF-Then-Else in VB code to check for "-1" value (i.e., user selected “All Students”) and assign appropriate query If (txtLastName.Text <> String.Empty Or txtFirstName.Text <> String.Empty) Then querystring = querystring2 ElseIf (SelectedStudentID.Text = "-1") Then querystring = querystring3 Else querystring = querystring1 End If

DATA INTEGRATION

Metadata Management System Catalog Part of DBMS Data Dictionary Typically passive Extension of catalog metadata Information Repository Master Data Management G. Green 40

Master Data Management "Ensuring the currency, meaning, and quality of reference data within and across various subject areas" Identify Common Data Subjects Common Data Elements Sources of "the truth" Cleanse Update applications to reference Master Data repository Ensures consistency of key data (not ALL data) throughout organization 41 G. Gree n

CLOUD COMPUTING

Cloud Computing Business Model Computing resources on demand Need-based architectures Internet-based delivery Pay as you go History (VERY high-level and approximate) 43 G. Gree n Time-sharing Virtual Machines Utility Computing WWW Personal Computers Grid Computing Cloud Computing 50's60's70's80's90's 2000's

Cloud Computing Services Impacts to Data(base) Administration G. Green 44

BIG DATA

Complex Data Landscape NOTE: This diagram is for effect ONLY—it is incomplete (e.g., no MDDB, no OODB) AND contains some inaccuracies G. Green 46

Big Data Big Data = more than you're able to effectively process Influenced by Mobile, Social Networking, Web analytics, RFID, Atmospheric, Medical Research data, … Traditional RDMS Problems Transaction-focus Requires schema  maintenance issue ACID-focus Requires locks, db constraints, joins  performance & availability issues "Relatively" small amounts of operational data Exceptions require complex, $ actions  scalability issue G. Green 47 Reference:

Big Data Solutions  Hadoop Good for storing and retrieving large amounts of (mostly unstructured) data  HDFS: data distributor  MapReduce: request distributor  Columnar Databases Good for data warehouses, analytics  Files contain all column values vs. all row values  NewSQL Databases Good when high scalability needed with relational DBMSs  Re-written, highly optimized storage engines  "Sharding" (i.e. horizontal partitioning across multiple DB instances)  Distributed query processing  In-memory databases  NoSQL Databases… G. Green 48

Big Data Solutions, cont…  NoSQL Databases Focus (in most cases):  Scalability via Physical Distribution of data  No fixed schema  "Individual query systems" instead of SQL  Support for semi- and un-structured data Some provide consistency  Apache's Hbase  Amazon's DynamoDB Most provide "eventual consistency"  Google's BigTable  Facebook's Cassandra  Amazon's SimpleDB Use a variety of data models… G. Green 49

Big Data/NoSQL, cont… NoSQL Physical Data Models Column (Family) Store Key Value Store Document Store G. Green 50 Advantages Highly scalable Support for semi- and un-structured data Data Model (schema) does not have to be defined up-front Many are open source Cloud options available (e.g., Amazon's SimpleDB) Disadvantages No common query language Data inconsistency Reliance on client applications for data validation, consistency, etc…

"Big Data" Skills ETL (extract, translate, load) Data warehousing (MDDB) Data mining techniques Statistical modeling with tools such as R, SAS, or SPSS Data visualization tools Technologies for structured and unstructured data Hadoop (an Apache project to provide an open-source implementation of frameworks for reliable, scalable, distributed computing and data storage.) NoSQL "NewSQL" See Big Data University for (mostly) free self-study trainingBig Data University Source: 51 G. Gree n

DATA WAREHOUSE

Need for Decision-Optimized Data Storage Business people need information to make plans, decisions, and assess results What were sales volumes by region and product category for the last 3 years? Which of two new medications will result in the best outcomes (higher recovery rate and shorter hospital stay)? Data captured by complex operational systems (OLTPs) optimized to support well-defined transaction requirements Difficult to get needed information from data grounded in OLTPs 53

Operational vs. Informational Data Operational (ie, OLTP)Informational (ie, OLAP) Data ContentCurrent ValuesHistorical, derived, summarized Data StructureOptimized for transactionsOptimized for complex queries Data VolumeMB/GB of dataGB/TB/PB… of data Access FrequencyHighMedium to low Access TypeRead, update, deleteRead-only UsagePredictable, repetitiveAd hoc, random, heuristic Response TimeSub-secondsSeveral seconds to minutes UsersLarge number; operational & data workers Relatively smaller number; data & knowledge workers 54

Evolution of Decision Support Technologies Business people need information to make plans, decisions, and assess results 60's Batch reports 70’s DSSs 80’s Info Centers 90’s Early DWs 2000's Business Intelligence Issues: Dependency on IT resources Based on OLTP or extracts Functionality often pre-programmed "Big Data" Analytics 55

Data Warehouse 56 “… a subject-oriented, integrated, nonvolatile, and time variant collection of data in support of management decisions.” Managing the Data Warehouse, W. H. Inmon, John Wiley & Sons, December, “… a copy of transaction data specifically structured for query and analysis.” The Data Warehouse Toolkit, R. Kimball, John Wiley & Sons, February, Enterprise data, transformed, integrated, accumulated over time, optimized for decision-making, and accessible via analytical tools

Multidimensional Data Model  Data represented as cube > Modeling and analyzing of data across dimensions  Cube depicts business measures analyzed by dimensions  Optimized for decision-making vs. transaction processing  Data storage  Pre-aggregation  De-normalization  Basis for data warehouses G. Green 57

Multidimensional Data Model, cont… Advantages Faster performance for decision-based queries Read-only Summary data Data viewed in many ways/dimensions Data analyzed over time Easier analyses by decision-makers Visual interfaces Typically point/click, drag/drop Disadvantages ETL Data redundancy G. Green 58

Overall BI Architecture 59 Source: Chaudhuri et. al., An Overview of Business Intelligence Technology, Communications of the ACM, 54(8), August 2011, pp