1. Characteristic Functions

2. Want: YearCodeQ1AmtQ2AmtQ3AmtQ4Amt 2001e1 198 204 214 231 (from fin_data table in Sybase Sample Database) Have: Yearquartercodeamount 2001Q1e1 198 2001Q2e1 204 2001Q3e1 214 2001Q4e1 231

3. Characteristic Functions Want: Un-normalized YearCodeQ1AmtQ2AmtQ3AmtQ4Amt 2001e1 198 204 214 231 Have: Normalized Yearquartercodeamount 2001Q1e1 198 2001Q2e1 204 2001Q3e1 214 2001Q4e1 231

4. Characteristic Functions Pivot Table YearCodeQ1AmtQ2AmtQ3AmtQ4Amt 2001e1 198 204 214 231 Similar to Excel Database Format Yearquartercodeamount 2001Q1e1 198 2001Q2e1 204 2001Q3e1 214 2001Q4e1 231

5. Possible Solutions (without using Characteristic Functions) Self Join as in type 4 queries

6. Possible Solution – Self Join Start with just Q1 and Q2, code E1, Year 2001 YearcodeQ1AmtQ2Amt 2001e1198204 Use a self join as in type 4 queries Write the SQL query

7. Possible Solution – Self Join Code SELECT F1.year, F1.code, F1.amount AS Q1Amt, F2.amount AS Q2Amt FROM fin_data F1, fin_data F2 WHERE F1.year = 2001// Only 2001 AND F2.year = 2001 AND F1.code = ‘e1’ // Only financial code e1 AND F2.code = ‘e1’ AND F1.quarter = ‘Q1’ // Get Q1 amount AND F2.quarter = ‘Q2’ // Get Q2 amount YearcodeQ1AmtQ2Amt 2001e1198204

8. Possible Solution – Self Join Expand to all years YearcodeQ1AmtQ2Amt 1999e110193 2000e1153149 2001e1198204

9. Possible Solution – Self Join Code SELECT F1.year, F1.code, F1.amount AS Q1Amt, F2.amount AS Q2Amt FROM fin_data F1, fin_data F2 WHERE F1.year = F2.year // Same Year AND F1.code = ‘e1’ // Only financial code e1 AND F2.code = ‘e1’ AND F1.quarter = ‘Q1’ // Get Q1 amount AND F2.quarter = ‘Q2’ // Get Q2 amount YearcodeQ1AmtQ2Amt 1999e110193 2000e1153149 2001e1198204 Expand to all years

10. Possible Solution – Self Join Expand to all four quarters YearcodeQ1AmtQ2AmtQ3AmtQ4Amt 1999e110193129145 2000e1153149157163 2001e1198204214231

11. Possible Solution – Self Join Code All four quarters SELECT F1.year, F1.code, F1.amount AS Q1Amt, F2.amount AS Q2Amt, F3.amount AS Q3Amt, F4.amount AS Q4Amt FROM fin_data F1, fin_data F2, fin_data F3, fin_data F4 WHERE F1.year = F2.year // Same Year AND F2.year = F3.year AND F3.year = F4.year AND F1.code = ‘e1’ // Only financial code e1 AND F2.code = ‘e1’ AND F3.code = ‘e1’ AND F4.code = ‘e1’ AND F1.quarter = ‘Q1’ // One record for each quarter AND F2.quarter = ‘Q2’ AND F3.quarter = ‘Q3’ AND F4.quarter = ‘Q4’ YearcodeQ1AmtQ2AmtQ3AmtQ4Amt 1999e110193129145 2000e1153149157163 2001e1198204214231

12. Possible Solution – Self Join Problems Coding: Suppose we wanted months instead of quarters… Performance: Suppose fin_data had 100,000 records instead of 84… We need a better solution!

13. Possible Solutions (without using Characteristic Functions) SubQueries

14. Possible Solution – SubQueries Start with just Q1 and Q2, code E1, Year 2001 YearcodeQ1AmtQ2Amt 2001e1198204 Use a subquery as a field in the select clause Write the SQL query

15. Possible Solution – SubQueries Code SELECT F1.year, F1.code, F1.amount AS Q1Amt, ( SELECT F2.amount as Q2Amt FROM fin_data F2 WHERE F2.quarter = ‘Q2’ AND F2.code = ‘e1’ AND F2.year = 2001 ) FROM fin_data F1 WHERE F1.quarter = ‘Q1’ AND F1.code = ‘e1’ AND F1.year = 2001 YearcodeQ1AmtQ2Amt 2001e1198204 Use a subquery as a field in the select clause

16. Possible Solution – SubQueries Expand to all years YearcodeQ1AmtQ2Amt 1999e110193 2000e1153149 2001e1198204

17. Possible Solution – SubQueries Code SELECT F1.year, F1.code, F1.amount AS Q1Amt, ( SELECT F2.amount as Q2Amt FROM fin_data F2 WHERE F2.quarter = ‘Q2’ AND F2.code = ‘e1’ AND F2.year = F1.year ) FROM fin_data F1 WHERE F1.quarter = ‘Q1’ AND F1.code = ‘e1’ //AND F1.year = 2001 YearcodeQ1AmtQ2Amt 1999e110193 2000e1153149 2001e1198204 Expand to all years This is now a correlated subquery

18. Possible Solution – SubQueries Expand to all four quarters YearcodeQ1AmtQ2AmtQ3AmtQ4Amt 1999e110193129145 2000e1153149157163 2001e1198204214231

19. Possible Solution – SubQueries Code All four quarters SELECT F1.year, F1.code, F1.amount AS Q1Amt, ( SELECT F2.amount as Q2Amt FROM fin_data F2 WHERE F2.quarter = ‘Q2’ AND F2.code = ‘e1’ AND F2.year = F1.year ), ( SELECT F2.amount as Q3Amt FROM fin_data F2 WHERE F2.quarter = ‘Q3’ AND F2.code = ‘e1’ AND F2.year = F1.year ), YearcodeQ1AmtQ2AmtQ3AmtQ4Amt 1999e110193129145 2000e1153149157163 2001e1198204214231 ( SELECT F2.amount as Q4Amt FROM fin_data F2 WHERE F2.quarter = ‘Q4’ AND F2.code = ‘e1’ AND F2.year = F1.year ) FROM fin_data F1 WHERE F1.quarter = ‘Q1’ AND F1.code = ‘e1’

20. Possible Solution – Self Join Problems Coding: Again, Suppose we wanted months instead of quarters… Performance: Our example requires the effective execution of 10 queries The F1 query is run once to return the year, code and Q1Amt columns The F2 query is run 9 times return the Q2, Q3, and Q4 amounts for 1999, 2000 and 2001 If our table had 100,000 rows F2 would run 300,000 times! We still need a better solution!

21. Possible Solution – Temporary Table Strategy Create a table with fields for year, code, Q1Amt, Q2Amt, Q3Amt, and Q4Amt Insert Query to add records and fill in year, code, Q1Amt Update Query to add Q2Amt Update Query to add Q3Amt Update Query to add Q4Amt

22. Possible Solution – Temporary Table Create a Table Create a table with fields for year, code, Q1Amt, Q2Amt, Q3Amt, and Q4Amt CREATE TABLE QAmt ( year char(4), code char(2), Q1Amt numeric(9), Q2Amt numeric(9), Q3Amt numeric(9), Q4Amt numeric(9) );

23. Possible Solution – Temporary Table DROP a Table There will be times when you want to get rid of a table you have created. Perhaps you made an error when you created it. Or, you may simply be through using it. To get rid of a table you DROP it from the database. DROP TABLE QAmt

24. Possible Solution – Temporary Table INSERT Query Insert Query to add records and fill in year, code, Q1Amt INSERT INTO QAmt (year, code, Q1Amt) SELECT year, code, amount FROM fin_data WHERE quarter = 'Q1' AND code = 'e1'

25. Possible Solution – Temporary Table Delete Query There may also be times when you want to keep a table, but get rid of all the records in the table… to empty it. If, for example, you find a problem with your INSERT query, you may want to empty the table before you run the corrected INSERT query. To empty a table you DELETE all the records FROM the table. To empty the QAmt table you: DELETE FROM QAmt

26. Possible Solution – Temporary Table After INSERT Query Check the results: SELECT * FROM QAmt YearcodeQ1AmtQ2AmtQ3AmtQ4Amt 1999e1101 (NULL) (NULL) (NULL) 2000e1153 (NULL) (NULL) (NULL) 2001e1198 (NULL) (NULL) (NULL)

27. Possible Solution – Temporary Table Update Query YearcodeQ1AmtQ2AmtQ3AmtQ4Amt 1999e1101 93 (NULL) (NULL) 2000e1153 149 (NULL) (NULL) 2001e1198 204 (NULL) (NULL) Update Query to add Q2Amt UPDATE QAmt Q SET Q2Amt = (SELECT amount FROM fin_data F WHERE F.year = Q.year AND F.code = Q.code AND F.quarter = 'Q2‘ ) We do two more update queries to fill in Q3Amt and Q4Amt

28. Possible Solution – Temporary Table Is this really different At this point you may have noticed that the temporary table approach looks a lot like the subquery approach. In fact, it is virtually the same. It seems conceptually simpler because the temporary table allows us break down the problem into separate and distinct subtasks. We can complete part of the solution, check the results, complete some more, etc. For this reason, you will see it used. It has some legitimate applications, but this isn’t one of them. This point becomes even more evident if we use either the self-join approach or the subquery approach to populate the table in one step.

29. Possible Solution – Temporary Table One Step Using Self Join INSERT INTO QAmt (year, code, Q1Amt, Q2Amt, Q3Amt, Q4Amt) SELECT F1.year, F1.code, F1.amount, F2.amount, F3.amount, F4.amount FROM fin_data F1, fin_data F2, fin_data F3, fin_data F4 WHERE F1.year = F2.year AND F2.year = F3.year AND F3.year = F4.year AND F1.code = 'e1' AND F2.code = 'e1' AND F3.code = 'e1' AND F4.code = 'e1' AND F1.quarter = 'Q1' AND F2.quarter = 'Q2' AND F3.quarter = 'Q3' AND F4.quarter = 'Q4'

30. Possible Solution – Temporary Table One Step Using SubQuery INSERT INTO QAmt (year, code, Q1Amt, Q2Amt, Q3Amt, Q4Amt) SELECT year, code, amount, (SELECT amount FROM fin_data F WHERE F.year = Q.year AND F.code = Q.code AND F.quarter = 'Q2' ), (SELECT amount FROM fin_data F WHERE F.year = Q.year AND F.code = Q.code AND F.quarter = 'Q3' ), (SELECT amount FROM fin_data F WHERE F.year = Q.year AND F.code = Q.code AND F.quarter = 'Q4' ) FROM fin_data Q WHERE quarter = 'Q1' AND code = 'e1'

31. Possible Solution – Temporary Table Problems We still need a better solution! So… using a temporary table really isn’t a distinct approach at all. It can make the solution conceptually simpler by allowing us to divide the solution into discrete steps, but it does nothing to reduce overall coding complexity or to improve performance. In addition, it introduces potential problems with regard to updates made between the time the table is created and the time it is used (currency). If you really do want a snapshot of the data at a particular time then this can be useful. In general, it’s undesirable You can overcome the currency problem by creating a view instead of a table but, so far, you’re still basically using either a self-join or a subquery method. If you want to create a View you’re better off using Characteristic Functions to do it!

32. Better Solution Characteristic Functions

33. Characteristic Functions Strategy Distribute amount to one of four new columns: yearcodequarteramountQ1AmtQ2AmtQ3AmtQ4Amt 1999e1Q1101101000 1999e1Q29309300 1999e1Q3129001290 1999e1Q4145000145 2000e1Q1153153000 2000e1Q2149014900 2000e1Q3157001570 2000e1Q4163000163 etc.

34. Characteristic Functions Strategy and GROUP BY year, summing the amounts in the new columns for each year. yearfin_codeQ1AmtQ2AmtQ3AmtQ4Amt 1999e110193129145 etc. yearcodequarteramountQ1AmtQ2AmtQ3AmtQ4Amt 1999e1Q1101101000 1999e1Q29309300 1999e1Q3129001290 1999e1Q4145000145 etc.

35. Characteristic Functions WHERE can be Bad In the previous examples, the reason we needed four copies of the table or four different queries is that the determination of which rows to include was made in the WHERE clause. WHERE quarter = ‘Q1’ allows us to put only the amounts for Q1 in the Q1Amt column. That’s good. But… it also removes the possibility of filling in the Q2Amt column in the same query. So Q2Amt must be filled in using a separate copy of the table or a separate subquery with a WHERE clause that allows us to see the amounts in the rows WHERE quarter = ‘Q2’

36. Characteristic Functions Characteristic Functions allow us to control which rows get represented in a particular column without using a WHERE clause. With Characteristic Functions the determination of whether amount is represented in Q1Amt or Q2Amt or Q3Amt or Q4Amt is made in the SELECT clause.

37. Characteristic Functions 1 or 0 A Characteristic Functions is an expression that evaluates to: 1 if a field should be represented in a particular column 0 if it should not. For example, a characteristic function to determine whether an amount should be represented in Q1Amt would return 1 if quarter is ‘Q1’, 0 if it isn’t. We might call this particular characteristic function CF1.

38. Characteristic Functions Assume for a moment that we have created CF1. We could then use it to control whether amount is represented in Q1Amt by defining Q1Amt as CF1 * amount. SELECT year, code, quarter, amount, CF1, CF1 * amount AS Q1Amt FROM fin_data WHERE code = ‘e1’ Would yield: yearcodequarteramount CF1 Q1Amt 1999e1Q11011101 1999e1Q29300 1999e1Q312900 1999e1Q414500 2000e1Q11531153 2000e1Q214900 etc. Note: this obviously won’t run yet since we haven’t actually created CF1 at this point.

39. Characteristic Functions We then create CF2 to return 1 if quarter contains ‘Q2’, 0 otherwise. We use CF2 in the same SELECT clause to control whether amount is represented in Q2Amt SELECT year, code, quarter, amount, CF1, CF1 * amount AS Q1Amt, CF2, CF2 * amount AS Q2Amt FROM fin_data WHERE code = ‘e1’ Would yield: yearcodequarteramount CF1 Q1AmtCF2Q2Amt 1999e1Q1101110100 1999e1Q29300193 1999e1Q31290000 1999e1Q41450000 2000e1Q1153115300 2000e1Q2149001149 etc.

40. Characteristic Functions We then GROUP BY year and SUM the Q1Amt and Q2Amt for the year SELECT year, MAX(code), SUM (CF1 * amount) AS Q1Amt, SUM (CF2 * amount AS Q2Amt FROM fin_data WHERE code = ‘e1’ GROUP BY year Would yield: yearcode Q1AmtQ2Amt 1999e110193 2000e1153149 2001e1198204

41. Characteristic Functions Implementing CF1 Recall that CF1 evaluates to: 1 if quarter is ‘Q1’, 0 otherwise. Both Sybase and SQL Server contain a CASE statement that can be used to implement CF1 as (CASE WHEN quarter = ‘Q1’ THEN 1 ELSE 0 END)

42. Characteristic Functions CF1 and CF2 SELECT year, code, quarter, amount, (CASE WHEN quarter = ‘Q1’ THEN 1 ELSE 0 END) as CF1, (CASE WHEN quarter = ‘Q2’ THEN 1 ELSE 0 END) as CF2 FROM fin_data WHERE code = ‘e1’ Yields: YearcodequarteramountCF1CF2 1999e1Q110110 1999e1Q29301 1999e1Q312900 1999e1Q414500 2000e1Q115310 2000e1Q214901

43. Characteristic Functions Using CF1 & CF2 SELECT year, code, quarter, amount, (CASE WHEN quarter = ‘Q1’ THEN 1 ELSE 0 END) * amount AS Q1Amt, (CASE WHEN quarter = ‘Q2’ THEN 1 ELSE 0 END) * amount AS Q2Amt FROM fin_data WHERE code = ‘e1’ Yields: YearcodequarteramountQ1AmtQ2Amt 1999e1Q11011010 1999e1Q293093 1999e1Q312900 1999e1Q414500 2000e1Q11531530 2000e1Q21490149

44. Characteristic Functions Group By Year SELECT year, MAX(code) AS fin_code, SUM((CASE WHEN quarter = ‘Q1’ THEN 1 ELSE 0 END) * amount) AS Q1Amt, SUM((CASE WHEN quarter = ‘Q2’ THEN 1 ELSE 0 END) * amount) AS Q2Amt FROM fin_data WHERE code = ‘e1’ GROUP BY year Yields: yearfin_codeQ1AmtQ2Amt 1999e110193 2000e1153149 2001e1198204

45. Characteristic Functions All 4 quarters SELECT year, MAX(code) AS fin_code, SUM((CASE WHEN quarter = ‘Q1’ THEN 1 ELSE 0 END) * amount) AS Q1Amt, SUM((CASE WHEN quarter = ‘Q2’ THEN 1 ELSE 0 END) * amount) AS Q2Amt, SUM((CASE WHEN quarter = ‘Q3’ THEN 1 ELSE 0 END) * amount) AS Q3Amt, SUM((CASE WHEN quarter = ‘Q4’ THEN 1 ELSE 0 END) * amount) AS Q4Amt FROM fin_data WHERE code = ‘e1’ GROUP BY year Yields: yearfin_codeQ1AmtQ2AmtQ3AmtQ4Amt 1999e110193129145 2000e1153149157163 2001e1198204 214 231

46. Characteristic Functions Can you modify the query to show all codes, not just e1? yearcodeQ1AmtQ2AmtQ3AmtQ4Amt 1999e110193129145 1999e2403459609632 1999e31437203321842145 1999e46237848561043 1999e5381402412467 1999r11023203329983014 1999r2234459601944 2000e1153149157163 2000e2643687898923 etc.

47. Characteristic Functions Can you modify the query to show all codes, not just e1? yearcodeQ1AmtQ2AmtQ3AmtQ4Amt 1999e110193129145 1999e2403459609632 1999e31437203321842145 1999e46237848561043 etc. SELECT year, code, SUM((CASE WHEN quarter = 'Q1' THEN 1 ELSE 0 END) * amount) AS Q1Amt, SUM((CASE WHEN quarter = 'Q2' THEN 1 ELSE 0 END) * amount) AS Q2Amt, SUM((CASE WHEN quarter = 'Q3' THEN 1 ELSE 0 END) * amount) AS Q3Amt, SUM((CASE WHEN quarter = 'Q4' THEN 1 ELSE 0 END) * amount) AS Q4Amt FROM fin_data GROUP BY year, code ORDER BY year, code

48. Characteristic Functions Result: We create a “Pivot” with only a single pass through the table and without resorting to programming! yearcodeQ1AmtQ2AmtQ3AmtQ4Amt 1999e110193129145 1999e2403459609632 1999e31437203321842145 1999e46237848561043 1999e5381402412467 1999r11023203329983014 1999r2234459601944 etc.

49. Characteristic Functions Add Slide Timing for Join Timing for SubQuery Timing for Temporary Table Timing for Characteristic Function