Programming Techniques :: Searching Data jamie@drfrostmaths.com www.drfrostmaths.com @DrFrostMaths Last modified: 25th June 2019

www.drfrostmaths.com ? Everything is completely free. Why not register? Registering on the DrFrostMaths platform allows you to save all the code and progress in the various Computer Science mini-tasks. It also gives you access to the maths platform allowing you to practise GCSE and A Level questions from Edexcel, OCR and AQA. With Computer Science questions by: Your code on any mini-tasks will be preserved. Note: The Tiffin/DFM Computer Science course uses JavaScript as its core language. Most code examples are therefore in JavaScript. Using these slides: Green question boxes can be clicked while in Presentation mode to reveal. Slides are intentionally designed to double up as revision notes for students, while being optimised for classroom usage. The Mini-Tasks on the DFM platform are purposely ordered to correspond to these slides, giving your flexibility over your lesson structure. ?

Databases DrFrostMaths runs off a database: Each row represents an entry in the table, in this case an individual school. We previously saw that this is an example of a record. A database consists of a number of tables… Each table consists of a number of fields of various types. For example, sid is the school identifying number (an integer), the name is a string, and so on.

SQL SQL (Structured Query Language) is a language to search, edit, insert and delete data from databases. It is the global standard for querying databases. Fro Note: SQL is just the language to query the database: it is not the database itself or the software used to store/manage the data, which is known as a Relational Database Management System (RDBMS). There are many different well-known RDBMSs, including MySQL (which is what DrFrostMaths uses), SQL Server and Oracle. SELECT sid, name, domain, phase, totalpoints FROM drfrostmaths_school The most basic syntax for a SQL query is as follows: SELECT fields FROM table SELECT means we’re retrieving data. We’d used UPDATE for modifying data and DELETE for deleting data. A NULL value means there is no value for this field for this row.

Special Fields SELECT * FROM drfrostmaths_school * is a special ‘wildcard’ character, which matches all fields.

Aggregate Functions SELECT COUNT(sid) FROM drfrostmaths_school (technically not in the GCSE syllabus, but super important) SELECT COUNT(sid) FROM drfrostmaths_school COUNT(sid) gives the count of the number of values (not necessarily distinct) in the sid column. Aggregate functions, such as COUNT, produce a table of just one row, in this case containing your count. Note that COUNT doesn’t include NULL values. COUNT(*) will give a count of rows in the table unless your table has rows where the value for all fields NULL. The phase field had some NULL values (see previous slide) so the count was less. SELECT SUM(totalpoints), AVG(totalpoints) FROM drfrostmaths_school Other aggregate functions include SUM (the total), AVG (the mean), MAX (the maximum) The totalpoints field is the total number of points each school has. So SUM(totalpoints) gives the total points of all schools.

Aggregate Functions – Advanced Notes Original table: (This really absolutely is not in the GCSE syllabus!) SELECT SUM(totalpoints) FROM drfrostmaths_school SUM(totalpoints) isn’t one of the original fields of our table, but a field we’ve created (based on an existing field), so it doesn’t have a field name. SELECT SUM(totalpoints) AS tot FROM drfrostmaths_school We can use the AS keyword to give new fields (usually aggregate functions) a name, known as an alias. This might seems useless on its own, but is useful if we want to use our table as a subquery within a more complicated query, and need to refer to this field.

? Aggregate Functions – Advanced Notes Original table: (This really absolutely is not in the GCSE syllabus!) You may wonder what happens if you combine an aggregate function with a normal field (here sid). sid -10 1 2 3 4 … SUM(totalpoints) 170189679 The fields do not have the same number of rows, so what would happen if we selected sid and SUM(totalpoints)? ? SELECT SUM(totalpoints), sid FROM drfrostmaths_school The answer is: nothing interesting. Due to the inconsistent number of rows, the overall table is restricted to one row, and sid is just restricted to the value in the first row for that field. You shouldn’t in general combine aggregate functions with normal fields. This wouldn’t matter so much if all the sid values were the same value, and this indeed will happen when we look at the GROUP BY keywords later.

Distinct Values SELECT phase FROM drfrostmaths_school (again, not in GCSE syllabus) SELECT phase FROM drfrostmaths_school Selecting a field gives the values in all rows for that field, including any duplicate values! SELECT DISTINCT phase FROM drfrostmaths_school Putting DISTINCT before a field name restricts it to only distinct (i.e. non-duplicated) values. Note: The DISTINCT keyword will be ignored if you select more than one field, as the values for each field in each row would no longer be lined up. SELECT COUNT(DISTINCT country) FROM drfrostmaths_school We can combine DISTINCT with aggregate functions. The above query counts the number of distinct values in the country field.

Conditions using WHERE SELECT name, totalpoints FROM drfrostmaths_school WHERE totalpoints > 10000 Using the WHERE keyword, followed by a condition, allows us to restrict our new table to rows which satisfy that condition.

Multiple conditions SELECT name, totalpoints FROM drfrostmaths_school WHERE totalpoints > 10000 AND phase = ‘Primary’ We can have multiple conditions which are combine with AND or OR, in exactly the same manner we would combine Boolean conditions with normal programming languages. SELECT SUM(totalpoints), COUNT(*) FROM drfrostmaths_school WHERE country != ‘England’ AND totalpoints > 0 Use != or <> for “not equal to”. We could also use: NOT(country = ‘England’) We can combine aggregate functions like SUM and COUNT with conditions. This query gets the total points earned by schools and number of schools outside of England, for those who have scored more than 0 points.

Sorting Results (weirdly, ordering results is not in the GCSE syllabus) SELECT name, totalpoints FROM drfrostmaths_school ORDER BY totalpoints DESC We specify the field we want to sort on, and whether to put in ascending order (ASC) or descending order (DESC).

Test Your Understanding So Far All fields in drfrostmaths_school: Write queries to get results for the following: 1 The postcodes of all schools in Wales. ? SELECT postcode FROM drfrostmaths_school WHERE country = ‘Wales’ 2 The average age of students leaving the school. ? SELECT AVG(maxage) FROM drfrostmaths_school 3 All fields for all schools scoring between 1000 and 2000 points. SELECT * FROM drfrostmaths_school WHERE totalpoints >= 1000 AND totalpoints <= 2000 ? 4 The number of all-girl primary schools. There are just 3, vs 16744 mixed! ? SELECT COUNT(*) FROM drfrostmaths_school WHERE gender = ‘Girls’ AND phase = ‘Primary’

Using % within WHERE (This *is* in the GCSE syllabus) Sometimes you might want to match rows where the value ‘starts with’ a string or ‘ends with’ a string, or contains a string somewhere within it. For example, when registering on DrFrostMaths, there is an ‘autocomplete’ facility when entering the name of your school: SELECT name, sid, thumb FROM drfrostmaths_school WHERE name LIKE ‘%Tiff%’ % indicates “0 or more of any character”. So %Tiff% means ‘Any value where ‘Tiff’ is potentially surrounded by some characters’. In other words, any value containing “Tiff”! If you use any wildcard symbols such as %, you need to use the keyword LIKE. Using LIKE also makes your matches case-insensitive. For example “tiffin” would match. Advanced: You can use the ‘_’ symbol for “any single character” (whereas % means ‘0 or more’)

Test Your Understanding Describe what the following queries will do: 1 SELECT name FROM drfrostmaths_school WHERE name LIKE ‘T%’ ? All schools whose name begin with the letter T. 2 ... WHERE name LIKE ‘A%a’ ? All schools starting and ending with the letter A. 3 ... WHERE name LIKE ‘% Academy’ ? All schools ending with the word ‘Academy’. N ... WHERE name LIKE ‘_i%’ ? All schools whose second letter is ‘i’. N ... WHERE name LIKE ‘_%_’ ? All schools whose name is at least 2 letters long.

Advanced :: GROUP BY (Not in GCSE syllabus) Sometimes you might want to use aggregrate functions like SUM and COUNT, but where the rows are first grouped in some way. For example, we might want to find the number of schools by country. SELECT country, COUNT(*) FROM drfrostmaths_school GROUP BY country Name Country Hogwarts England Tiffin School England Country COUNT(*) England 2 Name Country Hogwarts England Tiffin School England Beauxbatons Academy France La Baguette Ecole France British School of Paris France Durmstrang Bulgaria Name Country Beauxbatons Academy France La Baguette Ecole France British School of Paris France Country COUNT(*) France 3 Name Country Durmstrang Bulgaria Country COUNT(*) Bulgaria 1 The table is first split by whatever field we’re grouping by. The fields are then selected (including any aggregrate functions) for each table. Country COUNT(*) England 2 France 3 Bulgaria 1 The tables are then put back together.

Further Examples Q “Get the total number of points for schools by gender” ? SELECT gender, SUM(totalpoints) FROM drfrostmaths_school GROUP BY gender “Get the most points earned for a school by town, restricted to English towns beginning with the letter ‘K’” Q ? SELECT town, MAX(totalpoints), COUNT(*) FROM drfrostmaths_school WHERE country = 'England' AND town LIKE 'K%' GROUP BY town

Advanced :: Joins (Not in GCSE syllabus) Sometimes two database tables might have a field in common. For example drfrostmaths_user, containing all user accounts, has a field sid with the id number of the student’s school. drfrostmaths_school also has this field! drfrostmaths_user drfrostmaths_school uid firstname surname sid 3011 Jamie Oliver 204 3012 Jamie Theakston 203 3013 Freddie Kruger 37 … sid name 37 Evil High … 203 Grange Hill 204 Hogwarts “Produce a table of rows with the name of each student and the name of the school they attend.” firstname surname name Jamie Oliver Hogwarts Jamie Theakston Grange Hill Freddie Kruger Evil High … SELECT firstname, surname, name FROM drfrostmaths_school s INNER JOIN drfrostmaths_user u ON u.sid = s.sid And merges the rows using sid as the common column. Note we’ve named the tables s and u (known as aliases), so that we can be explicit about which table we’re using the sid field from. INNER JOIN joins the first table to the second…

Advanced :: Joins (Not in GCSE syllabus) “Find the names of any school who has a student called Jamie.” ? SELECT DISTINCT s.name FROM drfrostmaths_school s INNER JOIN drfrostmaths_user u ON u.sid = s.sid WHERE u.firstname = ‘Jamie’ Note that without the DISTINCT keyword, then were Abingdon School for example to have 5 students called Jamie, “Abingdon School” would appear 5 times in our table. DISTINCT ensures each name only appears once.

Advanced :: Joins a b 5 ‘cat’ 7 ‘dog’ 11 ‘rabbit’ b c ‘cat’ 0 ‘cat’ 1 (Not in GCSE syllabus) Table B Table A INNER JOIN ON A.b = B.b a b 5 ‘cat’ 7 ‘dog’ 11 ‘rabbit’ b c ‘cat’ 0 ‘cat’ 1 ‘rabbit’ 2 ‘rabbit’ 3 ‘rabbit’ 4 ‘dog’ 5 INNER JOIN works by giving all possible combinations of rows in table A merged with rows in table B, subject to them having the same value for the field we’re joining on, as per this example. While for example the 3rd row in table A can ‘join’ with three different rows in table B, in practice we’d usually only expect one thing to match in the second table. a b c 5 ‘cat’ 0 5 ‘cat’ 1 7 ‘dog’ 5 11 ‘rabbit’ 2 11 ‘rabbit’ 3 11 ‘rabbit’ 4

N Super Advanced :: Nested Queries “Find the names of all schools with at least 1000 registered users.” Initially, let’s group users in the main user table by school, and get the count within each school: SELECT sid, COUNT(*) FROM drfrostmaths_user GROUP BY sid We can use a join to also get the names of these schools: SELECT u.sid, COUNT(*) AS cnt, s.name FROM drfrostmaths_user u INNER JOIN drfrostmaths_school s ON u.sid = s.sid GROUP BY u.sid Unfortunately, because SQL evaluates the WHERE clause before the SELECT, we can’t just plop “WHERE cnt > 1000” near the end of our query (as the alias cnt won’t have been declared yet). We instead have to nest the table outputted by the above query within another SELECT statement! * Note that any nested table must be given an alias, in this case s2. SELECT name, cnt FROM (SELECT u.sid, COUNT(*) as cnt, s.name FROM drfrostmaths_user u INNER JOIN drfrostmaths_school s ON u.sid = s.sid GROUP BY u.sid) s2 WHERE s2.cnt>1000

Advanced :: Database Efficiency Databases can get very large. The table on DrFrostMaths which stores students’ answers for example is 27 million rows! Even simple queries like: SELECT * FROM drfrostmaths_useranswer WHERE uid=‘19483’ (to find all the answers by that student) would take a long time if we had to check all 27 million rows… The solution is to create ‘indexes’ on particular fields of a table. These are separate (usually tree-like) data structures which efficiently store the values for that field, allowing algorithms such as binary search (which we will cover separately). It’s possible to have an index of multiple columns. For example, I often have to make points calculations for a particular school in a particular time range. There’s therefore an index on the school id (sid) combined with the date the question was answered (dateanswered) uid 200 400 600 50 100 75 Row 274 Row 276 This tells us the indexes use BTREEs as their data structure.

Review ? ? ? ? ? What is the difference between the wildcards * and %? ‘*’ represents ‘all fields’, e.g. SELECT * FROM mytable. ‘%’ is used within a string to match any text, e.g. SELECT * FROM mytable WHERE fullname LIKE ‘J%’ would match anyone whose name starts with J. How would select people from a table (people) where either their age is above 40 or they earn more than £100,000? SELECT * FROM people WHERE age > 40 OR earnings > 100000 How would we then sort these people by name alphabetically ascending? SELECT * FROM people WHERE age > 40 OR earnings > 100000 SORT BY name ASC How would we determine the total earnings of people aged under 40? SELECT SUM(earnings) FROM people WHERE age < 40 How could we work out the average earnings by country? (Not in GCSE syllabus) SELECT country, AVG(earnings) FROM people GROUP BY country ? ? ? ? ?

Coding Mini-Tasks Return to the DrFrostMaths site to complete the various mini-coding tasks on searching data.