1. Joins and Relationships The Theory, the Pearls and the Perils
IDEA User Group 8th June 2017

2. Relationships
Nothing is perfect. Life is messy. Relationships are complex. Outcomes are uncertain. People are irrational. Hugh Mackay
This is as true with datasets as it is in life!

3. Many to one relationship
Relationships
Many to one relationship
Key
For sales, substitute work records (with product code equating to employer id)
Products would be employee details (with tax code equating to income tax code)
Mention that there may be compound keys as well, which adds to the level of complexity (and potential for error)
Foreign key
Foreign key

4. Some Diagrammatic Joins
Inner Join
(Matches Only)
Left Join
(All records in primary file)
Left Excluding Join
(Records with no secondary match)
Right Excluding Join
(Records with no primary match)
Outer Join
(All records in both files)
Outer Excluding Join
(Not available in IDEA)
Did anyone do set theory in school?
I wondered what the point of it was at the time, but now I know.
These show examples for 2 datasets – the reality can be much more complex.

5. Example Tables I’ve got 3 very small and simple datasets here.
Lets look at the results of different joins – For now I’m just using the first two tables to keep things sensible.

6. Inner Join Inner Join (Matches Only)
Apologise for potentially lack of line bre4aks in the IDEAScript code. Didn’t paste as I’d expected.
Key here is WI_JOIN_MATCH_ONLY.
IDEAScript in following slided is almost identical, except to join specification
Set db = Client.OpenDatabase("Table1.IMD")
Set task = db.JoinDatabasetask.FileToJoin "Table2.IMD"task.IncludeAllPFieldstask.IncludeAllSFieldstask.Add
Match Key "PRIMARYKEY", "PRIMARYKEY", "A"task.CreateVirtualDatabase = FalsedbName = ".IMD" task.PerformTask dbName, "", WI_JOIN_MATCH_ONLY
Set task = Nothing
Set db = Nothing
Client.OpenDatabase (dbName)

7. Inner Join Result
There is no uptput for Arizona, Lincoln, Lucent (table 1), and Microsoft, Apple and Scotch (table 2) as the key fields are not present in both databases.

8. (All records in primary file)
Left Join
Left Join
(All records in primary file)
Set db = Client.OpenDatabase("Table1.IMD")
Set task = db.JoinDatabasetask.FileToJoin "Table2.IMD"task.IncludeAllPFieldstask.AddSFieldToInc "VALUE"task.AddMatchKey "PRIMARYKEY", "PRIMARYKEY", "A“
task.CreateVirtualDatabase = False
dbName = ".IMD" task.PerformTask dbName, "", WI_JOIN_ALL_IN_PRIM
Set task = Nothing
Set db = Nothing
Client.OpenDatabase (dbName)
Keyword here is WI_JOIN_ALL_IN_PRIM

9. Left Join Result

10. (Records with no secondary match)
Left Excluding Join
Left Excluding Join
(Records with no secondary match)
Set db = Client.OpenDatabase("Table1.IMD")
Set task = db.Join
Databasetask.FileToJoin "Table2.IMD"task.IncludeAllPFieldstask.AddMatchKey "PRIMARYKEY", "PRIMARYKEY", "A“
task.CreateVirtualDatabase = FalsedbName = ".IMD" task.PerformTask dbName, "", WI_JOIN_NOC_SEC_MATCH
Set task = Nothing
Set db = Nothing
Client.OpenDatabase (dbName)
Keyword here in IDEAScript is WI_JOIN_NOC_SEC_MATCH
Question: “ Is Records with no Secondary Match” (Left Excluding Join) preferable to “All Records in Prmary File” (Left Join)
Discusion next slide maybe?

11. Left Excluding Join Result
Only 3 records returned, so we know that there are keys in primary file which don’t exist in secondary file.
This could be significant if you’re trying to match, say, ledger transactions with a list of ledger codes. If the number of records in the output is less that the number of records in the primary database, You immediately know that there are missing ledger codes in the master file.
On the other hand the left join would ensure that you get all the transaction records, but you would need to analyse the result for null values from the secondary database.
Exactly the ame applies to right and right exclusing joins.

12. (Records with no primary match)
Right Excluding Join
Left Excluding Join
(Records with no primary match)
Set db = Client.OpenDatabase("Table1.IMD")
Set task = db.JoinDatabasetask.FileToJoin "Table2.IMD"task.IncludeAllPFieldstask.IncludeAllSFieldstask.AddMatchKey "PRIMARYKEY", "PRIMARYKEY", "A"task.CreateVirtualDatabase = Falsedb
Name = ".IMD"
task.PerformTask dbName, "", WI_JOIN_NOC_PRI_MATCH
Set task = Nothing
Set db = Nothing
Client.OpenDatabase (dbName)
Keyword WI_NOC_PRI_MATCH

13. Right Excluding Join Result
None of the records in the primary database have been returned.
Easy to spot with small datasets like this, but may not be obvious when you’ve got datasets with tens of thousands (or more) records.

14. (All records in both files)
Outer Join
Outer Join
(All records in both files)
Set db = Client.OpenDatabase("Table1.IMD“
)Set task = db.JoinDatabasetask.FileToJoin "Table2.IMD"task.IncludeAllPFieldstask.IncludeAllSFieldstask.AddMatchKey "PRIMARYKEY", "PRIMARYKEY", "A"task.CreateVirtualDatabase = FalsedbName = ".IMD" task.PerformTask dbName, "", WI_JOIN_ALL_REC
Set task = Nothing
Set db = Nothing
Client.OpenDatabase (dbName)
Keyword WI_JOIN_ALL_REC

15. Right Excluding Join Result
So 2 null values in primary database, and 3 in secondary database.
Can’t think of why you may want to use this, but it’s available

16. Visual Connector
Very easy to use, bu as with all easy things use with caution.
You can create multiple . And you only have elationships, but IDEA doesn’t (I think) allow you to explicitly define each individual relationship.
I’m not quite sure what the results prove here.

17. Watch out for the Cartesian Join
Other Joins
Outer Excluding Join
(Why?)
If you really need to, it’s a left excluding join and a right exluding join, then append the two databases.
Watch out for the Cartesian Join
A sales ledger with 50,000 records relating to 150 account codes will result in an output of 7.5 million records (and the extract could take some time to run.)

18. Cartesian Join
IDEA won’t allow you to do this, but it’s very easy to do accidentally in Access. You just need to forget to create a relationship between two tables.

19. A Recommendation for Polling Day
Party
Vote
Optimism
Defeatism
IDEAlism X
Antidisestablishmentarianism
Humanism
Pragamatism
Schism

20. Thank you! Any questions?
Dave Pendreigh