1. Using Veera with R and Shiny to Build Complex Visualizations
Bharathwaj Vijayakumar, Senior Business Intelligence Analyst,
Rowan University
#RIConf2019

2. Overview of the presentation
Using Veera with R and Shiny to Build Complex Visualizations
Documentation and Veera Workflow

3. Objectives
Visualize the number of number of students migrating from one major to another
Visualize the sequence of courses that students take in a specific program
Visualize the effect of grades in one course, on the courses that follow that

4. Tools and Technology Used
SQL
Veera R
Shiny
R and Shiny was used because Tableau did not directly support the chart types needed for this type of data

5. Visualizations Used and Why?
Sankey Chart
Advantages
Look at magnitude of a flow from point A to point B
Very easy to interpret and visually pleasing
Can be created as multilevel, A to B to C
Disadvantages
Not Excel or table! (Can be an advantage too!)
Sometimes flows can looks similar
Scale can be hard to interpret sometimes
Needs dataset to be in a specific way
Cannot Handle cyclic flows (A to B to A)

6. Visualizations Used and Why?
Chord Diagram
Advantages
Look at magnitude of a flow from point A to point B
Visually pleasing
Can handle cyclic bidirectional flows A to B to A
Disadvantages
Not Excel or table! (Can be an advantage too!)
Sometimes flows can looks similar
Needs dataset to be in a specific way
Can be difficult to interpret compared to Sankey

7. Visualizations Used and Why?
Sankey can be used to show the flow of students from one major to another
Chord Diagram can help to show the sequence of courses that students take and grades they get.
Since students can take courses out of order, there can be bidirectional flows. Chord diagram works better than Sankey for course data

8. Overall Workflow Choose the type of visualization
Prepare the dataset using Veera
Write Code in R
Write the necessary code for Shiny
Publish to Shiny Server

9. Objectives
Visualize the number of number of students migrating from one major to another

10. Data Preparation in Veera
Migration between Majors
Current Format
One row per student per term enrolled
It has major, ids, college and other details
Required Format
Source
Target
Flow
Source Major (Original Major of students)
Target Major (Majors they transferred into)
Number of Students

11. Data Preparation in Veera
Migration between Majors
Final Format
Final Format with years

12. Data Preparation in Veera - Migration between Majors
Fall 2010
Counts the number of students
Fall 2011

13. Data Preparation in Veera - Migration between Majors
Calculates the diff bet the first term and the enrolled term to find the year
Job called multiple time for different years
Writes to two different outputs

14. Visualizations in R Shiny - Migration between Majors
All Years - all_years/
All Years by Source College - all_years_by_source_college/
Each Year by Source College each_year__by_source_college/

15. Objectives
Visualize the sequence of courses that students take in a specific program
Visualize the effect of grades in one course, on the courses that follow that

16. Data Preparation in Veera
Course Sequence
Current Format
One row per student per term per course registered
It has major, course, grades, ids, college and other details
Required Format
Source
Target
Flow
Source Course
Target Course
Number of Students

17. Data Preparation in Veera – Course Sequence
Rest of the data cleanup

18. Data Preparation in Veera
Course Sequence
Final Format

19. Visualizations in R Shiny – Course Sequence
Chord Diagram

20. Documentation & Workflow in Veera

21. Documentation & Workflow in Veera
Setup the users and connections
Development in Sandbox Folder
Create Documentation
Move the job to QA folder and share for QA
Prod &Test and share with user and schedule
Moveup folder share with Admin
Admin
Users

22. Documentation & Workflow in Veera
Setup the users and connections
Admin sets up the connections and the folders.
Windows active directory accounts are not used
All are veera based accounts maintained by admins
Connections are setup using service accounts

23. Documentation & Workflow in Veera
Development
Always use shared drives for input excel or text files and use full path. Create separate input and output folders for Veera
Use Query nodes and put in SQL with a basic select *, rather than importing tables
Save the job under sandbox folder.
Outputs
The output of test jobs should always write to Test schema using the service account
They should all have _test in the file names or table names
Create veera outputs under shared drive by creating appropriate folders

24. Documentation & Workflow in Veera
 Add numbers to your nodes by right clicking and renaming it. This helps to debug the steps when a job fails. 
Add notes to each node by right clicking it

25. Documentation & Workflow in Veera
Source control veera job using Github by exporting the jobs

26. Documentation & Workflow in Veera
Document everything in a web page or a link
Each step number in the veera job will have the details in the documentation page.

27. Documentation & Workflow in Veera
When you are inside a job, click on job-edit notes and add a detailed note about the job and its documentation link

28. Documentation & Workflow in Veera
QA and Prod Moveup
Once the job is done, copy it to QA folder and share it with some other user to review it
Once QA is done, copy the job to the Moveup folder and share it with admin
Admin will check the outputs, make sure it is changed to prod and then schedule the jobs.
The admin will share the jobs in Prod and Test folder with the user, for running or debugging in the future
Test jobs will be properly labeled by admins

29. Includes only First Time UG cohorts
Pulls in courses and grades for each term
ID = 12345
Good Grade
Fall 2010
Spring 2011
Math 1
Math 2
Chem 1
Chem 2
Pulls in courses and grades taken after the current course in the terms greater than the current term

30. Thank you! Questions/Comments?
Bharathwaj Vijayakumar, Senior Business Intelligence Analyst,
Rowan University
#RIConf2019