February 11-13, 2019 Raleigh, NC

Getting the Most Out of Your Data Advanced Flow Steps Getting the Most Out of Your Data

Agenda What are Flow Steps? Basic Flow Steps Advanced Flow Steps PowerScript

What Are Flow Steps? Calculations performed as data streams in from query Manipulate existing data or derive new values Can have any number of steps Steps are performed sequentially

Examples of Basic Flows

Basic Flow Steps - Calculated Column Similar to Informer 4 Can’t reference Java objects Must return a value Regular JavaScript Use for basic calculations

Basic Flow Steps - Concatenate Joins multiple fields together Space=default delimiter

Basic Flow Steps – Coordinates from Lat/Long

Basic Flow Steps - Counter

Basic Flow Steps – Fields from another Dataset Need common key value Pull back first or all matching records Not recommended for large Datasets

Basic Flow Steps – Java Calculated Field New in 5.0.18 Same as Informer 4 Can reference Java objects Migration bucket All migrated 4 calculations Should be converted for performance

Basic Flow Steps – Percent of Total

Basic Flow Steps – Fields from another Datasource Links to another Datasource Must have common key field Can pull first or all matching records Specify which fields to retrieve Should not be used in lieu of direct linking

Basic Flow Steps – Templated Field Same as Informer 4 Can not perform calculations Does not change data Similar to mail merge

Basic Flow Steps – Time Between Calculates time between two dates Use fields and/or literal date values Time units: seconds, minutes, hours, days,

Basic Flow Steps – Zip2Geo Converts zip code to geolocation value Relative center of zip code Use for heat maps

Advanced Flow Steps - Transform

Transform Flows – Field Settings Change the column’s label (header), Data type and/or Default Value

Transform Flows – Find & Replace Finds values in the result set and substitutes the replacement value. Good for data cleansing where common misspellings or inconsistencies in data entry occur.

Transform Flows – Merge Duplicates Merges rows together, displaying the unique matched values once and the remaining fields as either an array or the first/last value of the array. Useful when dealing with data having a one-to-many relationship.

Transform Flows - Normalize Opposite of Merge Duplicates. Expands array values into separate rows and repeats any non-array values for each array entry.

Advanced Flow Steps - Remove

Remove Fields

Advanced Flow Steps - Advanced

Advanced Flows - Flush

Advanced Flows - PowerScript Use the NodeJS vm library Plain old JavaScript with a sandboxed script context. No more Java objects!

PowerScript vs. Calculated Column PS does not return a value PS can create multiple columns in single step PS can reference reserved variables and exposed libraries

PowerScript Context Variables/Objects $record object The current row. PowerScript are free to modify the record. $index number The current row number (row 1 = index 0) $omit() function When called, removes the current row $fields Field metadata $field A function to describe a field: $field(‘location’).label(‘Location’) $inputs User inputs $local Whatever you want! Is not cleared from one record to the next _ object / function Lodash v3. Learn more at https://lodash.com moment Date manipulation and formatting: https://momentjs.com

PowerScript $record The current record being processed Access existing columns via alias var name = $record.firstName + ‘ ‘ + $record.lastName; Can replace existing values $record.firstName = “Tim”; Create new columns $record.helloWorld = “Hello World”;

PowerScript $local Creates local, internal variables $local.myLocalVariable = “123”; Persists across rows Important to initialize $local.xyz = $local.xyz || 0; Does not appear in output

PowerScript $fields, $query, $datasource Access to various objects Typically would not need these See Help Center article at https://informer5.zendesk.com/hc/e n-us/articles/115005113823 for more information

PowerScript $inputs Access inputs Key/Value pair, key=input Alias $inputs[‘myInputValue’] $inputs.myInputValue

PowerScript $index Returns the current row number var rowNumber = $index;

PowerScript Functions and Libraries $omit() Removes the current record from the result set if ($record.ShipCountry === "France") { $omit(); }

PowerScript Functions and Libraries Lodash Library for manipulating arrays, numbers, strings, objects, etc. lodash.com – be sure to use version 3.x documentation Referenced by _ (underscore) $record.orderTotal = _.sum($record.orderAmount);

PowerScript Functions and Libraries Moment Library to parse, validate, manipulate, and display dates and times momentjs.com Can take a string, date object, moment object, number, or array as argument if (moment($record.ShippedDate).isAfter($record.RequiredDate)) { $record.shipStatus = "Shipped Late"; } else { $record.shipStatus = "Shipped On Time"; }

Useful PowerScript Examples Pivot arrays

JavaScript Object Notation (JSON) An object is a list of Key:Value pairs Value can be any data type Access Value with dot (.) notation or bracket [] notation

Pivot Arrays Using for loop Using lodash for (var i = 0; i < $record.s_tudent_non_courses_admission_tests_assoc_stncNonCourse.length; i++) { var name = $record.s_tudent_non_courses_admission_tests_assoc_stncNonCourse[i].replace(/\./g, '_'); $record[name] = $record.s_tudent_non_courses_admission_tests_assoc_stncScore[i]; $field(name).type('integer'); } Using lodash _.forEach($record.s_tudent_non_courses_admission_tests_assoc_stncNonCourse, (test, ndx) => { $record[_.snakeCase(test)] = $record.s_tudent_non_courses_admission_tests_assoc_stncScore[ndx]; $field(_.snakeCase(test)).type('integer'); });