1. E-mail Merge: Automating Financial Aid Award Letters Stirling Crow, IT Services Specialist Kim Luu, Financial Aid Technical Analyst Jagruth Peddineni, IT Student Technical Specialist June 9 th, 2016

2. Requirements Demo Issues and Troubleshooting Solution Questions? Agenda

3. Mail Merge: – A process made popular by Microsoft Word that allows for a template to be populated by data from a list. – Quick Demo Terminology

4. Each year the Financial Aid department is responsible for sending out e-mails/letters for the following: Athletic Award Renewals Athletic Award Reductions Athletic Award Cancellations Fin Aid prefers to send customized emails with a customized PDF (that cannot be edited) that provides specific information to the recipient. 4 Requirements

5. Kim uses the following process each year: 5 Requirements Generates a list with items for the merge (Student ID, Name, Address, Award Info, etc…) Using list and a letter template, a very large Mail Merge document (with the merged letters) is created. Copy the student’s particular letter out of the Mail Merge document into a blank Word Document and save. Convert the Word Document to a PDF. Create a customized email with student info. Attach customized PDF to customized email. Send email with PDF.

6. Fin Aid needs a way to do the following: Send emails with a large number of customized fields. Send emails with a large number of customized fields and… the email needs to have an attached PDF letter that has a number of customized fields. Information in Banner (RUAMAIL) needs to be updated so that there is a record of the student being sent the email and/or the attachment. It would be ideal if there was a way to document exactly what was sent to each student 6 Requirements

7. Available options: Banner Delivered ROREMAL job. 7 Requirements

8. Available options: Banner Delivered ROREMAL job. Advantages: You can send emails to a large number of students. Email sent is logged via the RUAMAIL form. Limitation: The ROREMAL job only has a limited number of customizable fields. You can’t create a customized email with more than 10 fields. You can’t send attachments. 8 Requirements

9. Available options: MS Email Merge: Link Advantages: You can send customized emails. Limitation: There’s no clear way to add customized documents to the customized email. There’s no way to update RUAMAIL for logging purposes. 9 Requirements

10. Goals: It should be a Web Application. It should do the following : Send customized emails Send customized PDFs Update Banner. Create copies of everything it sends out and store the contents somewhere accessible. 10 Application Development

11. The user needs to be able to: Upload an e-mail template Upload Word Template. Upload a.csv list. Provide an Email subject. Provide the name for the attached PDF. Provide the code and aid year for the RUAMAIL logging. 11 Application Development

12. The Application must: Allow for uploading documents. Send e-mails. Parse Word Documents. Created PDFs from Word Documents. Connect to and update a database. 12 Application Development

13. Development Options: PLSQL: This will not work. The libraries for parsing Word documents and and PDFs do not exist. GRAILS: Banner XE Self-Service applications are built and deployed with same technology. Java-based. Any open-source Java library (API) can be used. Stirling had experience building a demo application a few years ago. 13 Application Development

14. Technologies Used: Grails 2.5 WebLogic (for deployment) Spring – uploading files Spring Security – Authentication and Authorization Apache Commons – Sending e-mails Apache POI - Parses Word Files and Creates PDFs https://poi.apache.org/document/ 14 Application Development

15. Initial Demo: 15 Application Development

16. Initial Demo: 16 Application Development

17. Kim began to test using a real Financial Aid Letter. Demo To Do: Get documents to show demo. 17 Issues and Troubleshooting

18. Kim’s Test - Word Document: 18 Issues and Troubleshooting

19. Kim’s Test - CSV File: 19 Issues and Troubleshooting

20. Some of the Mail Merge Fields are not working! 20 Issues and Troubleshooting

21. Some of the Mail Merge Fields are not working! 21 Issues and Troubleshooting

22. Why aren’t the Mail Merge Fields working??! Stirling tried to update the code to use different delimiters instead of > That did not work. The problem isn’t the delimiters. 22 Issues and Troubleshooting

23. Jagruth began to debug. He discovered that the parser that brings in the XML and the information in the Word Document was splitting up the Variable Names! 23 Issues and Troubleshooting

24. FNAME is being parsed correctly: 24 Issues and Troubleshooting

25. So the program can appropriately do a find and replace: 25 Issues and Troubleshooting

26. LNAME is not being parsed! 26 Issues and Troubleshooting

27. FA_MAIL_DATE is also not being parsed correctly! 27 Issues and Troubleshooting

28. As a result the application is not able to do a find and replace on those fields (along with the others)! 28 Issues and Troubleshooting

29. Troubleshooting – Creating a Simpler Document: The parser must be having a hard time with the formatting. The XML must be too much for it to handle. What if we could make the XML a bit simpler for the parser? Stirling copied the text from the Word document into Notepad++ to strip away any formatting and then copied the text from Notepad++ into a new, clean Word document. 29 Issues and Troubleshooting

30. Troubleshooting – Creating a Simpler Document: 30 Issues and Troubleshooting

31. Troubleshooting – Creating a Simpler Document: 31 Issues and Troubleshooting

32. Troubleshooting – Creating a Simpler Document: 32 Issues and Troubleshooting

33. Troubleshooting – Creating a Simpler Document: Stirling tested and it worked. 33 Issues and Troubleshooting

34. Troubleshooting – Creating a Simpler Document: Kim began to add simple formatting with spaces and fonts and it began to break. 34 Issues and Troubleshooting

35. Troubleshooting – It’s NOT the Parser!! Jagruth began to dig deeper. Why was the parser messing up the XML? 35 Issues and Troubleshooting

36. Troubleshooting – It’s NOT the Parser!! We did some research on.docx files (Word files since 2007). We learned that these files are actually zip files. You can use an unzip utility to unzip these files. And when you do… Demo 03-UnzipWordDocDemo\CancelTestVersion2\word\document.xml 36 Issues and Troubleshooting

37. Troubleshooting – It’s NOT the Parser!! …you see that the XML is formatted strangely within the Word document itself! The parser is doing what it’s supposed to! The problem is with how MS Word saves its data. 37 Issues and Troubleshooting

38. What do we do now? How can we replace fields if the XML in the original Word Document splits up our variable names? 38 Issues and Troubleshooting

39. Options and Outcomes: Pre 2007 Word Documents (.doc). Can we parse these? Jagruth attempted to do this using the available APIs and it didn’t work. We saved the letter as a.doc, modified our code, and it didn’t work. He got errors. Parsing the.docx file to an HTML file? docx  HTML  docx  PDF ? Two problems: The Find and Replace didn’t work (The weird format of the XML came over into the HTML) We lost page formatting with the HTML. 39 Issues and Troubleshooting

40. Options and Outcomes: Find and Replace within PDF? If we’re having so many issues with docx files, perhaps we can try to do a find and replace within a PDF file that we create? PDF files can’t be as difficult as Word files, right? docx  PDF  find and replace within PDF  save PDF Jagruth looked into the following APIS: 40 Issues and Troubleshooting - Aspose – Using this API costs money. - iText – Not open source. This may have functionality but parts of it you probably have to buy. - pdfBox – This is an open source project supported by Apache! Surely this will have find and replace functionality!!!

41. Options and Outcomes: Find and Replace within PDF? The Following is from the pdfBox site: 41 Issues and Troubleshooting

42. Options and Outcomes: Find and Replace within PDF? Jagruth tried to use a previous version of pdfbox and used it’s find and replace functionality. It did not work. 42 Issues and Troubleshooting

43. Options and Outcomes: We can’t do “find and replaces” in PDFs. Word splits out our variables so we can’t do consistent find and replacing. What can we do? 43 Issues and Troubleshooting

44. Single Character Approach: Well, you can’t split a single character in half. What if you used a rare character (like an ampersand “&”) as an indicator for the application to replace a value? Not ideal – but… let’s give it a try. We updated the code that “finds and replaces” in the Word Document to go through the values in the csv file and replace them in order of the ampersand values found in the Word document. Demo 44 Issues and Troubleshooting

45. Single Character Approach: Word Document: 45 Issues and Troubleshooting

46. Single Character Approach: CSV file (fields must be in the order of the ampersands): 46 Issues and Troubleshooting

47. Single Character Approach: Demo 47 Issues and Troubleshooting

48. Single Character Approach: Results – it works! 48 Issues and Troubleshooting

49. Athletic award letters will be sent next week! Other Fin Aid letters will present challenges since they contain tables and other items that may not be formatted correctly. Testing has to happen. Possible fix using.doc files that still needs to be tried. 49 What’s Next

50. 50 Questions? Stilring Crow – stirlingcrow@unm.edustirlingcrow@unm.edu Kim Luu – kimluu@unm.edukimluu@unm.edu Jagruth Peddineni – jkrishnapeddineni@unm.edujkrishnapeddineni@unm.edu