1. Shared Services with Spotfire
Christian Turri
Lead BI Engineer
Spotfire London User Group
May 2016

2. Our Spotfire History
Background
Started with v3.3 in 2011, moved to v4.5 in 2013 and to v6.5 on 2014/2015
Initially just Market Risk but now being used by several departments and across multiple divisions
Some stats
8 “active” Production and 8 “passive” Contingency servers (services stopped)
25 environments in total (Development, Testing, Production, Contingency)
124 CPU Cores and 1TB of RAM installed Prod capacity
~200 Scheduled Updates reports
1200 Unique Web Player users per month
~22k “Open Analysis” success events per month
~32k Scheduled Updates refreshed per month
450k Information Link executions per month
72 Professional Client unique logins to Production per quarter
~5000 Production Library Analysis (362 IT controlled, rest user generated)
LDAP sync with 4 Active Directory domains

3. Operating Model
How we use Spotfire in our organisation?
Global User base mainly based in the UK, India and the US
Servers running 24x7 with a maintenance window on Sunday morning
Shared Services model allows for:
Core Spotfire Team developers
Project Team Spotfire developers
Power Business Users developing their own reports (self-service) and developing EUDAs (End User Developed Applications) for their business users
Some applications embedded Spotfire within their applications (mashups)
Core Spotfire Team manages infrastructure, supports and controls the platform

4. Using Spotfire in a large enterprise deployment
Issues we found
Large number of changes (avg 20 changes per month) but no versioning control
Insufficient logging available
No historical reporting
Read only access to Administration tools not supported
Extremely difficult to monitor and enforce standards and best practices
Building new Spotfire environment took 2 man days
- Large number of changes: Audit of changes
- Insufficient logging available: to keep servers running smoothly and monitor server and user activity
- No historical reporting (3.3/4.5). Tibco added Action Logging in Spotfire v5.5 but even in 7.5 it still does not cover basic logging information like how long a Scheduled Update took to run and why it failed if it did
- Read only access to Administration tools: Prevented Production Support from troubleshooting access issues without breaking policy
Extremely difficult to monitor and enforce standards: with such a large Spotfire Developer base, many of which not even on the Spotfire Core Team and are based on different regions
Building new Spotfire environment took 2 man days: as we had to apply lots of custom config changes, lots manual steps required, lots of chances for humans to make mistakes.

5. What we did to address the different platform issues
Solutions we implemented
Persisted both Spotfire Server and Web Player logs to the Spotfire metadata database
Reverse engineered Spotfire metadata database and created views to expose data in a useful way
Hooked to our enterprise monitoring tool
Developed set of “Administration” reports
Developed several PowerShell scripts to monitor and automate tasks
Developed report to refresh Scheduled Updates manually
Automated check-in of Scheduled Updates files into SVN
Persisted both Spotfire Server and Web Player logs to the Spotfire metadata database: This gave us a lot of data about our Spotfire use, “who is doing what” and historical reporting.
Reverse engineered Spotfire metadata database: We looked at the Spotfire schema and were able to build views on top that showed all the library and user directory objects in an easy to report way.
Hooked our own monitoring tool (ITRS Geneos): so that we could develop rules and alerts based on specific criteria. Sample alerts: New server config deployed, new AD group, new library folder, data source changed, library folder has incorrect permissions, etc. We also hooked to the Spotfire JMX interface and to the Web Player performance counters to get lots of server stats into our monitoring tool.
Developed set of “Administration” reports: This gave Production Support access to platform information about User Access, library permissions, Scheduled Updates status, etc.
Developed several Powershell scripts: We were able to reduce new environment build from 2 days to 2 hs.
Developed report to refresh Scheduled Updates manually: This gave Production support the ability to request scheduled updates to be refreshed where an audit of the action could be recorded, an sent to our support mailbox and no administration access will be required.
Automated check-in of Scheduled Updates files into SVN: Simple script that watches for changes on the SU file on the server and then check-ins to SVN if there are changes.

6. What we did to address the different platform issues
Solutions we implemented
Persisted both Spotfire Server and Web Player logs to the Spotfire metadata database
Reverse engineered Spotfire metadata database and created views to expose data in a useful way
Hooked to our enterprise monitoring tool
Developed set of “Administration” reports
Developed several PowerShell scripts to monitor and automate tasks
Developed report to refresh Scheduled Updates manually
Automated check-in of Scheduled Updates files into SVN

7. Screen shots

8. Screen shots

9. Screen shots

10. What we did to address the different platform issues
Solutions we implemented
Persisted both Spotfire Server and Web Player logs to the Spotfire metadata database
Reverse engineered Spotfire metadata database and created views to expose data in a useful way
Hooked to our enterprise monitoring tool
Developed set of “Administration” reports
Developed several PowerShell scripts to monitor and automate tasks
Developed report to refresh Scheduled Updates manually
Automated check-in of Scheduled Updates files into SVN

11. Screen shots

12. Screen shots

13. Screen shots

14. What we did to address the different platform issues
Solutions we implemented
Persisted both Spotfire Server and Web Player logs to the Spotfire metadata database
Reverse engineered Spotfire metadata database and created views to expose data in a useful way
Hooked to our enterprise monitoring tool
Developed set of “Administration” reports
Developed several PowerShell scripts to monitor and automate tasks
Developed report to refresh Scheduled Updates manually
Automated check-in of Scheduled Updates files into SVN

15. Screen shots

16. Screen shots

17. What we did to address the different platform issues
Solutions we implemented
Persisted both Spotfire Server and Web Player logs to the Spotfire metadata database
Reverse engineered Spotfire metadata database and created views to expose data in a useful way
Hooked to our enterprise monitoring tool
Developed set of “Administration” reports
Developed several PowerShell scripts to monitor and automate tasks
Developed report to refresh Scheduled Updates manually
Automated check-in of Scheduled Updates files into SVN

18. Screen shots

19. What we did to address the different platform issues
Solutions we implemented
Persisted both Spotfire Server and Web Player logs to the Spotfire metadata database
Reverse engineered Spotfire metadata database and created views to expose data in a useful way
Hooked to our enterprise monitoring tool
Developed set of “Administration” reports
Developed several PowerShell scripts to monitor and automate tasks
Developed report to refresh Scheduled Updates manually
Automated check-in of Scheduled Updates files into SVN

20. Screen shots
Microsoft Log Parser to parse IIS logs

21. Screen shots
Automatic export of the Spotfire Library on a folder by folder basis.
Autodump of orphaned process in IIS
Weekly restart of Web Player

22. What we did to address the different platform issues
Solutions we implemented
Persisted both Spotfire Server and Web Player logs to the Spotfire metadata database
Reverse engineered Spotfire metadata database and created views to expose data in a useful way
Hooked to our enterprise monitoring tool
Developed set of “Administration” reports
Developed several PowerShell scripts to monitor and automate tasks
Developed report to refresh Scheduled Updates manually
Automated check-in of Scheduled Updates files into SVN

23. Screen shots

24. How we did it? Log4J changes - custom Java loader for Log4J log files
Some technical details
Log4J changes - custom Java loader for Log4J log files
Log4Net changes - log4net.Appender.AdoNetAppender for live logging (see Web Player installation manual Log to Database Example)
Oracle PL/SQL package to handle Scheduled Updates logic to enhance data logged
Database views on top of Spotfire Metadata objects
PowerShell scripts interacting with Spotfire config.bat, SVN command line, Spotfire Automation Services, Event Logs and Microsoft Log Parser for IIS logs

25. Screen shots

26. Screen shots

27. Future Enhancements planned
Scheduled automated SVN check-in of Information Link SQL, IronPython Scripts and other various configuration files (web.config, log4net, log4j.*, etc)
Automated for users trying to access a report they don’t have access
Automated to the relevant support team when a scheduled update fails
Automated way of distributing new config files
Look at automating Production releases

28. Q & A