1. Audit & Reporting with Alfresco & NoSQL architecture Lucas Patingre Alfresco consultant and technical lead at Zaizi

2. Summary Some context Alfresco audit Scaling the audit Benefits Expanding on the architecture Going forward

3. Why auditing? A bit of context

4. Zaizi for Alfresco: Platinum Partner and Best System Integrator Partner of Year 2012 & 2013 Specialist Alfresco ECM partner. Implemented the biggest and most complex Alfresco projects in the UK. Help global enterprises manage their growing information volumes by leveraging Alfresco’s outstanding performance and scalability. Alfresco Partner of the Year 2012 & 2013

5. Audit data from Alfresco Who When What Action Target Details /alfresco-access /transaction /action=<actionNamegt /sub-actions=<sub action listgt /path=<prefixPathgt /type=<prefixTypegt /node=<nodeRefgt /user=<usergt /copy /from /node=<nodeRefgt /path=<prefixPathgt /type=<prefixTypegt /move /from /node=<nodeRefgt /path=<prefixPathgt /type=<prefixTypegt /properties /from=<mapOfValuesgt /<propertyNamegt=<propertyValuegt /to=<mapOfValuesgt /<propertyNamegt=<propertyValuegt /add=<mapOfValuesgt /<propertyNamegt=<propertyValuegt /delete=<mapOfValuesgt /<propertyNamegt=<propertyValuegt /aspects /add=<mapOfNamesgt /<aspectNamegt=null /delete=<mapOfNamesgt /<aspectNamegt=null /version-properties=<mapOfValuesgt /sub-action/<sequencegt /action=<actionNamegt /copy /move /properties /aspects

6. Existing Alfresco audit Sharing Alfresco database

7. Components overview

8. Alfresco's audit dashlet

9. Alfresco's audit storage

10. SQL to retrieve audit entries SELECT entry.id 'Id', entry.audit_time 'Time', user_string.string_value 'User', act_string.string_value 'Application', sv.string_value 'Value' FROM alf_audit_entry entry INNER JOIN alf_prop_value user ON (entry.audit_user_id = user.id) INNER JOIN alf_prop_string_value user_string ON ((user.persisted_type = 3 OR user.persisted_type = 5) AND user.long_value = user_string.id) INNER JOIN alf_audit_app app ON (entry.audit_app_id = app.id) INNER JOIN alf_prop_value act ON (app.app_name_id = act.id) INNER JOIN alf_prop_string_value act_string ON ((act.persisted_type = 3 OR act.persisted_type = 5) AND act.long_value = act_string.id) INNER JOIN alf_prop_link pl on (pl.root_prop_id = entry.audit_values_id) INNER JOIN alf_prop_value pv on (pl.value_prop_id = pv.id) LEFT JOIN alf_prop_string_value sv on (sv.id = pv.long_value and (pv.persisted_type = 3 OR pv.persisted_type = 5))

11. Alfresco's RM audit

12. Scaling the audit Presentation of the components

13. The challenges Make the audit scale without hindering Alfresco Keep the audit queries fast Not delay too much time until stored Backward compatible with Alfresco's default Not break existing RM view audit Have a similar look-and-feel

14. The Alfresco search approach Move from Lucene Embedded in Alfresco Limited inspection tools To SOLR Externalised Can be clustered Comes with an administration console

15. Components overview

16. Syslog Standard, efficient and well integrated in Java Easy to implement a file rotation Possibility to re-compute all the audit data from file Lighten the weight on the database that is no longer a bottleneck

17. Logstash Open Source Works well with log files Able to handle our audit.log But potentially others too (OSSEC) Already has an ElasticSearch connector OOTB

18. Elastic search Open source Powerful indexing capabilities Easily scalable Can be queried from Alfresco

19. Kibana Open source Web UI for easy access

20. Scaling the audit (2) Overview of the implementation

21. Override the audit component

22. Specialise the audit component auditDAO.createAuditEntry(applicati onId, time, username, auditData); logAudit.createAuditEntry(applicati onId, time, username, auditData); Create JSON for action Add non-action-related parameters to JSON Configure Syslog Log the resulting JSON to audit.log

23. Quality-of-life improvements Poll Syslog availability If not available, switch system to readonly When back available, re-enable Toggle logging system Only file/only database/both Availability through JMX

24. Retrieve audit data

25. Replace audit webscript var nodeRefAuditURI = '/api/node/' model.nodeRef +'/rmauditlog'; var auditURI = "/api/rma/admin/rmauditlog"; var nodeRefAuditURI = '/api/node/'+ model.nodeRef +'/esauditlog?appname=RM'; var auditURI = "/api/es/admin/esauditlog"; audit.get.js Switch at the share level Enables us to handle a richer result returned Modifying the Alfresco webscript would be a viable approach too

26. Reap the benefit What this whole work was for

27. Performance / stability Asynchronous Processing after audit.log non-blocking Independent Audit falling doesn't bring Alfresco down Alfresco falling doesn't prevent to consult ES No stress on Alfresco database when querying audit data

28. Extend use of “view audit log”

29. Draw real time statistics out of it

30. Expanding on the architecture Non-alfresco-generated audit data

31. Auditing the logins <RecordValue key="user" dataExtractor="simpleValue" dataSource="/alfresco-access/loginUser" dataTrigger="/alfresco-access/login" /> <RecordValue key="user" dataExtractor="simpleValue" dataSource="/alfresco-access/loginUser" dataTrigger="/alfresco-access/loginFailure" /> /alfresco-access /login/user= /loginFailure/user= /logout/user=

32. Introduction to OSSec Open Source Intrusion Detection System log analysis file integrity checking rootkit detection Grabs data from most of our systems Software OS Inject in ElasticSearch via logstash

33. Auditing the security

34. Going forward How does it fit with Alfresco 5?

35. Alfresco5 analytics

36. Take-away Intellectual doggy bag

37. Conclusion A lot Of open source products Of scaling potential A reasonable amount Of Alfresco customisation A little Changed from the Out Of The Box Alfresco UI No Code change for non-alfresco technologies

38. http://xkcd.com/1256/