1. #SummitNow Super Size Your Search 14 th November 2013 Fran Alvarez (Zaizi)

2. #SummitNow Agenda Myself & My company Background Our Solution Scenario Demo Conclusions

3. #SummitNow About me Director of Zaizi Iberia and Lead Architect Alfresco Certified Engineer Responsible of large Alfresco architectures Semantic Consultant for Sensefy Alfresco Meetups Organizer

4. #SummitNow We are an Open Source Development Company that helps people work together more effectively HQ: London (UK) Seville (Spain) Colombo (Sri Lanka) Singapore

5. #SummitNow What we offer Open Source System Integrator Specialist in ECM Platinum Alfresco partner Best Systems Integrator Partner EMEA 2012 Best Systems Integrator Partner EMEA 2013 Million $ Club Awarded in 2013 Support 24/7

6. #SummitNow Overview How to build and manage your search server: 1.Scenario 2.Introducing Apache ManifoldCF 3.Zaizi Integrated Search Solution

7. #SummitNow Scenario An overview about the typical complex search architecture

8. #SummitNow Scenario - Alfresco limitations Alfresco supports these search engines: Apache Lucene (embedded) Apache Solr (provided by Alfresco) needs development if other repositories must be involved Every other approach must be implemented (ScheduledActions, WebScripts, etc..)

9. #SummitNow Scenario – Embedded Simple Search Architecture Alfresco is the only one repository involved in the architecture using the embedded search engine: the repository must take care of indexes also managing index transactions Indexes Alfresco FrontEnd applications Apache Lucene

10. #SummitNow Scenario – Embedded - Cluster Embedded Not easy to scale out with Lucene 1.every cluster must have its own search indexes 2.The cluster must synchronize indexes Indexes Alfresco Apache Lucene Indexes Alfresco Apache Lucene JGroups

11. #SummitNow Scenario – Simple Architecture Simple search architecture Alfresco is the only one repository involved in the architecture with an external search server 1.The search server can be used for publish contents in the front end architecture 2.The repository will stay in the logic backend Search Engine Indexes Alfresco FrontEnd applications

12. #SummitNow Scenario – Publish with search A search engine can be used for: advanced management of search indexes scaling out executing complex search on contents publishing contents in the FE architecture

13. #SummitNow Scenario – Publish with search Publish with search architecture Alfresco is the only one repository involved in the architecture with an external search server 1.The search server can be used for publishing contents in the front end architecture (HTML) 2.The repository will stay in the logic backend Search Engine Indexes Alfresco FrontEnd applications BackEndFrontEnd Lucene / Solr Indexes

14. #SummitNow Scenario – Simple Architecture Simple Search Architecture Alfresco is the only one repository involved in the architecture with an external search server 1.The search server can be used for publish contents in the front end architecture 2.The repository will stay in the logic backend Search Engine Indexes Alfresco FrontEnd applications

15. #SummitNow Scenario – Complex Architecture 1.Alfresco is only one of the platforms that must be involved in your search architecture 2.You don’t want to increase the development effort 3.You want just something to configure

16. #SummitNow Scenario – Complex Architecture Architecture with different ECM systems Alfresco is one of the content platforms that must be involved in the indexing process Alfresco Search Engine Indexes SharePoint FileNet CMIS JIRA Google Drive DropBox

17. #SummitNow Scenario – Complex Architecture Architecture with different ECM systems Alfresco is one of the content platforms that must be involved in the indexing process Alfresco Search Engine Indexes SharePoint FileNet CMIS JIRA Google Drive DropBox

18. #SummitNow Scenario – Complex Architecture Architecture with different ECM systems Alfresco is one of the content platforms that must be involved in the indexing process Alfresco Search Engine Indexes SharePoint FileNet CMIS JIRA Google Drive DropBox

19. #SummitNow Introducing Apache ManifoldCF

20. #SummitNow Apache ManifoldCF - History ManifoldCF code base was granted by MetaCarta to the Apache Software Foundation in December 2009. The MetaCarta effort represented more than five years of successful development and testing in multiple, challenging enterprise environments. The project was graduated as Apache Top Level Project in July 2012.

21. #SummitNow Apache ManifoldCF – What is? Open Source crawler crawling model (add, change, delete) schedule jobs to create indexes get contents from repositories push contents on search servers

22. #SummitNow Apache ManifoldCF – What is? Repository 1 Repository 3 Repository 4 Repository 2 Apache ManifoldCF Search Server 1 Search Server 2 Search Server 3 Search Server 4

23. #SummitNow Apache ManifoldCF – What is? Out-Of-The-Box it is distributed as a webapp REST API Authority Service ACL indexes Crawler UI can be embedded in any Java application

24. #SummitNow Apache ManifoldCF – Why? Reliability Incremental Flexible Multi repositories Security model Monitoring

25. #SummitNow ManifoldCF – Why? - Reliability Jobs scheduling and configuration are stored in the database to maintain the state of all the executions Repository 1 Repository 3 Repository 4 Repository 2 Apache ManifoldCF Search Server 1 Search Server 2 Search Server 3 Search Server 4 Pull Agent Daemon Database

26. #SummitNow ManifoldCF – Why? - Incremental get content changesets obtained from the repository API Repository 1 Apache ManifoldCF Pull Agent Daemon Database query Complete Changesets

27. #SummitNow ManifoldCF – Why? - Flexible If the repository can't supply all the changes Manifold can discover them through crawling Apache ManifoldCF Pull Agent Daemon Database query Incomplete Changesets Change Discovery NN

28. #SummitNow ManifoldCF – Why? – Multi repo Jobs can retrieve contents from the following repositories: Google Drive Dropbox HDFS CMIS-compliant Alfresco IBM FileNet EMC Documentum Microsoft SharePoint OpenText LiveLink Autonomy Meridio Memex Patriarch Windows Share/DFS Generic JDBC Generic Filesystem Generic RSS and Web

29. #SummitNow ManifoldCF – Why? – Multi repo Jobs can ingest contents to the following search servers: Apache Solr ElasticSearch OpenSearchServer MetaCarta GTS

30. #SummitNow ManifoldCF – Why? - Security Retrieve per-content ACLs Repository 1 Repository 3 Repository 4 Repository 2 Apache ManifoldCF Search Server 1 Search Server 2 Search Server 3 Search Server 4 Authority Service Authority 1 Authority 2 access tokens

31. #SummitNow ManifoldCF – Why? - Security Retrieve per-content ACLs Repository 1 Repository 3 Repository 4 Repository 2 Apache ManifoldCF Search Server 1 Search Server 2 Search Server 3 Search Server 4 Authority Service Authority 1 Authority 2 user access tokens user specific search results

32. #SummitNow ManifoldCF – Why? – Monitoring UI Crawler allows you to: configure jobs and connectors monitor jobs execution monitor contents ingestion status reports document status queue status history reports simple history maximum activity maximum bandwidth result histogram

33. #SummitNow ManifoldCF – Architecture Repository Job Search Server ACLs

34. #SummitNow ManifoldCF – Architecture Repository Job Search Server ACLs Repository Connector

35. #SummitNow ManifoldCF – Architecture Repository Job Search Server ACLs Repository ConnectorOutput Connector

36. #SummitNow ManifoldCF – Architecture Repository Job Search Server ACLs Repository ConnectorOutput Connector Authority Connector

37. #SummitNow ManifoldCF – Architecture Repository Job Search Server ACLs Repository Connector query to retrieve contents Output Connector Authority Connector

38. #SummitNow ManifoldCF – Architecture Repository Job Search Server ACLs Repository Connector query to retrieve contents Output Connector metadata mapping content ingestion Authority Connector

39. #SummitNow ManifoldCF – Architecture Repository Job Search Server ACLs Repository Connector query to retrieve contents Output Connector metadata mapping content ingestion Authority Connector retrieve content ACEs

40. #SummitNow ManifoldCF – Architecture Repository Job Search Server ACLs Repository Connector query to retrieve contents Output Connector metadata mapping content ingestion Authority Connector retrieve content ACEs verbal description crawling model scheduling

41. #SummitNow Who is using ManifoldCF?

42. #SummitNow ManifoldCF - Resources The project is available at http://manifoldcf.apache.org/ From this website you can access to the mailing lists, documentation and download links for binaries and source.

43. #SummitNow ManifoldCF – Resources - Book ManifoldCF in Action by Karl Wright published by Manning Karl is the original developer and the principal committer of Apache ManifoldCF The book is available at http://www.manning.com/wright http://www.manning.com/wright

44. #SummitNow Background Let’s put a bit of context

45. #SummitNow Those Old Days… Only Lucene in Alfresco 3.4- Indexes were managed within Alfresco context Permissions were checked after Lucene returned all results

46. #SummitNow Alfresco 4 is… Common Enemies Find a single document Return large data sets Filter by permissions Be fast! “Sometimes one superhero is not enough”

47. #SummitNow Present Solr as Search Subsystem Indexes are managed outside Alfresco context Permissions are checked at query time No in-transaction index

48. #SummitNow Alfresco + Solr Approach Quite a good architecture Takes care of both performance and usability Flexibility in deployment and installations However… Sometimes we just need to use something else

49. #SummitNow Future Don’t freak out dude! We can arrange something

50. #SummitNow Our solution Uses Apache Manifold Decoupled from Alfresco Can be integrated with either Alfresco or any other repository vendor Preserve security and permissions within results It’s included in our Semantic solution: Sensefy! API to manage Manifold Services API for searching, decoupling Search engine chosen Simple Bundled UI Lots of Manifold Customization

51. #SummitNow Apache ManifoldCF Open Source Apache SF Project Get content from repos Push content on search services Crawling model (add, change, delete) Respect permissions, bitch!

52. #SummitNow Our ManifoldCF Contribution Alfresco Repository Connector: New implementation Amazon Cloud Search Output Connector Alfresco Authority Connector: Design & Development

53. #SummitNow Some of our most famous villains

54. #SummitNow Several Alfresco instances Current Alfresco instances don’t share indexes Indexes can’t be merged Can’t have federated search No good approach for presenting results to users

55. #SummitNow Several Alfresco instances Our solution Once index to rule them all Data origin is irrelevant (or not if we don’t) Single search across repositories You choose your search engine!

56. #SummitNow Alfresco + Other data providers Current Alfresco Search subsystem != Other provider Search services Alfresco can’t reach external data No way to merge results uniformly to end users

57. #SummitNow Alfresco + Other data providers Our solution Search engine is shared All of them speak ‘our language’ Alfresco can reach external data through Results are present and accessible between data providers

58. #SummitNow Alfresco + O(TB) data Current Alfresco Search subsystem Single or clustered Solr Every Solr its own index No chance to apply scale techniques Huge server are required and performance might be compromised

59. #SummitNow Alfresco + O(TB) data Our Solution Alfresco uses our index Indexing techniques can be applied according to use cases Sharding, Replication… Search strategy can be adopted with best suitable search solution

60. #SummitNow Other benefits Extract, index and map information from any other sources Putting them together in a single index Permissions are checked just once Search capabilities: facets, highlighting… Red Link Apache ManifoldCF Search Server Authority Service Alfresco Permissions Alfresco

61. #SummitNow Demo

62. #SummitNow Demo : Architecture

63. #SummitNow Demo: Who are these guys? Christian Bale, Actor Christopher Nolan’s Batman Gareth Bale, footballer Real Madrid latest star

64. #SummitNow Conclusions Searching & Indexing in most popular Cloud Search solutions Retrieving information from most popular repositories and data providers altogether Manage permission and security for data Fully supported by us!

65. #SummitNow Conclusions

66. #SummitNow What’s coming How can we improve it, dude? - Powerful UI - New connectors - Large data volume benchmarking - Share integration

67. #SummitNow We are not Batman But we can be your Superhero Zaizi Ltd.Fran Álvarez enquiries@zaizi.com enquiries@zaizi.com falvarez@zaizi.com (+44) 20-3582-8330(+34) 666-424-364

68. #SummitNow Thank you! May you want to help us with this one?