Best Practices for Alfresco Replication, Backup and Disaster Recovery Richard McKnight Principal Consultant Brian Long Principal Consultant

Agenda  Intro and a bit of history  BC/DR, Replication Primer  Our BC/DR Choices  About our Implementation  Demo At the end of this presentation you should have enough information to make an informed decision about BC/DR and Global Replication

A Bit of History Sub Caption

It’s Been a Long Time Coming!  First proposed in 2009 for Replication Good feedback but seen as being a lot of work.  Socialized with Customers from 2011 through 2013 The general push back was rooted in the reluctance to pay for this as a bespoke engagement.  Socialized Internally circa 2012 There was not enough demand to justify prioritizing this over other priorities.  First Prototype Commissioned in 2013 We verified that Transfer Services was not designed with this use case in mind.

So What Changed?  A wave of deals requiring global replication and BC/DR  A couple of POC projects willing to try Active/Active replication.  A couple of projects willing to use code built leveraging the POC projects.  A framework to allow consulting to build and price reusable IP.

Why is Global Replication so Hard? Global Replication ≠ WAN level HA Clustering  You cannot guarantee that the system will run without failures all the time  During outages trade-offs must be made between consistency and availability.  Detecting and recovering from outages add complexity to global implementations  Under normal operations, latency between data centers must be considered

BC/DR and Replication Primer This section goes into some of the basics of Business Continuity, Disaster Recovery and Replication.

Disasters MUST Be Planned For  Outages Happen  Interruptions in operations can be costly.  Disaster Recovery costs money too!  What BC/DR Method you chose is a balancing act!

BC/DR Requirements  Recovery Point Objective The time period for which missing data could be tolerated  Recovery Time Objective The length of time that the organization could tolerate a service interruption.

Systems Have Become Global  Companies want to collaborate across the globe.  Global systems must operate properly even in the face of outages.  The mechanisms that support global systems can also support BC/DR

Backing Up Alfresco An Alfresco repository consists of content, metadata and a search index. The various pieces of the repository must be backed up in a specific order. Some BC/DR strategies are simply a remote backup.

BC/DR Approaches We will use these definitions to distinguish between the methods discussed. Hot standby and active/active use the same replication method.  Warm Standby The infrastructure is up and running with periodic backups. Alfresco not running. The backup site is a replica of the main site.  Hot Standby The far system is running in read only mode. Replication happens at the Application level.  Active/Active Replication at the application level. All sites in read/write mode.

Compare and Contrast CharacteristicWarmHot Identical InstancesYesNo Replication MethodInfrastructure components Application level Area of ComplexitySynchronizing the snapshots Application code to support replication RTO ConsiderationsDR site must be started DR site already running RPO ConsiderationsSupports only fully replicated transactions Supports partially replicated transactions

The Problem with Warm Standby The biggest challenge to supporting really tight RPO criteria with warm standby is getting rid of integrity errors. Currently the only way to remove all integrity errors from a continuous back is to roll back the DB transactions until no content integrity errors exist.

Warm Standby at its Worst #TransactionDBFile 1Add 10 filesCompletely Transferred10 files Transferred 2Add 100 more FilesCompletely Transferred99 Files transferred 3Update 20 ObjectsCompletely TransferredAll updates complete 4Delete 20 FilesCompletely TransferredN/A 5Add 10 more filesPartially Transferred7 Files transferred In the current warm standby we would only be able to include transaction #1. Transaction #5 would always be thrown out because the database would not commit a partial transaction. Because transaction #2 would be incomplete (due to missing files), it and all subsequent transactions would need to be discarded.

Warm Standby #FAIL Using current methods, updates to 140 objects are sacrificed because of one missing file. In the absence of any coordination between the database and file system replication, this scenario is not far fetched The inability to synchronize the replication of content and metadata makes warm standby unattractive. What if…. We only had to throw out the bad object(s)?

Warm Standby Alternative The following has been discussed as a way to handle a less than perfect backup (which is exactly what could happen after an incident).  Accepting objects with “hanging content URLs”  Adding a “Lost Content” aspect to identify the object as being “broken” and to record the original content URL.  Updating the content URL to point to a “missing content” file. This sort of recovery process would allow the 139 objects that were updated to be included in the recovered repository. This would support very aggressive RTO and RPO criteria, but it has not been developed

Our BC/DR Choices All choices are trade offs

And the Nominees are  Warm Standby Usually the first choice, but lack of synchronization between the database and file replication makes it unattractive  Hot Standby and Active/Active Attractive because of the ability to support BC/DR and Global repositories. Early clients wanted this but as a supported part of the product.  Modified Warm Standby This could potentially provide the best RTO/RPO for BC/DR only. Risky because it could potentially require changes to core Alfresco code.

And the Winner is Hot Standby – Active/Active  Support for both Global Repositories and BC/DR  Progressive replication to allow for soft failures.  Met requirements for multiple customers and prospects with business critical use cases

The Runner Up & Supporting Cast Modified Warm Standby  Could work in concert with Active/Active  Warm standby could provide preferred BC/DR  Active/Active could be BC/DR if warm standby is unavailable Active/Active paired with Modified Warm Standby is a killer combination

About our Implementation This sections just has slides built from the Infosys Engagement report

Business Continuity

High Availability + BC

Disaster Recovery + HA/BC

Flexibility / Configurability Wide Range of Business Requirements Wide Range of Technical Requirements Segmented Multi-Stage Replication

Node Segments & Events Stub Property/Properties (non-content) Association(s) Access Control(s) Content(s) Etc… DeleteNode MoveNode DeleteAssociation InvalidateProperties InvalidateContents Etc…

Scheduling Options Synchronous Asynchronous Batch Including Previously Failed Actions On-Demand Typically for Content Never Typically for Invalidate Events

What is Distributable? Folders declared as Distribution Roots Configured by an Administrator All Children of Distribution Roots Ignored Aspects Renditions Thumbnails Etc…

Journal

Transport

Conflict Resolution Simple/Smart Merging by Default Supports Custom Implementations Spring Pluggable

Demonstration

White: Section Head

White: Title and Content Level 1 Level 2 Level 3 – DO NOT USE

White: Two Column Level 1 Level 2 Level 3 – DO NOT USE

White: Comparison Heading Left Level 1 Level 2 Level 3 – DO NOT USE Heading Right

White: Title

White: Title (no brand)

Sample Code { // extract avm store id and path var fullpath = url.extension.split("/"); if (fullpath.length == 0) { status.code = 400; status.message = "Store id has not been provided."; status.redirect = true; break script; } var storeid = fullpath[0]; var path = (fullpath.length == 1 ? "/" : "/" + fullpath.slice(1).join("/")); }

White: Blank

White: Blank (no brand)

White: Picture and Caption

Image Library

LemonTangerineSkyBlueberryLeaf Chalkboar d SoilClaySand Color Palette

Iconography

Logo and Tagline

Other Graphics

A Sample Slide This is a bullet This is a level 2 bullet This is a level 3 bullet Header without bullet This is a level 2 bullet This is level 3, avoid going below this level. 60% Hybrid 20% Cloud 20% On-Prem Example Graphic

Fonts & Styling Helvetica Neue Medium Helvetica Neue Medium (Bold) Helvetica Neue Light Helvetica Neue Light (Bold) Helvetica Neue Thin Helvetica Neue Thin (Bold)

Color Palette & Sample Shapes LemonTangerineSkyBlueberryLeaf Chalkboar d SoilClaySand

Alfresco in information intensive industries Government & Intelligence Banking & Insurance ManufacturingMedia & Publishing High Tech