Migrating to uPortal 2 at UBC Paul Zablosky University of British Columbia Copyright Paul Zablosky 2003. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author. Slide 1 I’m here to talk about our deployment of uPortal Release 2 at the University of British Columbia; specifically how we moved our Enterprise Portal – called myUBC -- to the uPortal release 2 software platform.

History uPortal 1.0 The inevitable conversion Why we did it Paper for EDUCAUSE 2001 www.educause.edu/asp/doclib/abstract.asp?ID=EDU0109 The inevitable conversion Why we did it Stay current with the community Take advantage of new rendering model and database Recover on our investment in database design and implementation Slide 2: We built our first version of myUBC on uPortal 1.0 in the fall of the year 2000 If you’re interested in our trail-blazing experiences, there’s a paper I wrote for Educause which you can find at the URL. (Search for EDU0109 with Google to find the PDF file) As we were going into production with uPortal Release 1, Release 2 was already being designed. We knew that if we were successful we would be doing a massive conversion at some time in the future. But we didn’t have much choice: release 2 was far from operationally ready. It wasn’t until the summer of 2002 that we felt that it was ready to take over from release 1 in our environment. At which point we started to make plans to migrate. We wanted to stay current with the uPortal community, and be able to take advantage of all the developing capabilities of uPortal, with its flexible rendering model and improved database. Also we wanted to make use of the work we had put into the database design and implemenation for release 2.

The Project Objectives Re-implementation of myUBC on uPortal Release 2 Maximized compatibility Preservation of existing channels Preservation of user profiles (layouts) Improved functionality for selected channels Equivalent or improved response and performance Redesigned User Interface No changes required by content providers Slide 3 Our basic objective was to re-implement the myUBC service on the uPortal Release 2 platform We wanted compatibility wherever possible – there’s no reason to change things unless it involves substantial improvement, and we certainly didn’t want to lose any of our users’ customizations and selections. We needed to preserve all the existing channels to which our users held subscriptions We needed to preserve all the user’s layouts – their personally selected channels We wanted to improve the functionality of some selected channels We had to achieve at least equivalent responsiveness and reliability We felt that the overall appearance of the User Interface was ready for a redesign And we didn’t want to force any of our external providers in different departments to make any changes to their content.

Challenges Compatibility Equivalent Robustness, Reliability Performance Integration with UBC Infrastructure CWL Authentication CWL Roles CWL User ID’s (internal keys) Oracle database Slide 4: Some of our objectives presented real challenges As far as compatibility goes, there were some things we just couldn’t do much about. Release 2 has: A Different mechanism for layout management Doesn’t provide for manipulation of colours And there is no mechanism for users to reset their layouts to the default Robustness and reliability – had to be at least as stable as what we were running before, and we would have to prove this to ourselves before going into production The same goes for performance Performance – had to support the same number of concurrent logins (200-300) The same login rate – up to 1600 per hour Integration with UBC’s Campus-wide Login Use the CWL Authentication service – easy Use the CWL’s Roles – in place of uPortal Groups Use the CWL’s internal keys as User Names ( User’s can change their login names, and yet still have the same profile in myUBC so we use an imutable key we get from the CWL as the user_name) And we use Oracle as our standard production database service

The Team - organized around roles Project Sponsor Champions the project Project Manager Creates the project plan. Organizes and directs resources throughout the project Database Administrator Responsible for all aspects of managing the database resource, including conversion. Chief Programmer Responsible for all framework programming tasks Slide 5 In putting together our project team, we started with a list of all the roles that we would need to complete the project. We then gathered team members that covered the skills list. This scales up or down depending on how large your implementation is. You can map the roles onto a very small team or a very large one. We ended up with about a dozen people on the project at any one time.

The Team – roles continued Java Developers Design software solutions for assigned tasks Platform Constructor Builds operational service platforms for development and production – hardware and operating system Testers – functional Responsible for all functional and regression testing Testers – technical Perform volume, load, and stress tests Slide 6:

The Team – roles continued Style and Presentation Coordinator Manages the presentation structure (themes & skins), graphic design, navigation, and usability. Business Analyst Responsible for user issues, usability, compatibility, communications, documentation. Also liaison with Help Desk, stakeholders, and partners. Software librarian Performs all builds, integrations and promotions Technical Writer Creates technical and user documentation Slide 7

The Team – roles continued Webmaster Installs and configures the Apache Web Server and the Tomcat container. Manages their interactions Graphic Designer Designs and creates images (skin & theme work) HTML Developer Creates HTML (skin & theme work) XSLT/Java Developer - Develops XSLT transforms (skin & theme work) Slide 8

The Development Environment The base platform (uPortal 2.0.3) UBC Channels CVS Repository UBC Infrastructure (CWL, Oracle) Shared Instances: DEVL EVAL VERF PROD Promotion between instances Channel Definition Tool Slide 9: The Development Environment Most important to establish the conventions of your development environment if you have a non-trivial team Took the basic uPortal 2.0.3 file and put it in a jar file, placed in CVS repository UBC Channels are compiled into war files and placed in CVS repository We integrated our authentication facility (Campus-wide Login) and local database Developers do their work in their personal environments, but there are 4 shared instances through which we promote code: DEVL where developers do integration testing EVAL where functional testers do their testing VERF where we verify that things work in the real production environment, and do load and stress testing PROD the production service that the users see With all these instances it’s cumbersome to create channel definitions through and interactive user interface. It’s hard to ensure consistency between the instances. So we created a “channel definition tool” to read an XML description of a channel and write its definition to the database. The XML is stored in the CVS repository. So we have an external canonical source for channel definitions.

The Development Environment (cont.) Database Instances DEVL VERF PROD Slide 10 Of course we also have separate database instances for development, verification, and production. These are not necessarily locked to the myUBC instance of the same name. The DEVL database is used for both integration and functional testing.

Platforms Shared Development Platform Production Platform Sun V120 2GB Server: SunFire 280R 2x750MHz CPU 4GB, Solaris 8 - Java Version: 1.4.2-beta Database: Oracle 8.1.7.2 Connection Pooling: Oracle with thin JDBC drivers Container: Tomcat 4.1.24 Web Server: Apache 2.0.45 (mod_jk2) Slide 11: We had one shared platform for the developers to do integration and testing Our production platform is a modest 2 processor SunFire 280

The Development Process Spanned several months Iterative with ongoing testing Framework and database access Data conversion tools Channel refinement and publication User interface and skins Operational environment, container, and Web Server Slide 12 The development process took place over several months We worked on as many things as possible in parallel, because we never knew where we might encounter problems that might slow things down. Testing was an ongoing activity throughout the development process We tested everything and gradually developed a regression suite.

The Migration Challenge UBC-specific code (e.g. CWL authentication) The channels The user data Layouts Address-book entries Bookmarks Mail preferences The conversion process (40 hours) Slide 13 This is a quick summary of the challenges we faced in migrating the service Integrating the UBC specific code – Installing the channels Developing a mechanism for transferring all the user data Finding a way to convert all the data – a process which took roughly 40 hours – without experiencing a 40 hour outage of the service.

Performance & Reliability Challenges Connection pool blocking DBCP degradation Poolman option Oracle pooling Read-only connection Slide 14 Many of the technical problems we ran into had to do with managing the database connection pool We found a blocking situation which could create a deadlock on the connection pool under heavy load We found that DBCP pooling (the pooling provided by Tomcat) can degrade seriously under load Tried Poolman but encountered some problems and decided not to invest a lot in it since it appears to be moribund. Ended up with Oracle pooling which has worked well for us Created a read-only connection to handle the bulk of the database queries.

Cutover process Existing operational service Orchestrated procedure Backout procedure New service running on a “test” URL Shutdown, reconfigure, startup 7 minute outage Slide 15 When we were ready to bring up the new service we had to deal with the fact that we had an existing operational service with 100-200 people logged in. We made a detailed plan, orchestrating each step of the cutover, with tasks divided across several members of the team for maximum parallelism to minimize the time of actual service outage. For each step in the procedure, we made sure that it could be easily and quickly backed out Just before the cutover we had the service running on a test URL with the new database to ensure everything was working correctly. Testing was an ongoing process throughout the project. At the point of cutover, it was a simply a matter of shutdown, a flurry of reconfiguration and startup of the new service. We managed the cutover with a 7-minute outage, which we thought minimal, given that just converting the database was a 40 hour process

And it ran perfectly… Needed to back out once Second time lucky 22,000 logins in our first 26 hours of operation User feedback Slide 16 We were very glad we had rehearsed our backout plan. The first time we cut over, we ran for 5 minutes, logged in 94 users, and the Java Virtual Machine crashed. We felt we’d been blindsided by this, the JVM isn’t supposed to crash [Unexpected signal 11]. We quickly backed out, apologized to the users, and tried another JVM (the beta version). After a few days more testing we tried again, and everything went smoothly at cutover. And it has run very smoothly ever since. We saw 22,000 logins in our first 26 hours of operation. User feedback – some very positive, and some critical, but in most cases they were very specific about what they wanted. The service looked different, and behaved somewhat differently. The users all went into evaluation mode as they tried things out and sent their comments to the Help Desk. It’s very useful to get this sort of feedback, we now have a lot of this new material to review for for our enhancement planning

Working with Partners and Providers Institutional conventions and consistency Content providers Other partners Slide 17 Guest page is just another web page in the institution. Broadcast information, incorporate institutional links or navigation. This page is public, and can be targeted at a general audience. Once the user is logged in, the page is personal, and the individual user’s interests prevail, for content, personalization and customization. The services lives by its content, which comes from providers in all the different units of the institution, and it’s important to engage them at the appropriate level of what they can step up to technically. Some departments may be in a position to create high function native channels, whereas others may be comfortable with just an RSS stream. We found that for most departments at UBC, with our somewhat decentralized structure of autonomous units, that a WebProxy approach worked best for most providers. Partners aren’t only content providers – In our case they include such units as Public Affairs which coordinates conventions for institutional image and the CWL group which provides the institutional authentication service.

Lessons Learned The technology is still maturing Development times are hard to estimate Test test test Slide 18 The technologies we are working with here are still undergoing extremely rapid development. New versions of things come out very frequently. It’s necessary to perform regression tests with each component change, whether it’s the JVM, a new version of Xalan (the XSLT transformation processor). It’s difficult to estimate how long a particular task will take on a project like this which involves so much new technology. Things that you think will take a couple of days, sometimes consume a couple of weeks. Sometimes it goes the other way. The only area where we could get accurate estimates was Database Management, where we have deep experience and the technology is mature. We tested everything we could think of throughout the project. We were glad we did. We could never assume anything would work before we tried it.

http://my.ubc.ca Email: Paul.Zablosky@ubc.ca Visit us at http://my.ubc.ca Email: Paul.Zablosky@ubc.ca Slide 19