DB2z LOG RBA Reset Jay Seiler(Systems Engineer) Jseiler@LLBean.com Laurie Pratt(DBA) Lpratt@LLBean.com Laurie & Jay introductions L.L. Bean, Inc

Agenda LLBean overview LLBean DB2 history LLBean’s DB2z current environment The problem Planning Testing Implementation Questions Laurie L.L. Bean, Inc

LLBean Overview Founded in 1912 by Leon Leonwood Bean Our 100th anniversary this year ! Headquartered in Freeport, Maine L.L.Bean began as a one-room operation selling a single product, the Maine Hunting Shoe We have grown to a global, multi-channel organization Laurie catalog company Company remains committed to the same honest principles upon which it was built: Focus on the customer; 100% satisfaction guarantee Respect for people Preservation and stewardship of the natural environment; staying true to our outdoor heritage New buildings constructed in accordance with the US Green Building Council's LEED (Leadership in Energy & Environmental Design) program Company's fleet contains biodiesel trucks, buses and hybrid vehicles; company supports a  fleet of propane buses at Acadia National Park L.L.Bean recycles more than 83% of its waste; donates unused office supplies and equipment to local schools L.L.Bean products are recognized around the world for their practical design and high quality; many items have been in the product line for several decades: Maine Hunting Shoe & L.L.Bean Bean Boot, 1912 Field Coat, introduced in 1924 Chamois Shirt introduced in 1928 Widely recognized Boat and Tote® Bag was introduced in 1944 as Bean's Ice Carrier Today, over 20,000 items are stocked for the catalog, Web site and stores. L.L. Bean, Inc

LLBean Overview L.L.Bean products are rigorously tested, guaranteed to last and always shipped free; trusted source for quality apparel, reliable outdoor equipment and expert advice for 100 years Our 220,000 sq. ft. Flagship campus of stores in Freeport is open 24 hours a day, 365 days a year and welcomes more than three million visitors each year Laurie Company remains committed to the same honest principles upon which it was built: Focus on the customer; 100% satisfaction guarantee Respect for people Preservation and stewardship of the natural environment; staying true to our outdoor heritage New buildings constructed in accordance with the US Green Building Council's LEED (Leadership in Energy & Environmental Design) program Company's fleet contains biodiesel trucks, buses and hybrid vehicles; company supports a  fleet of propane buses at Acadia National Park L.L.Bean recycles more than 83% of its waste; donates unused office supplies and equipment to local schools L.L.Bean products are recognized around the world for their practical design and high quality; many items have been in the product line for several decades: Maine Hunting Shoe & L.L.Bean Bean Boot, 1912 Field Coat, introduced in 1924 Chamois Shirt introduced in 1928 Widely recognized Boat and Tote® Bag was introduced in 1944 as Bean's Ice Carrier Today, over 20,000 items are stocked for the catalog, Web site and stores. L.L. Bean, Inc

LLBean Overview Winner of National Retail Federation’s “Top 10 Customers' Choice Award”; rated in the top 3 for past 4 years; nominated once again this year L.L.Bean products are recognized around the world for their practical design and high quality; many items have been in the product line for several decades Laurie Company remains committed to the same honest principles upon which it was built: Focus on the customer; 100% satisfaction guarantee Respect for people Preservation and stewardship of the natural environment; staying true to our outdoor heritage New buildings constructed in accordance with the US Green Building Council's LEED (Leadership in Energy & Environmental Design) program Company's fleet contains biodiesel trucks, buses and hybrid vehicles; company supports a  fleet of propane buses at Acadia National Park L.L.Bean recycles more than 83% of its waste; donates unused office supplies and equipment to local schools L.L.Bean products are recognized around the world for their practical design and high quality; many items have been in the product line for several decades: Maine Hunting Shoe & L.L.Bean Bean Boot, 1912 Field Coat, introduced in 1924 Chamois Shirt introduced in 1928 Widely recognized Boat and Tote® Bag was introduced in 1944 as Bean's Ice Carrier Today, over 20,000 items are stocked for the catalog, Web site and stores. L.L. Bean, Inc

LLBean DB2 History 1991 Brought in DB2z v2.3 1996 Launched LLBean website (DB2 LUW) 1996 Implemented Data Sharing 2003 Implemented a distributed front end gui client accessing DB2z data via stored procedures for our customer service reps to take orders on the phones and handle telephone inquiries 2006 Eliminated Data Sharing 2010 migrated to v9.1 Laurie Our shop was primarily a VSAM and Datacom shop prior to DB2z Orders reformatted from the website and processed on DB2z L.L. Bean, Inc

LLBean’s DB2z Current Environment Two subsystems in our Production environment DBQ1: originally intended as decision support and it is small. DBP1: large, 3TB V9.1 nfm/non data sharing Virtual tape for image copies Had a z9 (2674 mips) at start of planning, but a z10 (same mips) by the time we did the RBA Reset Jay DBQ1 Q stood for Query 3TB of DB2 data (table and index pagesets) on DBP1 DBP1 is the subsystem that was approaching the end of the log rba range L.L. Bean, Inc

The Problem On March 8, 2011, from the console: *DSNJ032I -DBP1 DSNJW307 WARNING - APPROACHING END OF 761 THE LOG RBA RANGE OF FFFFFFFFFFFF. CURRENT LOG RBA IS F00035BCE000.  DSNJ033E -DBP1 DSNJW307 THE LOG RBA MUST BE RESET 762 BEFORE THE END OF THE LOG RBA RANGE IS REACHED This was the beginning of our journey Jay L.L. Bean, Inc

How long do we have before the end of the log RBA is reached? Looks like more than a year based on our processing rates in March: FFFF00000000 (high log RBA) minus F065441C3F42 (current RBA) equals F99BBE3C0BE Divided by the average RBA increase per a day in March equals 397 DAYS Divided by the average RBA increase per a day in our peak (busiest) season equals 78 DAYS The decision was made to start planning the RBA Reset immediately Jay L.L. Bean, Inc

Planning L.L. Bean, Inc

LOG RBA Reset process for non data sharing environment IBM link to their documentation on the process: http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2/index.jsp?topic=/com.ibm.db29.doc.admin/db2z_subsystemendrba.htm In a nutshell: Use IDCAMS to delete and redefine the table spaces SYSUTILX, SYSCOPY, and SYSLGRNX and their corresponding indexes. Then, re-initialize these page sets. APAR PK28576 – new DSNZPARM, SPRMRRBA, to enable special processing to reset the RBAs in the DB2 data for non data sharing DB2 subsystems using DB2’s COPY utility. Cold Start the subsystem back to an RBA value of 0 Image copy all tablespaces and indexes with zparm SPRMRRBA on Stop DB2, Turn off zparm SPRMRRBA, Start DB2 for normal access Jay L.L. Bean, Inc

RBA vs. LRSN RBA Relative Byte Address Used by non data sharing subsystem LRSN Log Record Sequence Number Used by data sharing subsystems Jay About five years ago we went from data sharing to non-data sharing. So prior to now when we were data sharing DB2 used a LRSN rather than an RBA. My understanding is that when we went from datasharing to non-datasharing our DB2 would continue to use LRSN's right? I imagine for all intents and purposes that they are interchangeable. Are there concerns or something that I might need to know with this in mind? Response: Would you be thinking to restart the quiesced member restarted again in the future? Did you keep the logs and BSDS from this member? As far as the member is continued to be quiesced, I do not see any problem with this member. DB2 only uses LRSN when DB2 is datasharing and used in order to merge the members each log in chronological order. LRSN is the STCK value using 6 bytes. When you go back to NON-DATASHARING, DB2 would be cold started beyond the highest LRSN observed from the group. So, it looks as though the values are LRSN but actually they are RBAs. As an interesting aside; this is probably why my RBA value was that much closer to max since when I went to non-datasharing I had to cold start at a higher LRSN value than my DB2 instances. Bottom line here is LRSN or RBA made no difference L.L. Bean, Inc

Planning Continued IBM was contacted and provided great support: information from other companies that had done an RBA Reset Bank of Italy’s plan Using DIAGNOSE TYPE(576) in the image copy for indexes which reduces what is written for an image copy dataset Formalized as a project with a project manager, systems programmer, DBA and a DBA who knew DPROP Attended March NEDB2UG. John Iczkovits talked about LBI (large block interface) for image copy datasets. Never benchmarked, but did use this for the image copies. Laurie L.L. Bean, Inc

Planning Continued Looked for obsolete tables to drop Looked for tables where purges could be more aggressive Used real time stats to determine how to break up image copy jobs. Turned COPY YES on for all indexes Created validation queries to determine if any tablespaces or indexes had been missed during the image copy process Planned backout for RBA implementation; this would use SRDF (mirrored data) Laurie Ran 17 concurrent jobs to image copy user tablespaces and indexes. Broke the link for the mirror before the process, so that we could go back in the event of a disaster. L.L. Bean, Inc

DataPropagator(DPROP) aka: Infosphere Replication Server DataPropagator captures inserts/updates/deletes from the log to be able to propagate the data changes to other systems. DataPropagator keeps track of the last RBA propagated to know where to start in the log when DPROP is stopped and started. This added time to our outage to insure that DPROP was caught up with the log when we shut down DB2 and making sure that DPROP worked appropriately when DB2 was restarted for normal access after the RBA Reset. Jay There were recent ptfs that were related to RBA Reset, Data Propagator and compressed tables that indicated to us that not many companies had handled an RBA Reset for a DB2 subsystem that used DPROP. IBM told us if we were in a data sharing environment, there would be no special handling of DPROP. The DPROP DBA handled all the DPROP logistics and understanding of what needed to be done to make sure that DPROP was able to restart without issues. We had to treat it like we were doing a cold start of DPROP. L.L. Bean, Inc

Testing Tested on 2 non prod subsystems Only one non prod subsystem had DataPropagator; needed to test with DPROP twice Did a dry run just image copying all page sets in production to help determine how many jobs could run concurrently Jay L.L. Bean, Inc

Implementation Sat 11pm Brought down prod systems Once everything was down, broke the SRDF links 11:45pm DBA made sure DPROP was completely caught up midnight Started image copies It took 1.5 hours to do the image copies Sun 1:30am Restarted DB2 for normal access, Validation, cold start of DPROP and DPROP steps 4:30am Developers ran batch jobs to insert/delete/update 4:45am Go decision was made, Reconnect SRDF, Bring up prod servers & apps 6:30am Call centers online 8am Batch started Laurie (17 concurrent jobs) If any errors later because of a problem with a pageset, the plan was to reorg a tablespace or rebuild the index Planning and practice helped us to successfully complete the Log RBA reset on Sept 11, 2011. L.L. Bean, Inc

Questions? Thank you. L.L. Bean, Inc