1. DB2z LOG RBA Reset Jay Seiler(Systems Engineer) Jseiler@LLBean.com
Laurie Pratt(DBA)
Laurie & Jay introductions
L.L. Bean, Inc

2. Agenda LLBean overview LLBean DB2 history
LLBean’s DB2z current environment
The problem
Planning
Testing
Implementation
Questions
Laurie
L.L. Bean, Inc

3. LLBean Overview Founded in 1912 by Leon Leonwood Bean
Our 100th anniversary this year !
Headquartered in Freeport, Maine
L.L.Bean began as a one-room operation selling a single product, the Maine Hunting Shoe
We have grown to a global, multi-channel organization
Laurie
catalog company
Company remains committed to the same honest principles upon which it was built:
Focus on the customer; 100% satisfaction guarantee
Respect for people
Preservation and stewardship of the natural environment; staying true to our outdoor heritage
New buildings constructed in accordance with the US Green Building Council's LEED (Leadership in Energy & Environmental Design) program
Company's fleet contains biodiesel trucks, buses and hybrid vehicles; company supports a  fleet of propane buses at Acadia National Park
L.L.Bean recycles more than 83% of its waste; donates unused office supplies and equipment to local schools
L.L.Bean products are recognized around the world for their practical design and high quality; many items have been in the product line for several decades:
Maine Hunting Shoe & L.L.Bean Bean Boot, 1912
Field Coat, introduced in 1924
Chamois Shirt introduced in 1928
Widely recognized Boat and Tote® Bag was introduced in 1944 as Bean's Ice Carrier
Today, over 20,000 items are stocked for the catalog, Web site and stores.
L.L. Bean, Inc

4. LLBean Overview
L.L.Bean products are rigorously tested, guaranteed to last and always shipped free; trusted source for quality apparel, reliable outdoor equipment and expert advice for 100 years
Our 220,000 sq. ft. Flagship campus of stores in Freeport is open 24 hours a day, 365 days a year and welcomes more than three million visitors each year
Laurie
Company remains committed to the same honest principles upon which it was built:
Focus on the customer; 100% satisfaction guarantee
Respect for people
Preservation and stewardship of the natural environment; staying true to our outdoor heritage
New buildings constructed in accordance with the US Green Building Council's LEED (Leadership in Energy & Environmental Design) program
Company's fleet contains biodiesel trucks, buses and hybrid vehicles; company supports a  fleet of propane buses at Acadia National Park
L.L.Bean recycles more than 83% of its waste; donates unused office supplies and equipment to local schools
L.L.Bean products are recognized around the world for their practical design and high quality; many items have been in the product line for several decades:
Maine Hunting Shoe & L.L.Bean Bean Boot, 1912
Field Coat, introduced in 1924
Chamois Shirt introduced in 1928
Widely recognized Boat and Tote® Bag was introduced in 1944 as Bean's Ice Carrier
Today, over 20,000 items are stocked for the catalog, Web site and stores.
L.L. Bean, Inc

5. LLBean Overview
Winner of National Retail Federation’s “Top 10 Customers' Choice Award”; rated in the top 3 for past 4 years; nominated once again this year
L.L.Bean products are recognized around the world for their practical design and high quality; many items have been in the product line for several decades
Laurie
Company remains committed to the same honest principles upon which it was built:
Focus on the customer; 100% satisfaction guarantee
Respect for people
Preservation and stewardship of the natural environment; staying true to our outdoor heritage
New buildings constructed in accordance with the US Green Building Council's LEED (Leadership in Energy & Environmental Design) program
Company's fleet contains biodiesel trucks, buses and hybrid vehicles; company supports a  fleet of propane buses at Acadia National Park
L.L.Bean recycles more than 83% of its waste; donates unused office supplies and equipment to local schools
L.L.Bean products are recognized around the world for their practical design and high quality; many items have been in the product line for several decades:
Maine Hunting Shoe & L.L.Bean Bean Boot, 1912
Field Coat, introduced in 1924
Chamois Shirt introduced in 1928
Widely recognized Boat and Tote® Bag was introduced in 1944 as Bean's Ice Carrier
Today, over 20,000 items are stocked for the catalog, Web site and stores.
L.L. Bean, Inc

6. LLBean DB2 History 1991 Brought in DB2z v2.3
1996 Launched LLBean website (DB2 LUW)
1996 Implemented Data Sharing
2003 Implemented a distributed front end gui client accessing DB2z data via stored procedures for our customer service reps to take orders on the phones and handle telephone inquiries
2006 Eliminated Data Sharing
2010 migrated to v9.1
Laurie
Our shop was primarily a VSAM and Datacom shop prior to DB2z
Orders reformatted from the website and processed on DB2z
L.L. Bean, Inc

7. LLBean’s DB2z Current Environment
Two subsystems in our Production environment
DBQ1: originally intended as decision support and it is small.
DBP1: large, 3TB
V9.1 nfm/non data sharing
Virtual tape for image copies
Had a z9 (2674 mips) at start of planning,
but a z10 (same mips) by the time
we did the RBA Reset
Jay
DBQ1 Q stood for Query
3TB of DB2 data (table and index pagesets) on DBP1
DBP1 is the subsystem that was approaching the end of the log rba range
L.L. Bean, Inc

8. The Problem On March 8, 2011, from the console:
*DSNJ032I -DBP1 DSNJW307 WARNING - APPROACHING END OF 761
THE LOG RBA RANGE OF FFFFFFFFFFFF. CURRENT LOG RBA IS F00035BCE000. 
DSNJ033E -DBP1 DSNJW307 THE LOG RBA MUST BE RESET 762
BEFORE THE END OF THE LOG RBA RANGE IS REACHED
This was the beginning of our journey
Jay
L.L. Bean, Inc

9. How long do we have before the end of the log RBA is reached?
Looks like more than a year based on our processing rates in March:
FFFF (high log RBA)
minus F065441C3F42 (current RBA)
equals F99BBE3C0BE
Divided by the average RBA increase per a day in March equals 397 DAYS
Divided by the average RBA increase per a day in our peak (busiest) season equals 78 DAYS
The decision was made to start planning the RBA Reset immediately
Jay
L.L. Bean, Inc

10. Planning
L.L. Bean, Inc

11. LOG RBA Reset process for non data sharing environment
IBM link to their documentation on the process:
In a nutshell:
Use IDCAMS to delete and redefine the table spaces SYSUTILX, SYSCOPY, and SYSLGRNX and their corresponding indexes. Then, re-initialize these page sets.
APAR PK28576 – new DSNZPARM, SPRMRRBA, to enable special processing to reset the RBAs in the DB2 data for non data sharing DB2 subsystems using DB2’s COPY utility.
Cold Start the subsystem back to an RBA value of 0
Image copy all tablespaces and indexes with zparm SPRMRRBA on
Stop DB2, Turn off zparm SPRMRRBA, Start DB2 for normal access
Jay
L.L. Bean, Inc

12. RBA vs. LRSN RBA Relative Byte Address
Used by non data sharing subsystem
LRSN
Log Record Sequence Number
Used by data sharing subsystems
Jay
About five years ago we went from data sharing to non-data sharing. So prior to now when we were data sharing DB2 used a LRSN rather than an RBA. My understanding is that when we went from datasharing to non-datasharing our DB2 would continue to use LRSN's right? I imagine for all intents and purposes that they are interchangeable. Are there concerns or something that I might need to know with this in mind?
Response:
Would you be thinking to restart the quiesced member restarted again in the future? Did you keep the logs and BSDS from this member? As far as the member is continued to be quiesced, I do not see any problem with this member. DB2 only uses LRSN when DB2 is datasharing and used in order to merge the members each log in chronological order. LRSN is the STCK value using 6 bytes When you go back to NON-DATASHARING, DB2 would be cold started beyond the highest LRSN observed from the group. So, it looks as though the values are LRSN but actually they are RBAs.
As an interesting aside; this is probably why my RBA value was that much closer to max since when I went to non-datasharing I had to cold start at a higher LRSN value than my DB2 instances.
Bottom line here is LRSN or RBA made no difference
L.L. Bean, Inc

13. Planning Continued IBM was contacted and provided great support:
information from other companies that had done an RBA Reset
Bank of Italy’s plan
Using DIAGNOSE TYPE(576) in the image copy for indexes which reduces what is written for an image copy dataset
Formalized as a project with a project manager, systems programmer, DBA and a DBA who knew DPROP
Attended March NEDB2UG. John Iczkovits talked about LBI (large block interface) for image copy datasets. Never benchmarked, but did use this for the image copies.
Laurie
L.L. Bean, Inc

14. Planning Continued Looked for obsolete tables to drop
Looked for tables where purges could be more aggressive
Used real time stats to determine how to break up image copy jobs.
Turned COPY YES on for all indexes
Created validation queries to determine if any tablespaces or indexes had been missed during the image copy process
Planned backout for RBA implementation; this would use SRDF (mirrored data)
Laurie
Ran 17 concurrent jobs to image copy user tablespaces and indexes.
Broke the link for the mirror before the process, so that we could go back in the event of a disaster.
L.L. Bean, Inc

15. DataPropagator(DPROP) aka: Infosphere Replication Server
DataPropagator captures inserts/updates/deletes from the log to be able to propagate the data changes to other systems. DataPropagator keeps track of the last RBA propagated to know where to start in the log when DPROP is stopped and started.
This added time to our outage to insure that DPROP was caught up with the log when we shut down DB2 and making sure that DPROP worked appropriately when DB2 was restarted for normal access after the RBA Reset.
Jay
There were recent ptfs that were related to RBA Reset, Data Propagator and compressed tables that indicated to us that not many companies had handled an RBA Reset for a DB2 subsystem that used DPROP. IBM told us if we were in a data sharing environment, there would be no special handling of DPROP.
The DPROP DBA handled all the DPROP logistics and understanding of what needed to be done to make sure that DPROP was able to restart without issues. We had to treat it like we were doing a cold start of DPROP.
L.L. Bean, Inc

16. Testing Tested on 2 non prod subsystems
Only one non prod subsystem had DataPropagator; needed to test with DPROP twice
Did a dry run just image copying all page sets in production to help determine how many jobs could run concurrently
Jay
L.L. Bean, Inc

17. Implementation Sat 11pm Brought down prod systems
Once everything was down, broke the SRDF links
11:45pm DBA made sure DPROP was completely caught up
midnight Started image copies
It took 1.5 hours to do the image copies
Sun
1:30am Restarted DB2 for normal access,
Validation, cold start of DPROP and DPROP steps
4:30am Developers ran batch jobs to insert/delete/update
4:45am Go decision was made, Reconnect SRDF,
Bring up prod servers & apps
6:30am Call centers online
8am Batch started
Laurie
(17 concurrent jobs)
If any errors later because of a problem with a pageset,
the plan was to reorg a tablespace or rebuild the index
Planning and practice helped us to successfully complete the Log RBA reset on Sept 11, 2011.
L.L. Bean, Inc

18. Questions?
Thank you.
L.L. Bean, Inc