1. Get Into the Goldilocks Zone
Managing SQL Transaction Log VLF Growth
Michelle Poolet, MCIS Zachman-Certified Enterprise Architect™ 
4/2/2016

2. What We’ll Do Today
Talk about VLFs, their value, impact and relationship to the SQL transaction log.
What causes the VLF count to grow?
How to rein in VLF and subsequent SQL Server transaction log growth.
How to get into the Goldilocks Zone, that is, get a handle on stabilizing VLFs and the SQL Server transaction log size.
4/2/2016

3. What’s a VLF & Why Do I Care?
Some background
All changes to a database (data & structure) are preserved
First write to the transaction log
Then write to physical database
One transaction log per database
SQL Server writes changes to available space in the log file
SQL Server must have enough tlog space so that active (uncommitted) transactions can be rolled back
one transaction log per database, can be comprised of multiple files
one log, sequential write only
space can be reused after
1) transaction is committed, and
2) database synched (CHECKPOINT)
3 |
4/2/2016

4. What’s a VLF & Why Do I Care?
VLF structure
Tlog is comprised of blocks of space -- VLF
Space management inside the transaction log file
No single fixed size for a VLF
No fixed number of VLFs per physical log file
When using Full Recovery mode:
transactions are copied to the tlog until they fill up log file;
SQL continues to write the new transactions from the beginning of the tlog file ONLY IF
a tlog backup has been taken (CHECKPOINT),
old transactions are marked for deletion (CHECKPOINT), and
ALL transactions within a VLF have been committed AND checkpointed and the VLF is marked for reuse.
If a transaction log backup hasn’t been taken:
the tlog file is forced to expand enough to accommodate newly executed transactions;
no instant initialization for tlog file, operation will negatively impact performance.
4 |
4/2/2016

5. Why Would a VLF Not Be Reusable?
Long-running transactions that don’t commit…
No commit = VLFs remain “active”, cannot reuse
Lack of transaction log backup…
a backup issues a CHECKPOINT
CHECKPOINT confirms that committed transactions on the tlog have been written to the database
committed/checkpointed transactions on the tlog are marked for overwrite
It all boils down to...
A VLF is reusable ONLY when ALL the transactions stored in that VLF have been committed and CHECKPOINTED
number of VLFs and size of the transaction log file are indirectly related
FULL or BULK-LOGGED recovery mode, SQL Server MUST capture all changes to the database
SIMPLE recovery mode transaction log freezes tlog file size and VLF count
What happens when a VLF is “cleared”?
it’s marked as inactive
nothing is truncated
nothing is overwritten
the log file size does not change
only whole VLFs are marked for reuse
5 |
4/2/2016

6. What Causes the VLFs (& TLog) to Grow?
If no VLFs available for re-use
SQL Server must auto-grow
Allows SQL Server to allocate additional space to the tlog
must pause processing (no instant initialization)
Increases the number of VLFs based on auto-grow size
SQL Server must stop processing
If auto-grow is not enabled and
“out of space in the log”
6 |
4/2/2016

7. VLF Trivia 2012 TRIVIA FACT for <= SQL Server 2012 TRIVIA QUESTION
Tlog auto-grows between 1-64 MB, +4 VLFs
Tlog auto-grows > MB, +8 VLFs
Tlog auto-grows >= 1GB, +16 VLFs
TRIVIA QUESTION
SQL Server created with tlog file = 2 MB & auto-growth by 10%
By the time the tlog reaches 500 MB, how many VLFs will it have?
7 |
4/2/2016

8. VLF Trivia 2012 TRIVIA ANSWER 416 VLFs…
264 VLFs < 1 MB in size (0.25 to 0.93 MB)
56 VLFs < 2 MB in size (1.03 to 1.82 MB)
40 VLFs < 3 MB in size (2.0 to 2.93 MB)
24 VLFs < 4 MB in size (3.23 to 3.9 MB)
16 VLFs < 5 MB in size (4.3 to 4.72 MB)
16 VLFs < 6 MB in size (5.2 to 5.72 MB)
8 |
4/2/2016

9. VLF Trivia 2014 More Trivia for >= SQL Server 2014
If the impending growth unit is < 1/8 size of the current log:
A single VLF of size = auto-grow unit will be created
Else
Use old algorithm
IS THIS A TRADEOFF, an attempt to optimize the VLF count for situations where the transaction log auto-growth is left at 10%
Are default values taking on the mantle of “best practice”??????
9 |
4/2/2016

10. VLF Algorithm Comparison
Comparing old & new auto-grow algorithms
The old formula: auto-grow using a 10% increment causes a fast build-up of VLF count, even with very small transaction logs. The new formula controls VLF count much more efficiently, but the VLF segment size increases more quickly.
10 |
4/2/2016

11. VLF Algorithm Comparison
Comparing old & new auto-grow algorithms
As the transaction log size approaches 1GB, VLF counts skyrocket under the old formula, but the VLF segment size stays very stable. Under the new formula, VLF count is still low, but the VLF segment sizes are getting quite large – this may cause problems with VLF segment reuse.
11 |
4/2/2016

12. VLF Algorithm Comparison
Comparing old & new auto-grow algorithms
The transaction log is in the 10-20GB range. The old formula has very high VLF count but small VLF segment size, good for VLF reuse. The new formula has a small VLF count but very large VLF segment sizes. VLF reuse is at risk, the transaction log could uncontrollably balloon to an "out of space on disk" condition.
12 |
4/2/2016

13. Why Is This Such A Big Deal?
Too many VLFs adversely impact database operations…
Database startup takes longer;
Database restores take longer;
Inserts, updates and deletes can take longer;
Backups are larger because active VLFs (uncommitted, not cleared for reuse) must be included;
Tlog file fragmentation - SQL Server has to search for the appropriate VLF whenever writing to the log file or clearing a VLF for reuse
13 |
4/2/2016

14. Can VLFs Get Too Large?
Overly large VLFs (4 GB?) = performance problems…
File I/O
Buffer cache crunch
Clearing large VLFs for reuse takes time…
SQL Server search for correct VLF to commit
Tlog backups take longer
Inability to clear VLFs rapidly = more VLFs created
New VLFs will be large = more disk space required
14 |
4/2/2016

15. Can There Be Too Many VLFs?
There is no hard and fast rule as to how many VLFs any database should have…
Too many VLFs = tlog fragmentation, negative impact on performance, slower database startup & restore.
Too few and very large VLFs = negative impact on performance, larger backups, ballooning transaction logs.
15 |
4/2/2016

16. Help! What Do I Do With A Ballooning TLog?
Check for open transactions
Take a tlog backup
Shrink the tlog file, reduce the VLF count (no maintenance window required)
16 |
4/2/2016

17. Help! What Do I Do With A Ballooning TLog?
CHECK FOR OPEN TRANSACTIONS
Dbcc opentran;
What is a tlog file waiting on?
SELECT name,
log_reuse_wait,
log_reuse_wait_desc
FROM sys.databases;
is there a way to make open transactions commit more often?
ANSWERS TO THE TLOG WAIT STATE…
Nothing…there’s no impediment.
Log backup needs to be run.
Active backup running (a full backup needs the transaction log to be able to restore to a specific point in time)
Active transaction (somebody typed BEGIN TRAN and walked away)
Database mirroring, replication, or AlwaysOn Availability Groups (is there a failed replica? Until all nodes can sync, the transaction log will continue to grow)
17 |
4/2/2016

18. Help! What Do I Do With A Ballooning TLog?
TAKE A TLOG BACKUP
You think that tlog backups have been running regularly?
Try running a SQL tlog backup…
VM or Hyper-V snapshots will not shrink the tlog
Rethink your backup strategy
Tlog backups will clear VLFs for reuse but probably will not shrink tlog
What happens when a checkpoint is issued?
checkpoint identifier (LSN) is written to transaction log file
“dirty” data pages are written to:
transaction log file
physical database file
then the pages in memory can be reused
What happens when a VLF is “cleared”?
it’s marked as inactive
nothing is truncated
nothing is overwritten
the log file size does not change
only whole VLFs are marked for reuse
18 |
4/2/2016

19. Help! What Do I Do With A Ballooning TLog?
SHRINK THE TLOG FILE (no maintenance window required)
TEMPORARY SOLUTION ONLY, reduces VLF count and releases log space to the operating system
Root cause has not been addressed
Tlog files will re-grow over time, will have to repeat process
No instant file initialization as SQL tlog grows, performance hit
USE MyDatabase GO
dbcc loginfo -- check VLF count before shrink
DBCC SHRINKFILE (N‘MyDatabase_log' , 100)
dbcc loginfo -- check VLF count after shrink
ROOT CAUSE = auto-grow value
19 |
4/2/2016

20. What’s the Ultimate Goal?
GET INTO THE GOLDILOCKS ZONE
Study & monitor tlog growth and behaviour
Schedule a Maintenance Window
Resize and pre-grow the tlog file
Monitor tlog growth and behaviour after-the-fact
Repeat if necessary
If you understand well how each database uses its tlog file, you may not have to do this step. Ever. Until things change.
20 |
4/2/2016

21. Goldilocks Zone: Pre-Grow A Tlog File
Schedule a Maintenance Window
determine a new size for the log file
shrink the transaction log
re-grow the transaction log
change tlog auto-grow settings
21 |
4/2/2016

22. Goldilocks Zone: Pre-Grow A Tlog File
1. How large should the tlog be?
How much data is being added to the database in one year’s time?
How often are index rebuilds being done?
Are there excessively-long transactions that are running against the database? Can they be shortened?
Is data being bulk-loaded into the database on a regular basis? How much and when?
What’s the disk usage history? (applies only to mature databases…)
Does this database take part in mirroring, replication, or AlwaysOn Availability Groups?
22 |
4/2/2016

23. Goldilocks Zone: Pre-Grow A Tlog File
2. Shrink the tlog to smallest possible size
Use dbcc shrinkfile to shrink the log file
Shrink slowly, in small increments
Allow SQL Server to transfer “active” transactions to the newer portion of the tlog
how much of the tlog file is in active use? dbcc sqlperf(logspace)
example
10GB database
3GB tlog file with 10% auto-grow
0.2% of the log is active
determine that optimal tlog size = 500MB, with auto-grow at 100MB (8 VLFs, 12.5MB each)
SHRINK In 100 MB increments
23 |
4/2/2016

24. Goldilocks Zone: Pre-Grow A Tlog File
3. Re-grow the tlog file
Use alter database modify file command
Incrementally grow the log file to an appropriate size
Grow it slowly
Grow too fast results in too-large VLFs
IDEAL CASE: size the tlog so it will not have to auto-grow in future operations
Determine that optimal tlog size = 500MB, with auto-grow at 100MB (8 VLFs, 12.5MB each)
REGROW in 100 MB increments
24 |
4/2/2016

25. Goldilocks Zone: Pre-Grow A Tlog File
4. Change auto-grow settings
Adjust auto-grow to new growth increment
If dynamically-created databases (Sharepoint)
Adjust model database to minimize tlog growth
ALTER DATABASE PROD
MODIFY FILE (NAME=PRODLOG, FILEGROWTH = 100 MB) GO
25 |
4/2/2016

26. DEMO
4/2/2016

27. Goldilocks Database
3/25/2017

28. Goldilocks – what do I know?
3rd-party package used to make reservations.
Normal processing behaviour is bursty, based on upcoming events.
Transactions are short (ticket purchase, integrate with PayPal, send to purchaser).
Business & activity reports are run against the same database.
Tlog size is <2% of the database file (seems very small, out of proportion, especially given the large auto-grow size).
This is a copy sent for review.
DBA may have shrunk transaction log file prior to backup?
3/25/2017

29. Goldilocks Assumptions…
IF...this transaction log is indeed a good size for this database...
the VLF count is high, higher than optimal for the type of transactions (short, succinct),
auto-grow value is disproportionately large,
larger VLF will take longer to clear. During high activity periods, SQL may need to acquire additional log space...
...having to pause SQL while the tlog initializes during high-activity periods is not good.
3/25/2017

30. Goldilocks Recommendations
Let's restructure the tlog file, reduce the VLF count, to better align with the type of processing and the type of data.
3/25/2017

31. Goldilocks: Shrink the tlog file slowly…
REPEAT as necessary until shrinking stops
3/25/2017

32. Goldilocks: REGROW THE Tlog file
Use growth increments until target size is reached
grow to 500 MB, growth unit = 100 MB
3/25/2017

33. Goldilocks: Change the tlog auto-grow value
to match the value used to re-grow the file
and make a final check
3/25/2017

34. Real-world Case: Personify
ERP solution, SQL Server 2012
PPROD data file = 22.9 GB, FULL recovery, growth unit = 20 MB
log file = GB, VLFs growing over time, 30 MB growth unit, 4.55% of log space used
log file should be regrown to 4 GB, growth unit = 500 MB
data file = 23,486 MB, FULL recovery model, growth unit = 20MB
log file = 16,638 MB, VLFs growing over time, 30 MB growth unit, 4.55% of log space used
log file should be regrown to 4000 MB, growth unit = 500 MB
USE PPROD GO
-- SHRINK THE TRANSACTION log file without truncating, SHRINK in small increments, allowing THE VLFs to clear
-- REPEAT as necessary until TARGET SIZE OF 1000 is reached, or until shrinking stops …
DBCC SHRINKFILE (2, 10); SHRINK log file by units of 1GB each time
dbcc loginfo VLFs
-- REGROW THE TRANSACTION log file, using growth increments until target size is reached
USE master;
ALTER DATABASE PPROD
MODIFY FILE (NAME = T6LOG, SIZE = 4000MB); regrow log in 500MB units to a total size of 4000MB
dbcc loginfo (8 * 8 = 64 + original 3 = 67, opted to create smaller VLFs when increasing by 500MB; the VLFs will clear for reuse faster if smaller.)
-- CHANGE THE AUTO-GROW AMOUNT
USE master
MODIFY FILE (NAME=T6LOG, FILEGROWTH = 500 MB);
34 |
4/2/2016

35. Real-world Case: Sharepoint 2013
SQL Server 2012
msdb data file = 22.9 GB, FULL recovery, growth unit = 20 MB
log file = GB, VLFs growing over time, 30 MB growth unit, 4.55% of log space used
log file should be regrown to 4 GB, growth unit = 500 MB
msdb data file = 116 MB, SIMPLE recovery model, growth unit = 10%
log file = log file = 195 MB, VLFs stable over time, 10% growth unit, 3.3% of log space used
log file should be regrown to 200MB, growth unit = 50MB
-- SHRINK THE TRANSACTION log file without truncating, SHRINK in small increments in units of 50MB, allowing THE VLFs to clear, until the target size of 50MB is reached, or until the shrinking stops …
USE msdb GO
DBCC SHRINKFILE (MSDBLog, 1);
dbcc loginfo VLFs
-- REGROW THE TRANSACTION log file to 200 MB, using growth increments of 50MB until target size is reached
USE master;
ALTER DATABASE msdb
MODIFY FILE (NAME = MSDBLog, SIZE = 200MB);
dbcc loginfo (4 * 4 = 16 + original 4 = 20, opted to create smaller VLFs; the VLFs will clear for reuse faster if smaller.)
-- CHANGE THE AUTO-GROW AMOUNT
USE master
MODIFY FILE (NAME=MSDBLog, FILEGROWTH = 50 MB);
35 |
4/2/2016

36. Is There a Methodology? Sort of…
Track changes over time to determine transaction activity and log growth needs.
Set up a maintenance window
Reduce VLF count
Regrow tlog size while controlling number and size of VLFs
Reset database > file properties > auto-grow size to align with changes made
Continue to monitor; if necessary, fine-tune assumptions and repeat process.
36 |
4/2/2016

37. What We’ve Covered…
VLFs, what they are, what impact they have on your SQL Server
Why VLFs to grow?
How to shut down a ballooning transaction log
How to get into the Goldilocks Zone - get a handle on stabilizing VLFs and the SQL Server transaction log size.
37 |
4/2/2016

38. Michelle Poolet, MCIS Zachman-Certified Enterprise Architect™
Questions?
Michelle Poolet, MCIS Zachman-Certified Enterprise Architect™ 
3/25/2017

39. Good References
Ivan Stankovic
Kim Tripp,SQLSkills.com
Eric Darling
3/25/2017

40. Thank You Sponsors
Platinum
Gold
Silver
Bronze
Swag
3/25/2017