1. Some less known facts about log file sync and other LGWR-related waits
Nikolay Savvinov,
snr. database performance specialist,
Deutsche Bank TechCentre

2. A few words about me 10 years with Oracle databases
was doing particle physics before
last 5 years focus on performance optimization
Twitter: oradiag
Blog: savvinov.com
10 June 2015, HARMONY – 2015, TALLINN
N. Savvinov, Some less known facts about LGWR

3. Outline Well known stuff about LGWR Not-so-well-known stuff about LGWR
log parallelism and how it can backfire
contention-related log file sync waits
excessive commits aren’t always excessive
10 June 2015, HARMONY – 2015, TALLINN
N. Savvinov, Some less known facts about LGWR

4. What is log file sync (LFS)?
Changes generate redo (redo makes changes recoverable)
Redo needs to be written to persistent storage
For performance reasons, this is done asynchronously
When user commits changes, he wants to be sure they’re protected
LFS is an delay between the commit and the confirmation that redo is on the disk
Normally, the main part of LFS is log file parallel write (LFPW)
Other components: latch manipulation, inter-process communication, CPU etc.
10 June 2015, HARMONY – 2015, TALLINN
N. Savvinov, Some less known facts about LGWR

5. Why care about LFS? LFS measures the delay introduced by a commit
I.e. systems that commit a lot, can spend a lot of time on LFS
LFS is critical for low-latency OLTP systems
LFS is one of major sources of replication delays
LFS is responsible for “commit gaps” => errors in logic
10 June 2015, HARMONY – 2015, TALLINN
N. Savvinov, Some less known facts about LGWR

6. How redo generation works
redo is simple (flows in, flow out)
log buffer is small
log files are written to in a circular manner
redo incoming rate ≈ rate of change
can be affected by hot backup
flush triggered by:
commit
rollback
log buffer 1/3 full
log buffer 1MB full
every 3 seconds
no balance between redo in- and out-flow => delays
Redo generation rate
Log buffer
Redo flush
Redo write speed
To log files
10 June 2015, HARMONY – 2015, TALLINN
N. Savvinov, Some less known facts about LGWR

7. Piggyback commit: mechanics
10 June 2015, HARMONY – 2015, TALLINN
N. Savvinov, Some less known facts about LGWR

8. Averages lie BIG TIME
(1 ms x ms x 1)/(10 + 1) ≈1.8 ms LFPW, (1 ms x ms x 10) / ( ) = 3 ms LFS
the higher (and the more frequent) the outliers are, the bigger gap between LFS and LFPW
10 June 2015, HARMONY – 2015, TALLINN
N. Savvinov, Some less known facts about LGWR

9. I/O related LFS most common scenario
mark of I/O related LFS: LGWR waiting almost exclusively on LFPW
I/O performance cannot be judged by time alone, need redo size as well
must take into account RAID write penalties
synchronous storage-level replication – another common scenario
a (rather common) special case: storage-level contention
10 June 2015, HARMONY – 2015, TALLINN
N. Savvinov, Some less known facts about LGWR

10. “log file parallel write” => “log file serialized write”
on the database level, LFPW is a simple single event
on the OS level: a bunch of write requests to several destinations (multiplexity!)
LFPW parameters:
select p1, p1text, p2, p2text, p3, p3text from v$active_session_history where event = ‘log file parallel write’ =============================================== files 2050 blocks 2 requests
when requests > multiplexing, we see log parallelism in action
introduced to reduce latching, has nasty side effects when many CPUs
_log_parallelism_max, _log_parallelism_dynamic (note )
10 June 2015, HARMONY – 2015, TALLINN
N. Savvinov, Some less known facts about LGWR

11. CPU-related LFS waits redo logging is NOT CPU intensive
still, it does need CPU
CPU starvation, priority inversion etc. can lead to CPU-related LFS
identified by % of time spend “ON CPU” by LGWR (or big outliers)
sometimes can be fixed by changing LGWR priority in OS, or by using database parameter _high_priority_processes
on the plot: spikes correspond to CPU scheduler quanta
10 June 2015, HARMONY – 2015, TALLINN
N. Savvinov, Some less known facts about LGWR

12. Contention-related LFS
Apart from I/O and CPU, another scenario for LFS is contention
e.g. writes to control file required when switching log files
Signature: LGWR spending significant % of time on “enq: CF – contention” (or big outliers)
Small log files increase the risk of this problem!
Rather than relying on reducing log switch frequency alone, the best approach is to identify the root cause (e.g. RMAN issues) causing excessive (or slow) writes to control file that lead to the contention
10 June 2015, HARMONY – 2015, TALLINN
N. Savvinov, Some less known facts about LGWR

13. Excessive commits?
“You’re committing too much” is one of the most popular responses when complaining about LFS
Record: the smallest commit frequency that was declared “excessive” by investigating DBA was … 30 commits per second
In reality, an Oracle database can handle thousands, and even tens of thousands of commits per second
Excessive commits are primarily a problem for transactional integrity
Excessive commits slow down the process that issues them
Removing them can LFS => LBS or cause a bottleneck elsewhere
10 June 2015, HARMONY – 2015, TALLINN
N. Savvinov, Some less known facts about LGWR

14. Bad reasons to worry about LFS
because without LFS everything would go faster (no it won’t)
because it’s in top events in AWR (so what?)
because LFS is contributing to locking/contention (no it doesn’t)
because LFS is increasing CPU consumption (no it doesn’t)
10 June 2015, HARMONY – 2015, TALLINN
N. Savvinov, Some less known facts about LGWR

15. Summary LFS is a common problem, more common for OLTP
not always a performance issues as such, could also be a “staleness” issue, or cause errors in logic
% DB time in AWR is rarely a useful measure for LFS impact
caused by redo I/O, CPU/scheduling issues, or contention (e.g. for CF)
I/O problems can be related to “serialized parallelism” issue, workaround: disable log parallelism
reducing commits can transform LFS into LBS
when troubleshooting LFS, understanding the scope is of key importance (or performance can be made worse instead of better)
10 June 2015, HARMONY – 2015, TALLINN
N. Savvinov, Some less known facts about LGWR