Using Hard Disks in Real-Time Systems Mark Stanovich

Context Real-time systems Raw disk I/O Hard disks with built-in scheduler/queue Mixed Workload Disk requests with deadline/response time requirements Background/best effort requests Want to guarantee real-time deadlines

Difficulties Real-time schedulability analysis generally relies on knowing worst-case execution times (WCET) Non-preemption makes guarantees even more difficult Variability of disk service times is extreme (tens of milliseconds to several seconds) Result is that hard disks are not prevelant in the critical path of a real-time system Meeting guarantees vs optimizing disk utilization

Value of Research Provides more information to provide schedulability analysis Large capacity of hard disks can be utilized in the critical path of a real-time system Less burden on other resources Lower latencies

Applications Multimedia Data Logging Webservers Data Analysis

Disk Scheduling Allow internal scheduler to schedule requests less burden on the CPU device driver can be at a lower priority fine-grained internal state does not have to be maintained disk specific characteristics can be utilized All scheduling performed inside the OS more control over request service order scheduling policy can be changed

Disk Scheduling Rotational-Position-Aware Real-Time Disk Scheduling Using a Dynamic Active Subset (DAS) A Real-Time Disk Scheduler for Multimedia Integrated Server Considering the Disk Internal Scheduler

Dynamic Active Subset upon each scheduling decision, the calculation of a subset of the outstanding disk requests such that all service guarantees can be enforced under worst-case assumptions schedule the subset based on the rotational position of requests in order to improve scheduling decision

Response Times

Worst-Case Execution Time seek time rotional delay * number of rotations to settle access time for some number of sectors time per sector varies depending on the zone overhead time disk controller processing data transfer between disk and host system skew time * v time to switch next cylinder and next disk head v depends on maximum request size and minimum size of a single track

Number of Rotations to Settle Rare cases the disk head needs some additional rotations to settle on the destination track Provoke worst-case by alternately issuing requests to the innermost and outermost region of the disk

Worst-Case Execution Times

Hiding Overhead Times substantial amount of time communicating with the disk without media access use TCQ to minimize these times send 2 requests so that as one request is transferred to or from the disk the other will be executing

Real-Time Disk Scheduling Execute all real-time requests at the beginning of each period limits the scope of scheduling optimizations to request classes DAS construct a subset of the outstanding requests such that no service guarantee will be violated regardless of which request is executed all scheduling algorithms can be used while ensuring deadlines dynamic nature of DAS does not allow scheduling inside the disk controller's hardware

DAS

Performance

Autonomous disk head location = data locations to retrieve

Autonomous Worst-case Rotational Latencies 5400 RPM = 11.1 msec 7200 RPM = 8.3 msec 10K RPM = 6 msec disk head location

Autonomous cons without detailed knowledge of hard disk internals, service times are difficult to predict for real-time requests

Lack of Preemption Capablility Real-time requests must wait for current request to finish if current request takes too long, even if we start the real-time request immediately, it may fail to meet its deadline NCQ does not allow requests to be pushed to the head of the queue now we may have to wait for all requests on the disk to be processed first

Response Time Simplistic Bound rotation latency + full stroke seek time Example: Maxtor 73G 10K RPM drive worst case latency: 6 msec + 11 msec = 17 msec

Response Time background task real-time task

Response Time

RT I/O Scheduler simple no merging no sorting accomodates I/O priorities VFS block I/O device driver elevator application user space kernel space

Response Times

Problems disk unaware of request priorities starvation of requests new background requests sent to the disk are serviced before older requests better performance to keep disk head in a certain region, less disk head movement

Response Times

Solutions use round based scheduling with ordered tag to prevent background requests from being serviced before real-time requests [Kim 2003] rely on disk starvation prevention algorithm draining of disk queue limiting on disk queue depth

Draining

allow disk to service request already on the disk without sending any new requests drain_time(n) maximum time to service n disk requests with no subsequent requests being sent to the disk condition to send new request: if (current time + drain_time(x)) <= earliest deadline, where x is the size of the on-disk queue depth

Draining determing drain_time(n) on-disk scheduling logic unknown, therefore makes analytical analysis difficult can empirically determine send n number of requests to the disk and measure the time to completion how to know when the worst case response time has been reached

Draining

Limiting On-Disk Queue Depth max_depth maximum number of outstanding requests permitted to be sent to the disk condition to send new request: if number of outstanding requests < max_depth

Implementation VFS block I/O device driver elevator application user space kernel space

Experimental Verification periodic real-time task requests data from disk period = deadline = 250 msec 256KB request size 450 constant background asynchronous requests sent to same disk

Experimental Verification

Conclusion more intelligent, autonomous hard drives increase the complexity of scheduling requests command queuing provides some assistance, but does not address all real-time disk I/O issues draining and limiting the on-disk queue can be used to maintain deadline constraints several aspects of disk behavior is still unexplained and until these are resolved, no absolute guarantees can be made

Previous Work Pro's takes advantage of drives internal mechanisms can guarantee most requests in a round uses Linux, a commodity OS uses an admission controller for the real-time requests Con's constrained to the time interval of a round all requests of one round are treated as equal priority does not mention about priority of disk device driver

constrain the internal features of a disk in order to provide some idea of reserved bandwidth start with periodic requests (benefit from the knowledge of upcoming requests) as time gets close to the real-time request reduce admission of best-effort requests to the disk (gradually lower on-disk requests) may also want to constrain the region in which the disk can do work if the region of the real-time request is known reserve bandwidth for some time interval Work in progress

disk drive has a worst-case bandwidth admission control can allocate up to this parameter after that cannot guarantee anything else for hard real-time constraints best-effort requests can fit before real-time requests as long as the requests do not jeopardize the upcoming real-time request (worst-case service time) real-time requests usually will not fill the entire time allocated for the worst case bandwidth therefore time will be available for best-effort requests use ordered tag to send best-effort requests after ALL known real-time requests are issued to the disk

Work in progress Differences between read/writes writes normally require a longer settle time time for head position to stabilize on the selected track

Work in progress Metrics extent to which real-time request exceeded deadline (for hard real-time this should be 0) average response time of real-time request in comparison with calculated worst-case time bandwidth of best-effort requests average/min/max of on-disk queue depth used NCQ allows for only 32 on-disk requests are more really needed such as the SAS drives with 256 requests (SCSI TCQ has a maximum queue length of 2^64) stress an actual implementation displaying video with best-effort applcations in the background (compile kernel, copy large files, etc.)

Constraining the Disk Reduces Efficiency Not necessary to send ordered tag right away sending the ordered tag may put unnecessary constraints on the internal scheduler may be better to stop future requests until the real- time request is completed or to some minimum internal disk queue length