1. An Overview of Insertion Sortby
Shanen Cross

2. What is Insertion Sort?
Insertion Sort is a simple sorting algorithm, ideal for ordering short arrays.
Insertion Sort has the advantage of being simple and intuitive, but is less efficient than other algorithims, especially when organizing larger arrays.
Because Insertion Sort is an in-place algorithim, its memory cost is very low.

3. How Insertion Sort Works
Here is an arbitrary, unsorted array of numbers, which we will sort using Insertion Sort:
42, 33, 9, 13, 1, 100
In many languages, including C/C++, array/list elements are ordered beginning with element 0, so we will assume the same here. Thus, 42 is the 0th element in the array, 33 is the 1st element (despite being second from the left!), 9 is the 2nd, etc.

4. How Insertion Sort Works
With 0-based numbering in mind...
Our list: 42, 33, 9, 13, 1, 100
Imagine previous array is laid out on a table as a series of playing cards (numbered 0-5). Insertion Sort begins with Card #1, and compares its value to Card #0. If Card #1's value is less than Card #0's, it swaps them.
Since 33 < 42, Card #1 and Card #0 are swapped. Now 33 is designated Card #0, and 42 is designated Card #1.
New Array: 33, 42, 9, 13, 1, 100

5. How Insertion Sort Works
Current Array: 33, 42, 9, 13, 1, 100
Now imagine that you have placed Cards #0-1 in your left hand, leaving the remaining cards on the table. The cards in your left hand represent the ordered cards, as they have already been sorted into an order.
Next, Insertion Sort looks at Card #2. It compares Card #2 to each of the ordered cards in your left hand, from right to left (#1, then #0), and inserts it into the appropriate location. Thus: New Array: 9, 33, 44, 13, 1, 100

6. How Insertion Sort Works
Current Array: 9, 33, 44, 13, 1, 100
Now, Cards #0-2 are the ordered cards held in your left hand, and Insertion Sort looks at the next unordered card, Card #3 (which is 13). It compares Card #3 to each of the ordered cards from right to left (#2, then #1, then #0) until it finds the right place to insert it.
New Array: 9, 13, 33, 44

7. How Insertion Sort Works
Current Array: 9, 13, 33, 44, 1, 100
This process continues, appropriately inserting each unordered cards still on the table into the ordered cards held in your left hand until all the cards are ordered. Here are the next few iterations of this process:
Inserting Card #4 (1) into ordered list:
New Array: 1, 9, 13, 33, 44, 100
Inserting Card # 5 (100) into ordered list:
New Array: 1, 9, 13, 33, 44, 100 (unchanged).
Now all the cards are ordered in your left hand, and none remain on the table.

8. How Insertion Sort Works
Current Array: 9, 13, 33, 44, 1, 100
This process continues, appropriately inserting each unordered cards still on the table into the ordered cards held in your left hand until all the cards are ordered. Here are the next few iterations of this process:
Inserting Card #4 (1) into ordered list:
New Array: 1, 9, 13, 33, 44, 100
Inserting Card # 5 (100) into ordered list:
New Array: 1, 9, 13, 33, 44, 100 (unchanged).
Now all the cards are ordered in your left hand, and none remain on the table.

9. How Insertion Works One more example array: 6, 20, 19, 100, 44, 5
Inserting Card #1 (20) into "ordered cards":
6, 20, 19, 100, 44, 6 (unchanged)
Inserting Card #2 (19):
6, 19, 20, 100, 4, 5
Inserting Card #3 (100):
6, 19, 20, 100, 4, 6 (unchanged)
Inserting Card #4 (4):
4, 6, 19, 20, 100, 5
Inserting Card #5 (5):
4, 5, 6, 19, 20, 100

10. Insertion Sort: Conceptual Breakdown of Steps
Place Card #0 in "left hand" (ordered cards). Sort Card #1 into "left hand" by comparing with Card #0.
Sort Card #2 into "left hand" by comparing with each of the cards in the "left hand" from right to left (first Card #1, then Card #0).
Repeat by sorting each of the remaining cards on the table into the "left hand" group of ordered cards.
How do we sort a "table" card into a "left hand" card?
If Card #2's value is less than the current left-hand card's value, shift that left-hand card's position in the array one over to the right, and move onto the next left-hand card comparison (from right to left). Otherwise, insert the table card in front of the current left-hand card (or at the beginning if there are no more left), and stop moving through the left-hand cards.

11. Example Implementation (C++)
void int_sort(int list[], int list_size) {
int i;
for (i=1; i<list_size; i++) // iterate through "table cards",
{ // beginning with Card #1
int current = list[i]; // extract current table card's value for
// comparison
int j = i-1; // First "left-hand card" precedes current table card
while (j>=0 && current<list[j]) // Iterate through left-hand cards
{ // If current table card should come
list[j+1] = list[j]; // earlier than current left-hand card,
j = j-1; // shift the latter one to the right }
list[j+1] = current; // Otherwise, place "table card" in front of
} // current left-hand card and stop iterating
} // through left-hand cards

12. Note on Implementation
Note that this is but one possible implementation of Insertion Sort. In our method, in order to insert each "table card" into the ordered "left hand cards", we successively pushed each card to the right until we found the place to put our new card.
An alternative implementation would be to swap pairs of cards as we go through the "left hand" cards, rather than "pushing" each left hand card to the right.

13. Big O Efficiency
Assuming the array has at least 2 elements (elements #0 and #1):
Best cased scenario: The array is already in order.
Example array: 4, 8, 9, 13
What will Insertion Sort do?
First: Place Card #0 (4) in "left hand", take Card #1 as first "table card". Compare #1 and #0: 8 > 4, so place table card one ahead of left-hand card (where it actually already is). Array is unchanged.
Next: Compare Card #2 (9) to Card #1 (8); Card #2 > Card #1, so place Card #2 in front of Card #1 (where it already is) and stop iterating through the sorted left-hand cards.

14. Big O Efficiency Current Array: 4, 8, 9, 13
Next, compare Card #3 to Card #2. Since #3's value > #2's value, place #3 in front of #2 (where it already is) and stop iterating through the "left hand" cards.
Finally, compare Card #4 to Card #3. Since #4's value > #3's, #4 goes in front of #3, and iterating through the "left hand" cards stops.
As you can see, we loop through the table cards (let's say there are a number n of them), and only make one comparison for each. So we iterate through only n loops.
So in the best case scenario, the efficiency is O(n). Nearly sorted arrays that require only 1 comparison per "table card" will also have O(n) efficiency.

15. Big O Efficiency
Worst case scenario: The array is in completely reversed order.
Example array: 10, 7, 5, 2
We compare #1 and #0. 7 < 10, so we push the 10 to the right (into the #1 slot), and place 7 in the #0 slot.
Array: 7, 10, 5, 2
We compare #2 and #1. 5 < 10, so we push 10 to the right and compare. Then we compare 5 and #0. 5 < 7, so we push 7 to the right and insert 5 in slot #0.
Array: 5, 7, 10, 2
We compare #3 and #2. 2 < 10, so we push 10 to the rightand compare 2 and #1. 2 < 7, so we push 7 to the right and compare 2 and #0. 2 < 5, so we push 5 and insert 2 into slot #0.

16. Big O Efficiency Final array: 2, 5, 7, 10.
Not only do we have to iterate through n table cards, for each table card we must also iterate through every single preceding card (the ordered cards held in your "left hand"). Since this is the worst case, no preceding cards are left out of the comparison.
As such, in the worst case scenario, the complexity becomes O(n*n), or O(n^2).
Here, the average Big O efficiency is essentially the same as the worst case scenario efficiency. So the average Big O Efficiency for Insertion Sort is O(n^2).

17. Memory Usage Insertion Sort is an In-Place Algorithim.
But what does that mean?
This means that Insertion Sort uses a constant amount of additional storage space to perform its task. This additional storage space is very small, so this is an advantage for Insertion Sort.

18. Other properties
Insertion Sort is Adaptive, meaning "its performance adapts to the initial order of the elements" (Source #3).
It is Stable, meaning it "retains relative order of the same elements" (Source #3).
It is Online, meaning "new elements can be added during the sort)" Source #3).
It does not depend on any other data structures aside from the array that it is operating on, as far as my research has shown.

19. Conclusion
Insertion Sort is an in-place sorting algorithm ideal of O(n^2) efficiency ideal for operating on small arrays/lists (say 8-12 elements according to Source #3) but is not practically useful for larger lists due to its inefficiency. It is however, simple and intuitive to understand.

20. Sources
#1: Rashid Bin Muhammad's Home Page, from Kent State University:
#2: eHow: The Advantages and Disadvantages of Sorting Algorithims
#3: Algorithms and Data Structures: Insertion Sort: