1. Hashing, B - Trees and Red – Black Trees using Parallel Algorithms & Sequential Algorithms
By Yazeed K. Almarshoud

2. Road map Introduction. Definitions. Sequential algorithms
Hashing.
B Trees
Red Black Trees.
Sequential algorithms
Parallel algorithms
February 25, 2019

3. Introduction
In this presentation I ‘m glad to present to you the importance of parallel computations and how it is a form of computation in which many calculations are carried out simultaneously, operating on the principle that large problems can often be divided into smaller ones which are then solved concurrently
February 25, 2019

4. Hashing

5. Definitions
Hashing: A hash function is any well – defined procedure or mathematical function which converts a large, possibly variable-sized amount of data into a small datum, usually a single integer that may serve as an index into an array.
i.e. keys made up for alphabetical characters could be replaced by their ASCII equivalents.
Two standards hashing techniques:
Division methods
Multiplication method.
February 25, 2019

6. Symbol – table problem
February 25, 2019

7. Hash functions
February 25, 2019

8. Choosing a hash function
The assumption of simple uniform hashing is hard to guarantee, but several common techniques tend to work well in practice as long as their deficiencies can be avoided.
Desirata:
A good hash function should distribute the
keys uniformly into the slots of the table.
Regularity in the key distribution should
not affect this uniformity.
February 25, 2019

9. Division method Assume all keys are integers, and define
h(k) = k mod m.
Deficiency: Don’t pick an m that has a small divisor d. A preponderance of keys that are congruent modulo d can adversely affect uniformity.
Extreme deficiency: If m = 2r, then the hash
doesn’t even depend on all the bits of k:
If k = and r = 6, then
h(k) =
February 25, 2019

10. Division method (continued)
h(k) = k mod m.
Pick m to be a prime not too close to a power
of 2 or 10 and not otherwise used prominently
in the computing environment.
Annoyance:
Sometimes, making the table size a prime is
inconvenient.
But, this method is popular, although the next
method we’ll see is usually superior.
February 25, 2019

11. Multiplication method
Assume that all keys are integers, m = 2r, and our
computer has w-bit words. Define
h(k) = (A·k mod 2w) rsh (w – r),
where rsh is the “bit-wise right-shift” operator
and A is an odd integer in the range 2w–1 < A < 2w.
Don’t pick A too close to 2w.
Multiplication modulo 2w is fast.
The rsh operator is fast.
February 25, 2019

12. Multiplication method example
February 25, 2019

13. Resolving collisions by chaining
February 25, 2019

14. Analysis of chaining
February 25, 2019

15. Search cost
February 25, 2019

16. Resolving collisions by open addressing
No storage is used outside of the hash table itself..
The hash function depends on both the key and probe number:
h : U X {0, 1, …, m–1}  {0, 1, …, m–1}.
E.g. h(k) = (k+i) mod m ; h(k) = (k+i2 ) mod m
Inserting a key k:
we check T[h(k,0)]. If empty we insert k, there. Otherwise,
we check T[h(k,1)]. If empty we insert k, there. Otherwise,…
otherwise etc for h(k,2), h(k,1), …, h(k,m–1).
Finding a key k:
we check if T[h(k,0)] is empty, and if =k. If not
we check if T[h(k,1)] is empty, and if =k. If not
Deleting a key k
Find it are replace with a dummy (why)
February 25, 2019

17. Example of open addressing
February 25, 2019

18. Example of open addressing
February 25, 2019

19. Example of open addressing
February 25, 2019

20. Example of open addressing
February 25, 2019

21. Probing strategies Linear probing:
Given an ordinary hash function h ‘(k), linear
probing uses the hash function
h(k,i) = (h’(k) + i) mod m.
This method, though simple, suffers from primary
clustering, where long runs of occupied slots build
up, increasing the average search time. Moreover,
the long runs of occupied slots tend to get longer.
February 25, 2019

22. Red-Black Trees 6 v 3 8 z 4

23. Roadmap Definition Height Insertion Deletion restructuring recoloring
adjustment
February 25, 2019

24. Red-Black Tree
A red-black tree can also be defined as a binary search tree that satisfies the following properties:
Root Property: the root is black
External Property: every leaf is black
Internal Property: the children of a red node are black
Depth Property: all the leaves have the same black depth 9 4 15 2 6 12 21 7
February 25, 2019

25. Height of a Red-Black Tree
Theorem: A red-black tree storing n items has height O(log n)
The search algorithm for a red-black search tree is the same as that for a binary search tree
By the above theorem, searching in a red-black tree takes O(log n) time
February 25, 2019

26. Insertion
To perform operation insertItem(k, o), we execute the insertion algorithm for binary search trees and color red the newly inserted node z unless it is the root
We preserve the root, external, and depth properties
If the parent v of z is black, we also preserve the internal property and we are done
Else (v is red ) we have a double red (i.e., a violation of the internal property), which requires a reorganization of the tree
Example where the insertion of 4 causes a double red: 6 6 v v 3 8 3 8 z z 4
February 25, 2019

27. Remedying a Double Red
Consider a double red with child z and parent v, and let w be the sibling of v
Case 1: w is black
The double red is an incorrect replacement of a 4-node
Restructuring: we change the 4-node replacement
Case 2: w is red
The double red corresponds to an overflow
Recoloring: we perform the equivalent of a split 4 4 w v w v 2 7 2 7 z z 6 6
February 25, 2019

28. Local invariants example:
It involves only the fields of an object and the fields of its tree-children
We specify local invariants using the repOkLocal method.
February 25, 2019

29. Restructuring
A restructuring remedies a child-parent double red when the parent red node has a black sibling
It is equivalent to restoring the correct replacement of a 4-node
The internal property is restored and the other properties are preserved z 4 6 w v v 2 7 4 7 z w 6 2
February 25, 2019

30. Restructuring (cont.)
There are four restructuring configurations depending on whether the double red nodes are left or right children 6 4 2 6 2 4 2 6 4 2 6 4 4 2 6
February 25, 2019

31. Recoloring
A recoloring remedies a child-parent double red when the parent red node has a red sibling
The parent v and its sibling w become black and the grandparent u becomes red, unless it is the root
The double red violation may propagate to the grandparent u 4 4 w v w v 2 7 2 7 z z 6 6
… 4 … 2
6 7
February 25, 2019

32. Analysis of Insertion Algorithm insertItem(k, o)
1. We search for key k to locate the insertion node z
2. We add the new item (k, o) at node z and color z red
3. while doubleRed(z)
if isBlack(sibling(parent(z)))
z  restructure(z)
return
else { sibling(parent(z) is red }
z  recolor(z)
Recall that a red-black tree has O(log n) height
Step 1 takes O(log n) time because we visit O(log n) nodes
Step 2 takes O(1) time
Step 3 takes O(log n) time because we perform
O(log n) recolorings, each taking O(1) time, and
at most one restructuring taking O(1) time
Thus, an insertion in a red-black tree takes O(log n) time
February 25, 2019

33. Deletion
To perform operation remove(k), we first execute the deletion algorithm for binary search trees
Let v be the internal node removed, w the external node removed, and r the sibling of w
If either v of r was red, we color r black and we are done
Else (v and r were both black) we color r double black, which is a violation of the internal property requiring a reorganization of the tree
Example where the deletion of 8 causes a double black: 6 6 v r 3 8 3 r w 4 4
February 25, 2019

34. Remedying a Double Black
The algorithm for remedying a double black node w with sibling y considers three cases
Case 1: y is black and has a red child
We perform a restructuring, equivalent to a transfer , and we are done
Case 2: y is black and its children are both black
We perform a recoloring, equivalent to a fusion, which may propagate up the double black violation
Case 3: y is red
We perform an adjustment, equivalent to choosing a different representation of a 3-node, after which either Case 1 or Case 2 applies
Deletion in a red-black tree takes O(log n) time
February 25, 2019

35. Red-Black Tree Reorganization
Insertion
remedy double red
Red-black tree action
(2,4) tree action
result
restructuring
change of 4-node representation
double red removed
recoloring
split
double red removed or propagated up
Deletion
remedy double black
Red-black tree action
(2,4) tree action
result
restructuring
transfer
double black removed
recoloring
fusion
double black removed or propagated up
adjustment
change of 3-node representation
restructuring or recoloring follows
February 25, 2019

36. Binary Trees 6 3 8

37. Binary Trees A tree in which no node can have more than two children
The depth of an “average” binary tree is considerably smaller than N, even though in the worst case, the depth can be as large as N – 1.
February 25, 2019

38. Example: Expression Trees
Leaves are operands (constants or variables)
The other nodes (internal nodes) contain operators
Will not be a binary tree if some operators are not binary
February 25, 2019

39. Binary Trees Possible operations on the Binary Tree ADT Implementation
parent
left_child, right_child
sibling
root, etc
Implementation
Because a binary tree has at most two children, we can keep direct pointers to them
February 25, 2019

40. Compare: Implementation of a general tree
February 25, 2019

41. Binary Search Trees
Stores keys in the nodes in a way so that searching, insertion and deletion can be done efficiently.
Binary search tree property
For every node X, all the keys in its left subtree are smaller than the key value in X, and all the keys in its right subtree are larger than the key value in X
February 25, 2019

42. Binary Search Trees A binary search tree Not a binary search tree
February 25, 2019

43. Binary search trees
Two binary search trees representing
the same set:
Average depth of a node is O(log N); maximum depth of a node is O(N)
February 25, 2019

44. Implementation
February 25, 2019

45. Searching BST If we are searching for 15, then we are done.
If we are searching for a key < 15, then we should search in the left subtree.
If we are searching for a key > 15, then we should search in the right subtree.
February 25, 2019

46. February 25, 2019

47. Searching (Find)
Find X: return a pointer to the node that has key X, or NULL if there is no such node
Time complexity
O(height of the tree)
February 25, 2019

48. Inorder traversal of BST
Print out all the keys in sorted order
Inorder: 2, 3, 4, 6, 7, 9, 13, 15, 17, 18, 20
February 25, 2019

49. FindMin/ FindMax
Return the node containing the smallest element in the tree
Start at the root and go left as long as there is a left child. The stopping point is the smallest element
Similarly for findMax
Time complexity = O(height of the tree)
February 25, 2019

50. Insert Time complexity = O(height of the tree)
Proceed down the tree as you would with a find
If X is found, do nothing (or update something)
Otherwise, insert X at the last spot on the path traversed
Time complexity = O(height of the tree)
February 25, 2019

51. Delete
When we delete a node, we need to consider how we take care of the children of the deleted node.
This has to be done such that the property of the search tree is maintained.
February 25, 2019

52. Delete Three cases: (1) the node is a leaf (2) the node has one child
Delete it immediately
(2) the node has one child
Adjust a pointer from the parent to bypass that node
February 25, 2019

53. Delete (3) the node has 2 children
replace the key of that node with the minimum element at the right subtree
delete the minimum element
Has either no child or only right child because if it has a left child, that left child would be smaller and would have been chosen. So invoke case 1 or 2.
Time complexity = O(height of the tree)
February 25, 2019

54. Sequential algorithms

55. Sequential algorithms
Is the search space a tree or a graph?
The space of a 0/1 integer program is a tree, while that of an 8-puzzle is a graph.
This has important implications for search since unfolding a graph into a tree can have significant overheads.
February 25, 2019

56. Sequential algorithms
Two examples of unfolding a graph into a tree.
February 25, 2019

57. Best-First Search (BFS) Algorithms
BFS algorithms use a heuristic to guide search.
The core data structure is a list, called Open list, that stores unexplored nodes sorted on their heuristic estimates.
The best node is selected from the list, expanded, and its off-spring are inserted at the right position.
If the heuristic is admissible, the BFS finds the optimal solution.
February 25, 2019

58. Best-First Search (BFS) Algorithms
BFS of graphs must be slightly modified to account for multiple paths to the same node.
A closed list stores all the nodes that have been previously seen.
If a newly expanded node exists in the open or closed lists with better heuristic value, the node is not inserted into the open list.
February 25, 2019

59. Best-First Search: Example
Applying best-first search to the 8-puzzle: (a) initial configuration; (b) final configuration; and (c) states resulting from the first four steps of best-first search. Each state is labeled with its -value (that is, the Manhattan distance from the state to the final state).
February 25, 2019

60. Search Overhead Factor
The amount of work done by serial and parallel formulations of search algorithms is often different.
Let W be serial work and WP be parallel work. Search overhead factor s is defined as WP/W.
Upper bound on speedup is p×(W/WP).
February 25, 2019

61. Parallel algorithms

62. Parallel algorithms
How is the search space partitioned across processors?
Different subtrees can be searched concurrently.
However, subtrees can be very different in size.
It is difficult to estimate the size of a subtree rooted at a node.
Dynamic load balancing is required.
February 25, 2019

63. Parallel algorithms
When a processor runs out of work, it gets more work from another processor.
This is done using work requests and responses in message passing machines and locking and extracting work in shared address space machines.
On reaching final state at a processor, all processors terminate.
Unexplored states can be conveniently stored as local stacks at processors.
The entire space is assigned to one processor to begin with.
February 25, 2019

64. Parallel Best-First Search
The core data structure is the Open list (typically implemented as a priority queue).
Each processor locks this queue, extracts the best node, unlocks it.
Successors of the node are generated, their heuristic functions estimated, and the nodes inserted into the open list as necessary after appropriate locking.
Termination signaled when we find a solution whose cost is better than the best heuristic value in the open list.
Since we expand more than one node at a time, we may expand nodes that would not be expanded by a sequential algorithm.
February 25, 2019

65. Parallel Best-First Search
A general schematic for parallel best-first search using a centralized strategy. The locking operation is used here to serialize queue access by various processors.
February 25, 2019

66. Parallel Best-First Search
The open list is a point of contention.
Let texp be the average time to expand a single node, and taccess be the average time to access the open list for a single-node expansion.
If there are n nodes to be expanded by both the sequential and parallel formulations (assuming that they do an equal amount of work), then the sequential run time is given by n (taccess + texp).
The parallel run time will be at least n taccess.
Upper bound on the speedup is (taccess+ texp)/taccess
February 25, 2019

67. Parallel Best-First Search
Avoid contention by having multiple open lists.
Initially, the search space is statically divided across these open lists.
Processors concurrently operate on these open lists.
Since the heuristic values of nodes in these lists may diverge significantly, we must periodically balance the quality of nodes in each list.
A number of balancing strategies based on ring, blackboard, or random communications are possible.
February 25, 2019

68. Parallel Best-First Search
A message-passing implementation of parallel best-first search using the ring communication strategy.
February 25, 2019

69. Parallel Best-First Search
An implementation of parallel best-first search using the blackboard communication strategy.
February 25, 2019

70. Parallel Best-First Graph Search
Graph search involves a closed list, where the major operation is a lookup (on a key corresponding to the state).
The classic data structure is a hash.
Hashing can be parallelized by using two functions - the first one hashes each node to a processor, and the second one hashes within the processor.
This strategy can be combined with the idea of multiple open lists.
If a node does not exist in a closed list, it is inserted into the open list at the target of the first hash function.
In addition to facilitating lookup, randomization also equalizes quality of nodes in various open lists.
February 25, 2019

71. Speedup Anomalies in Parallel Search
Since the search space explored by processors is determined dynamically at runtime, the actual work might vary significantly.
Executions yielding speedups greater than p by using p processors are referred to as acceleration anomalies. Speedups of less than p using p processors are called deceleration anomalies.
Speedup anomalies also manifest themselves in best-first search algorithms.
If the heuristic function is good, the work done in parallel best-first search is typically more than that in its serial counterpart.
February 25, 2019

72. Speedup Anomalies in Parallel Search
The difference in number of nodes searched by sequential and parallel formulations of DFS. For this example, parallel DFS reaches a goal node after searching fewer nodes than sequential DFS.
February 25, 2019

73. Speedup Anomalies in Parallel Search
A parallel DFS formulation that searches more nodes than its sequential counterpart
February 25, 2019

74. To here, The Presentation ends
By Yazeed K. Almarshoud
CS6260

75. Presentation Question:
What are the two methods of Resolving collusions in addressing hashing functions mentioned in the presentation? In Parallel systems, when a processor runs out of work, it gets more work from another processor (How?).
Part One:
The two methods are:
Resolving collusion by chaining.
Resolving collusion by open addressing.
Part Two:
This is done using work requests and responses in message passing machines and locking and extracting work in shared address space machines.