1. Towards an Archival Intermemory
Paper authored by Andrew Goldberg and Peter N. Yianilos
cs791
Aravind Elango

2. Aim of this paper
To provide a concrete mechanism to preserve data against time and adversarial attacks.
To explain the design of certain parts of such a mechanism.
Finally, a note about the outstanding issues.

3. What is the deal ?
Each subscriber of the system donates some amount of space for a certain period of time.
After the time period, the user may withdraw the space provided.
But the subscriber’s data will always be retained forever.
Payback is usually about 20%.

4. Where is all the data stored ?
In the Intermemory
A vast, distributed, de-centralized storage space on the internet.
It is the collection of space on the internet that every subscriber provides.

5. Structure of the Intermemory
The Intermemory is divided into memory blocks in which data is stored.
An example
2 ^ m blocks form the Intermemory.
Unique block address contains m bits.
Each block contains n words.
Each word will contain w bits.

6. Erasure Codes Commonly used for computer security.
Keys are replicated to preserve them.
A key is encoded into n subparts.
The key can be decoded from d encoded subparts where d < n.
But attackers cannot decode the key if they have only d-1 sub parts.

7. Storage in Intermemory
Each memory word (d bits) is encoded into 2*d bits.
The 2*d bits are divided into n parts and dispersed to n processors.
Each divided part is stored in a memory block of IM.
Since we are using erasure codes, we only need d out of the n parts to get back the original data.
Typically d in the paper is n/2.
Thus even if only half the processors are online, we will still be able to regenerate the original data.

8. Advantages of this method
If a file of size x bytes is replicated in 4 mirror sites, the file might be lost if 4 nodes go down.
For the same increase in size, the IM might be able to retrieve the file even after (2^18)-1 nodes in the network have been lost.
The level of dispersal is limited by the overhead it might produce.

9. Trading a moment for eternity
It is assumed that IM would grow for every successive unit of time.
The more older a file is, the more difficult it is to loose it.
Potential problems
Increasingly lower efficiency for new users.
IM might actually shrink or experience uneven growth.

10. An Example G – unit of IM growth per unit of time.
rs – rate of increase in IM
rd – rate of decrease in IM (1/processor life time)
Example:
rs = rd = 0.001
Number of days a user should provide storage = 1000 days
G = 1 – =
(G – 1)/ G ~= (Interest gained per unit of time(day))
For 1000 days, 1000* = 0.5 units of perpetual storage.
A user could store in IM for eternity, 50% of the memory he/she initially hosted.

11. Addressing VP# * VP# * …….*VP#
IM-addr
VP#
IP# * Port # N A P
VP# * VP# * …….*VP#
Nbd(P)
In their prototype implementation A is 128 bit and P is 64 bit. N is IP address + port number.

12. Concept of neighborhood
A set of virtual processor numbers are generated with the Physical number P as seed.
These processors would form the neighborhood of P, VP’s to which blocks could be dispersed from P.

13. Outstanding Issues
What’s been proposed is a write alone memory. Read-Write memory would be more challenging due to inconsistencies of distributed storage.
IM addresses migration to newer media as old nodes go down and new ones are introduced, but does not address the issue of preserving the environment of the data.

14. Other possible services
Distributed file system over IM.
Will help users store information in a hierarchical fashion.
Provide better facilities for searching through classification.

15. Conclusion
They have implemented a working model of the Intermemory with over 100 users.
While the feasibility of the project hinges on many unknowns, I think it is a very cool idea !

16. Thank You