Progress Report 2010/1/04

Compared to the last report… Due to a miscommunication/misunderstanding about the last report, it covered up to Dec. 15. That was only 3 weeks ago. So there is not a great amount of new progress. The students are still trying to resolve the same issues. So I, the professor, have kept their slides and just added some new explanations in italics. I have made some progress on collaborations. This will be discussed at the end.

Add pipe in hadoop For now on hadoop can only do one thing, in one command like bin/hadoop fs –ls Pipes have potential of reducing communication & increasing the distributedness of computation We are still work on fully understanding hadoop’s local –cloud-node mechanisms – Students were offered a bonus if they could produce a specific outcome: “A file is created, not using Hadoop’s file creation feature, on one of the cloud nodes that is not the server, when some hdfs command is run.” So far, no student has won the bonus.

About MapReduce To add the pipe feature in hadoop, we need to create local temp file in each client The challenge about that is how to tell JobTracker to create the file The short-term reason that we are trying to create a file is as a test-of-understanding. For a new file to appear on a client is undeniable evidence that that code which we added is, in fact, running on the client. (Print statements won’t work, because the client doesn’t have terminal.)

WordCount example(1) We traced the wordCount example and tried to figure out how it works: The above code definitely seems to be the place where the word count program really runs. We inserted code here to create a file. The file only creates when the computer with the modified code is the server.

WordCount example(2) We used wireshark to find how the server told client to do the job

But we’re still working From the last slide, we see that the job is executed locally as a java code. But what exact java code is that? The answer is not as obvious as you would think. We hope to understand it soon. We still hope so…

ISPAN The professor and several students recently attended ISPAN (it was in Kaohsiung) There were several sections ostensibly about clouds and grid computing, but in reality they were all about grids. But I had fruitful discussions with Dr. Anthony of ANU and Dr. Wang of HKU, plus his student, King Tin. – Dr. Antony had good industry knowledge that suggests that our work is desirable. – Dr. Wang works with JESICA2 (now 3), which I have been looking at for possible use. I will visit HKU this month to discuss it further.