WINTER 2016 – TERM PRESENTATION MICHAEL O’KEEFE

PAST RESEARCH - SUMMER 2015 Continued Jason Woodring’s research on UWCA Main issue with UWCA is the slow file reading performance A number of possible reasons (UWCA code, MASS, Netcdf, etc) Netcdf and text file performance testing Sequential read performance vs distributed read performance Performed tests using different storage methods Results confirmed that both text files and Netcdf files are read faster on multiple computing nodes

PAST RESEARCH - SUMMER 2015 (CONT)

WHAT ARE NETCDF FILES? Stands for Network Common Data Form Created by Unidata, part of the University for Atmospheric Research (UCAR) Array-oriented scientific data Consists of three parts: Dimensions (UWCA: longitude, latitude, time) Variables (UWCA: tasmax) Attributes (metadata)

FALL PAST RESEARCH Spent majority of my time looking through UWCA for causes of its slow read performance – very slow process since UWCA is such a large application Found some minor issues Was not making any significant progress Doctor Fukuda thought of a way to solve the performance problem and benefit the MASS Library Began designing how to implement Parallel I/O in the MASS Library

MOTIVATION FOR PARALLEL I/O Since the MASS library does not have any parallel I/O, users must read and/or write files via the master node, (i.e., the main program), which will become a bottleneck between the disk and all slave nodes participating in the same computation Parallel I/O would solve UWCA’s performance issue, and simultaneously benefit other MASS applications that could use parallel I/O capabilities and increase application performance

PARALLEL I/O - INITIAL DESIGN

PARALLEL I/O – INITIAL DESIGN (CONT)

PARALLEL I/O – CURRENT DESIGN Implemented in Place.java of the MASS Library Fields:

PARALLEL I/O - CURRENT DESIGN (CONT) Private class within Place.java – class FileAttributes:

PARALLEL I/O – OPEN CURRENT DESIGN Open method: Must be synchronized Requires a path to the file to be opened and a specified I/O type (0 for read, 1 for write) Returns a unique file number (descriptor) * Explain the process view *

NETCDF FILE OPEN VS OPEN IN MEMORY Recently found the method openInMemory(fileName) for Netcdf files Ran performance tests to see if this method could be useful: Created a 3D Netcdf File writing program, where the size of the file can easily be adjusted Created a Netcdf File reading program that has the option to open the written file in memory or on the disk, and then reads the file and outputs the time it took to read in milliseconds

RESULTS The reading time was on average 40% faster using openInMemory() (tested with varying file sizes from 500MB to 2GB) Note that occasionally the first read after opening in memory will only be 20% faster Note that a file size over 2GB cannot be opened in memory Note that the opening in memory process does take longer than just opening (of course!) Ran tests on Juno

RESULTS (CONT) One set of results:

PARALLEL I/O – CLOSE CURRENT DESIGN Close Method: Synchronized Returns true if the given file descriptor is closed properly; otherwise, false Takes a file number (descriptor) as a parameter If the file descriptor exists in the file table, the file descriptor is closed and the file is removed from the file table

UNIT TESTING – THANKS MATT EasyMock & Junit Allows developers to isolate pieces of code and test them using mock objects Can create any test cases you can think of Using random generators could help catch some edge cases Gives you peace of mind when changing your code (can check if tests pass) Isolating the code to be tested allows developers to not worry about other components of the application that might have errors (you know exactly where the bugs are)

MY UNIT TESTING Created a class called PlaceTest.java Created Place test subjects Created multiple text and Netcdf files to perfromed I/O operations on Can run the tests and compare output with expected output Right now: open and close functionality works for both text and Netcdf files Read is working for text files

UNIT TEST EXAMPLE Note that no mock objects are needed Would you like to see the test run?

TO BE IMPLEMENTED Netcdf reading – need to figure out how to determine the structure of an unknow Netcdf file Text and Netcdf writing – need to determine how the user specifies what is to be written File partitioning and distribution tool – splits a file depending on how many computing nodes are being used and distributes each file piece to their respective node – in the /tmp directory File collecting and merging tool – collects the file pieces from each node after I/O operations have been preformed, and merges pieces back to one file

QUESTIONS – ME ASKING YOU To implement write: Need to know what to write to a file How do you want the user to give us that information? Will read be non-synchronized if it is called from a synchronized method, open? I found this earlier this week, which brings up some questions:

WHAT I HAVE LEARNED File I/O in Java – synchronized, FileChannel, ByteBuffer, OpenOptions Linux – file descriptos, working in Terminal Unit Testing Source Tree, Bitbucket Avoiding race conditions – testAndSet() Netcdf Files - reading, writing, determining structure

WHAT I WOULD DO DIFFERENTLY (GOOD LESSON) Design before for implementation: I was eager to get start coding, especially when Doctor Fukuda wrote out the basic design (whiteboard pictures) A lot of that original design was changed It would have been better if I tried to break down the entire task before writing a single line of code Draw flow charts Write pseudocode Ask questions

QUESTIONS? – YOU ASKING ME Thanks for listening!