1. Jason Stowe Condor Week 2009 April 22 nd, 2009

2. Coming to Condor Week since 2005. Started as a User

3. Users hunger for features

4. AccountingGroups (2004/2005) Configuration w/Pipes (2005/2006) GroupResourcesUsed (2006/2007) Condor in Cloud (2007/2008) Resource Weights (2008/2009) Based upon customer requests

5. Focus on software development for managing Condor at any scale, and provide services that complement the technology

6. Universities, Fortune 500s, Government Labs, Small/Medium Businesses, that use Condor

7. Users like Condor because... It’s open, it works, flexible, (corporations) no lock-in API/Operating System, and...

8. The Community

9. Today, let’s talk about a few challenges, solutions

10. War Story #1: Compute & Data

11. Whenever you find or solve a computation problem, you discover a data problem.

12. “Dark” or Latent, Unused Storage on any OS/Device

13. Empty space dispersed across machines in unusable sizes

14. “We need more filer space, but we have empty space on all our machines.”

15. So we looked at Hadoop

16. New type of storage: Aggregated or “Cloud” Storage

17. Block Store Architecture

18. But how do we use it?

19. 1.5 years ago: It works well to access it in Java, but what about mounting?

20. So we tried WebDAV

21. Next up, open source FUSE driver

22. Need: Windows/Linux, Reliable, Large Files, scalable, and Read/Write

24. Mountable drivers Linux(FUSE) / Windows (IFS)

25. CloudFS Architecture

26. When we rolled it out...

27. Customers Asked for Surprising Features HTTP/REST Protocols similar to Amazon S3 HTTP/REST Protocols similar to Amazon S3Reasons: Installing mountable driver across servers/workstations prohibitive Want similar interface to various cloud storage providers => Internal Cloud FTP Interface – Because it is simple! FTP Interface – Because it is simple!

28. Status Today

29. Mountable Multi-platform Drivers. Linux: SUSE 10, RHEL/CentOS 4&5, Windows 2k3 +, OSX 10.3+

30. Encryption to avoid snooping sensitive data

31. Data Nodes built on Java: Linux, Windows, OSX, Solaris

32. RESTful Storage Service & FTP interface

33. Management interface for controlling storage features (Integrating with CycleServer)

34. Looking forward to condor_hadoop!

35. War Story #2: Cloud Calculations

36. Condor users Peak vs. Median usage Problem

37. Need for compute power comes up suddenly

38. Condor Users hunger for resources

39. Condor users balance “We need more servers for big runs” and “Our servers are 40% utilized”

40. Many ways to solve this problem using EC2

41. Use cases do exist for adding nodes to a local condor pool using Amazon EC2

42. We favored entire pools in cloud

43. Data Scheduling, Performance issues

44. Run workflows faster using resources you could never buy...

45. can test CycleServer at a scale our users have and we don’t

46. Need 1000 node Condor Pool Wait 15 minutes

47. Dynamic Resources => Pool can be sized to the jobs Dynamic Resources => Pool can be sized to the jobs

48. 1 corex 1000 hrs = 1000 core x 1 hr = ~$200

49. Sounds good, but how do we do this for a Workflow like BLAST?

50. From e-science 2008: For 64x the processors Hadoop Running Blast: 57x mpiBLAST: 52.4x

51. High-CPU Amazon EC2 nodes have best price/performance

52. Scalability: 2x CPUs = 1.9825x 64 CPUS = 60.7x Speed-up

53. Why High Throughput leads to Efficient Computing

54. Another User: Worked with Varian - Mass Spectrometers Other High-Tech Lab Equipment

55. Problem: Coming up on a conference, needed to run a large simulation

56. Six Weeks On an internal Condor pool

57. Deployed a Condor pool in CycleCloud

58. Same 6-week Job

59. Ran < 1 Day

60. War Story #3: Management

61. Condor Tutorial mentions “Why use a personal Condor?” i.e. Condor on few nodes...

62. Condor on 1 computer Gets you policies, fault-tolerance, Etc.

63. Similarly, management issues come up even on small pools

64. Collaborating with U. of W. Madison

65. Managing Configuration Files (our Config with Pipes CW2006)

66. Exploring ClassAds/LogFiles becomes problematic

67. Visualization, Reporting, etc.

68. Man-decades on development of tools to assist running Condor

69. Have demo against Madison pool Come see me. We’d love more use cases

70. Questions? Thank you For more information go to: http://www.cyclecomputing.com We constantly see opportunities for talented Condor folks, so please feel free to contact us! Jason Stowe jstowe - cyclecomputing.com