Taming Development and Data Chaos in Military Logistics: The ICIS Experience With Lightweight Languages Gary Kratkiewicz BBN Technologies Geoffrey Knauth BAE Systems

LL2 - November 9, Gary Kratkiewicz & Geoffrey Knauth2 ICIS Background Web-based software system Analyzes future combat and humanitarian operations Measures ability of Defense Logistics Agency to satisfy wartime requirements for following items: Fuel Food Replacement items (repair parts, clothing, medical items, etc.) Identifies potential problems When will I run out of stock? What should I buy today for tomorrows needs? Do I have adequate sources & transportation?

LL2 - November 9, Gary Kratkiewicz & Geoffrey Knauth3 Accomplishments Early Web-based DOD system Saved over $100 million By computing reduced inventory needs for fuel and food Adopted as part of DOD doctrine Months of calculations reduced to hours What-if capability leads to better decisions Short analyses expanded role to crisis action Used in military operations to: Assess readiness Compute operational requirements

LL2 - November 9, Gary Kratkiewicz & Geoffrey Knauth4 Development Notes Started in 1994 Grew from small prototype to large system in 4 phases User interface moved to Web in stages from 1996 to 1999 Approximately a third of code is in Lightweight Languages Significant data issues

LL2 - November 9, Gary Kratkiewicz & Geoffrey Knauth5 Data, Data, Data Large amounts of data input files; 200 different formats Gigabytes of input and output per analysis Hundreds of analyses “active” Data management issues Received from many different agencies Regular and irregular updates Meandering formats Duplication between analyses: Computations Intermediate & output files

LL2 - November 9, Gary Kratkiewicz & Geoffrey Knauth6 1. Prototype ICIS – 1994/1995 Original prototypes completely coded in Tcl/Tk Vague, rapidly changing, and conflicting requirements Less than 1 MB of data Ease of developing/tailoring lightweight prototypes greatly contributed to initial success

LL2 - November 9, Gary Kratkiewicz & Geoffrey Knauth7 2. Production ICIS – 1995/1996 Lots of data: approaching 1 GB Addition of (mostly) well-specified computational modules Real use with classified data in mid-1996 Mix of languages: Tcl/Tk for GUI and glue for module execution Perl for data preprocessing C/C++ for computational modules

LL2 - November 9, Gary Kratkiewicz & Geoffrey Knauth8 3. Web ICIS – Address remote access issues Moved to Web in stages On server: Tcl/Tk – glue for module execution Perl – data preprocessing C/C++ and Java – computational modules Perl and C++ – dynamic generation of HTML and JavaScript In browser: HTML – static GUI elements JavaScript – form-based GUI elements Java AWT applets – interactive GUI elements

LL2 - November 9, Gary Kratkiewicz & Geoffrey Knauth9 4. Redesigned Web ICIS – 2001/2002 Improve GUI and better manage data Data approaching 1 TB On server: JScheme – managing data & module execution (sophisticated “glue”) Perl – data preprocessing C/C++ and Java – computational modules In browser: HTML – static GUI elements JavaScript (much less) – simple GUI elements Java AWT applets – more complex GUI elements Java Swing applets – most complex GUI elements JScheme for understanding scenario

LL2 - November 9, Gary Kratkiewicz & Geoffrey Knauth10 Source Code Evolution

LL2 - November 9, Gary Kratkiewicz & Geoffrey Knauth11 Summary of Lightweight Language Use in ICIS Rapid prototyping LL replaced by HL: completely for computations, partially for GUI Highly iterative, innovative design & development LL used in Data Set Manager, patent pending Processing of meandering data Gluing together heavyweight modules LL replaced with more sophisticated LL Building little languages Test code

LL2 - November 9, Gary Kratkiewicz & Geoffrey Knauth12 JScheme — Application Flexibility Interpreter for debugging Scripts for component functionality Can be a server for execution management of multiple tasks Example implementation: BBN’s Data Set Manager

LL2 - November 9, Gary Kratkiewicz & Geoffrey Knauth13 User Asks ICIS Questions User interaction with ICIS is web-based User defines composite scenario User specifies results of interest ICIS runs the assessment Numerous modules might be run It might take a while… Results appear in browser Example: “How much Fuel do I need?”

LL2 - November 9, Gary Kratkiewicz & Geoffrey Knauth14 An Example Small Assessment

LL2 - November 9, Gary Kratkiewicz & Geoffrey Knauth15 Problems Obvious approach of using shell scripts and files doesn’t cut it here Too much data Many computations recomputed needlessly

LL2 - November 9, Gary Kratkiewicz & Geoffrey Knauth16 A Full ICIS Assessment Does Many Computations The readable version of this graph takes up a large wall.

LL2 - November 9, Gary Kratkiewicz & Geoffrey Knauth17 Data Set Manager Data is described by features, not file names A module spec describes Inputs and outputs of a module Process relationships with other modules Modules can be implemented in any language A module spec is applied to a composite scenario to produce an execution script The script is executed All computations are memoized

LL2 - November 9, Gary Kratkiewicz & Geoffrey Knauth18 Execution Management Needs Dynamic Capabilities Data Set Manager takes full advantage of Scheme

LL2 - November 9, Gary Kratkiewicz & Geoffrey Knauth19 Implementing the Data Set Manager in JScheme was fairly easy because: Quasiquote was used to go from module spec to execution script A full language was required to write module specs Executing the script was just walking a tree structure

LL2 - November 9, Gary Kratkiewicz & Geoffrey Knauth20 Murphy Strikes Start Goal Mistrial!

LL2 - November 9, Gary Kratkiewicz & Geoffrey Knauth21 MVC: JScheme = M+C, Java = V Show Details Try Again Generate User Options Draw graph from knowledge of assessment state

LL2 - November 9, Gary Kratkiewicz & Geoffrey Knauth22 Data Set Manager — Conclusion Core implementation is 3804 lines of JScheme code Supported by 287 lines of Perl, 102 lines of SQL ICIS Module specs add another 1853 lines of JScheme Current APIs for other languages are: 688 lines of C 496 lines of Perl 356 lines of Java (1 class) 120 lines of JScheme BBN has applied for two patents