1. YOLT Yuan Zheng Omar Ahmed Lukas Dudkowski T. Mark Kuba

2. Overview of YOLT Simple scripting language Simple scripting language Easy for coding and maintenance. Easy for coding and maintenance. Regular expression support Regular expression support := and @ := and @ “Web-scraping” uses “Web-scraping” uses Natural Language Processing Natural Language Processing Generating RSS Feeds Generating RSS Feeds Reformatting HTML for other uses (XML,etc) Reformatting HTML for other uses (XML,etc)

3. A Useful YOLT Program

4. Semantics YOLT Semantic checker is extremely simple. It serves a few main tasks: YOLT Semantic checker is extremely simple. It serves a few main tasks: Make sure that functions are declared properly, i.e. function declarations match functions, and function calls match the declarations Make sure that functions are declared properly, i.e. function declarations match functions, and function calls match the declarations Make sure that variables are initialized before they are used (or, in some cases, un-initialized) Make sure that variables are initialized before they are used (or, in some cases, un-initialized) (redundant) Make sure that the tree is properly formed (i.e. make sure that an if-then-else node has exactly three children, etc) (redundant) Make sure that the tree is properly formed (i.e. make sure that an if-then-else node has exactly three children, etc) *note*: there was once basic type-checking, but no longer.

5. Semantics Lessons Learned It is very easy to do too much in semantic checking It is very easy to do too much in semantic checking Either there are types, or no types (NO MIDDLE GROUND) Either there are types, or no types (NO MIDDLE GROUND) Scripting languages are an enormous relief to a semantic checker--they take away the biggest hassles Scripting languages are an enormous relief to a semantic checker--they take away the biggest hassles The tree walker should know EXACTLY what the structure of the AST will look like and cannot make ANY assumptions--things, as evident, can break down when you least expect them to. The tree walker should know EXACTLY what the structure of the AST will look like and cannot make ANY assumptions--things, as evident, can break down when you least expect them to.

6. Code Generation Written in Java Written in Java Input: correct AST Input: correct AST Output: Perl program Output: Perl program AST Code generator Perl Program Java

7. Implementation Walk AST Walk AST According to the information of the node, generate code or go down to the child node According to the information of the node, generate code or go down to the child node e.g.: := $a http://www.columbia.edu Go down to the tree at node “:=“ Generate code at node “$a” and “http://www.columbia.edu”

8. Implementation (tricks) The httpget := The httpget := invoke UNIX system call “wget” to download the web page into a temp file invoke UNIX system call “wget” to download the web page into a temp file Read the file line by line and store them into an perl array Read the file line by line and store them into an perl array Invoke another UNIX system call “rm” to remove the temp file Invoke another UNIX system call “rm” to remove the temp file Keep the web address in an perl scalar Keep the web address in an perl scalar Scalar and arrays use same syntax Scalar and arrays use same syntax Compiler (code generator) “guesses” whether the variable is a scalar or an array Compiler (code generator) “guesses” whether the variable is a scalar or an array Arrays can only appears in certain places (e.g.. Foreach) Arrays can only appears in certain places (e.g.. Foreach)

9. Documentation and Testing Lexer/Parser - Semantic Checker Lexer/Parser Semantic Checker Reference File: What I think it should produce Diff Log result: Good should be good. Bad should be bad. Test Cases ● Good ● Bad

10. Integration Testing Goal: display any comics that have the word hamster in the URL of www.toothpastefordinner.com, Summer 2002 archive. Goal: display any comics that have the word hamster in the URL of www.toothpastefordinner.com, Summer 2002 archive. Trying little YOLT programs to see functionality, code generation, etc. Working out bugs in implementation & design. Example: Yolt Program Generated Perl Resultant HTML $toothpaste_home ="http://www.toothpastefordinner.com/"; system('wget -q -O - http://www.toothpastefordinner.com/archives-sum02.php > toothpaste.txt'); open INFILE, "toothpaste.txt"; @toothpaste= ; close INFILE; system ('rm toothpaste.txt'); $toothpaste = "http://www.toothpastefordinner.com/archives-sum02.php"; $tags =".*hamster.* "; @tmp1=(); foreach ( @toothpaste) { if ($_=~m/($tags)/i){ push @tmp1, $2} } @elements = @tmp1; foreach $x ( @elements ) { print " "; print "\n"; } begin $toothpaste_home="http://www.toothpastefordinner.com/"; $toothpaste:="http://www.toothpastefordinner.com/archives- sum02.php"; $tags=".*hamster.* "; $elements = $tags @ $toothpaste; foreach $x in $elements { echo " "; echo "\n"; } end

11. The Result The source site The end result

12. Lessons Learned Develop and test incrementally Develop and test incrementally There are ALWAYS bugs, you just haven’t found them yet There are ALWAYS bugs, you just haven’t found them yet CLIC is not designed to be lived in CLIC is not designed to be lived in

13. One More Example