1. Visual Languages (Outline)
Motivation
Control Flow Diagrams
Data Flow Diagrams
Iconic Programming, Iconic Representation
Visual Languages for Computer-based Human Communication
Programming by Demonstration
CSE S. Tanimoto Visual Languages

2. CSE 341 -- S. Tanimoto Visual Languages

3. CSE 341 -- S. Tanimoto Visual Languages
Motivation
Programming is difficult because it’s hard to imagine what the state of the program is.
Programming is difficult because language constructs are cryptic.
Most PL concepts were developed in the text-only days, before graphical interfaces were widespread.
Human visual perception is capable of high-bandwidth input if images, rather than text, are used.
Visual languages can sometimes be learned more quickly.
CSE S. Tanimoto Visual Languages

4. CSE 341 -- S. Tanimoto Visual Languages
Control Flow Diagrams
Flow charts:
originally used only for specification and documentation
later, executable flow charts were developed.
Read n
product = 1 N
n > 1 Y
Print product
product *= n
n--
CSE S. Tanimoto Visual Languages

5. CSE 341 -- S. Tanimoto Visual Languages
Data Flow Diagrams
Captures potential parallelism
Usually requires scheduling
Allows cueing of types by line attributes
sin
sqr
Input double +
Output
cos
* 5
Input image
To Black&White
Output image
CSE S. Tanimoto Visual Languages

6. CSE 341 -- S. Tanimoto Visual Languages
Iconic Programming
Icons can sometimes be learned more quickly and remembered more easily than textual names.
An icon can represent an object or an operation.
“Icons on strings” -- replacement of the boxes in flowcharts and data flow diagrams by icons
Scripts composed of “iconic sentences”
Example: Pictorial Language for Animation by Youngsters (PLAY) -- Runyan & Tanimoto, 1985.
CSE S. Tanimoto Visual Languages

7. CSE 341 -- S. Tanimoto Visual Languages

8. CSE 341 -- S. Tanimoto Visual Languages

9. Iconic Representation
Icons can sometimes be learned more quickly and remembered more easily than textual names.
Icon design involves metaphor -- use of analogy.
Stylization -- graphical simplification and exaggeration.
Decontextualization -- the trashcan is not in the kitchen.
Abstract concepts are difficult to represent.
Composing new images takes longer than new names.
CSE S. Tanimoto Visual Languages

10. CSE 341 -- S. Tanimoto Visual Languages

11. Example Iconic Language
Vedo-Vedi: An iconic language for computer-based human communication.
Prototype supports multi-lingual communication
Meaning is expressed in an animation, with subtitles
developed by S. Tanimoto & C. Bernardelli, 1998.
CSE S. Tanimoto Visual Languages

12. Programming by Demonstration
The user performs a sample computation.
The system may generalize the sequence.
Requires an interface in which users can manipulate data directly.
Controlling the generalization process is tricky.
CSE S. Tanimoto Visual Languages

13. CSE 341 -- S. Tanimoto Visual Languages
Example Systems
Gael Curry: Programming by Abstract Demonstration -- UW, 1977.
Stagecast Creator: A simulation environment using programming by demonstration and rule-based simulation. (based on Cocoa & KidSim). Avail. at
ToonTalk: A robot programming environment supporting concurrent computation and complex control structures via a visual programming-by-demonstration method. (developed by Ken Kahn, available at
CSE S. Tanimoto Visual Languages

14. Example Commercial Software Development Language
Topological sort algorithm in Prograph. Prograph, by Pictorius, Inc. of Halifax, N.S., Canada. (uses data-flow paradigm)
CSE S. Tanimoto Visual Languages

15. Issues with Visual Languages
Scaling up: Diagrams can turn into “spaghetti”.
Editability: Programming by demonstration avoids an editable representation.
Naming without text: iconic naming conventions tend not to be as natural for programmers as textual ones.
Visual representations for complex concepts: Metaphors are more difficult to design and understand for abstract notions than simple ones.
CSE S. Tanimoto Visual Languages