Visual Programming Languages ICS 539 Icon System Visual Languages & Visual Programming, Chapter 1, Editor Chang, 1990 ICS Department KFUPM Sept. 1, 2007

Theory of generalized icon Based on work by Chang Definitions Icon (generalized icon): An object with the dual representation of: logical part (the meaning) Physical part (the image) iconic system: structured set of related icons iconic sentence (visual sentence): spatial arrangement of icons from iconic system

Theory (con.) visual language: set of iconic sentences constructed with given syntax and semantics syntactic analysis (spatial parsing): analysis of an iconic sentence to determine the underlying structure semantic analysis (spatial interpretation): analysis of an iconic sentence to determine the underlying meaning

Theory (con.) A visual Language Models an Icon System. In the Icon System, a program (Visual Sentence) consists of a spatial arrangement of pictorial symbols (elementary icons). The spatial arrangement is the two-dimensional counterpart of the standard sequentialization in the construction of programs in the case of traditional programming languages.

In traditional languages, a program is expressed by a string in which terminals are concatenated. As this operation (concatenation) is the only construction rule, it does not appear explicitly in the vocabulary of the language grammar. In the case of visual languages, three construction rules are used to spatially arrange icons: Horizontal concatenation & Vertical concatenation ^ Spatial overlay +

The elementary icons are partitioned into: For this reason, there is a need to include the spatial operators and elementary icons in the vocabulary of terminals of the visual language grammar. The elementary icons are partitioned into: object icons process icons. Elementary object icons identify objects while process icons express computations. The meaning depends on the specific visual sentence in which it appears.

Example Consider the following primitive icons from the Heidelberg icon set. where the square denotes a character, and its interpretation is unique in the system. the arrow can be used to express insertion or moving operations in different visual sentences.

A Visual Language (VL) is specified by the triple (ID, Go, B) where ID is the icon dictionary Go is a context-free grammar B is a domain-specific knowledge base.

The icon dictionary ID It is a set of triples of the form i = (il , ip , type(i)) Each triple i describes a primitive icon where il is the icon name (logical part or meaning) ip is the icon sketch (physical part) type(i) is the icon type (process or object)

The grammar Go It is a context-free grammar (N, T, S, P). Where N is the set of nonterminals T = Tl U Tsp Tl = { il | i = ( il, ip, type(i))  ID} Tsp = { &, ^, + } S is the start symbol of the grammar P is the set of productions of Go

The grammar Go specifies how more complex icons can be constructed by spatially arranging elementary icons. Usually, nonterminals in Go & the start symbol represent composite object icons. Composite object icons can be derived in one or more steps starting from a given nonterminals, N, which is different from S. In other words, composite object icons are obtained by spatial arrangement of elementary object icons only and are assumed to be of type object.

Visual Sentences contain at least one process icon. Visual Sentences have no type. Here, object icons denote both elementary object icons & composite object icons. It is easy to prove that all the icons involved in the icon system (elementary object, composite object, and visual sentences) are objects with dual representation of a logical and a physical part as follows:

Elementary object --- by definition in ID. Composite object and visual sentences --- The physical part is their image The logical part (the meaning) can be derived from the meaning of the elementary objects occurring in them. The meaning of a visual sentence is expressed in terms of a conceptual tree, which represents an intermediate code for later execution.

Conceptual Trees The meaning of a visual sentence is expressed in terms of a conceptual tree (CT), which represents an intermediate code for later execution. In CT notations a concept is denoted by [ ], and a relation is denoted by ( ).

The concepts are expressed in the form [ OBJECT : object-name] [ EVENT : event-name] where object-name represents the logical part (meaning) of an object icon (elementary icon or composite object icon). event-name are event names of a process icon, i.e., procedure names, which accomplish the task expressed by the visual sentence.

The meanings of these CTs are as follows: The following CTs are basic substructures occuring in all the CTs which represent the logical part of a visual sentence. [EVENT: event-name] → (rel1) → [OBJECT: object-name1] → (rel2) → [OBJECT: object-name2] [EVENT: event-name] → (rel) → [OBJECT: object-name] [OBJECT: object-name] → (rel) → [OBJECT: object-name] The meanings of these CTs are as follows:

1) A process icon has two object icons as arguments Example: consider the following icon from the Heidelberg icon set

It is composed of the object icons Selected-string String and the process icon Up-arrow

Its meaning is expressed by the CT [EVENT: replace] (object) [OBJECT: String] (place) [OBJECT: Selected-string]

2) A process icon requires only one argument. Example:

It is obtained by composing in spatial overlay: the object icon Selected-string and the process icon cross

Its meaning is described by the CT [EVENT: delete] (object) [OBJECT: Selected-string]

The object "name” identifies the object "file" 3) Here we have two combined object icons. This is the case in which one of the two objects involved in the visual sentence represents a qualitative or quantitative specification of the other object. Example: The object "name” identifies the object "file"  File name

The meaning is [OBJECT: file] (identifier) [OBJECT: name]

Complex structures can be obtained by composing the basic substructures as shown below: 1 2 3 This visual sentence could mean: append file1 to file2 and copy the result into file3   

The sentence could be decomposed into two subsentences: 1 2 and 2 3    

These subsentences are described, respectively, by the following CTs [EVENT: append] (object) [OBJECT: file1] (place) [OBJECT: file2] [EVENT: copy] (source) [OBJECT: file2] (destination) [OBJECT: file3]

The whole visual sentence is described by [EVENT: copy] (destination) [OBJECT: file3] (source) [EVENT: append] (object) [OBJECT: file1] (object) [OBJECT: file2]

The Knowledge Base B The Knowledge Base B contains domain-specific information necessary for constructing the meaning of a given visual sentence. It contains information regarding Event names Conceptual relations Names of the resulting objects References to the resulting objects.

It is structured in the following seven tables: 1) EVENT-NAME [proc-name, obj-name1, obj-name2, op1, op2] This table returns the name of the event (procedure) to be associated to the process icon proc-name when the objects obj-name1, obj-name2 are spatially arranged (via op1, op2) to proc-name. 2) LEFT-REL [proc-name, obj-name1, obj-name2, op1, op2] 3) RIGHT-REL [proc-name, obj-name1, obj-name2, op1, op2] These two tables hold the conceptual relations existing between EVENT-NAME [Proc-name, obj-name1, obj-name2, op1, op2] and obj-name1 or obj-name2 respectively.

4) RESULT-NAME [Proc-name, obj-name1, obj-name2, op1, op2] This table returns the name of the object resulting from the execution of the event. EVENT-NAME [Proc-name, obj-name1, obj-name2, op1, op2]. With arguments obj-name1 and obj-name2. 5) RESULT-REF [Proc-name, obj-name1, obj-name2, op1, op2]. This table returns a reference to the object representing the result.

The following two tables take into account cases in which object icons are composed (composite object icons). 6) RELATION [obj-name1, obj-name2, op]. This returns the conceptual relation existing between obj-name1 and obj-name2 when they are spatially combined by means of the operator op. 7) RESULT-OBJECT [obj-name1, obj-name2, op]. This returns the name of the object resulting by the spatial combination (via op) of obj-name1 and obj-name2. In the previous tables, obj-namei refer to the logical part of the icons.

Besides domain-specific relations the tables can also contain three special relations: <new> :This relation means that the combination obj-name1 op obj-name2 gives rise to a new object, different from obj-name1 and obj-name2. <null>:This relation means that the combination obj-name1 op obj-name2 generates an object that has the same logical part of obj-name1 or obj-name2. In other words, it points out that no additional information is derived from such a combination.

<l-obj> , <r-obj>: These two relations mean that the combination obj-name1 op1 proc-name op2 obj-name2 generates an object that has the same reference of the icon whose name is obj-name1 or obj-name2 respectively.

Example file3 file2 Obj-name1 Op1=& Obj-name2 Op2=& Proc-name  file2  Obj-name1 Op1=& Obj-name2 Op2=& Proc-name

Table 1: returns the name of the event “copy” Table 2, 3: LEFT-REL [EVENT: copy] (source) [OBJECT: file2] (destination) [OBJECT: file3] RIGHT-REL

Table 5: Return a reference (pointer) to file3 Table 6: Table 4: Return file 3 Table 5: Return a reference (pointer) to file3 Table 6: [OBJECT: file] (identifier) [OBJECT: file3] returns this Table 7: Return the name of the resulting object in 6

The Attribute Grammar An attribute grammar consists of a semantics rules for synthesizing the meaning of a visual sentence AG consists of an underlying context-free grammar where i) each nonterminal in the context-free grammar has associated two attribute sets, namely, synthesized and inherited attributes.

ii) each production has associated a set of semantics rules, used to determine the values of the attributes of the nonterminals depending on the values of the attributes of the remaining nonterminals appearing in the same production. The initial nonterminal has only synthesized attributes, and one of them is designated to hold the meaning of the derivation tree.

The production rules in Go are of the following form: a) X t t  T b) X Y where type (X) = type (Y) c) X M op L | L op M | M1 op M2 d) X M1 op1 L op2 M2 where type (L) = PROCESS type (M) = type (M1) = type (M2) = OBJECT. op, op1, op2 = nonterminals for spatial operators ( &, * , +)

The type for a given nonterminal X is determined as follows: Let X t be a derivation in Go and t  T* Then type (X) = spatial operator if t  Tsp type (X) = type (t) otherwise. The set of inherited attributes is empty for each nonterminal in Go.

The set of synthesized attributes associated to each nonterminal X is defined as follows. For a given non-terminal X 1. if type (X) = OBJECT : NAME (X) is the name of the object icon represented by X. CG(X) is the CT associated to the subtree rooted in X. REF(X) is the reference number. The reference number is an index associated with each elementary object icon occurring in a visual sentence. It allows one to distinguish between different occurrences of the same elementary object icon in a visual sentence.

2. if type (X) = PROCESS: NAME(X) is the name of the process icon represented by X. 3. if type (X) : "spatial operator" : OP(X) is the spatial operator derived from X.

The semantic rules associated with the four production rules are: a) X t a1) If t = icon-id where ( icon-id, icon-sk, type( i ) )  ID and type ( i ) = OBJECT (i.e. t is a terminal for an elementary icon). Then NAME(X) icon-id REF (X) ref-set CG (X) [OBJECT : NAME(X), REF(X)]

a2) If t = icon-id where ( icon-id, icon-sk, type( i ) )  ID and type ( i ) = PROCESS Then NAME(X) t a3) If t  {&, * , +} OP(X) t

b) X Y b1) If type (Y) = OBJECT Then NAME(X) NAME (Y) REF (X) REF (Y) CG (X) CG (Y) b2) If type (Y) = PROCESS b3) If Y op and op  {&, * , +} OP(X) OP(Y)

c) X M op L c1) If type (M) = OBJECT and type (L) = PROCESS Then NAME(X) RESULT-NAME [NAME(L), NAME (M), ‘null-obj’, OP(op), ‘null-op’] REF(X) SET-REF (L, M, ‘null-obj’, op, ‘null-op’) CG (X) SELECTIVE-MERGE (L, M, ‘null-obj’, op, ‘null-op’)

c2) If type (M) = PROCESS and type (L) = OBJECT Then NAME (X) RESULT-NAME [NAME(M), ‘null-obj’, NAME(L), ‘null-op’, OP(op)] REF(X) SET-REF (M, ‘null-obj’, L, ‘null-op’ , op) CG (X) SELECTIVE-MERGE (M, ‘null-obj’, L, ‘null-op’, op)

c3) If type (M) = type (L) = OBJECT Then NAME (X) RESULT-OBJECT [NAME (M), NAME(L), OP(op)] REF(X) REF (M) U REF (L) CG (X) LINK (X, M, L, op)

d) X M1 op1 L op2 M2 If type (L) = PROCESS and type (M1) = type (M2) = OBJECT Then NAME (X) RESULT-NAME [NAME (L), NAME(M1), NAME (M2), OP(op1), OP (op2)] REF(X) SET-REF (L, M1, M2, op1, op2) CG (X) SELECTIVE-MERGE (L, M1, M2, op1, op2)

Semantic rules c and d make use of the following procedures: Procedure LINK (X, M1, M2, op) rel RELATION [NAME(M1), NAME (M2), OP(op)] case rel of <new> : if NAME (X)  ID then - add the new composite object NAME(X) to ID - return [OBJECT: NAME (X), REF(X)] endif <null> : return CG(M1) else : return APPEND (CG(M1), CG(M2), rel) end case end LINK

Procedure SELECTIVE-MERGE (L, M1, M2, op1, op2) C [EVENT:EVENT-NAME [NAME(L), NAME(M1), NAME(M2), OP(op1), OP(op2)] if OP(op1) = ‘null-op’ then G1 C else rel1 LEFT-REL [NAME(L), NAME(M1), NAME (M2), OP(op1), OP(op2)] G1 APPEND (C, CG(M1), rel1) endif if OP(op2) = ‘null-op’ then G2 C else rel2 RIGHT-REL [NAME(L), NAME(M1), G2 APPEND (C, CG(M2), rel2) return HEADER-MERGE (G1, G2) end SELECTIVE-MERGE

Procedure SET-REF (L, M1, M2, op1, op2) ref RESULT-REF [NAME(L), NAME(M1), NAME (M2), OP(op1), OP(op2)] case ref of <l-obj> : return REF(M1) <r-obj> : return REF(M2) <new-obj>: return NEXT-REF( ) end case end SET-REF

The Icon Interpreter: The system diagram of the icon interpreter is as follows: G0 ID B Visual Sentencce S Construction of the Logical part Pattern Analysis Syntax Analysis Meaning of S CG Pattern String of S Parse Tree of S

The module ”Pattern Analysis" transforms the visual sentence into pattern string . Example The visual sentence is represented by the pattern string (char & char & char) + cross

The icon interpreter is logically divided into two phases: 1. The syntax analysis is accomplished by a parsing algorithm for context-free grammars. 2. Constructing the meaning of the given visual sentence by evaluating the attributes of the nonterminal on the root of the parse tree using the ATTRIBUTE_EVALUATOR procedure The attribute evaluation is accomplished by means of a postorder visit of the parse tree t of the visual sentence S

Procedure ATTRIBUTE-EVALUATOR (X). if X is a nonterminal in Go then /* let p be the production applied to the node X in t, and let 1, ..., np be the indices of nonterminals in the right-hand side of p*/ for j = 1 to np do call ATTRIBUTE-EVALUATOR (jth nonterminal son of X in t) end for apply the semantic rules associated to p so as to evaluate the attributes of X endif end ATTRIBUTE-EVALUATOR