1. Visual Programming Languages ICS 539 Icon System Visual Languages & Visual Programming, Chapter 1, Editor Chang, 1990
ICS Department
KFUPM
Sept. 1, 2007

2. Theory of generalized icon
Based on work by Chang
Definitions
Icon (generalized icon): An object with the dual representation of:
logical part (the meaning)
Physical part (the image)
iconic system: structured set of related icons
iconic sentence (visual sentence): spatial arrangement of icons from iconic system

3. Theory (con.)
visual language: set of iconic sentences constructed with given syntax and semantics
syntactic analysis (spatial parsing): analysis of an iconic sentence to determine the underlying structure
semantic analysis (spatial interpretation): analysis of an iconic sentence to determine the underlying meaning

4. Theory (con.) A visual Language Models an Icon System.
In the Icon System, a program (Visual Sentence) consists of a spatial arrangement of pictorial symbols (elementary icons).
The spatial arrangement is the two-dimensional counterpart of the standard sequentialization in the construction of programs in the case of traditional programming languages.

5. In traditional languages, a program is expressed by a string in which terminals are concatenated.
As this operation (concatenation) is the only construction rule, it does not appear explicitly in the vocabulary of the language grammar.
In the case of visual languages, three construction rules are used to spatially arrange icons:
Horizontal concatenation &
Vertical concatenation ^
Spatial overlay +

6. The elementary icons are partitioned into:
For this reason, there is a need to include the spatial operators and elementary icons in the vocabulary of terminals of the visual language grammar.
The elementary icons are partitioned into:
object icons
process icons.
Elementary object icons identify objects while process icons express computations.
The meaning depends on the specific visual sentence in which it appears.

7. Example
Consider the following primitive icons from the Heidelberg icon set.
where
the square denotes a character, and its interpretation is unique in the system.
the arrow can be used to express insertion or moving operations in different visual sentences.

8. A Visual Language (VL) is specified by the triple
(ID, Go, B)
where
ID is the icon dictionary
Go is a context-free grammar
B is a domain-specific knowledge base.

9. The icon dictionary ID It is a set of triples of the form
i = (il , ip , type(i))
Each triple i describes a primitive icon where
il is the icon name (logical part or meaning)
ip is the icon sketch (physical part)
type(i) is the icon type (process or object)

10. The grammar Go It is a context-free grammar (N, T, S, P). Where
N is the set of nonterminals
T = Tl U Tsp
Tl = { il | i = ( il, ip, type(i))  ID}
Tsp = { &, ^, + }
S is the start symbol of the grammar
P is the set of productions of Go

11. The grammar Go specifies how more complex icons can be constructed by spatially arranging elementary icons.
Usually, nonterminals in Go & the start symbol represent composite object icons.
Composite object icons can be derived in one or more steps starting from a given nonterminals, N, which is different from S.
In other words, composite object icons are obtained by spatial arrangement of elementary object icons only and are assumed to be of type object.

12. Visual Sentences contain at least one process icon.
Visual Sentences have no type.
Here, object icons denote both elementary object icons & composite object icons.
It is easy to prove that all the icons involved in the icon system (elementary object, composite object, and visual sentences) are objects with dual representation of a logical and a physical part as follows:

13. Elementary object --- by definition in ID.
Composite object and visual sentences ---
The physical part is their image
The logical part (the meaning) can be derived from the meaning of the elementary objects occurring in them.
The meaning of a visual sentence is expressed in terms of a conceptual tree, which represents an intermediate code for later execution.

14. Conceptual Trees
The meaning of a visual sentence is expressed in terms of a conceptual tree (CT), which represents an intermediate code for later execution.
In CT notations a concept is denoted by [ ], and a relation is denoted by ( ).

15. The concepts are expressed in the form
[ OBJECT : object-name]
[ EVENT : event-name]
where
object-name represents the logical part (meaning) of an object icon (elementary icon or composite object icon).
event-name are event names of a process icon, i.e., procedure names, which accomplish the task expressed by the visual sentence.

16. The meanings of these CTs are as follows:
The following CTs are basic substructures occuring in all the CTs which represent the logical part of a visual sentence.
[EVENT: event-name] → (rel1) → [OBJECT: object-name1] → (rel2) → [OBJECT: object-name2]
[EVENT: event-name] → (rel) → [OBJECT: object-name]
[OBJECT: object-name] → (rel) → [OBJECT: object-name]
The meanings of these CTs are as follows:

17. 1) A process icon has two object icons as arguments
Example:
consider the following icon from the Heidelberg icon set

18. It is composed of the object icons
Selected-string
String
and the process icon Up-arrow

19. Its meaning is expressed by the CT
[EVENT: replace] (object) [OBJECT: String]
(place) [OBJECT: Selected-string]

20. 2) A process icon requires only one argument.
Example:

21. It is obtained by composing in spatial overlay:
the object icon Selected-string
and the process icon cross

22. Its meaning is described by the CT
[EVENT: delete] (object) [OBJECT: Selected-string]

23. The object "name” identifies the object "file"
3) Here we have two combined object icons. This is the case in which one of the two objects involved in the visual sentence represents a qualitative or quantitative specification of the other object.
Example:
The object "name” identifies the object "file"

File name

24. The meaning is
[OBJECT: file] (identifier) [OBJECT: name]

25. Complex structures can be obtained by composing the basic substructures as shown below:
This visual sentence could mean: append file1 to file2 and copy the result into file3




26. The sentence could be decomposed into two subsentences:
and





27. These subsentences are described, respectively, by the following CTs
[EVENT: append] (object) [OBJECT: file1]
(place) [OBJECT: file2]
[EVENT: copy] (source) [OBJECT: file2]
(destination) [OBJECT: file3]

28. The whole visual sentence is described by
[EVENT: copy] (destination) [OBJECT: file3]
(source) [EVENT: append]
(object) [OBJECT: file1]
(object) [OBJECT: file2]

29. The Knowledge Base B
The Knowledge Base B contains domain-specific information necessary for constructing the meaning of a given visual sentence.
It contains information regarding
Event names
Conceptual relations
Names of the resulting objects
References to the resulting objects.

30. It is structured in the following seven tables:
1) EVENT-NAME [proc-name, obj-name1, obj-name2, op1, op2] This table returns the name of the event (procedure) to be associated to the process icon proc-name when the objects obj-name1, obj-name2 are spatially arranged (via op1, op2) to proc-name.
2) LEFT-REL [proc-name, obj-name1, obj-name2, op1, op2]
3) RIGHT-REL [proc-name, obj-name1, obj-name2, op1, op2]
These two tables hold the conceptual relations existing between EVENT-NAME [Proc-name, obj-name1, obj-name2, op1, op2] and obj-name1 or obj-name2 respectively.

31. 4) RESULT-NAME [Proc-name, obj-name1, obj-name2, op1, op2]
This table returns the name of the object resulting from the execution of the event.
EVENT-NAME [Proc-name, obj-name1, obj-name2, op1, op2]. With arguments obj-name1 and obj-name2.
5) RESULT-REF [Proc-name, obj-name1, obj-name2, op1, op2].
This table returns a reference to the object representing the result.

32. The following two tables take into account cases in which object icons are composed (composite object icons).
6) RELATION [obj-name1, obj-name2, op].
This returns the conceptual relation existing between obj-name1 and obj-name2 when they are spatially combined by means of the operator op.
7) RESULT-OBJECT [obj-name1, obj-name2, op].
This returns the name of the object resulting by the spatial combination (via op) of obj-name1 and obj-name2.
In the previous tables, obj-namei refer to the logical part of the icons.

33. Besides domain-specific relations the tables can also contain three special relations:
<new> :This relation means that the combination
obj-name1 op obj-name2
gives rise to a new object, different from obj-name1 and obj-name2.
<null>:This relation means that the combination
obj-name1 op obj-name2
generates an object that has the same logical part of obj-name1 or obj-name2.
In other words, it points out that no additional information is derived from such a combination.

34. <l-obj> , <r-obj>:
These two relations mean that the combination
obj-name1 op1 proc-name op2 obj-name2
generates an object that has the same reference of the icon whose name is obj-name1 or obj-name2 respectively.

35. Example file3 file2 Obj-name1 Op1=& Obj-name2 Op2=& Proc-name

file2

Obj-name1
Op1=&
Obj-name2
Op2=&
Proc-name

36. Table 1: returns the name of the event “copy” Table 2, 3: LEFT-REL
[EVENT: copy] (source) [OBJECT: file2]
(destination) [OBJECT: file3]
RIGHT-REL

37. Table 5: Return a reference (pointer) to file3 Table 6:
Table 4: Return file 3
Table 5: Return a reference (pointer) to file3
Table 6:
[OBJECT: file] (identifier) [OBJECT: file3]
returns this
Table 7: Return the name of the resulting object in 6

38. The Attribute Grammar
An attribute grammar consists of a semantics rules for synthesizing the meaning of a visual sentence
AG consists of an underlying context-free grammar where
i) each nonterminal in the context-free grammar has associated two attribute sets, namely, synthesized and inherited attributes.

39. ii) each production has associated a set of semantics rules, used to determine the values of the attributes of the nonterminals depending on the values of the attributes of the remaining nonterminals appearing in the same production.
The initial nonterminal has only synthesized attributes, and one of them is designated to hold the meaning of the derivation tree.

40. The production rules in Go are of the following form:
a) X t t  T
b) X Y where type (X) = type (Y)
c) X M op L | L op M | M1 op M2
d) X M1 op1 L op2 M2
where
type (L) = PROCESS
type (M) = type (M1) = type (M2) = OBJECT.
op, op1, op2 = nonterminals for spatial operators ( &, * , +)

41. The type for a given nonterminal X is determined as follows:
Let
X t be a derivation in Go and t  T*
Then type (X) = spatial operator if t  Tsp
type (X) = type (t) otherwise.
The set of inherited attributes is empty for each nonterminal in Go.

42. The set of synthesized attributes associated to each nonterminal X is defined as follows.
For a given non-terminal X
1. if type (X) = OBJECT :
NAME (X) is the name of the object icon represented by X.
CG(X) is the CT associated to the subtree rooted in X.
REF(X) is the reference number. The reference number is an index associated with each elementary object icon occurring in a visual sentence. It allows one to distinguish between different occurrences of the same elementary object icon in a visual sentence.

43. 2. if type (X) = PROCESS:
NAME(X) is the name of the process icon represented by X.
3. if type (X) : "spatial operator" :
OP(X) is the spatial operator derived from X.

44. The semantic rules associated with the four production rules are:
a) X t
a1) If t = icon-id
where
( icon-id, icon-sk, type( i ) )  ID and
type ( i ) = OBJECT (i.e. t is a terminal for an elementary icon).
Then
NAME(X) icon-id
REF (X) ref-set
CG (X) [OBJECT : NAME(X), REF(X)]

45. a2) If t = icon-id
where
( icon-id, icon-sk, type( i ) )  ID and
type ( i ) = PROCESS
Then
NAME(X) t
a3) If t  {&, * , +}
OP(X) t

46. b) X Y
b1) If type (Y) = OBJECT
Then
NAME(X) NAME (Y)
REF (X) REF (Y)
CG (X) CG (Y)
b2) If type (Y) = PROCESS
b3) If Y op and op  {&, * , +}
OP(X) OP(Y)

47. c) X M op L
c1) If type (M) = OBJECT and type (L) = PROCESS
Then
NAME(X) RESULT-NAME [NAME(L), NAME (M), ‘null-obj’, OP(op), ‘null-op’]
REF(X) SET-REF (L, M, ‘null-obj’, op, ‘null-op’)
CG (X) SELECTIVE-MERGE (L, M, ‘null-obj’, op, ‘null-op’)

48. c2) If type (M) = PROCESS and type (L) = OBJECT
Then
NAME (X) RESULT-NAME [NAME(M), ‘null-obj’, NAME(L), ‘null-op’, OP(op)]
REF(X) SET-REF (M, ‘null-obj’, L, ‘null-op’ , op)
CG (X) SELECTIVE-MERGE (M, ‘null-obj’, L, ‘null-op’, op)

49. c3) If type (M) = type (L) = OBJECT
Then
NAME (X) RESULT-OBJECT [NAME (M), NAME(L), OP(op)]
REF(X) REF (M) U REF (L)
CG (X) LINK (X, M, L, op)

50. d) X M1 op1 L op2 M2
If type (L) = PROCESS and
type (M1) = type (M2) = OBJECT
Then
NAME (X) RESULT-NAME [NAME (L), NAME(M1), NAME (M2), OP(op1), OP (op2)]
REF(X) SET-REF (L, M1, M2, op1, op2)
CG (X) SELECTIVE-MERGE (L, M1, M2, op1, op2)

51. Semantic rules c and d make use of the following procedures:
Procedure LINK (X, M1, M2, op)
rel RELATION [NAME(M1), NAME (M2), OP(op)]
case rel of
<new> : if NAME (X)  ID then
- add the new composite object NAME(X) to ID
- return [OBJECT: NAME (X), REF(X)]
endif
<null> : return CG(M1)
else : return APPEND (CG(M1), CG(M2), rel)
end case
end LINK

52. Procedure SELECTIVE-MERGE (L, M1, M2, op1, op2)
C [EVENT:EVENT-NAME [NAME(L),
NAME(M1), NAME(M2), OP(op1), OP(op2)]
if OP(op1) = ‘null-op’ then G C
else rel LEFT-REL [NAME(L), NAME(M1),
NAME (M2), OP(op1), OP(op2)]
G1 APPEND (C, CG(M1), rel1)
endif
if OP(op2) = ‘null-op’ then G C
else rel2 RIGHT-REL [NAME(L), NAME(M1),
G2 APPEND (C, CG(M2), rel2)
return HEADER-MERGE (G1, G2)
end SELECTIVE-MERGE

53. Procedure SET-REF (L, M1, M2, op1, op2)
ref RESULT-REF [NAME(L), NAME(M1), NAME (M2), OP(op1), OP(op2)]
case ref of
<l-obj> : return REF(M1)
<r-obj> : return REF(M2)
<new-obj>: return NEXT-REF( )
end case
end SET-REF

54. The Icon Interpreter:
The system diagram of the icon interpreter is as follows: G0 ID B
Visual
Sentencce S
Construction
of the
Logical part
Pattern
Analysis
Syntax
Analysis
Meaning of S CG
Pattern String of S
Parse Tree of S

55. The module ”Pattern Analysis" transforms the visual sentence into pattern string .
Example
The visual sentence
is represented by the pattern string
(char & char & char) + cross

56. The icon interpreter is logically divided into two phases:
1. The syntax analysis is accomplished by a parsing algorithm for context-free grammars.
2. Constructing the meaning of the given visual sentence by evaluating the attributes of the nonterminal on the root of the parse tree using the ATTRIBUTE_EVALUATOR procedure
The attribute evaluation is accomplished by means of a postorder visit of the parse tree t of the visual sentence S

57. Procedure ATTRIBUTE-EVALUATOR (X).
if X is a nonterminal in Go
then /* let p be the production applied to the node X in t, and let 1, ..., np be the indices of nonterminals in the right-hand side of p*/
for j = 1 to np do
call ATTRIBUTE-EVALUATOR (jth nonterminal son of X in t)
end for
apply the semantic rules associated to p so as to evaluate the attributes of X
endif
end ATTRIBUTE-EVALUATOR