Inf 723 Information & Computing Jagdish S. Gangolly Interdisciplinary PhD Program in Information Science Department of Informatics, College of Computing & Information State University of New York at Albany 1/2/2019 Inf 723 Information & Computing (Gangolly)

Inf 723 Information & Computing (Gangolly) Information Reports Information context (factive, not truth-functional) Information content ( Information carrier Indicating fact 1/2/2019 Inf 723 Information & Computing (Gangolly)

Inf 723 Information & Computing (Gangolly) An example: The acoustic waves from the speaker carry the information that the announcer said, “Nancy Reagan is irritated.” 1/2/2019 Inf 723 Information & Computing (Gangolly)

Inf 723 Information & Computing (Gangolly) Information context: The acoustic waves from Information carrier: the speaker carry the information Information content: that the announcer said, Indicating fact: “Nancy Reagan is irritated.” 1/2/2019 Inf 723 Information & Computing (Gangolly)

Inf 723 Information & Computing (Gangolly) Principles Facts carry information. The informational content of a fact is a true proposition. The information a fact carries is relative to a constraint. The information a fact carries is not an intrinsic property of it. The informational content of a fact can concern remote things and situations. 1/2/2019 Inf 723 Information & Computing (Gangolly)

Inf 723 Information & Computing (Gangolly) Principles Informational content can be specific; the propositions that are informational contents can be about objects that are not part of the indicating fact. Indicating facts contain such information only relative to connecting facts; the information is incremental, given those facts. 1/2/2019 Inf 723 Information & Computing (Gangolly)

Inf 723 Information & Computing (Gangolly) Principles Many different facts, involving variations in objects, properties, relations and spatiotemporal locations, can indicate one and the same informational content—relative to the same or different constraints. Information can be stored and transmitted in a variety of forms. 1/2/2019 Inf 723 Information & Computing (Gangolly)

Inf 723 Information & Computing (Gangolly) Principles Having information is good; creatures whose behavior is guided or controlled by information (by their information carrying states) are more likely to succeed than those which are not so guided. 1/2/2019 Inf 723 Information & Computing (Gangolly)

Inf 723 Information & Computing (Gangolly) To give form to a message by moulding it into a shape or pattern that can be communicated. Measurement, Quantitative Meaning, Qualitative 1/2/2019 Inf 723 Information & Computing (Gangolly)

Inf 723 Information & Computing (Gangolly) Entropy Measure of information content Communications example Claude Shannon’s work 1/2/2019 Inf 723 Information & Computing (Gangolly)

Inf 723 Information & Computing (Gangolly) Entropy Consider the proposition “it will rain tomorrow”. Let p = Probability{It will rain tomorrow} The information content I of a message with probability of occurrence p is I = log (1/p) = -log p 1/2/2019 Inf 723 Information & Computing (Gangolly)

Inf 723 Information & Computing (Gangolly) Entropy I = log (1/p) = -log p If p = 0, I = infinity. In other words, if a priori you consider an even to be impossible, then the message that the event occurred has information content infinity, since you would be very surprised If p = 1, I = 0. In other words, if a priori you consider an even to be a certainty, then the information content of the message that it occurred is 0 since you are not surprised at all. 1/2/2019 Inf 723 Information & Computing (Gangolly)

Inf 723 Information & Computing (Gangolly) Entropy The expected information content of a certain message about the rain is given by: H(p) = p . log (1/p) + (1 - p) . log (1 - p) 1/2/2019 Inf 723 Information & Computing (Gangolly)

Inf 723 Information & Computing (Gangolly) Entropy If p = 0, then H(0) = 0 . log (1/0) + 1 . log (1/1) = 0 If p = 1, then H(1) = 1 . log (1/1) + 0 . log (0) = 0 If p = 1/2, then H(1/2) = 1/2 . log (2) + 1/2 . log (2) = 1 1/2/2019 Inf 723 Information & Computing (Gangolly)

Inf 723 Information & Computing (Gangolly) Entropy The information content is usually measured in bits which (take values 0 or 1), and therefore the logarithms in the formulae are to the base 2 However, one can measure such information content in other measurement units such as 10 (decimal) or e (natural). If it is measured in units of 10, they are called Hartleys, named after the statistician who developed the concept. 1/2/2019 Inf 723 Information & Computing (Gangolly)

Entropy - An Application in Coding Weather in California: sunny or cloudy Probabilities: P{sunny}=7/8, P{cloudy}=1/8 Weather reports for two days to be sent The weather on the two consequent days are independent, ie., P(w1 & w2) = P(w1) . P(w2) 1/2/2019 Inf 723 Information & Computing (Gangolly)

Entropy - An Application in Coding Suppose you use code alphabet for weather {0, 1} where 0 stands for sunny and 1 stands for cloudy. . If the two days weather information is to be sent in a message, the average length of the message code is 2 (00, 01, 10, or 11) bits, or 2 .(8/7) + 2 . (1/7) = 2 1/2/2019 Inf 723 Information & Computing (Gangolly)

Entropy - An Application in Coding The entropy of the message is H = (7/8) .log( 1/(7/8)) + (1/8) .log(1/(1/8)) = 0.54 bits The question is, if we can exploit the fact that the weather on two consecutive days is independent, in order to reduce the average code length og the message. This can be accomplished by the following code: 1/2/2019 Inf 723 Information & Computing (Gangolly)

Entropy - An Application in Coding Weather Probability Code SS (7/8) .(7/8)=49/64 SC (7/8) .(1/8)=7/64 10 CS (1/8) .(7/8)=7/64 110 CC (1/8) .(1/8)=1/64 11 1/2/2019 Inf 723 Information & Computing (Gangolly)

Entropy - An Application in Coding The average length of this code is given by 1/2/2019 Inf 723 Information & Computing (Gangolly)

Entropy - An Application in Coding Therefore, the average length of a daily report is L/2 = 0.5 .(86/64) = 0.68 By considering a larger sequence for the weather report, we have reduced the average length of the report closer to the entropy of 0.54. 1/2/2019 Inf 723 Information & Computing (Gangolly)

Inf 723 Information & Computing (Gangolly) Entropy In the transmission of messages over wires, one can achieve higher reliability by repeating each character in the message by repetition and polling. The greater the repetition the greater the reliability However, such repetition adds overheads to the payload message. At the limit, we can achieve perfect reliability by repeating the characters in the message infinite times, but the message except for the first character would never be sent. 1/2/2019 Inf 723 Information & Computing (Gangolly)

Inf 723 Information & Computing (Gangolly) Entropy Shannon showed that it was possible, by appropriate coding schemes, it is possible to achieve reliability without sacrificing efficiency This is accomplished by two strategies assigning shorter codes to highly likely events grouping events 1/2/2019 Inf 723 Information & Computing (Gangolly)

Forms of Information (Bates) Information is the pattern of organization of matter and energy. All information is natural information, in that it exists in the material world of matter and energy. Represented information is natural information that is encoded or embodied. 1/2/2019 Inf 723 Information & Computing (Gangolly)

Forms of Information (Bates) “an organizing mechanism which provides an ability to deal with the environment. It is a symbolic description having modes of interpreting and interacting with the environment” (Goonatilake) 1/2/2019 Inf 723 Information & Computing (Gangolly)

Forms of Information (Bates) Information flow lineages (Goonatilake) Genetic information Neural-cultural (Experienced, enacted, expressed) Exo-somatic (information stored outside the animal as the “externalization of memories”) (embedded, recorded) Residue (trace) 1/2/2019 Inf 723 Information & Computing (Gangolly)

Forms of Information (Bates) Information as a sign (semiotics) Sign, interpretation, reference to object 1/2/2019 Inf 723 Information & Computing (Gangolly)