Dr. Lev Manovich Director, Software Studies Initiative, Calit2 + UCSD Professor, Visual Arts Department with Dr. Jeremy Douglass Post-doctoral researcher, Software Studies Initiative, Calit2 + UCSD You can capture this lecture using any media and share it. You can also download this softwarestudies.com presented on June 22, 2009 as Humanities 2009 Cultural Analytics: theory, methodology, practice

PRESENTATION CONTENTS: analysis and visualization of large cultural data: - theoretical implications - interfaces / visualization techniques - methods for analysis of visual media and born- digital culture

Our research is made possible by the support from: NEH Office of Digital Humanities Singapore Ministry of Education California Institute for Telecommunication and Information (Calit2), UCSD Division Center for Research in Computing and the Arts (CRCA) UCSD Visual Arts Department UCDARnet UCHRI

Software Studies Initiative Researchers: Lev Manovich | Director Noah Wardrip-Fruin | Associate Director Jeremy Douglass | Postdoctoral Researcher Cicero Silva | Software Studies Brazil William Huber | Researcher | PhD student, UCSD Chanda Carey | Researcher | PhD student, UCSD Daniel Rehn | Researcher | MFA student, UCSD

Software Studies Initiative Collaborators: Yuri Tsivian, Department of Art History, University of Chicago: cinemetrics.lv | film analysis Adele Eisenstein: Digital Formalism project (Department for Theatre, Film and Media Studies (TFM), Vienna University; the Austrian Film Museum; Interactive Media Systems Group, Vienna University of Technology) | film analysis Isabel Galhano Rodrigues, University of Porto, Portugal | gesture analysis David Kirsh, Cognitive Science, UCSD | dance video analysis Jim Hollan, Cognitive Science, UCSD | visualization | cultural analytics software Falko Kuester, Structural Engineering, UCSD + Calit2 | visualization | cultural analytics software Yoav Freund, Computer Science and Engineering, UCSD | machine vision | cultural analytics software Kay OHalloran, Multimodal Analysis Lab, National University of Singapore | Mapping Asian Cultures project Giorgos Cheliotis: Communication and New Media, National University of Singapore | Mapping Asian Cultures project Matthew Fuller: Goldsmiths College, University of London | software studies Benjamin H. Bratton: Visual Arts, UCSD + Calit2 | software studies

BACKGROUND

The joint availability of large cultural data sets (the exponential growth of user generated content, digitization efforts by museums and libraries, digital traces and self-presentation ), cultural information (web presence of all professional cultural agents), the tools already employed in the sciences to analyze and visualize big data, and the techniques developed in new media art makes feasible new methodologies for the study, teaching and presentation of cultural processes and artifacts - including contemporary cultural production, sharing and consumption.

fMRI of a global cultural brain today neuroscience combines single neuron recordings, tracking activities of neural networks (1mm) and neural maps (1cm), fMRI of the neural activity of the whole brain, and other methods. Traditional analysis of culture can be compared to recording and analyzing activity of a single cell or a small cell population. We need to start tracking, analyzing and visualizing larger cultural structures - (including their connectivity and dynamics over space and time) - equivalents of neural networks, maps, cortical columns, and the whole brain. Applying other techniques from neuroscience such as staining cells. Applying the basic method of contemporary neuroscience: combining results from different research methods which measure cultural processes at diff. resolutions (MRI, PET, staining cells, genomics, etc.)

new scale: 1. digitization of existing cultural assets 2. web presence 3. social media 4. cultural globalization

The rapid growth of professional educational and cultural institutions in many newly globalized countries along with the instant availability of cultural news over the web, the availability of software tools, and cheap travel has also dramatically increased the number of "culture professionals" who participate in global cultural production and discussions. Hundreds of thousands of students, artists, designers have now access to the same ideas, information and tools. In many cases, it is no longer possible to talk about centers and provinces. The students, culture professionals, and governments in newly globalized countries are often more ready to embrace latest ideas than their equivalents in "old centers" of world culture.

growth of a global culture space after Example: Fashion Weeks, 2005.

growth of a global culture space after Example: cumulative number of new art biennales,

world heat map made up from 35 million Flickr geo-coded photos).

NEW SCIENCE OF CULTURE? Until now, the study of human beings/cultural processes relied on two types of data: shallow data about many people/objects (statistics, sociology) or deep data about a few people/objects (psychology, psychoanalysis, ethnography, thick description, humanities). The development of high performance computing, mobile devices, web and social media allows for a fundamentally new methodology for the study of human beings and society: We can now collect detailed data about very large numbers of people/objects/cultural processes. We no longer have to choose between size and depth. (Example: reality mining project at MIT; Citysense; Blogpulse.) Software, hardware, and networks people use can capture detailed data about some dimensions of peoples cultural behavior, imagination and thinking (creating, remixing, reusing, communicating, etc.)

CULTURAL ANALYTICS

Why Cultural Analytics? existing terms: knowledge discovery from data to knowledge Google Analytics web analytics business analytics visual analytics = the science of analytical reasoning facilitated by interactive visual interfaces

Our goals: - being able to better represent the complexity, diversity, variability, and uniqueness of cultural processes and artifacts than the current cultural technologies allow for - democratize cultural research by creating sets of open-source tools for cultural analysis and visualization (in particular, study of visual and interactive media) - develop techniques to describe the dimensions of cultural artifacts and cultural processes which until now received little or no attention - create much more inclusive cultural histories and analysis - ideally taking into account all available cultural objects created in particular cultural area or time period (art history without names) - create representations and interfaces for visual exploration which operate across multiple scales - from details of structure of a particular individual cultural artifact/processes to massive cultural data sets/flows

INTERFACES

Platform for Cultural Analytics research environment: HIperSpace (287 megapixels).

What kind of interface do we need to create situational awareness for cultural analysts? The interfaces used today whenever a person/group monitors a performance of system/machine/process, makes decisions and controls (i.e., a human-machine system) all share the common principle: - multiple displays which present information about a system/process using diff. visual techniques. Examples: vehicle interfaces, patient monitoring in a hospital, control room of a plant, information dashboards, financial news.* Culture is a complex system / process / environment - therefore we should use similar interface design for studying and monitoring it.

Text Example of an existing interface: Barcos iCommand

Example of an existing interface: AT&T control center (2001).

Cultural Analytics research environment: interface mockup. Focus: geo map.

IDEOLOGY OF DIGITAL TRACES. Not all cultural and social activities leave digital traces on the web. A significant part of todays global culture is digitally invisible. Therefore, we cant only do projects which rely on web data or existing databases - we also need to take on digitally invisible cultural activities using available and original etnographic research. Example: envisioned analysis of the development of shanty towns in Mexico city (using research of the local architects working with these communities.)

Cultural Analytics research environment: interface mockup. Focus: long tail.

Example of long tail visualization: looks on lookbook.nu (sorted by number of hypes). Created by softwarestudies.com

Example of long tail visualization: looks titles on lookbook.nu (sorted by number of hypes). Created by softwarestudies.com.

Close-up of visualization of looks titles on lookbook.nu (sorted by number of hypes). Created by softwarestudies.com.

Map of Science visualization

Cultural Analytics research environment: interface mockup. Focus: relationships map.

Cultural Analytics research environment: interface mockup running on HIPerWall

The aim pursued with visual exploration is to give an overview of the data and to allow users to interactively browse through different portions of the data. In this scenario users have no or only vague hypotheses about the data; their aim is to find some. In this sense, visual exploration can be understood as an undirected search for relevant information within the data. To support users in the search process, a high degree of interactivity must be a key feature of visual exploration techniques. Christian Tominski, Event-Based Visualization for User-Centered Visual Analysis, PhD Thesis, Institute for Computer Science, Department of Computer Science and Electrical Engineering, University of Rostock, Exploration denotes an undirected search for interesting features in a data set. Kreuseler, M., Nocke, T., and Schumann, H. A History Mechanism for Visual Data Mining. In Proceedings of the IEEE Symposium on information Visualization (infovis'04) - Volume 00 (October , 2004). INFOVIS. IEEE Computer Society, Washington, DC, source: INTERFACES which combine media browsing and visualization to enable visual exploration of data

Cultural Analytics software running on HIPerSpace (May 2009)

GRAPHS which consist from the actual media objects

VISUALIZATION of a structure of a cultural artifact (borrowing and extending techniques from new media art and artistic visualization)

Visualization of complete text of Hamlet. Source: project Guthenberg.

Visualization of complete text of Hamlet. Lines of text are rendered as solid rectangles.

Visualization of complete text of Hamlet. Hamlet = red, King = black.

NBC TV news montage. data: 2 shows per year,

Representation of video structure: Betty Boop cartoon (left) vs. music videos (center and right)

THEORETICA L ISSUES

SOME THEORETICAL ISSUES AROUND CULTURAL DATA MINING / CULTURE VIS: Culture does not equate cultural artifacts. How can we automatically analyze context in a meaningful way? If cultural process/activity more important than the outputs being produced, how to visualize it? Statistical paradigm (using a sample) vs. data mining paradigm (analyzing the complete population). Modernity/normal distribution vs. Software Society/power law. Pattern as a new epistemological object. From meaning to pattern: humanities have been focused on interpreting the meanings of a cultural artifact/process. Today we can easily uncover the meanings of each cultural artifact - but we dont know the larger patterns they form. The new scale of culture points toward a pattern as a new unit of analysis (because we can not afford to consider meanings of every single artifact.) New digital divide - between social and cultural activities/people which leave digital traces and whose that do not. From small number of genres to multi-dimensional space of features where we can look for clusters and patterns.

Learning from a Search Engine: size: cultural criticism: very small samples of cultural production a search engine: every accessible web document categorization: cultural criticism: cultural objects are placed into small number of genres/categories a search engine: analysis of each web document to generate its unique description (not reducing to small number of categories) links: cultural criticism: analysis of some links (influences) between a given object and others a search engine: systematic consideration of all links between a given object and others features: cultural criticism: manual analysis of a small number of features of cultural objects a search engine: automatic analysis of the same (very large) set of features of web documents and links between them. Examples: Sense Networks attributes 487,500 dimensions to every place in a city. Google: PageRank reflects our view of the importance of web pages by considering more than 500 million variables and 2 billion terms. Our technology analyzes the full content of a page and factors in fonts, subdivisions and the precise location of each word. Sense Networks

Analysis of Born Digital Culture: 1. While patterns of interaction between readers and cultural objects/processes have always been a part of culture, recent technologies offers possibilities to capture and analyze such interactions. -> Big data has a new meaning in the case of interactive digital media. For example, we can deal with a single video game – but include in our analysis hundreds of hours of video which represent gameplay sessions by many users. 2. Because digital objects are driven by software, they have self-describing property - their structure and behavior are specified in their "software layer" (HTML code, project file of a AE composition, game engine, etc.) -> in addition to) analyzing the "surfaces" of digital objects (as we will do with the books or films), we also need to develop methods to analyze, summarize and visualize the structures and dynamics of "software layers" of these objects. 3. Media objects today are encountered by users/authors in a larger digital environment - the commands and interface of a desktop media viewer/authoring application, the commands and HCI of a social media site such as YouTube, etc. We need to develop methods for the analysis of digital environments - including functions and UI of desktop and mobile applications, webware, web sites, social media sites, etc.

4. Rather than existing in final fixed form as media inscriptions (film print, printed photo, printed book, etc.), digital objects are sets of possibilities. In the case of commercial interactive media, software specifies what can happen at every moment in interaction. This can take such simple form as a tree graph - for instance, in early games the player is given a choice to go left or right etc. The fields of HCI, interaction design and information architecture already developed a range of graphical and non-graphical techniques to describe the structures of possible interaction - so we can adopt some of them for the humanities, and also develop new ones. In the case of generative/procedural art and design, software specifies possible range of work "states." For example, a program which generates an abstract image implicitly specifies all possible images which it can produce - with each actual output being one instance from this sea of possibilities. (Think of typical works done with Processing.) In the case of time-based procedural media, such range of possibilities will change over time.

TECHNIQUES AND APPLICATION S

media: cinema view: macro data: average shot length stats on feature films from cinemetrics.lv

media: cinema and other types of moving images view: micro data: Dziga Vertov, A Man With a Movie Camera (1929); Betty Boop cartoons ( ); contemporary motion graphics. source: archive.org; designers web sites

Shot lengths and a distribution of particular shot type over time in A Man With a Movie Camera. Each bar corresponds to one shot. Bar height = shot length. Scale: complete film (68 minutes)

Shot lengths and distribution of different shot type over time in A Man With a Movie Camera (film length = 68 min). Each bar corresponds to one shot. Bar length = shot length. scale: complete film zoomed view

Combining manual annotations with automatic measurements or visual/spatial/temporal structure. Example: visualization of temporal patterns in A Man with a Movie Camera.

Comparing different temporal patterns on a micro-level in A Man With a Movie Camera. X axis = time. Y axis = brightness mean (dark grey line). Vertical bar = shot boundaries (light grey line). scale: 1 minute

Temporal patterns of low-level visual features in Common Go music video. Shown: brightness mean, brightness median, saturation median, hue median.

Comparative visualization: center of mass in Betty Boob cartoon (black) vs. music video (red). The center of mass for all frames are projected on top of each other.

Visualizations created by UCSD undergraduate students on HIPerSpace (June 2009)

media: visual art view: macro + micro data: 35 paintings by canonical artists representing transition from realism to modernism, source: artstor.org

FROM MAPPING TO SIMULATION

from visualization of cultural patterns to modeling and simulation to include all professional cultural content produced in the world

Studying cultural processes: from taxonomy to evolutionary biology We usually think of culture in terms of taxonomies - styles, genres, market segments - equivalent of biology in the 18th century. The exponential increases in the number of cultural objects (lifestyle products, films, photographs, songs, graphic designs, etc.) created by both professional producers and non-professionals - and the similar increase in the number of producers - create a new cultural universe. The sheer size of the new cultural universe, and its much higher connectivity[1] makes appropriate to apply scientific paradigms and methods (including mathematical models and computational modeling) used in evolutionary biology, environmental biology, genomics, bioinformatics, and other live sciences to study large-scale biological phenomena. [2] [2] Future collaboration with Calit2 Center for Algorithmic and Systems Biology (CASB)

[1] Higher cultural connectivity: globalization: access to the same ideas and tools + more people using English as a global cultural platform; large-scale globally distributed production in games industry, film industry, etc.; social media software which encourages remix and media mobility; desktop software production environment which similarly makes it very easy to borrow elements from other cultural products; increase in travel; access to cultural objects produced by everybody else (Youtube, Flickr, portfolio web sites of professional designers, etc.)

Mathematical simulations of global cultural flows based on the existing data? Blue Brain Markram's Brain and Mind Institute at the École Polytechnique (EPFL) in Lausanne Henry Markram anticipates that a simulation of a complete human brain down to the molecular level (based on all existing experimental neuroscience data) will be available before 2020.Henry MarkramÉcole PolytechniqueLausanne Henry Markram

Text a forest of neurons: a dye is injected into each neuron and then developed in order to reveal the morphology. Image by Blue Brain project.

If we do simulation-based research into global cultural flows, what kinds of models would we need? More complex than brain simulation? More simple? If simulation-based research is now being applied to new scientific areas, let us use it to study global cultural flows and dynamics. What new analytical categories can we develop if we aim for an equivalent of a Blue Brain project - simulating all of contemporary global cultural developments and relationships using all available data?

Thank you. follow Cultural Analytics at softwarestudies.com softwarestudies.com