Make Me Think: Composing the Narrative Interface

“…when I look at a web page it should be self-evident. Obvious. Self-explanatory.” - Steve Krug (11)
“…why would we– as users as much as creators or writers–want our interactions with interfaces to be “unthinking” so that we have no sense of how the interface works on us, delimiting reading, writing, even thinking?” - Lori Emerson

Steve Krug’s Don’t Make Me Think is a popular, introductory text on usability and interface design. It is short, easy to digest and very useful in outlining the basic principles of organizing and displaying information. In my multimedia authoring classes, students build web design skills through a series of short assignments whose aim is readability, clarity and cohesiveness. Krug’s book is a good resource, especially for non-designers, on the norms of visual hierarchy and spatial navigation. By applying simple principles to their interfaces (isolation, proximity, contrast, repetition), many students get immediate gratification with the results and a subsequent desire to dig deeper and to challenge themselves with the next project. The ability to create beautiful, attention-grabbing and useable interfaces is personally rewarding, educationally motivating and professionally useful.

Where I run into pedagogical stumbling blocks with visual design principles is when I ask my students to create a narrative web site. Narrative implies a voice, a point of view, an authored assemblage of data from the continuum of experience. What I teach up until the narrative project, are best practices for looking like everything else on the web. I consider the narrative project as a chance for students to break free from many of the rules and norms of interface design that they learned in the first part of the class. It is my hope that they have internalized web authoring tools just enough to be able to think with the materiality of code; to see mark-up, design and programming as something more than efficient vehicles for content, but as integrated processes of creative writing that are continuous with text and image creation. However, most of these final projects, with a few exceptions, end up being familiar story forms framed inside familiar web interfaces. While elegant and easy to navigate, the interfaces are flat as narrative. It is not a question of having too high expectation for undergraduates learning new skills. It is a problem of properly defining and framing the act of creative composition to arrive at a narrative interface. The design principles that I have taught them have overtaken the rich soil of their imaginations; the soil where narrative art and narrative composition, I believe, find their roots.

Visual design heuristics, visible in modern advertising and in the interfaces all around us, have their roots in early 20th century experimentation with typography and layout. According to Johanna Drucker, in The Visible Word, these avant-garde practices by poets and artists would later be codified toward the corporate interests of modern graphic design. (1996, 238-243)

“What had begun , in the 1910s, as a vivid and exuberant exploration of the materiality of signification, became, by the end of the 1920s,...an ordering of visual graphics which caused that very materiality to efface itself, to disappear, under the style of a graphics whose very adjectival character –elegant, clean, streamlined, balanced, correct–betrayed its repressive force.” (1996, 239)

Today, a “well-designed” GUI uses principles of visual hierarchy to guide the eye to the content with as little friction as possible. Facebook’s usability (fast access to friends status updates) would suffer if each member expressed themselves with their own fonts, colors and layouts. It is not that design principles are, in themselves, bad. They are undeniably effective at grabbing and guiding attention. But design principles in the pursuit of uniformity, clarity, impersonality and invisibility can easily suppress other possibilities of communication. Attention, that precious commodity of the web, can also be lured by slowness, ambiguity, contingency and semantic instability. In even the most popular songs and movies, the delays and gaps in a semiotic surface can set thought in motion and into deeper levels of engagement. How do we translate those qualities that make us engage with linear narratives (movies, literary fiction, songs) into interface design? How do we conceive a poetics of the interface that makes room for ambiguity and excess alongside clarity and economy? In other words, how do we teach the composition of interfaces that “make us think”?

I approach the problem of composing a narrative interface from the inside out; that is, from the computational and narrative thinking that produces interface surfaces, rather than from the more conventional “outside” approach; working through visual design prototypes to “properly” contain and frame the content. Computational thinking and narrative thinking are overlapping, yet distinct process. We use computational thinking to plan our day efficiently, and narrative thinking to reflect on how the day actually went. Both require a decomposition of “the day” into small units (times and events) so as to extract patterns (sequences) that cohere around meaningful abstractions (goals, desires). A carefully planned work schedule, the temporal ordering of meetings, breaks and tasks, can go wrong with an unexpected storm, or traffic jam or encounter with an old friend. Narrative helps us make sense of this unpredictable flux of experience and, in turn, contributes, computationally, to the planning of the next day’s schedule. All narrative composition involves both narrative and computational thinking: What is my tale? And how am I going to tell it? The narrative composition of digital media, however, engages both processes directly in the materiality of the code.

In this paper, I will expand upon five simple “rules of thumb” for narrative interface composition: 1) improvise data collection, 2) amplify narrative patterns, 3) structure narrative absences, 4) narrativize interactions and 5) iterate all of the above. These are intended as simple guidelines for individuals and small-group collaborations using common, accessible digital tools (HTML5, CSS3 and Javascript). Though the term interface now includes many kinds of semiotic surfaces, I am most interested in two-dimensional screens with touch or mouse navigation as the threshold for engaging intimately with an authored digital object. That said, the following heuristics are in the interest of producing more affective and narratively effective interfaces, of any kind, by making computational thinking more intimately related to narrative thinking in the act of composition.

Improvise Data Collection

Storyboards of user interaction and interface design prototypes, the equivalents of a film script, are used to work out relationships of media elements so that the creation of the discrete content units–the text, design elements, images, video and audio–can find ultimate unity and coherence when they are put together. However, a more improvisational approach to this data collection keeps open and fluid the ultimate form of the interface. Improvising through the computational processes of iteration and decomposition, working on macro and micro elements in a productive feedback loop, is what determines the final interface design, not the other way around: structuring the content to fit a predetermined template. Structure becomes just a framework or constraint for the act of composition. The interface itself–its spatial arrangements, navigation and graphic elements–grows out of the processes, needs and desires of content generation.

In his two volumes of Poetics of Cinema, the late Chilean filmmaker Raoul Ruiz explores computational approaches to narrative cinema production. He argues that any film has in it many more potential films. The viewer generates these other films at the level of the edit, between what happened and what might happen next, from his or her storehouse of memory: personal memories, memories of other films and memories from the film being viewed. A conventional and narrative-driven filmmaker will seek to cancel out these invisible other films by reinforcing a seamless representation and continuity of action, pre-established in the script. Another kind of filmmaker, what Ruiz calls a “shamanic” filmmaker, works with combinatory procedures, shuffling small story units and patterns, amplifying multiplicities generated between shots (montage), in order to create for the viewer this proliferation of phantom films. (2005, 73-90) The composition of such a film must involve decomposition and improvisation. “This is not just a way of writing, but a way of filming. These combinations work better if they arise during the shoot.” (2005, 88-89). A film script provides a structure, necessary for producers, actors and crew, to begin a film production. However, in directing and editing the film, the original structure is decomposed into shots that are improvised within environmental constraints: the weather, the actors’ emotional states, technical limitations, the director’s whims. In a Ruiz film, these production contingencies are brought into the composition. (2007, 42-56) Improvisation is being responsive and open to changes at the micro level–the unexpected constraints and contingencies of any media or text production–and being willing to adjust or even abandon the governing macrostructure.

Remix, the improvised appropriation of media fragments, is so natural to the web experience that it hardly requires an explanation for the authoring of multimedia texts. We search, copy and paste all types of content for all types of communication. But when the goal is a narrative work, in which media elements must be collected and/or produced for a somewhat cohesive user experience, improvisation can give way to more fixed, goal-oriented procedures. Scripts or outlines provide helpful guidelines for linear sequencing, but narrative interfaces most often offer a spatial and atemporal arrangement that rely on the user filling in gaps, detecting patterns, making associations. Through juxtaposition, in the layout or in nonlinear navigation, narrative is accessed, traversed, discovered and interpreted. Narrative interface composition is, therefore, less about following a specific structure than about growing patterns around the montage of fragments. An improvised rather than schematic approach to the creation of these media fragments will allow the author to perform a reading of the collected material in the same way the user/reader will perform a reading of the finished work.

2. Amplify Narrative Patterns

Pattern recognition in computational thinking is the process of isolating similarities or common differences that will help make predictions or shortcuts in the solving of problems. In narrative thinking, patterns are worked out in character traits, relationships, settings and events by accentuating differences and parallels. Such narrative patterns not only reinforce the sense of unity of a text, but also make it a dynamic space for engagement.

Ian Bogost’s notion of “unit operations”, a helpful tool for the textual analysis of all kinds of narrative artifacts, can also be employed as a cognitive tool for composition, particularly composition that seeks to integrate computational and narrative thinking.

“Unit operations are modes of meaning-making that privilege discrete, disconnected actions over a deterministic, progressive systems...I contend that unit operations represent a shift away from system operations, although neither strategy is permanently detached from the other.” (2006, 113)

Unit operations–in a videogame, a novel or a movie–are the discrete instances of meaning that work dynamically throughout a narrative to create detectible patterns: navigation gestures, motifs, character traits and actions. System operations, on the other hand, are largely “static” blocks of meaning, culturally determined, that impose order on the parts that make up a narrative whole. In his “unit analysis” of the movie The Terminal, Bogost isolates various forms of waiting that texture the scenes and character stories. Despite its trite system operations, a man’s search for identity, the film “challenges the viewer to abstract... specific representations of waiting into general, individual units of meaning that the viewer naturally recombines with his or her own experience.” (2006, 336) Unit operations can aggregate into closed and static systems, but they also can create open and relational systems.

In conventional interface design, with an aim toward for clarity and invisibility, content elements are put into visual hierarchies, layout schemes, and navigation sequences, primarily following system operations. A unit-operation approach, in the service of narrative exploration, would be less concerned with familiar blocks of narrative meaning–character arcs, cause-and-effect chains–and instead distribute small units of meaning in repeating and varying patterns throughout the text.

The historical timeline–ubiquitous in books, museums, magazines and websites– works to spatialize temporal sequence in an orderly and accessible way. The dominating pattern, the way image and text boxes are uniformly displayed along a horizontal line, is invisible; that is, the timeline’s visual organization, artifice and abstraction is in the service of the content only. A conventional timeline is not only invisible on the macro-level, it essentially removes any productive patterning on the micro-level. Everything has its place and exists only in temporal relation to everything else. There are, of course, exceptions. Many digital artists have found innovative approaches to the timeline as a form. Jonathan Harris’s site The Whale Hunt combines a macro-level timeline with granular patterning and multiplicities at the micro-level. Through scroll, zoom and left/right navigation, the interface reveals itself slowly, in a temporal rhythm that combines the user’s eager exploration and the constraints imposed on that exploration. The content itself, individual photos of the hunt, is both globally revealed and locally concealed. At the macro-level the interface give a sense of the quantity of data, but the photos are blurred. At the micro-level, the moving slideshow of photos, only one image is fully accessible at a time. Multiple timelines of varying visual scale are layered on top of each other. User-selected filters can restrict the timeline to cast (individuals), concepts (themes), context (settings), and cadence (speeds). The site amplifies these unit operations and relational patterns, creates difference and repetition, with the help of visual design principles: proportion, contrast, color harmony, proximity and isolation. But these design principles are in the service of narrative discovery, rather than organizational efficiency.

3. Structure Narrative Absences

Once a programmer has decomposed a problem or task into its component parts and identified patterns of similarities and common differences, then it becomes necessary to filter out unnecessary information and to generalize the patterns into abstractions, such as variables, so that algorithms can be designed. Narrative creation also relies on a similar honing and pruning so that generalized patterns, abstractions such as tags (hero, villain), schemas (goals, obstacles) and navigation instructions (genre), can emerge in a text. But narrative also filters information, reveals and conceals data, in order to activate a field of desire for the “user.” Narratives are full of gaps and ellipses that create openings for reflection and imaginative engagement. In a linear narrative, absences in the plot may generate the user’s associations, visualizations of alternatives, speculations, deductions, decodings on the outcomes of cause-and-effect chains. Narrative absences transform a mere series of selected details into a story-network that is always more than the sum of its parts. How can an interface, to a nonlinear field of data, structure such narrative absences?

The interface, being the site of narrative exploration, must present some of these narrative absences visually; that is, in the graphic design of the semiotic surface. White-space, the “empty” areas in visual design, does much to structure content, create hierarchies of content blocks through isolation and repetition. In Understanding Comics, an important text for the study and practice of visual narrative design, Scott McCloud extends his insights on the role of “the gutter” in comics, the empty areas between panels of content, to the role of absence in all visual narratives. “Closure” is the cognitive and perceptual effort by a reader/viewer to make a whole out of selected fragments. This visual principle is what allows us to “understand” complex narrative meaning in comics and movies. (1994, 60-93) But, there are other ways, beyond the visual juxtaposition of panels, in which narrative absence can be structure across the nonlinear field of an interface.

Important Artifacts and Personal Property from the Collection of Lenore Doolan and Harold Morris, Including Books, Street Fashion, and Jewelry by Shapton, Leanne is a love story in the form of an auction catalog. The novel’s title helps to frame the narrative structure, but the reader explores the collection with distributed attention, not really knowing what to look for. Through browsing, the reader becomes aware that something pushed the lovers apart and that the clue might be embedded in the object details and accompanying notes. The pages present an interface to story elements through visual repetition, making it easy to compare and contrast micro-narratives about the objects in the auction. These micro-narratives reveal patterns and relations around what the couple valued together, what they valued separately and what they thought or desired their partner to value. What is absent is the narration of the love plot itself. It is, rather, the material remains of the love plot that opens a reflective space for readers to navigate and narrate their own paths. Structuring narrative absence is a natural part of any narrative design, but the absences in a narrative interface might be approached differently. How might absence, the desire for narrative closure and meaning, be structured around the user’s navigation as well as in the juxtaposition of content fragments?

Narrativize Interactions

A frame narrative is a common literary technique whereby one story embeds one or more other stories to, for example, reinforce themes, question the reliability of the narration or reflect on the act of narration itself. There are many types and uses of literary frame narratives, but digital narratives often rely on frame narratives as a practical way to hold together, thematically or by a common story thread, a non-linear collection of micro-narratives. Such a frame narrative will typically present a story problem or question that the other nested narratives will illuminate in some way.

An interface not only frames “content”, but also provides an entry-point, context and desire for deeper levels of exploration and interaction. Icons such as tabs, buttons, folders, sliders, trashcans are familiar metaphors for simple interactions. Analogical spaces, representations of real world settings such as the leaves of a book, a room with drawers and closets, a private console with buttons and information panels, can provide context for more complex, situational interactions. A narrative interface can also use a “story-shell”–an introductory backstory or event that is visualized in the interface–to provide both a setting and frame for user motivation through the text. The familiar interfaces of browsers, blogs, social network feeds and personal desktops can act as “stage sets” for dramas made of emails, messages, web searches, photo galleries and video chat. All of these representational approaches to the interface can be effective in preparing a path for user engagement, but the interfaces are mostly treated as backdrops rather than as narrative elements in themselves. Once the metaphor or story-shell is understood, the interface effectively disappears. One rarely is encouraged to think deeply about an interface in the same way one would think deeply about Humbert Humbert’s convoluted narration of his obsession with Lolita.

Instead of using a story-shell or analogical setting as the framing device, why not appropriate the abstractions (the navigation gestures, the layout patterns and graphic elements) of interfaces as a starting point for generating a digital narrative? Consider a simple swipe as a user action on a tablet. If a user swipes left, it is assumed (from conventions of book reading) that there will be movement along a horizontal line from left to right, either as a continuous scroll or as segmented “pages.” Such an interface, even without content, is familiar enough as a setting for linear story progression. A swipe right, along a horizontal line that moves left-to-right, might, for western readers at least, imply a movement backwards in time and suggest remembrance. But what if this swipe backwards changed or altered the content–held in the user’s memory– of what had once been there before the swipe forward? What narrative effect might that produce? A vertical swipe, pulling down or up a content panel, across the horizontal line, might be taken as an associative or poetic gesture, outside of time. Familiar navigation gestures behaving unexpectedly can contain narrative information. This is not to say that there are fixed meanings to navigation gestures or layouts, nor that interfaces should somehow be focalized (present a perspective of narration), only that interaction as a system can transform the user’s orientation, interpretation and affective reception of the work. In computational thinking, algorithm design, the step-by-step instructions to complete a task of solve a problem, requires a certain level of abstraction in the form of variables and conditionals. Narrative generation similarly makes use of variables, character types and plot schemas, to activate meaning. To view interface interactions as abstractions, metaphors for narrative thinking, rather than representations of familiar stories or settings, is to open new ways of approaching narrative composition. An interface then becomes integral to the narrative act, a modulated field or “voice” that must be considered along with the content.

Iterate All of the Above

The above rules or principles are not for isolated stages of composition; they are to be applied, as needed, at each and every stage, iteratively. Digital media arts still carry influences from analog production, particularly, film production. Adopting a more organic, iterative approach to multimedia authoring, means treating every element and process of creation as writing: decomposing fragments, finding patterns, playing with arrangements, removing what is redundant or unnecessary in ever-expanding loops of macro/micro activity. The work of digital composition, as with writing and programming, is a type of thinking/doing.

Conclusion: Towards Affective-Expressive Interfaces

What should I make my narrative site about? This is a question that plagues my students of multimedia authoring. It is a creative problem for any artist in any media, but with digital media the problem is compounded by the endless possibilities of inscription. What does a narrative do in this plentitude? A book or film has a codified structure; one page or scene comes after another. An interface can be navigated through clicks, taps, zooms, scrolls and body movements. The term “narrative” creates even more complication, because of the inherited forms in other storytelling media. Most students recognize a narrative as involving a sequence of events in time with one or more characters pursuing a goal; a framework that seems antithetical to a website with hyperlinks and multimedia. Analysis of narrative interfaces–games, electronic literature and online journalism–can certainly stimulate ideas; and digital media theory can shape a new understanding of narrative, but these don’t necessarily help students navigate their own subconscious processes and intuitions. For some students, theory can even be a hinderance by setting up models of thinking that are removed from their everyday lives. Theory can be most productive after students have spent some time making digital objects, when they can connect abstract ideas with direct, hands-on experience. The five heuristics I am suggesting are to push the question, “What should my project be about?” towards “What steps can I take to discover what my project will be about?” The first question involves selecting system operations - templates, schemas, themes - the second involves improvising and experimenting with unit operations - navigation gestures, image and text juxtapositions. If students engage narrative thinking with computational thinking in their compositions they may arrive at what Earl Miner calls “affective-expressive” narratives. Unlike Western mimetic narrative founded on drama, plot and Aristotle's "imitation of an action, Asian non-mimetic or affective-expressive narratives do not require plot to hold events together as they are often episodic "ensembles" of event, voice and image in which “the narrator and the reader and the object of attention take on their being in respect to each other.” (1983, 87-94) Affect in the narrative arts, the layered patterns and temporal rhythms in “the telling”, can sometimes give pleasure in itself without much consideration to the developing content or plot of “the tale.” The content may be nonsensical, obscure, banal or even disagreeable, but the semiotic surface can still maintain a sensual and/or cognitive fascination. Because it is made of the stuff of lived, bodily experience (materiality, thought, desire, event), narrative is uniquely suited to presenting the ephemerality, paradoxes and flux of that experience. Narrative frightens, soothes, puzzles, frustrates, bores, arouses, because of its affective powers. It can explore the limitations and validity of itself as a framing device and can even make narration itself disappear. Narrative interfaces that “make us think” in this way– that make the user conscious of absences, navigation abstractions, patterns in content fragments and their spatial arrangements and juxtapositions–are more affective-expressive, less impersonal and invisible, in the framing of a narrative experience.

Works Cited

Barthes, Roland, and Stephen Heath. (1978) Image - Music - Text: Essays. New York: Hill and Wang. Print.

Bogost, Ian. (2006) Unit Operations: An Approach to Videogame Criticism. Cambridge, Mass.: MIT Press. Kindle AZW file.

Drucker, Johanna. (1996) The Visible Word: Experimental Typography and Modern Art, 1909 - 1923 [...] [...]. Chicago [u.a.: Univ. of Chicago Press. Print.

Emerson, Lori. (2013). “Activist Media Poetics: Electronic Literature Against the Interface-free (MLA 2012) | Loriemerson.net.” Web. Harris, Jonathan. (2013)“The Whale Hunt / thewhalehunt.org / by Jonathan Harris.” Web.

Krug, Steve. (2009) Don’t Make Me Think: A Common Sense Approach to Web Usability (2nd Edition) (Voices That Matter). 2nd edition. Berkeley: New Riders. Print.

McCloud, Scott. (1994) Understanding Comics: The Invisible Art. New York: William Morrow. Print.

Miner, Earl. (1983) "The Grounds of Mimetic and Nonmimetic Art: The Western Sister Arts in a Japanese Mirror," in Articulate Images: The Sister Arts from Hogarth to Tennyson, ed. Richard Wendorf. Minneapolis: University of Minnesota Press. Print.

No Author (2013) Un“Google: Exploring Computational Thinking.” | google.com/edu/computational-thinking/what-is-ct.html Web. 23

Ruiz, Raúl, and Brian Holmes. (2005) Poetics of cinema. 1, 1,. Paris: Dis voir. Print.

--------. (2007) Poetics of cinema. 2 2. Paris: Dis voir. Print.

Shapton, Leanne. (2009) Important Artifacts and Personal Property from the Collection of Lenore Doolan and Harold Morris, Including Books, Street Fashion, and Jewelry. New York: Farrar, Straus, and Giroux. Kindle AZW file.