Story World and Screen

Story World and Screen

EDWARD BRANIGAN

In chapter 2 of Narrative Comprehension and Film Branigan considers filmic narrative from the point of view of the spectators, analysing the way in which they transform the two-dimensional sequence of visual and aural effects on the screen into a three-dimensional world, which he calls the 'story world'. Grard Genette, in chapter 2 of his Narrative Discourse, had pointed out as a basic characteristic of all narratives -- whether aural, written or cinematographic -- the co-existence of a dual temporality: the time of the thing narrated, or story time (the time of the signified); and the time of the narrating (the time of the signifier). Building on this, Branigan proposes three dual frames of reference for narrative in film: the temporal, the spatial and the causal. Each of these frames, Branigan contends, is simultaneously at work on two levels -- on the screen and in the story world -- producing two fundamental, but often divergent, systems of space, time and causal interaction, which a theory of narrative must be able to analyse.

Branigan further distinguishes between the diegetic and the nondiegetic parts of the story world, that is, between the collection of sense data accessible to each character and those that are not accessible to any of the characters. As he explains, the spectator's organization of information into diegetic and nondiegetic story worlds is a critical step in the comprehension of a narrative and of the relationship of story events to our everyday world. Drawing on cognitive psychology, Branigan differentiates two kinds of perception processes in the spectator watching a film: 'top-down' and 'bottom-up' processing. These processes yield different and often conflicting interpretations of data which the spectator has to combine into a system. Branigan's contention is that the ongoing process of constructing and understanding a narrative consists of a 'moment by moment regulation of conflicts among competing spatial, temporal, and causal hypotheses'. He ends with the suggestion that the same kind of scheme may be successfully applied to the analysis of change in space. This suggestion remains undeveloped, however, as he reduces the notion of space to the binary opposition 'foreground/background'. Indeed, the main shortcoming of the analytical system he proposes is his simplistic, a priori assumption that 'virtually all phenomena [in the physical world] can be explained in terms of interactions between parts taken two at a time' (emphasis in the original).

Nevertheless, Branigan's efforts to systematize the spatial, temporal and causal frames of reference at work in filmic narrative and to tabulate the spectator's complex cognitive processes in the assimilation of on-screen data and of equivalent experiences which may not derive directly from screen constitute a productive alliance of narratology with cognitive psychology for the analysis of cinematographic narrative.

A preliminary delineation of narrative in film

Narrative in film rests on our ability to create a three-dimensional world out of a two-dimensional wash of light and dark. A bare facticity of graphics on the screen -- size, color, angle, line, shape, etc. -- must be transformed into an array of solid objects; and a texture of noise must be transformed into speech, music, and the sounds made by solid objects. Light and sound in narrative film are thus experienced in two ways: virtually unshaped on a screen as well as apparently moving within, reflecting and issuing from, a world which contains solid objects making sounds. Every basic spatial and temporal relationship, such as position and duration, thus has a double interpretation. A green circle might be seen to the left of a square in the same plane, or alternatively, it might seen to lie behind the square along a diagonal line to the left. In the latter interpretation, the circle may become a 'sphere,' the square a 'box,' and 'size' and 'color' will be adjusted according to our judgments about how distance and light are being represented in a given perspective system. Similarly, the green circle may appear for ten seconds on the screen but represent many hours of world time for the green sphere, especially if there is no other 'action' by which to gauge duration. Rudolf Arnheim asserts:

It is one of the most important formal qualities of film that every object that is reproduced appears simultaneously in two entirely different frames of reference, namely the two-dimensional and the three-dimensional, and that as one identical object it fulfills two different functions in the two contexts. ¹

The spectator, therefore, encounters at least two major frames of reference in film: the space and time of a screen as well as (a sample of) the space and time of a story world. ² More than space and time, however, is at stake. Causality also has a double interpretation. Changes in light and sound patterns will be perceived in at least two ways: as motion across a screen and as movement among objects in a story world. Causality on a screen will involve patterns of a purely visual, phenomenal logic where, for example, one blob smashes into another but the resulting transformations in motion and color may not be analogous to the interactions of three-dimensional objects like billiard balls; the blobs may even 'pass through' each other on the screen. Bizarre pictorial compositions and animation are clear examples of on-screen causality. In short, light and sound create two fundamental systems of space, time, and causal interaction: on a screen and within a story world. One of the tasks of narrative is to reconcile these systems.

It seems likely, in fact, that more than two frames of reference are active in our comprehension of film. It has even been argued that there is a stage of visual processing located halfway between two and three dimensional perception which produces a 21/2 dimensional representation of space. ³ Clearly, major changes occur during the conversion from phenomenal appearances on the screen to functions in a story world. One of the essential tasks of a narrative theory is to specify the various stages through which we represent and comprehend a film as a narrative. On-screen patterns of light, sound, and motion do not denote and hence cannot be true or false; they are fully present and neither narrative nor fictive. Moreover, the time in which these patterns are present on the screen is determined initially by the film projector. By contrast, a story builds complex spatial and temporal contexts, makes references to things which are not present (and may not exist), and allows broad conclusions to be drawn about sequences of actions. Moreover, time in the world of the story may be quite different than the time of the projection of the film. For example, in Letter from an Unknown Woman ( Ophuls, 1948) screen time is ninety minutes while the story covers three hours of an early morning during which a letter is read, and the letter, in turn, dramatizes events spanning fifteen years at the turn of the century in the world of Vienna.

Many concepts have been proposed to help describe how on-screen data is transformed through various spatial, temporal, and causal schemes culminating in a perceived story world. The various stages have been described with concepts like script, set decoration, technology, technique, performance, material, shot, form, style, plot, diegesis, code, narration, and referent. Since nonnarrative ways of organizing data may coexist with narrative, one might also recognize a conflict among discursive schemes, an 'excess' within the story. ⁴ The processing of film data has an important effect on how a spectator feels about the conceptual structures which are ultimately constructed. Some of the metaphors offered by film theorists suggest that our comprehension of film proceeds only forward, one step at a time, and depends simply on local and immediate juxtapositions, but other metaphors are less restrictive. Rudolf Arnheim speaks of picture postcards in an album while Noel Burch speaks of picture postcards suspended in a void, 'radically autonomous'. ⁵ Early in his career Eisenstein argued that shots are perceived not next to each another in a horizontal or vertical chain, but on top of each other in collision. ⁶ Later he refined the idea to include layers of pictures 'rushing towards the spectator,' but not necessarily in a straight line. He proposed that film data might be perceived as arranged vertically in matrix form, exhibiting a multiplicity of criss-crossing relationships in an instant. ⁷ Finally, the psychologist Julian Hochberg mentions three types of perceptual analysis in film: simple summation, directional patterns, and cognitive maps. ⁸ [. . .]

It will be useful now to separate the concept of 'story world' into two parts: the diegetic and the nondiegetic. In talking about a 'story,' we often refer to certain events which surround a character, events which have already occurred, or might occur in a particular manner, in a certain sequence and time span, and so forth. We understand such events as occurring in a 'world' governed by a particular set of laws. I will refer to that imagined world as the diegesis. The spectator presumes that the laws of such a world allow many events to occur (whether or not we see them), contains many objects and characters, contains other stories about other persons, and indeed permits events to be organized and perceived in nonnarrative ways. The diegetic world extends beyond what is seen in a given shot and beyond even what is seen in the entire film, for we do not imagine that a character may only see and hear what we observe him or her seeing and hearing. The diegesis, then, is the implied spatial, temporal, and causal system of a character -- a collection of sense data which is represented as being at least potentially accessible to a character. ⁹ A sound in a film, for example, is diegetic if the spectator judges that it has been, or could have been, heard by a character. However some on-screen elements (e.g., 'mood' music) are nondiegetic and addressed only to the spectator. These elements are about the diegetic world of a character and are meant to aid the spectator in organizing and interpreting that world and its events. Nondiegetic elements are not accessible to any of the characters. The spectator's organization of information into diegetic and nondiegetic story worlds is a critical step in the comprehension of a narrative and in understanding the relationship of story events to our everyday world.Let us now attempt a preliminary delineation of narrative in film. This definition will aid us in examining narrative comprehension more precisely and will also provide a basis for outlining five recent types of narrative theory, each of which stresses and interprets a different aspect of the narrative process.

Narrative is a way of comprehending space, time, and causality. Since in film there are at least two important frames of reference for understanding space, time, and causality, narrative in film is the principle by which data is converted from the frame of the screen into a diegesis -- a world -- that frames a particular story, or sequence of actions, in that world; equally, it is the principle by which data is converted from story onto screen. To facilitate analysis, narrative may be divided into a series of relationships. For example:

[. . . ] First we must examine the relationship of diegesis and screen, namely, how is data on the screen transformed into a story world? In order to answer this question, a distinction will be made between two types of perception operative in watching a film. These two types of perception produce different kinds of hypotheses about space, time, and causality. Distinguishing between them will allow us to examine closely how a spectator makes separate use of judgments about space, time, and causality, as well as how a spectator may integrate these judgments to produce an overall narrative rendering of experience.

Top-down perception

The movement from screen to story world does not proceed along a smooth path and in only one direction. Many of our abilities are brought to bear simultaneously on a film, producing at least some conflict and uncertainty. As a first step toward unravelling some of these abilities and specifying the kinds of conflicts that arise, I will use a fundamental cognitive psychological distinction to divide perception into two kinds of process according to the 'direction' in which they work. Some perceptual processes operate upon data on the screen in a direct, 'bottom-up' manner by examining the data in very brief periods of time (utilizing little or no associated memory) and organizing it automatically into such features as edge, color, depth, motion, aural pitch, and so on. Bottom-up perception is serial and 'data-driven,' and produces only short-range effects. Other perceptual processes, however, are based on acquired knowledge and schemas, are not constrained by stimulus time, and work 'top-down' on the data, using a spectator's expectations and goals as principles of organization. Top-down processes are indirect in the sense that they may reframe data in alternative ways independently of the stimulus conditions which govern the initial appearance of the data. Top-down processes must be flexible and general in order to be effective across a wide range of situations while allowing for (unpredictable) variations among specific cases. Top-down processes often treat data as an inductive sample to be projected and tested within a variety of parallel frames of reference while bottom-up processes are highly specialized and atomistic (e.g., detecting motion). Both kinds of process operate simultaneously on the data creating a variety of representations with varying degrees of compatibility. ¹⁰

Because top-down processes are active in watching a film, a spectator's cognitive activity is not restricted to the particular moment being viewed in a film. Instead the spectator is able to move forward and backward through screen data in order to experiment with a variety of syntactical, semantic, and referential hypotheses; as Ian Jarvie notes, 'We cannot see movies without thinking about them.' ¹¹ By experimenting with various methods for ordering data, the spectator creates spatial, temporal, and causal experiences which do not derive directly from screen time. Also critical in top-down processing are procedures which test the degree of 'progress' which has been made toward solving a perceptual problem. Such procedures are active, for example, when we search for the 'end' of a story. If we are unable to detect progress, we may begin to doubt the particular techniques we have been using, or even whether we have properly understood the goal. Because of the diversity of top-down and bottom-up processes which may be at work at a given moment in a text, perception as a whole is perhaps best thought of as a system which struggles to manage different and often conflicting interpretations of data.

In addition, the fact that comprehension may be divided into topdown and bottom-up kinds of activity helps explain some inconsistencies in the terminology employed by film writers. For example, some writers prefer to use the concept of 'voice' in film as a means to identify the source of words that are actually heard by a spectator while other writers prefer to apply the concept more broadly in order to include a number of top-down factors that influence a spectator's perception. Bill Nichols argues for an expansive notion of 'voice.'

[I]n the evolution of documentary [as a genre] the contestation among forms has centered on the question of 'voice.' By 'voice' I mean something narrower than style: that which conveys to us a sense of a text's social point of view, of how it is speaking to us and how it is organizing the materials it is presenting to us. In this sense 'voice' is not restricted to any one code or feature, such as dialogue or spoken commentary. Voice is perhaps akin to that intangible, moire-like pattern formed by the unique interaction of all a film's codes, and it applies to all modes of docummtary. ¹²

For Nichols, the concept of 'voice' is not confined to words literally spoken, or written, nor confined to fictional narrative; instead, 'voice' includes powerful, nonverbal patterns even in nonfiction (documentary) films. Accordingly, the 'person' whose voice is 'heard' in a text may be a much more complex (invisible and inaudible) entity than a voice-over narrator or someone being interviewed. Thus Nichols's approach is well-suited to an analysis of narration. [. . . ] Film narration cannot be limited to, say, an explicit commentary, or defined by literal, material, purely formal, stylistic, technical, technological, or 'bottom-up' kinds of categories. Narration, and narrative, are preeminently top-down phenomena that require for their analysis the use of wide-ranging, complex concepts like 'point of view,' or Nichols's 'social point of view.'

When we think of narrative as a general phenomenon that may appear in many physical forms (conversation, pictures, dance, music, etc.), we are thinking of it as a top-down cognitive effect. Wallace Martin may put it too strongly when he says that 'narratives may be the source of the varied visual resources of the movies, rather than vice versa.' ¹³ Nevertheless, much can be learned by concentrating on top-down processes in an attempt to isolate the psychological conditions that allow narrative to be understood in all media.

I wish to examine some of the top-down processes which seem to be relevant to our comprehension of narrative. I will begin by considering how our top-down search for a coherent causal system helps to organize screen data into diegetic and nondiegetic story worlds, each with a coherent temporal system. Later in the chapter, I will consider how judgments about screen space are related to judgments about story space and the causality of a story world. In general, we will discover that conflicts arise between top-down and bottom-up processing, between story and screen, and between the diegesis and what seems external to it. Hence we will find that the ongoing process of constructing and understanding a narrative is perhaps best seen as the moment by moment regulation of conflicts among competing spatial, temporal, and causal hypotheses. 'Narrative' in film is therefore the overall process as well as the result of searching among hypotheses for an equilibrium, however precarious.

Temporal and spatial order

There is a sequence in The Lady from Shanghai ( Welles, 1948) where three distinct actions are intercut through fifteen shots in such a way that it appears that when a woman presses a button, a door flies open allowing a dying man to drag himself into a room; when she presses the button again, a car is sent speeding down a road as a truck pulls up to a stop sign; and, when she presses the button a final time, the car is sent crashing into the back of the truck as the two men in the car react with horror at their helpless condition. The problem for the spectator of this film is how to interpret these events which can have no causal connection and yet are presented as if they were causally connected so that it seems that pushing a button brings a dying man into a room and creates a car accident. ¹⁴ In effect, we are being asked to accept a special fiction ('as if') within an already fictional mystery story.

In order to solve this causal problem the spectator must evaluate the temporal relationships posed by the sequence. Four important principles of causal reasoning are that a cause must precede an effect, an effect cannot work backward in time to create a cause, certain patterns of repetition among events make a causal connection more likely (e.g., pushing a button three times . . .) and a prior event which is temporally or spatially more proximate to the outcome than others is more likely to be a cause of the outcome. ¹⁵ Many different sorts of temporal situations, bearing on our judgments of causality, may be created through the juxtaposition of spatial fragments from different shots. As Arnheim emphasized, 'the fact that two sequences follow each other on the screen does not indicate in itself that they should be understood as following each other in time.' ¹⁶ Thus before tackling the causal problem, we must briefly survey some of the possible temporal, and spatial, situations which may arise.

An extraordinary fact about the physical world is that virtually all phenomena can be explained in terms of interactions between parts taken two at a time. According to Marvin Minsky, 'One could conceive of a universe in which whenever three stars formed an equilateral triangle, one of them would instantly disappear -- but virtually no three-part interactions have ever been observed in the physical world.' ¹⁷ I will assume that explanations of phenomena are constructed on this basis; specifically, that the spectator constructs temporal, spatial, and causal situations by assembling parts two at a time. Thus in figure 1 [p. 244 ], temporal situation AB ₁ in the story is created by imagining a particular relationship between the durations of two on-screen spaces, A and B, resulting in such story relationships as temporal continuity, ellipsis, overlap, simultaneity, reversal, or distortion.

More specifically, these temporal relationships in the story may be described as follows:

B ₁ represents the time of A as continuing into B such that the story order AB ₁ is presented as identical to the screen order AB.

B ₂ represents the time of A as continuing into B but with an initial ellipsis so that the screen order is interpreted as having omitted something from the story (which must be restored by the spectator's imagination); that is, the true order is: A, X, B ₂, where X is not represented on the screen. If the ellipsis is large, but later disappears when completed by new screen events, then B ₂ is a flashforward.

B ₃ represents the time of A as continuing into B but only after an initial overlap in which there has been a partial replay of time already experienced in A.

B ₄ represents a complete overlap with the time of A so that story event B is understood to be simultaneous with story event A even though B is seen to occur after A on the screen; that is, story time overrides the literal order on the screen.

B ₅ represents an overlap with the time of A but with an initial brief jump back in time. This produces a fleeting but curious story time in which an effect (in A) has apparently been shown prior to its cause (in B). The spectator, in fact, is tempted to mentally reverse A and B ₅ (creating a relation like A and B3) in order to restore the forward arrow of time in which causes precede effects (i.e., prospective time). It is also possible that AB ₅ may require the spectator to imagine an even earlier time (e.g., B ₆ ) which is then taken as an explanation of A in the story -- an implicit flashback -while B5 continues to represent the 'present time' of A. Using Noel Burch's terms, I will refer to the AB ₅ type of story order as a retrospective or retroactive story time. ¹⁸ Our usual expectation is prospective time -- A and so B. Less usual is retrospective time -- A because of B.

Here is an example of retrospective time: Shot A shows an object from a certain position, but then shot B shows a person looking at the object from that previous position. In this way, we discover that the object we saw in A had already been seen by a character (and in fact, without knowing it at the time, we were seeing how

Figure 1 Story time Graphic display of several varieties of story time created as a spectator relocates the on-screen time of spatial fragment B relative to the onsceen time of spatial fragment A resulting in a new and imaginary temporal order in the story, relationship AB _n. Some of the new relationships that may be created include temporal continuity, ellipsis, overlap, simultaneity, reversal, and distortion. The general principles illustrated here for time may also be applied to describe the ordering of space into such patterns as chains, gaps, reversals, and distortions (see text).

that character saw the object). Thus with shot B we are forced to mentally readjust the order of events and reconceive shot A using the character as a new reference point, as a new condition for our seeing. We now conceive of the story event as composed of, first, a character who looks, followed by our view of what can be seen from that character's viewpoint. Part of shot B then comes to stand for either a literal, brief jump back in time, or else an approximation of what it would have looked like to have seen the character first looking. In any event, what is important is that the shots require the spectator to refigure the temporal scheme.

B ₆ represents a time prior to A -- a past time, or flashback, which requires the spectator to reorder story events and imagine other events which have been omitted and not seen on the screen between B ₆ and A.

B ₇ and B ₈ represent temporal distortions. The on-screen duration of B (with respect to A!) is radically altered in such a way that it is not immediately clear what relationship with A is appropriate. For example, the duration of B may be compressed or expanded by running the film at a new speed, showing it backwards, repeating

B, showing alternative takes, omitting frames by step printing, using freeze frames, and so forth. ¹⁹ In these situations A and B do not seem commensurate and hence we cannot immediately decide what story order is appropriate. Also included in these categories is indeterminable time. For example, in Jean-Luc Godard Weekend ( 1967) there is a shot (involving hippie-guerrillas) that is so carefully arranged that its time cannot be ascertained. In fact, the shot is a flashforward but it cannot be recognized as such until much later in the film when the event being depicted actually occurs! The shot is thus a retrospective, nonsubjective flashforward.

The above scheme is a way of talking generally about principles of temporal ordering. However, it also applies to principles of spatial ordering. Although my discussion of the particular causal problem of the car crash in The Lady from Shanghai will center on time, it should be evident that, in general, space is just as relevant to the solution of causal problems. Therefore I wish to indicate briefly how the above scheme may be interpreted as an overview of some spatial principles of ordering. The scheme is not meant to be restrictive but merely to provide a way of comparing various narrative theories each of which will use specialized terms to examine space and time in still finer detail.

In order to demonstrate how figure 1 may be applied to space, I will for the moment make an artificial, but simplifying assumption about space. I will assume that space comprises only two sectors: a foreground and a background. The question then becomes, how may we recognize a change in space? How may a given space 'connect to' and be related to another space to form a new ordered whole? For convenience I will also assume -- as in the case of time -- that the change is effected through the editing of shots even though the scheme applies to changes effected in other ways (e.g., through camera or character movement, sound, changes in lighting level). The result of these assumptions is that space may evolve in only three basic ways: a new background may appear with the old foreground; a new foreground may appear with the old background; or, both a new foreground and a new background may appear. In the first two cases, what is new is introduced in conjunction with what has already been seen (an old foreground, or else an old background). This means that spaces are being connected into a chain. In the third case (a new foreground and a new background) the relationship of the new space to the spaces which have already been seen is open and not yet defined; that is, there is a 'gap' of some type between new and old space. Such a gap is indicated in figure 1 by the gap between fragment A and fragment B ₂. On the other hand, fragments B ₁, B ₃, and B ₄ represent a new space which either adjoins, overlaps, or repeats an old space so as to compose a chain of spaces.

There is a special case of the open space (A-B ₂ ) which must be mentioned. When the new foreground is simply the old background and the new background is the old foreground, there has been no real change: foreground and background have simply been interchanged across the two shots. Space has been reversed, or mirrored between the shots. Fragments B ₅ and B ₆ are meant to represent this general class of reverse angles in film. A typical example is shot/reverse-shot editing which depicts a conversation between two characters by alternating shots taken over each character's shoulder. ²⁰

What can spatial 'reversals' like B ₅ and B ₆ mean in terms of the story world being created by the spectator from a series of on-screen spaces? David Bordwell offers a proposal:

Shot/reverse-shot editing helps make narration covert by creating the sense that no important scenographic space remains unaccounted for. If shot two shows the important material outside shot one, there is no spatial point we can assign to the narration; the narration is always elsewhere, outside this shot but never visible in the next. ²¹

In other words, when space is reversed we do not see a camera, sets, or technicians but only more diegetic space which seemingly is part of a consistent and unified group of spaces with no disturbing (causal) outside influences (e.g., by an 'author'). The new and larger space being represented through a reversal is an imaginary space -- a diegetic space of the characters that is seemingly like itself in every direction.

There are, of course, degrees and kinds of chains, gaps, and reversals of space; and our recognition of the kinds will depend on the nature of other conventions governing, say, camera placement (for example, whether spaces are oriented toward a 180 degree axis of action). Connecting screen spaces to a pattern of story space does not prohibit also using gaps (B ₂ and B ₆ ), or other distortions (cf. B ₇ and B ₈ ), to create a story space which is not the sum of spatial fragments on the screen. Such a gap between screen and story space leads to degrees and kinds of 'impossible' space; that is, to space which can not be justified as existing wholly within the diegesis. Impossible space leads to perceptual problems of a new kind that force the spectator to reconsider prior hypotheses about time and causality.

Notes

My discussion of this scene in The Lady from Shanghai is based on GEORGE M. WILSON'S suggestive comments in Narration in Light: Studies in Cinematic Point of View ( Baltimore: Johns Hopkins University Press, 1986), pp. 1-4, 10, 202-4. There are, however, many inaccuracies in Wilson's description of the shots and story events, including the number of shots and their order.