"Comic Books: An Evolving Multimodal Literacy"

About the Author

Taylor Quimby graduated from Keene State College in 2010 with an individualized major in Aesthetic Studies. He currently works as a board operator and afternoon host for New Hampshire Public Radio.  An avid reader of both comics and philosophy, Taylor also enjoys spending his time playing the ukulele, video games, and writing plays.  Taylor lives in Hooksett, New Hampshire, with his family.

Contents

Introduction

Defining the Comic

The Speech Balloon...

Consider the speech...

Literacy and content

Literacy events    

Other Representations

Conclusion

Works Cited

2. The Speech Balloon, the panel, and multimodality

Have you ever tried to describe a funny comic strip to a friend, only to discover that the joke loses its humor when read aloud?  In order to convey the meaning of the strip, you cannot simply read the speech balloons, you also have to describe the characters and relate the peripheral action.  These log-jammed retellings tend to ruin the pace of the strip.

The reason comics don’t translate verbally has everything to do with how they communicate information.  You don’t just read the words; you also read the image.  James Paul Gee uses textbooks as an example of how word/image texts are more complicated than we imagine.

In many modern textbooks images not only take up more space, they carry meanings that are independent of the words in the text.  If you can’t read the images, you will not be able to recover their meanings from the words in the text alone … In such multimodal texts (texts that mix words and images), then, the images often communicate different things from the words.  Further, the combination of the two modes communicates things that neither of the modes does separately. (17-18)

When a comic inserts dialogue (or any other symbolic language) into its visual narrative, it becomes a multimodal text.

Let’s look at an example (Fig.1).  In this strip from Calvin and Hobbes, Calvin complains about New Year’s resolutions to Hobbes while they are sledding.  The humor is not merely physical; it’s multimodal.  Calvin is seemingly unaware of his own fragile physicality, as we can see by the unconcerned expression on his face before they crash into the tree.  Hobbes, on the other hand, is rooted to the action.  As they pick themselves up from the frigid river in the last panel, Hobbes implies that perhaps Calvin should resolve to watch where he is going, but Calvin unequivocally rejects the notion that he has any faults.

[Fig. 1,  Excerpt from Homicidal Psycho Jungle Cat, A Calvin and Hobbes Collection by Bill Watterson]

Calvin's refusal to take part in the visual world around him prevents him from learning from his own philosophical discussion.  Calvin’s comedic hypocrisy in this strip has everything to do with the relationship between the visual and the linguistic. In the Calvin and Hobbes 10th Anniversary Collection, Bill Watterson describes how

Calvin’s wagon [or this case, his sled] is a simple device to add some physical comedy to the strip, and I most often use it when Calvin gets longwinded or philosophical.  I think the action lends a silly counterpoint to the text … Sometimes the wagon ride even acts as a visual metaphor for Calvin’s topic of discussion. (104)

Notice how Watterson first acknowledges the individual elements of the strip, but then describes how meaning is formed in the relationship between them.  This relationship is part of what Gee calls an internal design grammar, or, “the principles and patterns in terms of which one can recognize what is and what is not acceptable or typical content in a semiotic domain” (28).  A semiotic domain, Gee explains, is “an area or set of activities where people think, act, and value in certain ways” (19).  In this case the semiotic domain is comic books, and the visual/linguistic relationship is part of its unique design grammar. There are other multimodal relationships at work, too.