Writing with Sound: Composing Multimodal, Long-Form Scholarship

Historically speaking, the fields of digital humanities and media studies have remained parallel at best, with the former anchored more in computational practices and technical competencies than the latter. However, recent work – such as Sharon Daniel’s ‘Public Secrets’ (2007), Kathleen Fitzpatrick’s Planned Obsolescence (2011), Matthew Kirschenbaum’s Mechanisms (2008), Kari Kraus’s ‘Conjectural Criticism’ (2009), and Lev Manovich’s cultural analytics (2012) – suggestively troubles this parallelism. From the vantage of media studies, digital humanities allows scholars to shift from commenting about new media and technologies to constructing arguments with and through them (McPherson 2009: 120). Informed by claims from experience and anchored in embodied acts of building, digital humanities arguments necessarily become ‘hands on,’ and scholarly distance from technologies no longer holds. Meanwhile, media studies investments in cultural criticism and situated knowledge-making are increasingly important to today’s digital humanities practitioners, involved such as they are in multimodal communication (e.g., interactive visualizations, geospatial representations, rich exhibits, and gaming). For instance, Alan Liu argues that ‘digital humanities should enter into fuller dialogue with the adjacent fields of new media studies and media archaeology so as to extend reflection on core instrumental technologies in cultural and historical directions’ (Liu 2012: 501, emphasis added). This fuller dialogue would enhance the field’s awareness of how work with technologies and data intersects with the relevant social, economic, and political issues of our time. Assuming those conversations are inevitable (and that media studies and digital humanities should continue to overlap and intersect), a key question thus emerges: what would be an appropriate platform for such forms of scholarly communication? How would it function, and under what assumptions about how digital humanities and media studies are practiced?

With these questions in mind, this talk uses the author’s ongoing book project (tentatively titled, How Text Lost Its Source: Magnetic Recording Cultures) to present his findings on the yet-to-be-released Scalar platform. Supported by the Andrew W. Mellon Foundation and the U.S. National Endowment for the Humanities, Scalar is designed for authoring and publishing multimodal books. Built using PHP ARC2 (a MySQL-based Semantic Web/RDF framework), HTML, CSS, jQuery, XLST, Dublin Core, and other ontologies including SIOC, it enables users to assemble media (e.g., audio files) from multiple sources and juxtapose them with their own writing. The platform particularly facilitates work with visual materials; however, it also lends itself to audio, which is central to the history articulated in How Text Lost Its Source. Throughout the talk, ‘multimodal’ is preferred over ‘multimedia’ because the term stresses systems of ‘sensory or perceptual experience’ over the ‘means of conveying [or storing] a representation’ (Anastopoulou, Baber & Sharples 2001). And in the particular case of a Scalar book, it assumes that: (1) attention behaviors such as reading, watching, and listening are not inherent to or determined by a medium, (2) digital communication frequently entails blending approaches to composition (e.g., through images, audio, video, text, and databases), (3) layers of digital content are materially distinct and yet function relationally, and (4) scholarly interpretation demands several sensory modes (e.g., listening closely, scanning, and clicking).

The author’s preliminary findings on Scalar show that, most importantly, it encourages reflexive approaches to the computational processes involved in digital humanities research. Influenced by the work of LaDona Knigge and Meghan Cope (2006) as well as Mei-Po Kwan (2002), the author organizes these findings according to how Scalar fosters: (1) exploratory research across multiple archives, (2) iterative and recursive argumentation, (3) an oscillation between abstract and concrete expressions, and (4) multiple interpretations of media and cultural history. Throughout, an emphasis is placed on juxtaposing critical writing with the critical use of audio in Scalar, including a few demonstrations of How Text Lost Its Source. 

In the case of exploratory research across multiple archives, the platform is persuasive because it affords composites of text, audio, video, and images drawn from the various histories of magnetic recording, enabling the author to represent audio across the material specificities of multiple media (rather than reducing audio to simply sound or text). It also allows historical evidence to be modeled and exhibited independently of the author’s writing (e.g., audiences can navigate all audio files collected for the history without reading the author’s interpretations). However, any given audio file in a Scalar book can be annotated through discrete, time-stamped commentary by the author, and this commentary is displayed within the medium’s own temporality. While such features are now typical in visual culture (e.g., annotating lexia in Commentpress or tagging an image in Flickr), few such mechanisms exist for the scholarly treatment of sound.

Additionally, Scalar facilitates iterative and recursive expression because it not only keeps a version history of all contributions and makes that history public; it also allows authors to duplicate media content (e.g., a magnetic tape recording) in another context and then re-evaluate it. Evidence in a scholarly argument thus becomes subject to constant re-visitation and re-use (e.g., the same tape recording is interpreted several times), underscoring the fact that media histories emerge from multiple (and often conflicting) perspectives, worldviews, and accounts. Importantly, these media – together with the author’s writing – are presented throughout a Scalar book both in the aggregate and in the particular. Using RDF/XML and D3.js dynamic visualizations, any single instance (i.e., anything with a URL) in the book can be situated in relation to the balance of the content. Consequently, the book’s historical materials are structured and expressed in such a way that they can be studied in isolation but cannot exist independently. And when a book is modeled effectively, its audiences can study complex cultural relationships (e.g., between two audio files) established by the author. Importantly, such relationships are frequently constructed and encoded based on what has not been written or recorded in media history. While RDF/XML and D3.js help scholars organize and convey it, no computational practices exist (as of yet) for mining or visualizing implicit content, especially in the case of sound.    

Finally, the Scalar platform excels at fostering various interpretations of media history by allowing audiences to – in the fashion of context-sensitive design – select how a book’s content is viewed. These views range from ‘text-only’ and ‘media-emphasis’ to a radial visualization, a force-directed graph, and a history browser. Again, this array of perspectives brushes against any totalizing account of media history, the cultural history of magnetic recording included. It also destabilizes a scholar’s authority over an audience’s interpretation as it allows them to arrange and re-arrange content. Such an emphasis on the constructedness of arguments is especially key in a moment when visualizations are gaining traction in the field. The dynamic juxtapositions of writing and media in Scalar – D3 visualizations among them – actively demonstrate how our data and content are taken, not given (Drucker 2011). They also facilitate robust and often contradictory accounts of material histories, where scholars can: (1) argue with and through the media they study; (2) present material relationally and discretely, either with or without commentary; (3) author within a medium’s temporality; (4) exhibit the otherwise ignored processes of research and revision; (5) duplicate and interpret media in multiple contexts; and (6) underscore the contingent character of historical artifacts, to such an extent that any complete description of a given media object is difficult to say the least.