Mnemotheque: An experiment in dynamic interactive memorial

Mnemotheque is a journey into collective memory where you are both director and audience.
You guide the retelling, and then relive the memories as they wash over you.

PREFACE

As anyone who has listened to a first-person account of the Holocaust will agree, hearing the history of this incredible tragedy directly from those who lived it has an impact that is far stronger than any book, film, or lecture. As the years pass and the survivors grow older, it becomes increasing difficult for succeeding generations to experience this intimacy.

In 1999, with this in mind, we set about to develop a platform for a dynamic, living interactive memorial to the Holocaust that would allow us to preserve and share this history for all time. The program is called The Mnemotheque.

INTERACTIVE MEMORY; THE MNEMOTHEQUE VISION

We designed the Mnemotheque in the hopes of allowing future generations to experience the history of the Holocaust, and more specifically, first-person accounts of the Shoah, from an intimate and uniquely visitor/user-centric perspective: a perspective that we believe enriches one's understanding of history and its unique context in a way that is sublimed by interactive media.

What makes the Mnemotheque experience different, and unique, is that we, the viewers, are invited to become an active participant in the retelling of our own history. We are not only rapt listeners of heart-rending holocaust testimonials but, via the Mnemotheque, we become explorers and directors of a living documentary, winding our own way through memory as we follow the thread of our own interests and emotions.

The Mnemotheque technology allows us to explore our collective memory through the individual memories of the people who have experienced History from their own unique point of view. Their stories (text, video, voice) are illustrated by other media (photos, sound, personal letters, etc.) assembled from archives of the appropriate period. The technology assembles these media dynamically based on an algorithm that is a model of memory itself, engendering a new form of narrative that is guided by the user/visitor.

A MNEMOTHEQUE INTERACTIVE LIVING MEMORIAL

A Mnemotheque Interactive Living Memorial could assume many different forms. As a web-based, permanent exhibit, visitors could access data assembling worldwide historical archives via interactive booths or multimedia installations at museums and/or from their homes around the globe. Another application could be developed for on-site "Virtual Reality" theaters in which individuals and/or groups could experience these intimate histories via an appropriately designed interface and equipment; Imagine moving through Memory by reaching out to caress the image of a child's face, or by virtually turning the pages of Anne Frank's diary.

The Mnemotheque Grinberg instantiates this vision in a dynamic, interactive documentary of our family's experience during The Shoah. The user - the "explorer" - guides the Mnemotheque along paths of interest through the photographs, texts, and video segments that record this history. At each step, the Mnemotheque dynamically composes Blocs Souvenir (memory units) from conceptually related media that span time, place, and person. Topic leads to topic, and a story unfolds through an interaction between the explorer and the testimony, as though related by a family elder. And each telling enriches our understanding of history and its unique context.

click to see full size of any image

The Mnemotheque enables the explorer to interactively focus on concepts, places, or events of interest, and to view these from the perspective of different witnesses. An appreciation of perspective is among the most important contributors to tolerance. By helping people grasp the richness of the points of view inherent in history, Mnemotheque-based exploration may help lead to greater mutual understanding and tolerance.

SPREADING ACTIVATION; THE MNEMOTHEQUE TECHNOLOGY

When Vannevar Bush envisioned "Memex", the predecessor of The Web, he proposed that information access could work like human memory, using "Selection by association, rather than by indexing" (Bush, 1945): "The human mind [...] operates by association. With one item in its grasp, it snaps instantly to the next that is suggested by the association of thoughts, in accordance with some intricate web of trails carried by the cells of the brain. [...] [T]he intricacy of trails, the detail of mental pictures, is awe-inspiring beyond all else in nature."

Cognitive Scientists have since proposed an algorithm, called Spreading Activation, that implements Bush's vision (Quillian, 1968; Collins and Loftus, 1974). The Mnemotheque uses Spreading Activation to identify conceptually related memories, dynamically forming pathways in its media-base. As the explorer interacts with Mnemotheque Grinberg, the center of activation shifts from one Bloc Souvenir to the next, spreading through their historical and conceptual context, and accessing other conceptually related media. The explorer guides the spread of activation through themes, places, times, and people. History becomes a dynamic story, every interaction forming new Blocs Souvenir, and new pathways among them.

click to see full size of any image

THE MNEMOTHEQUE FAMILY

Mnemotheque was conceived by Monique Gabrielle Shrager (fmz@numericable.fr), who also produced the database for Mnemotheque Grinberg. Monique studied the design and management of multimedia projects at L'Ecole des hautes tudes en sciences de l'information et de la communication at the Universit de Paris IV, La Sorbonne. She holds a BA in Political Philosophy from Amherst College, and has worked for over ten years as a writer, interactivity designer and editorial consultant with Disney, Microsoft, Kodak, Hitachi, Cryo Interactive, Nevrax. She is presently lives and works in France. The Mnemotheque is the culmination of Monique's experience as a writer, designer, and poet.

Mnemotheque was implemented by Jeff Shrager (jshrager@stanford.edu), Ph.D., who holds degrees in Computer Science and Cognitive Psychology from the University of Pennsylvania and Carnegie Mellon University. Jeff is an expert in Artificial Intelligence and Cognitive Science. He is a research fellow at The Carnegie Institution of Washington, and teaches Human Computer Interaction at Stanford. His current research involves automated scientific discovery. In addition to being a personal exploration, The Mnemotheque embodies research pioneered by Jeff while he was at the Xerox Palo Alto Research Center, integrating models of human memory with multi-media information access.

All of the music in Mnemotheque Grinberg was written and performed by Frederic Grably, a musician and architect living in France. In addition to producing several of his own albums, Fred has written and performed music for numerous multimedia presentations and games, both on- and off-line. Fred also designed and produced the introductory video segment, and wrote and performed its music.

Eric Simard, Helene Bajard, Frederic Xicluna, and Salem Kadouri helped us in preparing the material content of Mnemotheque Grinberg. Most importantly, none of this would have been possible without the love, support, and tolerance of the four survivors: Simone Shrager, Berthe Muflaz, Jeannette Zamora, and Renee Nedjar.

PATHS TO PARTNERSHIP

We are seeking a partner to bring the Mnemotheque vision and technology to a larger community with a larger multi-media collection. The method scales up well, and is fairly easy to implement. A Mnemotheque-inspired interface to a given collection would permit users to explore the collection in a richly connected way, as with Mnemotheque Grinberg. It is an easy conceptual step from this to a separable Mnemotheque distribution for schools and museums, tailored to each specific audience, and delivering the relevant experience of the collection to these groups.

In an even more extensive partnership, we envision the development of a personalizable Mnemotheque, enabling families to set up and explore dynamic histories composed from their own photographs, texts, and video. If these personalized Mnemotheque databases were connected to one another through The Web, and to a larger database as well, Spreading Activation would form associations between the experiences of one family and other families, both in their own communities, and throughout the world. The resulting worldwide Mnemotheque would enable people to explore dynamic histories that span not only time and space, but also family, nationality, and race.

To the extent that Mnemotheque can help people to relate their own experiences to those of others, forming a story that crosses boundaries of many sorts, it may help lead people to greater mutual understanding and tolerance.

REFERENCES

Bush, V. (1945). As we may think. The Atlantic Monthly. 176(1), 101-108.
Collins, A. & Loftus, J. (1975). Spreading Activation theory of semantic processing, Psych. Rev., vol.82, pp.407-428.
Quillian, M.R. (1968). Semantic memory. In M. L. Minsky (Ed.), Semantic information processing. MIT Press

A MNEMOTHEQUE FAQ

Are the Bloc Souvenir canned? No. Each Bloc Souvenir is dynamically composed from the most highly activated media (according to Spreading Activation), modulated by heuristics that prevent the same media from appearing over and over again.

How big is Mnemotheque Grinberg? The database contains about 600 items, composed of about 13 meg of photographic, and 330 meg of video media. There are about 224K of text, and over 150 keywords.

Technically speaking, what is Spreading Activation? Spreading activation is an iterated matrix multiply between a keyword weight vector and the media connectivity matrix. Although the latter can be very large, it is generally sparse, and so Spreading Activation can be quite efficient, even for large databases.

Where do the (3) guide words come from? They are the most discriminating terms among the next most activated media (from the ones included in the current Bloc Souvenir).

On what platform is Mnemotheque Grinberg implemented? Java 2, with JMF (the Java Media Framework).

Will it run across the web? Yes. It is pure Java, and so will run across the web. At the moment it is not web-ready because the media are downloaded instead of streamed. (It should be fairly easy to fix this.) We also do not wish to put our personal family video online.

How can I get a demo? Write either Jeff (jshrager@stanford.edu), or Monique (fmz@numericable.fr), and we will be happy to arrange a private demo.

Where are the keywords, and how would a remote application get to them? The keywords are compiled into the Java program in Mnemotheque Grinberg. The Java program is actually "written" by a somewhat complex Lisp pre-compiler that produces the data structures required for the spreading activation engine. In a remote environment the keyword table would live permanently in memory on a large server that executes spreading activation requests from remote clients.

Where did you get the keywords for Mnemotheque Grinberg? We manually developed them. This turns out not to be so hard. All you have to do is listen through the interview, and type in whatever major conceptual words you hear go by. It isn't necessary to have a transcript, and usually only takes one pass to get the media indexed.

What about missing keywords? Spreading activation is robust in the face of missing words. It has the effect of statistically "filling in" missing words because of commonality of statistical context. For example, if the word "Camp" is often used in the context of the word "Concentration," then the Spreading Activation algorithm will tend to "see" both words when only one actually appears. This can sometimes lead to unexpected behavior, but most often it does what you want it to do. (It is similarly robust to typos, and, interestingly, can have some of the effect of automatic translation from a partial dictionary!)

What are you offering a partner? A free license, in perpetuity to the Mnemotheque concept, for use with specific materials, and our technical assistance in producing and implementing it.

What do you want in return? In the best case, a partner in developing Mnemotheque into a general purpose historical documentary product that can be used to form dynamic interactive documentaries from any media. At minimum, consideration in licensing back extensions made to the software for use with other media, and advice for developing funding for this external project.

Okay, how can I find out more? Write either Jeff (jshrager@stanford.edu), or Monique (fmz@numericable.fr). We'll be happy to discuss any aspect of the Mnemotheque with you, and arrange for a demo.