Informedia Digital Video Library:  Digital video library research at Carnegie Mellon School of Computer Science
nav graphic
















  Carnegie Mellon University
  School of Computer Science
  5000 Forbes Avenue
  Pittsburgh, PA 15213

  About CCRHE  |  Sponsors
Project Title:
Capturing Coordinating and Remembering Human Experience
Howard Wactlar
Mike Christel, Alex Hauptmann, Mark Derthick, Takeo Kanade
National Science Foundation under the Division of Information and Intelligent Systems

Project Description

Building upon technology developed under Informedia, this project (CCRHE) will develop algorithms and systems enabling people to query and communicate a synthesized record of human experiences derived from individual perspectives captured during selected personal and group activities. CCRHE will transform this record into a meaningful, accessible information resource, available contemporaneously and retrospectively. We validate our vision with two societally relevant applications:

1.) Geriatric Care and Clinical Studies: CareMedia is a current project that is creating a meaningful, manageable information resource that enables more complete and accurate assessment, diagnosis, treatment, and evaluation of behavioral problems for the elderly. Through activity and environmental monitoring in a skilled nursing facility, a continuous, voluminous audio and video record is captured. Through work in information extraction, behavior analysis and synthesis, this record is transformed into an information asset whose efficient, secure presentation empowers geriatric care specialists with greater insights into problems, effectiveness of treatments, and determination of environmental and social influences on patient behavior. The results would not only assist in individualizing patient treatment plans but would also improve the ability of clinician-researchers to demonstrate the impact of their interventions.

2.) Emergency Response Management: Many disaster reports lack documentation, objectivity, and perspective. These shortcomings can be addressed by automatically capturing multiple experiences occurring during overlapping periods of time and origin as part of an emergency response activity. The collective view generated from such experiences may improve the ongoing response activity through better allocation of resources, coverage of particular areas, and faster communication between respondents. After the event, the captured experience becomes a resource for emergency responders and planners, so that they may profit from the lessons learned by others.

This research will prototype personal experience capture units to record audio, video, location and sensory data, and electronic communications. Each constituent unit captures, manages, secures and associates information from its unique point of view. Each operates as a portable, interoperable, information system, allowing search and retrieval by both its human operator and remote collaborating systems. An individual cannot see everything, nor remember everything that was seen or heard. The integration of multiple points of view provides more comprehensive coverage of an event, especially when coupled with support for vastly improving the memory from each perspective. The research thus enables the following technological advances:

  • Enhanced memory for individuals from an intelligent assistant utilizing an automatically analyzed and fully indexed archive of captured personal experiences.
  • Coordination of distributed group activity, such as management of an emergency response team in a disaster relief situation, utilizing multiple synchronized streams of incoming observation data to construct a "collective experience."
  • Expertise synthesized across individuals and maintained over generations, retrieved and summarized on demand to enable example-based training and retrospective analysis.
  • Understanding of privacy, security and other societal implications of ubiquitous experience collection.

The foundation for this work, the Informedia Digital Video Library, has demonstrated the successful application of speech, image, and natural language processing in automatically creating a rich, indexed, searchable multimedia information resource for broadcast-quality video. The proposed work builds from these technologies, moving well beyond a digital video library into new information spaces composed of unedited personal experience video augmented with additional sensory and position data. Tools will be created to analyze large amounts of continuously captured digital experience data in order to extract salient features, describe scenes and characterize events. The research will address summarization and collaboration of multiple simultaneous experiences integrated across time, space, and people.

A distinguished panel of experts will serve as the Advisory Board for the project, providing domain knowledge and sharpening our application focus for elder care memory aids and emergency response coordination. In addition, the Board will aid in assessing the broad societal implications for the work in capturing, coordinating, and remembering human experience.

This effort provides cohesion and application focus to a broad research program in numerous disciplines addressing information gathering, extraction, and communication, including wireless networking, distributed real-time systems, knowledge management, social and knowledge networks, and information security, data privacy and survivability. Recognizing the significant cross-disciplinary system implementation requirements of the research, the project will also create internships for several new post-baccalaureate professional degree programs in information technology established at Carnegie Mellon University, including those specializing in software engineering, data mining, language technologies, e-commerce, data privacy, and human computer interaction.

Figure: Conceptual overview of experience information gathering, extraction and feedback.



nav graphic

About CCRHE  |  Sponsors  |  INFORMEDIA HOME