White Paper for NRO Director’s Innovation Initiative

Mindtel, LLC

November 1, 2001 [third version]

Methods for Perceptually Enhanced Human Interaction with Information: the Perceptualization Environment

A.1 Statement of the Problem

In recent years, the technological capacities of the National Reconaissance Office to access and collect diverse information have increased substantially. And yet, there is clear awareness within the organization that the ability to collect is still insufficient, particularly in this time of growing concern over urban warfare and the atrocious threat of biological and chemical weapons being used as a means of homeland assault. In line with this need for even greater capacities to gather intelligence, Mindtel is aware of the urgent need which exists for a rigorous re-consideration of the human user as a site for the examination of how intelligence processes are created and conducted for analysis. The complex and advancing nature of information gathering has only made salient the need for a more robust integration of the human user into a multipoint and immersive knowledge discovery matrix.

Information gathering and manipulation technologies will ultimately be limited in their ability to make a difference to the extent that the human user is limited in his ability to exploit that information. It is the lack of conceptual and methodological consideration of the human’s extraordinary abilities for perception and cognition that is the problem with current interface models for humans interacting with information. With the typical emphasis on technologies for gathering, handling, and storing information, the question as to how to best present the information to the user has been practically ignored. What is the best way to not only present information to a user, but more importantly, to give the user tools which allow them to manipulate information in ways which ‘feel’ more productive? How can collection be made more valuable by being renderable according the needs and preferences of particular nervous systems tasked to interact with that information?

A.2 Solution

Mindtel seeks to develop an inclusive system based on specific methodolgies and emergent instrumentations. We propose to develop an approach to the problem which leverages sensory physiological prinicples with an operational knowledge of human-information system integration. The priority of our approach is to connect the human to the content via an interface which exploits the visual, auditory, and the tactile capacities of the human body for perception. The human has a much greater capacity for perceptual interaction with information than the current interface model permits. .

MindTel will propose the ‘perceptualization environment’ as a multi-modal, multi-sensory, high performance communications tool for handling the escalating quantities and types of information the Office will confront in the near and distant future. Mindtel’s primary goal is to develop an interface medium for drastically increasing the accessibility and, perhaps more importantly, the intelligibility of variant intelligence data. Intelligibility, or the graspable, available meaning within information, is determined by the manner(s) in which it is presented to a user. The representation and comprehensibility of information are the conceptual and technological challenges we seek to address with the perceptualization environment. This environment is a unified interface system comprised of a core infrastructure and experimental methodology which optimises the integration of human intelligence through increased multi-sensory perceptualization and enhanced expressive capacity. This tool will provide an order of magnitude increase in human-information interaction. Finally, our solution will also propose an exploration of states of mind and how their manipulation may augment the processes of both intelligently and expressivity. In this area we will introduce the ideas of state specific sciences and perceptual state space modulation, cognitive state enhancements.

A.2.1. Integration of Human Intelligence?

The perceptualization environment concept requires a shift in thinking about how humans interact with machines. Up to this point, the nature of the interaction was in how humans adapted to what the machine could perform and to then determine the structures of their use of machines accordingly. In contrast, our approach is to develop a system which generates and modifies information in terms of the user’s preferences for how he would like to confront and manipulate information for his own specific, and at times highly contingent, purposes. We are after an emergent fusioning of information. We desire the end user to be able to rather spontaneosly communicate their intention about the content at hand and its form. We want the user to be able to determine their interaction with information and then for the machine to adapt and render accordingly. The goal substantially includes the following methods for intentional expressivity:

• Communication
• Fusion
• Creation
• querying

Human integration into the gathering and presenting of information opens the space for discernment of knowledge which would have remained invisible. If the human is more deeply woven into the processes whereby the machine makes decisions about data and representation, more powerful options for response are available. This involves, for example, the capacity to specify queries and the results thereof. Or if I, as the user, am able to use the search and representation capacities of the system to capture the characterization of a complex chunk of some data, I am then in a powerful position to share that information with another expert who will be sent much farther along in the process of action. Part of what this technology will allow is the opportunity to not just interact with content, but to interact with processes and patterns within information which will then be open to modification. The pattern is often more significant than the content contained within the pattern. Attention to the patterns within information is a means of developing sensitivity to a process of optimally handling incoming information which can be modeled for and then modified by other users. Thus an increase in the intelligence quotient, if you will, for performing tasks.

A.3. Physicality & Functionality

Mindtel has developed powerful and generic hardware/software solutions to handle core functional elements of this communications environment. Our efforts, upon acceptance of our proposal, will be in the development and implementation of both the conceptual models and functional prototypes. The latter will be initial demonstrations of how the many streams of critical, time sensitive information may be innovatively woven together and presented to the user with a high degree of content dynamicity and user control. Also, the final presentation of information to the user will manifest perceptual diversity; diversity in using the tool to integrate and render information in perceptual styles which depend on mission needs and user biases (See Appendix B).

We currently conceive of the perceptualization environment as a physical space, as opposed to a kind of head mounted visual and auditory display. A room of some size, yet to be determined, will be built to contain an array of current and high-end visual and auditory displays. For the tactile displays, the user will have some surfaces of the body covered with the tactile rendering devices themselves. The perceptualization environment will be consitituted by the following elements:

1. Hardware/Software

2. Primary physicality of system

3. Functionality of integrated components

4. Open & adaptable operational processes and methods

The goal of the perceptualization environment is to simultaneously explore and enhance both the

• the perceptual form in which information is being represented &
• the cognitive state of the human during perception of represented information

A.4 Cognitive State Enhancements

‘State-specific sciences’ refers to the idea that states of brain affect states of the mind (i.e., perception and cognition). Therefore, it is reasonable to assume that alterations in brain state will bring out some correlative alterations in perceptual cognitive state; changes in perception disclose heretofore invisible information. State specific take into account and exploit ‘states of consciousness’ (SoC) when interacting with particular data. We do not know all that is to be accessed in a collection of data. Thus, altering our perceptual cognitive states at calculated points during the design and execution of the interaction process may open us to entire realms of information which would have remained totally invisible.

A.4.1 Perceptual State Space Modulation

The idea of perceptual state space modulation is our attempt to conceptualize what it would look like to introduce state specific sciences into the discussion of human-information interaction. Not only do new representations of information promise to show us new and powerful contents of our information which we could not have otherwise seen, but so might changes in brain state give us new perceptions of that information. Ultimately, it is the thoughts which one has about information which leads to decisions and thus processes set in motion to benefit most from that information. How then might we alter user states of consciousness?

A.4.2 Modalities

How might one think about modulating states of the brain so as to modulate states of the perceiving mind? We have identified several:

1. Bioenergetic modulations. Microwaves and other kinds of energy modulations have been shown to have diverse influences on neurophysiology. Perhaps it would be possible to also include certain frequency modulations into the perceptualization environment.

2. Acoustic modulations may also be an avenue for perceptual state space modulation. Manipulated sound can have effects on the central nervous system which may enhance perception and cognition in important ways. For example, the Monroe Institute has long conducted acoustic-neurologic research. With modulation of specific sounds into both ears at the same they are suggested a re-entraining of the two hemispheres of the brain. Hemispheric synchronization is the phenomena whereby communication between the hemispheres is optimized/altered along some axis resulting in perceptual and cognitive alterations.

3. Pharmacology

Mindtel’s research in this area thus far has centered on acoustic state modulation. This aspect of our thinking about the perceptualization environment is likely to show value once substantial progress has been made in the development of the perceptualization technologies.

A.5 Summary

In this paper, our purpose has been to climb out of the box to think in new ways about the perceptual basis of human interaction with vast and diverse data sources . What we are proposing is far outside the normative realm of how to construct interfaces and represent information. Thinking in these ways will require potential users to understand the challenge which confronts them in this time of intelligence gathering. Without pushing the boundaries of what can count as an acceptable means for accessing information and making decisions based upon it, unrealized possibilities to help go unseen. We are aiming towards a new kind of perceptually based literacy to aid any process of high quantity, high performance interaction with information. Arriving at this place in our thinking has been the result of conceptual cross linking based on a desire for both a flexibility of functionality and massively enhanced perceptual dimensionality. The perceptualization environment is a representational knowledge matrix which addresses the problem of the insufficient language structures we currently use in digital communications. The new language we are working on will no doubt lead to a kind of perceptual cross pollination in the different realms where this technology is eventually applied. Our desire is to be funded in sufficient measure to construct a robust prototype which collects data from sensors, human input, and databases and turns it into “sentences” in our new perceptual languages.