The Remote Monitor, Control, and Identify Input Suit (REMCIIS) Using BioElectric Signals
BioControl Systems is proposing the development of the Remote Monitor, Control, and Identify Input Suit (REMCIIS) for control of both simulated environments (AVATAR) and real combat situations. The unique goals of REMCIIS are:
1. To create a light weight input suit that would allow for unencumbered movement with no problem of mechanical failure;
2. To create an input suit that can remotely monitor all bioelectric phenomena including muscle activity, heart rate, galvanic skin response, respiration rate, and brain activity;
3. To create an input suit that can control in real-time all aspects of a virtual or real environment using muscle motion and gestures, eye gaze (in both two and three dimensions), and brain activity; and
4. To create an input suit that can identify the user for security and database collection purposes.
This white paper first describes the development of BioSignal Processing by R. Benjamin Knapp, Ph.D. and Hugh S. Lusted, PhD. both from Stanford University and founders of BioControl Systems, Inc.. It then summarizes the import aspects in the development of REMCIIS in four parts: the input hardware, the monitoring software and verification, the control software and verification, and the identification software and verification.
BioControl Systems (BCS) has been involved with biosignal processing and the development of biosignal controlled interfaces since 1987. Currently, BCS has three issued patents on eye controller technology and three patents pending on various other biocontroller technologies. In 1992, BCS released its first product ‑ the BioMuse, which is a biocontroller development platform. The heart of a biocontroller is a three stage process involving: (1) bioelectric signal acquisition, (2) signal processing for pattern recognition or extraction of desired elements of the biosignal, and (3) mapping of the results of the signal processing algorithm to some desired output code which controls external electronic devices. This configuration allows flexibility in mapping biosignal inputs to output code and also allows for specific applications to be created in software without the need for specialized transducers on the body. The device has eight input channels with programmable gains and filters that can be configured to process virtually any bioelectric signal. Currently, BCS is directing its development efforts toward utilization of four main types of bioelectric signals: Electrooculogram (EOG), Electromyogram (EMG), Electroencephalogram (EEG), and Electrocrardiogram (EKG). To date, BCS has implemented various signal processing algorithms which enable these biosignals for use as biocontrollers.
BCS has an ongoing R & D effort devoted to advancing the state of the art in EOG controller technology, and also to develop several application areas for the eye controller. For instance, BCS has developed eye controlled joystick and mouse applications that allow users to control graphical objects with eye movements. The mouse applications give users the ability to do “hands free “ word processing or use other menu driven software. This type of system is ideal for persons with physical disabilities who do not have access to computers through the traditional manual inputs (Knapp and Lusted 1993). BCS has also developed the ability to track the movement of each eye independently, allowing the system to ascertain where the user is looking in 3‑D space, and also control virtual objects in 3‑D space. Current research is aimed at improving the control accuracy of the eye controller system, and BCS recently completed an ARPA funded (Contract no. MDA972‑94-M-0011) feasibility study for use of the this technology as an endoscopic camera controller.
EMG signal processing and controller work is currently proceeding in two directions. The first is in the development of continuous EMG signal monitoring for use as a graphical controller. This enables the physically disabled to use word processing programs and other personal computer software. It also enables manipulation of robotic devices for telepresense and telesurgery.
The second developmental area is in pattern recognition using fuzzy set theory and neural networks. The EMG controller can be programmed to perform gesture recognition based on signal analysis of individual muscle groups or groups of muscles working together (Putnam and Knapp 1993). BCS has demonstrated the feasibility of a hand gesture recognition system using a sensor array on the forearm which leaves the hand unencumbered. An advantage of a biosignal based system for gesture or movement detection is that these applications can be implemented in software without the need for expensive extremity mounted movement detectors, such as a data glove. The biosignal data can also be used to gain additional information about a user. By monitoring the magnitude and dynamics of muscle tension, performance and fatigue parameters can be established. Thus, individual training and achievement programs can be designed for different kinds of motor skill tasks.
BCS is also conducting research in the area of pattern recognition on specific EEG signals from the brain. This is an analogous process to the hand gesture recognition mentioned earlier, the difference being that the signal originates from the brain. Ultimately, with this type of controller, the user could visualize action which would be translated to a command set for a computer to implement Additionally, the user could think a word or command which would be interpreted and implemented by the computer. To date, BCS does not have a controller that can interpret “subvocalized” commands of this sort. However, BCS is working on brain signal analysis algorithms with that eventual aim in mind. Dr. Knapp has done studies with neural network pattern recognition in order to classify sleep stage EEG records (Hu & Knapp 1991). This system was able to recognize EEG sleep patterns with 90% reliability. In quite a different application area, BCS has implemented EEG pattern detection programs that output MIDI (Musical Instrument Digital Interface) commands for sound synthesis (Knapp and Lusted 1990).
BCS is developing similar pattern recognition techniques to examine pathologies of the EKG. Eventual use of this for cardiac monitoring using sonification techniques is being explored.
REMCIIS will incorporate advanced biosensors with multi-use functionality. BCS proposes using dry electrode biosensors which will be integrated into a light weight, unencumbering suit, that can easily be worn under normal clothing. In addition to basic autonomic biosignal monitoring (e.g. EKG), multi-channel EMG monitoring of key muscle groups will render complete stress, attention, and fatigue evaluation, as well as dynamic motion capture capabilities for real-time task performance evaluation.
In order to create a biosensor that can survive in a battlefield situation or a heavily used simulation based design (SBD) environment, the biosensor must meet the following criteria:
1. Durable
2. Reusable
3. Require little or no preparation
The biosensors (electrodes) that BCS presently uses are a patented semi-aqueous hydrogel preparation. They are reusable up to one week and they require little or no skin preparation. They are not, however, very durable due to their semi-aqueous nature. BCS has proposed and performed preliminary research on a new dry electrode system. A dry electrode would create a biosensor that would be very durable, reusable for a long period of time, and require little or no skin preparation, i.e., a biosensor extremely well suited for placement inside REMCIIS for the simulation based design environment as well as ultimately for the battlefield environment.
For personnel that wear head gear, either a standard helmet or visor type HMD, biosensors can be incorporated into the headgear for dual use functionality. The advantage of this integrated biosignal processing is that the same biosensors can be used for both vital signs monitoring and biocontroller operations. In the case of the HMD, interactive control of data displays using eye movement is possible, as well as EEG monitoring for attention and stress evaluation.
The following is a step by step description of how BCS will create a the hardware for a complete input suit and HMD dry electrode system:
REMCIIS
Hardware Development
|
Project Description |
Deliverable |
Advantages |
Time |
Cost |
|
Phase 1: Discrete Active Electrodes for EMG (Muscle Tension and Movement Patterns) Monitoring and Control |
A set of prototype electrodes with attached discrete active electronics capable of recording EKG and EMG activity with a bandwidth of 1Hz -1kHz and a sensitivity of 1.0 uV. Electrodes will interface directly to the BioMuse biological signal processing system. |
These new biosensors would: · be more durable than any aqueous or semi-aqueous electrode system. · be reusable without any special storage technique · require little or no skin preparation · integrate easily into a inputsuit system |
9 Months |
|
|
Phase 2: REMCIIS without HMD: Integrated Circuit Active Electrodes System |
A completely integrated dry electrode prototype system with microprocessor and RF transmitter. Prototype receiver system to interface to the BioMuse biological signal processing system |
The REMCIIS hardware would integrate the capabilities of the Phase #1 system into a complete remote monitor / control / identify system. There would be no associated wires and could monitor a combatant at a great distance. |
18 months |
|
|
Phase 3: REMCIIS with HMD: |
A set of prototype electrodes with attached discrete active electronics capable of recording EOG with a bandwidth of 0.01Hz -100Hz and a sensitivity of 1.0 uV. Electrodes will interface directly inside HMD of REMCIIS. |
These new biosensors would improve on the Phase #1 sensors by adding the capability for eye tracking. These new sensors would: · have long term drift that is only a fraction of the drift of aqueous or semi-aqueous electrodes · be able to be incorporated directly into an HMD |
12 Months |
|
|
Total Cost for REMCIIS
Hardware without HMD: |
|
|||
|
Total Cost for REMCIIS
Hardware with HMD: |
|
|||
BCS has begun investigation into the creation of a Virtual Reality Software Application designed specifically for the SBD environment. The creation of such an application will require the expansion of BCS’s UNIX software library. The library will facilitate the process of applications building and initiate the process of establishing an industry standard for the development of biosignal processing and biocontrollers. The objectives of the proposed REMCIIS monitoring software are:
1. Database: The BioMuse signal processing system is a high bandwidth measurement tool for monitoring human performance and human-computer interaction. The BioMuse’s self-calibration and self-diagnostic capabilities make it ideal for measuring bioelectric data and patterns within the bioelectric data for quantitative analysis and storage. The stored information can then be structured as a database that will have detailed information on personnel performance for a specific complex task.
2. Feedback and Training: In a training situation, all of the trainee’s bioelectric activity including muscle tension (stress), motor patterns and eye movement can be graphically displayed in real-time. In addition the bioelectric activity can be mapped to Musical Instrument Digital Interface (MIDI) for audio feedback (sonification) of the procedure. Automated, or command personnel monitored, comparison of the training procedure versus an ideal procedure will further help instruct the trainee. The inputsuit would allow for unencumbered movement in the training procedure with no problem of mechanical failure.
In order to implement these advanced applications, BCS proposes to complete the UNIX software library and then integrate the inputsuit and HMD controllers into the UNIX development environment.
Special purpose software will be developed for evaluating stress and attention factors. Follow-on stage software development will focus on human performance evaluation factors in both simulated and real task environments. BCS envisions using this technology to assess psychomotor performance for complex tasks. In addition, the data gathered during assessment can be used to create more effective training programs, individually tailored for the strengths and weaknesses of each trainee.
Methodologies of
control using an individuals (combatants) nervous system for real or simulated
environments.
The EEG (brain), EOG
(eye), and EMG (muscle) controllers in Simulation:
· By measuring both the combatants' eyes we are able to monitor the eye's convergence and divergence for 3 dimensional control. In recent tests on the Onyx SGI computer we were able to determine four levels of depth in a virtual environment. BioControl Systems, on March 8, 1994, was issued a patent (# 5,293.187) for this device.
· In a simulation system using the signals from both eyes in a HMD or HUD the combatant will have all the control functions that they have in the real world. The ability to view and control a virtual environment by natural eye gaze in 2 and 3 dimensions. Where I look is what I see (WILIWIS).
· In a HMD or HUD the combatants can dynamically change the view by their eye movement, as opposed to head movement, to reduce the lag time that is currently associated with head trackers. This method is more natural, to the combatant, and may have a positive effect on motion sickness.
· The combatant can point and click on text or objects in the HMD or HUD for access of information and for communication with others combatants or the command post.
· A combatants’ use of eye control in a virtual battlefield is “The Third Arm”. Allowing them control of other virtual objects while they still have the use of their hands to hold weapons.
· An Avatar’s virtual eyes can be made to replicate the combatants. The motion capture of real eye movement mapped to a virtual Avatar’s eyes.
· Whole body, real time, motion or gestures of a combatant are captured and mapped to a virtual environment. As the combatant, in real time, make's motions or gestures the EMG signal pattern is recognized, processed, and mapped to an Avatar to create the same motion or gesture.
· By processing the EMG signels from the combatants, Flexor and Extensor, forearm muscles we are able to determine grip and hand control of the Avatar. This method will improve the functionallity of the current mechanical data glove input device.
· The EEG and EOG combined can act as an accurate point and click device for virtual environments. BioControl’s current development has shown that using both of these signals we can create a fast and accurate method of landing a cursor on a small target in virtual environments. The EOG saccadic eye movement is extremely fast in getting to within 2 degrees of the target that a combatant is gazing at. Using a subliminal, evoked potential, EEG controller the system will allow the cursor to land directly on the target that is being looked at.
· The alpha EEG waveform is ideal as a triggering device. The combatant can be trained to use this waveform to trigger action in a virtual environment. Also other EEG waveforms are being considered as controllers.
The EOG (eye), EEG (brain), and EMG (muscle) controllers in real
environments:
· A combatant can target objects in the battlefield using an eye gaze system. This system is not limited to field combatants and can be used in any military situation. Night vision devices would be an ideal example for the use of an EOG eye gaze system. The EEG evoked potential capability can be used in this scenario for greater accuracy.
· BioControl has recently demonstrated, and filmed, how the eye gaze system can control a surgical robotic endoscope. The AESOP robotic endoscope was controlled by natural eye gaze. We were able to target the organ of choice and move the endoscope to that position. This type of system would also be beneficial for remote surgical procedures in the battlefield.
· The EOG gaze and EMG motion capture systems can be used for tele-presence and tele-robotics applications. The control of electronic devices in remote locations by combatants. In particular, in hazardous areas' robots can be sent in that are being controlled by a combatant, nervous system, wearing the (biological) motion capture input suit. This device will allow them to have natural control of a 3D camera by their eyes’ and control robotic vehicles in the remote location by their natural motion.
Summary of Controllers: The use of EOG, EEG, and EMG in virtual environments is now a necessity. If we are to create more natural and realistic virtual environments we must consider the way we interact with our own three dimensional world. The EOG, EEG, and EMG signals have been processed and digitized to meet this requirement. When using devices to control virtual environments and characters (Avatars) within them it is essential that we create something that is natural and akin to real world situations. With this approach in mind the human nervous system supplies all the needs required for this purpose. In addition to virtual environment control the processed signals can be deployed to control any digital device. This system is a true, real time, human-to-machine interface.
To complete the system, BioControl is proposing the addition of a pattern matching and recognition feature to identify the soldier using REMCIIS. There are two compelling reasons for the addition of this feature:
1. Monitoring Security: To determine that the critical physiological information is coming from the targeted soldier in the battle field (or in simulation) and not from some other hostile (or friendly) source.
2. Control Security: To determine that only combatants/individuals with proper security clearance may use BioSignals to control given devices in the battlefield (or in simulation).
BioControl proposes to develop a pattern recognition system using fuzzy logic to identify the bioelectric signals of the user of REMCIIS. Dr. Knapp has already developed a system for the NSA for recognizing truth vs. deception in the polygraph (Knapp,1994). He has also developed pattern recognition systems for recognizing bioelectric patterns in arm gestures (Knapp,1993) and in eye motion (Knapp,1992)
The development of the identification portion of REMCIIS will take several stages:
1. Development of EMG Signature Data Base
2. Development of off-line pattern recognition to 100% accuracy
(The following work must be completed after the dry electrode sensors are added to the PSM)
3. Development of Real-Time Identification System to 100% accuracy to complete REMCIIS
REMCIIS Software Development
|
Project Description |
Deliverable |
Advantages |
Time |
Cost |
|
Phase 1: UNIX software library expansion to enable development of BioMuse biosignal processing applications |
A complete library of commands to control the BioMuse system. The library will run on a Silicon Graphics Iris, Challenger, or Reality Engine platform |
This new library would: · allow rapid development of biosignal processing applications by BCS and any other ARPA contractor · allow networking of multiple BioMuse systems for remote operation and massive parallel data gathering |
9 Months |
|
|
Phase 2: EMG monitor and control integration |
A turnkey application based on the phase #1 UNIX library which will integrate BCS’s proprietary EMG data acquisition and pattern recognition algorithms into REMCIIS |
This new application would: · eliminate the need for mechanical biometric techniques which are prone to failure · automate data collection and manipulation for massive data collection · allow EMG intensity and gesture control of the SBD environment |
12 Months |
|
|
Phase 3: Pattern recognition for combatant identification |
This system will introduce advanced pattern matching software to identify the individual’s biosignals |
This new system will: · keep secure access to parts or all of an SBD system · provide field ID in case of possible hostile use of REMCIIS |
12 months |
|
|
Phase 4: EOG (2 and 3 dimensional) monitoring and controller integration for simulation and training applications |
A turnkey application based on the phase #1 UNIX library and the phase #2 application which will integrate BCS’s proprietary and patented EOG eye controller algorithms into REMCIIS for HMD applications |
This new application would: · incorporate an eye controller into an HMD environment for both eye tracking and virtual object controlling · allow for eye monitoring with no loss of field of view · enable faster image updating than can be achieved using information obtained from a Polhemus style head tracker |
12 months |
|
|
Total Cost for REMCIIS
Software without HMD: |
|
|||
|
Total Cost for REMCIIS
Hardware with HMD: |
|
|||