A Telepresence Mobile Robot Controlled
With a Non-Invasive Brain-Computer
Interface

T.Shanmugapriya; S.Senthilkumar

A Telepresence Mobile Robot Controlled With a Non-Invasive Brain-Computer Interface

T.Shanmugapriya¹, S.Senthilkumar²

Assistant Professor, Department of Information Technology, SSN Engg College, Chennai, Tamil Nadu, India
Assistant Professor, Department of Electronics and Instrumentation, Bharath University, Chennai, Tamil Nadu, India

Related article at Pubmed, Scholar Google

Visit for more related articles at International Journal of Innovative Research in Computer and Communication Engineering

Abstract

This paper reports an EEG-based brain-actuated telepresence system to provide a user with presence in remote environments through a mobile robot, with access to the Internet. This system relies on a P300-based BCI and a mobile robot with autonomous navigation and camera orientation capabilities. The shared-control strategy is built by the BCI decoding of task-related orders (selection of visible target destinations or exploration areas), which can be autonomously executed by the robot. The system was evaluated using five healthy participants in two consecutive steps: (i) screening and training of participants, and (ii) preestablished navigation and visual exploration telep- resence tasks. On the basis of the results, the following evaluation studies are reported: (i) technical evaluation of the device and its main functionalities, and (ii) users’ behavior study. The overall result was that all participants were able to complete the designed tasks reporting no failures, which shows the robustness of the system and its feasibility to solve tasks in real settings where joint navigation and visual exploration were needed. Furthermore, the participants showed great adaptation to the telepresence system.

INTRODUCTION

Brain-computer interfaces (BCIs) provide users with com- munication and control using only their brain activity. BCIs do not rely on the brain’s normal output channels of peripheral nerves and muscles, opening a new valuable communication channel for people with severe neurological or muscular dis- eases, such as amyotrophic lateral sclerosis (ALS), brainstem stroke, cerebral palsy, and spinal cord injury. The ability to work with non-invasive recording methods (the electroen- cephalogram or EEG is the most popular method) is one of the major goals for the development of brain-actuated systems for humans. Some examples of EEG-based applications include the control of a mouse on a computer screen [1], communica- tion such as spellers [2], Internet browsers [3], etc. The first noninvasive brain-actuated control of a physical device was demonstrated in 2004 [4] and since then, research has been mainly focused on wheelchairs [5], [6], [7], [8], manipulators [9], [10], small-size humanoids [11], and orthosis operated with functional electrical stimulation [12], [13], to name a few. All these developments have a property in common: user and robot are placed in the same environment

Very recent research has focused on BCI applications where the human and the robot are not co-located, such as in robot teleoperation. Some examples include a museum guide robot [14], the teleoperation of a manipulator robot [9], an .aircraft [15], or mobile robots [16], [17]. The ability to brain- teleoperate robots in a remote scenario could provide severely disabled patients with telepresence. Telepresence could be seen as an extension of the sensorial functions of daily life by means of a physical device, embodied in the real environment and placed anywhere in the world, which could perceive, explore, manipulate, and interact with the remote scenario, and controlled only by brain activity. Furthermore, it has been suggested that the use of these BCIs could have a neurorehabil- itation effect and/or a maintenance of neural activity, avoiding or delaying the extinction of thought, hypothesized to occur in patients like ALS [18].

There are three major engineering problems in the design of this type of systems: (i) current non-invasive BCIs are slow and uncertain; (ii) BCIs when used as input interfaces are highly cognitive-demanding; and (iii) the variable and uncertain communication time delays in any development involving robot teleoperation via Internet. For these reasons, researches have started to look at these systems from a shared- control point of view, where the robot is equipped with a degree of intelligence and autonomy that totally or partially manages the task (alleviating the previous problems). This principle was initially explored in the context of BCI control of wheelchairs [5], [6], [19] and very recently applied to BCI telepresence [17]. This paper is in line with these works.

The present BCI telepresence system relies on a syn- chronous P300-based BCI and a mobile robot with au- tonomous navigation and camera orientation capabilities (see [20] for reviews on BCIs and [21] on navigation systems). During operation, the user concentrates on the desired option on a computer screen, which displays live video sent by the robot along with relevant information related to robot motion or camera orientation tasks. Following the typical P300 visual stimulation processes, the BCI collects the EEG brain activity and decodes the user’s intentions, which are transferred to the robot via Internet. The robot autonomously executes the orders using the navigation system (implemented with a com- bination of dynamic online grid mapping with scan matching, dynamic path planning and obstacle avoidance) or the camera orientation system. Thus, the shared-control strategy is built by means of the mental selection of robot navigation or active visual exploration task-related orders, which can be autonomously executed by the robot. In principle, this shared- control design shapes the low information transfer rates of existing BCIs, avoids the exhausting mental effort of BCIs that require continuous control, and overcomes the Internet delays problems in the control loop. In relation to the shared-control approach used in [17] to teleoperate a mobile robot, which elies on a motor imagery BCI and low-level robot motion primitives, the contribution of the present engineering system is a shared-control design that incorporates a much higher degree of autonomy in the robotic layer.

An added value of this research is the experimental method- ology and validation protocol, which could guide future de- velopments. The telepresence system was evaluated using five healthy participants in two consecutive steps: (i) screening and training of participants, and (ii) pre-established navigation and visual exploration tasks performed during one week between two laboratories located 260 km apart. On the basis of the results, the following analyses are reported: (i) technical eval- uation of the device and its main functionalities, and (ii) users’ behavior study. The overall result was that all participants were able to complete the designed tasks reporting no failures, which shows the robustness of the system and its feasibility to solve tasks in real settings where joint navigation and visual exploration are needed. Furthermore, the participants showed great adaptation to the system.

In relation to our previous work, partial results were outlined in [22]. This paper reports the complete results of the inves- tigation, and is organized as follows: Section II describes the brain-actuated telepresence technology, Section III describes the experimental methodology, Section IV reports the results and evaluations, and conclusions are drawn in Section V.

TELEPRESENCE TECHNOLOGY

The telepresence system consisted of a user station and a robot station, both remotely located and connected via Internet (Figure 1). At the user station, the BCI decodes the user’s intentions, which are transferred to the robotic system via Internet. At the robot station, the user’s decisions are au- tonomously executed using autonomous navigation and active visual exploration capabilities. Furthermore, the robot station provides live video (captured by the robot camera), which is used by the user as visual feedback for decision-making and process control. From an interactional point of view, the user can switch between two operation modes: (i) robot navigation mode, and (ii) camera exploration mode. According to the operation mode, the graphical interface displays a set of augmented reality locations to navigate to, or visually explore. The user then concentrates on the desired location, and a visual stimulation process elicits the P300 visual-evoked potential enabling the pattern-recognition strategy to decode the desired location. Finally, the target location is transferred to the robotic system via Internet, which autonomously executes the relevant orders: (i) in the robot navigation mode, the autonomous navigation system drives the robot to the target location while avoiding collisions with obstacles detected by its laser sensor; and (ii) in the camera exploration mode, the camera is oriented to the target location, performing a visual exploration of the environment.

The next subsections outline the three main modules that compose the global system: brain-computer system (proto- col and EEG acquisition, graphical interface, and pattern- recognition strategy), robotic system, and integration between the systems.

A. BCI: Protocol and EEG Acquisition

The BCI was based on the P300 visual-evoked potential [23]. In this protocol, the user attends to one of the possible visual stimuli, and then the brain-computer system detects the elicited potential in the EEG. The P300 potential is characterized by a positive deflection in the EEG amplitude at a latency of approximately 300 ms after the target stimulus is presented within a random sequence of non-target stimuli (Figure 2a-b). Elicitation time and amplitude are correlated to fatigue of the user and to saliency of stimulus (color, contrast, brightness, etc.) [24]. This potential is always present as long as the user is attending to the process, and its variability among users is relatively low. BCIs based on this potential have been successfully used in patients for long periods of time in different assistive applications (see review in [25]). EEG was acquired using a commercial gTec EEG system (EEG cap, 16 electrodes, and a gUSBamp amplifier). The electrodes were located at Fp1, Fp2, F3, F4, C3, C4, P3, P4, T7, T8, CP3, CP4, Fz, Pz, Cz, and Oz, according to the international 10/20 system, as suggested in previous studies [26]. The ground electrode was positioned on the forehead (position Fpz) and the reference electrode was placed on the left earlobe. The EEG was amplified, digitalized with a sampling frequency of 256 Hz, power-line notch filtered and bandpass-filtered between 0.5 and 30 Hz. Graphical interface and signal recording and processing were developed through the BCI2000 platform [27], placed on an Intel Core2 Duo @ 2.10 GHz with Windows XP OS.

B. BCI: Graphical Interface

The brain-computer system incorporated a graphical inter- face with two functionalities: (i) to provide the user with the functionalities to control the robot and visual feedback of the robot environment, and (ii) to develop the visual stimulation process to trigger the P300 potentials.

a) Visual Display: In both operation modes (robot nav- igation and camera exploration), the visual display showed an augmented reality reconstruction of the robot environment overlapped with live video background (Figure 3). The re- construction displayed a predefined set of options, arranged in a 4 × 5 matrix to favor the next patternrecognition strategy. In the robot navigation mode, a set of possi- ble destinations were represented by a (1.5m, 2.5m, 4m) × (−20?, −10?, 0?, 10?, 20?) polar grid referenced on the robot. Destinations were selected as a compromise between utility and good visualization, and represented real locations in the environment that the user could select. Obstacles were depicted as semitransparent walls built from a 2D map con- structed in real-time by the autonomous navigation technology, hiding unreachable destinations. The row of icons in the lower part of the display represented the following options,

from left to right: (i) turn the robot 45? to the left; (ii)

refresh option; (iii) change to camera exploration mode; (iv)

validate the previous selection; and (v) turn the robot 45?

to the right. In the camera exploration mode, destinations were uniformly placed on a 2D grid, which mapped a set of locations that the user could select to orientate the camera in that direction. The row of icons in the lower part of the display represented the following options, from left to right: (i) align the robot with horizontal camera orientation and change to robot navigation mode; (ii) refresh option; (iii) change to robot navigation mode; (iv) validate the previous selection; and (v) set the camera to its initial orientation. The refresh option allowed the user to receive live video for 20 s, freezing the stimulation process in that interval. Further information on an improved version of the present visual display, which incorporates bidirectional communication along the lines of a video-conference, can be found in [28]. b) Stimulation Process: A visual stimulation process was designed to elicit the P300 visual-evoked potential. The options of the visual display were “stimulated” by flashing a circle on them. The Farwell & Donchin paradigm [29] was followed to reduce the magnitude of the posterior classification problem and sequence duration (a sequence is a stimulation of all options in a random order as required by the P300 oddball paradigm). Flashing of stimulus was accomplished by means of rows and columns instead of flashing each option individually, obtaining 9 stimulations per sequence (4 rows plus 5 columns) instead of 20.

All visual aspects of the elements shown on the visual display (color, texture, shape, size and location) as well as

all scheduling of the stimulation process (mainly stimulus duration, inter-stimulus interval, and number of sequences) could be customized to equilibrate the user’s capabilities and preferences with the performance of the system. Note that the P300 potential is correlated to these aspects.

C. BCI: Pattern-Recognition Strategy

A supervised pattern-recognition technique was used to recognize the P300 visual-evoked potential. This technique was applied offline to previously recorded EEG, where the user attends to a predefined sequence of targets. The tech- nique consisted of two steps: (i) feature extraction, and (ii) classification algorithm.

a) Feature extraction: In order to extract features, EEG data were first pre-processed following the technique described in Krusienski et al. [26]: one-second vectors of data were extracted after each stimulus onset for each EEG channel, and these segments of data were then filtered using the moving average technique and downsampled by a factor of 16. Selection of input channels for the classifier was based on the r2 metric [27]. For each channel, this metric computes the variance between target and non-target feature vectors (note that each feature vector could be labeled as target or non-target according to whether the stimulus was attended to or not by the user). Thus, r2 values for each channel were plotted (Figure 2c) and the channels with higher r2 were selected through visual inspection (a priori those channels will be the best to discriminate through a linear classifier). Finally, following Krusienski et al. study [26] again, the feature vectors of selected channels were normalized and concatenated, creating a singlefeature vector for the classification algorithm.

b) Classification algorithm: Two classification subprob- lems were obtained following the adoption of the Farwell & Donchin paradigm in the stimulation process. The StepWise Linear Discriminant Analysis (SWLDA) was used for each subproblem. SWLDA is an extension of the Fisher Linear Discriminant Analysis (FLDA), which performs a reduction in the feature space by selecting the most suitable features to be included in a discriminant function. This classification algorithm has been extensively studied for P300 classification problems, obtaining very good results in online communica- tion using visual stimulation [30].

The P300 signal-to-noise ratio is low but it can be improved by averaging the responses through repetitions of the stimu- lation process (number of sequences). This leads to higher classification accuracy at the cost of longer stimulation time (time of a sequence in the stimulation process is approximately 2 s). Number of sequences is usually customized per user (Figure 4).

D. Autonomous Robotic System

The robot was a commercial Pioneer P3-DX, equipped with a laser sensor, camera, back wheels (working in differential- drive mode), wheel encoders (odometry), and a network inter- face card. The main sensor was a SICK planar laser placed on the frontal part of the robot; the laser operated at a frequency of 5 Hz with a 180? field of view and 0.5? resolution (361 points). The camera, placed on the laser, was a pan/tilt Canon VC-C4 with a ±100? pan field of view and a 90?/ − 30? tilte field of view. The robot was equipped with a computer, an Intel @ 700 MHz with Linux OS (Debian distribution). The computer managed all computational tasks, provided access to the hardware elements through the player robot device inter- face [31], and integrated the autonomous navigation system. In the experiments, the maximum translational and rotational velocities were set to 0.3 m/s and 0.7 rad/s respectively. The objective of the autonomous navigation system was to drive the vehicle to a given destination, set by the BCI, while avoiding obstacles detected by the laser sensor. The general assumption is that the environment is unknown and dynamic (it can vary with time), which imposes a difficulty since precomputed maps and trajectories cannot be used. To deal with this problem, the navigation system implemented online modeling and dynamic planning capabilities [32], integrated into two modules: the model builder and the local planner (Figure 5).

a) Model Builder: The model builder integrates sensor measurements to construct a local model of the environment (static and dynamic parts) and to track the vehicle’s location. Free space and static obstacles are modeled by a ternary occupancy map. Dynamic objects are tracked using a set of extended Kalman filters. In order to accurately build both models, a technique is used to correct the robot’s position, update the map, and detect and track the moving objects around the robot [32]. The static map travels centered on the robot. This map has a limited but sufficient size to present the required information to the user (as described in the graphical interface section) and to compute the path so as to reach the selected target destination.

b) Local Planner: The local planner computes the local motion based on the hybrid combination of tactical planning and reactive collision avoidance [33], [34]. An efficient dynamic navigation function (D∗Lite planner [35]) is used to compute the tactical information (i.e., main direction of motion) required to avoid cyclic motions and trap situations. This function is well suited for unknown and dynamic sce- narios because it works based on the changes in the model computed by the model builder. The final motion of the vehicle is computed using the ND technique [36], which uses a case-based strategy, based on situations and actions to simplify the collision avoidance problem. This technique has the distinct advantage that it is able to deal with complex navigational tasks such as maneuvering in the environment within constrained spaces (e.g., passage through a narrow doorway). In order to facilitate comfortable and safe operation during navigation, shape, kinematics, and dynamic constraints of the vehicle are incorporated [37].

E. Integration Platform and Execution Protocol

The communications system performed the integration be- tween the brain-computer system and the robotic system. The software architecture was based on the TCP/IP protocol and on the client/server paradigm. It consisted of two clients (one for the brain-computer system and one for the robotic system) plus a link server that concentrated information flow and conferred scalability to the system. This design allows for teleoperation of the robot in any remote environment via Internet con- nection. The BCI client was integrated within the BCI2000 platform [27], cyclically executed every 30 ms, and communi- cated with the link server through an Internet connection. The robot client encapsulated the navigation system, synchronized the orders to the camera and to the navigation system, and communicated with the link server through a peer-to-peer (ad- hoc) wireless connection. This client also communicated with the robot hardware controllers using the player robot device interface [31]. Regarding the hardware components, the BCI client operated in a computer executing all the BCI software. The link server operated in a dedicated computer, an Intel Core2 Duo @ 2.10 GHz with Linux OS (Ubuntu distribution), equipped with an Ethernet and Wireless network card. The robot client operated in the computer embedded in the robot. The autonomous navigation system was a time-critical task, which was integrated in the robot computer within a threadbased system with time-outs to preserve the computation cycle (200 ms).

A typical execution of a navigation order is described next. The BCI infers the user desired goal location (8 Bytes of information), which is transferred via Internet from the BCI client to the link server. The link server transfers the goal location to the robot client via the ad-hoc wireless connection. The robot client makes the location available to the navigation system. Within a synchronous periodical task of 200 ms, the navigation system reads the location of the robot from the motor control system and the laser sensor, requests the robot odometry, executes the mapping and planning module, and sends the computed translational and rotational velocities to the robot controllers. While the robot is navigating, the robot client iteratively requests images from the camera, which are transferred to the BCI. Finally, when the robot reaches the final location, the navigation system triggers a flag to stop the image transfer process and sends three variables to the BCI to display the reconstruction of the environment: the map model (400 Bytes), the model location (12 Bytes), and the robot location within the map (12 Bytes). The upper boundary of the information transfer was set by the video transfer rate. The images captured by the camera were compressed to the jpeg standard format, obtaining an image size of approximately 30 KBytes. In the experimental sessions, 10 images/s were transferred, resulting in a transfer rate of approximately 300 KBytes/s, which is adequate for the typical bandwidth order of Internet networks.

a) Execution Protocol: The way the users interact with the system is modeled by a finite state machine with three states: Selection, Validation, and Execution. Initially, the state is Selection and the BCI develops the stimulation process while the robotic system is in an idle state. Then, the BCI selects an option and the state changes to Validation. In this state, a new stimulation process is developed and the BCI selects a new option. Only when the selected option is validation the previous selection is transferred to the robotic system, and the state changes to Execution. In this state, the robotic system executes the order (this will be referred as a mission). While the robot is executing the mission, the BCI is in an idle state (no stimulation process is developed) and live video captured by the robot camera is sent to the graphical interface. Once the robot accomplishes the mission, the state turns to Selection, video transfer stops (no interference stimuli, which could decrease BCI accuracy) and the BCI stimulation process starts again. Note that the validation option reduces the probability of sending incorrect orders to the robotic system, as BCI is always an uncertain channel.

EXPERIMENTAL METHODOLOGY

An experimental methodology was defined to carry out a technical evaluation of the system and to assess the degree of user adaptability. The experimental sessions were performed by healthy users in real settings. The recruitment of the participants and the experimental protocol are discussed next.

A. Participants

Inclusion and exclusion criteria were defined in order to obtain conclusions over a homogeneous population. The inclusion criteria were: (i) users within the age group 20 − 25 years of age; (ii) gender (either all women or all men); (iii) laterality (either all left-handed or all right-handed); and (iv) students of the University of Zaragoza. The exclusion criteria were: (i) users with history of neurological or psychiatric disorders; (ii) users under any psychiatric medication; and (iii) users with episodes of epilepsy, dyslexia or experiencing hallucination. Five healthy, 22 year-old, male right-handed students of the University of Zaragoza were recruited. They had neither utilized the telepresence system nor participated in BCI experiments before. The study was approved by the University of Zaragoza’s Institutional Review Board. All par- ticipants signed informed consents after being informed about the entire protocol.

B. Experiment Design and Procedures

The study was divided into two phases: (i) screening and training phase, and (ii) a brain-actuated telepresence phase. Both phases were carried out in the BCI Laboratory of the cognitive assessment forms. The training task consisted of a battery of online tests (facing the graphical interface without teleoperating the robot) to check whether the accuracy of the system was greater than a threshold value of 90%, qualifying the participant for the next phase. Duration of this phase was 3 hours per participant.

2) Evaluation of Brain-Actuated Telepresence: This phase consisted of a battery of online experiments with the telepresence system in order to carry out a technical evaluation of the system and to assess the degree of user adaptability. The experiments were carried out between the BCI Laboratory at the University of Zaragoza (Spain) and the University University of Zaragoza on different days. of Vilanova i la Geltru´ (Spain), separated by 260 km. Two

1) Evaluation of Screening and Training: This phase con- sisted of two tasks: (i) screening task to study the P300 response and validate the graphical interface design, and (ii) training task to calibrate the system and measure BCI accuracy. Initially, the visual aspects of the graphical interface were selected adapting the results of a parallel study [5]. Images were captured in black & white to preserve high saliency of stimuli; initial camera orientation was 0? pan and −11.5? tilt to provide a centered perspective of the environment starting approximately one meter in front of the robot. The final aesthetic factors of the visual display are shown in Figure 3. Stimulation process schedules were also set for both tasks according to [5]. The inter-stimulus duration was set to 75 ms and stimulus duration was set to 125 ms.

The screening task consisted of 8 offline trials to study the P300 response in the EEG. In each trial, the participants had to attend to a predefined sequence of 10 targets. After execution, participants were asked to fill out neuropsychological andtasks were designed, which combined navigation and visual exploration in unknown scenarios and under different working conditions. Each participant had to perform two trials for each task. Task 1 involved complex navigation in constrained spaces with an active search for two visual targets. Task 2 involved navigation in open spaces with an active search for one visual target. The maps of the circuits are shown in Figure 6. The maps were the only information of the remote environments shown to the participants, which had never been physically there. Regarding the stimulation process schedules, inter-stimulus duration was set to 75 ms and stimulus duration was set to 125 ms. After each trial, the participants were asked to fill out neuropsychological and cognitive assessment forms, one for each operation mode of the system. Duration of this phase was 4 hours per participant. It should be noted that execution of tasks was not counterbalanced (the two trials of Task 1 were performed before the trials of Task 2), thus the obtained results containing inter-task comparisons (especially in the users’ behavior evaluation) may reflect learning effects.

RESULTS AND EVALUATION

This section reports the results obtained during the experi- mental phases. Phase 1 was composed by a screening and a training task. Regarding the screening task, visual inspection of the recorded EEG data showed that the P300 potential was elicited for all participants. Furthermore, participants reported high satisfaction in the psychological assessments. Thus, the graphical interface design was validated. Regarding the training task, the pattern-recognition strategy was trained and the participants performed the online tests. All participants achieved more than 93% BCI accuracy; and thus, all were qualified to carry out the next phase.

Phase 2 consisted of execution of the predefined teleopera- tion tasks, which combined navigation and visual exploration. First, the participants performed 4 offline trials to train the pattern-recognition strategy. The number of sequences was customized for each participant according to the results pro- vided by the classifier in this calibration process (Figure 4). The number of sequences was set to the minimal number that allowed the participant to achieve a theoretical accuracy higher than 90%. Then, the experiments were performed. Technical evaluation of the telepresence system and the behavior study of users are described next. The overall result was that all participants were able to complete the designed tasks reporting no failures, which shows the robustness of the system and its feasibility to solve tasks in real settings where joint navigation and visual exploration were needed. Furthermore, participants showed great adaptation.

A. Technical Evaluation The technical evaluation consisted of a global evaluation of the brain-actuated telepresence system and a particular evalu- ation of the brain-computer system and the robotic system.

1) Global Evaluation: According to [38], [5], the following metrics were proposed: (i) task success; (ii) number of collisions of the navigation system; (iii) time elapsed until completion of task; (iv) length of path traveled by the robot; (v) number of missions1 to complete the task; (vi) BCI accuracy; (vii) BCI selection ratio: ratio between time spent selecting orders and total time to complete the task; and (viii) navigation ratio: ratio between time spent in robot navigation mode and the total time to complete the task,

which is complementary to the exploration ratio. Results are summarized in Table I. All participants succeeded to perform all trials reporting no collisions, highlighting the robustness of the system. Time elapsed, path length and number of missions were very similar for all participants indicating a similar performance among them (these metrics will be further discussed from the point of view of the participants in the users’ behavior section). The real robot trajectories are displayed in Figure 6. Although there were variations in BCI accuracy, the BCI interaction was satisfactory as the BCI accuracy was always higher than 78%, achieving a mean performance of 90%. BCI selection ratio was on average 52%, which shows the great importance of BCI accuracy in the global system performance. Regarding the ratio of usage of the operation modes, both operation modes were used to complete the tasks. It can also be inferred that the system provided enough functionalities to the users, so that they were able to adapt to the different working conditions of the tasks. Task 1 presented a higher exploration ratio because it involved more complex visual explorations. Task 2 presented a higher navigation ratio because it involved navigation in open spaces and simpler visual exploration.

In summary, results were very encouraging because they showed the feasibility of the technology to solve tasks combin- ing navigation and visual exploration under different working conditions. Furthermore, participants were na¨?ve to BCI usage and received a short briefing on the system operation. The system was calibrated in less than an hour. 2) Brain-Computer System: The evaluation of the brain- computer system was divided into two parts: evaluation of the pattern-recognition strategy performance (BCI accuracy) and evaluation of the visual display design. Based on [38], [5], the following metrics were proposed: (i) total errors; (ii) reusable errors; (iii) real BCI accuracy; (iv) practical BCI accuracy: BCI accuracy computed using the correct selections plus reusable errors; (v) selections per minute; (vi) selections per mission (usability rate); (vii) missions per minute; (viii) number of sequences; (ix) information transfer rate (ITR) according to the Wolpaw definition [39] 2 ; (x) number of errors caused by interface misunderstandings; and (xi) option usage frequency. Results are summarized in Table II, except for the option usage frequency, which is shown in Table III.

a) BCI Accuracy Evaluation: Participants were in- structed to report an error to the experimenter through a small movement of the right hand index finger. In some cases, although the BCI detects a non-desired target, the target is reused to complete the task (common situation in open spaces, where a task can be solved in many different ways). These errors are referred to as reusable errors and they do not increment the time to set a mission to the system. The distinction between a reusable error and a non-reusable error was made by the experimenter and then verified with the participants at the end of the experiment (when needed). Real BCI accuracy was high, above 85% on average. Reusable errors result in a practical BCI accuracy higher than real. Practical accuracy was on average 90%. The BCI system set only two incorrect missions in all executions, representing

0.78% of all missions (theoretical probability of this situation was 0.3%). The number of sequences was customized per participant according to their accuracy, between 6 and 10. The number of sequences determined the number of selections per minute, which was approximately 4. The usability rate was slightly greater than 2 (ideally it is equal to 2, i.e., a mission needs at least one selection plus validation) due to BCI errors and interface misunderstandings by the user. The number of missions per minute, determined by the number of selections per minute and the usability rate, was on average 1.65. The ITR of the BCI system was on average 15 bits/min.

b) Visual Display Design Evaluation: The design of the interface was valid, as participants achieved tasks with only a short briefing on its functionalities. There was only one incorrect selection due to interface misunderstandings, which arose at the very end of one trial (the participant set an unreachable mission, located behind the goal wall). The usage frequency for all options in the interface and participants shows that all functionalities were used, thus indicating that there were no useless options. Furthermore, it also suggests a usable visual display design. Change mode option was used once per trial in each operation mode due to the requirements of the designed tasks (participants changed to the exploration mode to visualize the targets and then switched to navigation mode to complete the tasks). Note that alignment and change mode options in the exploration mode were complementary, since both options allowed the participant to change to the navigation mode. The home option in the exploration mode was only used only once throughout all the experiments, probably because in the pre-defined tasks the home option did not provide an important advantage with regard to grid destinations. Refresh option was little used because of the execution of constrained tasks; this option could be useful in more uncontrolled tasks to increase the interaction capabilities. In summary, these results show a satisfactory integration between the visual display and the designed stimulation pro- cess as the participants successfully completed all trials with high BCI accuracies. Furthermore, the graphical interface was usable and easy to understand. The system presents low ITRs, which is a common problem of all event-related potential approaches, but it is in part overcome by the adoption of a shared-control approach.

3) Robotic System: Based on [38], [5], a set of metrics was proposed to evaluate the two operation modes of the robotic system: (i) number of navigation missions; (ii) length traveled per mission; (iii) mean velocity of the robot; (iv) mean clearance (average of the minimum distances to the obstacles); (v) minimum clearance (minimum distance to the obstacles); (vi) number of camera exploration missions; and (vii) total angle explored by the camera. Results are summarized in Table IV, which is divided into two sections, each relevant to an operation mode.

Regarding the navigation mode, a total of 177 navigation missions were carried out without collisions, with a total length of 325 meters and a mean velocity of 0.08 m/s (10 times less than usual human walking velocity). Mean velocity and length traveled per mission were greater in Task 2 than in Task 1, which denotes that the navigation system was able to deal with the different environmental conditions, resulting in a velocity increase in open spaces (Task 2) and a reduction when maneuverability became more important (Task 1). Mean and minimum clearances show that the vehicle carried out obstacle avoidance with safety margins, which is one of the typical difficulties in autonomous navigation [34]. Regarding the exploration mode, a total of 79 missions were carried out, exploring a total angular distance of 3.2 radians.

In general, the performance of the robotic system was remarkable as the navigation missions were successfully executed reporting no failures. The exploration system provided a good visual feedback of the remote environment and sufficient functionalities for active exploration.

B. Users’ Behavior Evaluation

An evaluation of the users’ behavior was carried out to measure the degree of participants adaptability to the brainactuated telepresence system. Three studies were defined to achieve such an objective: (i) execution analysis, to study the performance of participants; (ii) activity analysis, to study the interaction strategy with the robot; and (iii) psychological assessment, to study the participants’ workload, learnability and level of confidence.

1) Execution Analysis: A set of metrics based on [38], [5], were used: (i) task success; (ii) number of missions; (iii) path length traveled by the robot; (iv) time elapsed until completion of task; and (v) practical BCI accuracy. Results are summarized in Table V, which shows the two trials per participant and task.

The number of missions is an indicator of the intermediate steps required to complete the tasks. Although this metric is strongly related to the interaction strategy (discussed in the next subsection), it can be inferred that some participants presented a more efficient mission selection. Participants 1 and 4 showed a more efficient mission selection in Task 1; while participants 2, 3 and 5 presented a more efficient selection in Task 2. This metric suggests that these participants could be divided into two groups, according to the way they adapted to the environmental conditions. One group adapted better to the constrained environment of Task 1, and the other group adapted better to the open spaces in Task 2. Path length is another metric of individual performance in the use of the telepresence system. Participants 3 and 5 presented shorter path lengths in both tasks, showing a better adaptation to the automation capabilities of the system. Execution time involves BCI accuracy and mission selection performance, which are factors that can increase the number of selections required to complete the tasks. Due to the large amount of time needed to select an option with the BCI (13 s on average), the lower BCI accuracies lead to the longer execution times. Participants 2 and 4 presented lower BCI accuracies and consequently longer execution times. The fact that all participants succeeded in completing the tasks shows that all participants successfully adapted to the system, which is a good indicative to explore the transition of the technology towards end users.

2) Activity Analysis: The interaction strategy of the par- ticipants when teleoperating the robot is studied. Regard- ing robotic devices that provide automation facilities, two types of interaction strategies can be applied: supervisory- oriented interaction and direct control-oriented interaction [40]. Supervisory-oriented interaction extensively explores the automation capabilities (mainly trajectory planning and obsta- cle avoidance in navigation mode) minimizing user interven- tion. Direct control-oriented interaction is characterized by an increased user intervention, minimizing the use of automation capabilities. In the concrete case of the developed system, supervisory-oriented interaction will be characterized by a high number of far destinations in the navigation, while direct control-oriented interaction will be characterized by a higher number of near-range destinations or left/right turn selections. The following metrics, adapted from [38], [5], were defined to study whether the participants followed different interaction strategies in the two tasks: (DA ) activity discriminant: ratio between goal selections minus total of turn selections and the total number of selections; (PM ) path length per mission; (TM ) robot motion time per mission; (CA ) control activity descriptor: ratio between turn selections and total number of selections; (SA ) supervisory activity descriptor: ratio between first grid row destinations and total number of selections. According to the proposed metrics, high values of activity discriminant (DA ), path length per mission (PM ) and robot motion time per mission (TM ) indicate a tendency towards supervisory-oriented interaction. On the contrary, low values indicate a tendency towards control-oriented interaction. Furthermore, control-oriented interaction is also characterized by high values of CA , whereas supervisory interaction is characterized by high values of SA . Results are summarized in Table VI.

Values of DA , PM , and TM in Task 1 were comparatively lower than in Task 2, suggesting control interaction in Task 1 and supervisory interaction in Task 2. In Task 1, participants exhibited a propensity towards control interaction as CA values were higher in comparison to those values in Task 2. In Task 2, participants showed a propensity towards supervisory interaction as SA values were higher in comparison to those of Task 1. In summary, these results suggest that the participants adapted to the different working conditions of each task. Task 1 involved complex maneuverability and participants presented control-oriented interaction; Task 2 involved more simple navigation in open spaces and participants presented supervisory-oriented interaction.

3) Psychological Assessment: This subsection studies the adaptability of the participants to the telepresence system from a psychological point of view. The following metrics were used: (i) workload based on effort: amount of effort exerted by the participant during the tasks; (ii) learnability: easiness to learn how to use the system during the tasks; and (iii) level of confidence: confidence experienced by the participant during the tasks. Results obtained from the questionnaires (filled out after each trial by the participants) are outlined in Figure 7.

Participants 2 and 5 reported less workload than partici-pants 1, 3, and 4. All participants reported higher values of workload in Task 1. This result might be due to the fact that Task 1 involved more complex maneuverability. Regarding the learnability metric, participant 1 presented difficulties in learning how to solve the first task, but showed a great improvement in Task 2. This participant may have initially found the telepresence system complex. Regarding the level of confidence, participant 4 showed the lowest values, which might be explained by his low BCI accuracy (see Table V). In general, these three metrics showed a great improvement in Task 2 with regard to Task 1. An improvement in metrics can be observed in the second trial with regard to the first one (within each task), where the first trial may be seen as an adaption trial to complete the new task. These results suggest high adaptability of the participants to the telepresence system: participants experienced less effort, higher learning skills and felt more confident during the use of the system. However, these results should be interpreted with caution since tasks were not counterbalanced and thus they may reflect a learning effect.

All users participating in the experimental methodology were able to accomplish two different tasks, which covered typical navigation situations such as complex maneuverability and navigation in open spaces. The interaction with the BCI was satisfactory as na¨?ve BCI users obtained high accuracies (88% in mean) with short calibration time (less than an hour). The functionalities of the robotic system were sufficient to complete the tasks. The navigation system implemented task- level primitives that incorporated real-time adaptative motion planning and model construction, and thus it was able to deal with non-preprogrammed and populated scenarios. As demonstrated in other applications [5], [38], the navigation system demonstrated to be robust (the robot received 177 missions without any failure). The integration between the BCI system and the robotic system was satisfactory, achieving an overall high performance of the system. The evaluation of the users’ behavior suggested a high degree of adaptability to the telepresence system.

CONCLUSIONS

This paper has reported a synchronous P300-based BCI teleoperation system that can provide users with presence in remote environments through a mobile robot, both con- nected via Internet. The shared-control strategy is built by the BCI decoding of task-related navigation or visual exploration orders, autonomously executed by the robot. This design overcomes low information transfer rates, avoids exhausting mental processes, and explicitly avoids delay problems in the control loop caused by Internet communication (as happens in teleoperation systems with continuous control)

One feature of the current system is that no continuous feedback is perceived when the user is interacting with the BCI. With this feature, the user is never exposed to external stimuli while interacting with the BCI, and thus it allows to employ a methodology to explore the BCI accuracy in controlled scenarios. However, this certainly limits the de- gree of presence and shared-control interaction, and further investigation is required to understand the effects that the alleviation of this restriction could have. In order to increase the degree of presence, the adoption of an asynchronous P300 control to support an idle state would be an improvement, as given in [41]. Another improvement could be the adoption of a multi-paradigm BCI by the inclusion of an asynchronous error potentials detection protocol [42]. This improvement could have two effects. On one hand, this could reduce the interaction required by the BCI to control the robot (note that 50% of the total time is spent in decoding the BCI intentions due the safety nature of the device, implemented in the execution protocol through a validation step) by removing the validation protocol as incorrect P300 selections could be detected. On the other hand, the inclusion of this protocol could increase the shared-control interaction and system safety by detecting possible unrecognized risks by the robot’s sensors while navigating. However, the adoption of such solutions could impose the typical drawbacks of asynchronous proto- cols: lower accuracies, much higher calibration and training time with the user, and higher cognitive effort.

This study could be considered as a step towards the development of new telepresence-oriented systems using BCIs and mobile robots in which navigation and visual exploration problems are solved. Thus it could allow the designers to focus on specific interaction functionalities (e.g., incorporate bidirec- tional communication along the lines of a videoconference), which might be dependent on the patient pathology and needs. Although the utility of this technology was demonstrated for healthy users, the final objective is to bring these possibilities closer to patients with neuromuscular disabilities, which is the direction of work in the near future [28].

Tables at a glance


Table 1	Table 2	Table 3

Table 4	Table 5	Table 6

Figures at a glance


Figure 1	Figure 2	Figure 3	Figure 4

Figure 5	Figure 6	Figure 7

References

J. R. Wolpaw, D. J. McFarland, G. W. Neat, and C. A. Forneris, “An EEG-based brain-computer interface for cursor control,” Electroencephalographyand clinical neurophysiology, vol. 78, no. 3, pp. 252–259, 1991.
N. Birbaumer, N. Ghanayim, T. Hinterberger, I. Iversen, B. Kotchoubey, A. Ku¨ bler, J. Perelmouter, E. Taub, and H. Flor, “A spelling device forthe paralyzed,” Nature, vol. 398, pp. 297–298, 1999.
A. Karim, T. Hinterberger, and J. Richter, “Neural Internet: Web surfing with brain potentials for the completely paralyzed,” Neurorehabilitationand Neural Repair, vol. 20, no. 4, pp. 508–515, 2006.
J. del R. Milla´n, F. Renkens, J. Mourino, and W. Gerstner, “Noninvasive brain-actuated control of a mobile robot by human EEG,” IEEE Trans.on Biomedical Eng., vol. 51, no. 6, pp. 1026–1033, 2004.
Iturrate, J. M. Antelis, A. Ku¨ bler, and J. Minguez, “A non-invasive brain-actuated wheelchair based on a P300 neurophysiological protocoland automated navigation,” IEEE Trans. on Robotics, vol. 25, no. 3, pp. 614–627, 2009.
G. Vanacker, J. del R. Milla´n, E. Lew, P. W. Ferrez, F. G. Moles, J. Philips, H. Van Brussel, and M. Nuttin, “Context-based filtering forassisted brain-actuated wheelchair driving,” Computational Intelligence and Neuroscience, vol. 2007, pp. 3–3, 2007.
T. Luth, D. Ojdanic, O. Friman, O. Prenzel, and A. Graser, “Low level control in a semi-autonomous rehabilitation robotic system via a braincomputerinterface,” In proc. IEEE 10th Int. Conf. on Rehabilitation Robotics, pp. 721–728, 2007.
B. Rebsamen, E. Burdet, C. Guan, H. Zhang, C. L. Teo, Q. Zeng, C. Laugier, and M. H. Ang Jr., “Controlling a wheelchair indoors usingthought,” IEEE Intelligent Systems, vol. 07, no. 2, pp. 1541–1672, 2007.
Ferreira, R. L. Silva, W. C. Celeste, T. F. Bastos, and M. Sarcinelli, “Human-machine interface based on muscular and brain signals appliedto a robotic wheelchair,” Journal of Physics: Conference Series, vol. 07, pp. 1541–1672, 2007.
Iturrate, L. Montesano, and J. Minguez, “Single trial recognition of error-related potentials during observation of robot operation,” In ProcInt. Conf. of the IEEE Eng. in Medicine and Biology Society (EMBC), 2010.
J. Bell, P. Shenoy, R. Chalodhorn, and R. P. N. Rao, “Control of a humanoid robot by a noninvasive brain-computer interface in humans,”Journal of Neural Eng., vol. 5, no. 2, pp. 214–220, 2007.
G. Pfurtscheller, G. Mu¨ ller, J. Pfurtscheller, H. Gerner, and R. Rupp, “‘Thought’ - control of functional electrical stimulation to restorehand grasp in a patient with tetraplegia,” Neuroscience Letters, vol. 351, no. 1, pp. 33 – 36, 2003.
M. Tavella, R. Leeb, R. Rupp, and J. del R. Milla´n, “Towards natural non-invasive hand neuroprostheses for daily living,” In Proc. Int.Conf. of the IEEE Eng. in Medicine and Biology Society (EMBC), 2010.
Chella, E. Pagello, E. Menegatti, R. Sorbello, S. M. Anzalone, F. Cinquegrani, L. Tonin, F. Piccione, K. Prifitis, C. Blanda, E.Buttita, and E. Tranchina, “A BCI teleoperated museum robotic guide,” In Proc. Int. Conf. on Complex, Intelligent and Software Intensive Systems,pp. 783–788, 2009.
Akce, M. Johnson, and T. Bretl, “Remote teleoperation of an un- manned aircraft with a brain-machine interface: Theory and preliminaryresults,” In Proc. IEEE Int. Conf. on Robotics and Automation (ICRA), pp. 5322–5327, 2010.
L. Tonin, E. Menegatti, M. Cavinato, C. Avanzo, M. Pirini, A. Merico, L. Piron, K. Priftis, S. Silvoni, C. Volpato, and F. Piccione,“Evaluation of a robot as embodied interface for brain computer interface systems,” Int. Journal of Bioelectromagnetism, vol. 11, no. 2, pp. 97–104,2009.
L. Tonin, R. Leeb, M. Tavella, S. Perdikis, and J. del R. Milla´n, “The role of shared-control in BCI-based telepresence,” In Proc. IEEE IntConf. on Systems, Man, and Cybernetics, 2010.
Ku¨ bler and N. Birbaumer, “Brain-computer interfaces and commu- nication in paralysis: extinction of goal directed thinking in completelyparalysed patients?” Clinical Neurophysiology, vol. 119, no. 11, pp. 2658 – 2666, 2008.
F. Gala´n, M. Nuttin, E. Lew, P. W. Ferrez, G. Vanacker, J. Philips, and J. del R. Milla´n, “A brain-actuated wheelchair: asynchronous andnon-invasive brain-computer interfaces for continuous control of robots,” Clinical Neurophysiology, vol. 119, no. 9, pp. 2159–2169, 2008.
R.Wolpaw, N. Birbaumer, D. J. McFarland, G. Pfurtscheller, and T. M. Vaughan, “Brain-computer interfaces for communication and control,”Clinical Neurophysiology, vol. 113, no. 6, pp. 767–791, 2002.
Minguez, F. Lamiraux, and J. P. Laumond, “Motion planning and obstacle avoidance,” in Springer Handbook of Robotics, B. Sicilianoand O. Khatib, Eds. Springer Berlin Heidelberg, 2008, pp. 827–852.
Escolano, J. M. Antelis, and J. Minguez, “Human brain-teleoperated robot between remote places,” In Proc. IEEE Int. Conf. on Roboticsand Automation (ICRA), 2009.
S. Sutton, M. Braren, J. Zublin, and E. R. John, “Evoked potential correlates of stimulus uncertainty,” Science, vol. 150, no. 3700, pp.1187–1188, 1965.
] S. H. Patel and P. N. Azzam, “Characterization of N200 and P300: Selected studies of the event-related potential,” Int. Journal of MedicalSciences, vol. 2, no. 4, pp. 147–154, 2005.
del R. Milla´n, R. Rupp, G. R. Mu¨ ller-Putz, R. Murray-Smith, C. Giugliemma, M. Tangermann, C. Vidaurre, F. Cincotti, A. Ku¨bler, R. Leeb, C. Neuper, K. R. Mu¨ ller, and D. Mattia, “Combining brain- computer interfaces and assistive technologies: state-of-the-art andchal- lenges,” Frontiers in Neuroscience, vol. 4, 2010.
J. Krusienski, E. W. Sellers, F. Cabestaing, S. Bayoudh, D. J. McFarland, T. M. Vaughan, and J. R. Wolpaw, “A comparison ofclassification techniques for the P300 speller,” Journal of Neural Eng., vol. 3, no. 4, pp. 299–305, 2006.
G. Schalk, D. J. McFarland, T. Hinterberger, N. Birbaumer, and J. R. Wolpaw, “BCI2000: A general-purpose brain-computer interface(BCI) system,” IEEE Trans. on Biomedical Eng., vol. 51, no. 6, pp. 1034–43, 2004.
Escolano, A. Ramos, T. Matuz, N. Birbaumer, and J. Minguez, “A telepresence robotic system operated with a P300-based brain-computerinterface: initial tests with ALS patients,” In Proc. Int. Conf. of the IEEE Eng. in Medicine and Biology Society (EMBC), 2010.
A. Farwell and E. Donchin, “Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials,”EEG Clinical Neurophysiology, vol. 70, no. 6, pp. 510–523, 1988.
J. Krusienski, E. W. Sellers, D. J. McFarland, T. M. Vaughan, and J. R. Wolpaw, “Toward enhanced P300 speller performance,” Journalof Neuroscience Methods, vol. 167, no. 1, pp. 15–21, 2008.
T. H. Collett, B. A. MacDonald, and B. P. Gerkey, “Player 2.0: Toward a practical robot programming framework,” In Proc. AustralasianConf. on Robotics and Automation (ACRA), 2005.
Montesano, J. Minguez, and L. Montano, “Modeling dynamic sce- narios for local sensor-based motion planning,” Autonomous Robots,vol. 25, no. 3, pp. 231–251, 2008.
J. Minguez and L. Montano, “Sensor-based robot motion generation in unknown, dynamic and troublesome scenarios,” Robotics and AutonomousSystems, vol. 52, no. 4, pp. 290–311, 2005.
Montesano, J. Minguez, and L. Montano, “Lessons learned in integration for sensor-based robot navigation systems,” Int. Journalof Advanced Robotic Systems, vol. 3, no. 1, pp. 85–91, 2006.
Ranganathan and S. Koenig, “A reactive architecture with planning on demand,” In Proc. IEEE Int. Conf. on Robotics and Automation(ICRA), pp. 1462–1468, 2003.
J. Minguez and L. Montano, “Nearness diagram (ND) navigation: collision avoidance in troublesome scenarios,” IEEE Trans. OnRobotics and Automation, vol. 20, no. 1, pp. 45–59, 2004.
——, “Extending reactive collision avoidance methods to consider any vehicle shape and the kinematics and dynamic constraints,” IEEE Trans.on Robotics, vol. 25, no. 2, pp. 367–381, 2008.
Montesano, M. Diaz, S. Bhaskar, and J. Minguez, “Towards an intelligent wheelchair system for cerebral palsy subjects,” IEEETrans. on Neural Systems and Rehabilitation Eng., vol. 18, no. 2, pp. 193–202, 2010.
J. R. Wolpaw, N. Birbaumer, W. J. Heetderks, D. J. McFarland, P. H.Peckham, G. Schalk, E. Donchin, L. A. Quatrano, C. J. Robinson, and T. M. Vaughan, “Brain-computer interface technology: a review of the firstint. meeting.” IEEE Trans. on Rehabilitation Eng., vol. 8, no. 2, pp. 164–173, 2000.
Baker, R. Casey, B. Keyes, and H. A. Yanco, “Improved interfaces for human-robot interaction in urban search and rescue,” In Proc.IEEE Int. Conf. on Systems, Man and Cybernetics, vol. 3, pp. 2960–2965, 2004.
H. Zhang, C. Guan, and C. Wang, “Asynchronous P300-based brain- computer interfaces: a computational approach with statistical models,”IEEE Trans. on Biomedical Eng., vol. 55, no. 6, pp. 1754–1763, 2008.
Buttfield, P. W. Ferrez, and J. del R. Milla´n, “Towards a robust BCI: error potentials and online learning,” IEEE Trans. on NeuralSystems and Rehabilitation Eng., vol. 14, no. 2, pp. 164–168, 2006.