2004 P300 Data Analysis Competition

[ Introduction | Recent Competition | Schedule | Submission | Data Sets | Judging and Winning | Results | People ]

Introduction

What is a BCI?

A brain computer interface is a communication system in which users send information using brain activity alone. All other interfaces, such as keyboards, mice, or voice systems, require people to send information through peripheral nerves and muscles. Disabled individuals may have difficulty using conventional interfaces. Some people suffer from "locked in syndrome," meaning they are completely unable to control any muscles. For such users, a BCI is the only hope for ever communicating with loved ones, controlling even simple devices like televisions or lamps, or otherwise expressing oneself.

BCIs can be divided into two general categories. Most BCIs are noninvasive BCIs, meaning that no surgery is required. They instead utilize EEG activity measured at the scalp with an electrode cap. This sort of technology has been in widespread use for many decades. Electrode caps are safe, painless, and fairly easy to apply and remove. Some BCI, called invasive BCIs, instead record the brain's activity via sensors placed on the surface of the brain or even inside it. Obviously, the neurosurgery required for this implantation is a very serious procedure, and is only performed when medically necessary. Invasive BCIs can ultimately provide more information about brain activity than noninvasive BCIs, because the EEG signal recorded at the scalp is badly smeared by the skull and other tissues. However, the surgery is very difficult and expensive, and the jury is still out on the long term safety and effectiveness of implanted systems.

Here are a few websites with more information about BCIs:

BrainLab: This page is a bit old, but has some good information. Dr. Allison's PhD thesis can be found under "other writings," along with some slideshows providing a nice introduction to BCIs and other files.

UCSD: This is the lab where Dr. Allison worked as a graduate student. This site contains a slew of links as well as slideshows from Dr. Allison's class last summer about BCIs. Note these slideshows are meant for university students, while the ones on the BrainLab are meant for anyone.

Wadsworth Research Center: This site contains information about BCI2000, an excellent program developed by the Wadsworth Research Center that is being used by many research groups. It also contains some videos of BCIs in operation.

There are many more sites out there! Please see the links on the UCSD website, or just search the web for "brain computer interface" or "brain machine interface."

What is a P300 BCI?

There are several different types of BCI systems, each defined by the type of brain activity they utilize. They all have various advantages and drawbacks in terms of speed, training time, ease of use, whether surgery is needed, and other factors. A P300 BCI uses a type of brainwave called the P300. P300 BCIs are relatively fast and easy to use, require no training or training or surgery, and work with nearly all adults. The main drawback of P300 BCIs is that they can only provide one binary signal - that is, a YES or a NO. Other BCIs, such as mu BCIs, can provide provide more information.

Graphical representation of a P300 BCI. A user wears an electrode cap that measures the brain's response to flashes. She pays attention to flashes of a certain target letter (such as "K") while ignoring other flashes. The user's brain produces a different response to the flashes that contain the target letter (red line) than other letters (blue line). By determining which flashes produced a red line The more effectively a pattern recognition can discriminate the red line from the blue line, the better the BCI. This difference may seem obvious in the graph above, but this graph was derived by averaging together dozens of trials. It's much harder with fewer trials.

Conference objectives: Why have this competition?

One very important factor in determining the speed and accuracy of a P300 BCI (or any BCI) is the pattern recognition approach applied to the EEG data. In the case of P300 BCIs, a better pattern recognition system could recognize the P300 more quickly and accurately. This is not the first effort to explore different approaches. Farwell and Donchin (1988), Donchin et al (2000), Bayliss' doctoral thesis (2001), Meinicke et al. (2002), and Sajda et al (2003)  all compared different techniques. These papers generally concluded that different approaches can yield significantly different results. As there remain many approaches that have not been tested, it is likely that this competition will further elucidate the best approaches. This competition also utilizes a wider variety of P300 datasets, from different types of P300 BCIs, with both healthy and disabled subjects. Furthermore, all P300 BCI papers used healthy subjects. There are currently no papers published regarding P300 BCIs in locked in patients - the people who need these systems most.

Of course, it is impossible to ever determine the best pattern recognition approach with finality. The old gunslingers in the American Wild West had a saying, "There's always someone faster on the draw." Similarly, there will always be a better pattern recognition algorithm out there. This fact should not be discouraging to those interested in the competition. On the contrary, competitions such as these help identify new directions for research, and spur increased attention to the field.

Comparison to Recent Competition

A BCI data analysis competition was held in 2003, organized by Dr. Benjamin Blankertz. See his competition web page for more information.

The current competition differs from the 2003 competition in several ways:

1) All datasets in this competition involve P300 BCIs. The 2003 competition contained only one dataset with P300 data.

2) Further, the datasets in this competition are from different types of P300 BCIs, such as the classic "Donchin speller," Allison's "multiple flash approach," the Wadsworth Research Center's "Sequential P3 BCI," and other systems. These BCIs yield somewhat different P300s that may be best studied with different types of approaches.

3) Because there will be several data sets from different labs, the number of channels varies from 5 to 64. It is possible that different approaches will perform differently depending on the number of channels available. Though probably insignificant, the impedances, referencing, and sampling rates vary as well.

4) This competition will include P300 BCI data from locked in patients. No such data were available at the 2003 competition.

5) The previous competition had the problem of having five different winners for the same P300 dataset. This is because many of the competitors were so good, they attained 100% accuracy with all of the data available! We are taking three steps to prevent this. One is the aforementioned use of several different datasets, and thus a lot more data. Some of the data, such as our ALS patient data, is much more challenging. The second is that, in the previous competition, fifteen single trials were made available. In this competition, differing numbers of single trials will be made available, including in some cases only one single trial. It is highly unlikely that any competitor will attain 100% accuracy across all datasets. Finally, the judging rules below are different from the last competition and make a tie absolutely impossible.

6) As with any new competition, we expect to have some different competitors and approaches in this competition.

This competition is not meant to compete with Dr. Blankertz's 2003 competition nor his upcoming data analysis competition. On the contrary, both Dr. Blankertz and I feel these two competitions complement each other well. Participants in this competition are encouraged to contact Dr. Blankertz regarding his upcoming competition. For more information, please see his web page for his 2003 competition.

Schedule

Sample data are currently available for download on our FTP server. Please contact Dr. Allison for more information. More sample data will be placed online before the competition begins. These dates are tentative.

SOMETIME: Competition begins! All data will be online.
SOMETIME AFTER THAT:  Deadline for submission.
A LITTLE AFTER THAT: Announcement of the results on this web site.

Submission

• There is no prerequisite for entry. New people, ideas, and approaches are always welcome!
• To enter the competition, please contact Dr. Allison, whose contact information is below. You will receive information to access our FTP server. By entering the competition, you agree to all rules on this web page.
• One researcher/research group may submit results to one or to several data sets. Winners will be announced for each data set separately. However, researchers must submit to at least three different data sets to compete for the title of grandmaster. Please see the "judging rules" below to see how winners will be determined.
• One researcher may NOT submit multiple results to one data set. She/he has to decide for her/his favorite one. However ... (see next point)
• From one research group multiple submissions to one data set are possible. The sets of involved researchers do not have to be disjoint, but (1) the 'first author' (contributor) should be distinct, and (2) approaches should be substantially different.
• All submissions should be sent to Dr. Allison.
• We send confirmations for each submissions we get. If you do not receive a confirmation within 3 days please resend your email and inform Dr. Allison.
• The submission must include the estimated labels, names and affiliations of all involved researchers and a short note on the involved processing techniques.
• Anyone submitting a dataset must agree to sign the "grandmaster" certificate unless they are the overall winner (see Judging and Winning, below).
• SPECIAL EXCEPTION: The current rules may create an ethical quandary for P300 data from ALS patients. What if a participant develops a substantially better pattern recognition approach for ALS patients very soon after the competition begins? It would be inappropriate to keep this development secret for months until the submission deadline, as it could be of immediate benefit. Therefore, for the ALS dataset only, the submission rules differ from the above in one way. One researcher MAY submit different results to the ALS data set, with the understanding that these results may be field tested in an ALS patient. However, only the last submission will be judged for the competition. All submissions prior to the final submission will be ignored for purposes of judging. Furthermore, to prevent an unfair advantage for that competitor, the results of the field test will not be shared with anyone outside of this laboratory (except the patient and his/her caretaker) until after the submission deadline. This policy may be frustrating for competitors eager to hear advance results of their submission, but this is the only way to preserve the integrity of this competition.

Data sets

Each data set (except the ALS data set) will contain data from at least three different healthy subjects. Each subject's directory will contain two top level directories, named "labelled" and "unlabelled." Each of these two directories will contain at least three subdirectories, named 8, 3, and 1. Each of these numbered subdirectories will contain at least 10 different runs. The numbers 8, 3, and 1 reflect the number of single trials contained in its daughter subdirectories. Thus the subdirectories under the "1" directory contains only one trial. Within each of the other two directories, the single trials will all be from the same run, within a few minutes of each other. That is, the "8" directories will contain groups of 8 single trials from the same run, though the "3" and the "1" directories will each feature data from a different run.

Here is a graphical representation of this directory structure:

                                                                                            SUBJECT 1

                                                                                         /                           \

                                                                                labelled                    unlabelled

                                                                                /       |      \                  /       |      \  

                                                                               8       3       1            8        3        1

                                                                            /  |  \    /  |  \    /  |  \     /  |  \    /  |  \    /  |  \

                                         RUN NUMBER       1  2  3  ... etc

The "labelled" directory serves as a training set, and the "unlabelled" directory is the test set. These directories will never feature the exact same data, since that would eliminate any challenge in identifying the training set. The goal of each competitor is to identify whether each of the subdirectories in the "unlabelled' directories contain a P300 - that is, whether they reflect EEG evoked by a target flash or an ignored flash.

In the case of datasets with more than one condition (sets 2, 3, 4, and 6), each directory will also specify the condition. For example, data set 2 will specify whether the data are from the single or multiple flash condition and the target probability.

Data set I: 6 x 6 Donchin matrix
provided by the BrainLab at Georgia State University

Data set 2: 8 x 8 matrix with single vs. multiple flashes
provided by the Cognitive Neuroscience Laboratory at UC San Diego

Data set 3: 4 x 4 and 12 x 12 matrix  
provided by the Cognitive Neuroscience Laboratory at UC San Diego

Data set 4: Sequential BCI for answering questions
provided by the BrainLab at Georgia State University

Data set 5: Sequential BCI for answering questions (ALS patients)
provided by the BrainLab at Georgia State University

Data set 6: 1 x 6 matrix with robot control icons
provided by the BrainLab at Georgia State University

Data set 7: 6 x 6 Donchin matrix under extreme conditions
provided by the BrainLab at Georgia State University

Judging and Winning

The following rules will apply to each data set.

Each submission will earn one point for each correctly identified numbered subdirectory. For example, if data set 1 contained the minimum number of unlabelled directories (3 per each of 3 subjects, each containing 10 runs), a maximum of 90 points could be earned. Most data sets will contain well above the minimum number of unlabelled directories.

Please note that this differs somewhat from the 2003 competition. In it, each run reflected a subject choosing several targets in sequence to spell a word. The data in this competition instead reflect a subject choosing a single target. When data were collected for data set 1, subjects spelled a three letter sequence, but these will be broken up into individual letters for the competition.

In the event of a tie, points will be doubled for correct classification of all "1" directories, since recognizing a single trial is a more challenging feat than multiple trials. If a tie remains, points will be doubled for correct classification of any "3" directory. In the very unlikely event a tie remains, all people in the tie will be given a copious amount of new P3 data, and given one month for a special "tiebreaker" competition. Since this will delay the determination of who is the winner, the website shall reflect that this "tiebreaker" competition is pending until it is decided. If a tie remains after the tiebreaker, the organizer shall go insane, thus ending the competition.

There will be one winner for each data set. Winners are encouraged to submit their results to IEEE Transactions on Biomedical Engineering (TBME), IEEE Transactions on Neural Systems and Rehabilitation Engineering (TNSRE), or a similar journal. Winners will also be announced at the 2005 BCI conference and given a certificate designed and signed by Drs. Allison and Moore.

Participants must submit entries to at least three datasets to be considered for the title of overall winner. The scores for all the datasets that participant used will be averaged. Whoever has the highest average score shall be the overall winner, or "grandmaster." In addition to the fact that the grandmaster will very probably win at least one of the datasets, the grandmaster gains two additional prizes. The first is a crown designed by Dr. Allison. The grandmaster shall be crowned at the 2005 BCI conference, if present. To add to the excitement, please note that Dr. Allison has no relevant design experience and is a terrible artist. The second is a special certificate. All participants except the grandmaster are required to sign this certificate acknowledging the grandmaster has "bragging rights" until the next competition.

Dissemination of Results

The results of this competition, including a list of winners and their pattern recognition approaches, will be announced at the 2005 BCI conference in New York, posted on this website, and submitted to IEEE TNSRE or TMBE. Winning participants are also encouraged to submit their results to either of these or another relevant journal. The conference organizer is not affiliated with either of these IEEE journals, and decisions regarding publication rest solely with its editorial board. However, IEEE TNSRE has been receptive to articles from the recent data analysis competition.

Each participant agrees to deliver an extended description (one page in IEEE style) of the algorithm used for the publication in in case she/he is the winner for one of the data sets. In this publication and any other dissemination of results, each participant must reference the group that recorded the data and cite at least one of the papers listed in the respective description for each data set.

If interested in the IEEE TNSRE publication from the 2003 conference organizers, please see Sajda et al. (2003). An example of an IEEE publication from one of the winners of this competition is Mensh et al. (2003).

People

This conference is being coordinated by Brendan Allison, Ph. D. My PhD thesis involved P300 BCIs, but included a thorough review of all BCIs. I graduated from UC San Diego in 2003, where I worked in the Cognitive Neuroscience Laboratory under Dr. Jaime Pineda. After a 3 month internship at the Wadsworth Research Center in New York, I now work at the BrainLab at Georgia State University. Our lab director is Dr. Melody Moore.

I wish to thank:

Benjamin Blankertz, Melody Moore, and Gerwin Schalk for advice on this competition.

The Wadsworth Research Center for BCI2000, which was used to collect the data in data sets 1, 4, 5, and 6.

Andrey Vankov for ADAPT, which was used to collect the data in data sets 2 and 3.

Jaime Pineda for permission to use data sets 2 and 3, which Dr. Allison collected in the Pineda lab.

All subjects who participated in this research. While subjects' names must of course remain anonymous, none of this would be possible without their help. Special thanks to the ALS patients and their caretakers.

Dr. Allison can be reached at: