Frequently asked questions (October 2021)
Is late registration possible?
No. The registration period ended on October 3rd 2021. There will not be a second registration period.
Is it possible to change the composition of the teams?
As was indicated in the participation agreement, withdrawing from the project is possible at any time. Other changes to the current teams (e.g. adding new members) is unfortunately not possible, since this would require opening another registration period.
How do you exactly define what is an alpha oscillation?
We have received multiple questions about the exact definitions of oscillation frequency ranges. Since this project aims to investigate how EEG researchers choose their analysis pipelines, please analyze these data using your usual pipeline with the definition and frequency range you find most appropriate. If you haven’t earlier analyzed similar data, please refer to the published literature, talk to your colleagues, and use your own expert judgement.
May I still participate in this project, if there’s a concept I don’t know previously (e.g. the N1 component mentioned in the prior expectations questionnaire)?
The eligibility criterion for participating in this project is that each team must have at least one member with at least one previous EEG publication. Not knowing a particular concept at the onset of the project does not disqualify you or your team from participating in this project. Try to analyse the data in the way you find most appropriate as if you were analysing your own data to test the hypotheses. Also, please refer to published literature, talk to your colleagues, or use your own expert judgement.
I would like to suggest another hypothesis to be tested or an existing hypothesis to be rephrased. Can you make changes?
The current set of hypotheses are fixed. It is not possible to add a new question or change the way the hypotheses are currently phrased. Please try to give your answer to whether you confirm/reject the hypotheses as they are formulated.
Is it possible to answer only a subset of the hypotheses or submit otherwise incomplete pipelines but still receive co-authorship?
As was outlined in the participation agreement, submitting incomplete data or pipelines is not sufficient for eligibility for co-authorship.
Hypothesis X seems to presuppose a specific analysis method, I thought we were supposed to use our own analysis methods?
Some hypotheses might be more or less indicative of a given approach than others, but we do not suggest that they imply a fixed analysis strategy. Try to analyse the data in the way you find most appropriate as if you were analysing your own data to test the hypotheses.
Frequently asked questions (June 2021 – September 2021)
What is EEGManyPipelines?
EEGManyPipelines is a multi-analyst project investigating the variability in EEG preprocessing pipelines. In this project, many independent teams will analyze the same data set with the same questions. Such a multi-analyst approach has been used before (Silberzahn & Uhlmann, 2015; Silberzahn et al., 2018), e.g., for fMRI data (NARPS project, Botvinik-Nezer et al., 2020). Note that a multi-analyst project differs from the following: 1) meta-analysis; 2) Multilab; 3) Multiverse or specification curve analysis.
- A multi-analyst project is different from a meta-analysis in that only one single data set and one single set of hypotheses are used, which are then tested by all teams. It thus investigates variability that occurs purely as a function of different analysis choices, not as a function of different underlying data sets/ data collection procedures/ hypotheses.
- A multi-analyst project is also different from a Multilabs project (e.g. EEGManyLabs, Pavlov et al., 2021) in that only a single data set is used. Only variability in analysis choices, not variability in data collection is investigated.
- A multi-analyst project is also different from a multiverse or a specification curve analysis (Steegen et al., 2016; Simonsohn et al., 2016) in that it does not investigate just theoretically possible pipelines, but actual pipelines that independent EEG researchers deem their preferred way for analyzing the provided data set. Any variability in pipelines and results in this project will thus be closer to the naturally occurring variability of everyday EEG research.
Who can participate in this project?
Anyone with sufficient EEG experience can participate. Participants can sign-up in teams of up to three people. At least one person in the team must have at least one peer-reviewed journal publication featuring EEG (or MEG or eCoG) analysis. This eligibility criterion will be checked by the steering committee. This requirement is important because it allows us to make sure that our sample of teams is representative of people already participating in EEG research. Signing-up as a team allows for people of different levels of expertise and seniority (e.g. PhD, postdoc, PI) to work together. It is possible that multiple teams from the same lab or department participate. However, in the participation agreement, we ask you to not talk or share your pipelines and results with each other before the end of the analysis phase (see below).
I would like to participate in a team, but don’t have teammates. Can you match me with others?
Due to privacy concerns, we cannot share contact details of other analysts. We strongly advise interested analysts to team up with people who they know personally (same department, same university) and who use the same software and overall analysis approach. Having a shared vision on EEG analysis will greatly facilitate collaboration in a team. If you have no such option, we suggest you to ask around on Twitter using the hashtag #eegmanypipelines.
What are the benefits of participation?
If participants submit all required results, code and data at the end of the analysis phase, they will be invited to become co-authors on a journal submission based on the results from this data. The code, pre-processed data, and questionnaire reports of all different teams will be shared publicly after this project and will allow future researchers to use this data for additional analyses. We hope that this project will shed more light on how much EEG pipelines vary “in the wild” and to what extent this influences results and conclusions. In this way, it will hopefully lead to a better understanding of the consequences that different preprocessing and analysis steps have on the results and thus lead to a better grasp of what it means for results to be “robust” or not. This will in the end benefit the entire EEG field.
What does participation entail exactly?
Upon registration, you will be asked to sign a participation agreement. The participation agreement asks you to analyze the provided data set with the provided hypotheses during the analysis phase (October 2021 – April 2022) and at the end submit your results, code, and preprocessed data (see details below). In addition, it asks you not to share any information about the data set & hypotheses, your pipeline or your results with other people outside your team during the analysis phase. This is important for us to obtain independent pipelines and results that were not influenced by each other. Direct communication between teams as well as indirect discussions (e.g. on Facebook, Twitter) are thus prohibited; if we become aware of such discussions, we will intervene. For any questions, please contact the EEGManyPipelines team under email@example.com.
What if I have signed up but want to cancel my registration?
You can cancel your registration at any time by sending an email to firstname.lastname@example.org. Apart from not qualifying as a co-author anymore, there will be no downsides to you. The same holds in case you do not submit anything at the end of the analysis phase. Even if you canceled your registration, we ask you to not share any details on the dataset & hypotheses (or your pipeline approach/results) with other teams during the analysis phase.
What kind of dataset will be analyzed?
All analysis teams will receive the same dataset. The data set will comprise data from 33 subjects each completing about 1,200 trials, recorded with a 64-channel system, including 4 EOG channels and 2 mastoids. The task involves visual stimuli. There is ample previous literature on similar task designs. We will not disclose further details of the dataset to prevent public discussion and not sway analysts’ prior expectations/ analysis approaches.
What kind of analyses will be required?
Analysis teams will be given the raw EEG data and asked to test 5 hypotheses based on the dataset. These hypotheses are grouped into 3 bigger questions that each require comparing 2 (or more) experimental conditions. All analyses require that time-domain (ERPs) or time-frequency-domain (TFRs) data between two conditions are compared for differences. There is ample previous literature on the contrasts of interest. We will not disclose further details of the data set to prevent a public discussion and not sway analysts’ prior expectations/ analysis approaches. Participants are free to use any software or programming language for their analysis. There are no constraints.
Can I use my/our own pipeline/method/custom software to analyse the data?
Yes, you are free to use whatever method and software that you find necessary to process and analyse the data. How you want to analyse the data is up to you. Given that we are mostly interested in studying analysis pipelines representative for published studies, we actually encourage you to use whichever software and pipeline you have used in your previous work. The goal is for your team to answer the research questions that are provided when the analysis phase begins, using the procedures and methods that you would otherwise use if this was your own research project. However, when submitting your results, we will also ask you to submit your code (or log file of the analyses performed), the pre-processed data, and describe your pipeline using a structured questionnaire. More details will follow.
What will the teams have to submit after analyses are finished?
Over the course of this project, analysis teams will complete multiple questionnaires, starting with demographic questions upon registration and some questions about their a-priori expectations once they receive the data. At the end of the analysis phase, teams will have to report a binary yes/no decision about each hypothesis with regard to whether they found it supported by the data or not, and some extra questions on how confident they are about this result. In addition, teams will be asked to write a short results section (similar to a journal publication) describing and discussing what exactly they have found. In addition to the results, analysis teams will be asked to report the exact pipeline they used for analyzing the data. They will be asked to submit all the code they used and to additionally fill in a questionnaire about their pipeline. Finally, analysis teams will be asked to submit their preprocessed data (after preprocessing, before aggregation or statistical tests) for each subject in the data set. More details will follow.
Whom can I contact for questions?
For any questions, please send an email to email@example.com. We ask you to not ask any analysis- or results-related questions on public forums, chats, mailing lists, … to not influence other analysts.
Can I share my results with other people in lab meetings, on Twitter, or elsewhere?
No, we ask you to not share any details on your analysis pipeline or your results during the analysis phase (October 2021 – April 2022). After this phase, you are free to share your pipeline and results with other people, either in personal meetings or online. Also feel free to post your source code on GitHub after the analysis phase.
What will happen with all the data (pipelines, questionnaire results) after this project is finished?
All the data acquired in this project will be made publicly available under the CC-BY license after the results of this study have been published. This means that you and other researchers can use them for their own future projects (potentially resulting in further publications).
What does the team identifier mean?
If you want to work together with other people, they have to fill in your team identifier when they sign up for the project.