Frequently asked questions (May 2022)
Should I anonymize my analysis scripts?
We do not guarantee anonymity of analysts. If you do not want your code to be identifiable, you are responsible yourself to remove identifiable features (e.g. your name). If you want to be acknowledged, then please add your name in the files. The code of each team will be published alongside the paper we aim to write, using a CC-BY license. If you want your code to be citable/ acknowledgeable, please document yourself in the respective script files that you authored.
Can you provide us with an empty folder structure for the data upload?
We believe that the easiest way to upload your data is by organizing it in the folder structure described in the “Instructions to submit data” sent to analysts by email and uploading it as one piece to the submission portal. This would require that you organize your data in your local directory before uploading it. For reference, the empty folder structure, including all subfolders, can be now downloaded from GitHub. If you need help organizing your data files to match the template, please contact us at firstname.lastname@example.org.
I can only upload individual files to the submission portal. How can I upload a folder?
While the upload button (indicated with a ‘+’ sign in the submission portal) allows selecting individual files, you can always upload folder structures by dragging and dropping them to the active web window.
Can we upload additional data for all of the analyses that we did?
We are excited to see the growing numbers of data uploaded on our servers. However, we would like to kindly ask you to upload only the data that was requested in the emails sent to analysts as well as in our FAQ section (see below the Q: ‘What do we need to submit’). This is important due to our storage space restrictions. In case there is additional data and/or an observation that you would like to share with us or upload it, please contact us at email@example.com before uploading it.
We used different preprocessing pipelines for different hypothesis testing. Should we upload a single file per subject or multiple ones?
In case you used different pipelines to pre-process data (i.e. before any averaging and inference), please upload a single file per subject, namely the time-domain pre-processed data you used for hypotheses 4.b. This is the same hypothesis that you should use as a “default” when answering the analysis questionnaire. Note that hypothesis 4.b. is to be tested on time-frequency domain data but for the data submission, we ask you to submit the time-domain data before time-frequency decomposition. If you have any further questions on how to upload data in this situation, please contact us at firstname.lastname@example.org.
Should I also upload pre-processed data for subjects that I excluded from the analyses?
Please create an empty folder in the submission portal for the subjects that were excluded from your further analyses. In this case we would know that you didn’t overlook this subject during the data submission process but intentionally excluded him/her.
Frequently asked questions (April 2022)
What do we need to submit?
The data submission will consist of three parts that have to be submitted per team and one questionnaire that should be submitted by each analyst individually. These parts are:
- Upload pre-processed time-domain data and analysis scripts. Each team will receive an individualized link (see the question below) to their own upload folder.
- A free text results questionnaire summarizing the results of your analysis. You will have to write 1–2 paragraphs per hypothesis describing the results you obtained, similar to the results section of a scientific journal article. This section should include all relevant test statistics and decision criteria with high numerical precision (e.g. 7–8 digits after the comma or a numerically exact fraction)
- Analysis strategy questionnaire about the processing steps. This questionnaire has separate subsections for different preprocessing and analysis steps. You can complete each subsection separately. Note that this questionnaire might take several hours to complete.
- Results questionnaire asking for each hypothesis whether you confirmed it or not, as well as your confidence rating.
When and where can I upload my data, code, and results?
The submission phase starts on April 30th. All teams will receive emails before May 3rd with links to their individual submission folder and questionnaires. Instructions on how to upload your data can be found from the emails as well as from the folders that you will find in the submission portal. Instructions for filling in the questionnaires can be found at the beginning of each questionnaire. If you have not received such an email, please contact us at email@example.com. The deadline for submitting data, code, and results is on May 15th 2022!
I have technical problems with uploading my data, source code, or filling in one of the questionnaires. What should I do?
Please contact us at firstname.lastname@example.org. We will support you to solve the technical problems.
We need more time for uploading the data, source code, or filling in one of the questionnaires. Can the submission deadline be extended?
The end of the analysis phase was set for April 30th. Uploading and reporting of the results should be completed between May 3rd and May 15th. Unfortunately, we are unable to make major exceptions to this rule. If there are major obstacles that cause delays or prevent you from uploading and reporting your results within the allocated time frame, please contact us at email@example.com. We will consider requests for extensions on an individual basis.
How will you treat incomplete submissions?
Please begin uploading your data, code, and a report of your results as soon as possible and allow enough time to complete all the steps before May 15th. After the deadline, we will review all submissions and notify you if any subsections are incomplete or missing. You will have a few extra days to complete those subsections, but we are unfortunately unable to grant major deadline extensions.
I have not received an email to the submission portal. What should I do?
Please check your spam folder and make sure you are using the same email address as when you signed up. If you are still unable to find the email, please contact us at firstname.lastname@example.org.
Can I or can we publish our results and/or pipelines on our personal websites, blogs, GitHub, etc.?
Yes, but we kindly ask you to wait until June 15th before publishing anything about your answers to the research questions, your data analysis, or analysis code.
Frequently asked questions (October 2021)
Is late registration possible?
No. The registration period ended on October 3rd 2021. There will not be a second registration period.
Is it possible to change the composition of the teams?
As was indicated in the participation agreement, withdrawing from the project is possible at any time. Other changes to the current teams (e.g. adding new members) is unfortunately not possible, since this would require opening another registration period.
How do you exactly define what is an alpha oscillation?
We have received multiple questions about the exact definitions of oscillation frequency ranges. Since this project aims to investigate how EEG researchers choose their analysis pipelines, please analyze these data using your usual pipeline with the definition and frequency range you find most appropriate. If you haven’t earlier analyzed similar data, please refer to the published literature, talk to your colleagues, and use your own expert judgement.
May I still participate in this project, if there’s a concept I don’t know previously (e.g. the N1 component mentioned in the prior expectations questionnaire)?
The eligibility criterion for participating in this project is that each team must have at least one member with at least one previous EEG publication. Not knowing a particular concept at the onset of the project does not disqualify you or your team from participating in this project. Try to analyse the data in the way you find most appropriate as if you were analysing your own data to test the hypotheses. Also, please refer to published literature, talk to your colleagues, or use your own expert judgement.
I would like to suggest another hypothesis to be tested or an existing hypothesis to be rephrased. Can you make changes?
The current set of hypotheses are fixed. It is not possible to add a new question or change the way the hypotheses are currently phrased. Please try to give your answer to whether you confirm/reject the hypotheses as they are formulated.
Is it possible to answer only a subset of the hypotheses or submit otherwise incomplete pipelines but still receive co-authorship?
As was outlined in the participation agreement, submitting incomplete data or pipelines is not sufficient for eligibility for co-authorship.
Hypothesis X seems to presuppose a specific analysis method, I thought we were supposed to use our own analysis methods?
Some hypotheses might be more or less indicative of a given approach than others, but we do not suggest that they imply a fixed analysis strategy. Try to analyse the data in the way you find most appropriate as if you were analysing your own data to test the hypotheses.
Frequently asked questions (June 2021 – September 2021)
What is EEGManyPipelines?
EEGManyPipelines is a multi-analyst project investigating the variability in EEG preprocessing pipelines. In this project, many independent teams will analyze the same data set with the same questions. Such a multi-analyst approach has been used before (Silberzahn & Uhlmann, 2015; Silberzahn et al., 2018), e.g., for fMRI data (NARPS project, Botvinik-Nezer et al., 2020). Note that a multi-analyst project differs from the following: 1) meta-analysis; 2) Multilab; 3) Multiverse or specification curve analysis.
- A multi-analyst project is different from a meta-analysis in that only one single data set and one single set of hypotheses are used, which are then tested by all teams. It thus investigates variability that occurs purely as a function of different analysis choices, not as a function of different underlying data sets/ data collection procedures/ hypotheses.
- A multi-analyst project is also different from a Multilabs project (e.g. EEGManyLabs, Pavlov et al., 2021) in that only a single data set is used. Only variability in analysis choices, not variability in data collection is investigated.
- A multi-analyst project is also different from a multiverse or a specification curve analysis (Steegen et al., 2016; Simonsohn et al., 2016) in that it does not investigate just theoretically possible pipelines, but actual pipelines that independent EEG researchers deem their preferred way for analyzing the provided data set. Any variability in pipelines and results in this project will thus be closer to the naturally occurring variability of everyday EEG research.
Who can participate in this project?
Anyone with sufficient EEG experience can participate. Participants can sign-up in teams of up to three people. At least one person in the team must have at least one peer-reviewed journal publication featuring EEG (or MEG or eCoG) analysis. This eligibility criterion will be checked by the steering committee. This requirement is important because it allows us to make sure that our sample of teams is representative of people already participating in EEG research. Signing-up as a team allows for people of different levels of expertise and seniority (e.g. PhD, postdoc, PI) to work together. It is possible that multiple teams from the same lab or department participate. However, in the participation agreement, we ask you to not talk or share your pipelines and results with each other before the end of the analysis phase (see below).
I would like to participate in a team, but don’t have teammates. Can you match me with others?
Due to privacy concerns, we cannot share contact details of other analysts. We strongly advise interested analysts to team up with people who they know personally (same department, same university) and who use the same software and overall analysis approach. Having a shared vision on EEG analysis will greatly facilitate collaboration in a team. If you have no such option, we suggest you to ask around on Twitter using the hashtag #eegmanypipelines.
What are the benefits of participation?
If participants submit all required results, code and data at the end of the analysis phase, they will be invited to become co-authors on a journal submission based on the results from this data. The code, pre-processed data, and questionnaire reports of all different teams will be shared publicly after this project and will allow future researchers to use this data for additional analyses. We hope that this project will shed more light on how much EEG pipelines vary “in the wild” and to what extent this influences results and conclusions. In this way, it will hopefully lead to a better understanding of the consequences that different preprocessing and analysis steps have on the results and thus lead to a better grasp of what it means for results to be “robust” or not. This will in the end benefit the entire EEG field.
What does participation entail exactly?
Upon registration, you will be asked to sign a participation agreement. The participation agreement asks you to analyze the provided data set with the provided hypotheses during the analysis phase (October 2021 – April 2022) and at the end submit your results, code, and preprocessed data (see details below). In addition, it asks you not to share any information about the data set & hypotheses, your pipeline or your results with other people outside your team during the analysis phase. This is important for us to obtain independent pipelines and results that were not influenced by each other. Direct communication between teams as well as indirect discussions (e.g. on Facebook, Twitter) are thus prohibited; if we become aware of such discussions, we will intervene. For any questions, please contact the EEGManyPipelines team under email@example.com.
What if I have signed up but want to cancel my registration?
You can cancel your registration at any time by sending an email to firstname.lastname@example.org. Apart from not qualifying as a co-author anymore, there will be no downsides to you. The same holds in case you do not submit anything at the end of the analysis phase. Even if you canceled your registration, we ask you to not share any details on the dataset & hypotheses (or your pipeline approach/results) with other teams during the analysis phase.
What kind of dataset will be analyzed?
All analysis teams will receive the same dataset. The data set will comprise data from 33 subjects each completing about 1,200 trials, recorded with a 64-channel system, including 4 EOG channels and 2 mastoids. The task involves visual stimuli. There is ample previous literature on similar task designs. We will not disclose further details of the dataset to prevent public discussion and not sway analysts’ prior expectations/ analysis approaches.
What kind of analyses will be required?
Analysis teams will be given the raw EEG data and asked to test 5 hypotheses based on the dataset. These hypotheses are grouped into 3 bigger questions that each require comparing 2 (or more) experimental conditions. All analyses require that time-domain (ERPs) or time-frequency-domain (TFRs) data between two conditions are compared for differences. There is ample previous literature on the contrasts of interest. We will not disclose further details of the data set to prevent a public discussion and not sway analysts’ prior expectations/ analysis approaches. Participants are free to use any software or programming language for their analysis. There are no constraints.
Can I use my/our own pipeline/method/custom software to analyse the data?
Yes, you are free to use whatever method and software that you find necessary to process and analyse the data. How you want to analyse the data is up to you. Given that we are mostly interested in studying analysis pipelines representative for published studies, we actually encourage you to use whichever software and pipeline you have used in your previous work. The goal is for your team to answer the research questions that are provided when the analysis phase begins, using the procedures and methods that you would otherwise use if this was your own research project. However, when submitting your results, we will also ask you to submit your code (or log file of the analyses performed), the pre-processed data, and describe your pipeline using a structured questionnaire. More details will follow.
What will the teams have to submit after analyses are finished?
Over the course of this project, analysis teams will complete multiple questionnaires, starting with demographic questions upon registration and some questions about their a-priori expectations once they receive the data. At the end of the analysis phase, teams will have to report a binary yes/no decision about each hypothesis with regard to whether they found it supported by the data or not, and some extra questions on how confident they are about this result. In addition, teams will be asked to write a short results section (similar to a journal publication) describing and discussing what exactly they have found. In addition to the results, analysis teams will be asked to report the exact pipeline they used for analyzing the data. They will be asked to submit all the code they used and to additionally fill in a questionnaire about their pipeline. Finally, analysis teams will be asked to submit their preprocessed data (after preprocessing, before aggregation or statistical tests) for each subject in the data set. More details will follow.
Whom can I contact for questions?
For any questions, please send an email to email@example.com. We ask you to not ask any analysis- or results-related questions on public forums, chats, mailing lists, … to not influence other analysts.
Can I share my results with other people in lab meetings, on Twitter, or elsewhere?
No, we ask you to not share any details on your analysis pipeline or your results during the analysis phase (October 2021 – April 2022). After this phase, you are free to share your pipeline and results with other people, either in personal meetings or online. Also feel free to post your source code on GitHub after the analysis phase.
What will happen with all the data (pipelines, questionnaire results) after this project is finished?
All the data acquired in this project will be made publicly available under the CC-BY license after the results of this study have been published. This means that you and other researchers can use them for their own future projects (potentially resulting in further publications).
What does the team identifier mean?
If you want to work together with other people, they have to fill in your team identifier when they sign up for the project.