Ethical Reviewing Recommendations

This document aims at regrouping information and recommendations about ethical reviewing. It is split into different sections, each directed at different actors: conference chairs, ethical chairs, ethical reviewers, and authors/general reviewers.

This was made by combining information from different sources, either pasting them or rephrasing them. The structure and patchwork was done by Ella Li, Minzhi and Fanny Ducel. The conversion to Markdown was done by Guido Ivetta:

• – Karën Fort and Min-Yen Kan

• – Jin-Dong Kim and Min-Yen Kan

• – Vinodkumar Prabhakaran and Malihe Alikhani (ACL Rolling Review Ethics Chairs), with help from NAACL 2024 Ethics Chairs, EACL 2024 Ethics Chairs, and EMNLP 2024 ethics chair

• – Karen Fort and Min-Yen Kan (No need to formally cite)

• – developed by Marine Carpuat, Marie-Catherine de Marneffe and Ivan Vladimir Meza Ruiz, the NAACL 2022 program chairs, working with Jesse Dodge, and with the ARR editors in chief. Additional input was provided by the other NAACL 2022 reproducibility chairs, Margot Mieskes, Anna Rogers, and the ACL Ethics Committee. Updated for ARR October 2024 cycle by Anna Rogers, based on discussions with ARR board.

• – drafted for ARR by Amanda Stent and reviewed and edited by the ARR Editors in Chief, Tim Baldwin, Anna Rogers, the NAACL 2022 ethics chairs Kai-Wei Chang and Diyi Yang, and members of the ACL Ethics Committee.

• – Malihe Alikhani and Vinodkumar Prabhakaran

• – Luciana Benotti, Karën Fort, Min-Yen Kan, Yulia Tsvetkov

1. General introduction to ethics reviewing

The following section is a compilation of other documents: one written by Karën Fort and Min-Yen Kan and another written by Malihe Alikhani and Vinodkumar Prabhakaran.

Ethics reviews are a cornerstone of maintaining integrity and responsibility in academic research. They are not just a formality; they are critical to the integrity of academic discourse. In particular, the ethics reviews are meant to assess whether the submitted research exhibits any substantial ethical issues and could lead to harm. It is especially critical to uphold these standards as we engage in research that deals with sensitive data and/or controversial topics. Ethics reviewing plays a pivotal role in safeguarding these standards. We want to emphasize that an ethics review is not just about ticking boxes, nor is it a technical review; it’s about engaging critically with the material and its implications.

The existence of ethics committees should not inhibit, prevent, or discourage reviewers, ACs, or SACs from making decisions to reject a paper on the basis of ethical concerns. While our committee will presumably provide reviews and discussion for those papers, we believe it should be the ultimate goal of the ACL community that these considerations be considered as a relevant part of the standard review process.

Standardized Terminology

ACL Ethics Committee - In charge of “provid[ing] a continuous and consistent platform for dealing with potential ethical concerns raised within ACL events and the ACL community”, focusing on “provid[ing] guidance and address considerations of what constitutes ethical research”. Their role includes the coordination with Ethics chairs at *ACL conferences, the provision of guidelines on how to report ethical concerns and more generally, “the compilation and publication of resources to educate the broad community about the limitations and societal implications of the technologies we develop and deal with ethical concerns within the ACL”. The present document was created by members of the ACL ethics committee, in line with the last aforementioned goal. Composed of “three co-chairs serving a five-year term, and six members at large serving a three-year term, ensuring uniform coverage of the three geographic regions represented by AACL, EACL, and NAACL”. The members of the current committee are listed at LINK. Source.

ARR Ethics Chairs - In charge of considering submissions flagged for ethics review during ARR review. If they consider it necessary, they will assign the paper to ethics reviewers for an in-depth ethics review. They remain in charge of supervising all the ethics review(er)s. As of Dec 2024, the ARR Ethics Chairs are, according to source, Malihe Alikhani and Vinodkumar Prabhakaran. (Rephrased from source and source)

Ethics Review Chairs of a Convenor - Each conference has its own ethics chairs. Their role is similar to the one of ARR Ethics Chairs but for the papers submitted to the conference they are in charge of.

ACL Publications ethics committee - In charge of “developing policies and procedures related to publication ethics, including such issues as plagiarism, the use of LLMs in producing reviews, disclosures of e.g. authorial control of papers, and how to handle issues raised about papers post-publication”. A first policy was made available in June 2024 at source. As of Dec 2024, the current chairs are Leon Derczynski, Dr. Kokil Jaidka and Dr. Aoife Cahill. Source.

2. Recommendations for conference chairs

Make sure ethics chairs have enough time

The ethics chairs should be recruited early in the conference planning process, with sufficient time to participate in setting the review schedule to ensure there is time for ethics review and clarity about how papers will be referred and evaluated.

There should be at least 2-3 weeks in the review timelines allocated to the ethics review, and there should be a strict cut-off for when the ACs/SACs flag a paper for ethics review. Ethics chairs should not be getting requests after the Ethics review timeframe.

Process Flow (ARR Ethics Chairs, including conference ethics chairs)

During ARR review, some submissions are flagged for ethics review. These submissions are considered by the ethics chairs and if necessary, sent to ethics reviewers, who are supervised by the ethics chairs.

Timeline:

Phase 0: Preparation

1. Familiarize with the Open Review consoles:
   • You can find them by logging into the Open Review platform. If your profile is correctly set up in the system as Ethics Chairs for a particular ARR cycle, you should have two of the below consoles (in addition to others) under Your Active Consoles:
     • ACL ARR <Year> <Month> Ethics Chairs Console: This is the main console where you perform most of the actions related to this task, such as reviewing papers, ethics review flag and justification, assign ethics reviewers etc.
     • ACL ARR <Year> <Month> Ethics Reviewer Console: This console will be used in two contexts:
       1. When you want to see how a reviewer sees the assignment (for example, to check the review form).
       2. To Add a review to a paper acting as ethics reviewer (for which you’ll first have to assign yourself as an ethics reviewer to the paper from the Ethics Chairs console).
2. Familiarize with the ARR Ethics Review Cycle-specific Master Tracking Sheet, which contains the following sheets:
   • Timeline: A spreadsheet version of the above timeline, where you could add Cycle-specific dates for easy tracking etc.
   • Instructions: A set of common false flags and misconceptions about ethics reviewing, and suggested actions and how to deal with them (including comment templates, if applicable)
   • Reviewers: A sheet that keeps track of the reviewers available for the current cycle. This will help keep track of how many papers are assigned to each reviewer, what load they requested etc.
   • Chairs Coordination: To coordinate among the ethics chairs; to track the load handled by each chair, to track availability/OOO time etc.
   • Papers Tracking: A sheet that keeps track of each paper flagged for ethics review.
   • Papers in the OR queue:
3. Ensure that we have an adequately sized ethics reviewer pool.
   • Having an adequate # of reviewers depends on the submission volume. In the past, about 4% of all submissions get a full ethics review and we recommend a rate of 5%. Each paper needs just one ethics review. So assuming 2.5 papers per reviewer on average, for X total submissions, if you have between 0.02X/2.5 and 0.03X/2.5 reviewers, you should be good.
   • But keep track of trends in your cycle, and you may need to request new reviewers if you feel the existing pool doesn’t have the expertise to review a particular paper.
   • Add any new reviewers to the “Reviewers” sheet in the master tracking sheet
   • To check the reviewers in the system, click on the “Edit Group” button on the top of the Ethics Reviewer Console.
   • To add new reviewers to the system, follow the instructions here.
4. Ensure reviewer availability for the time period.
   • Please note that every ethics reviewer from past cycles will have been carried over to the new cycles on Open Review, but they may not be available to review for your cycle. So you will need to first check with them about their availability — either using the message functionality or directly emailing them (latter is more likely to succeed).
   • You may also use this form and sample recruitment email to recruit additional reviewers as well as collect their availability information.

Phase 1 & 2: Assigning & Tracking Ethics Reviews

Because this will need to be coordinated among all ARR ethics chairs, we usually do this using a spreadsheet for coordination and tracking.

1. [Monitoring]: Monitor the incoming papers and add them to the "Papers Tracking" sheet in the master spreadsheet. This is an ongoing task and has a few kinks that are outlined below. [Owner Assignment] Assign an Owner ethics chair who will be in charge of the paper
2. [Initial assessment] Decide which papers really need an ethics review. The usual process is to perform a quick check and flagging papers that don't have any justifications etc. You can check the "Instructions" sheet in the spreadsheet for some common patterns and suggested actions. You may also choose to read the abstract or quickly review the paper, in case of any doubt.
3. [Optional Note] If a paper does not need a full ethics review,
   • You may want to leave a note depending on what the case is. Some such scenarios in the Instructions sheet. Rule of thumb is whether the flagger or authors benefit from knowing that someone looked at it from an ethics perspective. If a paper was flagged with a justification "None", it is a false flag and doesn't require a response. But if, for instance, it was flagged for not having an ethical consideration section, it would be helpful for the flagger and the authors to hear that we looked at it and it (by itself) was deemed to be not enough justification for a full review.
   • If you need to add a note, you have to first assign yourself as a reviewer to the paper (instructions here). It may take a while for the assignment to trickle through. So you may have to check after a few minutes to an hour.
4. [Reviewer Assignment] If a paper is deemed to need a full ethics review:
   • Assign the reviewer in OpenReview. You can do it by going to the Ethics Chairs console and clicking on the paper and it should show an "edit assignments" button. Detailed instructions here.
   • Update the Papers Tracking sheet to keep track, and also making sure not to assign more than 2-4 papers (depending on the cycle) to each ethics reviewer, and respecting any reduced load they requested.
5. [Follow up] Owner follows up with the reviewer closer to the review deadline to make sure they submit their ethics review by then. Emailing directly is often more effective than reminders via OpenReview. This deadline varies between Phase 1 & Phase 2.

Ongoing Tasks

• [Monitoring Flagged Papers] Whenever a reviewer/AE flags a paper for ethics review, it automatically sends an email to all ethics chairs. You can also see the currently flagged papers in the “Paper Status” tab in the Ethics Chairs console. Periodically, make sure to update the “Papers Tracking” sheet in the Master Spreadsheet with these flagged papers so that you can coordinate on the initial assessment and review assignment. You may use a helpful script written by Ameeta Agarwal given below, or simply use a spreadsheet formula to do such batch transfer, periodically., There are a few issues associated with this:
  • [Inflow] They come in almost daily, and aligned with certain deadlines in the technical review timeline, there may be huge influxes of papers getting tagged. This is a manual step. So you may want to wait for a few days to get a big batch in one go, or update the sheet with papers as they come, based on what your work style is and bandwidth at that time. I have reached out to OpenReview to make this easier, and will report back if viable.
  • [Duplicates] Every time a reviewer/AE flags a paper we get an email. So if you are adding papers to the sheet from the email notifications, note that some of them may already have been flagged. Currently Column C in the Papers Tracking sheet has a formula to flag duplicates, to make it easier to detect them.
  • [Removals] Some reviewers/AEs may realize a paper was accidentally flagged, and update their ethics review flag to “No”. This will remove the paper from the Paper Status console on Open Review. Correspondingly I have added a column D in the Papers Tracking sheet to note if the paper is active on the console. In the beginning you can manually update it if you find the paper is no longer in the console. But we can also semi-automatically refresh this column periodically.

Suggestions for Call-for-paper

• The Call for Papers (CFPs) should incorporate ethics guidelines, making it clear that submissions may be subject to rejection due to ethical concerns.
• This proactive measure may contribute to deterring submissions presenting straightforward ethical issues, such as manual annotations lacking proper compensation disclosure.
• For conferences featuring multiple tracks (e.g., main, student, industrial), uniform ethical guidelines should be applied across all tracks. It's essential that these guidelines are clearly articulated in the CFPs for each track, ensuring consistency and awareness among prospective authors.
• Furthermore, extending these ethical standards to the Call for Tutorials and Workshop Proposals is prudent. Given that workshop papers are also archived in the ACL Anthology, adherence to the same ethical guidelines is imperative to maintain integrity across all conference proceedings.

3. Recommendations for ethics committee chairs

Form a diverse committee

Form a large and diverse ethics committee, including participants working in different countries from all continents and declaring a variety of genders. For an example, see the NAACL 2021 committee. We also recommend that ethics reviewers have term limits (where possible) to motivate and ensure that ethics reviewers come from a wide and diverse swath of the memberships and not by a fixed coterie of incumbent reviewers. Less experienced ethics reviewers can be paired with more experienced ones.

As members of such a committee may experience pressure to implement decisions from their institutions and/or governments, we suggest to give papers to ethics-review to reviewers from another country. This will not eliminate such risks, but will allow to limit them.

Plan enough time, anticipate the uncertainty

The ethics committee and the recommendations to authors should be publicized in advance (at least one month) on the conference website and on the usual lists. The overall process, including conditional-accept papers shepherding, takes quite a long time and this should be planned accordingly.

Take into account the fact that some papers might be flagged for ethics and be false positive, i.e. will not in fact need an ethics review.

Document the process

The ACL ethics committee’s website provides a dedicated form for documentation at LINK.

The documentation of the process will, among other things, help future ethics chairs to estimate their workload.

Documentation should especially include the number of papers: - flagged for ethics review - indeed reviewed for ethics - accepted conditionally - rejected on ethical grounds: both scientific and ethical OR ethical only

Scope of ethical reviewing

• The purview of Ethics Chairs should be limited to reviewing the ethical issues in the execution of the research presented in the paper.
• Perhaps the ethics section in the papers (or the checklist in the submission process) explicitly requires that authors state explicitly that they had legal access to this data.

4. Recommendations for ethics reviewers

These guidelines are aimed at ethics reviewers, i.e. people assigned to review papers that have already been flagged by other reviewers, focusing on ethics issues. These guidelines are not meant to be exhaustive (especially as ethics is a discussion and can not be diminished to a checklist), but to help reviewers to better understand their role and provide them practical examples.

Scope (what to focus on/how to write an ethics review)

Reminder: an ethics review is not a scientific, nor a complete technical review. Your focus is only on the ethical issues and their presentation. You do not need to check any equations, thoroughness of citations (except as they pertain to ethical issues), or provide any feedback on experimental design or grammar/typos/writing style.

• The goal of the ethics review is to decide whether or not there are any substantial ethical issues (i.e. increased risk of harm outside the current norms of NLP/CL research) with the research presented in the submission.
• Take into account the paper, supplementary materials, the ethics and reproducibility checklist completed by the authors. However, the paper should stand on its own.
• If there are ethical issues:
  • Describe each in terms of the (ACL code of ethics). If you cannot describe the issue in terms of the ACL code of ethics, please consider whether this is something the community agrees is an issue. Please refrain from relying on regional or national laws, regulations or practices in your ethics review, since the submission may be from another country or region. What governs acceptable practice in *ACL venues is the ACL code of ethics. Bear in mind that notions of ethical behavior and practice vary over time and across cultures. If you have questions, please contact the ACL ethics committee.
  • For each ethical issue you identify, describe how you would suggest the authors address it.

Examples of issues

• Issues with the method or approach, for example:
  • failure to consider and ameliorate the negative impacts of the approach on the political, social, or natural environment.
  • failure to consider and ameliorate biases that the approach may perpetuate
  • failure to consider possible risks and impacts of the approach.
  • failure to consider robustness and security, especially in papers about the development of systems close to or in production deployment.
• Issues with the use of code, data and use of people, for example:
  • failure to obtain informed consent where required, or to protect personally identifiable information as required by relevant institutional ethics boards.
  • failure to respect intellectual property and data/code ownership; failure to cite the creators of artifacts used, or to use these artifacts in the ways intended (for example, violations of licenses).
  • failure to consider and ameliorate biases in the data, or biased code/model outputs.
  • failure to consider how the data, the code or other outcomes of the research may cause harm.
  • failure to consider how the participation of people (e.g. annotators) in the research may cause benefits or harms to those individuals.
• Issues with the intended or potential applications of the research (i.e., dual use), for example:
  • failure to consider how applications of the research may harm individuals, groups or society at large
  • failure to discuss ways to mitigate such harms.
  • failure to consider other possible risks of applications of the research; failure to discuss ways to mitigate such risks.
• Other issues, including academic dishonesty:
  • dishonesty in the execution or presentation of the research, including **plagiarism**, deliberate **violation of anonymity, citation, review**, or **duplicate submission policies, falsifying results**, or **misrepresentation** (for example, claiming someone is a co-author when they are not).

Examples of ethics review

Examples of ethics reviews that adhere to the above guidelines

The paper considers the analysis of data from a website that hosts user contributed posts. The authors obtain access to private data by using an account that falsified information to deceive site moderators. The authors scrape the content of the site and release the resulting data, without removing personally identifiable information (including usernames). They do not obtain approval from the site owners or obtain informed consent from the site users. These practices violate the ACL code of ethics’ guidance to “Access computing and communication resources only when authorized or when compelled by the public good” and to “Respect privacy”. The authors are encouraged to (a) document whether and how they obtained permission to scrape and reshare the data; and (b) document how they will remove personally identifiable information from their dataset before releasing it.

The authors fail to discuss ways in which their solution may cause harm to individuals or groups; specifically, in this case, ways in which their solution, if applied, would “out” people who are LGBTQIA, including potentially people who are not out, and potentially falsely “out” people who are not LGBTQIA. This has significant potential to cause harm, rather than contributing to society and to human well-being, as outlined in the ACL code of ethics. I don’t see any way for the authors to avoid these potential harms as their solution currently stands; the best they can do is to promise not to release their model, data and code. (There are cultures and societies where being a member of one of these groups is against the law or considered to be immoral; however, an ethics reviewer in this case might refer the authors to the Universal Declaration of Human Rights).

The authors of this paper claim that their solution beats the performance of previous methods on a common benchmark dataset. To demonstrate the improvement, the authors reproduce a figure showing accuracy numbers of several methods from a previous paper. However, the authors include the figure without attribution and modify it to remove another method that obtains better results than their proposed method. Knowingly misrepresenting the work of others is a violation of the code of ethics (“Strive to achieve high quality in both the processes and products of professional work”). The authors should include all the relevant results with attribution.

Examples of ethics reviews that do **not** adhere to the above guidelines (with comments on what is wrong)

The authors didn’t pay their annotators at least £8.91 / hour, which is the minimum wage in the UK. This means they didn’t “Manage personnel and resources to enhance the quality of working life”.

-> The minimum wage in the UK is not a global minimum wage. An ethics reviewer could inquire if the authors paid the annotators at least “a local living minimum wage”.

There is no documentation of IRB approval for this research.

-> The specific term “IRB” applies to research conducted by USA-based researchers or in the USA that has federal funding and involves human participants only. It is not a universal regulation. An ethics reviewer could ask, if the research involves human participants - particularly in a high stakes setting such as medical care - whether the authors had worked with an ethics review board.

This work will lead to biases in how medical diagnoses are made.

-> Good problem statement, but incomplete: what part of the ACL code of ethics is violated, and how might the authors address or correct this?

This model achieves only 60% accuracy on this data set, even though the authors have deployed it in the educational system in their country. That means that they haven’t followed the ethical rule to “take special care of systems that become integrated into the infrastructure of society”. They should withdraw the model from use, or insist on a human in the loop at all times.

-> Depending on the nature of the model, 60% accuracy may very well be state of the art, and the model may have documented utility even at that level of accuracy. If there is an issue of ethical practice here - and contemporary NLP researchers may disagree - the authors should perhaps discuss how they are going to mitigate the weaknesses of the model; a human in the loop is one possibility, but not the only one.

5. Recommendations for authors and reviewers

Guidelines for Flagging Papers for Ethics Review

Based on: LINK. (the introductory paragraph was adapted by FD, but the “what to avoid?” and “common misconceptions” were simply pasted as I find them very clear as they are)

Ethics reviews are essential to ensure that academic research maintains integrity and responsibility. The number of submissions keeps increasing every year, and we must ensure that all papers adhere to ethical standards, rules, and regulations. Reviewers and action editors have the responsibility to flag papers that may need an in-depth ethics review (which will be performed by an ethics reviewer). The rest of this section provides guidelines for flagging papers for ethics review. More precisely, we list bad practices in ethics flagging, that result in false flags. Over-flagging should be avoided as it burdens the review process by overloading ethics reviewers.

What to avoid?

1. No justification: If you are flagging a paper for full ethics review, please try to give a clear and succinct justification so that ethics chairs can appropriately assign ethics reviewers for the paper. A huge majority of papers still get flagged without the justification field being filled in, or left as “N/A”, “None”, “No ethical concerns” etc. This unnecessarily adds to the workload of the ethics chairs and reviewers.
2. Flagging for Missing Section(s): Another substantial number of papers get incorrectly flagged because of missing ethical considerations section, or missing limitations section, or incompletely/incorrectly filled Responsible AI checklist. While these issues may need to be flagged to the AE’s attention, and in some cases may justify desk rejection (in case of Limitations section for certain conferences), they do not justify a full ethics review. For ACL conferences, these issues can be flagged in the "reviewer checklist". Similarly, adding an ethical considerations section is not mandatory; however if you believe there are specific ethical issues that warrant a full review, please flag the paper and provide a clear justification.
3. Flagging for Copyright, Consent, Transparency: While it is important to ensure that ARR process correctly flags submissions that may be in violation of copyright policies of datasets used, inadequate informed consent process, or transparency of presented artifacts, these issues in and of themselves do not justify a fuller ethics review. Instead, since you have already identified the issue, you can call out the authors in your review or meta review itself to address these issues. However, if there are aspects that require a deeper look, you can of course flag it for a full ethics review, outlining the concern.

Common Misconceptions

1. All Data-centric Papers require Ethics Review: A common misstep is assuming that any paper discussing data usage needs a full ethics review. This is **not** the case. Instead, focus should be on potential misuse or unethical handling of data. A mere description of data usage does not automatically warrant an in-depth review.
2. Use of Human Annotators require Ethics Review: The involvement of human annotators in a study does **not** on its own justify an ethics review. However, if there are specific concerns about the human annotation step, such as potential exploitation or ethical lapses, these may warrant an in-depth ethics review.
3. All Datasets on Sensitive Topics require Ethics Review: Datasets that feature critical content, such as misinformation detection, might raise eyebrows due to their potential dual use. But again, the mere presence of such content does not automatically demand a full ethics review. If it involves engaging with specific marginalized communities (e.g., linguistic minorities) or if the content of the annotation involves potentially traumatizing data (e.g., involving hate speech or other gory concepts), then it may justify an in-depth ethics review to assess whether adequate safeguards were taken. In such cases, explain your reasons as justification while flagging for ethics review.
4. All Papers on High-stakes Domains require Ethics Review: While all of NLP research can arguably have downstream human impact, data/methodological innovations on high-stakes domains may have an immediate impact on users or communities. These include educational use cases such as automated grading scenarios, or medical use cases such as in psychotherapy, or legal use cases such as criminality prediction. These may also include challenges in de-identification in high-stakes domains such as healthcare. However, just because a paper focuses on such a high-stakes domain doesn’t by default necessitate a full ethics review. What matters is the context and application of the research. If regulations have been followed and necessary precautions taken, and these are detailed in the paper, then sensitive research areas can be explored without automatically qualifying for a full ethics review.
5. All Papers on NLP ethics require Ethics Review: A paper engaging with questions on NLP ethics and fairness in and of itself does not justify an in-depth ethics review. However, many of these papers do tend to engage with topics that are potentially sensitive and may sometimes need an in-depth review. But this determination should be based on such additional contexts, rather than merely for the topic being NLP ethics.

Checklist for Responsible NLP Research

Most questions in this (checklist) address the kinds of information that is often asked by reviewers (e.g. experimental details, annotation protocols, IRB approval, etc.). For every submission, it checks whether the authors describe the limitations and potential risks of the work. If there’re scientific artifacts, it checks if the authors cite the creators of artifacts, and if the authors discuss the license or terms for use or distribution of any artifacts, if the use of existing artifact(s) was consistent with their intended use, the steps taken to check whether the data that was collected or used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect or anonymize it. Moreover, it checks for documentation and statistics description of the artifacts. When computational experiments are conducted, it checks if the authors report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used, discuss the experimental setup, including hyperparameter search and best-found hyperparameter values, report descriptive statistics about the results, and use any existing packages. For work involving human participants, it checks whether the authors report the full text of instructions given to participants, report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, discuss if such payment is adequate, discuss whether and how consent was obtained, get approval or exempt for the data collection protocol from an ethics review board, as well as report the basic demographic and geographic characteristics of the annotator population that is the source of the data. On top of that, it checks for the use of AI assistants for the work submitted. This allows the checklist to serve as a kind of FAQ for the reviewers, pre-empting questions that would otherwise be asked in the author response period. Reviewers are asked to use the checklist information as one of the factors in their evaluation.

Ethical concerns examples

Ethics cannot be “one size fits all” as it is often about dealing with dilemmas, details that change the nature of the problem, evaluation biases, etc. We therefore do not want to provide an ethics checklist, which would give the false impression that everything is in it and that your research is ethical provided you checked all the boxes. Instead we believe ethics is about thinking differently about one’s research and the ethical considerations section is here for authors to detail this part of their work.

We provide here some examples of ethical concerns that should be addressed, which are not exhaustive in any way.

Data collection/creation

Microworkers remuneration

In case microworking crowdsourcing was used, the paper should state how a fair rate of pay was determined and how the researchers ensured that crowd workers were compensated fairly.

Scraping Datasets

A distinction should be made between reusing existing scraped datasets (without redistributing them) and scraping new ones (with the intent to distribute them).

Identity characteristics

Well-known pitfalls in this area that should be avoided: treating identity characteristics as something which can be reliably ascribed (e.g. based on names) and also essential properties of humans and drawn from fixed sets.

Environmental impact

Emerging best practices include calculating how much energy a given experiment or methodology requires and publishing that information.

Appendix

Temporary, temporally sensitive bugs

The setup for platforms such as SoftConf or OpenReview should be meticulously prepared and thoroughly tested before implementation. Unfortunately, this was not the case for LREC-COLING, leading to several challenges:

• Submissions flagged for ethical issues had to be manually copied (or shared) to the "Ethics Review Track" by program committee chairs multiple times.
• Ethical issues reported by technical reviewers were not accessible to ethical reviewers. Instead, they had to be downloaded into a spreadsheet by track chairs, necessitating the maintenance of an external document.
• Due to the restriction of only one meta-review per submission, ethics chairs were unable to enter meta-reviews of ethics assessments.
• Submissions within the "Ethics Review Track" were also associated with another technical track. This created confusion for area chairs of technical tracks, as they were unable to distinguish reviewers assigned by ethics review chairs.
• It would be very, very helpful if there is an explicit additional functionality that could make managing Ethics flagged papers and their workflow directly in Openreview (instead of keeping things in spreadsheets)
• It would help with ethics review planning to clarify ahead of the conference how involved/available ARR ethics chairs can be for the specific event. Technical processes can also be shared with the conference ethics chairs ahead of time as it is not intuitive to understand what happens through Open Review and what has to be handled separately.
• 2/ the ability to be notified when ethics reviews are submitted. Currently, there is no other way for chairs to check that than accessing the paper channel on Open Review. 3/ the ability to communicate with ethics reviewers through Open Review, including some way for reviewers to differentiate notifications pertaining to scientific vs. ethics reviewing.

Other suggestions to facilitate ethics chairs’ work

• Clarify the extent of the ethics committee action: does it include the workshops? The demo or industry papers? The conditional-accept papers shepherding?
• We recommend that a checkbox on the submission form be provided to allow authors to aver that they have legally binding permission to publish their data; and to indemnify the ACL and committees for any legal repercussions.
• Review scheduling should incorporate the ethics review process from the initial planning stages. Streamlining the sharing of submissions flagged for ethics issues with the ethics committee would greatly enhance efficiency.