Skip to content

Reviewers

As reviewers, you play a critical role in shaping the research landscape of the NLP field. Your assessments influence the quality of publications alongside the kinds of practices and standards which are rewarded. Ethical reviewing is a key part of this responsibility: it ensures that published research is rigorous, transparent, and mindful of its broader impacts across societal scales.

This tutorial will guide you in using various tools as a starting point for your ethical reviewing practice. You’ll also have the opportunity to test and deepen your understanding of research ethics through case studies and simulated reviewing exercises.

1. Recognize That Ethics is Part of Scientific Quality

Ethical concerns are not separate from questions of validity, reproducibility, or contribution — they are integral to assessing a paper’s quality and impact. Raising an ethics flag is crucial to ensuring that potential risks, limitations, or harms are properly identified and addressed.

When reviewing, ask yourself:

  • Does the work show awareness of ethical implications in data collection, modeling, evaluation, or application?
  • Are sensitive aspects (e.g., privacy, bias, harmful content) clearly acknowledged and documented?
  • Is there adequate transparency about limitations, risks, or potential misuse?

A helpful starting point is the Responsible Research Checklist provided by authors at submission time. This can guide your evaluation and help you see whether key considerations have been thoughtfully addressed across potential papers.

2. Pay Attention to All Stakeholders

Any research project encompasses a wide range of stakeholders throughout its lifecycle. Many people contribute and have the potential to be affected by work in NLP:

  • Annotators, crowdworkers, interns: Are recruitment, training, demographics, and potential risks described? Are contributors treated fairly and respectfully?
  • Human participants: Is informed consent obtained? Is anonymity/pseudonymization handled responsibly? Are participants debriefed when appropriate?
  • Downstream users and impacted communities: Does the paper address who might benefit from the system, who might be disadvantaged, and whether performance varies across populations?

As a reviewer, you can flag omissions in such ethical reporting, ask clarifying questions, and encourage authors to be more transparent in their experimental setup and limitations.

3. Consider Broader Impacts

Beyond technical contributions, NLP research may carry risks or unintended consequences such as hate speech amplification, exacerbating adversarial vulnerability, or amplifying anti-democratic values. Look for whether authors discuss:

  • Limitations: Are the boundaries of the work clearly stated (languages, populations, contexts)? Are biases or conflicts of interest acknowledged within the research project?

  • Misuse and dual use: Could the work be applied in harmful ways, particularly for marginalized groups? Did the authors anticipate risks and suggest safeguards to mitigate such concerns?

  • Environmental impact: Are computational resources (GPUs, training time, etc.) reported? Did the authors consider less resource-intensive alternatives or estimate their carbon footprint?

  • Data sourcing and licensing: Is data collected or created under appropriate permissions? Does the dataset contain offensive, sensitive, or non-representative content? Is documentation provided?

4. Ethics Reviewing is a Discussion

You are not alone in making these judgments. The ACL ethics review process is designed to provide support and consistency. If you’re unsure whether a paper warrants further review, flag it — the dedicated ethics reviewers will handle deeper evaluation.

Engage with your fellow reviewers: in meta-reviews and discussions, raise ethical aspects alongside technical ones. This helps normalize ethics as part of standard reviewing practice. You can also build your own judgment through our tutorial resources:

  • Case Studies: Explore hands-on examples in our Tutorial section to see how ethical dilemmas can arise in NLP research.
  • Review Guidelines: Refer to the Ethical Review Recommendations available in the Resources section for structured guidance on assessing submissions.

5. Extend Your Knowledge on Ethics in Reviewing

The ACL Ethics Committee has gathered resources to help reviewers deepen their understanding:

Ethics reviewing is about helping authors improve their work, ensuring fairness and transparency, and protecting the integrity of our field. By engaging with these tutorials and resources, you become an essential part of building a more responsible and sustainable NLP community that can positively benefit society.