Reviewers Guidance

Thank you for agreeing to be a reviewer for RSECon24 submissions. You are an essential part of making the conference a success.

The Programme Team has put together this guidance to help you understand what is expected from you and to explain the Review Criteria to ensure consistency between reviews. If anything is unclear, please reach out to the Programme Team by email at [email protected].

Key Dates

• Reviewer guidance sent out to reviewers: May 7th
• Reviewers receive their allocated submissions: May 10th
• Deadline for all reviews to be completed: May 24th

Expectations

As a reviewer we ask that you:

1. Familiarise yourself with the conference’s mission statement and research software development principles. Ensure that your evaluations align with the conference’s focus areas and goals.
2. Read the submission guidelines and review criteria.
3. Maintain confidentiality and integrity. Treat all submissions as confidential documents and refrain from sharing or discussing them with others. Conduct your evaluations objectively and impartially, avoiding any conflicts of interest or bias.
4. Provide respectful, constructive and specific feedback. Offer detailed and constructive feedback to authors, focusing on strengths, weaknesses, and areas for improvement. Provide specific examples or suggestions to support your comments and recommendations. This feedback should be professional, respectful, and courteous, even if you have critical comments or suggestions.
5. Meet review deadlines. Adhere to the timeline and deadlines provided by the conference organisers for completing your reviews. Fulfil your responsibilities as a reviewer promptly and professionally, ensuring that your evaluations are submitted on time. If you are unable to meet the deadlines, please contact the Programme Team as soon as possible.
6. Seek clarification or assistance if needed. If you encounter any issues or have questions about the submissions or review process, don’t hesitate to seek clarification or assistance from the Programme Team.

Each submission will have a minimum of two reviewers. The combination of your grades and feedback with that of your (anonymous) co-reviewer/s will inform the final acceptance of the contribution. We fully expect to see divergence between reviewers; opinions from all members of the community are valid and welcome. The final decision of which submissions are accepted rests with the Programme Team.

Abstention

If there are any reviews you’ve been assigned but wish to decline due to, for example, a conflict of interest, or not feeling able to assess the technical aspect of the submission, please contact the programme team. Please note that sessions are expected to be of general interest to large sections of the RSE community, so abstracts should not expect extensive knowledge in any particular domain.

Bias and Inclusivity

The conference organisers are committed to creating a welcoming and diverse environment for the conference. We ask that you be mindful of common undesirable biases when making your decisions. The first line of defence is awareness [Nature 526, 163 (2015)] so, for your part, we ask that you review the definitions of relevant common forms of bias so that they are in the forefront of your mind as you read submissions.

Common forms of bias to watch out for:

• Affinity bias - favouring those with similar backgrounds, interests or beliefs.
• Confirmation bias - favouring information consistent with your own existing beliefs.
• Affect heuristics - relying on emotions, over objective information, to perform evaluation.

Bear in mind that submissions may come from people for whom English is not their native language, or text is not their preferred method of communication, and ultimately the sessions will be presented live, so text will not be the final format. Where the intent of the text is clear, please disregard any errors of spelling and grammar from your considerations, though constructive feedback is welcomed, as speakers will have an opportunity to amend their abstracts prior to the conference.

The Review Form

Content

When reviewing a submission, you are asked to grade it under each of the review criteria given below. Most of the criteria are graded in the range 1-5, whereby 1 corresponds to a strong negative, 3 corresponds to neutral or uncertain, and 5 a strong positive as shown in the image below.

• Relevance to Audience
  • Is the topic of interest to the RSE community?
  • Is this a novel topic?
• Abstract Quality
  • Does the abstract clearly communicate the idea?
• Internal Consistency
  • Do all components of the submission align (e.g. are prerequisites and outcomes appropriate?)
  • Is the length of the proposed session appropriate?
  • Does the popularity of the topic justify its requested length?
• Viability
  • How will they deliver the submission?
  • How will they support attendees both in-person and on-line?
  • Does the intended number of participants for workshops or BoFs seem achievable and realistic for the scale of the conference (~350 attendees).
  • Have they considered software, tooling and licences? How will they be providing access?
  • Have they provided a clear plan instructing attendees to prepare for the session?
  • Have they specified a plan for recruiting speakers for their panel or BoF and does it appear viable?

Viability is a much easier threshold to meet for most talks and posters but will require thorough consideration for interactive sessions. Viability is instead graded with one of three categories; yes, maybe and no. Please refer to the submission guidelines for the interactive sessions to help you to determine an appropriate grade.

Your Decision

Similar to grades that you provide for the review criteria, you are asked to provide a final grade as to whether you would accept or reject a submission. This is again a graded question where you select a decision ranging from a strong reject to a strong accept.

**Strong Accept: **This is judged to be a high-quality contribution which will add value either via seeding discussion, or presenting a complete and high-quality piece of work. It is well-suited to the audience of the conference.

**Strong Reject: **This submission does not fit well with the audience of the conference, it has not been well thought out, and/or it will not add significantly to existing or previous discussions on the same topic.

Submissions should not be explicitly scored for fit with the conference themes, this year the hope is that speakers will briefly reflect on the research software engineering principles outlined at RSECon23’s keynote during their submissions. These lay central to all research software engineering, so anything appropriate for the conference’s audience should be suitable.

Feedback

We ask that you write a sentence or two of constructive feedback to the submitter, and a sentence of justification to the programme team (which will not be shared with the submitter).

All constructive feedback will be moderated by the programme team, prior to returning it to submitters, to ensure that comments are respectful.

If you are finding it difficult to write constructive feedback, consider highlighting a strength (what you like about the submission) and a weakness (how you would improve the submission).

It is expected that a submission with positive grades for review criteria would also receive an acceptance grade, however if you have reason to diverge from this your reasoning should be clear in your justification to the programme team.

There will be many good submissions, so it is likely that the programme team will ultimately have to reject some abstracts which receive positive feedback. Therefore the strength of the justification that you provide may impact the likelihood of a submission’s selection.