| Number 2 |
Tips for managing successful conference abstract submissions & reviews.
Part II – Managing Reviews.
Trevor Strome
Note: This article is a continuation of Issue Number 1 in which we discussed tips for managing the submission aspect of an abstract review competition. Part II focuses on management of the review portion.Review Criteria Overview
As mentioned in the September 2006 issue, conference abstracts are very often the only published work of a given scientific investigation. The need therefore exists to ensure that the high-quality research worthy of publication is not lost due to a biased or arbitrary review process. To ensure this, abstract competitions need to ensure fair, unbiased, and consistent adjudication of submissions. The key to making this happen is to ensure that the abstract review criteria are constructed in such a way as to promote accountability and consistently to the abstract judging process.
While the simplest with which to review abstracts, the basic Likart scale approach is highly prone to reviewer bias (both intentional and unintentional) and makes substantiating a particular grade very difficult. For example, two reviewers might place very different weights on aspects of the abstract and therefore arrive at completely different scores.
Structured Review Criteria
To help address the issue of arbitrary weighting and scoring, an increasing number of organizations are adopting (and requiring) a more sophisticated approach to reviewing abstract submissions. This new way is the structured criteria approach. Essentially, a structured abstract is an enforced organization of an abstract's content into major headings (typically Introduction, Methods, Results, and Conclusions) rather than the traditional unstructured paragraph(s) format.(For more information about structured abstracts, visit http://research.mlanet.org/structured_abstract.html).
The structured review criteria usually encompasses several weighted categories (which mirror the structured abstract headings) with each encapsulating several criteria related to the category. For example, a structured abstract scoring criteria set worth a total of 25 points might be broken down into the following five criteria categories (each worth five points): Introduction, Methods, Results, Conclusions, and Overall.
This structured approach provides much more guidance to the reviewer on how to score a given category. For example, for the Introduction category, the criteria could be listed as follows:
Type-Specific Criteria
An even more sophisticated approach, building upon the structured review criteria concept, is to tailor the review criteria according to the various type of submissions expected. This approach can be helpful, for example, in the case when abstracts of both qualitative and quantitative research studies are expected to be submitted. Typically these types of studies are designed differently and employ very different methods for analysis. In this scenario, submitters are asked to self-select the investigation type (i.e., qualitative or quantitative) upon abstract submission, and this selection will trigger the appropriate review criteria to be displayed to reviewers upon review.
It is important the evaluation criteria for each investigation type do not differ too significantly. It is suggested that a "core" set of criteria are established first (based on the structured approach) but the various category descriptions (i.e., Methods) are customized for the various investigation types. The example below illustrates how "methods" criteria might differ between a controlled clinical trial and a survey.
| Score | Controlled Clinical Trial | Survey |
| 1 | Non-randomized | Non-representative sampling and <80% response rate |
| 2 | Pseudo-randomized (day of the week, flip coin, etc) | Non-representative sampling or <80% response rate |
| 3 | Randomized (random numbers tables, etc) | Representative sampling and >80% response rate |
Reviewer Workload
Once the abstract review criteria have been decided upon, it is important to carefully select the reviewers of the abstracts. It is important to know the strengths and weaknesses of each of your reviewers, because they will each have their own areas of expertise. Such expertise may be within a certain topic or content area, or may be related to methodology. Reviewers may be unwilling, unable, or uncomfortable reviewing abstracts that fall outside their area of expertise.When recruiting reviewers, it is important to have enough to cover all topic and methodological areas so that reviewers are able to apply their best judgment. To ensure that the workload for reviewers is not too great, we have found that 15 to 20 abstracts is about as many as should be assigned to any one reviewer to prevent reviewers from losing interest and not carefully scoring abstracts.
About the author: Trevor Strome MSc, PMP is Vice President (Business Development) of VS Communications Inc. and product manager of VS Review. Trevor can be reached by email at: trevor@vscommunications.com.