Peer Review of Search Strategies – AHRQ – June 2012

Posted on June 19, 2012. Filed under: Research | Tags: , , |

Peer Review of Search Strategies – AHRQ – June 2012

Prepared for: Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services

“Structured Abstract

Background. Many steps in the preparation of effectiveness and comparative effectiveness reviews (CERs) by the Agency for Healthcare Research and Quality‘s Effective Health Care (EHC) Program involve outside review and input (e.g., by Key Informants, the Technical Expert Panel, Draft Report Peer Reviewers). However, development of bibliographic database search strategies is currently not consistently peer reviewed. An opportunity exists for the EHC Program to leverage its expert searchers across the 14 Evidence-based Practice Centers (EPCs) in the Program by implementing a process of peer review of search strategies.

Objective. Due to the number and frequency of observed errors in published search strategies uncovered in the research literature, an evaluation of the feasibility of instituting a peer review process is warranted.
Specific Aims:
1. Evaluate whether the PRESS instrument or no-instrument (‗free-form‘ evaluations) is preferred by Technical Expert Peer Reviewers (TEPRs) of search strategies.
2. Evaluate the usefulness of a peer review process for database search strategies, that is, do peer reviews change search strategies.
3. Evaluate the costs of implementing a formal peer review of search strategies program as a part of the review process.

Methods. We identified current research protocol phase CER search strategies to review, TEPRs from across the EHC Program, and the original expert searchers who worked on developing each of the search strategies. Each TEPR was assigned to either the control group who only wrote ―free-form‖ reviews or to the group who initially wrote a ―free-form‖ review, then trained using the PRESS Instrument, and finally completed the last review using the PRESS instrument. Original expert searchers were asked to comment on the reviews of their searches. One study researcher administered the peer review process and log time required to perform it, while the other, blinded study researcher analyzed qualitative and quantitative data derived from the reviews, as well as demographic information about the TEPR and original expert searchers.

Results. As a whole, the group of people available to conduct and review search strategies in the EHC Program is professionally educated and very experienced. Of the 24 respondents to the demographic survey, 20 (83%) have master‘s degrees in library science. Most have more than 10 years‘ experience as a librarian or other information professional. Sixty-seven percent of respondents have more than 5 years experience contributing to systematic reviews, with 63 percent having contributed to more than 10 systematic reviews. These experienced searchers have a variety of relationships with the EPCs for whom they do searches. Of the 25 peer reviewers invited to participate, 24 completed the initial free-form round of reviews, and 15 completed either the PRESS review or a second free-form review as part of our control group. For the most part peer reviewers were positive about the review process, although many hesitated to incorporate the review process into their current workflow. All of the reviewers found the background material (systematic review protocol) helpful to the review. Of those who used the PRESS instrument, 82 percent (9) indicated that the instrument was helpful, 18 percent (2) reported that it was neither helpful nor limiting, and none of the reviewers indicated that the PRESS instrument was limiting. The PRESS instrument reviews contained more recommendations on the whole and in particular had more comments that could be termed error detection—specific comments about spelling or syntax indicating that a mistake had been made. In 97 percent of cases, the original searcher indicated that the comments did not cause them to alter their search strategies.

Conclusions. While the results of this study suggest that if a formal peer review process is to be valuable then it would need to be both timely and timed for a window of opportunity immediately prior to the finalization of the protocol. Even if a formal peer review process is not implemented, the PRESS instrument could be useful in informal peer review or even self review. If review of search strategies is to take place, then these results suggest that the use of the PRESS instrument would cut down the time taken, increase the likelihood of response and be more effective in identifying actual errors in search strategies. Additionally, the content of the reviews indicates that there are several search tactics for which there is no consensus, and further research could help us to understand variation in practice around such issues as limits, searching for observational studies, and searching for outcomes and comparators. The process of reviewing other searchers‘ work can bring these issues to light, and a peer review-like process could be used to start investigations and discussions of what techniques work and why. Finally, many of the reviewers commented on the difficulty of reading the search strategies as currently presented. The EHC Program currently has no standards for reporting search strategies, and there is no recognized standard for reporting search strategies. Adopting standard of reporting designed to facilitate review may make it easier to review search strategies both internally and when reported to the public.”

Read Full Post | Make a Comment ( Comments Off on Peer Review of Search Strategies – AHRQ – June 2012 )

Peer review in scientific publications – UK House of Commons Science and Technology Committee – Eighth Report – 18 July 2011

Posted on August 23, 2011. Filed under: Research | Tags: , |

Peer review in scientific publications – UK House of Commons Science and Technology  Committee – Eighth Report – 18 July 2011
Extract from the summary

“Peer review in scholarly publishing, in one form or another, has always been regarded as crucial to the reputation and reliability of scientific research. In recent years there have been an increasing number of reports and articles assessing the current state of peer review. In view of the importance of evidence-based scientific information to government, it seemed appropriate to undertake a detailed examination of the current peer-review system as used in scientific publications. Both to see whether it is operating effectively and to shine light on new and innovative approaches. We also explored some of the broader issues around research impact, publication ethics and research integrity.

We found that despite the many criticisms and the little solid evidence on the efficacy of pre-publication editorial peer review, it is considered by many as important and not something that can be dispensed with. There are, however, many ways in which current pre-publication peer-review practices can and should be improved and optimised, although we recognise that different types of peer review are suitable to different disciplines and research communities. Innovative approaches—such as the use of pre-print servers, open peer review, increased transparency and online repository-style journals—should be explored by publishers, in consultation with their journals and taking into account the requirements of their research communities. Some of these new approaches may help to reduce the necessary burden on researchers, and also help accelerate the pace of publication of research. We encourage greater recognition of the work carried out by reviewers, by both publishers and employers. All publishers need to have in place systems for recording and acknowledging the contribution of those involved in peer review.”  … continues

Alternatives to Peer Review in Research Project Funding – RAND – May 2011


“Peer review is considered the gold standard for reviewing research proposals. However, it is not always the best methodology for every research funding process. Public and private funders that support research as wide-ranging as basic sciences, defence technologies and social sciences utilise a diverse set of strategies to advance knowledge in their respective fields. This report highlights a set of established approaches that offer unique alternatives to traditional peer review – alternatives that address many of the shortcomings in peer review effectiveness and efficiency. The appropriateness of these different approaches will depend on the funder’s organisational structure and mission, as well as short- and long-term financial realities. We hope that the information presented in this folio of cards will inspire thinking amongst research funders by showing how the research funding process can be changed, and give funders the confidence to try novel methods by explaining where and how similar approaches have been used previously.”

Read Full Post | Make a Comment ( None so far )

Evaluating Grant Peer Review in the Health Sciences: A review of the literature RAND Corporation, August 2009

Posted on August 24, 2009. Filed under: Research | Tags: , |

Evaluating Grant Peer Review in the Health Sciences: A review of the literature RAND Corporation, August 2009
By: Sharif Ismail, Alice Farrands, Steven Wooding

“More than 95% of the £2 billion of public funding for medical research each year in the UK is allocated by peer review. Long viewed as a respected process of quality assurance for research, grant peer review has lately been criticised by a growing number of people within the scientific community and without. Detractors highlight its perceived inefficiency, and structural flaws that compromise its effectiveness in allocating funding. This report presents the findings of a wide-ranging literature review to evaluate these criticisms. It concludes with a short discussion of simple modifications to the peer review process that might help to address some of them. The research for the report was conducted with funding support from RAND Europe’s Health R&D Policy Research Unit with the Department of Health (England). It is available in English only.:

Read Full Post | Make a Comment ( None so far )

Liked it here?
Why not try sites on the blogroll...