The Statement
1. ISCB recommends that bioinformatics funding agencies encourage reviewers to follow those URLs in bioinformatics grant proposals that provide information important to the grant review. Funding agencies should not forbid reviewers from following URLs in grant proposals.
2. ISCB recommends that bioinformatics funding agencies establish proxy Web servers for use by reviewers to facilitate anonymous access to applicant web sites.
Background
This statement is motivated in part by the current grants policy of the US National Institutes of Health, which states "URLs may not be used to provide information necessary to the review because reviewers are under no obligation to view the Internet sites. Moreover, reviewers are cautioned that they should not directly access an Internet site as it could compromise their anonymity." (See URL http://grants2.nih.gov/grants/funding/phs398/phs398.pdf) That is, examination of web logs by an applicant might reveal the identity of a reviewer.
In addition, several NIH review panels now take the more restrictive position of forbidding reviewers to follow URLs.
Submission of software via a CD-ROM in conjunction with a grant would be one way to avoid using the Internet to demonstrate software or database capabilities to reviewers, however, the NIH also does not allow CD-ROMs to be submitted with a grant proposal.
Funding agencies are also concerned that URLs could be used to circumvent grant page restrictions if an applicant places additional information on their web site.
Another concern behind this policy is that there is no permanent record of the contents of any URL, and that if a grant PI later challenges a review by saying, for example, "This review is not competent because the reviewer's objection is clearly addressed by this information on my web site" there is no way for NIH to later validate what was on the PI's web site at an earlier time.
Rationale
There are several related issues here. One issue pertains to third-party Web sites ("Third-party URLs") such as articles in online journals. Another relates to Web sites maintained by the applicant PI or their associates ("PI URLs");
1. Third-party URLs. There is simply no question that reviewers should be allowed to access third-party sites. Grant applications already contain extensive references to third-party information, namely scientific publications. As the model of scientific publishing evolves, some publications are available only through the web, and others are most efficiently accessed via the web. It is a waste of the reviewers’ time to forbid them from accessing such sites. With third-party sites there are no issues of anonymity nor of circumventing page limits.
2. PI URLs. The deliverables of many bioinformatics projects are databases or software packages that are resident on the web, and can most efficiently be accessed by reviewers through the web. Direct review of database and software packages via the web is both extremely informative, and extremely time efficient for the reviewer. To prevent the reviewer from interacting directly with a database or software package prevents the reviewer from having first-hand knowledge of the database or software that is extremely valuable, and introduces a serious risk that their knowledge is inadequate to perform an informed review. In addition, for large projects, part of their funding typically covers a service component that is also important for reviewers to assess. However, reviewers should be cautioned against basing a negative judgment on a single problematic session that could be caused by network outages that are beyond the control of the project. And small projects in particular cannot be expected to provide perfect 24x7 service.
It should be left to the discretion of the reviewer which URLs they consider important to the grant review. Although bioinformatics database and software-related applications are likely candidate projects where consideration of Web information will be important, the reviewer will be the best judge of when to follow a URL.
The anonymity concern of the NIH can be solved if reviewers use proxy servers, which shield the identity of the person accessing a web site. We recommend that the NIH fund the creation of proxy servers for use by grant reviewers; ISCB would be willing to host and operate such proxy servers.
NIH's page limitation concern can be solved if reviewers are not REQUIRED to read anything accessible through URLs.
The NIH concern of the lack of a permanent record for URLs is valid, but is outweighed by the other factors. Note also that reviewers are already influenced by many other subjective factors that are not part of the grant application.
27 comments:
The Board of Directors was generally in favor of this policy statement and is eager to hear from the membership. Directors felt it was important not just to publish the statement (if it is approved) but also to take action to address this issue. One possibility is that ISCB might be able to help with technology to allow reviewers to visit grant URLs anonymously. ISCB members are requested to provide feedback on this idea as well.
Seems like a good proposal.
I agree that it can be limiting if reviewers don't have access to certain Internet resources to conduct their reviews. One potential danger we need to acknowledge is that some researchers may be tempted to "bloat" their proposal by putting a lot of the information on the web. Reviwers usually have to read a lot to begin with, so this might be a problem. I think that the NIH should certainly permit the use of URLs, but reviewers should not be required to visit the sites as part of their decision-making process unless there is some critical piece of information that could not be conveyed in the grant application (e.g. a video, protein conformational change animation, etc.). The writer should convey a sense of importance when citing a URL to let the reviewer know what's there, how important it is to understanding a point or evaluating the grant, and how long it might take. For example, writers shouldn't just say "For more information, see http://www.blah.blah". They should instead say something like "A demonstration of how the algorithm is able to learn from examples is at http://www.blah.blah, where users can pick two protein structures from the database and within 5 minutes have a visualization of how they are expected to interact with one another".
As both a recipient and a reviewer of NIH grants, I think this proposal is neither necessary nor a good idea.
Visiting a computer URL is analogous to visiting an experimental laboratory, and neither should be part of normal grant review.
In both cases, instead, the proposer should describe in clear, simple, lucid prose the significance and innovation of the ideas and approach, then back it up with data (such as number of URL hits, user base, letters of support, citations in which the URL is used, etc.).
It does a disservice to the computational biology community to advance the notion that we need to be treated specially because we cannot state clearly why our work is useful and important.
BTW, I am a computer scientist and a software developer.
I think this proposal is wrongheaded and would be damaging to the ISCB.
Every discipline has aspects of their work that do not fit perfectly in 25 11pt pages. However, this constraint allows for consistent and fair review, without unduly burdening reviewers.
Including a bioinformatics PI's database website URL would be analgous to including all the sequence traces from a sequencing lab or a vial of crystallized protein and the reams of diffraction data from a structural biology lab. None of these are necessary, and all can be well described in the clear lucid prose of a good grant proposal.
Indeed, for review of an engineering effort the sheen of a sexy website is often much less important than the the code itself. Should all the code also be included with the grant review? For science-based (as opposed to engineering-based) bioinformatics, I cannot think of any case where text would suffice.
I should note that I am a recipient of NIH grants for web-based databases. I have found many problems in NIH's review process for bioinformatics grants, but accessing URLs was never an issue.
It would be a travesty for ISCB to squander its political capital on such a trivial and debatable issue, when much more serious issues exist.
I agree with the above comments that this proposal is neither necessary nor a good idea. The NIH policy does not prohibit the inclusion of any URL in a grant proposal. It only states that URLs may not be used to provide information “necessary” to the review. It is true that current grant applications already contain extensive references to third-party information, but most of these references only provide supportive information, which can be omitted without affecting the integrity of the proposal (at least in theory). If a certain website or software package is deemed to be vital to the proposal, the applicant should summarize the main features of the website or software package clearly in the proposal, instead of asking the reviewers to “explore” these website or software themselves. Most of these websites and software packages are complicated and may not be used by the reviewer on a daily basis. It is extremely unproductive to ask the reviewers to “discover” the advantages of these website or software themselves.
a complete package should be provided online as hyperlinks for the reviewers, who can download free of cost,any kind of software that is needed to support him for the review. The cost of these should be met by the funds from the commercial sponsors
While I understand the argument "we do not have to visit wet labs to evaluate what they are doing there", my impression is that the very concept of a *user* - and this is usually not the case of a wet lab - makes that visits are in fact compulsory. A database, for example, has several features which need to be reviewed *in practice* (description with words will hardly tell enough in terms of user-friendliness, for example) and this should be part of the reviewing process. The second thing, of course, in genomics, is assessment of the *quality of annotation* this can hardly be tested by means other than URLs, where the reviewer tests his (her) own knowledge and compares it with what is offered. The only problem I see associated to the use of URLs is confidentiality, but this can certainly be sorted out.
In brief, I am quite puzzled by the old-fashioned reactions against using URLs for assessment of some relevant work. Also, this would be a move to improve recognition of those who use most of their time for supporting the community as a whole, precisely because databases are used much more than quoted! I did not say anything about software that provide new methods, but, here too, the reviewer needs to test it with his (her) own data.
I do not see how somebody doing "hands on" research can really be seriously negative about presentation of URL as part of a work in in silico biology.
Thank you very much for inviting my comment.
Grant proposal authors are in fact free to include URLs in their grant proposals. I have always found the admonishon to the reviewers that their identity might be compromised to be somewhat silly. A reviewer can access a URL from a library, for example, and remain completely anonymous. Moreover, most web sites that have been around a while have sufficient traffic to insure that the grant proposal author can remain anonymous by virtue of being one of many visitors.
I think individual researchers should remain free to put whatever information that may be pertinent to their research on the web that they choose, and that it should be up to the individual reviewers to decide whether they should consider that information. In general, anything that causes, during review, the reviewer to consider a grant proposals as one-off, stand-alone entity independent of anything else that exists in that proposal requires a degree of suspended disbelief that I personally find quite unnatural and limiting to a full interpretation what is being proposed. I am always interested in how new proposals fit into the larger context. If a reviewer is aware of previous effort of a grant proposal author or team, established or new, that provides evidence that they are productive and can deliver what they propose then I think reviewers should be able to bring that material into their consideration, overtly.
I support anything that increases the degree of communication of ideas without compromising the anonymity of the reviewers, so this is a good proposal.
What would ISCB ever do, however, if someone's cited content was unavailable during a critical review period... and (sorry to ask) what assurance are there that the ISCB won't use the information from the resulting traffic to their own, or someone else's, advantage?
Reading over the comments it strikes me that the membership is fairly divided on the issue. In addition it seems that there were some good points made about why and why not to allow it.
Instead of simply asking people's opinion on whether the proposal should be put forward, it might be advisable to ask people to recommend the policies they consider appropriate, collate them, and put to a vote which people think is best.
It seems to me a lot hinges on length constraints, the requirement to review the material, and the issue of permanence. I think it would therefore be preferable to argue for additional media being allowed for submission such as video which could be put on a CD-ROM. One advantage is this might be supported by experimental biologists who are starting to generate more movies as part of an experiment. Another is that there could be an explicit time limit placed on it. Third the database or software used to generate the demo could be placed on the CD-ROM to maintain a level of verifiability that the software can do what it claims.
There are certainly issues with such a proposal such as whether to allow editing to shorten the time taken to watch vs. usability, but it does seem to provide alternative advantages such as - easily enforced strict limits on volume, usefulness in other biological fields, permanent record, and the fact that a demo is in essence the succinct summary of a product, just like a proposal is a summary of one's research goals.
This proposal is a bad idea. It tends to favor flashy computer technique over solid biological science. In silico biology is just a set of tools for the pursuit of real biology. If the tools actually are scientifically useful, that can be explained and proven in old-fashioned literate text. If they aren't, that's easy to obscure with a glitzy web interface - which should be against the rules.
I totally agree with ISCB's initiative to support inclusion of URL's in grant proposals. Prohibiting reviewers from accessing information on the web makes them less efficient and adds to the reviewer's fatigue. Seeing the product is different from reading the description of the product no matter how effectively it is presented. Allowing reviewers to visit the URL's is a necessary change to ensure fair evaluation of bioinformatics proposals. Issues about reviewer's anonymity can be easily addressed by proxy servers.
I believe NIH is being too restrictive in what it will allow to be included as supporting data in the Appendices of NIH applications, to the great disadvantage of many applicants. I support your efforts to get NIH to permit all types of data to be included in Appendices, and also longer Addenda in Grant Application updates shortly before Study Sections meet to evaluate grant applications.
I agree that reviewers should be allowed to visit pertinent websites and that funding agencies should provide a proxy service to allow that to be done anonymously, and should remove these warnings and prohibitions. As to the use of websites to expand past page limits and the dangers of dynamic websites, I suggest that reviewers will be intelligent enough to be aware of those issues.
I think most have been said above. I just want to add that in *some* proposals, the use experience is key to the success of a project. For example, in project where a collaborative database is to be developed, demonstrating the a usable tool has already been developed is really important for the evaluation. With all due respect to serious programmers and algorithms developers, usable software is something that biology misses.
Perhaps the wording of the pledge should emphesize that suitable criteria be made public as to when using a URL is acceptible (i.e. when a technology presented in the proposal involves user interfaces or interactive graphics).
For complete disclousre, I never recieved or reviewed NIH grants, but I did participate in grant reviewes and wrote grant proposals for other aganecies.
Eitan Rubin, Ben Gurion Univeristy
This issue applies not only to NIH, but also to funding agencies in other countries (e.g. ARC and NHMRC here in Australia) that likewise prohibit or restrict the citation of URLs in grant proposals.
Sometimes it is necessary to refer not just to papers or databases, but to evolving collaborative projects, web services, new versions of software or markup languages, commercial products etc. where no adequately up-to-date print source exists, or where online availability is the very point. In my experience, funding agencies allow such citations in practice, if citing a URL is obviously the right (or only) thing to do and does not advantage one application over another.
NIH and most other funding agencies track evolving best practice among the communities they serve. If ISCB finds a consensus among its members -- or two or three frequent opinions -- bringing this to the attention of NIH should not be seen as "burning political capital"; it is simply ISCB contributing, on behalf of its members, to the debate.
I can see arguments both ways, but favor slightly the notion that grant-writers ought to be able to describe their contributions with clean prose, not relying on a website to make up for a poorly composed application.
However, if we do choose to move forward with this proposal, please consider rewriting it as a scientist rather than as a marketer: the "extremely" frequent use of the word "extremely" (inter alia), blows the claims all out of proportion. It is exactly this unscientific and overhyped style of writing that has caused the public to be very skeptical of scientists and their claims, not to mention contributing to the stridency of modern political discourse. Let's state the facts and let them argue for themselves.
None of the comments have mentioned the problem with electronic-only journal citations. It would seem that this part of the problem is a no-brainer, but do NIH guidelines need to change in order to accommodate it? A couple of people mentioned that proposals are allowed to have URLs in them, but this seems irrelevant if reviewers are banned from visiting them.
Does anyone know what the practice is for other granting agencies (government or private)? Surely this issue must have come up in other contexts than bioinformatics - is there any useful experience to draw on from other communities?
In weighing the various comments here, I think that the key question is whether the proposed project is worth the money requested. In light of this, any tools available that may help the reviewers evaluate this issue should be included. Although it is true that a lackluster proposal may get greater favor after reviewers can view the materials live and in action, is this a bad thing? What is critical is demonstration of tax dollar value.
However, the proposal should be written as if it must stand alone without URL support. Also, the submitter should have zero recourse to defend his proposal based on the materials of that site. Additionally, I think including a website is the best way to ensure submitters are providing their software freely.
At the risk of provoking incendiary responses, I move that we need to embrace the future. For those data junkies out there: What correlation do you expect between one's years post- PhD and support for internal URLs?
In short-- give reviewers the option. And lets also give reviewers the benefit of the doubt that they are savvy enough to reject entrancement by fancy yet data poor graphics and flash animations.
Monica Horvath
NIEHS/NIH
Once reviewer visits the URL, his/her identity may be uncoverd due to time zone. It's ridiculous for a reviewer get up at midnight to visit the URLs. So anonymous is not guaranteed using the three methods mentioned. An alternative is to proposer submit the demo/screenshots or something necessary to ISCB hosted server for reviewer's review.
I'm not clear what are the cases of citing URLs is a must although it adds more means of communication.
I agree with the policy
Let's assume there are two competitive proposals for the same pot of money; both are scientifically and semnatically at the same level, but one has a fancy website behind it. Although the information on it may not contribute to the proposal, it may make the reviewers tend more to the "fancy" one, leaving scientific reasoning behind. Hence format control of the URLs would be needed, but this is certainly not implementable. Any supplemental data should be given without links to fancy and blinding websites.
Many of the comments against this
statement assume that reviewers
will nonetheless visit web sites under
the current NIH policy. What particularly concerns me is that I
have been a member of an NIH review panel that was explicitly forbidden,
by NIH staff, from visiting web sites
from the projects we were reviewing.
When I pointed out this conflicted with the NIH policy, they refused
to alter their stand on this. Yes,
some reviewers will still disregard
that advice, but others may not,
and this atmosphere makes it much
less likely that aspects of a web
site would be discussed during a
review.
I think that URLs are absolutely a bad idea in proposals. If online projects or databases haven't been documented in publications or submissions, then they should have been. Personal online (unreferreed) publication is not an acceptable method to share prior work.
I support this proposal for reasons that are already outlined in several comments. I have visited websites under review in the past, reasoning that a rule designed to avoid circumvention of page limits simply doesn't apply if the product is a website. The analogy to published papers is apt. Should NIH prohibit reviewers from reading publications by the applicant? If not, then they should not prohibit reviewers from visiting websites that play the analogous role of presenting past research.
In any case, the notion that reviewers are under no obligation to visit a website as part of their review (which is reasonable, and in the spirit of the page limit) is quite different from a prohibition against doing so.
Manuel and Debra gave a talk to the Board of Directors regarding ISCB-SC. Board members asked whether Manuel thought ISCB-SC membership would help members' careers. One board member was dismayed aboutsportsbook segregating junior from senior members of ISCB, saying he did not see the necessity. Some members were enthusiastic about mentoring programs. ISCB President Michael Gribskov suggested that a senior ISCB member help apply for grant support to run some of the ISCB-SC programs.
http://www.enterbet.com
Post a Comment