FWF Rules on Quality and Transparency Evaluations, Studies and Research Policy-related Services

FWF contracts for evaluations, studies and other research policy-related services are awarded to independent experts with a strong professional background in the relevant field only. All contracts are awarded using a transparent selection procedure based on pre-defined criteria. In line with the principle of open access to scholarly knowledge, the FWF supports open access not only to research results arising from FWF-funded projects, but also to evaluations, studies and other research policy-related services commissioned by the FWF. Where legally possible, this applies to all results and data arising from these studies, evaluations and other research policy-related services.

This document defines the FWF's approach to commissioned evaluations, studies and other research policy-related services, and the technical criteria for open access to the resulting data and publications. The Appendix defines the structure of the metadata to be provided by evaluators and study authors.

Evaluations

For a funding agency like the FWF, it is crucial that any evaluation of the organisation’s programmes meets the same high quality standards imposed on the review and evaluation of FWF-funded research projects. For this purpose, it is necessary to define and publish the criteria applicable to evaluations of the FWF’s funding programmes.

One of the main objectives of these standards is to ensure independence and impartiality as key quality criteria for proper evaluation activities, and to address the issues raised by the close relationship between evaluators and the FWF as the organisation funding such programme evaluations. In this context, it is necessary to establish a set of rules which, as a whole, constitute a “code of conduct” for these evaluations. These rules mainly pertain to the competence of the evaluators, the systematic planning and execution of evaluations, the integrity and credibility of the evaluators, respect for third-party interests and rights worthy of protection, and the organisation’s responsibility to society1 2.

The award of evaluation contracts and their execution must be decided on and documented by the FWF Executive Board.

Quality

Evaluations of FWF programmes

  • are to be scheduled at appropriate intervals after the launch and in the course of each funding programme (i.e. every 5 to 10 years).
  • are generally only assigned to independent, professional evaluators (or institutions) with relevant experience and expertise. Depending on the specific context of the programme, evaluators (institutions) from outside Austria are given preference wherever possible.
  • With regard to design and execution, these evaluations must generally fulfill the quality standards stipulated by the Austrian Platform for Research and Technology Policy Evaluation; these standards must also be taken into account in the terms of reference (TORs).
  • FWF programme evaluations must be advertised in a public invitation to tender.

Transparency and supervision

For future evaluations of its programmes, the FWF will employ a neutral system of supervision which ensures adherence to international quality standards.

For the sake of maximum transparency and impartiality, these supervision mechanisms will in any case be handled by at least one international, independent expert, if necessary with the involvement of international sister organisations or other relevant institutions (e.g. Science Europe).

These experts will review and comment on the following

  • the content/wording of the TORs
  • the content/wording of the invitation to tender
  • the draft of the final report

These experts will also participate and cooperate in the following activities

  • project presentations by bidders (as necessary)
  • the selection of a bidder for the contract award
  • presentations of interim findings (as necessary)
  • the acceptance of the final report
  • presentations of the findings (as necessary)
  • public presentations of the final report (as necessary)

For each evaluation, the FWF has to prepare its own written comments, which like the evaluation report must be published. The supervising expert's final comments on the performance of the evaluation must also be included in these comments.

Studies and research policy-related services

The award and execution of studies and other research policy-related services must be decided and documented by the FWF Executive Board. Where the budget is €10,000 or higher, the contract must be awarded through a public invitation to tender based on transparent criteria.

One special case is the award of contracts in cooperation with other organisations. In such situations, a joint procedure for the contract award is to be defined on a case-by-case basis. In any case, however, an FWF Executive Board decision and the relevant documentation are required.

The results of research policy-related services (e.g. consultations and similar services) must be documented for the Executive Board, and at least a brief summary must be published on the FWF website in any case.

Quality and supervision

FWF contracts for studies and other research policy-related services are awarded only to independent experts with a strong professional background in the relevant field. Depending on the topic in question, experts residing outside of Austria are to be given preference wherever possible. These studies may be carried out with the involvement of third parties (national partners, international sister organisations or other relevant institutions) if necessary.

Publication policy of reports

The findings of FWF-commissioned evaluations, studies or other research policy-related services are generally published in accordance with the FWF's Open Access Policy, Berlin Declaration on Open Access to Knowledge in the Sciences and Humanities and the Open Government approach by the European Commission.

Technical criteria under the FWF's Open Access Policy

The following technical criteria for open access to evaluations, studies and data must be observed

Metadata

For every published report the following metadata template has to be used

Title
Authors
Author Afiiliations
DOI
Publishing Date
Publication Type
Publication Venue
Pages
Peer Review
Subject Areas
Keywords (max. 6)
Copyright
LicenceThis is an open access publication distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, reproduction and adaptation in any medium and for any purpose provided that it is properly attributed. For attribution, the original author(s), title, publication source and either DOI or URL of the article must be cited.
Cite as
Data Availability
Competing Interests
Funding
Author Contributions
Acknowledgements

1 A good overview of the procedures and best practices of various European funding agencies can be found in the report of the ESF Member Organisation Forum on Evaluation of Funding Schemes and Research Programmes: ESF MOF Report: Evaluation in National Research Funding Agencies: approaches, experiences and case studies (F.Bingen, B.Curran, A.M.Mugabushaka, A.Reinhard, October 2009)

2 Paraphrased from the evaluation standards of the Austrian Platform for Research and Technology Policy Evaluation (fteval).The fteval is a nonpartisan association of all stakeholders involved in evaluation in Austria, www.fteval.at

3 In this context the standards of the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) are to be applied.