Evaluation in Pilot-Based Research Projects: A Hierarchical Approach

Thomas Zefferer*

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference paper


The evaluation of project results is a crucial part of most research projects in the information-technology domain. In particular, this applies to projects that develop solutions for the public sector and test them by means of pilot applications. In these projects, involved stakeholders require meaningful evaluation results to derive lessons learned and to steer future research activities in the right directions. Today, most projects define and apply their own evaluation schemes. This yields evaluation results that are difficult to compare between projects. Sometimes, inconsistent evaluation schemes are even applied within a project. In these cases, even evaluation results of a single project lack comparability. In absence of coherent evaluation schemes, lessons learned from conducted evaluation processes cannot be aggregated to a coherent holistic picture and the overall gain of research projects remains limited. To address this issue, we propose and introduce an evaluation scheme for arbitrary pilot-based research projects targeting the public sector. The proposed scheme follows a hierarchical approach. Concretely, it organizes evaluation criteria on different layers of abstraction. Furthermore, the proposed scheme describes in detail necessary process steps to carry out evaluations using the defined criteria. This way, the proposed scheme enables in-depth evaluations of research projects and their pilot applications. At the same time, it assures that obtained evaluation results remain comparable anytime. The hierarchical evaluation scheme introduced in this article has been successfully applied to the international research project FutureTrust, demonstrating its practical applicability, and showing its potential also for future research projects.

Original languageEnglish
Title of host publicationWeb Information Systems and Technologies - 14th International Conference, WEBIST 2018, Revised Selected Papers
EditorsMaría José Escalona, Francisco Domínguez Mayo, Tim A. Majchrzak, Valérie Monfort
Number of pages22
ISBN (Print)9783030353292
Publication statusPublished - 2020
Externally publishedYes
Event14th International Conference on Web Information Systems and Technologies, WEBIST 2018 - Seville, Spain
Duration: 18 Sep 201820 Sep 2018

Publication series

NameLecture Notes in Business Information Processing
Volume372 LNBIP
ISSN (Print)1865-1348
ISSN (Electronic)1865-1356


Conference14th International Conference on Web Information Systems and Technologies, WEBIST 2018


  • Evaluation scheme
  • Hierarchical evaluation model
  • Project evaluation

ASJC Scopus subject areas

  • Management Information Systems
  • Control and Systems Engineering
  • Business and International Management
  • Information Systems
  • Modelling and Simulation
  • Information Systems and Management


Dive into the research topics of 'Evaluation in Pilot-Based Research Projects: A Hierarchical Approach'. Together they form a unique fingerprint.

Cite this