Replication of research experiments is important for establishing the validity and generalizability of findings, building a cumulative body of knowledge, and addressing issues of publication bias. The quest for replication led to the concept of scientific workflow, a structured and systematic process for carrying out research that defines a series of steps, methods, and tools needed to collect and analyze data, and generate results. In this study, we propose a cloud-based framework built upon open source software, which facilitates the construction and execution of workflows for the replication/reproduction of software quality studies. To demonstrate its feasibility, we describe the replication of a software quality experiment on automatically detecting code smells with machine learning techniques. The proposed framework can mitigate two types of validity threats in software quality experiments: (i) internal validity threats due to instrumentation, since the same measurement instruments can be used in replications, thus not affecting the validity of the results, and (ii) external validity threats due to reduced generalizability, since different researchers can more easily replicate experiments with different settings, populations, and contexts while reusing the same scientific workflow.

This research was partially funded by ISTAR’s projects FCT UIDB/04466/2020 and UIDP/04466/2020

Scientific workflow,Software quality,Quality assessment,Replication,Code smells,Open source

• Computer and Information Sciences - Natural Sciences

This publication is an output of the following project(s):

• Alternative Shaper
• Impact of Blockchains and Decentralised Ledger Technologies