Research Artifacts of Guidelines for Addressing Threats to Validity in Replication Studies: Insights from a Focus Group

📋 Overview This study examines how Software Engineering researchers perceive, interpret, and assess replication guidelines, with particular attention to methodological soundness, contextual applicability, and the need for refinement. Drawing on focus group discussions, the study analyses which guideline elements are accepted, adapted, or contested in practice, and identifies key tensions between standardization and context-sensitive decision-making in replication studies. To support transparency and reproducibility, this repository contains the datasets, analysis scripts, focus group transcripts, and supporting materials associated with the paper Guidelines for Addressing Threats to Validity in Replication Studies: Insights from a Focus Group. 📂 Repository Structure Focus Group Validation of Replication Guidelines/ Focus Group Validation of Replication Guidelines/├── data/ │ ├── material/ │ │ ├── Pilot transcript.pdf │ │ ├── Focus Group Transcript.pdf │ │ ├── Guidelines for Replication Studies.pdf │ │ ├── Focus Group Script │ ├── scripts/ │ │ ├── DotPlot.pdf │ │ ├── DotPlot.R │ │ ├── KrippendorffsAlpha.R │ ├── Focus Group-data/ │ │ └── Focus Group Validation of Replication Guidelines.xlsx├── LICENSE├── LICENSE-MIT├── Paper.pdf (This file will be added in the future)└── README.md 📂 data/ Contains all data, materials, and scripts used or produced during the focus group study for validating replication guidelines in Software Engineering. This directory centralizes the empirical evidence supporting the analysis reported in the paper. 📂 data/material/ Contains the materials used to plan, conduct, and document the focus group study. > Download files Pilot transcript.pdfTranscript of the pilot focus group session conducted to refine the discussion protocol and validate the clarity and relevance of the guideline items. Focus Group Transcript.pdfAnonymized transcript of the main focus group session, which serves as the primary qualitative dataset for the analysis. Guidelines for Replication Studies.pdfThe version of the replication guidelines that was evaluated and discussed by the participants during the focus group. Focus Group ScriptThe discussion guide used by the moderator, including prompts, questions, and facilitation structure adopted during the focus group sessions. 📂 data/scripts/ Contains scripts used to support quantitative summaries and reliability analysis derived from the focus group data. > Download files DotPlot.RR script used to generate dot plot visualizations summarizing participants’ assessments of the replication guidelines. DotPlot.pdfOutput figure generated by the DotPlot.R script. KrippendorffsAlpha.RR script used to compute Krippendorff’s Alpha, supporting the assessment of inter-rater agreement in the quantitative evaluations produced during the focus group analysis. 📂 data/Focus Group-data/ Contains structured datasets derived from the focus group discussions, combining quantitative ratings and qualitative annotations. > Download files Focus Group Validation of Replication Guidelines.xlsxSpreadsheet consolidating participants’ evaluations of each guideline, including ratings, comments, and coding used in the analysis. This file represents the primary structured dataset used to synthesize the focus group results reported in the paper. 📄 Paper.pdf The preprint will be shared following approval. 📄 LICENSE / LICENSE-MIT Define the usage terms for the data and scripts. 📄 README.md This file provides an overview of the repository and a guide for usage. 💾 Storage Requirements The total size of this repository is under 20 MB. No special storage requirements are needed. 🛠 How to Run the Scripts ⚙️ Requirements To run the scripts, you need: R installed on your machine. RStudio (optional, but recommended). 🖥️ System Requirements No specific hardware requirements are needed to execute the scripts. 🛠 Steps to Run Download and install R. Download and install RStudio (optional). Download the R script from the scripts folder. Open the R script. Run the script using: Mac: Cmd + Shift + Enter Windows/Linux: Ctrl + Shift + Enter The output is the statistical result and a plot, that will be saved in the same folder as the script. 💡 The script generates the Figure 4, presented in the paper. 💡 Make sure both the data file and R script are in the same directory. 🛡️ Ethical and Legal Considerations This study was conducted strictly for academic research purposes and adheres to institutional research ethics guidelines. No personal or sensitive data was made available. 📜 License R scripts are licensed under the MIT License. Research data and artifacts are licensed under the CC BY 4.0 License. Diretrizes para enfrentar ameaças à validade em estudos de replicação: percepções de um grupo focal.