Human Attention During Localization of Memory Bugs in C Programs

This zenodo repository accompanies the paper "Human Attention During Localization of Memory Bugs in C Programs." The .zip files contained are from our github repository here: https://github.com/apcl-research/human_attention_localize_c_bugs There are 3 important branches of the repository, so we have zipped each one. This is the main branch: human_attention_localize_c_bugs-1.0.0.zip This is the instructions branch: human_attention_localize_c_bugs-1.0.0-instructions.zip This is the analyze branch: human_attention_localize_c_bugs-1.0.0-analyze.zip Below is a copy of our README. # Replication Repository and Online Appendix for "Human Attention During Localization of Memory Bugs in C Programs" # 1 README Contents: overview of sections of this file <!-- TOC --> - [Replication Repository and Online Appendix for "Human Attention During Localization of Memory Bugs in C Programs"](#replication-repository-and-online-appendix-for-human-attention-during-localization-of-memory-bugs-in-c-programs) - [1 README Contents: overview of sections of this file](#1-readme-contents-overview-of-sections-of-this-file) - [2 Repository Organization: description of contents of each branch](#2-repository-organization-description-of-contents-of-each-branch) - [3 Replication Procedure for Study](#3-replication-procedure-for-study) - [3.1 Software Setup](#31-software-setup) - [3.2 Setup for Session](#32-setup-for-session) - [3.3 Session with Participant](#33-session-with-participant) - [3.4 After Participant Leaves](#34-after-participant-leaves) - [4 Replication Procedure for Data Analysis](#4-replication-procedure-for-data-analysis) - [4.1 Data Processing Steps](#41-data-processing-steps) - [4.2 Analysis Steps](#42-analysis-steps) - [4.3 Table/Figures Recap](#43-tablefigures-recap) - [5 More Details on `analyze` branch and Scripts](#5-more-details-on-analyze-branch-and-scripts) - [5.1 `analyze` Branch Contents](#51-analyze-branch-contents) - [5.2 Script Details](#52-script-details) - [5.3 FAQ](#53-faq) - [6 More Details on `instructions` branch](#6-more-details-on-instructions-branch) - [6.1 `StudyInstructions`](#61-studyinstructions) - [6.2 Scripts](#62-scripts) <!-- /TOC --> # 2 Repository Organization: description of contents of each branch <details> <summary><strong>Expand Section</strong></summary> This repository is organized into branches: - `main`: contains the raw itrace_core and itrace_eclipse .xml files - `instructions`: a template branch that contains the template consent form, pre-study questionnaire, post-study questionnaire, and bug reports. The branch was copied from to create the participant branches. - `analyze`: contains the data analysis scripts and intermediate versions of data </details> # 3 Replication Procedure for Study ## 3.1 Software Setup <details> <summary><strong>Expand Section</strong></summary> 1. We use the Tobii Pro Fusion Eye Tracker 120 Hz: https://www.tobii.com/products/eye-trackers/screen-based/tobii-pro-fusion and Tobii Eye Tracker Manager 2.6.1: https://connect.tobii.com/s/etm-downloads?language=en_US 2. Download Eclipse IDE for C/C++ Developers (2024-09) Version 4.33.0: https://www.eclipse.org/downloads/packages/release/2024-09/r/eclipse-ide-cc-developers 3. Download iTrace tools: https://www.i-trace.org/main/pages/downloads.html - Core - v0.2.0 - Plugin - v0.2.0 - Toolkit - v0.2.2 - ScreenRecording - v0.2.0 4. Add the iTrace plugin to Eclipse following the instructions on the iTrace wiki: https://github.com/iTrace-Dev/iTrace-Core/wiki/Getting-Started-with-iTrace-Core 5. Install the Remain AI Chat Plugin for Eclipse: https://marketplace.eclipse.org/content/remain-ai-chat-chatgpt#details 6. We use OBS Studio with the iTrace-ScreenRecording plugin. Install OBS: https://obsproject.com/ 7. Install and setup the iTrace-ScreenRecording: https://github.com/iTrace-Dev/iTrace-ScreenRecording#readme 8. Clone this repository 9. Switch to the `instructions` branch 10. Run the `get_repos.py` script to download the buggy versions of source code for each bug </details> ## 3.2 Setup for Session <details> <summary><strong>Expand Section</strong></summary> 1. Attach Tobii Eye Tracker to computer 2. Create Display Setup for Tobii Eye Tracker in Tobii Eye Tracker Manager 3. Make a copy of the `instructions` branch 4. Modify the `StudyInstructions/StudyProcedure.md` file if you want to change which bugs the participant will solve during the session. The default is `ladybug`, `stonefly`, `hornet`, `silverfish`, `praying_mantis`, and `spider` 5. Open Eclipse, and use File > Open Projects from File System to open the Study Instructions folder and the folder containing each bug's source code. 6. Rename the folders in Eclipse containing each bug's source code to the bug's code name (e.g. `ladybug`) 7. Make sure you see the iTrace Eclipse Plugin in Eclipse. Window > Show View > Other > search for iTrace > select iTrace plugin, and it should open in Eclipse 8. Open iTrace core; minimize command prompt that opens 9. In Eclipse, in the iTrace plugin window, click "Connect to Core" 10. Create folders to save iTrace data. It is important to create a new folder for each task because the mouse click data from iTrace saves to the same file name every time. 11. Open OBS and ensure iTrace-Recorder.py is loaded in the scripts. </details> ## 3.3 Session with Participant <details> <summary><strong>Expand Section</strong></summary> 1. Thank participant for participating in our study. 2. Introductions 3. Sit participant down at computer 4. Have participant read through the consent form and sign name 5. Conduct pre-study questionnaire 6. Walk participant through calibrating in Tobii Eye Tracker Manager 7. Walk participant through calibrating, starting, stopping in iTrace-core 8. Participant ready to read `StudyInstructions.md` now 9. Answer any questions that the participant has. 10. Start script to do pop ups called `prompt.py` with ``` python .\prompt.py --interval 5.0 --csv {participant_name}.csv ``` 11. Participant opens `StudyProcedure.md` and starts 12. Repeat steps 13-18 until time is up 13. In OBS > Scripts, click "Connect to Core" 14. In the Session Setup tab of iTrace-core, fill in Task Name, Researcher Name, and Participant ID in the iTrace-core window. Choose a Data Directory.In the iTrace Tracking tab, select Tobii Pro Fusion as the tracker, check the boxes for "Record with DejaVu" and "Enable Screen Recording", calibrate, and then start iTrace-core. 15. Start 30 minute timer for bug 16. Participant opens bug report and works on bug 17. After 30 minutes, ask participant to move on to next bug if not done. 18. When participant is done with this bug, stop iTrace-core. In the Data Directory, you should see 4 files: `itrace_core-*`, `itrace_eclipse-*`, `out.csv`, and `screen_rec-*` 19. When participant finishes bug that is closest to the 2 hour mark, tell them that they are done. 20. Stop `prompt.py` script which has been running the whole time 21. Conduct post-study questionnaire 22. Pay participant </details> ## 3.4 After Participant Leaves <details> <summary><strong>Expand Section</strong></summary> 1. Label and backup files created by iTrace (Data Directories) 2. Save Remain AI Chat window history using save button in Eclipse 3. Upload .xml files, AI Chat window history, and popup .csv to GitHub 4. Upload screen recordings to Google Drive (too big for GitHub) </details> # 4 Replication Procedure for Data Analysis Switch to the `analyze` branch ## 4.1 Data Processing Steps <details> <summary><strong>Expand Section</strong></summary> ### 4.1.1 Eye Tracking Data > [!NOTE] > - You may skip all of these steps if you would like to use our generated .pkl files located at this Google Drive Link: https://drive.google.com/drive/folders/1eTnnvSOuRE0g94TNWKcHzAgez_4G1xa9?usp=drive_link > - You may skip to step 6 if you would like to use our generated databases located at this Google Drive Links: https://drive.google.com/file/d/1W5Sg9k8IGHyC_9tHMhCjEZTh0IBeaD6K/view?usp=sharing > - Skip to step 3 if using study's original data which is included in this repository, `analyze` branch, `only_eclipse_data` folder. Note, you many need to unzip some large files. 1. Reorganize data to make sure there are no nested folders (iTrace-Toolkit does not handle nested folder structures well). You will see that some of the folder organization of `only_eclipse_data` in the `analyze` branch has been modified from `main`'s version to accommodate this. In situations where there are nested folders in `only_eclipse_data` in the `analyze` branch, these folders are used to create separate databases. 2. Convert all relevant references to .md files to references to .c files in the iTrace/eclipse .xml files. Use `convert_md_to_c_in_eclipse_xml.py` #### Note: This is the state that the files in `only_eclipse_data` in the `analyze` branch are in #### Note: You may skip steps 3-4 if you would like to use our generated srcML files located at this Google Drive Link: https://drive.google.com/drive/folders/1Y4HQGYNrKKFCap6LzHGmnmCDMB3xlQEa?usp=drive_link 3. Convert all .md files in the directory containing the code for the bug and in the study instructions directory to .c files. we do this because srcml cannot create .xmls for .md files. To do this, use the `rename_md_to_c.py` script 4. Generate the srcml for the directory containing the code for the bug and the instructions directory. (must install srcml on machine first) Command: ``` srcml --verbose --archive --position path\to\bug\directory path\to\instructions\directory -o {bug_name}_{participant_id}.xml ``` 5. For each bug subfolder for each participant, use iTrace-Toolkit to generate database (.db3 files; you should have at least one per task). Note: We excluded folders that contained data for just a few seconds of trackin