SIGCSE TS 2027 SUBMISSION: 152 TITLE: REDACTED: Academic Integrity in the Era of Generative AI ------------------------- METAREVIEW ------------------------ Summary This paper presents an automated tool for identifying programming assignments that may warrant further review for LLM-assisted code generation. The proposed approach combines static code heuristics and behavioral indicators into an H-score that helps instructors prioritize manual review while emphasizing that automated scores are not evidence of academic misconduct. The paper reports a longitudinal evaluation across six offerings of a large introductory C programming course and analyzes the relationship between H-scores and student performance on proctored examinations. Meta-review Reviewers commented positively on many aspects of this work. They highlighted the paper's timely motivation, the careful design of the proposed tool, the substantial longitudinal evaluation across six course offerings, and the strong empirical evidence supporting the approach's usefulness. The reviewers also appreciated the emphasis on human review over fully automated decision-making and noted the potential for adapting the methodology to other programming courses. The reviewers identified only minor issues that, if addressed, would further strengthen the paper. These include clarifying aspects of the dynamic normalization routine, providing additional contextual information about deployment (such as class size and instructional staffing), improving a few figures, and expanding the discussion of related work and practical considerations. One additional observation is that, as AI-assisted programming becomes increasingly prevalent, the long-term behavior of the normalization strategy may warrant further discussion in the paper. Although this was not considered to affect the overall assessment of the paper, briefly acknowledging this potential limitation or a future research direction would strengthen it. If the paper is accepted, we strongly encourage the authors to address these suggestions in the final version. In particular, clarifying the dynamic normalization routine, discussing its longer-term applicability, adding deployment context, incorporating the suggested related work, and addressing the minor presentation improvements identified by the reviewers would further strengthen the paper. If the paper is not accepted, we encourage the authors to address these same areas of concern and consider resubmitting to a future SIGCSE TS. ----------------------- REVIEW 1 --------------------- SUBMISSION: 152 TITLE: REDACTED: Academic Integrity in the Era of Generative AI ----------- Summary ----------- The paper discusses the use of a tool to detect potential AI use when writing C code. The tool is an automated step, but then the code requires human review. The authors compared scores from the tool with performance on exams. ----------- Hallucinated References ----------- SELECTION: yes ----------- Motivation (ERT) ----------- SCORE: 5 (excellent) ----- TEXT: The motivation for this paper is to use a tool to flag potential LLM-assisted code generation for review by teaching staff. ----------- Prior and Related Work (ERT) ----------- SCORE: 4 (good) ----- TEXT: Section 2 reviews related work pre-genAI and post-genAI. Paper discusses link to this work. ----------- Approach (ERT) ----------- SCORE: 5 (excellent) ----- TEXT: Paper discusses the motivation for the use of the tool. It also describes how the tool analyzes each commit for concepts not covered in the course and style patterns not used in the course. This analysis leads to a score that can be used to identify submissions that should be manually reviewed. The approach compares tool scores with exam performance, where exams are closed-note and proctored. ----------- Evidence (ERT) ----------- SCORE: 5 (excellent) ----- TEXT: Paper describes sample size for the six course offerings (six spring semesters). Data is presented in multiple ways, including graphs. A change in data can be seen as LLM-based assistance has become more widely used. ----------- Contribution & Impact (ERT) ----------- SCORE: 4 (good) ----- TEXT: Data is interesting to see as it mirrors what I have seen in class. If the tool is shared with others, it could impact the process of AI detection in other courses at other universities as well. ----------- Presentation (ERT) ----------- SCORE: 5 (excellent) ----- TEXT: Paper is well-written. ----------- Suggestions for improvement ----------- For the side-by-side graphs, it could be helpful to include label the vertical axis on each. ----------- Overall evaluation ----------- This paper is well-written. It clearly describes the motivation for tool creation along with the data from using the tool with six offerings of a C course. ----------------------- REVIEW 2 --------------------- SUBMISSION: 152 TITLE: REDACTED: Academic Integrity in the Era of Generative AI ----------- Summary ----------- The authors wrote a utility tool for teaching introductory classes in the C programming language, with respect to deciphering which students are most likely to have used LLM tools and thus warrant the time for a human to check their submissions more closely. They explain the inadequacy of other methods, both for similarity checking and for LLM detection. This will be directly useful for anyone teaching a course in C programming in the near future, and I believe the techniques shown in the development of this tool apply well to other languages. ----------- Hallucinated References ----------- SELECTION: yes ----------- Motivation (ERT) ----------- SCORE: 5 (excellent) ----- TEXT: The inadequacies in the previous tools are well explained in the previous work section, and the developed tool solves these problems well. ----------- Prior and Related Work (ERT) ----------- SCORE: 4 (good) ----- TEXT: The section is a good overview of recent related work with a focus on introductory programming courses and detecting LLM-generated code. This situates the work well. Given that significant space was (in my view, needed) for design and results, it is acceptable that no further branching (such as other plagiarism and software similarity) was covered here. One paper that you might want to consider adding to help making the point about the evolving nature of students using LLMs in their code is: Irene Hou, Hannah Vy Nguyen, Owen Man, and Stephen MacNeil. 2025. The Evolving Usage of GenAI by Computing Students. In Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 2 (SIGCSETS 2025). Association for Computing Machinery, New York, NY, USA, 1481–1482. https://doi.org/10.1145/3641555.3705266 ----------- Approach (ERT) ----------- SCORE: 5 (excellent) ----- TEXT: The tool's design is both appropriate and well described. The context of its deployment is clear as well. ----------- Evidence (ERT) ----------- SCORE: 5 (excellent) ----- TEXT: There is ample evidence as the tool was developed and the 2026 case study, complemented by the structured manual review process, drives home the point, and the retrospect data is valuable as well. ----------- Contribution & Impact (ERT) ----------- SCORE: 5 (excellent) ----- TEXT: A novel software tool that allows good pedagogical practices to more easily scale to large classes (the authors seem to have hundreds per semester) are, in my view, self-evidently making an important contribution. However, the authors make their case in the paper very well, including how well it can be adapted and they provide important best practices reminders (such as human review). The contrast in section 5.1 is a key point; if I were teaching a course in C, I would want this tool available, and if I were teaching a similar course in another language, I would want a version for that. ----------- Presentation (ERT) ----------- SCORE: 5 (excellent) ----- TEXT: I don't have anything to say here beyond that this is excellently presented. I do have some suggestions for improvement (see below). ----------- Suggestions for improvement ----------- How large are the classes and how many support staff (TAs, etc) do you have? This information did not seem to be in the paper, although it is implied to be in the several hundreds, and that information would help to evaluate the scaling factor. The MOSS software also has a similar best practice to what is described here, where they emphasize (as you do) that similarity score is not evidence of misconduct but rather is a way to determine where human inspection is most needed. It might be worth referencing this to formally establish that these views are long held in the community. ----------- Overall evaluation ----------- This appears to be a fantastic tool. I believe there are two reasons that this will be a large benefit to the community. First, (usefulness directly for C-based classes, as is common in architecture). Second (inspiring adoption of similar tools for other classes, as the methodology is clear and apparent and an improvement over basic similarity scores) ----------------------- REVIEW 3 --------------------- SUBMISSION: 152 TITLE: REDACTED: Academic Integrity in the Era of Generative AI ----------- Summary ----------- This paper is about a new way to score assignments in an introductory CS course for AI assisted code, creating what is called an H-index. It provides data that correlates the index with student outcomes on exams, which supports existing literature on the extent and deleterious impact of this phenomenon. It relates to both large and small universities, all CS or adjacent programs. ----------- Hallucinated References ----------- SELECTION: yes ----------- Motivation (ERT) ----------- SCORE: 5 (excellent) ----- TEXT: The paper is well motivated and the objectives are clear. While the conclusions are not unexpected, the data are compelling. By using a fixed measure of performance based on exams, the figures and results are quite fantastic. ----------- Prior and Related Work (ERT) ----------- SCORE: 5 (excellent) ----- TEXT: The authors do a great job of reviewing the current literature and couching their work in that context. They use and expand upon existing ideas to create their new H-index measure of AI assisted coding. ----------- Approach (ERT) ----------- SCORE: 5 (excellent) ----- TEXT: The methods are clearly explained overall. The one part that could use some additional structure, examples, or other writing techniques to help the reader better understand their approach is the paragraph that explains the "dynamic normalization routine." I found myself having to reread that a few times to understand it, but with a little effort, it can be made clearer. ----------- Evidence (ERT) ----------- SCORE: 5 (excellent) ----- TEXT: Excellent work on the data analysis and visualization! Fig 2 and 3 are amazing. The sheer amount of time this work spans is what really makes the story great: pre-AI, post-covid, and post-AI. It's quite dramatic and evident. ----------- Contribution & Impact (ERT) ----------- SCORE: 5 (excellent) ----- TEXT: Many folks at multiple institutions are studying this issue. Many anecdotally have seen the widespread misuse of AI, and its effect on outcomes. This paper just really makes this point clear: 45.7 percent! Sadly, it's not a surprise. But it's great to see a concrete measure of this. ----------- Presentation (ERT) ----------- SCORE: 5 (excellent) ----- TEXT: Wry good writing. I found no real issues. One minor comment is that I found Fig 1 to be unhelpful to the narrative. The paper stands on its own well without it. Also, Fig 5 is fine, but only marginally useful. ----------- Suggestions for improvement ----------- One minor comment is that I found Fig 1 to be unhelpful to the narrative. The paper stands on its own well without it. Also, Fig 5 is fine, but only marginally useful. ----------- Overall evaluation ----------- As detailed above, this paper provides a compelling new measure of AI misuse and demonstrates clearly the full extend of the problem. Fantastic figures. Only a few minor issues that are easily corrected.