🏆 A group project. Our team took first prize in the Neural Network and Deep Learning course competition.

Project Overview

A binary classifier that decides whether a short piece of text was written by a person or generated by a language model. The working domain was food and restaurant reviews: short pieces, averaging around 68 words, with the occasional non-English or code-switched fragment mixed in.

Two things made it interesting. The first was compute: everything had to train on a single free-tier GPU, which ruled out throwing the biggest available model at the problem. The second was generalisation: a detector that scores well on its own validation split is easy, one that survives text from unseen generators and adversarial tricks is not.

Choosing the Model

We surveyed three candidate approaches before committing to one.

Approach	Idea	Why we passed or picked it
Binoculars (Hans et al., 2024)	Zero-shot: score the ratio of a passage’s perplexity under an observer LLM to its cross-entropy under a performer LLM	Near-supervised AUROC with no labels, but needs two 7B models in memory at once. Too heavy for our hardware.
RoBERTa (Solaiman et al., 2019)	The default supervised recipe: classification head on `[CLS]`, fine-tune end to end	Efficient and well understood, but English-only pre-training and GPT-2-era detector weights
XLM-RoBERTa (Conneau et al., 2019)	Same recipe, multilingual backbone pre-trained across 100 languages	Picked. Matches RoBERTa on English, beats it on multilingual, at no extra compute cost

The deciding factor was the data. The Yelp corpus carries a non-trivial amount of non-English and code-switched text that RoBERTa’s English-only tokeniser handles poorly, so the multilingual backbone was effectively free accuracy.

The Data

Three corpora, each pulling in a different direction:

AIGenFoodData (Gambetti & Han, 2024): the target domain. Review-style text, roughly balanced human/AI, and the closest match to the deployment setting. It defines the validation and test distribution.
Yelp Open Dataset (2024): reviews and tips, all treated as human-written. Supplies genuine diversity of style, sentiment and subject matter.
RAID (Dugan et al., 2024): the robustness supplement. Machine and human text across many domains and model families, including adversarial samples with paraphrasing, homoglyph substitution and zero-width character insertion. About 97% AI-generated, used in Stage 1 only.

Each was sampled differently: reservoir sampling for Yelp, label-stratified splits for AIGenFoodData (kept in full), and stratified sampling over domain, generator and attack type for RAID so no subgroup could dominate.

The merged corpus came to 5,985,323 rows, heavily skewed at 4.3% human to 95.7% AI. Text was normalised to a common schema, stripped of control characters and stray markup, then tokenised to a fixed 256 tokens. Punctuation and casing were deliberately preserved: the descriptive statistics showed systematic differences there between human and AI text that the model could use.

Two-Stage Training

The core idea: learn breadth first, then specialise.

Stage 1, broad generalisation. One epoch of XLM-RoBERTa-base over the full ~6M-row corpus, linear warm-up then linear decay. Validation AUROC climbed monotonically to 0.9946, and gradient norms held steady around 10 to 15 with no divergence.

Stage 2, domain adaptation. Continue from the Stage 1 checkpoint on AIGenFoodData alone (20k rows) at a much lower learning rate (5e-6, linear decay). Validation AUROC hit 1.0 by step 141 and stayed there. Checkpoint 141 became the submitted model.

Results

Configuration	Kaggle AUROC	RAID AUROC
Baseline: RoBERTa-base, HC3 only (Guo et al., 2023)	0.7823	0.5812
Stage 1: XLM-RoBERTa, full dataset	0.8870	0.9976
Stage 2: + domain fine-tune (ckpt-141)	0.9511	0.9977
Ensemble (Stage 1 + Stage 2)	0.9398	0.9976

The two stages did exactly the different jobs we hoped for. Stage 1 delivered the robustness win: +0.4164 RAID AUROC over the baseline, which at 0.5812 was barely above chance on adversarially diverse text. But it moved Kaggle only +0.1047, so scale alone did not close the short-form domain gap. Stage 2 closed it, adding +0.0641 on Kaggle (0.8870 to 0.9511) while leaving RAID untouched at +0.0001. Targeted fine-tuning recalibrated the head toward short review-style text without eroding anything Stage 1 had learned.

Worth noting that the ensemble scored below Stage 2 alone on Kaggle, so we submitted the single checkpoint.

Where It Still Fails

Six samples out of a 1,000-row RAID evaluation were misclassified, and the pattern in them is consistent. The false positives were homoglyph and zero-width-space attacks: adversarial Unicode produces irregular SentencePiece subword splits that superficially mimic the low-diversity, high-frequency token distributions typical of AI text. The one notable false negative combined a code snippet with homoglyph substitution, where the structural regularity of short code already compresses the human/AI margin before the attack obscures what is left of the signal.

Also Explored

TF-IDF + XGBoost. A classical baseline (100k vocabulary, uni- and bi-grams) that scored 0.9998 AUROC on our own validation split. That near-perfect number was the finding, not the result: it proved the split was too easy, and held-out text from other generators scored far worse. It is a large part of why the final approach was built around cross-domain evaluation.

DetectGPT. Zero-shot scoring based on how log-probability behaves under small perturbations, tried with a LoRA-adapted scorer. A useful reference point, but too slow per example to be practical here.

My Role

The project was a team effort and the results above belong to the group. My own contribution was data selection, model training and evaluation: profiling the test set, choosing and sampling the corpora that matched it, running the two-stage fine-tuning, and benchmarking the checkpoints on Kaggle and RAID.

Tech Stack

PyTorch and Hugging Face Transformers / Datasets for fine-tuning
scikit-learn and XGBoost for metrics and the classical baseline
Pandas, NumPy, Matplotlib for data analysis and training curves