Kaggle / October 2026–January 2027
Name That LLM
An ongoing 18-class model-attribution project identifying which language model generated a passage, using text features, LoRA models, and calibrated inference.
Ongoing
- Public rank
- 427
- Balanced accuracy
- 0.599
As of 23 Oct 2026 View leaderboard ↗
Public leaderboard
The challenge
My approach
Solution notes — reviewed 23 October 2026. Methods, local experiments, and competition standings are identified separately.
Open pipeline diagram at full size
Model attribution under prompt shift
Name That LLM asks which of 18 language models generated a passage. Balanced accuracy gives equal weight to recall for each model. A classifier that performs well on common classes but never predicts a difficult sibling model can therefore be weak overall. The hidden set uses different prompts, which makes prompt-disjoint validation more relevant than a random row split.
An interpretable first model
The initial implementation combines word and character TF-IDF with 165 stylometric features. A linear ensemble mixes ridge and SGD-based classifiers. src/validation.py creates locked StratifiedGroupKFold assignments using prompt_id as the group and checks prompt separation and class coverage. The first verified experiment records mean five-fold balanced accuracy 0.2849 and public score 0.269. These baseline numbers are project results; the larger comparison scores quoted from community work in the README are not.
Public inference lineages and attribution
Later experiments adapt public two-stage LoRA pipelines with calibration and assignment logic. The experiment record names the Sangam and lightningv08 lineages, along with grouped-Gaussian, OLMo2, and ATLAS components in particular variants. The best verified public score of 0.599 comes from a public-pipeline derivative. That score must not be described as a model trained entirely from scratch in this repository.
A separately trained adapter
The owned-model route trains a Qwen3-8B LoRA adapter on an external text pool, then fine-tunes it on host data using prompt-grouped evaluation. The retained record gives stage-B mean out-of-fold balanced accuracy 0.5768, rising to 0.5798 with Sinkhorn calibration. An owned-adapter inference entry scored 0.557 publicly. These figures describe a different route from the best public derivative and should be compared with their training and calibration conditions intact.
Runtime is part of the model
The log retains unhandled errors, CPU-runtime failures, and invalid earlier cross-validation rather than presenting them as successful experiments. Time guards and fallback inference protect a code-competition submission from finishing without a usable output. Calibration and assignment assumptions also need checking against the actual hidden-data contract; they are not free improvements independent of the task distribution.
Current standing and next questions
On 23 October 2026, the public leaderboard shows rank 427, balanced accuracy 0.599, and nine entries. This is an ongoing public result. The experiment log still marks a later mixed owned/public variant as pending, so this write-up does not assign it an achieved score. The central lesson is to keep prompt leakage, external-data distribution, public-weight credit, and deployment runtime visible alongside the headline number.
Sources and provenance
The implementation and experiment records for this entry are private. This account summarizes the reviewed local code and logs without publishing private artifacts.