This article is currently an experimental machine translation and may contain errors. If anything is unclear, please refer to the original Chinese version. I am continuously working to improve the translation.
TL;DR & Demo
As of early 2026, mainstream LLM-generated text exhibits strong statistical patterns that can be effectively distinguished from human-written content using traditional machine learning models. I suspect this is how many so-called “AI plagiarism checkers” actually work under the hood.
The model used in this demo is not trained on general-purpose data, nor has it undergone rigorous optimization or iteration. Its single-sentence detection accuracy is approximately 85% on the test set. Please read through this article before use to understand potential limitations.
The core code (drafts) and trained model files are available on GitHub: lyc8503/AITextDetector
Background (aka Useless Rambling)
Back when I was still writing my thesis at school half a year ago, rumors were already spreading about checking papers for AIGC (AI-generated content). I tested several platforms—CNKI, Wanfang, and some third-party AIGC detection services—and found they could indeed distinguish between my hand-written text and LLM-generated text with decent accuracy.
That sparked my curiosity about how AIGC detection actually works(and how to bypass it).
But I was juggling too many things at the time—obsessed with radio, Minecraft, Touhou—and after a few failed attempts, I shelved the idea.
Eventually, I faked my way through the thesis, and life moved on. But recently, while browsing Lofter, I stumbled upon entire tags flooded with low-quality, wildly out-of-character AI-generated fanfics.
How can I tell they’re AI at a glance? Well, some folks(or gals) don’t even bother cleaning up Markdown formatting or AI-generated section headers before posting—and then they slap half the article behind a paywall 😓
Most AI-generated texts, however, are harder to spot—they’re buried among diverse writing styles, varied prompts, and not immediately obvious. By the time you realize something feels off, it’s too late. Some texts are borderline impossible to prove as AI, leaving me paranoid. After swallowing a few too many AI-generated turds, I’d finally had enough. Lofter browsing stops here—time to open VS Code!
Yes, that’s how I ended up reviving my weekend project idea: building an AI-generated text detector…
Research Attempt – No Luck
The internet is now almost entirely polluted with ads when searching for AIGC detection. Every result is just another essay-AI-rewriting service. Back then, I dug through the noise and found something called text perplexity.
The idea is simple: use an existing LLM to estimate the probability of each word appearing in a given sentence. If nearly every word ranks high in the LLM’s predictions (Top-N), the sentence is likely AI-generated. Conversely, if many words are unexpected, it’s more likely human-written.
Sounds promising, right? I spent some time trying this method, but results were disappointing—plenty of false positives and false negatives, and no reasonable threshold could be set. Plus, there are practical issues: high inference cost, poor cross-model generalization, difficulty deploying large models locally, and closed-weight models being hard to integrate. Overall, this approach isn’t elegant or reliable.
Failed attempt—got tricked by a bunch of soft-ad "tutorials"
A (Somewhat) Successful Attempt – scikit-learn SVM
Since online resources were useless, back to old-school alchemy.
Scikit-learn, activate! Following its Roadmap, we can directly pick Linear SVC and Naive Bayes as good starting points for our classification task.
(Whisper: this also matched my gut feeling—LLMs have detectable word-choice patterns; even a Naive Bayes classifier should pick them up. I just didn’t expect the signal to be this strong.)
Data Generation
Old-school alchemy traditional classifiers need labeled data—so we need human-written texts and confirmed LLM-generated texts for training.
My approach: I pulled data I’d scraped in 2023 from a certain Ford-like and River-like platform, filtering for articles published between 2010–2022 (pre-ChatGPT). I only filtered out extremely low-engagement or very short pieces, then randomly sampled nearly 10,000 multi-thousand-character texts as human-written samples.
Then, I used an LLM to generate chapter summaries of these texts, fed the summaries back into the LLM, and had it regenerate full articles. This gave me a roughly equal number of LLM-generated samples, diverse in genre and closely matching the original human content.
In theory, at least. But LLM APIs are expensive, and I wasn’t about to spend thousands on a weekend project. So I got creative—and skirted the rules leveraged multiple low-cost or free API channels:
- Gemini: Used CLIProxyAPI to convert Antigravity/Gemini CLI quota into API access—just pay ~$20 for an AI Pro account
- Qwen: qwen-code lets you reverse-engineer the Qwen Plus API—free
- GLM-5: Lucky timing—OpenRouter was offering free GLM-5 public beta (Pony Alpha)
- Kimi, Deepseek, Doubao, GLM-4.7: Signed up during a promotional coding plan—$8.9 first month, API access unlocked
Disclaimer: This is not a recommendation. These actions violate platform ToS and may get you banned. But the platforms are too busy with marketing hype to care, and I wasn’t about to pay full price.
Many programming-focused LLM APIs strangely charge per call, but we can abuse optimize this by batching tasks into massive inputs, forcing the LLM to generate more content per call. And so…
What do you mean I used over 300M Gemini tokens worth $2000 at full price?!
Ultimately, I used gemini-3-flash to generate summaries, and seven different models (gemini-3-pro, qwen-coder-plus, glm-5, glm-4.7, kimi-k2.5, doubao-seed-code, deepseek-v3.2) to generate seven sets of LLM-generated samples.
The generated files
Training
While I was halfway through data generation, I couldn’t wait and started training.
I asked Claude to write the classifier code, and it naively dumped the entire raw text into the model—achieving a suspicious 99.45% accuracy… Wait, really?
Claude’s useless. I’ll do it myself. For training, I split all texts into sentences using Chinese punctuation, cleaned non-Chinese/English characters, then applied scikit-learn’s TF-IDF → LinearSVC. After cleaning up some noise, sentence-level classification still hit ~85% accuracy!
Even this buggy first version hit 88% accuracy (later optimized to 85%)
Individual sentences carry limited info, but 85% accuracy per sentence means that for a longer article, we can be highly confident in judging whether it’s AI-generated. This performance far exceeded my expectations. Old-school ML still slaps—way better than those dumb online tools that just ask an LLM, “Hey, is this text AI-generated?”
After finishing all data, I tried training an 8-class model (human + 7 AIs), but the LLMs seem too similar—probably distilled from each other—so classification was messy, with only ~50% accuracy.
Multi-class results—apparently not separable. Maybe my model sucks, but whatever, not important
Eventually, I trained seven separate binary classifiers and used majority voting: a sentence is flagged as AI if ≥2 models detect it.
loaded 8536 samples |
All models achieved over 85% accuracy and over 80% F1—pretty solid! I also noticed that AI-generated texts were often flagged by multiple models, so voting made perfect sense.
I tried MultinomialNB and SGDClassifier, but accuracy dropped slightly. BERT gave a minor boost but required too much GPU time—discarded. Even tested AutoGluon, which somehow managed only 53% binary accuracy. Won’t dive into those.
JS Implementation for Web Demo
At this point, I could’ve just published the repo and called it a day. But launching Python every time is way too inconvenient. I could’ve hosted a Python API, but that means server maintenance—violates my strict Serverless philosophy.
My original plan: export model to ONNX, run inference via ONNX Web Runtime in Wasm. But when I asked my silicon servant Claude to help, I didn’t specify clearly—and it went off-script, trimming and exporting the model as a JSON… then implemented TF-IDF + SVM entirely in JavaScript for browser inference.
Hmm… actually not a bad idea. I tested it on a 1-million-character text—it took about 10 seconds on my machine, acceptable. For typical few-thousand-character inputs, it’s instant.
Fine, since this is just a demo, and the JS approach is more transparent, I’ll keep this slightly silly implementation. (Blame Claude, not me.)
As for accuracy: I tested different feature limits. Ultimately prioritized performance and kept 500k features. Stored as JSON, it’s a bloated 107MB (though gzipped server-side, it’s ~38MB). Smaller versions (50k–80k) only lost 3–4% accuracy, but final AI detection rates varied significantly—especially on human texts, with ±50% relative differences, leading to false positives. So I stuck with 500k.
Final accuracy drop: ~1%, as shown below:
1 | ============================================================ |
Testing Performance
All tests below use the pruned web version, which should perform similarly to the full joblib models.
Current logic: split input text into sentences, clean and classify using all 7 binary models. If ≥2 models flag a sentence, it’s marked as suspected AI and highlighted. Final AI score is the proportion of flagged characters. Classification:
- <50%: Human
- 50–70%: Maybe Human
70%: Maybe AI
First, test detection rate on common models like Doubao and Deepseek—both were in training data. Prompt: write me a 3000-word story. Easily caught:
Deepseek V3.2: 78.4%
Doubao Seed Code: 93.0%
Now test on unseen models—how’s generalization?
Claude Sonnet 4.6: 71.9%
GPT 5.2: 73.3%
I tested several other models not in training (MiMo-V2, Doubao-Seed-2.0, GPT-4o)—all detected at ~70%, some even >90%. Solid.
Also tested more complex prompts—e.g., feeding 20 chapters of human-written text and asking the LLM to mimic style and continue. Detection rate dipped slightly to 67.8% (but remember, we trained on complex prompts too). Results not shown due to space.
Then I picked 10 completed web novels (pre-2022) from my subscription list—diverse genres, authors, eras, and likely not in training data.
Their AI detection rates: 22.7%, 24.2%, 25.0%, 24.5%, 19.0%, 13.7%, 29.1%, 4.9%, 27.3%, 19.2%—all under 30%. I also sampled random Lofter fanfics; since they’re more casual, detection rates were often below 10%. But when I fed in texts I suspected were AI-generated, detection spiked to 83.4%, strongly suggesting LLM use without disclosure.
[Mar 5, 2026 Update] For more rigorous testing, I randomly sampled 10,000 high-engagement (views >5000), long-form (word count >2000) fanfics from Lofter, all posted before 2022. Their AI detection rate distribution (using 7-model voting, ≥2 votes):
1 | 0-5%:313 | 5-10%:1945 | 10-15%:3016 | 15-20%:2033 | 20-25%:1355 | 25-30%:594 | 30-35%:492 | 35-40%:123 | 40-45%:34 | 45-50%:62 | 50-55%:24 | 55-60%:5 | 60-65%:3 | 65-70%:1 |
Using 60% as threshold → false positive rate: 0.04%
Using 70% → false positive rate: <0.01% (effectively zero)
The four texts above 60% were all collection indexes, not actual stories—flagged due to excessive links.
Even at 50% threshold, false positive rate is only 0.33%.
Then, I scraped all articles from Lofter Android’s top 20 trending tags (weekly榜单), filtered by length, and ran detection:
1 | 0-5%:27 | 5-10%:138 | 10-15%:231 | 15-20%:245 | 20-25%:238 | 25-30%:137 | 30-35%:112 | 35-40%:87 | 40-45%:116 | 45-50%:112 | 50-55%:118 | 55-60%:157 | 60-65%:118 | 65-70%:109 | 70-75%:75 | 75-80%:56 | 80-85%:28 | 85-90%:15 | 90-95%:10 |
32.22% of articles scored >50% AI—likely partially or fully AI-generated… Is there any human left?? Moreover, not a single one has proactively declared AI-generated content.
“Age of Dharma’s decline, , ,” —a friend in the group
Attack and Defense – Bypassing Detection(?)
Alright, we’ve built an AIGC detector. Time to build an anti-detector now.
Nope, kidding. I’m not that bored.
But let’s test some common anti-AIGC detection tricks:
Classic Translation Method
Google Translate roundtrip (CN→EN→CN): 89.9% → 85.0%
Youdao Translate (CN→EN→CN): 89.9% → 79.2%
Sogou Translate (CN→EN→CN): 89.9% → 86.0%
Slight drop, but still clearly flagged.
LLM Prompt Method
Use “magic” prompts to make LLM output less “AI-like”—sounds ridiculous from the start!
Tested one-line prompt: Rewrite the above article to minimize AI flavor: 89.9% → 83.0%
Also tried more complex prompts: 89.9% → 79.3%
Slight improvement, but still meaningless in practice.
This detection method is way too robust!(flag
If I really wanted to bypass it, my only ideas would be fine-tuning an LLM on massive human text, or building a huge rule-based system to surgically disrupt SVM-matched features. But that’s beyond this article. Not sure if it’d even work. Or maybe there are better ways—left as an exercise for the reader.
Epilogue
Now, time for closing rambling.
I honestly didn’t expect this classification task to be so easy—simple enough that a scikit-learn “Hello World” script, with minimal iteration and some hardcoded rules, could produce a fairly robust and accurate detector. Most of the effort was just waiting for LLMs to generate data…
Ambitious readers could follow this approach to train detectors for other domains—say, academic paper AIGC detection. Add a flashy frontend, and you’ve got a tool you can sell to desperate college students. If you make money, don’t forget to donate a little.
Another idea: detect AI-generated images. But with Stable Diffusion and easy LoRA fine-tuning, visual styles are far more diverse than text—this task would be much harder. And after writing this, my three-minute enthusiasm is burned out. Maybe next time.
Lastly, a few words on AI-generated content: I don’t accept AI-generated entertainment as legitimate creative work. Just like AI coding tools produce bloated, unmaintainable code, AI-generated text, images, and audio may seem decent at first glance, but fall apart on closer inspection—repetitive, shallow, and so predictable that even word frequency stats can catch them. This pattern is fundamentally unsuitable for real creation, and as a reader, I’m deeply unsatisfied. I’m starting to suspect LLMs’ so-called “creative writing” is just a bunch of post-training data being endlessly recombined and regurgitated.
But then again, “the world should be” has never equaled “the world is.” While LLMs bring innovation and productivity gains, misuse and abuse are spreading relentlessly across every industry. And since LLMs are fine-tuned to exploit human perception, who knows whether they “understand” anything or just memorize patterns? After patching endless bugs like “which is bigger, 3.9 or 3.11” or “should I walk or drive 50m to a car wash,” can we really say the model understands the world?
Everyone’s stuck in debates: What is LLM? How will it disrupt my industry? Where will AI take humanity? No one has answers.
At least I’m glad I learned to code before the AI era. Otherwise, I might not realize how stupid today’s vibe-coded software really is. As for the future? Either generative AI smashes human social order to pieces, or the AI bubble bursts and memory becomes free. Either way, sounds fine to me, doesn’t it?
This article is licensed under the CC BY-NC-SA 4.0 license.
Author: lyc8503, Article link: https://blog.lyc8503.net/en/post/llm-classifier/
If this article was helpful or interesting to you, consider buy me a coffee¬_¬
Feel free to comment in English below o/