ROFL. I just ran a "detect AI generated code" tool on iocaine, and it's hilarious.

Scores:
  Claude     ████████████████████ 67.5%
  Gemini     ███████ 25.4%
  GPT        ██ 7.0%
  Copilot     0.0%
  Human       0.0%

The signals it uses for scoring is hilarious. Follow along in the next few toots for some mind bogglingly bad takes!

I've got 1.5 score for documentation comments. Apparently, thoroughly documenting my code is an AI smell.

Like, dude... LOOK AT MY LITERATE PROGRAMMED REPOS! I write more documentation than code, if I'm left to my own devices.

I received 0.5 points for zero trailing space and machine-perfect formatting.

I mean, this is Rust code. I run cargo fmt as a pre-commit hook. It is going to have no trailing space and will be machine-perfect formatting! Because I ran cargo fmt on it.

I can do that without an AI, surprising as it may be.

I was awarded 0.8 score for having no .unwrap() calls. Because I have unwrap_used = "deny" in my clippy lints, and I run cargo clippy religiously.

Pretty standard best practice is not an AI smell, my dude.

I got 0.4 score for all lines being under 100 chars. Thanks cargo fmt.

I got 1.0 for compact functions (14 lines average), because I prefer small, composable functions. I'm coming from a functional programming background (kinda), small composable functions are awesome.

Clippy also complains about long functions, so if it yells at me, I oblige, and split shit up.

Because I prefer understandable code. Unlike an LLM.

1.5 score for all public functions being documented is apparently an AI signal. Because a human can't obsessively document their APIs.

I do not have enough facepalms.

0.8 score for diverse, descriptive names - mostly because clippy yells at me if my identifiers are too similar. Thanks clippy, we're AI now.

0.5 score for low nesting depth (average 1.5), a direct consequence of having small functions - I don't need to nest deep!

Big Nesting is evil. Not liking Big Nesting makes me an AI, allegedly.

Also 0.8 points because I consistently use Self instead of naming the struct/enum. We're an AI, clippy and I.

In summary, this tool attributed pretty much all code quality improving properties to AI, and works under the assumption that human-written code is imperfect, and does not use any kind of automated, non-AI tooling for formatting and quality control and whatnot, and that documentation is something humans do not write.

Short, simple functions? AI!
Thorough documentation? AI!
Low nesting depth? AI!
Consistent formatting? AI!

Meanwhile, most AI code I saw were the exact opposite.

Now, the scores posted at the start of this thread are scores for one file only. This thing can't score the entire directory, only individual files.

But similar distribution applies to most other files.

Human signals seem to be:

• Short variable names (<= 3 chars), because why the fuck would I write for i in 0..100 differently, for a very short for loop in a very short function? Do you want me to write for index_of_some_bullshit in 0..100? Why?
• allow/cfg pragma directives. Ok, fine, I guess that's a human thing  to do.
• Ticket/issue references, because we never saw an AI hallucinate a ticket/issue id, so any such references is a human signal. Surely.

Also, a few more incredible opinions in this tool:

• Consistent use of ? is an AI smell, because why the fuck wouldn't human authors prefer consistent error propagation.
• Preferring exhausting matching in match expressions is an AI smell, because following best practices is now a thing only AI do.
• Long method chaining is also an AI smell, because composability and the builder pattern are inherently inhuman or something.
• Relying on type inference is also an AI smell, because a human will always write annotations, because their language server can't tell them the underlying type or something.

I'm going to stop here, because this is getting more sad than hilarious. That there are people who believe these kind of metrics, and this kind of scoring is correct-ish.

@algernon oh, which tool? I want to know what bullshit it comes up with for me

@mstar https://github.com/o-k-a-y/vibecheck

@algernon Which tool?

@kaleissin https://github.com/o-k-a-y/vibecheck

@algernon This whole thread is ... well, it would be funny if it didn't make me sad.

@liw Yeah. It started out being funny, but halfway through I became really sad.

Luckily, it is afternoon coffee time, so I'm gonna go and prepare my favourite coffee + cocoa + caramel + strawberry flavour + hint of vanilla + whatever-else-I-can-throw-in-before-my-wife stops me combo.

Drink that with a grin, while my wife looks at me in horror.

Always cheers me up.

@algernon A while ago I plopped some Lua code of mine into a slop-detecting slop machine and apparently I'm also not human.

@datarama Welcome to the Borg!

@algernon what did you use? i'd love to run this for my own code, all of which i've written by hand

@kate https://github.com/o-k-a-y/vibecheck

@algernon
AI: Actually, I just ran clippy

Okay, okay, just one more toot... I managed to coerce the tool into giving me an overall "which tool is responsible for this directory, at what percentage certainty?", and the iocaine-powder crate is allegedly 49% Claude.

(Reality: it's 80% human, and 10% emacs, because I use snippets for the copyright headers, and 10% clippy & cargo fmt, because we're best buddies, and they yell at me when my code is terrible.)

@algernon After my experience with being identified as a robot, I tried giving the same pile of Lua to a Gemma model on duck.ai and asked it to identify which parts were written by machine and which were written by filthy human paws.

It described the nicest parts where I'd used idiomatic Lua and made good use of abstraction as an "AI masterpiece" (!), and some generated snippets as definitely human, because they were "repetitive" in a way AI wouldn't do.

It is grim.

@datarama All of these slop detection tools are heavily AI biased, as these examples show, too.

You write idiomatic, well designed, thoroughly documented code? AI.

To be considered human, you have to make a mess. Because AI never makes a mess. Surely.

Nah. All of these tools are complete and utter bullshit. They do work as designed, perfectly in line with the rest of AI culture: to try and make us feel bad, and small, and unnecessary.

They're wrong.

@algernon

the conspicuous absence of TODO: fix this later

this is fucking satire

@algernon it also lines up with the somewhat cultish apologists / enthusiasts who are like "but aren't we all bots, really?"

My dude, I was cursed with the illusion of free will and I'm damn well going to lean into it, and that includes doing whatever my best is at a given moment. Doing the best you can is not a task we reserve for machines.

(also also, reminded of the precursors to this tooling that said, based on my blog, that I am clearly a woman. A pox on all conformative assumptions)

@algernon @liw "Yeah. It started out being funny, but halfway through I became really sad." ... and then you stopped, because you have "taste" (in a broad sense). A concept totally alien to AI and AI bros.

Taste, except when it comes to the atrocities you do to coffee.

;-P

@goedelchen @liw This ain't even the worst crime I committed against coffee.

Way back in my teens, as an act of nonconformance, to make Adults recoil in horror, I mixed cola with milk, and pretended to like it¹. This developed into a habit, where whenever I'm making a drink for myself, of any kind, I will always mix it with something else. The more the merrier.

Sometimes it even tastes good.

I also practiced this habit way back when in the early 2000s when we, a gathering of linux nerds went out for dinner every friday. Since I'm a picky eater, I always ordered pancakes. Everything pancakes. "Whatever filling you have on the menu for pancakes, throw all of them into one." is how you end up with a monster pancake filled with a dozen different things, with whipped cream and a cherry on top. That was surprisingly delicious, and the restaurant even put it on their menu for a while.

To this day, whenever my wife makes pancakes, our daughter makes an "everything pancake" for me. She loves doing it, because filling it with so much stuff, without it all falling apart or leaking out is an Art, and she's rightfully very proud of her work. (She also has good taste, much better than I do, so she only puts ingredients in that go well together.)

1. It was horrible. ↩︎

@algernon that “so-called evaluation” tool and the makers+suppliers of that “so-called evaluation” tool will make us all “illiterate-expert” button pushers, if it the last thing they do ! - https://mastodon.social/@dahukanna/116352210774566255

@dahukanna Nah, they won't. They'd have to make me use these things first.

I chose not to play, and am very happy to be "left behind" and watch them walk off the cliff.

@algernon what is this tool? i kinda want ot try it on my code now

@lobo https://github.com/o-k-a-y/vibecheck

Be aware, it's slop, and it is wrong, and it will make you weep.

@algernon Wait until you see what job recruiters think of spell check and hyperlinks in PDFs.

@drwho Urgh.... now I kinda want to know.

@algernon, honestly, it is distressing how much of all these "AI reviews" are literally random lint that could be trivially done with a much more efficient and much more reliable tool.

@mgorny /reviews//;, and still correct.

Anything these "AI" tools can solve, there are existing tools that can do it better.

Reviews? Linters.
Vulnerabilities? Fuzzers.
Typos? Spell checkers.
Boilerplate? Templates, DSLs, libraries, etc.

And so on and so forth.

@algernon agreed and am looking forward to the “reckoning”.

@dahukanna Kinda the same! At least the reckoning part.

Though, the financial collapse that'll follow the bubble popping will not be pretty, and I'm not looking forward to that part.

@algernon I recently read a good article about the equivalent of this but for prose: https://marcusolang.substack.com/p/im-kenyan-i-dont-write-like-chatgpt

@algernon Wonder what the AI detection tool makes of genuine tells like LLM has created its own equivalents of common libraries from scratch instead of just using them ?!

@m If it would score that kind of stuff, it would probably score it as a good thing because dependencies were avoided, increasing supply chain safety or something.

@algernon I'd love to see the score on older code (why not some unix code from way before LLM).

@lord Judging by the kind of scoring it uses, pretty sure it would declare the moon lander code human-free.

@algernon Just for shits and giggles, I ran a project I wrote in Go through this “detection” tool.

Yep, it gave me full “AI” credit for “machine-perfect formatting”.

This is *Go*, FFS. It’s kind of legendary for not only having a characteristic default formatting, but also for very strongly encouraging Go programmers to incorporate said standard formatting into their toolchain as an automated process.

Also got accused of being AI for the same sorts of things that you saw in its output for Rust, like sensible naming conventions, actually adhering to standards from a linter, and (my favorite) *proper exception handling*.

@algernon
... I wonder, is this the point of the tool? Classify bad code as "human" as a means of creating that negative association in our field?

@syeberman It's vibe coded, so... probably yes.

@algernon "fluid" interfaces bad. Hmm. (maybe Rust uses slightly different terminology; I can't remember)

@algernon I swear AI "detectors" - automated and human - are going to end up being nearly as big a threat as actual AI. The rapid destruction of trust and the accusations that going with it visible on social media are horrifying.

@srtcd424 yep. All of those "tools" do much more harm than good. By its nature, "AI" output will always resemble some form of human work, because it can't do anything else but plagiarize.

All these tools do is hurt those who have been robbed most.

@algernon Anybody who puts a ticket id into a comment, enjoys pain. Put the justification into the commit if you need to, and the comment should... Comment the code, as to why you're doing something...

@shaknais I can do both: include a ticket ID, and the justification.

In this case, however, there is no ticket id. There's a copyright header that reads SPDX-FileContributor: @someone.

In another case, ip saddr @allow_v4 accept (as used in nftables) was misdetected as a ticket/issue id, because clearly, anything starting with @ is a ticket id.