How a project gets scored

Every project here was read the same way, and this is that way.

What is measured, and what is judged

Magpie counts things itself — stars, commits, how old a repository is, how fast it is growing. It never asks a model to count. Those numbers are facts and they are checkable.

What a model is asked is a fixed set of 15 questions, each with a finite set of answers: a probability, or a pick from a named list, or a position on a written-out ladder. It is never asked to write anything. That matters because a sentence can be persuasive without being true, and a number on a ladder whose rungs are written down cannot.

Reading a project as it was

Each project is read as it stood four months into its life, from the README at the commit that was current then — not today's README, which for anything successful has been rewritten by people who already knew it succeeded.

The name and the author are removed before the model sees it. A project called react would otherwise get the benefit of everything that happened after 2013, and the whole point is to ask what could be told at the time.

The questions

Would this be hard to rebuild?

If it disappeared, could you put the same capability together from things that already exist? This is the one that separates a new idea from a nicer wrapper.

How much would change if it were adopted?

A ladder from "saves keystrokes on something already easy" up to "the tools around it become the wrong question".

Is there machinery of its own?

Added because the question above was reading claim size. A README can open with a large claim and be a thin layer over someone else's work; this asks what is actually behind it.

Is there an idea here to keep?

From "a compilation, pointers to other people's work" up to "a way of thinking that outlives the implementation". This is the one that puts a list of links at zero and keeps it there.

Could one person start today?

Whether anything has to be asked of anybody — an account, a sales conversation, a waitlist — and separately, how much has to happen before it runs.

Is there something to build in a weekend?

And how long from starting to having something worth showing, in units a person plans with: a day, a week, a month, longer.

How well it does

The whole pipeline was run backwards over fifty-one projects whose outcome is already settled — a third of them chosen to be turned down, because a ranking that says yes to everything scores perfectly and is worth nothing.

Star counts are deliberately not the target. The most-starred repository in that test set has half a million stars and is a list of links; it comes last. Across the set, the ranking and the star count disagree slightly more often than they agree, which is the intended behaviour rather than a defect.

What this cannot tell you