Scoring & Ranking

Filtering tells you what qualifies. But when 50 models pass your filter, which one should you actually use? Scoring evaluates each candidate across multiple dimensions and ranks them so recommend() can surface the best fit.

Each model gets a score between 0 and 1 based on weighted criteria. The highest-scoring model is the recommendation.

How Scoring Works

1. Criteria evaluate a single dimension (cost, recency, context size). Returns 0 to 1, or undefined when the model has no data for that dimension.
2. Weights set relative importance. Weight 5 matters 5x as much as weight 1.
3. Normalization: weights are normalized, then multiplied by each criterion’s score.
4. Final score: the weighted sum, always between 0 and 1.

const profile: PurposeProfile = {
  criteria: [
    { criterion: costEfficiency, weight: 7 },  // 70% of final score
    { criterion: contextCapacity, weight: 3 },  // 30% of final score
  ],
};

Built-in Criteria

pickai ships five scoring criteria. All use min-max normalization across the candidate set. Scores are relative to the other candidates, not absolute.

costEfficiency

Cheaper models score higher. Based on input cost per 1M tokens. Models without pricing data are uncovered (unknown cost gets no credit for being cheap).

recency

Newer models score higher. Based on releaseDate. Models without a release date are uncovered and excluded from the scale.

contextCapacity

Larger context windows score higher. Based on limit.context.

outputCapacity

Larger output limits score higher. Based on limit.output.

knowledgeFreshness

More recent knowledge cutoffs score higher. Based on the knowledge field (e.g., "2025-03"). Models without a knowledge cutoff are uncovered and excluded from the scale.

Missing Data and Coverage

“We have no data for this model” and “this model scored terribly” are different situations, and pickai keeps them apart:

• A criterion returns undefined when a model has no data for it. That contributes 0 to the score, same as before, but it is tracked.
• Every scored result carries coverage: the fraction of criterion weight backed by real data for that model. A new model missing one benchmark might rank low with coverage: 0.7; that reads “partly unmeasured”, not “weak”.
• criterionCoverage() reports how many candidates each criterion found data for. A criterion covering zero candidates usually means a wrong field key.
• recommend() reports dead criteria through onZeroCoverage and throws when every criterion is dead, so a broken profile fails loudly instead of producing a plausible-looking ranking.

Writing Custom Criteria

A ScoringCriterion is a function that receives a model and the full candidate list, and returns a number from 0 to 1, or undefined when the model has no data for the criterion:

type ScoringCriterion = (model: Model, allModels: Model[]) => number | undefined;

The allModels parameter lets you do relative scoring (min-max normalization). For simple binary criteria, you can ignore it:

// Binary: does the model support vision?
const supportsVision: ScoringCriterion = (model) => {
  return model.modalities.input.includes("image") ? 1 : 0;
};

For relative scoring, use the full model set to score against peers:

// Relative: boost models with above-median context windows
const aboveMedianContext: ScoringCriterion = (model, allModels) => {
  // Find the median context window across all candidates
  const sorted = allModels.map((m) => m.limit.context).sort((a, b) => a - b);
  const median = sorted[Math.floor(sorted.length / 2)];

  // Score 1 if above median, 0 if below
  return model.limit.context >= median ? 1 : 0;
};

Using scoreModels() Directly

If you want scoring without the full recommend() pipeline, use scoreModels() directly:

import { scoreModels, costEfficiency, recency } from "pickai";

const scored = scoreModels(models, [
  { criterion: costEfficiency, weight: 3 },
  { criterion: recency, weight: 1 },
]);

// scored is ScoredModel[] sorted by score descending
console.log(scored[0].id, scored[0].score);

Full Example

custom-scoring.ts

import {
  fromModelsDev, recommend,
  costEfficiency, contextCapacity, recency,
  type ScoringCriterion, type PurposeProfile,
} from "pickai";

const models = await fromModelsDev();

// Write a custom criterion: prefer models with vision
const supportsVision: ScoringCriterion = (model) => {
  return model.modalities.input.includes("image") ? 1 : 0;
};

// Build a custom profile with your criterion + built-in ones
const profile: PurposeProfile = {
  filter: { toolCall: true },
  criteria: [
    { criterion: supportsVision, weight: 5 },
    { criterion: recency, weight: 3 },
    { criterion: contextCapacity, weight: 2 },
    { criterion: costEfficiency, weight: 1 },
  ],
};

const results = recommend(models, profile, { limit: 5 });

for (const model of results) {
  console.log(`${model.score.toFixed(3)} | ${model.name}`);
}