Testing LLM clothing advice against real hourly forecasts

Wearton — Sat, 24 Oct 2026 07:06:16 GMT

Ask a language model "what should I wear today?" and you get a confident answer. Checking whether the answer is good is harder. We build Wearton, an AI stylist that suggests outfits from your own closet for the weather and your plans, so we needed a way to test this. The result is WeatherWearBench: a text-only benchmark with 765 tasks built from 72 real hourly forecasts in 16 cities. The paper and the dataset are public. This post is about the design decisions that you can reuse in your own LLM evaluations.

The task

Each task gives the model three things:

the user's outdoor plans for the day;
an optional log of how they felt on earlier days;
a closet of about 30 items.

The model reads the hourly forecast and the plans, and returns JSON: the items to wear or take, the feels-like temperature it dressed for, and flags for rain and sun.

Decision 1: no judge model

Code scores every answer, with fixed rules for warmth, removable layers, season errors and protection against rain and sun. No language model or person judges the answers. Runs are reproducible and cheap: the whole study was 18,090 calls through OpenRouter for USD 7.10.

Decision 2: opaque item ids

Items are named by opaque ids. The model has to pick from the user's own closet. It cannot answer with a perfect jacket that the user does not own.

Decision 3: the time of the request is a trap

A person who asks at 07:00 about a lunch walk needs clothes for noon, not for the cold morning. A rule that dresses for the feels-like temperature at the time of the request makes gross errors on 20.3% of tasks. The five language models we tested make them on 0.2–6.3%.

Decision 4: single answers are not enough

A baseline surprised us. One fixed outfit per closet, which states the middle of the outdoor range and the right flags, reaches a strict pass rate of 74.3%. The best model reaches 77.7%, on days not used to choose that outfit. When a constant answer scores 74%, a single-answer score tells you little about whether the model reads the weather.

So the main results come from paired checks: two tasks that differ in one thing, for example warmer outdoor hours with the same 24-hour mean, or a user who often feels cold. The main measure looks only at pairs in which the old outfit no longer fits the new weather. With the same 24-hour mean, the models' new answer passes and fits the new hours in 58.5–93.5% of these pairs, against 5.7–21.8% for another answer to the old prompt.

The lesson for any LLM evaluation: check that the output changes when the input changes in a way that matters.

What we tested

Five low-priced language models (Claude Haiku 5.5, GPT-6 Luna, Gemma 4 31B, Gemini 3.1 Flash Lite and GPT-5 nano), one decision model asked through an adapter, and 8 rule baselines.

Limits

A high score does not mean that the person will be comfortable. The benchmark checks the total warmth and the protection of the listed items against its rules. It does not measure real comfort, and the rules have not been checked with people yet.

Try it

Dataset: wearton/WeatherWearBench on Hugging Face (CC BY 4.0).
Paper: figshare, DOI 10.6084/m9.figshare.34618188.
The code and the prompts will be published on the research page. The runner skips calls that already have an answer, so adding a model does not repeat the others.

Research page and paper: wearton.app/research/weatherwearbench

Wearton Engineering