Method

UI generation tasks, 5 tools,Three metrics.

Each code agent received the same design scenarios. For every output, we captured the code, the rendered interface, and all user-facing reasoning, both in the thinking trace and in the final response.

Claude

Bolt

ChatGPT

Vercel v0

Firebase

The prompts are written the way non-designers will prompt

Each prompt is a design scenario with two UX principles embedded in it. The prompt never names them. It describes users, contexts and constraints, and leaves the code agent to recognize which principles apply.

Tier 1 - Structural

What the interface contains

Information architecture: grouping related content, ordering sections, building navigation paths, routing distinct audiences, surfacing what is urgent. Eight tasks.

Tier 2 - Styling

How it looks

Visual communication: color, typography, spacing, hierarchy, contrast, emotional tone. Eight tasks, from straightforward branding to interfaces for constrained devices and readers under stress.

Tier 3 - Functional

How it behaves

Interaction: state, error prevention and recovery, user control, role switching, accessibility features. Eight tasks, from a four-step reservation flow to interactive systems serving large, diverse populations.

The three metrics

TFS

Thinking Fidelity Score

Did the tool build what it said it would?

We extract checkable claims from the reasoning trace: “a three-column grid”, “keyboard-navigable tabs”. Vague claims are discarded. Two evaluators score each claim against the artifact: fully implemented (1.0), partially (0.5), not at all (0.0).

TFS = (1.0 · N_full + 0.5 · N_partial) / N_total

PAS

Principle Adherence Score

To what extent do generative UI tools recognize and implement UX principles that are implicitly embedded in natural-language prompts (tasks)?

Two target UX principles are chosen per prompt before evaluation: structural tasks from information architecture, styling from WCAG, functional from Nielsen’s heuristics. Each scores 1 or 0; PAS is their mean.

DHI

Design Homogeneity Index

How similar are the interfaces produced by different generative UI tools?

All 120 interfaces are rendered as 1200 x 1200 screenshots giving ten pairwise comparisons per prompt. Similarity is kept as three separate dimensions:

Visual: UIClip embeddings.
Color: CIELCh + Earth Mover’s Distance.
Layout: OmniParser v2 + tree edit distance.