Model Evaluation

Building Meta Horizon's first AI quality framework

For Meta's first generative AI clothing tool, I split quality into what experts could score and what only users could tell us. I then wrote a scoring scale precise enough that 2 reviewers could land on the same score. Over 90% of outputs met the bar at launch, and the framework became the standard across multiple AI systems in Meta Reality Labs.

Hype reel for the launch

Overview

At a glance

My role
Design lead and model designer. I designed the framework, ran an evals onsite to align the team, and wrote the scoring criteria.
Partners
  • A UX researcher who scored the subjective criteria
  • A 3D artist who set where the objective visual quality bar sat
  • Computer vision engineers on model limits and live experiments
  • Contract team of human raters scoring outputs against my evaluation criteria
Timeline
3 months
Model
A fine-tuned version of Emu, adapted to paint textures onto 3D clothing meshes
Audience
1 billion Meta avatar users across Instagram, Facebook, and Horizon, targeting teens and young adults

Headline metrics

>90%of outputs exceeded the ship bar
960Kshirts created and saved in 4 months
52%of people wore a shirt they created

Launch performance across Facebook, Instagram, and Meta Horizon.

Meta's avatars used across Instagram, Facebook, and Horizon had a reputation problem. Teens and young adults, the audience we most wanted to grow, thought they were ugly. Meta's answer was a new avatar art style, and my team was building the cornerstone of its launch: a text-to-image tool that let anyone design their own avatar shirt.

Clothing was a big part of the problem in avatars being appealing. In the creation funnel, 35% of people dropped off at the step where they chose what to wear, and even people who saved an avatar mostly kept the default outfit.

Avatar onboarding funnel

Drop-off at each stage of avatar creation

Avatar creator hair color and hairstyle screen

7%drop off

Hair style

Avatar creator facial features screen

10%drop off

Facial features

Avatar creator makeup screen

11%drop off

Makeup

Avatar creator body type screen

8%drop off

Body type

Avatar creator clothing selection screen

35%drop off

Clothing

To give teens and young adults another chance with our avatars, what they made with our t-shirt creation tool had to look good enough to wear.

The tool was the first AI feature at Meta that had to meet 2 very different quality bars at once:

  • Strict adherence to a tightly defined visual ecosystem
  • Genuine appeal to teens and young adults

Our art team judged quality technically. Is it rigged right, are the textures clean, does it fit the Horizon look? Our users judged it by feel. Is this cool, and do I feel good wearing it?

Both are real, but they kept blurring together in every conversation, and nobody had a way to measure either one. And you can't improve or ship what you can't measure.

How people think about digital clothing quality

Is the item rigged properly?
Technical quality
Are the textures accurate?
Perceptual quality
Does it feel trendy and desirable?
Experiential quality
Does it feel creative and novel?

Naming the split

I sorted everything we cared about into 2 buckets of criteria we could measure with either internal evals or external research.

  • Objective criteria ensured generated designs didn't create visual dissonance with our avatar and Horizon ecosystem and were things we could measure with internal evals, like object deformation, texture uniformity, and accuracy.
  • Subjective criteria ensured generated designs had a high level of social value and were things we could only measure through research with real users, like appeal, representation, and expression.

Measured with internal evals

Objective criteria

Ensured generated designs didn't create visual dissonance with our avatar art style.

Measured with external research

Subjective criteria

Ensured generated designs had a high level of social value and appeal.

Aligning a global team

Our team stretched from Tel Aviv to the West Coast, with maybe 2 overlapping work hours a day, and async messages weren't getting us aligned. So I called for a 5-day onsite and designed the week backward from what we needed to leave with: trust across a team that had mostly never met, a ranked list of quality gaps, a plan to measure improvement, and an agreed ship bar.

Seattle
San Francisco

10-hour time difference

Tel Aviv
ONSITE GOALS
Connect
Prioritize
Plan
Align

Each specialist started by teaching the room their own lens on quality. Then everyone used the tool themselves, trying to recreate the pattern on the dress our researcher was wearing that day. You judge a model differently after 20 minutes of using it yourself. When we critiqued real outputs together, the lenses collided usefully. Our artist caught deformations nobody else saw, our researcher heard intent mismatches, and engineering saw model limits.

Then we ranked the gaps against research, not opinion. Common patterns like tie-dye and polka dots became top priority, because if the tool can't render a pattern someone asks for by name, they stop trusting it. Abstract prompts like “happy” ranked lower, because research showed they were edge cases.

Making quality measurable

For the objective criteria, I built a 3-point scoring scale. Our 3D artist decided where the bar sat visually. My job was to turn that judgment into language precise enough that reviewers without art training could score outputs consistently at scale. That's the heart of evals — making “good” so concrete that 2 different people land on the same number.

Example scoring

Object deformation

Pattern of heavily deformed lion illustrations

1 = Fail

More than half of the objects in the image are severely deformed. If there are too many objects, the percentage can be estimated.

Pattern of lion illustrations with minor irregularitiesMinimum ship bar

2 = Pass

The vast majority of objects have either no deformations or are only minor in nature (e.g., anatomy of objects are mostly accurate with minor irregularities).

Pattern of detailed lion illustrations without deformities

3 = Ideal

All objects in the image appear normal without deformities (e.g., hands have 5 fingers, arms are in the right location, there are no irregular details).

On the subjective side, we tied quality to the business goal of growing people's affinity for their avatars. That came down to measuring representation, appeal, and social value, which our researcher scored using an existing Meta Reality Labs framework for product shippability.

Representation

Combined average metric of whether a final item matches people’s intentions and helps them better express themselves.

Appeal

Combined average metric of whether the final item increases affinity for their avatar and they feel confident wearing it.

Social value

Establish baseline measurement on if the final item allows them to connect with others the way they want to.

With a scale to measure against and our gaps ranked, the next question was how to close them in time for a launch date only 2 months away. Many of our top-ranked gaps required fine-tuning the model with better training data—something that would take our computer vision engineers a few months.

That helped me scope the launch between what they could solve with fine-tuning and what I could take on through prompt-engineering fixes. I brought leadership a launch plan that put every gap in one of 3 groups to keep the launch on track:

  • What prompt engineering could fix with a few days to weeks of work
  • What fine-tuning had to fix in a few months before we could ship
  • What dropped below the cutline and we could ship without solving, as the issues likely wouldn’t hurt adoption of the tool

What dropped below the cutline, like solving for abstract prompts such as “happy” and adding text to shirts, we were confident wouldn't hurt adoption at launch. This is because our research showed they were either uncommon prompts or features users didn't expect.

We shipped with confidence in the model. Over 90% of outputs scored a shippable 2 or higher on every objective criterion. And people went nuts for the tool. In the first 4 months, nearly half a million people created shirts, saved 926K creations, and 52% wore one they made on their avatar.

The framework outlived the launch because its foundation held, becoming the standard across 3 generative AI systems in Meta Reality Labs. Quality still ran from technical to experiential, even as the questions along that spectrum changed with each AI tool and audience.

Then we brought AI clothing creation for Meta avatars to Horizon creators. Their tool ran on Meta 3D TexGen, a shape-aware model. I kept most of the objective criteria and added 9 new ones for 3D outputs. With a researcher, I also reworked the subjective criteria around what creators cared about. Did an item represent their brand? Would they feel confident selling it?

Later, I brought the framework to an agentic system for automating video marketing assets for game developers. It scored 26 criteria across visual quality, audio quality, and game accuracy. Its subjective criteria, for both consumers and creators, measured a video’s potential to engage, its relevance, and whether it felt trustworthy.

Is the item rigged properly?
Technical quality
Are the textures accurate?
Perceptual quality
Does it feel trendy and desirable?
Experiential quality
Does it feel creative and novel?