When AI Reads Your Product Photos, Not Your Keywords

Devraj runs Foundry & Field, a $2.3M-a-year outdoor gear brand that does about 70% of its volume on Amazon and the balance on Shopify. In March his best-selling insulated flask, top three in its category for two solid years, slid to page four inside a week. Same price. Same 4.7 rating. Same title he’d split-tested into the ground.

He called us on a Tuesday, genuinely rattled, because nothing in his playbook explained it.

We pulled the listing apart and the keywords were fine. The images were the problem, or rather, what the images failed to say to a machine that had recently started doing the looking. A seller we got on a discovery call with put it more bluntly than we would have: Amazon’s Rufus is now deciding which products to show hundreds of millions of shoppers, and it reads your pictures, not your keywords. Most sellers have no idea.

He’s basically right. And that shift is the most under-discussed change in product discovery this year.

The flask that dropped to page four

Devraj’s hero image was gorgeous. Flask centered on pure white, dramatic lighting, not a single distraction. The kind of shot a brand manager approves in two seconds.

To a person, that image says premium. To a vision model trying to understand what the product is and who it’s for, it says almost nothing. No hand holding it, so no sense of size. No trail, no kitchen counter, no gym bag, so no use case. The lid was closed, the material was ambiguous under studio gloss, and there wasn’t a single frame in the carousel that showed the thing actually doing its job.

Rufus had quietly reweighted toward listings whose imagery it could understand. Competitors who shot their flasks in context, on a hike, beside a campfire, filled with visibly steaming coffee, were getting surfaced for questions like “flask that keeps coffee hot all day.” Devraj’s listing, built for a keyword era, was invisible to a question the model answered by looking.

That’s the whole story in miniature. The retrieval layer changed underneath him and the assets didn’t.

What the model actually sees when it looks at a photo

A vision model doesn’t see a flask. It builds a numerical representation of the image, then matches that against what a shopper asked for. Captioning models generate a sentence describing the frame. Embedding models turn the picture into a vector that sits near other pictures and phrases with similar meaning. Amazon’s Rufus and Google’s multimodal models both work in roughly this shape, even if the plumbing differs.

So the question stops being “what keywords did I stuff in the title” and becomes “what would an honest caption of this image say.”

Here’s the gap we draw on a whiteboard for almost every client.

What a human reads in your hero shot	What the model reads in the same shot
”Looks premium, clean, expensive"	"Cylindrical object, neutral background, no scale reference"
"I can tell that’s stainless steel"	"Reflective surface, material uncertain"
"Obviously a travel flask"	"Container, use context absent"
"Nice lifestyle vibe"	"Subject present, activity not depicted”

The left column sells to a person scrolling. The right column decides whether you show up at all when the shopper talks to an assistant instead of typing two words into a box. Both matter now. The mistake is assuming the left column gets you the right one for free.

The audit we run before touching a single image

Before anyone reshoots anything, we run every hero image and the first two carousel slots through a plain captioning pass and read what comes back. Not the marketing copy, the literal description.

If the caption for a $90 flask comes back as “a silver bottle on a white background,” that listing is starving the model. If it comes back as “a stainless steel insulated flask held by a hiker on a mountain trail, steam rising from the cup,” the model has something to rank.

We score each SKU on four things the caption either contains or doesn’t: scale (is there a hand, a body, a known object for size), context (where and how is it used), material (can the surface be named), and state (is the product open, in use, doing the thing it’s for). Most catalogs we see score two out of four on their hero and worse on the carousel. Devraj’s flask scored one.

The audit is boring and it’s the highest-leverage hour in the whole project. You can’t fix what you can’t see the model failing to see.

The attributes that move the needle

Once you know what’s missing, the fixes cluster.

Scale is the cheapest win and the one brands skip most. A product floating on white has no size. The same product in a hand, or beside a coffee mug, gives the model a reference it can reason about, and it gives the shopper one too. Nobody loses here.

Context is the one that actually changes retrieval. A model surfaces your SKU for “gift for someone who camps” because it saw a tent and a campfire in frame, not because your title said “great gift.” Show the product where it lives. The off-white seamless backdrop is a 2015 instinct that now actively costs you.

Material and texture need a frame that lets the surface read honestly. Heavy retouching that smooths everything into plastic perfection strips the exact cues a model uses to tell brushed steel from painted aluminum. Shoot at least one frame that looks like the thing, not like a render.

Then there’s state, which fashion and food brands get for free and everyone else forgets. A closed box, a folded garment, an unlit candle, these are the hardest things for a model to interpret. Show the product mid-use. The candle lit. The jacket worn. The flask pouring.

Alt text, filenames, and the data the model quietly reads

Pixels aren’t the only input. When a model is unsure about an image, it leans hard on the text wrapped around it, and most stores hand it garbage.

Alt text is the big one. Roughly speaking, the majority of product images we audit have no alt text at all, which means the model gets zero textual confirmation of what it’s half-guessing from the pixels. Good alt text isn’t a keyword dump, it’s the honest caption you wish the model would generate: “stainless steel insulated flask held on a mountain trail, steam rising.” Shopify lets you set this per image, and their own guidance is worth following to the letter.

Filenames matter more than people expect. A file called IMG_4471.jpg tells a crawler nothing. Rename it to insulated-steel-flask-hiking-trail.jpg and you’ve handed over a free signal. It costs one keystroke per image.

Structured data is the layer agencies fight over. Marking up your product imagery with a proper ImageObject schema gives the model a machine-readable claim about what each photo depicts, which it can weigh against the pixels. We treat this as grounding: the image says one thing, the schema confirms it, and the model trusts the SKU more for the agreement. Marco, an agency dev we trade notes with, told us in a Slack DM that schema-aligned listings were the only ones that held position through the Rufus reweighting his clients saw in spring. Small sample, but it tracks with what we watched happen.

Shooting a catalog the machine can parse

When it’s time to actually shoot, the brief changes shape.

Every SKU gets a hero that still wins the human glance, because conversion hasn’t gone anywhere. But the carousel now carries the model’s job. We plan one in-context frame (product where it’s used), one scale frame (product with a hand or a known object), one material frame (close, honest, lightly retouched), and one state frame (product in use). Four frames that between them give a vision model nowhere to be confused.

Tag as you shoot, not three weeks later when nobody remembers which file is which. Alt text, filename, and schema get written against the shot list, so the text and the pixels say the same thing from day one. That alignment is the entire game. A gorgeous photo with a contradicting caption confuses the model more than a mediocre photo with an accurate one.

And shoot for the question, not the keyword. The old instinct was to imagine the search box. The new one is to imagine a shopper describing a need out loud to an assistant, then make sure at least one frame answers it visually.

Knowing whether any of it worked

You can’t watch Rufus rank, but you can interrogate it. We keep a list of the natural-language questions a real buyer would ask for each hero SKU, and we ask the assistants those questions on a schedule. “What’s a good flask for long hikes.” “Insulated bottle that fits a car cup holder.” If your SKU surfaces and gets described accurately, the imagery is doing its job.

We log it monthly per client: which questions surface the product, which describe it correctly, which pull a competitor instead. The trend line matters more than any single check, because the models shift and you want to see direction, not a snapshot.

Devraj’s flask, six weeks after the reshoot, started surfacing again for the “keeps coffee hot all day” style questions it had vanished from. Not because we found a keyword. Because the new carousel finally showed the thing doing what shoppers were asking about.

What we keep telling clients

The reflex when rankings drop is to rewrite the title and tweak the bullets. That muscle was trained on a decade of keyword search and it’s pointed at the wrong layer now.

The catalog is becoming a thing machines read by looking, and most brands are still optimizing the words. The brands that will own discovery over the next two years aren’t the ones with the prettiest hero shots. They’re the ones whose imagery makes an honest, legible claim a model can verify and rank. Pretty and legible aren’t the same thing, and the gap between them is where listings quietly die.

None of this means firing your photographer or chasing the model with a thousand variations. It means shooting with two audiences in mind, the shopper and the machine, while making sure your text never contradicts your pixels. That discipline is cheap, it compounds, and the reshoots don’t have to.

Devraj didn’t reshoot his whole catalog. We fixed eleven hero SKUs, rewrote alt text across the store, and added schema. The flask is back in the top five for the questions that matter, and the rest of the line is on a rolling schedule. He spends less time guessing why rankings move now, which honestly was the bigger win.

Questions we get every week

Does this only matter on Amazon, or does it hit Shopify too?

It hits anywhere an AI layer sits between the shopper and your catalog, which increasingly is everywhere. Amazon’s Rufus is the loudest example, but Google’s multimodal search and a wave of on-site assistants read images the same way. Optimizing for one tends to help across all of them, because they’re solving the same problem.

Do I have to reshoot my entire catalog?

Almost never. Start with your top SKUs by revenue. Fix the hero and the first two carousel slots, and get alt text and schema right across the store. Most of the visibility lift comes from a small fraction of products, so triage by revenue and work down.

Are AI-generated product images good enough for this, or do I need real shoots?

Generated imagery is fine for backgrounds and context variations, and it’s genuinely useful for testing, but for the hero and the material frame real shots still win because buyers and models both pick up on the slightly-off texture of generated product photos. Use generation to extend a good shoot, not to replace it.

How fast do these models change what they reward?

Faster than you’d like, which is why we measure with questions instead of chasing a fixed checklist. The underlying principle, show the product honestly and label it accurately, has stayed stable even as the models churn. Build for that and you’re not redoing this every quarter.

If your catalog has quietly slipped in AI-driven search and you can’t see why, book an image audit with us and we’ll show you exactly what the model sees.