Why AI Reads More Than Just Your Website Text?

TL;DR 

1. AI search no longer reads only text. It pulls signals from images, videos, social conversations, and metadata simultaneously through a process called multimodal retrieval.
2. Image quality is now a ranking signal. Blurry images produce unclear visual tokens, which can lead AI to describe your brand inaccurately.
3. Product adjacency matters. The objects sitting alongside your brand in images influence how AI categorises your positioning, pricing tier, and audience.
4. Videos contribute through transcripts, descriptions, chapter markers, and thumbnails, not just the visual content itself. YouTube remains one of the strongest sources for AI systems to retrieve.
5. Social conversations on Reddit, LinkedIn, and other platforms now feed directly into how AI builds confidence in your brand. Silence on social is no longer a neutral signal.

Most people think AI search reads only words on a page. This is not true anymore.

Today, AI systems look at images, watch videos, listen to what people say on social platforms, and read text all at the same time. They put all of this together before they decide what answer to give. This is called multimodal retrieval.

If your brand only focuses on text, you are giving AI less than half the picture.

‍

What is multimodal retrieval in AI search? 

The word "multimodal" just means "many types."

Instead of one channel (text), the AI looks at many channels at the same time:

- Text on your page - Images on your page - Videos on YouTube or in your content - What people say about you on social platforms - Captions, alt text, and transcripts tied to your media

When all of these channels say the same thing about your brand, AI has more confidence in you. When they say different things, AI gets confused.

A research paper published on arXiv in 2025 explains this well. In real-world scenarios, humans naturally interact with multimodal data such as browsing web pages that combine text, images, and videos in mixed layouts. AI systems need to analyse images or videos alongside text to better understand the context.

‍

How does AI actually read the images on your website? 

When you put an image on your site, AI does not just see a picture. It breaks the image into small pieces called visual tokens. These are like words, but for images. Visual tokens enter the same retrieval pipeline that determines which content the AI pulls into its answers, meaning images are evaluated alongside text rather than separately. 

To large language models, images, audio, and video are sources of structured data. They use a process called visual tokenisation to break an image into a grid of patches, or visual tokens, converting raw pixels into a sequence of vectors. This unified modelling allows AI to process a picture as a single coherent sentence.

This means your image has to be sharp and clear. A blurry or heavily compressed image gives the AI bad data. When the data is bad, the AI can make wrong guesses about what your image shows.

Poor resolution can cause the model to misinterpret those tokens, leading to hallucinations in which the AI confidently describes objects or text that do not actually exist because the visual words were unclear.

Here is something most brands miss. The AI also looks at what objects are in your image and what they are sitting next to. AI identifies every object in an image and uses its relationships to infer attributes such as a brand, price point, and target audience. This makes product adjacency a ranking signal.

If you run a premium brand but your images show cheap-looking surroundings, AI picks that up. It will change how your brand is described.

Here’s what to do:

1. Use high-quality original images, not stock photos used by many other sites 
2. Write clear, specific alt text that tells AI what the image shows and why it matters 
3. Make sure the objects in your images match the story you want to tell about your brand

You can learn more about how Google reads images here: [Google Cloud Vision API documentation](https://cloud.google.com/vision/docs/detecting-web)

‍

How AI ranks images: a patent tells the story

Google holds a patent called "Multi-modal image ranking using neural networks" (US Patent 10642887). It shows exactly how this works.

The system generates a visual modality ranking by comparing an embedding of at least one visual feature of a digital image to an embedding of a textual query. It also generates a language modality ranking by comparing text features associated with the image to the same query. A multimodal neural network then determines the importance of each ranking type and combines them into a final score.

In simple words: AI scores your image on what it looks like AND on the text around it, then combines both scores into one number. If your image scores well on both, it shows up. If only one is strong, you fall behind.

You can read the patent here: [US Patent 10642887 - Multi-modal image ranking using neural networks] (https://image-ppubs.uspto.gov/dirsearch-public/print/downloadPdf/10642887)

Another Google patent (US11782970B2) goes further. The scoring function ranks search results based on various signals, including where and how often the query text appears in the surrounding document text, in an image caption, or in alt text.

This tells us that text near your image matters just as much as the image itself. If your image sits alone on a page with no good text around it, it scores poorly.

‍

How does AI use videos to build answers? 

Videos are now a major source for retrieval. AI does not need to watch your whole video. It pulls signals from many places at once.

Current commercial systems, such as YouTube search, rely heavily on non-visual metadata like titles, descriptions, and user engagement signals. Newer retrieval systems also capture signals from visual content, audio, and embedded text within the video itself.

Research from a 2025 academic paper on multilingual video retrieval found that to capture signals from visual content, researchers select key frames from a video, encode them, and use them in the search pipeline. Through testing, the benefit of sampling more frames saturates after 16 key frames.

This means AI does not need to see every second of your video. It looks at snapshots. If those snapshots are clear and relevant, you are in good shape.

Here’s what to do for your videos:

1. Add a full text transcript to every video 
2. Write a detailed description that covers the main points of the video 
3. Use clear chapter markers with titles 
4. Make sure the thumbnail image is sharp and relevant 
5. Post the video on YouTube, which is a primary retrieval source for many AI systems

‍

Why are social signals now part of AI retrieval? 

This is the part most brands are missing.

When people talk about you on Reddit, LinkedIn, or Twitter, AI picks that up. It uses these conversations to form a picture of what your brand does and how trustworthy it is.

Reddit has become one of the most visible domains in Google's organic search results, now ranking as the fifth-highest visibility domain. Google increasingly prioritizes authentic, human-led discussions, and Reddit threads frequently appear in both traditional search results and AI search engines and LLMs.

A social network's performance directly influences how Google understands, trusts, and ranks the entity as a whole. Its performance also strengthens brand signals, influencing LLM knowledge and grounding searches.

In simple words: if people are talking about you in real, honest conversations on social platforms, AI trusts you more. If no one is talking about you, AI has less confidence in including you.

More online mentions typically lead to more credibility. In some cases, content with strong social engagement also appears directly in search results. 

For example, if you search for a product experience, you will often see top results from YouTube and Reddit, two platforms where social activity drives visibility.

‍

How do AI systems combine text, images, video, and social signals? 

AI does not look at images, then look at video, then look at social separately. It fuses signals from every modality into one answer, the same way it fuses text chunks from different sources.

A 2025 research paper explains that multimodal retrieval-augmented generation systems process unified information across text, image, table, and video modalities together. The system uses all of these formats to generate a single coherent answer.

What this means for your brand: if your text says one thing, your images suggest another, and social conversations tell a third story, AI gets a conflicted picture of you. It either drops you from the answer or describes you in a way you did not intend.

The brands that show up consistently are the ones where all signals agree.

‍

What should brands actually do to optimise for multimodal AI search? 

Here is what you can start doing this week:

• For images: - Every image needs clear, specific alt text - Use original images, not generic stock photos - Make sure the objects in your image match your brand story - Put good descriptive text near every image
• For videos: - Add a full transcript to every video - Write a proper description, not a one-liner - Use chapter markers with clear titles - Upload to YouTube, not just your own site
• For social: - Be present in the places your audience talks (LinkedIn, Reddit, Twitter/X) - Answer questions in public threads so AI can find your real voice - Build a presence on YouTube since it is one of the strongest retrieval sources for AI
• For everything together: - Make sure your brand description is the same everywhere - The story your text tells, and the story your images show, should match - When people search for problems you solve, they should find you in text, video, and conversation.

‍

Why is multimodal optimisation now a visibility requirement? 

Search is no longer a web page indexed by text alone. Vision models interpret images, ASR models decode speech, and large language models reconcile context across modalities to synthesize answers. That means your visibility depends on how well your assets communicate meaning to these systems, not just to human readers.

This is not about doing more work. It is about making sure the work you already do is legible to machines.

A great blog post with blurry images and no video presence is a half-finished signal. A YouTube video with no transcript is a missed opportunity. A LinkedIn presence with no engagement is invisible to the retrieval layer.

Multimodal retrieval is the reason why some brands appear everywhere, and others with better text content get left out. The ones that appear everywhere have made themselves readable across all signal types.