How Search Engine Crawlers See Your Page (and What They Miss)

Search engine crawlers see your page as code, not as a design. They download the raw HTML, follow links found in <a href> tags, read text, headings, meta tags and structured data, and (in Google's case) render JavaScript later to see content that scripts add. Anything that only appears after a click, lives inside an image, or is blocked by robots.txt may be invisible to them.

Key takeaways

• Crawling, rendering and indexing are separate steps. A page can be crawled but not rendered or not indexed.
• Google renders JavaScript with an up-to-date Chromium, but rendering can happen after the initial crawl. Many other bots do not run JavaScript at all.
• Crawlers follow <a href="..."> links. They do not click buttons, open menus or fill in forms.
• robots.txt controls crawling; noindex controls indexing. Mixing them up is a common and costly mistake.
• Compare what the bot sees with what users see before assuming a page is fine.

The three steps: crawl, render, index

Crawl. The crawler requests a URL, just like a browser does. Before that, it checks the site's /robots.txt file to see if it is allowed. The server responds with a status code (200, 301, 404, 500 and so on), HTTP headers and the HTML body.

Render. Google puts the fetched page in a queue for its Web Rendering Service, which runs the JavaScript in a headless Chromium browser and builds the final DOM. This can happen shortly after crawling or later, depending on resources. Content that exists only after rendering is therefore seen later than content in the raw HTML.

Index. Google analyzes the rendered content, picks a canonical URL among duplicates, and decides whether to store the page in its index. Being crawled does not guarantee being indexed.

What a crawler actually reads

Here is a simplified page and what a crawler extracts from it:

HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
X-Robots-Tag: index, follow

<!doctype html>
<html lang="en">
<head>
  <title>Blue Ceramic Mugs | Example Store</title>
  <meta name="description" content="Handmade blue ceramic mugs, 350 ml.">
  <link rel="canonical" href="https://www.example.com/mugs/blue/">
</head>
<body>
  <h1>Blue Ceramic Mugs</h1>
  <p>Each mug is thrown by hand and glazed twice.</p>
  <a href="/mugs/green/">Green mugs</a>
  <span onclick="go('/mugs/red/')">Red mugs</span>
  <img src="/img/blue-mug.jpg" alt="Blue ceramic mug on a wooden table">
</body>
</html>

• Status code and headers: 200 means "this page exists". The X-Robots-Tag header can carry indexing directives, just like a meta robots tag.
• Title, meta description, canonical: read from the <head>.
• Headings and text: the <h1> and paragraph text are indexable content.
• Links: /mugs/green/ will be discovered. The red mugs "link" will not, because it is a <span> with a JavaScript click handler and no href.
• Images: the crawler cannot read text inside the picture, but it reads the alt attribute, file name and surrounding text.

A Search Engine Spider Simulator shows you this text-and-links view of any URL, which is a fast way to spot missing content or links that are not real links.

JavaScript: what gets seen and what does not

Google can index JavaScript-generated content, and Google's own JavaScript SEO basics guide explains how. But "can" is not "always, instantly". Keep these rules in mind:

• Put critical content in the initial HTML when you can. Server-side rendering or static generation means every crawler sees the content on the first fetch.
• Do not rely on user actions. Googlebot does not click "Load more", open tabs that fetch content on click, or scroll like a human. Content that only loads on click may never be seen.
• Use real links for navigation. Single-page apps must output <a href="/real-path"> links, not just click handlers. URL fragments such as #/products are generally not treated as separate pages.
• Do not block resources. If robots.txt blocks your JavaScript or CSS files, Google cannot render the page properly.
• Remember other crawlers. Bing renders JavaScript too, but many other bots, including social media preview bots and a number of AI crawlers, read only the raw HTML. Do not assume they will run your scripts.

Raw HTML vs rendered DOM

To compare the two views, open the page with Get Source Code of Webpage to see the raw HTML the server sends, then compare it with what you see in your browser. If your product description, price or internal links are missing from the raw source, they depend on JavaScript. That is not automatically a problem for Google, but it is a risk worth knowing about. Google Search Console's URL Inspection tool shows the rendered HTML Google actually produced.

Signals that control crawlers

The robots.txt and noindex trap

If you block a URL in robots.txt, Google cannot fetch it, so it never sees a noindex tag on that page. The URL can still appear in results (usually without a description) if other pages link to it. To remove a page from the index, allow crawling and use noindex. Use robots.txt for crawl control, such as keeping bots out of infinite filter combinations:

User-agent: *
Disallow: /search
Disallow: /*?sort=

Sitemap: https://www.example.com/sitemap.xml

The robots.txt format is standardized in RFC 9309. Note that robots.txt is a request, not a security measure. Well-behaved crawlers obey it; bad bots ignore it.

Things crawlers commonly miss

• Text in images, including headlines baked into banner graphics.
• Content behind logins or forms, such as search results pages users reach by submitting a query.
• Content loaded on click, such as tabs or accordions that fetch content from the server only when opened. Content that is already in the HTML but visually collapsed is fine.
• Infinite scroll without paginated URLs. Give each chunk a real, linkable URL.
• Orphan pages with no internal links pointing to them. A sitemap helps discovery, but links are how crawlers understand importance.
• Very large HTML files. Googlebot processes only the first 15 MB of an HTML file. Few pages hit this, but bloated inline data can.

Crawl budget: should you worry?

For most sites with a few thousand pages, no. Google has said crawl budget is mainly a concern for very large sites or sites that generate huge numbers of URLs, for example through faceted navigation or session IDs. If you do have that problem, the fixes are: stop generating useless URL variations, return proper 404s for removed pages, keep server responses fast, and avoid long redirect chains. Frequent 5xx errors and slow responses make Google crawl more cautiously.

A five-minute crawler check

1. Fetch the page in a spider simulator and confirm the main text, headings and important links are present.
2. View the raw source and check the title, meta description, canonical and meta robots tags.
3. Confirm the status code is 200 for live pages and that redirects go straight to the final URL.
4. Open /robots.txt and make sure nothing important, including CSS and JS folders, is disallowed.
5. In Search Console, run URL Inspection and view the rendered HTML and screenshot.

Next steps

• Pick your three most important page templates and run the five-minute check on each.
• Replace any click-handler navigation with real <a href> links.
• Move critical text out of images and into HTML.
• Audit robots.txt for rules that block pages you actually want indexed, and switch to noindex where removal from results is the goal.