Blog post: robots.txt and sitemap guide with real code examples for site owners and developers

This robots.txt and sitemap guide covers both files with actual syntax examples, not just theory. If Google cannot crawl your pages efficiently or cannot find your sitemap, your content will not rank regardless of how well it is written. These two technical files are the foundation of every successful SEO setup, and getting them right takes less than an hour when you know exactly what to do.

What robots.txt and sitemap files do and why both matter

A robots.txt file tells search engine crawlers which parts of your website they are and are not allowed to access. It does not block pages from appearing in search results on its own, but it does control where Googlebot and other crawlers spend their crawl budget. A sitemap.xml file does the opposite job: it tells search engines exactly which URLs exist on your site and when they were last updated, making discovery faster and more reliable. Together, these two files form the crawl and indexation layer of your technical SEO. If your robots.txt is misconfigured, you can accidentally block your most important pages from being crawled. If your sitemap is missing or outdated, new content may take weeks longer to be indexed. For a broader view of how these files fit into your overall technical health, our SEO services Kuala Lumpur post covers the full ranking picture.

How to write a correct robots.txt file

• The basic structure A robots.txt file lives at the root of your domain, for example yourdomain.com/robots.txt. It is a plain text file read by crawlers before they visit any other page. Every rule block begins with a User-agent directive, followed by Allow or Disallow directives, and optionally a Sitemap directive pointing to your XML sitemap URL.
• A minimal, correct robots.txt example User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://yourdomain.com/sitemap.xml — This tells all crawlers to stay out of your WordPress admin area while allowing the AJAX endpoint that front-end features may depend on, and it points every crawler directly to your sitemap.
• Blocking specific crawlers You can target individual bots by replacing the asterisk with a specific user-agent name. For example, to block a known content scraper while keeping Googlebot unrestricted, you would write a separate User-agent block for that bot with a Disallow: / directive. Google Search Central publishes the full official robots.txt documentation, which is the definitive reference for syntax rules and edge cases.
• Common mistakes to avoid Never add Disallow: / without a preceding Allow rule unless you deliberately want to block all crawlers from your entire site. A single misplaced slash has deindexed entire websites. Also avoid blocking CSS and JavaScript files that Googlebot needs to render your pages correctly. If Google cannot render your pages, it cannot assess their quality or relevance.
• The Sitemap directive You can declare multiple sitemap locations inside your robots.txt file by adding a Sitemap line for each one. This is useful if you maintain separate sitemaps for blog posts, product pages, and images. Each Sitemap directive must use the full absolute URL including the protocol.
• What a valid XML sitemap looks like A sitemap.xml file is written in XML and must conform to the Sitemaps protocol. The minimum required element for each URL entry is the loc tag containing the full canonical URL. You can also include lastmod to indicate when the page was last updated, changefreq to suggest how often it changes, and priority to indicate relative importance within your site. Most SEO plugins for WordPress generate this automatically, but understanding the structure helps you audit what is actually being submitted.
• A basic sitemap.xml structure The XML declaration opens the file, followed by a urlset element that declares the sitemap namespace. Inside urlset, each page is wrapped in a url element containing at minimum a loc element with the full page URL. A well-formed entry might include loc, lastmod with an ISO 8601 date, changefreq set to weekly or monthly, and priority set to a value between 0.0 and 1.0. Your homepage typically carries a priority of 1.0 while supporting pages sit at 0.8 or 0.6.
• Image and video sitemaps Google supports extended sitemap formats for images and videos. An image sitemap includes an image:image element nested inside each url block, with an image:loc tag pointing to the image file. This helps Google discover images that may not be easily found through standard crawling, which is particularly useful for ecommerce and portfolio sites where visual content drives traffic.
• Sitemap index files If your site has more than 50,000 URLs or you maintain multiple content types, use a sitemap index file. This is an XML file that lists the locations of your individual sitemaps rather than listing URLs directly. A sitemap index follows the same XML structure but uses a sitemapindex root element and sitemap child elements instead of urlset and url. You submit only the index file URL to Google Search Console.
• Keeping lastmod accurate Only update the lastmod date when the page content genuinely changes in a meaningful way. Setting lastmod to today's date on every page in your sitemap, regardless of whether the content changed, trains Google to ignore the field. Accurate lastmod signals help Googlebot prioritise recrawling pages that have actually been updated.

A correct sitemap is not just a list of URLs. It is a prioritised signal to Google about what matters most on your site and when it changed.

How to submit your sitemap in Google Search Console

Submitting your sitemap through Google Search Console is the fastest way to tell Google your sitemap exists and prompt indexation of new or updated URLs. To do this, log in to Google Search Console and select your verified property. In the left sidebar, navigate to Indexing and then Sitemaps. In the Add a new sitemap field, enter the path to your sitemap file relative to your domain root, for example sitemap.xml or sitemap_index.xml, and click Submit. Google will then attempt to fetch and process the sitemap. The Sitemaps report shows you how many URLs were submitted, how many were indexed, and flags any errors it encountered. You should also reference the Google Search Central robots.txt documentation for up-to-date guidance on how Googlebot interprets crawl directives alongside your sitemap submission. If you have implemented local business schema on your site, our post on local business schema markup explains how structured data works alongside your sitemap to improve how Google understands your pages. For a complete pre-launch technical review, our SEO audit Kuala Lumpur checklist walks through every layer of technical SEO you should verify before investing in content or link building.

How EEOS helps you get technical SEO right

EEOS works with site owners and developers across Malaysia to audit, fix, and future-proof the technical foundations that determine whether a site ranks. That includes reviewing robots.txt for unintentional blocks, validating sitemap structure and submission status in Google Search Console, and ensuring crawl budget is not being wasted on low-value URLs. If you are evaluating how technical SEO fits into your overall digital marketing investment, our post on SEO services Kuala Lumpur covers the ranking factors Google rewards most in 2026, including the technical signals covered in this guide. Every engagement starts with a full audit of your current crawl and indexation setup before we recommend a single change.