Your website might look perfectly fine when you visit it in a browser. Every page loads, every image appears, and every link seems to work. But search engines like Google experience your site very differently from how you do.
Google sends automated bots called crawlers to discover, read, and index your pages. When those crawlers run into problems, they log what are known as crawl errors. These errors mean Google either cannot find a page, cannot read it properly, or is being blocked from accessing it entirely.
The result is straightforward: pages with crawl errors often do not appear in search results. For a business owner, that translates directly to lost visibility, lost traffic, and lost potential customers.
The good news is that crawl errors are not mysterious. They are identifiable, categorized, and fixable. You do not need a developer background to find them. You just need the right tools and a clear process.
This guide walks you through exactly how to check your website for crawl errors, understand what each error type means, and fix the most common issues so your site stays healthy and visible in search. Whether you run an e-commerce store, a local service business, or a B2B company, these steps apply universally.
By the end, you will have a repeatable process you can run monthly to keep crawl errors from quietly undermining your search performance.
Step 1: Set Up and Verify Google Search Console
Before you can check for crawl errors, you need access to the tool that reports them. Google Search Console is the primary free resource for identifying crawl issues directly from Google's perspective. It shows you exactly what Googlebot has encountered when visiting your site, which makes it far more valuable than any third-party estimate.
If you have not set up Search Console yet, start at search.google.com/search-console. Click "Add Property" and enter your website's domain or URL prefix.
You will then need to verify that you own the site. Google offers several methods:
DNS Record Verification: You add a TXT record to your domain's DNS settings through your domain registrar (like GoDaddy, Namecheap, or Cloudflare). This is the most reliable method and covers all subdomains automatically.
HTML Tag Verification: Google gives you a meta tag to paste into the section of your homepage. This works well if you have easy access to your site's code or CMS settings.
Google Analytics Verification: If Google Analytics is already installed on your site, you can verify ownership through your existing Analytics account. This is often the fastest option for business owners already using Analytics.
Once verified, Search Console begins collecting data. It does not show you historical data from before verification, so the earlier you set this up, the better. Give it 24 to 48 hours after verification before expecting data to appear.
One important note for anyone who has searched for older tutorials: Google replaced the legacy "Crawl Errors" report with the current Pages report found under the Indexing section. If you have been following outdated instructions and cannot find the old interface, that is why. All crawl-related issues now surface in the Pages report, which we will walk through in the next step.
If you are new to Search Console and want a broader overview of how to use it alongside other foundational practices, our guide on checking if your website is indexed on Google covers the basics of verifying your site's presence in search results.
Success indicator: You can log into Search Console and see your site's data beginning to populate in the dashboard, including impressions, clicks, and coverage information.
Step 2: Navigate the Pages Report to Find Indexing Errors
With Search Console verified and active, head to the left-hand navigation and click Indexing, then select Pages. This is where all crawl-related issues now live.
The Pages report divides your site's URLs into two main categories: Indexed and Not indexed. The "Not indexed" section is where crawl errors and indexing problems appear. Click on that tab to see a breakdown of every reason Google is failing to index pages on your site.
Google organizes these issues into specific reason categories. Here is what each one means in plain language:
Not found (404): Google tried to visit a URL on your site and received a "page does not exist" response. This often happens when content is deleted, a URL is changed without a redirect, or there is a typo in an internal link. These are among the most common crawl errors on any site.
Server error (5xx): Google reached your server but the server failed to respond properly. This usually points to a hosting issue, an overloaded server, or a misconfigured plugin or application. Unlike 404 errors, these are not about missing content. They are about your server's ability to respond at all.
Blocked by robots.txt: Your robots.txt file contains rules that are telling Googlebot not to access certain pages. Sometimes this is intentional (blocking admin pages, for example), but it is also a common source of accidental blocking, especially after a site migration or redesign. Understanding how to manage these rules is part of broader Googlebot optimization techniques that keep your site crawlable.
Redirect error: Google followed a redirect and encountered a problem, such as a redirect chain that is too long, a redirect loop where pages redirect back to each other, or a redirect pointing to a URL that itself returns an error.
Crawled - currently not indexed: Google visited the page and was able to read it, but chose not to include it in the index. This is usually a content quality signal. The page may be thin, duplicative, or not considered useful enough to rank.
Discovered - currently not indexed: Google knows the page exists (perhaps from your sitemap or an internal link) but has not crawled it yet. This can indicate a crawl budget issue or simply that the page is very new.
Soft 404: The page returns a 200 "success" status code to Google, but the content on the page is essentially empty or nearly identical to a "page not found" message. Google treats these as functional 404s even though the server says everything is fine.
As you review the list, sort by the number of affected pages to see which error types are causing the most widespread problems. A single error type affecting dozens of pages deserves attention before an isolated issue affecting one or two URLs.
Success indicator: You have a clear list of error types and know how many pages each one affects. You can see the scale of each problem before deciding where to focus your energy.
Step 3: Run a Complementary Site Crawl with a Dedicated Tool
Google Search Console tells you what Google has already found and reported. But it does not catch everything. A dedicated site crawling tool fills in the gaps by simulating how a search engine spiders your site in real time, often surfacing issues that Search Console has not yet flagged.
Three tools worth knowing:
Screaming Frog SEO Spider: Free for up to 500 URLs. This is the most widely used option among SEO professionals and works well for small to mid-size sites. You download it as a desktop application, enter your domain, and it crawls every page it can find.
Sitebulb: A paid tool with a strong visual interface that makes it easier to interpret results without a technical background. Particularly useful if you want guided recommendations alongside the raw data.
Ahrefs Site Audit: Part of the Ahrefs subscription, this cloud-based crawler runs automatically on a schedule and integrates with other SEO data like backlinks and keyword rankings. A good option if you are already using Ahrefs for other purposes.
To run a basic crawl with Screaming Frog, open the application, paste your domain URL into the search bar at the top, and click Start. The tool will begin spidering your site the same way a search engine would, following internal links from page to page.
Once the crawl is complete, focus on these specific areas within the results:
Broken links (4xx errors): Look at the Response Codes tab and filter for 4xx responses. These are pages that returned a "not found" error when the crawler attempted to visit them.
Server errors (5xx): Also visible in the Response Codes tab. These indicate server-level failures that need immediate attention.
Redirect chains: Under the Redirects tab, look for chains where multiple redirects fire in sequence before reaching the final URL. These slow down crawling and dilute link equity.
Orphan pages: Pages that exist on your site but have no internal links pointing to them. Search engines may never find these pages because there is no path leading to them. Screaming Frog can identify these when you upload your sitemap alongside the crawl. If you want to learn more about connecting your pages effectively, our guide on internal linking strategy for small businesses covers the fundamentals.
Pages blocked by robots.txt: The tool flags these so you can cross-reference with what you found in Search Console.
The value of this step is that it gives you a second, independent data set. Where Search Console reflects Google's historical experience with your site, a fresh site crawl reflects the current state of your site right now.
Success indicator: You have a second data set that complements your Search Console findings, giving you a fuller picture of what needs to be fixed.
Step 4: Categorize and Prioritize Your Crawl Errors
Here is where many business owners get stuck. They open their crawl report, see dozens or hundreds of errors, and either feel overwhelmed or start trying to fix everything at once. Neither approach is efficient.
Not all crawl errors carry the same weight. A 404 error on a high-traffic page with backlinks pointing to it is a serious problem. A 404 on a test page that was never published and never received a single visitor is essentially harmless.
The most practical approach is to organize your errors into three priority tiers. You can do this in a simple spreadsheet or use the filtering features built into your crawl tool. This kind of structured prioritization is a key part of knowing how to audit your website SEO effectively.
Critical (Fix Immediately): Server errors (5xx) affecting any page. Broken links or 404 errors on pages that receive meaningful traffic or have backlinks pointing to them. Important pages (homepage, service pages, product pages) that are blocked by robots.txt or returning redirect errors. These errors directly impact your ability to rank and your users' ability to navigate your site.
Moderate (Fix Within the Month): Redirect chains involving key pages. Soft 404s on secondary content pages. Orphan pages that contain content you want indexed but have no internal links pointing to them. These issues do not cause immediate ranking drops but compound over time if left unaddressed.
Low Priority (Monitor or Ignore): 404 errors on old URLs that were intentionally removed and had no traffic or backlinks. Pages marked as "Discovered - currently not indexed" that are very new. These can often be left alone or addressed during routine maintenance.
To determine where each error falls, cross-reference your error list with two data sources. First, check Google Analytics to see if the errored URLs received any traffic in the past three to six months. Second, run the URLs through a backlink checker (Ahrefs, Moz, or even Google Search Console's Links report) to see if any external sites are linking to those broken pages.
A URL that received traffic and has backlinks pointing to it gets elevated to critical, regardless of what type of error it is.
Success indicator: You have a prioritized action list with the most impactful errors at the top, organized in a way that lets you work through them systematically rather than randomly.
Step 5: Fix the Most Common Crawl Errors
With your prioritized list in hand, it is time to resolve the errors. Here is how to approach each of the most common types:
404 Not Found Errors
Start by determining why the page is missing. Did the URL change? Was the content deleted? Is there a typo in an internal link pointing to a non-existent URL?
If the page moved to a new URL, set up a 301 redirect from the old URL to the new one. A 301 tells both users and search engines that the page has permanently moved, and it passes most of the original page's authority to the new destination.
If the page was removed by mistake, restore it. If the page is genuinely gone and had no meaningful traffic or backlinks, you can leave it. Google will de-index it naturally over time. Do not create a redirect to an unrelated page just to clear the error. That creates a poor user experience and Google can detect it.
5xx Server Errors
These are hosting and infrastructure problems, not content problems. Start by contacting your hosting provider and describing the error. Ask them to review your server logs for the time periods when the errors occurred.
Common causes include exceeding your hosting plan's resource limits during traffic spikes, a plugin or application consuming too much memory, or a misconfigured server setting after an update. If 5xx errors are appearing frequently, it may be a sign that your current hosting plan is no longer adequate for your site's needs.
Blocked by robots.txt
Visit yourdomain.com/robots.txt in your browser to view your current robots.txt file. Look for "Disallow" rules that might be blocking pages you actually want Google to crawl.
A very common mistake: during website development or migration, developers set the robots.txt to block all crawlers to prevent the staging site from being indexed. If that setting carried over to the live site, you may have been accidentally blocking Google from your entire website. This is more common than most people realize. If your site has been invisible in search results because of this, our guide on why your website isn't showing up on Google covers additional causes and solutions.
If you find a rule that should not be there, remove it and save the updated file. You can test your robots.txt directly in Google Search Console using the robots.txt tester under the Settings section.
Redirect Chains and Loops
A redirect chain occurs when Page A redirects to Page B, and Page B redirects to Page C. Every hop in that chain adds load time and dilutes the link equity being passed. The fix is to update the original redirect so it points directly to the final destination URL, cutting out the middle steps.
A redirect loop occurs when pages redirect to each other in a circle, causing both browsers and crawlers to get stuck. These need to be identified and broken by pointing at least one URL in the loop to a real, final destination.
Soft 404s
Review each page flagged as a soft 404. If the page has thin or placeholder content, either add meaningful, substantive content to justify its existence or redirect it to a relevant page that does have real content. If the page was meant to be a "not found" page, configure your server to return an actual 404 status code rather than a 200.
Success indicator: Your highest-priority errors are resolved and you have documented what you changed, including the original URL, the error type, and the fix applied. That documentation will be valuable when validating fixes in the next step.
Step 6: Validate Your Fixes and Request Re-Crawling
Fixing errors on your end is only half the process. You also need to confirm that Google recognizes the fixes. Do not assume the work is done just because you made changes on your site.
Return to Google Search Console and navigate back to Indexing > Pages. Click on the specific error type you addressed. At the top of that report, you will see a "Validate Fix" button. Click it to notify Google that you have resolved the issues in that category and ask Googlebot to re-check those URLs.
Validation is not instant. Depending on your site's size and how frequently Google crawls it, the process can take anywhere from a few days to a couple of weeks. Search Console will update you on the validation status as Google works through the affected URLs. For a deeper look at what to do when pages remain stubbornly out of the index, see our walkthrough on how to fix pages not indexed in Google.
For individual pages that are high priority, such as a key service page or a top-traffic landing page, use the URL Inspection tool in Search Console. Enter the specific URL, and once the inspection loads, click "Request Indexing." This prompts Google to crawl that specific page sooner than it might otherwise get to it.
While waiting for Google's validation, re-run your third-party site crawl from Step 3. This gives you an immediate, independent confirmation that the fixes are working at the technical level before Google's validation completes. If the same errors still appear in your fresh crawl, something in the fix did not take hold and needs to be revisited.
A common mistake at this stage is checking once and moving on. Check back in Search Console after a week and again after two weeks to confirm the error counts are decreasing and that previously affected pages are moving into "Indexed" status. Keep in mind that SEO takes time to show results, so patience during validation is essential.
Success indicator: Search Console shows the error count decreasing for the categories you fixed, and previously errored pages are appearing in the Indexed section of the Pages report.
Step 7: Build a Monthly Crawl Error Monitoring Routine
Crawl errors are not a one-time problem you fix and forget. They are an ongoing reality of running a website. Every time you publish new content, delete old pages, restructure your navigation, or update plugins, you create the potential for new crawl errors to appear.
The businesses that maintain strong search visibility over the long term are the ones that treat technical SEO as a recurring maintenance task, not a one-time project.
Set a monthly calendar reminder to complete two tasks: check the Search Console Pages report for new errors and run a fresh site crawl with your chosen tool. The entire process should take 30 to 60 minutes once you are familiar with the workflow. That small time investment prevents errors from compounding quietly in the background while your search rankings slowly erode.
Pay extra attention to crawl errors in the weeks following any significant site changes. A redesign, a platform migration, a URL restructure, or even a major content update are all high-risk moments for introducing new crawl issues. Running a crawl immediately after these events can catch problems before Google has a chance to process them.
If you want to reduce the manual burden, AI-powered monitoring tools and automated alert systems can flag new crawl errors as they appear, sending notifications when something breaks rather than waiting for your monthly check. Many SEO platforms now include automated crawl monitoring as a standard feature, which is particularly useful for business owners who are managing multiple priorities simultaneously.
Think of crawl error monitoring as one component of a broader SEO strategy built from scratch. Alongside monitoring crawl errors, a healthy site also requires attention to page speed, mobile usability, structured data, and internal linking. Addressing these areas together creates the kind of clean, crawlable site that both traditional search engines and AI-powered search systems can access and reference accurately.
Success indicator: You have a recurring monthly process that catches new crawl errors before they compound into larger visibility problems, and you are addressing issues proactively rather than reactively.
Your Quick-Reference Checklist and Next Steps
Here is a condensed version of the full process you can return to each month:
1. Open Google Search Console and navigate to Indexing > Pages. Review the Not Indexed tab for new error categories.
2. Note the error types and volume. Identify which categories have the most affected pages and which pages are most important to your business.
3. Run a fresh site crawl using Screaming Frog, Sitebulb, or Ahrefs Site Audit. Export the results and cross-reference with your Search Console findings.
4. Prioritize errors into critical, moderate, and low tiers based on traffic, backlinks, and page importance.
5. Fix errors by type: set up 301 redirects for moved or deleted pages, contact your host for server errors, review your robots.txt for accidental blocks, and clean up redirect chains.
6. Validate fixes in Search Console using the Validate Fix button, and use the URL Inspection tool for high-priority individual pages.
7. Re-run your site crawl to confirm fixes are working technically before Google's validation completes.
8. Repeat monthly and schedule an immediate crawl after any major site changes.
Crawl errors are one of the most overlooked factors in search performance, precisely because they are invisible to the naked eye. Your site can look completely functional while Google is quietly being blocked from half your pages. Building this routine into your monthly operations is one of the most practical things you can do to protect and grow your organic visibility over time.
If you would like expert help building a technical SEO foundation that supports consistent rankings and lead generation, the team at North Valley Solutions works with business owners to create scalable systems that turn website traffic into booked customers. Rank higher and grow by scheduling a quick call to see how we can help.