It is important to diagnose potential technical issues to keep a website running. This process used to be time-consuming. Google Search Console allows inspecting only one URL at a time.
Since 2022, this process has become smooth. This is when Google Search Console URL Inspection API was tested and launched.
What is the URL inspection API in Google Search Console?
Google Search Console URL Inspection API allows users to check bulk URLs present in the Search Console.
You can inspect multiple URLs through the console. The results can be exported in Excel or a CSV file. The URL inspection API provides a convenient way for developers to debug and optimize pages.
Which data types can the Google Search Console URL Inspection API provide?
You can get a wide range of data from Google’s URL inspection API:
Sitemap
You can pull out data and information to find which URLs form part of the sitemaps.
RobotsTxtState
You can check whether any of your robots.txt rules prevent Googlebot from crawling your website. A manual search can do this. Conducting the testing at a scale and using Google’s data can add leverage and precision to your testing process.
Referring URLs
The referring Urls data shows the linkage of each page as per Google.
PageFetchState
You can diagnose server errors and other issues using the pageFetchState field. A list of such errors and corresponding responses is given below:
| Field | What Does It Imply |
| PAGE_FETCH_STATE_UNSPECIFIED | Unknown fetch state |
| SOFT_404 | Soft 404 |
| NOT_FOUND | Not found (404) |
| SUCCESSFUL | Successful Fetch |
| BLOCKED_ROBOTS_TXT | Blocked by robots.txt |
| ACCESS_DENIED | Blocked due to unauthorized request (401) |
| SERVER_ERROR | Server error (5xx) |
| ACCESS_FORBIDDEN | Blocked due to access forbidden (403) |
| BLOCKED_4XX | Blocked due to other 4xx issues (other than 404,403) |
| INVALID_URL | Invalid URL |
| INTERNAL_CRAWL_ERROR | Internal error |
| REDIRECT_ERROR | Redirection error |
LastCrawlTime
This data could only be accessed only through log file analysis or when someone spot-checked individual URLs using Google Search Console. SEOs can now find out when Googlebot crawled on their website last with this field. It helps in understanding the frequency of such crawls on your site.
GoogleCanonical and userCanonical
Sometimes, there are chances that Google may select a different canonical than what is specified in the code. If such a situation arises, you can compare Google Canonical and user Canonical and make relevant changes.
Crawled As
The crawled AS field is useful for checking the kind of user agent that crawled your website. It could be a mobile or a Desktop agent. The response codes for both are:
- DESKTOP for Desktop user agent
- MOBILE for mobile user agent
Indexing State
Indexing state field can be used to know your URL’s current indexation status. The Inspection API provides other responses beyond Pass and Fail. Some other responses are:
- NEUTRAL: This response is the same as the ‘Excluded’ message in Search Console.
- PARTIAL: This response equates to the ‘Valid with warnings’ message on the Search Console.
- VERDICT_UNSPECIFIED: This response pops up when Google cannot conclude the indexation status of the URL/s in question.
Coverage State
You can find out about a particular URL being submitted and indexed in your sitemap.
Other uses for the API
You can inspect it using Google’s URL inspection API if you have an AMP site.
Step-by-step instructions for using the Google Search Console URL inspection API.
We enumerated several functionalities of using Google’s URL inspection API. Let’s now focus on the steps to use the Google Search Console URL Inspection API.
- Begin by requesting Google for the URL inspection. You must start inputting the request parameters. This should include the URLs you want to inspect and the URL of the property in the Console. The body of the request would follow the structure shown in the image:
Google has extensively documented information on how to use the API. You can use it to learn more about the API usage.
- The response that you get from Google API would resemble something that is shown below:
- You can avoid getting into the technicalities of the code and use the free Google Bulk Inspect URLs tool from valentin.app. The tool is a great option for people not comfortable with code to query the API without using any coding.
- Go to https://valentin.app/inspect.html and give it authorized access to your Google account. Next, choose the Search Console property you want to begin testing.
- Whatever URLs you want to test, paste them in the box. The data will not be shared with anyone or uploaded on a server. It will be processed right on your browser only.
- Click on the ‘Inspect URL’ button, and the data will begin pulling itself from the API.
- Once done, export the results on an Excel sheet or a CSV file.
- Go through the data and check for any potential technical issues.
In what ways can you make use of the Google Search Console URL inspection API?
Using the URL inspection API seems like a fantastic idea. This can help you understand and test your website. The drawback is that analyzing the data you have pulled out will be difficult.
However, here are a few use cases for the same:
Site migration – resolving any technical issues
During site migrations, several issues crop up. For example, developers may mistakenly block Google from crawling your website or pages through robots.txt. You can save this situation by using Search Console URL inspection API to bulk test robot.txt by calling robotsTxtState.
By doing so, you come to know which robots.txt is disallowing Google from crawling your website/pages and fix them immediately.
Note if Google has Respected your declared canonical
Check if Google accepts the canonical tag changes once you have implemented them on your website.
Google can ignore your canonical tag for reasons such as:
- The declared canonical tag may not be HTTPS – the format Google prefers for canonicals.
- There are chances that Google may opt for a different page, believing it to be a better canonical page than the one you declared.
- The canonical you declared is a noindex page
You can export the data from the Inspection tool to a CSV file and use an IF formula. If your googleCanonical data is in the first column and the userCanonical data in the second column, put this formula to find canonicals that don’t match:
=IF(A2=B2, “Self referencing”,”Non Self Referencing”)
Note when Google recrawls after you make changes to your site
You should know the impact of the changes and updates you have made on your website.
It may be a possibility that you may not be able to access your website’s log files. If that happens, wait for Google to recrawl your site.
Once this happens, you can use the lastCrawlTime field in Google Search Console URL inspection API to note the exact time at which Googlebots crawled your website. This information is also useful to gauge the impact of updates made to the website.
Conclusion
Now that you know what is URL inspection and how to conduct one using Google’s URL inspection API, testing your website for issues will be a breeze. Google limits querying the URL inspection API to 2,000 queries per day. But if your properties are verified separately in Google Search Console, you can bypass this limit.
The URL Inspection API is a great way for developers and SEOs to scale their testing and attain bulk data pulled from Google. This data can be employed in multiple use cases to optimize your website.
Popular Searches
