Does it seem like Google is painfully slow at indexing your site, especially the inside web pages? There might be good reason to look into why your website isn’t being crawled, indexed, and cached as frequently.
1) Google Will Never Index All Your Pages
Plain and simple: Google Webmaster Trends Analyze John Mueller has publicly stated that Google will “never index all pages”. As disheartening as that sounds, there are just some things that are beyond your control. In the mean time, take a look at all the back end things that are within your control to make sure that you are doing everything possible to ensure the proper indexing of your website.
2) Check Your robots.txt File and Meta
Sometimes, after a redesign or website changes, funny things can happen to your robot.txt file or meta tags. This is especially true for WordPress sites where “NOFOLLOW,NOINDEX” can easily be applied to all or some of your web pages. Make sure that these options are disabled and that pages are set to FOLLOW,INDEX.
3) Domain or Partial Domain Penalty
If your website or some web pages have a penalty, the consequence is that they won’t show up or be indexed by Google. A good place to check for that is in the Manual Actions section in Google Search Console. This is extremely helpful for manual penalties but sites facing algorithmic penalties may have a harder time. If you still suspect some type of website penalty, check for duplicate and thin content.
4) Is Your Website Hacked or Infected?
Search Console is also a good place to check to see if Google thinks your site has been infected. If this is the case, Google has a legitimate reason to not index your website to prevent users from harm from the malicious content.
5) Not Enough Signals to the Web Page
If you have a really good, informative web page, it’s probably not the best idea to just add it to your website and have that be the end of things. Google will surely never see or find it that way. That is why it’s important to keep an updated sitemap and sitemap.xml file, to make it easier for Googlebot to crawl. Linking from the home page and relevant places in the navigation is also important. Aside from linking on your own website, it’s also necessary to link elsewhere: social media websites, third party review sites, relevant directories, and niche listings. The more helpful a website or web page is, the more likely it is that it will be linked to from a diverse amount of other websites.