Black Friday 2026 50% off every plan Our biggest deal of the year on #1 Joomla products.
Ends in 11d 22h 26m 57s
Claim 50% off
Staff replied SP Page Builder

Unwanted URLs being crawled despite Disallow in robots.txt

Asked by Sascha 2 years ago Last activity 2 years ago

Hello everyone,

I have a technical question regarding the crawling of specific URLs on my website.

The situation:
The main URL of my website is, for example, https://homepage.de. However, during crawling, automatically generated URLs are also picked up, which I would like to exclude. One example of these unwanted URLs is: https://homepage.de/de/component/sppagebuilder/page/1.

What I’ve tried:
To prevent these URLs from being crawled, I added the entry Disallow: /components/ in my robots.txt file. Despite this, these links continue to be crawled.

My question:
Is there a reliable way to prevent these links from being crawled, even though they’re excluded in the robots.txt? Would any other methods or configurations be needed?

Thanks in advance for any help!

1 reply

Paul Frankowski Senior Staff
2 years ago · edited

HI Sascha.

Indeed, as you noticed sometimes Google index links that we (webmasters) don't want to. I have similar problem but with different component, brrr.
Why it happends, hard to say in all cases. I don't work in Google to be sure.

  1. I would search in database (use PHPMyAdmin in cPanel) if you used in Page, Article or Module (even by mistake) that phrase "component/sppagebuilder/page". If yes, correct link, update with short one.

  2. Then in Google Search Console > Removals > New Request > add one by one, add all links with long URL.

  3. Check also sitemap - maybe that link was included already. If yes, remove , upload and then resend to Google for reindex.

  4. Later wait 2-7 days for Google response.

Log in to reply.