A page can be public, crawlable, and still not be the URL you want displayed in search. That is why robots.txt, noindex, and canonical links should not be treated as interchangeable switches.
Before changing any of them, describe the desired outcome in ordinary language. “This private document must not be accessible” is very different from “These two public URLs show the same article.”
Robots.txt addresses crawling
Robots.txt tells cooperating crawlers which paths they may request. It is not a password system and does not protect confidential information. If a page must remain private, restrict access with appropriate authentication.
Blocking crawling is also not a dependable way to remove a URL from search. A search engine may learn about an address through links without being able to read its contents. Google explains this distinction in its robots.txt introduction.
For a small business site, review broad rules with care. A rule covering an entire directory could unintentionally block service pages or articles you want discovered.
Noindex addresses search inclusion
A noindex instruction tells supporting search engines not to include the page in their index once they can read and process the instruction. It may appear in page markup or an HTTP header.
This creates an important dependency: if you block the crawler from accessing the page, it may not see the noindex instruction. Ask the developer to test the combined behavior rather than checking each setting separately.
A public thank-you page might intentionally be excluded. A new service page accidentally carrying a staging noindex setting is a different case. Keep a list of intentional exclusions so future maintenance does not undo them.
Canonical identifies a preferred version
A canonical link suggests which URL should represent duplicate or very similar content. It is a signal rather than a guarantee that Google will choose that address. It does not make unrelated pages equivalent and does not stop people from visiting another version.
Use an absolute production URL and check it after a domain move. Google’s canonical documentation describes how canonical signals interact.
A decision check before editing
- Is this content private? Restrict access; do not rely on search settings.
- Should this public page be excluded from search? Review noindex and crawler access together.
- Is this a duplicate public URL? Review canonical signals and internal links.
- Has the page permanently moved? Plan a relevant redirect.
After any change, inspect the live response and record the intended result. If discovery is the underlying issue, continue with the new website troubleshooting guide rather than experimenting with multiple settings at once.