HTTP vs HTTPS and WWW vs Non-WWW: Crawl Examples
My coach stands at both the front and the back of the hedge, and at neither will I tell you which is the real one.
The short answer
- What it is
- HTTP sends data in plain text, and HTTPS encrypts it with TLS. A site with both, or with www and non-www versions, can have four addresses for every page.
- Why it matters
- If every version answers with a page, search engines see duplicates and must choose one. Visitors on the HTTP version get no encryption and a browser warning.
- How a crawler should report it
- Fetch all four host and protocol versions, and report each one's status and final URL. Warn when more than one answers 200, when a redirect is a 302 instead of a 301, or when a version takes more than one hop.
- Live examples
http://crawltestlab.com/, which answers without a redirecthttps://www.crawltestlab.com/, a twin host with an apex canonical/lab/canonical-host-variant/(positive), on the second shelf/lab/insecure-links/(positive)
Four Ways to Reach One Home Page
HTTP vs HTTPS comes down to protection in transit. HTTP sends requests and pages as plain text, so anyone on the network path can read or change them. HTTPS wraps the same traffic in TLS. That gives encryption, integrity, and proof of the server's identity through a certificate. HTTP uses port 80 by default, and HTTPS uses port 443. Browsers also need HTTPS for HTTP/2 and for many newer features.
Every site can also be reached with or without www. Add both protocols, and you have four start points: http://example.com/, http://www.example.com/, https://example.com/, and https://www.example.com/. Neither the www nor the non-www form is better for rankings. The goal is to pick one and stick to it. Problems start when a server answers on more than one, or when a new certificate does not cover both host names.
Browsers mark HTTP pages as not secure. Google confirmed HTTPS as a light ranking signal in 2014, and it prefers to index the HTTPS version when both exist and the secure page works. Search engines treat each of the four forms as a different URL. If they all load, signals get split until one is chosen. The choice may not match yours. HSTS tells browsers to use HTTPS from then on, but it does not replace redirects.
To test your setup, request all four versions and record the status and final URL of each. Exactly one should answer 200. The other three should reach it in a single 301 hop. Avoid chains such as HTTP to HTTPS to www. To fix it, add a server or CDN rule that sends every variant straight to your preferred URL. Set self-referencing canonical tags on that host. Update internal links and sitemaps too. Check that your certificate covers both names.