cerez.io crawler

This crawler is used by cerez.io customers to run technical SEO scans of their own sites. A scan starts only from the site owner's account, and only on sites whose ownership is verified in the cerez.io panel.

Identity

The raw HTML crawl uses this user agent:

cerez.io-seo/1.0 (+https://cerez.io/bot)

Sampling, in which a small number of selected pages are opened with JavaScript, carries HeadlessChrome and cerez.io-seo/1.0 in its user agent. robots.txt group name: cerez.io-seo

Rules

• Follows robots.txt: applies the group for its own name if there is one, otherwise the User-agent: * group. Honours Crawl-delay.
• The default rate is 1 request per second. The site owner can raise it to at most 4 requests per second on a site with verified ownership.
• If the site returns 429 or 503, has repeated server errors or responds clearly slower, the rate is halved automatically.
• Makes only GET and HEAD requests; does not submit forms or log in, and records redirects step by step.
• The number of pages per scan is limited by the site owner's plan.

Blocking

To block the crawler on your whole site, add this to your robots.txt:

User-agent: cerez.io-seo
Disallow: /

Contact

For any issue with the crawler, write to us: destek@cerez.io