Short answer: An automated cookie scanner gives you breadth across many pages and a dated baseline you can re-run. A manual pass gives you what a crawler structurally cannot reach: tags that fire only after consent, storage set behind a login or during checkout, region and experiment variants, event-triggered scripts, and non-cookie storage such as localStorage, IndexedDB and pixels. A defensible inventory needs both, on a schedule, with a last-verified date on every row.
Most teams find the gap the same way. The scanner reports eleven cookies, the cookie policy lists eleven, then somebody opens DevTools on the checkout page and finds four more plus a dozen undocumented localStorage keys. Nothing was misconfigured; checkout was never part of what a crawler could do.
Note: This article is general information, not legal advice. Classification decisions, especially whether a cookie is strictly necessary, depend on your setup and should be reviewed with qualified counsel.
Table of contents
- Why the inventory, not the banner, is the foundation
- What an automated cookie scanner actually does
- The five blind spots automation has by design
- The two-run method: reject, accept, then diff
- The manual pass in eight steps
- The inventory fields you actually need
- A repeatable workflow that combines both
- From inventory to policy and banner
- How the cerez.io scanner is built and what it does not do
- Frequently Asked Questions
- Sources
Why the inventory, not the banner, is the foundation
Teams start with the banner because the banner is visible. The inventory is what carries the weight.
Accountability sits on the operator, and it attaches to storage rather than to the word "cookie". GDPR Article 5(2) makes the controller responsible for demonstrating compliance with the processing principles, and Article 7(1) requires it to demonstrate that consent was given; both assume you can state what is being set, by whom, and for what. The ePrivacy Directive (2002/58/EC) Article 5(3) then attaches that requirement to storing or accessing information in a user's terminal equipment, which is technology neutral: localStorage writes, IndexedDB records, service worker caches and SDK identifiers all sit inside it even though none is a cookie. An inventory built only from Set-Cookie headers misses part of the regulated surface by definition. If you serve Turkish traffic, the Turkish DPA's 2022 Guideline on Cookie Applications starts in the same place.
Everything downstream inherits from it. The cookie policy is a rendering of the inventory, the banner categories are a grouping of it, and the blocking rules are keyed to those categories. Get the inventory wrong and the rest is quietly wrong with it, through every banner redesign.
What an automated cookie scanner actually does
The blind spots are easier to see once you know the pipeline. Almost every scanner runs the same five stages.
- Crawl. Start from a seed URL, keep same-host links, queue them, stop at a page budget. Breadth-first is common because it reaches many templates quickly.
- Load. Fetch each page. Some scanners read the markup over plain HTTP; others drive a headless browser so client-side JavaScript executes. That single choice explains most of the accuracy difference between tools.
- Observe storage. Record
Set-Cookieresponse headers and, when a browser is actually running, readdocument.cookieand sometimes the other storage areas once the page settles. - Match scripts against a signature database. Compare every
<script src>and inline snippet against known provider fingerprints. This is how a scanner reports a provider whose cookie never appeared during the crawl. - Classify. Match each name against a known-cookie database for purpose, provider, duration and a proposed category, then fall back to pattern rules, then leave the rest unrecognized.
A scanner is an excellent inventory generator and a poor inventory validator, and those five stages are where a human takes over.
The five blind spots automation has by design
These are not bugs, and no vendor fixes them by trying harder. They follow from what a crawler is.
1. Consent-gated tags. If your site is doing the right thing, non-essential scripts do not run until the visitor chooses, and a default crawl never chooses. The better your blocking works, the emptier your scan looks, and teams misread that as "we only have four cookies".
2. Authenticated and transactional areas. Account dashboards, checkout steps, payment iframes, support chat after identification. These carry the densest storage on the site, and an anonymous crawler reaches none of it.
3. Geo and experiment variants. Tag configurations branch on region, consent region logic, A/B assignment or personalization rules, and a crawler arrives from one IP address, with one user agent, in one bucket. If you serve both EU and non-EU traffic, the comparison of KVKK and GDPR consent requirements covers what those branches need to differ on.
4. Event-triggered storage. Tags that fire on scroll depth, add to cart, video play, form submission, exit intent or a delayed timer. A crawler loads the page and leaves, so anything waiting for a human never fires.
5. Non-cookie storage. localStorage, sessionStorage, IndexedDB, Cache Storage, service workers, and pixels carrying an identifier in the request URL without setting a cookie. ePrivacy Article 5(3) covers this ground, but plenty of tools report only cookies, and a missing row is a missing policy line.
The two-run method: reject, accept, then diff
One scan always produces the wrong number. Not an inaccurate number, the wrong kind of number, because a single run cannot tell you whether a cookie is absent or merely waiting.
Run the same page set twice, from a clean browser profile each time. In Run A you reject all optional categories and walk the pages: whatever appears is what your site sets without consent. In Run B you clear the profile, accept all of them, and walk the identical pages: that is your full disclosure surface. Then diff the two lists.
| Result | What it tells you | What to do next |
|---|---|---|
| Present in Run A and Run B | Fires regardless of consent | Justify it as strictly necessary with a documented rationale, or fix the blocking |
| Absent in Run A, present in Run B | Correctly gated behind consent | Document it in the policy under the category that unlocked it |
| Present in Run A, non-essential in nature | Blocking failure | Trace the injection point, usually a hardcoded tag or a trigger with no consent condition |
| In neither run, but reported by the scanner | Provider detected in markup, never fired | Check whether it fires on an event, a variant, or a page you did not walk |
The third row costs teams the most: usually a tag pasted straight into a template outside the tag manager, so no consent condition applies.
On "strictly necessary": resist the urge to let a scanner decide this. The classification is assessed case by case by the controller, and supervisory authority practice varies by country, so a tool's default label is a hypothesis, not a verdict. Document three things per candidate: is the storage required for a service the user explicitly requested, would that service break without it, and is it used only for that purpose rather than reused for analytics.
The manual pass in eight steps
Budget two to four hours for a mid-sized site. It is a checklist, not a research project.
1. DevTools Application panel, per storage type. Read Cookies, Local Storage, Session Storage, IndexedDB, Cache Storage and Service Workers separately, on five to ten representative templates rather than every URL.
2. Network tab, filtered to third-party hosts. Sort by domain and look for hosts you do not recognize, including requests that set nothing: pixels carrying an identifier in the query string never appear in a cookie list.
3. A logged-in run. Repeat steps 1 and 2 while authenticated, including a full add-to-cart and checkout flow in test mode. This is where the densest and least documented storage usually is.
4. A mobile-viewport run. Some widgets, app install banners, chat SDKs and ad units load only under mobile conditions; emulation catches most of it.
5. Tag manager review, tag by tag. List every tag with its trigger and consent setting, then compare that list against what the crawl found. Any tag with a rare trigger is invisible to every scanner you will run.
6. Vendor reconciliation. Every third-party host should map to a named vendor, the contract appropriate to that relationship (a processor agreement, or a controller-to-controller or joint-controller arrangement), and a documented basis for any transfer outside your jurisdiction.
7. Sample the long-lived cookies. Sort by duration and inspect anything unusually long or unclear in purpose. Give each an owner and a written justification, or remove it.
8. Date and file the findings. Record the date, the scope walked, who ran it, browser state and tool versions, and keep the raw evidence. An undated audit is an anecdote.
The inventory fields you actually need
Most inventories are a three-column table of name, purpose and duration. That is enough to render a policy and not enough to defend one.
| Field | Why it belongs in the row |
|---|---|
| Name | The identifier exactly as stored, including any prefix |
| Host / domain | Shows first-party versus third-party at a glance |
| Provider and transfer destination | The named vendor and where data goes, linking to the contract and transfer record |
| Storage type | Cookie, localStorage, sessionStorage, IndexedDB, pixel, service worker |
| Purpose | One plain sentence, not a category name repeated |
| Category | Necessary, functional, analytics, marketing, and so on |
| Duration | Session, or the real expiry taken from the attribute rather than the vendor's docs |
| Set by | The script or tag that writes it, which is what you fix when it misbehaves |
| Where observed and how | Template or URL, plus whether the row came from a scan or a manual check |
| Consent run | Whether it appeared in the reject run, the accept run, or both |
| Basis note | The short rationale, especially for anything claimed as strictly necessary |
| Last verified and owner | The date a human last confirmed the row, and who can answer questions about it |
Last verified is the field reviewers ask about and most inventories lack. A "first seen" date tells you when a cookie appeared; only "last verified" tells you whether the document describes the site as it is today or two redesigns ago. Any row older than your review cycle is a task, not a record.
A repeatable workflow that combines both
The goal is a loop, not a project. Match scan cadence to release frequency: ship weekly, scan weekly; ship quarterly, monthly is fine. But the events that add tags are not evenly spread across the calendar.
| Trigger | Action |
|---|---|
| New marketing campaign or agency access granted | Scan within 48 hours, review new third-party hosts |
| Tag manager container published | Scan, plus a tag-by-tag review of what changed |
| New vendor, plugin, theme or SDK | Scan, plus vendor reconciliation for the new host |
| Site redesign or template change | Full scan plus a manual pass on the changed templates |
| Quarterly, regardless of changes | Full manual pass, refresh every last-verified date |
Use scan-diff alerts rather than re-reading full reports. Reading a fresh 40-row list every week is work nobody sustains; surfacing only the delta takes two minutes. A new marketing tag should reach you as a change notification, not as a row you might notice.
Keep the quarterly manual pass on the calendar, and regenerate the policy from the inventory. Automation tells you something changed; the manual pass tells you whether the change is acceptable. Hand-editing the published policy is how drift starts.
From inventory to policy and banner
Three artifacts should be derived from one table, in this order.
Rows to categories. Each row gets exactly one category. The mapping is mechanical once purposes are written honestly, and it is where unrecognized rows must be resolved rather than parked: an uncategorized cookie cannot be blocked or disclosed correctly.
Categories to blocking rules and platform signals. Every non-essential script is tagged with its category and held until the visitor chooses. This is where the reject run pays off: anything that appeared in Run A has a blocking rule that is missing or attached to a tag bypassing the tag manager. A consent management setup enforces the rule; the inventory tells it which scripts to hold. The same categories map onto Google Consent Mode v2 signals such as analytics_storage and ad_storage, and mapping from the inventory rather than from memory keeps the signal aligned with the storage you set.
Rows to policy text, then a drift check. The published table should list name, provider, purpose and duration straight from the inventory, and a cookie policy generator that reads the inventory keeps the two in sync; a hand-maintained policy diverges within a quarter. Once a quarter, verify every policy row still exists in the inventory and every banner category still has a row behind it. Per-category consent analytics is how you notice early that a category no longer corresponds to anything you run.
How the cerez.io scanner is built and what it does not do
The free public scanner. Our cookie scanner takes a URL without an account and fetches the homepage plus up to two internal pages on the same host. Results are cached for 24 hours per URL, and the endpoint is rate limited to five scans per hour per IP address. It runs without a headless browser, fetching pages over HTTP and reading response headers and markup, so cookies written by JavaScript at runtime do not appear in it. The same fetch also runs seven surface-level accessibility checks. Anonymous visitors see the first five cookies and the first three accessibility findings; leaving an email sends a link to the full result, and a free account keeps a history instead of a one-off page. It is a first look, not an inventory.
The account scan. Signed-in scans crawl breadth-first, following same-host links up to a page budget that depends on the plan, and add two things the public tool does not: third-party script signature analysis across 16 provider fingerprints, and classification against a 217-entry known-cookie database. Cookies derived from signature analysis rather than an observed Set-Cookie header carry an inferred flag, so you can tell a hypothesis from an observation. Scans run on a schedule (daily, weekly or monthly depending on plan), and each is compared with the previous scan, so new cookies surface as a notification and an email.
How classification works, including where it guesses. Classification runs in four stages: exact name match, prefix match and regex match against the known-cookie database, then a keyword fallback that infers a category from the name itself. That fallback is a hypothesis, not a finding, and it can label an unrecognized name as necessary purely because the name contains a string like session, token or auth. Only names that fail every stage are left uncategorized. Review every auto-assigned category, starting with the ones labeled necessary, for exactly the reason set out earlier in this article.
What it does not do. It does not drive a real browser, so runtime-set cookies are inferred from provider signatures rather than observed. It does not enumerate localStorage, sessionStorage or IndexedDB keys, log in to your site, walk your checkout, or run the accept-all second pass for you. All five blind spots above apply to our scanner too, which is why the manual pass here is written as a requirement and not as an upsell.
What the product does is measure, report and document: a dated inventory, tracked changes between scans, and policy text generated from what was found. It does not make anyone compliant and it does not replace legal review. If you are still choosing tooling, our cookie consent tool guide covers the evaluation criteria.
Frequently Asked Questions
Are cookie scanners accurate enough for a GDPR audit?
Short answer: They are accurate enough to be the starting point, not the finished document. A scanner captures what loads on public pages under the conditions it can reproduce, which is genuine coverage and genuinely incomplete. Because GDPR Article 5(2) puts the burden of demonstrating compliance on you, the defensible artifact is a scan plus a documented manual pass, with a last-verified date on every row.
How often should I re-scan my website for cookies?
Short answer: Match the cadence to your release frequency: ship weekly, scan weekly; a site that changes quarterly is fine with monthly. Independently of the calendar, re-scan after any tag manager publish, new campaign, new plugin or SDK, or redesign. The workflow table lists the triggers worth automating.
Does a cookie scanner find localStorage and tracking pixels?
Short answer: Some do, many do not, and ours does not enumerate storage keys today. It matters because ePrivacy Article 5(3) attaches the consent requirement to storing or accessing information on a user's device rather than to cookies specifically, so localStorage, IndexedDB, service worker caches and identifier-carrying pixels are in scope. The reliable coverage is manual: read each storage area separately in DevTools, then check the Network tab for requests carrying identifiers without setting anything.
Why does my scan show different cookies than my browser DevTools?
Short answer: Almost always because the two are looking at different sites. Your browser has a consent choice recorded, a session, extensions and a region; the crawler has none of those, and many scanners fetch over HTTP instead of executing JavaScript. Run the two-run method and the differences usually explain themselves.
Do I need to inventory cookies on pages behind login?
Short answer: Yes. Obligations attach to the storage you set, not to whether the page is crawlable, and authenticated areas carry the densest storage on the site: session and fraud-prevention identifiers, payment provider cookies, support chat SDKs, conversion tags. No anonymous crawler reaches them, which makes the logged-in walkthrough one of the highest-value hours in the audit.
Sources
- Regulation (EU) 2016/679 (GDPR), full text
- ePrivacy Directive 2002/58/EC, full text
- Google Consent Mode developer documentation
- Turkish DPA, Guideline on Cookie Applications (2022)
Want a dated baseline to start your inventory from? Start free, run a scan across your site up to your plan's page budget, and get an alert when a new cookie appears between scans. Plan details, including scan depth and frequency per plan, are on the pricing page.