← Identity home

OPERATION / POLICY

Crawling Policy

The operating rules for VigiaLabsBot.

This policy describes the operating rules for VigiaLabsBot.

01

Operator identification & scope

VigiaLabsBot is operated by ABE International, SL, through VigiaLabs. It performs periodic technical privacy audits of cookies, tracking, fingerprinting and consent implementations on publicly accessible websites.

Requests use Mozilla/5.0 (compatible; VigiaLabsBot/1.0; +https://vigialabs.net/). This information website does not crawl, render or scan third-party websites.

02

robots.txt & exclusions

The crawler retrieves and honours robots.txt using RFC 9309. It checks applicable disallow rules before collecting a resource. When server or network failures make robots.txt unreachable, it suspends collection according to the protocol’s handling rules. No access-control permission is implied by robots.txt.

To express a complete exclusion for this bot, publish:

User-agent: VigiaLabsBot
Disallow: /

Domain owners can also request exclusion via the operator’s contact channel. Include the domain, requested scope and evidence of authority.

Crawl-delay is not standardised by RFC 9309. This site does not claim support for that extension.

03

Rate limits & retries

The crawler limits concurrency and request frequency per origin, avoids bursts, and reduces load when a server reports difficulty.

HTTP 429 responses trigger backoff. The crawler honours valid Retry-After values, including HTTP dates and delay seconds, on 429 and 503 responses. Repeated failures pause activity to avoid aggressive retries.

04

Access restrictions & security boundaries

The crawler respects authentication, paywalls, access restrictions and explicit denials. It does not bypass CAPTCHAs, evade blocks by changing identities, exploit vulnerabilities or attempt access to private networks. A cryptographic identity does not grant authorisation to access a resource.

Logins, form submissions, purchases, destructive actions and vulnerability exploitation are outside the documented scope. Redirects and resource destinations must remain within these access boundaries.

05

Technical evidence & responsible operation

Audits may require HTML, cookies, browser storage, network metadata, screenshots and historical observations. Only evidence relevant to the audit should be collected. This does not authorise unrestricted collection of personal data.

Unexpected traffic, privacy concerns and security incidents should be reported through the contact instructions. The privacy notice explains the applicable data handling and retention policy.

06

Crawler activity and third-party data

The service includes examining publicly accessible websites without prior permission from their owners. Public availability does not remove data protection, copyright, database or access restrictions. A client’s contract does not itself provide a contractual legal basis for processing unrelated people’s data.

Processing personal data requires a specific purpose, necessity and an applicable legal basis. Reliance on legitimate interests requires balancing individual rights and applying safeguards. Indirect collection requires the applicable transparency measures; a public notice alone is not an automatic exemption from individual information duties. High-risk processing requires the corresponding impact assessment.

A domain exclusion is an operational request, separate from a person’s data protection rights. You can request action on your personal data without proving ownership of a website. An exclusion does not necessarily erase lawfully preserved evidence, and preserving a minimal exclusion rule may be necessary to avoid collecting again.

Technical findings are evidence for review, not a legal ruling or certification of compliance. Applicable rules depend on the operator, audience, processing and territorial scope, not just a node’s IP address.

Retention policy