AubeActuBot, the AubeActu collection robot

Who runs this robot

AubeActuBot is the collection robot of AubeActu, a local news service for the Aube department, published by Adnan TOURAK EI (Tourak Digital). Publication director: Adnan TOURAK.

A question, an incident, an unusual load on your server: write to contact@aubeactu.fr. The robot carries this address in its header so that you can reach us rather than have to block it.

How to recognise it

It identifies itself with this User-Agent header, and no other:

User-Agent: AubeActuBot/1.0 (+https://aubeactu.fr/robot; contact@aubeactu.fr)

In your robots.txt, its name is AubeActuBot.

What it reads

RSS feeds and news pages published by the municipalities, the groupings of municipalities and the State services in the Aube, and open datasets: weather, weather warnings, fuel prices, product recalls, openly licensed event listings. It never enters a restricted area and never fills in a form.

What it does with what it reads

It publishes nothing itself. What it reads feeds the pool from which the newsroom draws its stories: the information is checked and rewritten there, and its source is cited under the article, with a link to your page. Only official alerts, openly licensed event listings and service data may appear without rewriting.

How it behaves, in figures

These limits are the ones the program applies; this page reads them from its configuration, it does not copy them.

• It reads a site's robots.txt before entering it, and reads it again at least every 24 hours. A Disallow rule aimed at it, or at all robots, is applied; a Crawl-delay longer than its own spacing is honoured.
• Two requests to the same site are at least 2 seconds apart.
• At most one pass per site every 5 minutes, and at most 5 requests per pass, pagination included.
• At most 12 requests per hour and 60 requests per day to the same local authority site.
• At most 1500 requests per day, all sites together.
• No reading of municipal sites at peak hours: from 8:00 to 9:00 and from 14:00 to 15:00.
• On a local authority site, it gives up a connection after 5 seconds, a response after 15 seconds, and any document larger than 8 MB.
• It sends conditional requests (If-None-Match, If-Modified-Since): a feed that has not changed is not downloaded again.
• A 429 or 503 response suspends every request to that site for the time asked for in the Retry-After header, two hours at most; without that header, one minute.
• Public open-data APIs (weather, warnings, fuel prices, product recalls, geocoding) are queried under their own terms of use: the per-site quotas above apply to local authority sites; the spacing between two requests applies to all.

Keeping the robot off your site

Through your robots.txt, without writing to us: add these two lines. The rule is taken into account within 24 hours at most.

User-agent: AubeActuBot
Disallow: /

To exclude only part of the site, replace the slash with the path concerned.

By e-mail to contact@aubeactu.fr, giving the address of your site: the newsroom switches the source off and confirms it by return e-mail.