UGYBot

Bu sayfa Türkçe de var: UGYBot hakkında

UGYBot is the web crawler of UGY, a free app for iOS and Android that brings what's on in your city (concerts, theatre, exhibitions, festivals, family days out and more) into one feed. UGYBot reads public event listings so that people can find those events. Each event in the app links back to its source.

How to recognise it

UGYBot sends this User-Agent with each request:

UGYBot/1.0 (+https://ugy.app/bot; [email protected])

It never presents itself as a web browser. In robots.txt it follows the rules written for UGYBot.

UGY staff sometimes send the same User-Agent when they check a site by hand, for example to find where its event data comes from. They follow the same robots.txt rules and send only a handful of requests, at least 2 seconds apart.

What it reads

Public event listing pages, the event pages they link to, sitemaps, public event APIs, and the JSON data that a site's own pages load their events from. UGYBot does not log in, does not run a page's JavaScript and does not download images.

What we keep

From each event we keep the facts: its title, date and time, venue and address, price (or that it is free), ticket links and the address of the page we read it from. We also give it one of our own categories.

From some sources we also keep the event's description and the address of its image; the app then loads the image from that address. From some of these sources we may also translate the title and description into English; when we do, we keep the original text too. From other sources we keep only the facts and the link back. If you want us to keep only the facts from your site, write to us.

To read these details we use the structured data on the page, rules we write for a site and, for some pages, an AI language model.

We keep the page content we fetch only to process it, and delete it within 90 days. The exception is a small number of fetched pages and data files that we keep as test samples for our parsers; ask us and we will take your site out of those samples. Events are removed from UGY within 30 days after they take place. English translations of titles and descriptions are also kept in a translation cache, which has no deletion date yet: a translation can stay there after its event has left UGY, even after a removal request. We never sell or license this data. Our Privacy Policy sets out our general retention periods; the two exceptions above are not listed there yet.

How it crawls

• robots.txt comes first. UGYBot reads your robots.txt before it requests a page, and follows it (RFC 9309): the rules of a User-agent: UGYBot group if your file has one, otherwise the User-agent: * rules. We cache your robots.txt for at most one hour, so a change applies within the hour.
• If robots.txt fails. If your robots.txt answers 429 or a server error (5xx), or cannot be reached, UGYBot requests nothing else from your site until it can read the file again. If it answers 404 or another 4xx, UGYBot treats the site as allowed, as RFC 9309 says.
• Redirects. UGYBot checks robots.txt for the address it requests. When that address redirects, UGYBot follows the redirect without checking robots.txt again for the new address.
• Light load. While UGYBot crawls one of our sources, it has at most three requests open at a time and pauses for a short random time (0.15 to 0.45 seconds) before each event page; some sources are read at one request per second. A site can get more requests at once when it hosts several of our sources, which can be crawled at the same time, or many upcoming events: the cancellation check below can have up to ten requests open to one site, in each copy of UGY (see Schedule).
• Schedule. Each source has its own schedule. Most are read twice a day, and a source is never scheduled more often than once an hour (a failed run can be restarted sooner, see Errors during a crawl below). A test copy of UGY also reads some of the same sources, usually less often than the live app, with the same User-Agent and the same rules, and runs the same cancellation check.
• Cancellation check. Every 4 hours, UGYBot checks events from today through the next 7 days, to catch events that were cancelled or removed. It requests the event's page and the page we read the event from: a HEAD request and, unless the HEAD fails or answers 404 or 410, a GET, even after a 401, 403 or 429. The check does not retry and does not read Retry-After. Up to ten of these checks run at the same time. This check follows robots.txt too.
• Errors during a crawl. A 4xx answer is not retried during that visit, except a 429 (and, rarely, a 413 that carries Retry-After). A 429 or 5xx answer is retried at most twice. Before retrying a 429 or 503, UGYBot waits for the time in your Retry-After header, but only until the run reaches its time limit (under ten minutes): a longer Retry-After ends the run early, and the run can then be restarted, up to twice, within minutes. UGYBot does not remember a Retry-After from one visit to the next. When your server cannot be reached at all, UGYBot tries again up to three times over about a minute. After five failed visits in a row, a source is tried at most once a day.
• No workarounds. UGYBot never changes its User-Agent and never uses proxies or a headless browser. It never tries to get around a block, a login or a challenge page.
• One exception. The only exception today is the Ticketmaster Discovery API, which we read with our own API key under Ticketmaster's API terms. It is a licensed API, not a crawl of a website, so robots.txt does not apply to it.

How to block UGYBot

Add this to your robots.txt:

User-agent: UGYBot
Disallow: /

From then on UGYBot asks your site only for robots.txt itself (see the note on redirects above). You can also block only some paths: UGYBot follows the longest matching rule and understands the * and $ wildcards.

Removal and contact

To have events from your site removed from UGY, write to [email protected]. We remove the events of a valid removal request from UGY within 7 days (section 8 of our Privacy Policy); cached translations (see above) are not yet removed. For any other question about UGYBot, write to [email protected].

UGY is a Viaworks product, operated by Tuncan Technology & Consulting DOO, Podgorica, Montenegro.