The JED crawler
JED builds and runs websites for small businesses. Our crawler visits public websites to learn which businesses have one, what it is built with and whether it works on a phone.
What it does
- It reads robots.txt first, and stops there if it is not allowed in.
- Then it loads the home page once. It does not follow links and does not download images or files.
- From that page it keeps only public facts: the title and description, the kind of business, the platform the site is built on, whether it works on mobile, and the contact details the business publishes on it. It never keeps a copy of the page.
- It makes one request at a time to each website, and visits each website once per pass.
What it never does
- Log in, fill in or submit forms, or probe for weaknesses.
- Visit a website that asked it not to.
How to recognise it
It identifies itself with this user agent:
Mozilla/5.0 (compatible; JedBot/1.0; +https://jedsite.com/bot)How to stop it
Add this to your robots.txt:
User-agent: JedBot
Disallow: /Or email support@jedsite.com with your domain. We take it off our lists and never visit it again.
The crawler is run by DS Maker LTD, Cyprus, the company behind JED.