The Site Medic crawler
Site Medic checks websites for broken links and other problems. If you found this page while reading your server logs, a Site Medic user asked us to check a website and our crawler visited yours. This page explains what it does and how to stop it.
How to recognise it
Its requests carry this User-Agent:
They come from these addresses:
- 192.241.158.139
Anyone can type this User-Agent into another program. If something claiming to be us is behaving badly, check the address it came from against the list above and tell us.
What it does
- It fetches pages of the site being checked with ordinary GET requests, follows the links it finds there, and records what each link returns.
- It also checks links that point to other sites. If a page belonging to one of our users links to yours, you may see a single request for that one address.
- The link checker does not run JavaScript, log in, submit forms, or send anything other than GET requests. (Screenshots, described below, are different: they do run JavaScript.)
- It reads at most 5 MB of any one page.
- Once per check of a site it asks for a short list of well-known addresses, such as /robots.txt, /sitemap.xml and /favicon.ico, and for files that should never be public, such as /.env, /.git/HEAD and /phpinfo.php. That is to warn the site's owner if one is readable. It only looks at what your server answers with; it does not try to exploit or change anything.
- It makes a few test requests to see how your server behaves, once per check of a site:
- Your home page in its four forms (https:// and http://, with and without www.), to see whether they all lead to one address.
- Up to 5 of your pages with the trailing slash added or removed, to look for duplicate pages.
- One address that cannot exist: /mysitemedic-probe- followed by 12 random characters. It tests whether your server answers a missing page with a real page (a “soft 404”) instead of a proper 404. A 404 is the answer we hope for.
- Your home page with the header Origin: https://mysitemedic-probe.invalid, and once with Origin: null. That checks whether your server lets any other website read its responses (CORS). The .invalid address is reserved and cannot belong to anyone.
- It opens a TLS connection to read the site's certificate, and looks up the domain's DNS records (for example SPF, DMARC and CAA).
Screenshots (Page Shots): this part runs JavaScript
A Site Medic user can also ask for screenshots of a single web page. To draw the page as a visitor would see it, we open it in a real web browser (Chromium), and that browser does run JavaScript. So, for these visits only:
- The request looks like a browser's, with our name added on the end:
- It loads that one page, then whatever the page itself pulls in to draw itself: scripts, stylesheets, images and fonts. It is repeated once for each screen size (a phone, a tablet and several desktop sizes; up to 8 in all), one after another. Each time it first makes a quick request to see where the address leads (following any redirect one step at a time, and checking robots.txt at every step) and then loads the page in the browser, so expect two requests to your page per screen size.
- Because your scripts run, your analytics may count these visits. They are recognisable by the User-Agent above.
- It only takes screenshots of web pages. An address that is an image, PDF or other file, or that the server answers with anything other than HTML, is refused. Video, audio and streaming connections are blocked.
- It does not log in, click, fill in or submit forms, and keeps no cookies from one visit to the next.
- It obeys robots.txt exactly as the link checker does, and there is no setting to override that. It reads the file before loading the page and again before following a redirect of it.
- Each user may make only a limited number of screenshot requests a day, set by their plan. The pictures are stored for the person who asked, and nobody else can see them.
How gently it behaves
- At most 2 requests to one website at the same time, and at least 100 ms between starting them.
- It does not yet follow the Crawl-delay directive in robots.txt. If our rate is too high for your server, tell us.
How to block it
Add this to your robots.txt:
You can also block only part of your site by changing Disallow: / to a path such as Disallow: /private/. The crawler reads robots.txt before it fetches a page and obeys it, and so do the screenshots.
One exception you should know about: the person who adds a site to Site Medic can choose to ignore robots.txt for it. That option is meant for people checking their own sites. Anyone can add any public address, and we cannot prove who owns it, so if you want crawls of your site to stop whatever the setting, contact us. Screenshots have no such setting: they always obey robots.txt.
Contact us
To stop crawls of your site, report a problem, or tell us it is going too fast, email crawler@mysitemedic.com with the address of your website and roughly when you saw the requests, and we will look into it promptly.