If you found TechnoUnion-HeaderWalker in your server logs, this page explains what it is
and how to stop it.
Allsource operates a crawler that identifies which technology products a company's website uses. It requests the home page of a domain, reads the HTTP response headers and the returned HTML, and records the names of technology vendors it recognises. It identifies itself with this User-Agent:
TechnoUnion-HeaderWalker/1.1 (+https://www.allsourcedata.io/crawler; honors opt-out; headers-only)
/robots.txt, before anything else./. One request.
A domain is visited no more than once every 30 days. Requests are made from
15.204.104.113.
The names of technology vendors detected on the site, and the account identifier a tag carries when one is present in the public page source. The page content itself is discarded after it is read and is not stored, republished, or used to reproduce the site.
The crawler does not collect personal information about visitors to your site. It does not execute JavaScript, set cookies, or interact with any analytics or advertising tag on the page.
The fastest way is robots.txt, which takes effect on the next visit and requires nothing
from us. Add this to the robots.txt file at the root of your domain:
User-agent: TechnoUnion-HeaderWalker Disallow: /
A wildcard rule blocking all crawlers also works, because this crawler applies the
User-agent: * group when no rule names it specifically.
If you cannot edit robots.txt, or you want a domain on a permanent
do-not-crawl list that does not depend on that file, email
crawler@optout.allsourcedata.io
with the domain or domains you want excluded.
Requests are processed automatically within about ten minutes, and you will get a
confirmation once it is done. Domains on that list are never requested again,
regardless of what robots.txt says, and any records we already hold for
them are deleted.
The crawler makes two requests to a domain once a month, so it is unlikely to affect a site's performance. If you are seeing something different from that, email the address above with a sample of your access log and we will look at it.