Unblocking the Haystack spider
- Applies to:
In rare circumstances, Haystack may be blocked from downloading the pages of a website by an anti-bot filter.
This is usually because Haystack’s requests originate in the AWS (Amazon Web Services) cloud, which some filtering systems block. This article gives the necessary technical information to unblock Haystack’s requests in your filtering system. Note – it is only possible to unblock websites for which you control the filtering.
The best way to unblock Haystack’s requests is to look for the following HTTP request header which is part of every request Haystack makes:
X-TESTING-APPLICATION: Silktide Haystack 1.0
What doesn’t work
Haystack’s server IPs are not static and change daily, so the best way to unblock Haystack is via request data.
Unfortunately, user agent is also not ideal as in order to circumvent some filtering tools, Haystack uses a user agent copied from a real browser and is subject to change. The current user agent for Haystack is:
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36
If you need further assistance, please contact your Haystack account manager.