Static assets scraping
In the context of Zoning and Session Replay, Contentsquare fetches static assets on your website.
To allow Contentsquare fetching these assets, select one of the following options:
- Allowlist Contentsquare IP addresses
- Use a static header to validate requests
- Use a dynamic signature header to validate requests
Allowing IP Addresses
Section titled Allowing IP AddressesAllow ports 80
(HTTP) and 443
(HTTPS) and the following IP addresses to prevent your proxy, firewall, or server configuration from blocking the scraper.
52.18.162.15720.75.90.236100.24.76.9034.192.98.14820.67.250.10954.247.44.19652.51.9.1235.72.153.3835.73.99.41
Using a custom static header
Section titled Using a custom static headerWhen selecting this option, Contentsquare adds a custom header to the project settings.
{ "headers": { "my-new-header-key": "myKeyValue" }}
You can then validate that scraper requests contain the specific header and value.
receivedHeaderExample = 'myKeyValue';
const CONTENTSQUARE_CUSTOM_HEADER = 'myKeyValue';if (receivedHeaderExample === CONTENTSQUARE_CUSTOM_HEADER) { //}
Using a custom dynamic signature header
Section titled Using a custom dynamic signature headerWhen selecting this option, Contentsquare adds the X-CONTENTSQUARE-SIGNATURE
header to incoming requests from the scraper.
The X-CONTENTSQUARE-SIGNATURE
header is a string generated in this format:
<TIMESTAMP>-base64(hmac('sha256', <SECRET>, <RESOURCE_DOMAIN>-<TIMESTAMP>))
with:
<TIMESTAMP>
: the time at which the request was sent usingDate.now()
,<RESOURCE_DOMAIN>
: the complete domain hosting the resource on your website,<SECRET>
: the secret shared between you and Contentsquare for the project, generated at project creation.
With a secret of abcde
, the Contentsquare scraper service has emitted the request below on contentsquare.com
on the 6th of August 2020 at 05:39 am, to fetch the official Contentsquare logo.
curl \ -H 'accept-language: en-GB,en-US;q=0.9,en;q=0.8,fr;q=0.7' \ -H 'accept: application/json, text/plain, */*' \ -H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36' \ -H 'X-CONTENTSQUARE-SIGNATURE: 1596706743675-BxmHtG6vu4CfFlzpHxc0qYOmR0iMajlIvA2B4404qk4=' \ -X GET https://contentsquare.com/wp-content/themes/kps3-contentsquare/public/assets/images/contentsquare-logo.svg?tv=1.3.0
You can compute the signature and verify it against the value of the X-CONTENTSQUARE-SIGNATURE
header by providing:
- The timestamp from the incoming request (
1596706743675
), - The resource domain (
contentsquare.com
), - The secret provided by Contentsquare (
abcde
).
const crypto = require('crypto');
const secret = 'abcde'; // Given by someone from CSreceivedHeaderExample = '1596706743675-BxmHtG6vu4CfFlzpHxc0qYOmR0iMajlIvA2B4404qk4=';
// Extract the timestamp and digest from the received headerconst [timestamp, receivedDigest] = receivedHeaderExample.split('-');
// Extra security step to make sure the timestamp is not too oldconst currentTimestamp = Date.now();if(currentTimestamp - timestamp > 5 * 60 * 1000) { throw new Error("Validation failed. Timestamp signature is older than 5 minutes");}
// Recreate the string that was used to generate the digestconst dataToSign = `${resourceDomain}-${timestamp}`; // e.g. 'contentsquare.com-1596706743675';
// Create a new digest using the same secret and algorithmconst hmac = crypto.createHmac('sha256', secret);hmac.update(dataToSign);const generatedDigest = hmac.digest('base64');
// Compare the newly generated digest with the one received in the headerif (receivedDigest === generatedDigest) { console.log('Header is valid'); // Code to validate request}