Less technical information is also available
This is a developers' guide to using our insight data; for a shorter, less technical introduction to insight data see our guide for non-developers.
Insight data is storable information related to your customers’ purchasing histories and habits. It allows you to store collections of arbitrary data against your contacts or account via our API.
So they’re additional contact data fields, extending the contact data fields we already store?
Yes – but with the prime difference being these are transaction-based ones. Any additional keyed data you have for a contact can be stored as insight data, with very little restriction on the type of data it can be. As a broad rule, you can store anything that is serialisable to JSON.
Just as you can currently segment upon contact data fields such as gender, age and geography, insight data gives you the ability to segment upon types of items purchased, regularity of purchases and amount spent on purchases, for instance. As ever, the quality of your insight data will dictate how well you can segment address books and personalise your campaigns and offers.
Here are some examples of datasets that could be stored as insight data:
- For an auction site, a list of all bids a user has made and if they succeeded or not, including the date and time of the bid, the amount of the bid and the category and name of the item bid on
- For a travel agent, a list of destinations a user has visited via bookings on the site, including the amount of bookings for the destination (solo, couple, family, etc.,), the amount spent and the type of accommodation
- A list of a user’s likes and dislikes
The key point is that many different types of insight data can be stored for each contact, and structured in a way of your choosing.
Once insight data is stored against your contacts, you can use the website to write queries to segment your contacts against this data and create new address books. This will enable you to send targeted, personalised campaigns based upon your contacts’ transactional history and habits. Using the above examples, you will be able to run segmentations based on bids made, countries visited and likes and dislikes.
For more on segmenting insight data, read our article on insight data segmentation.
This section provides information and guidelines on storing your insight data. It does get quite technical (if it hasn’t enough already!), so if you are reading this and are not a developer then you will probably want a developer to read it instead, or at least help you with understanding it.
At its heart, insight data is a key-value pair storage mechanism; you store a piece of data against a contact, using a key to refer to it. If you need to update or delete a piece of data at a later date, you use the key as a reference to retrieve the piece of data and alter it.
- Be unique (it is up to the user of the API to ensure uniqueness)
- Contain no more than 255 characters
A key can contain any character you like.
Data is stored against a contact or as account-scoped data
A contact identifier (email address or ID) is submitted with each insight data document uploaded. If the identifier used is "account" rather than an email address, then the record will be stored as account-scoped insight data (AccountInsight data). This data can be used for advanced personalisation and viewed via the account menu in application.
All data to be stored must be serialisable as JSON. This means complex object structures can be used to represent data. For example, you could represent the contents of a user’s shopping basket like this:
"name": "a dvd",
"name": "a book",
There are a few restrictions and extensions to the JSON format as outlined below.
Valid collection names can only contain the following characters:
- alphanumeric (a-z, A-Z, 0-9)
- hyphen ("-")
- underscore ("_")
Furthermore, collection names can't:
- begin with a number
- exceed 255 characters in length
- have exactly the same name as any other collection, even if the collections are differently scoped (e.g. one collection is contact-scoped and another collection is account-scoped (AccountInsight))
- The null value is not allowed
- String values are restricted to 255 characters in length.
- Date time values are also supported via a subset of the ISO 8601 standard (e.g. 2013-03-19T17:22:55.804Z)
- Lists must be of the same type. Lists may only contain either Booleans, strings, numbers, objects, lists, etc.
See the JSON specification for all other value types.
A note on date time formatting
Not all ISO 8601 formats are supported (for example the string "2013" would not be treated as a date). The following are valid for this implementation.Complete date plus hours, minutes and seconds: YYYY-MM-DDThh:mm:ssTZD (eg 1997-07-16T19:20:30+01:00) Complete date plus hours, minutes, seconds and a decimal fraction of a second YYYY-MM-DDThh:mm:ss.sTZD (eg 1997-07-16T19:20:30.45+01:00)
where:YYYY = four-digit year MM = two-digit month (01=January, etc.) DD = two-digit day of month (01 through 31) hh = two digits of hour (00 through 23) (am/pm NOT allowed) mm = two digits of minute (00 through 59) ss = two digits of second (00 through 59) s = one or more digits representing a decimal fraction of a second TZD = time zone designator (Z or +hh:mm or -hh:mm)
All times without a time zone designator will be treated as UTC.
While you can store any JSON structure as insight data, it is not strictly schemaless. Insight data requires that you store similarly structured data in a collection. Each collection may have a different schema, allowing you to store a variety of different data against contacts.
Schemas are defined implicitly upon uploading data via the API for the first time. Furthermore, schemas are extendable but not editable.
For example, let’s say you’ve just uploaded the following object into a collection you’ve created called “preferences”:
For all proceeding uploads, the preferences collection will expect an object with two properties, “likes” and “dislikes”, each of which refer to an object where each object has a property called “animals” which is a list of strings and “numbers” which is a list of numbers.
Once you’ve defined a named property, its type can’t be changed. For example, if you upload this JSON object to define your schema for a collection:
"name": "Tom Bloggs"
You couldn’t then upload the following object:
Why? Because the “id” property has been changed from a string to a number and the “name” property has been changed from a string to an object.
The only things that are mutable are objects because they are extendable. Properties can be added to an object at any time (however they are then bound by the above rule). For example, if this object was uploaded to define a collection’s schema:
"fullName": "John Bloggs"
Then uploading this object would extend the collection's schema to include the two new properties, “firstName” and “lastName”:
"fullName": "John Bloggs"
As standard, you will be adding your data on a single-transaction-at-a-time basis – for instance, at the point of online checkout.
However, to get you up and running, it is highly likely that you’ll want to make an initial bulk import of all your historical data. This can be done using the SOAP API methods ImportTransactionalData and GetTransactionalDataImportProgress, or the REST API methods Bulk add transactional data to contacts and Get transactional data import status.
You can perform multiple bulk imports at a time for an account. Typically we'd suggest five as a maximum, for optimal speed and performance.
Given the nature of Insight data storage, you will want to spend some time thinking about how your Insight data needs to work for you. A little bit of planning will go a long way! What sort of data do you want to store? How might you want to extend this data as you move forward?
The point to remember is that you don’t want to become backed into a corner by initiating a schema that will prove constrictive. For example, you will probably want to avoid entering your product ID as an integer. This won’t give you very easily identifiable information and you won’t be able to change it once it is uploaded. To change it, you would need to start again with a new schema and re-upload all of your data to conform to it. This is definitely something to be avoided!