Aggregating late arrivals

There may be times when data associated with your Client Audit Logs (CAL) arrives for processing later than expected. When this happens, the first CAL file that is available to download will be missing some data. The CAL file is updated once the data is finished processing but will contain both the new and old data. To avoid discrepancies caused by duplicate data, you must delete the original data and re-aggregate the CAL files.

The process for re-aggregating the data is different depending on the type of CAL files that you are subscribed to.

If you know the type of files that you are subscribed to, proceed to one of the appropriate sections below:

• Aggregating late arrivals in single bid event and impression CAL files

• Aggregating late arrivals in multi-part bid event CAL files

If you don't know the type of files that you are subscribed to, complete the following to find out:

1. Send two CAL API requests:
   • One to list single bid event or impression files ("/downloads?subscriptionName=bid events") or ("/downloads?subscriptionName=impressions")
   • One to list multi-part bid event files ("/downloads/bidevents")
2. Review the response returned by the CAL API. One of the routes should return the bid event list and the other should be empty. If the request for single event bid files returns a list, proceed to Aggregating late arrivals in single bid event and impression CAL files. If the request for multi-part bid event files returns a list, proceed to Aggregating late arrivals in multi-part bid event CAL files.

Aggregating late arrivals in single bid event and impression CAL files

To determine if there are late arrivals in single bid event and impression files, locate any files with a downloadStatus of Updated and delete the duplicate data:

1. Send a request to the CAL API to retrieve a list of files available for download ("/downloads?subscriptionName=bid events") or ("/downloads?subscriptionName=impressions").
2. Parse the hour and downloadStatus fields from each JSON object in the response. Review the downloadStatus field and locate all files with a status of Updated.
3. Download all files with a status of Updated.
4. Review each file and delete the original data.
5. After the duplicate data in all files is removed, run the queries a second time.

Aggregating late arrivals in multi-part bid event CAL files

To determine if there are late arrivals in multi-part bid event files, locate any files for a previously downloaded hour with a downloadStatus of New:

1. Send a request to the CAL API to retrieve a list of files available for download ("/downloads/bidevents").
2. Review the hour field to locate the date and hour of the last download.
3. Review the parts field for all files within that hour. Any files for a previously downloaded hour with a downloadStatus of New are late arrivals.
4. Download all late arrival files.
5. After the duplicate data in all files is removed, run the queries a second time.