Historical Data Migration

As part of the installation, Symbee Connect Business Intelligence supports optionally migrating your Amazon Connect and Symbee Connect historical data into the solution if you have this data already saved in S3 buckets in the original JSON/Firehose format.

This means you can immediately analyze and report on your historical activity from day one.

Migration of the following data types are supported:

• Amazon Connect Contact Trace Records (CTR's) - i.e. your historical contact (call, chat, task) activity
• Amazon Connect Agent Events - i.e. your historical agent state related activity
• Symbee Connect Interaction Trace Records (ITR's) - i.e. your historical non-Amazon Connect delivered interactions (email, webchat, preview-dial, sms, etc.) activity
• Symbee Connect Work Codes activity
• Symbee Connect Optimize Evaluations activity
• Amazon Connect Contact Lens data

When to set up and perform your Migration?

Historical Data Migration is performed by components that get deployed as part of the Symbee Connect Business Intelligence CloudFormation. The deployment of these additional components is optional, and temporary (once migration is complete, the migration components can be undeployed).

Essentially, a single input parameter in the CloudFormation stack determines if historical data migration components are deployed (or undeployed) - the Deploy Migration Components parameter (see here for reference).

You can perform the initial CloudFormation stack deploy without the Deploy Migration Components parameter set (which will immediately start collecting "new/live" data from that point on), and then perform an Update on the CloudFormation stack at a later time, changing the Deploy Migration Components parameter to Yes, which will cause the deployment and configuration of the migration components at that later time.

Or you can enable the Deploy Migration Components parameter during the very initial CloudFormation stack deployment.

Either way, once migration of your data is complete, always perform a subsequent Update on the SCBI CloudFormation stack, resetting the Deploy Migration Components parameter back to No.

Setting up for Historical Data Migration

The first step for migrating your historical data is to deploy (or Update) the Symbee Connect Business Intelligence CloudFormation with the Deploy Migration Components input parameter set to Yes. Further below is a link to the step by step guide for this, but review the below first for context:

• The Historical Data Migration section of the CloudFormation parameters asks for a Start and End date, used to scope how much data to migrate from the source S3 buckets you identify. Make sure to set these parameters carefully to avoid overlap with live data already being collected and creating artificially duplicated data. Review notes related to these dates here.
• The CloudFormation deploy/update does not actually start the migration process, it only deploys and configured it. Beginning the actual migration is a separate (intentionally manual) step you need to perform after the CloudFormation deploy/update has completed. See the next section for this.

See Download and Run the Business Intelligence CloudFormation.

Once your stack is finished deploying (or updating), see the next section.

Starting Migration

Once the SCBI CloudFormation deploy/update (with the Deploy Migration Components input parameter set to Yes) is complete, take the following steps to start the process of migrating your historical data (Note: the migration could take anywhere from hours to days to complete, depending on how much data is being migrated in. The system can still be used in the meantime):

1. Log Into your AWS Management Console.

2. Make sure you are in the correct AWS region where you deployed your Business Intelligence CloudFormation stack (on the top right of the Navigation bar).

3. Click on Services at the top, and under Application Integration, choose Amazon EventBridge (or search EventBridge)

4. From the EventBridge console, on the left-hand navigation bar under Buses, select Rules...

   

5. On the Rules list (above) locate the rule named sc-bi-trigger-migration... (it will be Disabled) - select it, and Enable it (per above).
   
   

Monitoring Migration Progress

Take the following steps to monitor the progress of your historical data migration:

1. From the AWS Console, click on Services at the top, and under Management & Governance, choose Systems Manager (or search SystemsManager)

2. From the AWS Systems Manager console, on the left-hand navigation bar under Application Management, select Parameter Store...

3. On the Parameter Store listing screen, add a search filter of 'SC_BI' and you will see up to 6 parameters (depending on what options you chose when running the CloudFormation) related to Symbee Connect Business Intelligence.

   

Each of the above parameters track the status of the migration for the respectively named data type. Click into each one, and check its value. You can also click on the History tab inside each parameter to see the last update of progress. The parameters will have one of the following values:

• NOT_STARTED
• DISCOVERING_RECORDS
• SENDING_FOUND_RECORDS_TO_FIREHOSE
• PROCESSING_BATCH_RECORDS_TO_FIREHOSE
• COMPLETE

In addition to the above, you can also monitor progress by looking at the contents of your sc-bi-data... S3 bucket in the S3 Console.

Note: The migration process works backwards, starting at the historical data migration End Date you configured in your CloudFormation stack, and walks back day-by-day until the Start Date is reached. You will be able to analyze and report on your historical data while the migration is in progress - as soon as the data has arrived in the S3 bucket it will be available in Amazon QuickSight to view.