Reviewing and Updating Evaluations
Completed evaluations can be reviewed, updated, and voided (if needed) from the Evaluations History view.
Evaluations History View
The Evaluations History view is accessed from the main Function menu within the Symbee Connect Administration Portal, using the Evaluations History option under the Optimize subsection.
The evaluations history view allows you to review recent evaluations performed, update evaluations to resolve disputes, and export evaluation data for further analysis.
Use the Time Range and Status criteria section to select an appropriate time range, and which evaluation statuses to show, and click the Search button. The results for your search are then displayed in the table below. The view is limited to display 500 results. A message will be displayed if your criteria would have returned more than 500 entries. If you see this message, you can either reduce your time range or add additional filters to further restrict the results returned.
Historical Evaluation Actions
With a set of evaluation results displayed, the Actions column allows certain actions to be performed, depending on the status of each Evaluation. An evaluation can be in one of the following states:
- Completed - this is the normal status for an evaluation - when a new evaluation is created, this is the initial state.
- Contested - only possible if Evaluation Disputes has been enabled (see Working with Disputed or Contested Evaluations). This state indicates the user or agent being evaluated has disputed the evaluation, and it is waiting for review.
- Resolved (Amended) - only possible if Evaluation Disputes has been enabled - after an Evaluator has reviewed a dispute, they change the state of the evaluation to either this state, or Resolved (Unchanged) below.
- Resolved (Unchanged) - only possible if Evaluation Disputes has been enabled - after an Evaluator has reviewed a dispute, they change the state of the evaluation to either this state, or Resolved (Amended) above.
- Voided - the evaluation has been voided. Once an evaluation has been voided, no further actions can be taken on it (other than a read-only view of the evaluation).
Up to three actions are available, based on the evaluation state:
- View Evaluation action - opens a read-only view of the Evaluation - this is effectively the same view the user or agent sees when they are reviewing the evaluation. The View action is available for evaluations in all states. For reference - the same User/Agent review link displayed on the last Evaluation Complete page when initially creating an evaluation (see here) is also available at the bottom of this view if you need to retrieve the link again.
- Update Evaluation action - opens the existing evaluation in the same Evaluation Create/Edit viewer that is used when new evaluations are created. The Update action is available for all evaluations except those in a Voided state. Allows you to modify question answers and comments in the evaluation as needed, which will update the total score if necessary. If Evaluation Disputes has been enabled (see Working with Disputed or Contested Evaluations), and the evaluation is either in a Contested or one of the Resolved states this mode also contains an additional section at the bottom of the evaluation that displays the users comments, any resolution comments added by the evaluator, and an evaluation state control allowing the reviewer to update the state (for example, from Contested to Resolved).
- Void Evaluation action - after confirmation, changes an evaluation's state to Voided. The Void action is available for all evaluations except those already in a Voided state. Once an evaluation is voided, it can no longer be updated.
Evaluation Data Storage and Format
When Optimize Evaluations are performed, all the data associated with each evaluation is serialized into JSON (JavaScript Object Notation) format, and stored in an S3 bucket within your AWS Account.
The S3 bucket holding your evaluation data is created when you download and run the Optimize CloudFormation. For further reference, see the Installing the Optimize CloudFormation step-by-step guide.
For reference, the S3 bucket in your AWS account will be named: sc-optimize-env-999999999999 (where 'env' is the name of your Symbee Connect Company Environment, and 999999999999 is your AWS Account number).
Analyzing your Optimize Evaluation data
After evaluations are complete, you can review your data using one of three approaches:
- Symbee Connect Business Intelligence (SCBI) provides pre-defined Amazon QuickSight Dashboards over the Evaluation data stored in your S3 bucket
- Use the Evaluations History view within the Symbee Connect Administration Portal
- Provides a real-time means for searching and reviewing your historical Evaluations data stored in S3.
- Allows you to export summary evaluation data to comma-separated-values (CSV) files for further analysis in tools like Excel and Google Sheets. The exported data is in tabular format with 1 row per evaluation made, and columns providing total actual and possible scores, and a breakdown of the actual and possible score received for each question in the form.
- Provide your own custom analysis and views of the evaluation data stored in S3 (Setting up this sort of approach is beyond the scope of this documentation). For example:
- You could configure S3 triggers to notify a process of each new evaluation (JSON object) arriving (or being updated) in S3, and then process the results in a Lambda and store the data in your own in-house (or third-party) reporting solution
- Use AWS Glue and Amazon Athena services to process the S3 data
Optimize Evaluation payload format (when stored in S3)
The following provides a description for reference, of the Optimize Evaluation JSON payload stored in your Optimize S3 bucket.
S3 Object key structure
Evaluations are stored within your Optimize S3 bucket using the following object naming convention:
Evaluations/CCYY/MM/DD/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX-AAAAAA-EEEEEE.json
Where:
- Evaluations/ is static
- CCYY/MM/DD is the year month and day of the time the evaluation was initially created
- XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX is the unique identifier created for the Evaluation
- AAAAAA is the User Id of the user or agent being evaluated
- EEEEEE is the Admin User Id of the Evaluator that performed the evaluation
- .json is the static file extension
Evaluation JSON object example
The following is an example Evaluation JSON payload stored in the S3 bucket - this example uses an Evaluation Form that has at least one question of each response type for reference.
The JSON payload is intentionally a completely self-contained object containing all the details of the original Form used to create the Evaluation, as well as all the answers the Evaluator responded with to each form question.
{
"EvaluationId": "xxxxxxxx-66a1-42e3-a0e5-xxxxxxxxxxxx",
"FormGroup": "Customer Servicing Dept",
"FormName": "CS Standard Phone Evaluation",
"FormVersion": 0,
"EvaluationDateTime": "2023-01-18T17:06:17.007Z",
"LastUpdated": "2023-01-18T17:09:23.121Z",
"FormStatus": "NOT_DISPUTED",
"FormDescription": "Standard Evaluation Form for Customer Servicing representatives when handling Inbound Phone calls (across all clients).",
"FormComments": "Great job!",
"ActionThreshold": 60,
"FormTotalScore": 73,
"FormTotalPossibleScore": 85,
"Evaluator": "mike.mars@yourcompany.com",
"EvaluationUser": "john.smith@yourcompany.com",
"UserReportHierarchy": "",
"InteractionId": "xxxxxxxx-b9fa-437a-a74c-xxxxxxxxxxxx",
"InteractionDateTime": "2023-01-14T22:19:06Z",
"InteractionType": "VOICE",
"InteractionLength": 367,
"InteractionDirection": "INBOUND",
"InteractionFarEnd": "+15551002000",
"InteractionQueue": "Customer Service",
"Sections": [
{
"SectionName": "Greeting and Introduction",
"SectionOrder": 1,
"SectionTotalScore": 13,
"SectionTotalPossibleScore": 15,
"Questions": [
{
"QuestionName": "Q1",
"QuestionOrder": 1,
"QuestionText": "Did the Representative begin the using the correct and approved Client Introduction Greeting? ",
"QuestionExplanation": "",
"QuestionType": "YESNO",
"QuestionScale": 5,
"QuestionPotentialPoints": 5,
"QuestionRequired": true,
"QuestionResponse": "YES",
"QuestionScore": 5,
"QuestionComments": "",
"QuestionAvailableOptionInfo": null
},
{
"QuestionName": "Q2",
"QuestionOrder": 2,
"QuestionText": "Rate the Representatives spoken demeanor and attitude when answering the call. Were they polite, friendly, and welcoming?",
"QuestionExplanation": "Provide a rating between 1 and 5 (1 being poor, 5 being exemplary)",
"QuestionType": "RATING1TO5",
"QuestionScale": 10,
"QuestionPotentialPoints": 10,
"QuestionRequired": false,
"QuestionResponse": "4",
"QuestionScore": 8,
"QuestionComments": "",
"QuestionAvailableOptionInfo": null
}
]
},
{
"SectionName": "Caller Identification",
"SectionOrder": 2,
"SectionTotalScore": 0,
"SectionTotalPossibleScore": 0,
"Questions": [
{
"QuestionName": "Q3",
"QuestionOrder": 3,
"QuestionText": "Did the Representative prompt and receive at least 2 forms of additional personal information associated with the account before progressing further with the call?",
"QuestionExplanation": "",
"QuestionType": "YESNO",
"QuestionScale": 0,
"QuestionPotentialPoints": 20,
"QuestionRequired": false,
"QuestionResponse": "N/A",
"QuestionScore": 0,
"QuestionComments": "",
"QuestionAvailableOptionInfo": null
}
]
},
{
"SectionName": "General Call Handling",
"SectionOrder": 3,
"SectionTotalScore": 15,
"SectionTotalPossibleScore": 20,
"Questions": [
{
"QuestionName": "Q4",
"QuestionOrder": 4,
"QuestionText": "Rate the Representatives ability to clearly understand the Caller's issue, and then achieve a good outcome with the Customer so the Customer doesn't end up calling back multiple times relating to the same issue",
"QuestionExplanation": "Looking to understand if we are achieving First Call Resolution.",
"QuestionType": "PICKLIST",
"QuestionScale": 20,
"QuestionPotentialPoints": 20,
"QuestionRequired": false,
"QuestionResponse": "Thorough",
"QuestionScore": 15,
"QuestionComments": "",
"QuestionAvailableOptionInfo": "Poor,0|Average,10|Thorough,15|Exemplary,20"
}
]
},
{
"SectionName": "Call Completion",
"SectionOrder": 4,
"SectionTotalScore": 45,
"SectionTotalPossibleScore": 50,
"Questions": [
{
"QuestionName": "Q10",
"QuestionOrder": 5,
"QuestionText": "Did the Representative ask the Customer the required 'Is there anything else we can help you with today?' question before closing out the call? ",
"QuestionExplanation": "",
"QuestionType": "YESNO",
"QuestionScale": 10,
"QuestionPotentialPoints": 10,
"QuestionRequired": false,
"QuestionResponse": "YES",
"QuestionScore": 10,
"QuestionComments": "",
"QuestionAvailableOptionInfo": null
},
{
"QuestionName": "Q15",
"QuestionOrder": 6,
"QuestionText": "Who completed the call? Did the Representative disconnect first? Or did the Representative wait for the Customer to leave the call?",
"QuestionExplanation": "",
"QuestionType": "PICKLIST",
"QuestionScale": 20,
"QuestionPotentialPoints": 2,
"QuestionRequired": false,
"QuestionResponse": "Customer First",
"QuestionScore": 20,
"QuestionComments": "",
"QuestionAvailableOptionInfo": "Agent First,1|Customer First,20"
},
{
"QuestionName": "Q20",
"QuestionOrder": 7,
"QuestionText": "Give the Representative an overall rating for their handling of the call, between 0 and 20 points. ",
"QuestionExplanation": "This is an optional opportunity to award the Representative with additional points if you think they earned it.",
"QuestionType": "RATINGUNBOUND",
"QuestionScale": 20,
"QuestionPotentialPoints": 20,
"QuestionRequired": false,
"QuestionResponse": "15",
"QuestionScore": 15,
"QuestionComments": "",
"QuestionAvailableOptionInfo": "1-20"
}
]
},
{
"SectionName": "General Feedback and Training",
"SectionOrder": 5,
"SectionTotalScore": 0,
"SectionTotalPossibleScore": 0,
"Questions": [
{
"QuestionName": "Q50Training",
"QuestionOrder": 8,
"QuestionText": "Provide any additional feedback or training tips as required.",
"QuestionExplanation": "",
"QuestionType": "FREETEXT",
"QuestionScale": 0,
"QuestionPotentialPoints": 0,
"QuestionRequired": false,
"QuestionResponse": "",
"QuestionScore": 0,
"QuestionComments": "Additional training notes.",
"QuestionAvailableOptionInfo": null
}
]
}
],
"EvaluationFilePath": "sc-optimize-prod-876399999956/Evaluations/2023/01/18/xxxxxxxx-66a1-42e3-a0e5-xxxxxxxxxxxx-john.smith@yourcompany.com-mike.mars@yourcompany.com.json"
}
Evaluation JSON payload field descriptions
Evaluation top-level fields
EvaluationId
A unique identifier for the Evaluation.
FormGroup
The Form Group the evaluation form used for this evaluation was part of.
FormName
The name of the Evaluation Form used for this evaluation.
FormVersion
The published version number of the Evaluation form used for this evaluation.
EvaluationDateTime
The initial date and time the evaluation was created, in UTC.
LastUpdated
The last date/time this evaluation payload was updated within the S3 bucket, in UTC. After initial creation in S3, an evaluation can sometimes be later updated (as the result of a modification made by the evaluator, for example, when resolving a dispute).
FormStatus
The status of the evaluation. Possible values are:
- NOT_DISPUTED - this is the normal status for an evaluation - when a new evaluation is created, this is the initial state.
- DISPUTED - only possible if Evaluation Disputes has been enabled. Indicates the user or agent being evaluated has disputed the evaluation, and it is waiting for review.
- RESOLVED_AMENDED - only possible if Evaluation Disputes has been enabled. After an evaluator has reviewed a dispute, they change the state of the evaluation to either this state, or RESOLVED_SUSTAINED (below).
- RESOLVED_SUSTAINED - only possible if Evaluation Disputes has been enabled. After an evaluator has reviewed a dispute, they change the state of the evaluation to either this state, or RESOLVED_AMENDED above.
- VOIDED - the evaluation has been voided.
FormDescription
The description text from the Evaluation Form used for this evaluation.
FormComments
The overall feedback notes added to the evaluation by the Evaluator when performing the evaluation.
ActionThreshold
The (optional) Action Threshold score value configured on the Evaluation Form used for this evaluation
FormTotalScore
A numeric value containing the total number of points (the score) the user or agent was awarded for this evaluation.
FormTotalPossibleScore
A numeric value containing the total possible score that could be achieved based on the Evaluation Form used for this evaluation.
Evaluator
The Symbee Connect Administration User Id of the Evaluator that performed the evaluation.
EvaluationUser
The User Id of the user or agent being evaluated.
UserReportHierarchy
The reporting hierarchy of the user or agent being evaluated.
InteractionId
The unique Id of the interaction that was being reviewed as part of this evaluation. For an Amazon Connect delivered contact (for example, a phone or chat contact) this value will match the Amazon Connect ContactId. For a Symbee Connect delivered interaction, this value will match the Symbee Connect GatewayTaskId.
InteractionDateTime
The initiation date/time (in UTC) of the interaction that was being reviewed as part of this evaluation.
InteractionType
The type of the interaction that was being reviewed as part of this evaluation. Currently valid values are:
- VOICE
- ROUTED_EMAIL
- WEBCHAT
- SMS
- SMS_PINPOINT
- SMS_WEBHOOK
- PREVIEW_DIAL
- SOCIAL_WHATSAPP
- SOCIAL_LINE
- SOCIAL_FBMESSENGER
- SOCIAL_TWITTERDM
- SOCIAL_OTHER
- INSTANT_MESSAGE
- UC_CALL
- GENERIC_TASK
InteractionLength
The duration (in seconds) of the interaction that was being reviewed as part of this evaluation.
InteractionDirection
The direction of the interaction that was being reviewed as part of this evaluation. Valid values are:
- INBOUND
- OUTBOUND
- OTHER
InteractionFarEnd
Reference information about the far end party of the interaction that was being reviewed as part of this evaluation. For example, for a phone call interaction, this is the customer's calling phone number; or for an Email interaction this is the customer's email address.
InteractionQueue
The name of the Amazon Connect Queue or Symbee Connect Skill Queue the interaction that was being reviewed as part of this evaluation was delivered to the agent via.
Sections
The sections field is a JSON Array - each object within the array represents a section of the Evaluation Form that was used for this evaluation. See the next section for a description of each of the fields within Section objects.
Section fields
Each object in the Sections array has the following fields:
SectionName
The name of the Section configured in the Evaluation Form being used for this evaluation.
SectionOrder
A numerical value used for ordering, to present the sections in the intended order to the Evaluator.
SectionTotalScore
A numeric value containing the total number of points the user or agent was awarded for questions within this section.
SectionTotalPossibleScore
A numeric value containing the total possible number of points that could be achieved based on the Evaluation Form for the questions in this section.
Questions
The Questions field is a JSON Array - each object within the array represents a question within the current section of the Evaluation Form that was used for this evaluation. See the next section for a description of each of the fields within Question objects.
Question fields
Each object in the Questions array has the following fields:
QuestionName
The Question Identifier (a short text string) assigned to this question in the Evaluation Form being used for this evaluation.
QuestionOrder
A numerical value used for ordering, to present the questions within the section in the intended order to the Evaluator.
QuestionText
The actual question text presented on the evaluation form for this question.
QuestionExplanation
Optional additional text associated with the question that may have been configured in the Evaluation Form for this question.
QuestionType
The type of response allowed for this question. Possible values are:
- YESNO
- RATING1TO5
- RATINGUNBOUND
- PICKLIST
- FREETEXT
QuestionScale
A numeric value containing the number of points that could have been awarded for this question, considering whether the question was required or not, and whether or not the evaluator provided a response of N/A (Not Applicable).
The difference between this value and the QuestionPotentialPoints value below is, when a question is answered with N/A its possible score is removed from the SectionTotalPossibleScore and FormTotalPossibleScore values which reduces the possible score the user or agent could have received for the evaluation.
QuestionPotentialPoints
A numeric value containing the possible number of points that could have been awarded for this question if it was not answered with an N/A (Not Applicable) response.
QuestionRequired
A boolean (true/false) value indicating whether a response was required for this question or not.
QuestionResponse
A text string containing the actual (unscaled) response the evaluator chose or entered as an answer to this question.
QuestionScore
A numeric value containing the actual (scaled) true number of points awarded for this question by the evaluator.
QuestionComments
A text string containing any additional feedback comments the evaluator provided for the user or agent being evaluated when they answered this question.
QuestionAvailableOptionInfo
Will be null unless the QuestionType is one of: PICKLIST or RATINGUNBOUND.
When QuestionType is PICKLIST, the value is the (pipe-delimited) list of choices provided to the evaluator for selection, and for each choice, the number of points awarded for that choice (the choice label and point value is separated by comma).
When QuestionType is RATINGUNBOUND, the value is the minimum and maximum allowed values, separated by a dash (minus sign).