Partial Outage: Reports / KPI System

Incident Report for SMRT Systems

Resolved

Together with the AWS operations team, we've monitored the reports / KPI database through the night without issues. Processes are being implemented to ensure report / KPI availability during future AWS datacenter outages. Please do not hesitate to reach to reach out to the SMRT support team if you have any questions or concerns regarding the report / KPI system.
Posted Mar 08, 2019 - 02:46 PST

Monitoring

The original database has been restored via a previous snapshot and we've synced all data was added after that point.
To further prevent any future outages we've also configured our KPI database to exist over multiple data-centers.
We will continue to monitor the results.
Posted Mar 07, 2019 - 20:23 PST

Update

We are continuing to work on a fix for this issue.
Posted Mar 07, 2019 - 14:02 PST

Update

All systems are 100% operational. KPI’s and Reports are generating with new information; however, a re-sync of the database is required to restore historical data to the reports. We will work with Amazon Web Services to resolve and update you once the re-sync is complete.
Posted Mar 07, 2019 - 14:00 PST

Update

We’re currently directing traffic to another data center to resolve issue, ETA is 30min.
Posted Mar 07, 2019 - 11:10 PST

Update

KPI/Reports are still experiencing issues, we are in constant contact with AWS to resolve the datacenter issues.
Posted Mar 07, 2019 - 10:24 PST

Identified

Our provider Amazon Web Services has confirmed one of their datacenters are experiencing an outage, we're currently redirecting traffic to other datacenters to bring the system back to normal. While reports/KPIs are experiencing issues, mission critical components such as POS and Assembly should operate normally.
Posted Mar 07, 2019 - 08:35 PST

Investigating

We're currently investigating an issue where many Reports & KPIs are failing to load.
Posted Mar 07, 2019 - 07:49 PST
This incident affected: SMRT POS (Reporting / KPI System).