Partial Outage: Reports / KPI System
Incident Report for SMRT Systems
Resolved
Together with the AWS operations team, we've monitored the reports / KPI database through the night without issues. Processes are being implemented to ensure report / KPI availability during future AWS datacenter outages. Please do not hesitate to reach to reach out to the SMRT support team if you have any questions or concerns regarding the report / KPI system.
Posted 7 months ago. Mar 08, 2019 - 02:46 PST
Monitoring
The original database has been restored via a previous snapshot and we've synced all data was added after that point.
To further prevent any future outages we've also configured our KPI database to exist over multiple data-centers.
We will continue to monitor the results.
Posted 7 months ago. Mar 07, 2019 - 20:23 PST
Update
We are continuing to work on a fix for this issue.
Posted 7 months ago. Mar 07, 2019 - 14:02 PST
Update
All systems are 100% operational. KPI’s and Reports are generating with new information; however, a re-sync of the database is required to restore historical data to the reports. We will work with Amazon Web Services to resolve and update you once the re-sync is complete.
Posted 7 months ago. Mar 07, 2019 - 14:00 PST
Update
We’re currently directing traffic to another data center to resolve issue, ETA is 30min.
Posted 7 months ago. Mar 07, 2019 - 11:10 PST
Update
KPI/Reports are still experiencing issues, we are in constant contact with AWS to resolve the datacenter issues.
Posted 7 months ago. Mar 07, 2019 - 10:24 PST
Identified
Our provider Amazon Web Services has confirmed one of their datacenters are experiencing an outage, we're currently redirecting traffic to other datacenters to bring the system back to normal. While reports/KPIs are experiencing issues, mission critical components such as POS and Assembly should operate normally.
Posted 7 months ago. Mar 07, 2019 - 08:35 PST
Investigating
We're currently investigating an issue where many Reports & KPIs are failing to load.
Posted 7 months ago. Mar 07, 2019 - 07:49 PST
This incident affected: SMRT POS (Reporting / KPI System).