Together with the AWS operations team, we've monitored the reports / KPI database through the night without issues. Processes are being implemented to ensure report / KPI availability during future AWS datacenter outages. Please do not hesitate to reach to reach out to the SMRT support team if you have any questions or concerns regarding the report / KPI system.
Posted Mar 08, 2019 - 02:46 PST
Monitoring
The original database has been restored via a previous snapshot and we've synced all data was added after that point. To further prevent any future outages we've also configured our KPI database to exist over multiple data-centers. We will continue to monitor the results.
Posted Mar 07, 2019 - 20:23 PST
Update
We are continuing to work on a fix for this issue.
Posted Mar 07, 2019 - 14:02 PST
Update
All systems are 100% operational. KPI’s and Reports are generating with new information; however, a re-sync of the database is required to restore historical data to the reports. We will work with Amazon Web Services to resolve and update you once the re-sync is complete.
Posted Mar 07, 2019 - 14:00 PST
Update
We’re currently directing traffic to another data center to resolve issue, ETA is 30min.
Posted Mar 07, 2019 - 11:10 PST
Update
KPI/Reports are still experiencing issues, we are in constant contact with AWS to resolve the datacenter issues.
Posted Mar 07, 2019 - 10:24 PST
Identified
Our provider Amazon Web Services has confirmed one of their datacenters are experiencing an outage, we're currently redirecting traffic to other datacenters to bring the system back to normal. While reports/KPIs are experiencing issues, mission critical components such as POS and Assembly should operate normally.
Posted Mar 07, 2019 - 08:35 PST
Investigating
We're currently investigating an issue where many Reports & KPIs are failing to load.
Posted Mar 07, 2019 - 07:49 PST
This incident affected: SMRT POS (Reporting / KPI System).