KCF SMARTdiagnostics Unplanned Service Degradation

Incident Report for KCF Technologies

Postmortem

Our cloud provider has provided a postmortem on their outage, which may be viewed here:
https://aws.amazon.com/message/11201/

We sincerely apologise for any inconvenience this outage may have caused. KCF’s engineering team worked around the clock to mitigate the impact of the cloud provider’s outage, and service was restored as soon as possible. We deployed extra server resources to process the accumulated backlog of sensor data during the incident, and were able to restore service without losing any sensor data.

While we believe this outage to be a rare, isolated incident, KCF is working on architectural changes to SMARTdiagnostics to be more resilient if such an event occurs again.

Posted Nov 30, 2020 - 15:29 EST

Resolved

All issues with the cloud provider have been resolved. The backlog of sensor data has been fully processed, and SMARTdiagnostics has returned to normal operation.

Posted Nov 26, 2020 - 16:54 EST

Monitoring

The cloud provider has begun to restore service. SMARTdiagnostics is beginning to recover. Due to the duration of the outage, there is an extensive backlog of sensor data to process. Customers may see gaps in sensor data until the backlog is fully processed. We will continue to monitor the situation and make back-end adjustments to SMARTdiagnostics as necessary.

Posted Nov 25, 2020 - 17:25 EST

Update

The cloud provider has not yet resolved the issues and has not yet provided an ETA. SMARTdiagnostics is currently completely offline. We will provide updates when available. Base stations will continue to buffer sensor data and re-transmit when service is restored.

Note this a region-wide issue affecting a major cloud provider. Customers should be advised that numerous other websites and Internet-based services are affected as well.

Posted Nov 25, 2020 - 13:27 EST

Identified

SMARTdiagnostics is currently under degraded performance due to an issue with our cloud service provider affecting multiple services. Users may see intermittent errors while logging into the SMARTdiagnostics website. KCF's engineers are monitoring the situation and are in contact with the cloud hosting provider. We will post updates when available.

Posted Nov 25, 2020 - 10:28 EST

This incident affected: SMARTdiagnostics.