May 30th, 2020. All times are in UTC.
- 12:52 UTC: A customer reached out to support pointed out a problem connecting to a database
- 13:05 UTC: We identified it as a problem with certificates and raised an internal incident
- 13:25 UTC: A working group of 5 persons was formed to resolve the incident.
- 14:00 UTC: The problem was identified as an expiration of one of our intermediate certificates in our certificate chain. We started testing the suggested resolution from our reseller for this issue.
- 14:24 UTC: We applied a patch to production to prevent the problem from happening in the future. We also started patching affected instances from customers that reached out.
- 15:40 UTC: Our team started working on automating the update all affected Standard and Performance stations.
- 18:00 UTC: We started launching the automated process to resolve the issue on the affected instances in batches and were able to resolve the problem on all the affected production-ready instances (Standard and Performance).
June 1st, 2020. All times are in UTC.
- 13:56 UTC We finished patching also all our Hobby instances.
One root certificate used to sign one of the intermediate certificate that we use on our certificate chain expired, resulting in certification errors for some customers connecting from older systems.
We updated our chain certificate following the instructions of our certification reseller and applied the patch to all affected stations.
We were not notified by our reseller about the upcoming expiration, thus one of the actions we're going to take is change reseller for all certificates of new plans, focusing on better support and small certificate chain to simplify management of expirations
We're also adding a mechanism to get notified of expiration of intermediate certificates internally.