Problem with expiration of intermediate certificate on our certificate chain
Incident Report for GrapheneDB
Postmortem

Timeline

May 30th, 2020. All times are in UTC.

  • 12:52 UTC: A customer reached out to support pointed out a problem connecting to a database
  • 13:05 UTC: We identified it as a problem with certificates and raised an internal incident
  • 13:25 UTC: A working group of 5 persons was formed to resolve the incident.
  • 14:00 UTC: The problem was identified as an expiration of one of our intermediate certificates in our certificate chain. We started testing the suggested resolution from our reseller for this issue.
  • 14:24 UTC: We applied a patch to production to prevent the problem from happening in the future. We also started patching affected instances from customers that reached out.
  • 15:40 UTC: Our team started working on automating the update all affected Standard and Performance stations.
  • 18:00 UTC: We started launching the automated process to resolve the issue on the affected instances in batches and were able to resolve the problem on all the affected production-ready instances (Standard and Performance).

June 1st, 2020. All times are in UTC.

  • 13:56 UTC We finished patching also all our Hobby instances.

Root cause

One root certificate used to sign one of the intermediate certificate that we use on our certificate chain expired, resulting in certification errors for some customers connecting from older systems.

We updated our chain certificate following the instructions of our certification reseller and applied the patch to all affected stations.

Remediation

We were not notified by our reseller about the upcoming expiration, thus one of the actions we're going to take is change reseller for all certificates of new plans, focusing on better support and small certificate chain to simplify management of expirations

We're also adding a mechanism to get notified of expiration of intermediate certificates internally.

Posted Jun 30, 2020 - 17:39 BST

Resolved
This incident has been resolved.
Posted Jun 01, 2020 - 14:55 BST
Monitoring
We've applied the patch to all affected databases. We're monitoring this closely.
Posted May 30, 2020 - 21:54 BST
Identified
We have identified today a problem with our certificates that was affecting a small number of customers. We are already applying a patch to all deployments to make sure nobody suffers from this issue in the future. Thanks!
Posted May 30, 2020 - 17:06 BST
This incident affected: Hobby Database Tier, Standard Database Tier, and Performance Database Tier.