Last update: 2nd May, 2022
This page will be updated with details of any major incidents or scheduled maintenance affecting the Pando services.
Details about past incidents can be found below, along with uptime and API performance information.
Data published from February 2021 onwards.
|Month||Average API response time||Average messaging system response time|
Data published from September 2020 onwards.
Past incidents (all times are Europe/London)
6 March 2022 – Messaging Platform Degradation
We encountered temporarily reduced performance in our messaging system that caused problems with message delivery from Mar 6, 2022 8:36:07 AM till Mar 9, 2022 5:58:08 PM. For this time period message delivery may have been impacted for a small number of users.
10 March 2022 09:00:00
31 August 2021 – API Performance Degradation & Short Outage
A database server issue caused a period of degraded API performance between 10:18:00 and 10:21:00. Message delivery was unaffected during this time, but users would have experienced delays in logging in, searching and patient cards. There was a short outage between 10:31:11 and 10:31:41 to apply mitigation measures to prevent this from re-occurring.
31 August 2021, 10:35:00
24 August 2021 – API Performance Degradation
A database server issue caused a period of degraded API performance between 08:55:56 and 09:08:00. Message delivery was unaffected during this time, but users would have experienced delays in logging in, searching and patient cards.
24 August 2021, 11:00:00
7 July 2021 – Pando Messaging System Outage
The Pando Clinical Messaging System was unavailable from 16:04:48 to 16:12:46. Users were able to log in, but were unable to send or receive messages during this time.
8 July 2021, 08:00:00
10 May 2021 – Pando Service Outage
The Pando Service was unavailable from 12:36:00 to 13:14:00. A software error caused a database server to become unresponsive. Mitigation measures have been applied to prevent this issue from reoccurring.
10 May 2021, 17:08:30
7 April 2021 – Short Unplanned Outage
The Pando Service was disrupted from 16:14:18 to 16:15:34. A configuration error prevented API operations from working correctly. Some message deliveries were delayed. Full normal service was quickly restored. Mitigation measures have been applied to prevent this issue from reoccurring.
7 April 2021, 16:55:00
8 March 2021 – Severe Performance Degradation
The Pando Service was disrupted from 09:53:30 to 09:57:00. A scheduled daily job failed to run correctly causing database performance problems. The job should have taken one or two seconds to run, but instead took six minutes. The problem with the daily job was quickly identified and fixed, and will not reoccur.
9 March 2021, 11:00:00
2 March 2021 – Short Messaging System Outage
All Messaging Services were unavailable from 11:47:10 to 11:47:50, during which time users were unable to send or receive messages. This was caused by a configuration change being pushed out to all messaging servers simultaneously rather than the servers being updated one-by-one. The problem with the deployment system was quickly identified and fixed, and will not reoccur.
2 March 2021, 16:00:00
1 February 2021 – Pando API Outage
EC2 and RDS failures in the AWS euw-az3 availability zone caused a failure in the Pando API at 11:08:00. Messaging services were unaffected for logged in users. Although the API has been engineered (and demonstrated) to be tolerant of availability zone failure, this particular set of circumstances was unusual and unforeseen. Full service was restored at 11:19:55, and we have subsequently extended our ability to cope with AZ failure.
9 February 2021, 09:35:00
Incident history published from February 2021 onwards.
Advisory – Apache Log4J vulnerability
Forward Clinical Ltd trading as Pando and Juno was alerted (Saturday 18th December 2021) to a Common Vulnerabilities and Exposures notice (CVE-2021-45105) that Apache Log4j2 does not always protect from infinite recursion in lookup evaluation.
This means that Log4j2 is vulnerable to potential Denial of Service attacks.
Forward Clinical Ltd treated this as an emergency/critical issue. We have conducted a full dependency analysis on the Android APKs that our products utilise alongside their transitive dependencies. None of them use Apache Log4j2.
Any Amazon Web (AWS) services that use or used Apache Log4j2 will be subject to patching by AWS.
In this respect, we consider that the Pando and Juno services are not vulnerable to this exploitation.