Service Information

Last update: 5th November, 2022

This page will be updated with details of any major incidents or scheduled maintenance affecting the Pando services.

Details about past incidents can be found below, along with uptime and API performance information.

Uptime

MonthUptime
October 2022100%
September 202299.9%
August 2022100%
July 2022100%
June 2022100%
May 2022100%
April 2022100%
March 2022100%
February 2022100%
January 2022100%
December 2021100%
November 2021100%
October 2021100%
September 2021100%
August 202199.988%
July 202199.982%
June 2021100%
May 202199.913%
April 202199.997%
March 202199.990%
February 202199.973%
Uptime/SLA as defined by standard Pando commercial contract.
Data published from February 2021 onwards.

Performance   

MonthAverage API response timeAverage messaging system response time
October 202297ms4.5ms
September 2022168ms7.7ms
August 2022134ms4.6ms
July 2022123ms4.3ms
June 2022130ms4.2ms
May 2022210ms4.3ms
April 2022138ms4.1ms
March 2022102ms3.5ms
February 2022101ms3.4ms
January 2022105ms3.5ms
December 2021100ms3.2ms
November 2021114ms3.5ms
October 202196ms3.9ms
September 2021110ms3.9ms
August 2021140ms6.2ms
July 202184ms5.9ms
June 202180ms5.0ms
May 2021111ms4.0ms
April 202172ms3.7ms
March 202181ms3.5ms
February 2021190ms4.2ms
January 2021153ms3.3ms
December 2020121ms2.8ms
November 2020111ms2.7ms
October 202090ms2.6ms
September 202072ms2.6ms
Target (non-contractual) maximum time 200ms for API response, 50ms for messaging system response.
Data published from September 2020 onwards.

Past incidents (all times are Europe/London)

12 September 2022 – Pando Service Outage

The Pando Service was unavailable from 9:07 to 10:15. A database server outage was the reason for the platform to become unresponsive. Mitigation measures have been applied to prevent this issue from reoccurring.
12 September 2022 14:00:00

6 March 2022 – Messaging Platform Degradation

We encountered temporarily reduced performance in our messaging system that caused problems with message delivery from Mar 6, 2022 8:36:07 AM till Mar 9, 2022 5:58:08 PM. For this time period message delivery may have been impacted for a small number of users.
10 March 2022 09:00:00

31 August 2021 – API Performance Degradation & Short Outage

A database server issue caused a period of degraded API performance between 10:18:00 and 10:21:00. Message delivery was unaffected during this time, but users would have experienced delays in logging in, searching and patient cards. There was a short outage between 10:31:11 and 10:31:41 to apply mitigation measures to prevent this from re-occurring.
31 August 2021, 10:35:00

24 August 2021 – API Performance Degradation

A database server issue caused a period of degraded API performance between 08:55:56 and 09:08:00. Message delivery was unaffected during this time, but users would have experienced delays in logging in, searching and patient cards.
24 August 2021, 11:00:00

7 July 2021 – Pando Messaging System Outage

The Pando Clinical Messaging System was unavailable from 16:04:48 to 16:12:46. Users were able to log in, but were unable to send or receive messages during this time.
8 July 2021, 08:00:00

10 May 2021 – Pando Service Outage

The Pando Service was unavailable from 12:36:00 to 13:14:00. A software error caused a database server to become unresponsive. Mitigation measures have been applied to prevent this issue from reoccurring.
10 May 2021, 17:08:30

7 April 2021 – Short Unplanned Outage

The Pando Service was disrupted from 16:14:18 to 16:15:34. A configuration error prevented API operations from working correctly. Some message deliveries were delayed. Full normal service was quickly restored. Mitigation measures have been applied to prevent this issue from reoccurring.
7 April 2021, 16:55:00

8 March 2021 – Severe Performance Degradation

The Pando Service was disrupted from 09:53:30 to 09:57:00. A scheduled daily job failed to run correctly causing database performance problems. The job should have taken one or two seconds to run, but instead took six minutes. The problem with the daily job was quickly identified and fixed, and will not reoccur.
9 March 2021, 11:00:00

2 March 2021 – Short Messaging System Outage

All Messaging Services were unavailable from 11:47:10 to 11:47:50, during which time users were unable to send or receive messages. This was caused by a configuration change being pushed out to all messaging servers simultaneously rather than the servers being updated one-by-one. The problem with the deployment system was quickly identified and fixed, and will not reoccur.
2 March 2021, 16:00:00

1 February 2021 – Pando API Outage

EC2 and RDS failures in the AWS euw-az3 availability zone caused a failure in the Pando API at 11:08:00. Messaging services were unaffected for logged in users. Although the API has been engineered (and demonstrated) to be tolerant of availability zone failure, this particular set of circumstances was unusual and unforeseen. Full service was restored at 11:19:55, and we have subsequently extended our ability to cope with AZ failure.
9 February 2021, 09:35:00

Incident history published from February 2021 onwards.

Advisory – Apache Log4J vulnerability

Forward Clinical Ltd trading as Pando and Juno was alerted (Saturday 18th December 2021) to a Common Vulnerabilities and Exposures notice (CVE-2021-45105) that Apache Log4j2 does not always protect from infinite recursion in lookup evaluation.

This means that Log4j2 is vulnerable to potential Denial of Service attacks.
References

Forward Clinical Ltd treated this as an emergency/critical issue. We have conducted a full dependency analysis on the Android APKs that our products utilise alongside their transitive dependencies. None of them use Apache Log4j2.

Any Amazon Web (AWS) services that use or used Apache Log4j2 will be subject to patching by AWS.

In this respect, we consider that the Pando and Juno services are not vulnerable to this exploitation.