Service Information

Last update: 4th November, 2024

This page will be updated with details of any major incidents or scheduled maintenance affecting the Pando services.

Details about past incidents can be found below, along with uptime and API performance information.

Uptime

MonthUptime
November 2024100%
October 2024100%
September 2024100%
August 2024100%
July 2024100%
June 2024100%
May 2024100%
April 2024100%
March 2024100%
February 2024100%
January 2024100%
December 2023100%
November 2023100%
October 2023100%
September 2023100%
August 2023100%
July 202399.92%
June 2023100%
May 2023100%
April 2023100%
March 2023100%
February 2023100%
January 2023100%
December 2022100%
November 202299.8%
October 2022100%
September 202299.9%
August 2022100%
July 2022100%
June 2022100%
May 2022100%
April 2022100%
March 2022100%
February 2022100%
January 2022100%
December 2021100%
November 2021100%
October 2021100%
September 2021100%
August 202199.988%
July 202199.982%
June 2021100%
May 202199.913%
April 202199.997%
March 202199.990%
February 202199.973%
Uptime/SLA as defined by standard Pando commercial contract.
Data published from February 2021 onwards.

Performance   

MonthAverage API response timeAverage messaging system response time
November 2024144ms5.0ms
October 2024148ms5.3ms
September 2024282ms6.3ms
August 2024190ms6.3ms
July 2024187ms6.7ms
June 2024182ms6.9ms
May 2024181ms6.8ms
April 2024210ms6.5ms
March 2024168ms6.5ms
February 2024170ms6.9ms
January 2024194ms6.4ms
December 2023183ms6.0ms
November 2023190ms6.2ms
October 2023190ms6.1ms
September 2023183ms5.5ms
August 2023171ms5.3ms
July 2023217ms7.0ms
June 2023169ms8.7ms
May 2023167ms7.8ms
April 2023145ms6.0ms
March 2023141ms6.1ms
February 2023126ms5.9ms
January 2023124ms5.4ms
December 2022114ms5.6ms
November 2022175ms5.8ms
October 202297ms4.5ms
September 2022168ms7.7ms
August 2022134ms4.6ms
July 2022123ms4.3ms
June 2022130ms4.2ms
May 2022210ms4.3ms
April 2022138ms4.1ms
March 2022102ms3.5ms
February 2022101ms3.4ms
January 2022105ms3.5ms
December 2021100ms3.2ms
November 2021114ms3.5ms
October 202196ms3.9ms
September 2021110ms3.9ms
August 2021140ms6.2ms
July 202184ms5.9ms
June 202180ms5.0ms
May 2021111ms4.0ms
April 202172ms3.7ms
March 202181ms3.5ms
February 2021190ms4.2ms
January 2021153ms3.3ms
December 2020121ms2.8ms
November 2020111ms2.7ms
October 202090ms2.6ms
September 202072ms2.6ms
Target (non-contractual) maximum time 200ms for API response, 50ms for messaging system response.
Data published from September 2020 onwards.

Past incidents (all times are Europe/London)

10 July 2023 – Pando Service Outage

The Pando Service was unavailable for some users between 08:08 and 08:41. A traffic spike caused a performance degradation on the database server. Mitigation measures have been applied to prevent this issue from reoccurring.
11 July 2023 15:03:00

23 November 2022 – Pando Service Outage

The Pando Service was unavailable from 08:02:00 to 09:19:00. Performance degradation following software upgrade caused a database server to become unresponsive. Mitigation measures have been applied to prevent this issue from reoccurring.
23 November 2022 15:00:00

12 September 2022 – Pando Service Outage

The Pando Service was unavailable from 9:07 to 10:15. A database server outage was the reason for the platform to become unresponsive. Mitigation measures have been applied to prevent this issue from reoccurring.
12 September 2022 14:00:00

6 March 2022 – Messaging Platform Degradation

We encountered temporarily reduced performance in our messaging system that caused problems with message delivery from Mar 6, 2022 8:36:07 AM till Mar 9, 2022 5:58:08 PM. For this time period message delivery may have been impacted for a small number of users.
10 March 2022 09:00:00

31 August 2021 – API Performance Degradation & Short Outage

A database server issue caused a period of degraded API performance between 10:18:00 and 10:21:00. Message delivery was unaffected during this time, but users would have experienced delays in logging in, searching and patient cards. There was a short outage between 10:31:11 and 10:31:41 to apply mitigation measures to prevent this from re-occurring.
31 August 2021, 10:35:00

24 August 2021 – API Performance Degradation

A database server issue caused a period of degraded API performance between 08:55:56 and 09:08:00. Message delivery was unaffected during this time, but users would have experienced delays in logging in, searching and patient cards.
24 August 2021, 11:00:00

7 July 2021 – Pando Messaging System Outage

The Pando Clinical Messaging System was unavailable from 16:04:48 to 16:12:46. Users were able to log in, but were unable to send or receive messages during this time.
8 July 2021, 08:00:00

10 May 2021 – Pando Service Outage

The Pando Service was unavailable from 12:36:00 to 13:14:00. A software error caused a database server to become unresponsive. Mitigation measures have been applied to prevent this issue from reoccurring.
10 May 2021, 17:08:30

7 April 2021 – Short Unplanned Outage

The Pando Service was disrupted from 16:14:18 to 16:15:34. A configuration error prevented API operations from working correctly. Some message deliveries were delayed. Full normal service was quickly restored. Mitigation measures have been applied to prevent this issue from reoccurring.
7 April 2021, 16:55:00

8 March 2021 – Severe Performance Degradation

The Pando Service was disrupted from 09:53:30 to 09:57:00. A scheduled daily job failed to run correctly causing database performance problems. The job should have taken one or two seconds to run, but instead took six minutes. The problem with the daily job was quickly identified and fixed, and will not reoccur.
9 March 2021, 11:00:00

2 March 2021 – Short Messaging System Outage

All Messaging Services were unavailable from 11:47:10 to 11:47:50, during which time users were unable to send or receive messages. This was caused by a configuration change being pushed out to all messaging servers simultaneously rather than the servers being updated one-by-one. The problem with the deployment system was quickly identified and fixed, and will not reoccur.
2 March 2021, 16:00:00

1 February 2021 – Pando API Outage

EC2 and RDS failures in the AWS euw-az3 availability zone caused a failure in the Pando API at 11:08:00. Messaging services were unaffected for logged in users. Although the API has been engineered (and demonstrated) to be tolerant of availability zone failure, this particular set of circumstances was unusual and unforeseen. Full service was restored at 11:19:55, and we have subsequently extended our ability to cope with AZ failure.
9 February 2021, 09:35:00

Incident history published from February 2021 onwards.

Advisory – Apache Log4J vulnerability

Forward Clinical Ltd trading as Pando and Juno was alerted (Saturday 18th December 2021) to a Common Vulnerabilities and Exposures notice (CVE-2021-45105) that Apache Log4j2 does not always protect from infinite recursion in lookup evaluation.

This means that Log4j2 is vulnerable to potential Denial of Service attacks.
References

Forward Clinical Ltd treated this as an emergency/critical issue. We have conducted a full dependency analysis on the Android APKs that our products utilise alongside their transitive dependencies. None of them use Apache Log4j2.

Any Amazon Web (AWS) services that use or used Apache Log4j2 will be subject to patching by AWS.

In this respect, we consider that the Pando and Juno services are not vulnerable to this exploitation.