Authenticator based MFA not receiving notification

Studentnet apologises for this morning’s failure to send 2nd factor authentication notifications via the Cloudwork generic authenticator app.

The root cause of this incident was a failure in our system for monitoring the certifications of one of our app environment providers, in this case Apple for iOS apps. This failure then had down stream consequences that affected all users of the Cloudwork generic app regardless of environment. The details can be found below.

Root Causes

• Studentnet has a management system to maintain and monitor the status of perishable assets. These assets include:

  • certificates
  • domain names
  • security credentials
  • skill credentials
  • licences etc

• Studentnet’s management system sends reminders to relevant personnel at 30, 14, 7 days and 1 day intervals prior to a perishable asset expiring.

• The Apple play store certificate for the Cloudwork generic authenticator iOS app is such a perishable asset. It was last renewed in September 2023.

• No entry for either the creation of the asset or its renewal in September 2023 was created in the Studentnet management system.

• Renewal notices for the expiring certificate were not generated and sent to personnel.

• The Apple play store certificate for the Cloudwork generic authenticator iOS app expired over the weekend of September 7&8, 2024.

• Upon expiry, any MFA request involving an iOS device failed to generate a 2nd factor notification stopping successful login by the user.

• The repeated failed iOS notification attempts backed up in the Cloudwork processing queues ultimately blocking successful sending of all notification regardless of processing environment. This meant that Android users also commenced to fail to receive notifications.

Remediation Taken

• The certificate for the Cloudwork generic authenticator app was renewed with Apple Push Notification Services
• The Cloudwork instances for affected schools were redeployed to utilise the renewed certificate
• An entry for the Apple play store Cloudwork iOS certificate was created in the Studentnet asset management system
• All schools were inspected to ensure:

1. the expiry dates for relevant assets were accurate
2. entries for all app certificates existed in the Studentnet management system

• All schools were contacted advising of redeploys
• Schools were reminded that any emergency outage such as this should immediately be notified to Studentnet even over the weekend.

Safeguards Taken

• All Studentnet staff are receiving refreshed training on:

  • how certificates are used within app processing environments
  • how to create and maintain entries in the Studentnet management system

• An audit of all certificates used within Studentnet will be undertaken

• More stringent, and thorough, certificate management workflows are being internally documented and enforced with regular diligence inspections

Logging an out of hours support call with Studentnet

• Some schools reported issues with logging out o hours calls with Studentnet at 8am on Monday 9/9/24
• Studentnet has moved to a new phone system necessitating further adjustment in the manner in which out of hours phone calls are handled
• Studentnet business hours will be adjusted to 8am to 5pm Monday to Friday