DC Connectivity Issues
Incident Report for Studentnet
Postmortem

Studentnet Cloudwork

Post Incident Report - 2021-01-29

Descriptive Name: SSO connection incident

Incident Reference Number: 20210129

Date of Incident: 27/01/2021 Wednesday, 28/01/2021 Thursday & 29/01/2021 Friday

Time & Duration of incident:

  • 27/1/2021:

    • 09:30-10:22, 52mins
    • 11:51-12:48, 57mins
    • 20:15-22:41, 2hrs 26mins - intermittent disruption to small number of schools
  • 29/1/2021:

    • 07:55-08:10, 15mins
    • 13:55-14:30, 35mins

Severity: - Service Availability Affecting

Locations Affected: All schools for all occurrences except 2021-01-27-20:15 incident

Services Affected:

  • SSO
  • MFA
  • Password Reset
  • Web site

Incident Cause:

  • Sustained Distributed Denial of Service attack directing enormous traffic loads from all over Asia originating addresses specifically targeted at the Cloudwork service delivery IP address and port

Incident Resolved: Yes

Date & Time of Resolution: 2021-02-05-15:00

Issued By: Technical Support, support@studentnet.net

PIR Issue Date: 2021-02-05

Contact Information: Please report any continued service disruption immediately to :

**Studentnet NOC Support**: \+61 2 9281 3905 

**Support Email**: [_support@studentnet.net_](mailto:support@studentnet.net) or [_support@coherentcloud.com_](mailto:support@coherentcloud.com)

Incident Description

  • 27/01/2021

    • 09:30 - Studentnet monitoring alerts that connectivity to DC has been lost for all external services. Initial reports appear to be hardware not coping with traffic volume. Initially observed traffic was believed to be legitimate. This later proved on Friday 29/1/2021 to be incorrect
    • 11:55 - Studentnet monitoring recommences alerts that connectivity to DC has been lost for all external services. Incident again mis-diagnosed to be hardware/traffic related.
    • 14:00 - Replacement higher capacity hardware installed
    • 20:00 - Configuration of new hardware inappropriate for a small number of schools causing new outage for those schools. Configuration is adjusted rectifying symptoms
  • 29/01/2021

    • 07:55 - Monitoring again advises that connectivity is failing
    • 13:55 - Monitoring again advises that connectivity is failing. Closer analysis of traffic for first time identifies high volume of malicious traffic originating from Asian IP addresses specifically targeted at an IP address in the Studentnet range on the port used to deliver the service. NTT contacted. NTT confirm observation of malicious traffic and null route all packets destined for the Studentnet IP address. Connectivity instantly is re-established
    • 14:30 - Studentnet starts blocking malicious traffic from IPs that participated in the attack
    • 17:00 - As a registered partner of the ACSC Studentnet reports DDoS attack to Australian Signals Directorate(ASD) of the Dept of Defence flagging report to be forwarded to Australian Federal Police(AFP)
    • 18:00 - NTT contacts Studentnet advising DPS Max DDoS auto-mitigation solution. NTT commences to establish commercial arrangement to implement DPS Max.
    • 23:30 - NTT completes establishment of commercial arrangement to implement DPS Max.
  • 30/01/2021

    • NTT USA commences implementation of DPS Max
  • 31/01/2021

    • 16:30 - NTT USA completes implementation of DPS Max. Configuration tuned for expected Monday morning legitimate traffic

Analysis

  • A perpetrator has specifically targeted the Studentnet Cloudwork service employing a dark web botnet to flood the service with traffic with the aim of denying service to users of Cloudwork

  • The attack was highly targeted and very effective due to its sheer scale and volume but it was very unsophisticated in that it was just raw packets. In particular, there was no attempt to breach our security.

  • Studentnet network security was not breached.

  • There was no network intrusion.

  • There was no loss of data.

  • There was no extraction of any Cloudwork data out of our network.

Unresolved Questions

  • Will the perpetrator attempt again in the near future?
  • What was the motive for the attack?
  • Can the identity of the perpetrator be established?

Root Cause

  • DDoS attack

Recommendations/Preventative Measures

  • Continue to capture data with a view to tracking and establishing the identity of the perpetrator
  • Continue to fine tune the NTT DPS Max auto-mitigation facility
  • Continue to exercise the NTT DPS Max logging options

oOo

Posted Feb 05, 2021 - 14:52 AEDT

Resolved
This incident has been resolved. Postmortem to follow. Please report any issues immediately to the office (+61 2 9281 1626) during business hours and the NoC (+61 2 2981 3905) out of hours. Thank you.
Posted Feb 05, 2021 - 14:49 AEDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Feb 02, 2021 - 01:34 AEDT
Investigating
We are currently investigating this issue.
Posted Feb 01, 2021 - 23:48 AEDT
Update
NTT Security in the USA have now established our DPS Max DDoS protection for our connectivity. NTT DPS Max is configured and fully operational. All Studentnet systems are operational. Please report any issue immediately to our NoC on +61 2 9281 3905. Thank you.
Posted Feb 01, 2021 - 04:28 AEDT
Monitoring
NTT Australia have been engaged to establish automated DDoS mitigation on all of our network connectivity via their DPS Max product. NTT Security in the USA are now establishing our DPS Max instance. We expect to have the DPS Max instance configured and operational over this weekend. All systems are operational. Please report any issue immediately to our NoC on +61 2 9281 3905. Thank you.
Posted Jan 30, 2021 - 08:06 AEDT
Identified
DDoS attacks! From China, India, Indonesia etc!

We've just received confirmation from NTT Security of the attacks. They have implemented protections on our network connection. In addition we have upgraded our firewall rules to drop packets from suspect IP ranges.

That this should happen on the first week of the academic school year is too much of a coincidence. This incident will be reported to the Australian Cyber Security Centre (we are a registered partner) and the Australian Federal Police for further investigation.
Posted Jan 29, 2021 - 14:33 AEDT
Update
Confirmation has been received from our upstream provider(NTT) that we are being subjected to a DDoS attack with traffic being targetted at our network from China, India and Indonesia. The NTT security team are investigating and will be providing us with an update. We will be requesting geoblocking to be enforced limiting traffic to only domestic Australian traffic. There may be other options that the provide to us. Further update will follow.
Posted Jan 29, 2021 - 13:42 AEDT
Investigating
Outage has recommenced - no further information at this stage.
Posted Jan 29, 2021 - 13:17 AEDT
Update
We are continuing investigate this incident. We are reaching out to our upstream providers who may be able to assist with filtering of our traffic. We are also engaging with DevOps/Architecture consultants to undertake a root and branch review of our network solution. Bluntly put... we will not rest until this is resolved to everyone's satisfaction.
Posted Jan 29, 2021 - 12:24 AEDT
Identified
Services have been re-instated but investigation is continuing. Please immediately report any further issues to 02 9281 1626.
Posted Jan 29, 2021 - 09:28 AEDT
Investigating
We are currently investigating this issue.
Posted Jan 29, 2021 - 07:57 AEDT
This incident affected: System Core (Web site, DNS) and Identity Management (Cloudwork SmartID - Authentication SSO, Cloudwork AuthentID SMS-MFA/Password Reset).