Responding to System Incidents: Reporting, Repair, Notification and Post-Incident Reports

A system incident needs a handling process that is settled beforehand: who takes over, when an issue counts as an incident, how it is reported, and how users are notified after the fix are all written down. POLISH sets reporting, reproduction, the interim workaround, the correction and notification as one fixed path, so the project team knows the next step at any point in time. The review afterward assigns no blame and keeps its focus on contributing factors and improvement items. When a public-sector system is maintained under contract, the process also has to connect with the supervising agency's reporting procedures, so the explanatory statement and the change record are part of the process too.

Which situations are declared incidents

An ordinary report and an incident call for different levels of handling. Writing the declaration criteria down in advance spares those handling the issue from making a judgment call under pressure. Stribblehill, 2017, in the fourteenth chapter of Google's Site Reliability Engineering book, explains that the more explicit the criteria for declaring an incident, the better, and gives three examples, any one of which makes an issue an incident: a second team needs to be involved, the impact is visible to users or customers, or the issue remains unresolved after one hour of focused analysis. The chapter describes Google's scale, and a small project can adjust the thresholds to the size of the team.

The criteria are written in terms of what users can see, such as a page that will not open, a form that cannot be submitted, or data that displays inconsistently. Once settled, the criteria are written into the maintenance agreement together with the severity levels and targets of the SLA, so that the project team and the engineering team judge by the same basis.

Every report follows the same path from intake to closure

A report that does not meet the incident criteria still needs to be logged, owned and closed with a reply. POLISH sets reporting, reproduction, additional information, the interim workaround, the correction and notification as one process, and the diagram below marks who is responsible for each step.

1. After the project team receives a call or email from field staff or a user, the report is first logged in the shared report list, with the page, the time of occurrence and a screenshot, along with the reason it needs priority.
2. A comment is left in the shared thread naming the engineering lead and attaching the list item, and the engineering team first replies that the report is received and under investigation.
3. An engineer first tries to reproduce the problem in the engineer's own environment. If it cannot be reproduced, the reporter is asked to supply the time of the operation and a screenshot of what was entered, or a video of the test case is attached with a request for the other side to add the scenario.
4. A workaround and an expected completion time are given, for example adjusting the browser's permissions for the site so that operations on site can continue for the time being.
5. The fix goes to the test site first and, once confirmed, to the production site.
6. A reply in the original thread states that the update is live, so the report and its outcome stay in one place.

For a report that needs manual data, the project team collects supporting evidence from users, and the engineering team handles it in the admin back end and then replies; where necessary, the project team confirms the original state with users so that it can be restored. This pattern continues throughout the long-term maintenance period.

A workaround addresses only the moment. For problems of the same kind, POLISH also feeds the lesson back into the page, adding clearer checks and prompts.

Problems found during testing are managed with a numbered list. The engineer marks the handling status number by number: those that can be reproduced are handled, those that cannot are given a test-case video so that the other side can add the scenario, and those that are prompt-related get an explanatory popup first. After handling, the engineering lead groups the causes under the issue numbers and explains how each was aligned, and finally sends a one-line notice that the new version is done, asking the product side to confirm.

For problems that span the front end and the back end, POLISH posts the scenario and screenshots in the thread and names the developers responsible for both ends, who first agree on the basis for the diagnosis and then make the fix. After a new version is released, POLISH retests and records the results in the quality checklist. Report handling, periodic checks and maintenance reports during routine maintenance are covered in routine maintenance after launch.

Roles and communication are assigned beforehand

When a problem occurs in a system, POLISH will assign staff to handle it immediately. Who takes over, who explains to outside parties and who confirms with users are written into the process document beforehand.

Stribblehill, 2017 describes the hazards common in incident handling: focusing only on the technical problem, poor communication, and unilateral changes to the system. The practice the chapter recommends separates four roles, command, operations, communications and planning, sets up a fixed command post, maintains a live incident document, and prepares and rehearses the process in advance.

Google operates at a far larger scale than a typical project. In a small team one person may hold two roles, provided it is written down who is in command and who explains to outside parties. POLISH's approach is a single fixed thread in which assessments, decisions and progress are all recorded in one place, with the parts handled by the project team and by the engineering team listed separately in the process document.

Items on the host side are passed by the project team to the host's maintenance contact, with the operating window confirmed in advance.

Explanatory statements and change records for the supervising agency

When a public-sector system is maintained under contract, the order of handling for security notices forwarded by the supervising agency is fixed in advance as follows.

1. The project team forwards the notice email and names the engineering lead in the thread.
2. POLISH first replies that handling is under way.
3. The engineer determines the source and drafts an explanatory statement for the supervising agency's contact, containing the preliminary cause, the direction for reinforcement and the method of retesting.
4. The project team reviews it and sends it.
5. The project team asks about the repair schedule, and POLISH gives an expected completion date.
6. After the repair is made and the production site is updated, POLISH reports back.
7. For the parts involving code changes, a change record is filled in as the security management requirements call for, for later audit review.

Before any repair step, logs and screen captures are preserved as the basis for the investigation. The business section of the Administration for Cyber Security, Ministry of Digital Affairs, 2026 explains that when a cybersecurity incident occurs at an agency, the agency should report and respond immediately under the regulations, quickly complete damage control or recovery, and ensure that traces and evidence are preserved; the Administration also publishes an operating procedure for agencies' incident reporting and response, for agencies to refer to. The engineering team's fields and retained records follow the supervising agency's operating procedure.

The contractor is also within the scope of the law. Article 7 of the Enforcement Rules of the Cybersecurity Management Act provides that when an agency outsources the building or operation of an information system or the provision of information services, selecting and supervising the contractor should take into account matters such as staffing, subcontracting, security testing of customized systems, incident reporting and the handling of data when the contract ends, and that the contractor must be audited periodically, or reviewed when an incident becomes known. When the contractor is audited, the change record is material that can be provided directly, so it is completed together with the release of the repair.

When results from an external test or vulnerability scan come in, the engineering lead posts a screenshot of the report in the thread with the handling deadline marked, first determines the cause, and then proposes one or two options, such as adjusting a default value, switching to a simplified error page or showing plain text only, and asks the developer to confirm that nothing else will break as a result. After adjusting, the developer first releases the corrected version and then replies that it has been handled, and the engineering lead rechecks. Details of remediation priority, appeals against false positives and retesting are in after a vulnerability scan report arrives. What evidence a public-sector project needs to keep is covered in delivery and maintenance evidence for public-sector projects.

Backup restoration drills start by asking who sets the plan

A restoration drill verifies that a backup can in fact be restored when needed, and leaves the evidence the supervising agency requires. The drill involves the host side, so the first step is to ask who sets the plan.

1. POLISH first asks the project team whether the engineering team should propose the drill plan or the supervising agency already has a version to be adjusted. When the supervising agency provides notes and a procedure, POLISH prepares according to that material.
2. The project team first lists the requirements: after the standby system is activated, the team must verify that the data is correct, and the verification steps, the comparison method (record counts, content, file sizes) and the screenshots used as evidence are written down.
3. POLISH schedules the drill window and invites the project team to take part.
4. During the drill, the previous night's backup is restored, the service is confirmed to be running normally, and the record counts and the number of uploaded files are compared.
5. The project team fills in the restoration record test form and attaches screenshots taken after the restoration; how the form is written is adjusted to the actual architecture of the project.
6. The drill records and evidence are placed on the shared cloud drive, and a reachable test URL is provided so that screenshots can be added.
7. Only after the project team confirms that the data volume is sufficient is the host's maintenance contact asked to delete the drill host and the related settings.

The drill environment is cleaned up last, and the evidence must be complete before cleanup. Nelson et al., 2025 is incident response guidance published by NIST in the United States; it places incident response within an organization's cybersecurity risk management, covers the whole span from preparation to recovery, and can serve agencies in planning incident response operations. A backup restoration drill is part of advance preparation.

Post-incident reports assign no blame and record contributing factors

Lunney and Lueder, 2017 argue for blameless postmortems that focus on the contributing factors of an incident and do not blame an individual or a team. The chapter holds that people hide problems when punishment is feared, so systems and processes should be improved, and the report itself is never used as grounds for punishment.

Allspaw, 2012 adds the engineer's perspective: a blameless review lets the engineers involved describe in detail, without fear, the actions taken, the observations made, the expectations held and the assumptions at the time. People who fear punishment withhold details, the mechanism of the incident cannot be understood, and the same incident will inevitably recur. The article also offers the "second story", which treats human error as a result of weaknesses in the organizational system rather than as the cause of failure. Both sources come from engineering practice in internet services.

POLISH's practice is to prepare an investigation report from the logs once service is restored, with an improvement plan attached to the full version. The report covers the course of the incident, the scope of impact and the contributing factors; when the causes divide between the application side and the host side, each is listed and tracked separately. A post-incident report is recommended for every declared incident.

When the full process does not fit

• One-off pages limited to an event period, taken down when the event ends, with no later maintenance. Logging and a closing reply are enough; no post-incident report is needed.
• Trial builds or internal test environments with no real users yet. Reports can go through the test list, and neither declaration criteria nor a command role is needed.
• Systems whose maintenance contact and owner are not yet settled. Decide who takes over reports first, then turn to incident declaration; with no one to take over, even detailed criteria go unexecuted.

What to prepare before starting

• The location of the shared report list and the fixed thread, and who is authorized to log reports.
• The declaration criteria, written in terms of what users can see.
• A role roster: command, operations, communications and planning, plus the contacts for the host maintenance window and the supervising agency's window.
• The fields of the explanatory statement for the supervising agency, and the format of the change record.
• The responsible party, verification steps and evidence items for backup restoration drills.
• A template for the post-incident report, and the agreement that reviews assign no blame.

Under the Ministry of Education project, POLISH carried the work from the design tender through to the engineering tender and has maintained the Outdoor Education Resource Platform over the long term. POLISH will fold this checklist into the maintenance agreement of long-term maintenance and platform development and adjust the fields to the supervising agency's operating procedure. To compare an existing system against these items, contact POLISH and provide the current process documents, and the engineering team will check the documents item by item.