- Python CLI tool for rolling updates of CL-AppPipe-* and CL-SvcPipe-* stacks - Async update engine with configurable concurrency (asyncio.Semaphore) - Exponential backoff retry for API throttling - Dry-run mode for safe preview - IAM permission pre-validation - Comprehensive test suite (80 tests: 11 property-based + 69 unit) - Full spec documentation (requirements, design, tasks)
7 KiB
Requirements Document
Introduction
This feature provides a one-click mechanism to update 22+ CloudFormation stacks in an audit account. All stacks follow the CL-AppPipe-* naming convention (e.g., CL-AppPipe-9894aa72, CL-AppPipe-ca6dca90) and use the same nested template URL (https://s3.amazonaws.com/solutions-reference/centralized-logging-with-opensearch/latest/AppLogS3Buffer.template) from the Centralized Logging with OpenSearch solution (SO8025-s3b). The number of stacks grows over time as new pipelines are added. When a new version of that template becomes available, the operator needs a single action to trigger a rolling update across every stack without manually updating each one.
Glossary
- Stack_Updater: The system that orchestrates the discovery and update of all target CloudFormation stacks in the audit account.
- Target_Stack: A CloudFormation stack whose name starts with the
CL-AppPipe-prefix, belonging to the Centralized Logging with OpenSearch solution for log ingestion pipelines. - Stack_Name_Prefix: The string
CL-AppPipe-used to identify Target_Stacks by their CloudFormation stack name. - Nested_Template_URL: The S3 URL (
https://s3.amazonaws.com/solutions-reference/centralized-logging-with-opensearch/latest/AppLogS3Buffer.template) that always resolves to the latest version of the pipeline template. - Solution_ID: The identifier
SO8025-s3bfor the Centralized Logging with OpenSearch AWS Solution that provisions the Target_Stacks. - Update_Run: A single execution of the Stack_Updater that attempts to update all Target_Stacks.
- Operator: The person who triggers and monitors the update process.
- Concurrency_Limit: The maximum number of CloudFormation stack updates that the Stack_Updater executes in parallel during an Update_Run.
Requirements
Requirement 1: Discover Target Stacks
User Story: As an Operator, I want the system to automatically discover all stacks matching the pipeline naming convention, so that I do not have to maintain a manual list and newly added stacks are picked up automatically.
Acceptance Criteria
- WHEN an Update_Run is triggered, THE Stack_Updater SHALL discover Target_Stacks by listing all CloudFormation stacks whose name starts with the Stack_Name_Prefix
CL-AppPipe-. - THE Stack_Updater SHALL perform discovery dynamically on each Update_Run so that stacks added after the previous run are included automatically.
- WHEN discovery completes, THE Stack_Updater SHALL report the total number of Target_Stacks found before proceeding with updates.
- IF no Target_Stacks are found, THEN THE Stack_Updater SHALL log a warning message and terminate the Update_Run without error.
Requirement 2: One-Click Trigger
User Story: As an Operator, I want to trigger the update of all Target_Stacks with a single action, so that I do not have to update each stack individually.
Acceptance Criteria
- THE Stack_Updater SHALL expose a single invocation endpoint that starts an Update_Run.
- WHEN the Operator invokes the endpoint, THE Stack_Updater SHALL begin an Update_Run that updates all discovered Target_Stacks.
- THE Stack_Updater SHALL accept an optional Concurrency_Limit parameter that controls how many stacks are updated in parallel.
- IF no Concurrency_Limit is provided, THEN THE Stack_Updater SHALL default to updating 5 stacks in parallel.
Requirement 3: Stack Update Execution
User Story: As an Operator, I want each stack to be updated using its existing parameters, so that the only change is the refreshed nested template.
Acceptance Criteria
- WHEN updating a Target_Stack, THE Stack_Updater SHALL call the CloudFormation UpdateStack API using the existing parameter values of the Target_Stack.
- WHEN updating a Target_Stack, THE Stack_Updater SHALL use the current Nested_Template_URL so that CloudFormation resolves the latest template version.
- WHILE an Update_Run is in progress, THE Stack_Updater SHALL respect the Concurrency_Limit and wait for an in-progress update to complete before starting the next one.
- IF CloudFormation returns a "No updates are to be performed" response for a Target_Stack, THEN THE Stack_Updater SHALL treat that Target_Stack as successfully updated and continue.
Requirement 4: Error Handling and Resilience
User Story: As an Operator, I want the update process to continue even if individual stacks fail, so that one failure does not block the remaining stacks.
Acceptance Criteria
- IF a Target_Stack update fails, THEN THE Stack_Updater SHALL log the stack name and error details and continue updating the remaining Target_Stacks.
- IF a Target_Stack is in a non-updatable state (e.g., ROLLBACK_COMPLETE, UPDATE_ROLLBACK_IN_PROGRESS), THEN THE Stack_Updater SHALL skip that Target_Stack and log a warning.
- IF the CloudFormation API returns a throttling error, THEN THE Stack_Updater SHALL retry the request using exponential backoff with a maximum of 3 retries.
- IF all retries for a Target_Stack are exhausted, THEN THE Stack_Updater SHALL mark that Target_Stack as failed and continue with the remaining stacks.
Requirement 5: Progress Reporting and Summary
User Story: As an Operator, I want to see the progress and final results of an Update_Run, so that I know which stacks succeeded and which failed.
Acceptance Criteria
- WHILE an Update_Run is in progress, THE Stack_Updater SHALL log the status of each Target_Stack update as it completes (succeeded, failed, skipped, no-update-needed).
- WHEN an Update_Run completes, THE Stack_Updater SHALL produce a summary report containing: total stacks found, stacks updated successfully, stacks skipped, stacks failed, and stacks with no update needed.
- WHEN an Update_Run completes, THE Stack_Updater SHALL return a non-zero exit code if any Target_Stack update failed.
- THE Stack_Updater SHALL log the start time and end time of each Update_Run.
Requirement 6: Dry Run Mode
User Story: As an Operator, I want to preview which stacks would be updated without actually performing updates, so that I can verify the scope before committing.
Acceptance Criteria
- THE Stack_Updater SHALL accept a dry-run flag as an input parameter.
- WHEN the dry-run flag is set, THE Stack_Updater SHALL discover and list all Target_Stacks without performing any updates.
- WHEN the dry-run flag is set, THE Stack_Updater SHALL output the name and current status of each Target_Stack that would be updated.
Requirement 7: IAM Permission Validation
User Story: As an Operator, I want the system to verify it has the required permissions before starting updates, so that I catch permission issues early.
Acceptance Criteria
- WHEN an Update_Run is triggered, THE Stack_Updater SHALL verify that the executing role has
cloudformation:DescribeStacks,cloudformation:UpdateStack, andcloudformation:ListStackspermissions before proceeding. - IF the executing role lacks required permissions, THEN THE Stack_Updater SHALL report the missing permissions and terminate the Update_Run without attempting any updates.