01c2a5df2e
- Drop existing indexes on DeploymentJobTargets and DeploymentJobSteps - Alter Status column to nvarchar(450) in DeploymentJobTargets, DeploymentJobSteps, and DeploymentJobs - Add new columns: Finished, OutputMetadataJson, Started to DeploymentJobTargets; ErrorMessage, Finished, OutputMetadataJson, Started to DeploymentJobSteps; CorrelationId, HeartbeatAt, Priority, RowVersion, ScheduledAt, WorkerName to DeploymentJobs - Create new indexes for improved query performance - Add constraints to ensure OutputMetadataJson is valid JSON - Record migration in __EFMigrationsHistory
146 lines
8.7 KiB
Markdown
146 lines
8.7 KiB
Markdown
# SelfService Portal Restructure ToDo
|
|
|
|
This file tracks the planned restructure from the current single-template deployment model to a generic configuration composition platform for on-prem, hybrid, and cloud workloads.
|
|
|
|
## Status Discipline
|
|
|
|
- Keep this file updated after every implementation step.
|
|
- Reflect completed work, deferred cleanup, migration status, and DemoData changes here before moving to the next package.
|
|
- Keep `Context/DemoData.cs` aligned with model/API changes so the API can be tested after every migration.
|
|
|
|
## Goals
|
|
|
|
- Support reusable configuration building blocks for many services, such as Active Directory, SQL, SharePoint, Exchange, Teams, Azure, and M365.
|
|
- Allow shared catalog data, such as domains and environments, to be linked instead of duplicated.
|
|
- Compose one environment deployment from multiple deployments, for example AD, SQL, SharePoint, and cloud policy workloads.
|
|
- Keep deployments reproducible by referencing immutable template versions.
|
|
- Keep the worker independent from a single renderer so PowerShell DSC v2 and DSC v3 JSON can both be supported.
|
|
|
|
## Work Packages
|
|
|
|
### 1. Architecture Baseline
|
|
|
|
Status: Completed
|
|
|
|
- Define the target domain terms: Environment, Domain, Service, Target, DeploymentGroup, Deployment, Template, TemplateVersion, TemplateSelection, DeploymentTarget, QueueJob, Artifact.
|
|
- Document the intended relationships.
|
|
- Decide how `JsonData` should be stored and validated.
|
|
- Define compatibility rules for the existing API and database.
|
|
- Output: `Docs/Architecture/DeploymentModel.md`
|
|
|
|
### 2. Template Versioning
|
|
|
|
Status: Completed
|
|
|
|
- Add `TemplateVersionModel`. Completed.
|
|
- Move mutable template document fields from `TemplateModel` into `TemplateVersionModel`. Completed for new writes while legacy fields remain for compatibility.
|
|
- Keep `TemplateModel` as the stable catalog entry. Completed.
|
|
- Add `JsonHash`, `SchemaVersion`, `IsPublished`, `PublishedAt`, and `PublishedBy`. Completed.
|
|
- Add unique constraints for `TemplateId + Version`. Completed.
|
|
- Migrate existing `Templates.JSONData` into an initial `TemplateVersions` row. Completed in migration.
|
|
- Keep API responses backward compatible at first. Completed.
|
|
- Add Contoso demo data for template versions. Completed.
|
|
- Add reset/seed migration for local development data. Completed and applied locally.
|
|
- Defer removal of legacy `TemplateModel.Version` and `TemplateModel.JSONData` until Web/Worker no longer depend on them.
|
|
|
|
### 3. JSON Document Contract
|
|
|
|
Status: Completed
|
|
|
|
- Define the canonical JSON document shape for templates and deployments. Completed in `Docs/Architecture/JsonDocumentContract.md`.
|
|
- Add schema versioning, for example `schemaVersion`. Completed for template documents and tolerant deployment overrides.
|
|
- Add server-side JSON validation. Completed through `JsonDocuments/ConfigurationDocumentValidator.cs`.
|
|
- Add database checks where supported, for example `ISJSON(...)`. Completed for `TemplateVersions.JsonData`; other JSON columns remain validated in code until they are migrated.
|
|
- Add content hash calculation for version comparisons. Completed for `TemplateVersionModel.JsonHash`.
|
|
- Decide which document fields should be projected into queryable columns. Completed in the contract documentation.
|
|
|
|
### 4. Deployment Composition
|
|
|
|
Status: Completed
|
|
|
|
- Add `DeploymentTemplateSelectionModel`. Completed.
|
|
- Allow a deployment to reference multiple template versions with role and sort order. Completed through deployment batch composition endpoints.
|
|
- Add `DeploymentParameterValueModel` for explicit user overrides. Completed.
|
|
- Add target assignment model for deployment targets. Completed as `DeploymentTargetAssignmentModel`; generic `TargetModel` follows in Step 5.
|
|
- Keep old `DeploymentGroup.TemplateId` as legacy until Web and Worker are migrated. Completed for this slice.
|
|
- Update Web GUI to test deployment composition directly. Completed in `DeploymentBatchDetailsPage`.
|
|
|
|
### 5. Generic Target Model
|
|
|
|
Status: Completed
|
|
|
|
- Generalize beyond virtual machines. Completed with `TargetModel`, `TargetController`, `TargetRepository`, and target DTOs.
|
|
- Introduce target types such as `VirtualMachine`, `Tenant`, `Subscription`, `ResourceGroup`, `User`, `Group`, `Site`, and `PolicyScope`. Completed through `TargetType`.
|
|
- Link targets to environments/domains where applicable. Completed for optional domain links; environment targeting remains composition/worker-specific.
|
|
- Preserve VM-specific details in `MetadataJson` or a specialized extension table. Completed with generic `MetadataJson`; no extension table yet.
|
|
- Add provider scoping for on-prem, Azure, Microsoft 365, Teams, and related target providers. Completed through `ProviderType`.
|
|
- Add migration `20260707211957_AddGenericTargetModel`; migration script generated at `buildcheck/AddGenericTargetModel.sql`.
|
|
- Update rudimentary Web GUI to use `/targets` and expose target type/provider fields. Completed.
|
|
|
|
### 6. Worker Pipeline Refactor
|
|
|
|
Status: Completed
|
|
|
|
- Load deployment composition from API/database. Completed in Worker `DeploymentCompositionLoader`.
|
|
- Convert JSON documents to ordered PowerShell hashtables for the existing Merge/Resolve modules. Completed as `PowerShellDscV2Renderer`, which writes `ConfigurationData.psd1` with `[ordered]@{...}` output for the composed deployment document.
|
|
- Add a renderer abstraction. Completed through `IDeploymentRenderer`.
|
|
- `PowerShellDscV2Renderer`. Completed.
|
|
- `DscV3JsonRenderer`. Completed.
|
|
- future cloud/action renderers. Extension point added.
|
|
- Store generated artifacts and logs. Completed as filesystem artifacts under configurable `WorkerPipeline:ArtifactRoot`; structured logs are written through `ILogger`.
|
|
- Replace the old dry-run provisioning provider path with `DeploymentPipeline`. Completed.
|
|
- Note: actual invocation of `Merge-DSCConfigurationData` / `Resolve-DSCConfigurationData` can now be added inside the DSC v2 renderer when the worker moves from render-only to execute mode.
|
|
|
|
### 7. Queue Hardening
|
|
|
|
Status: Completed
|
|
|
|
- Add worker claim/lock semantics based on `LockedBy`, `LockedUntil`, and a concurrency token. Completed with atomic `ExecuteUpdateAsync` claim logic in the worker and SQL `rowversion` on deployment jobs.
|
|
- Add `CorrelationId`, `Priority`, `ScheduledAt`, `HeartbeatAt`, and `WorkerName`. Completed in model, DTOs, API mapping, and Web TypeScript contracts.
|
|
- Add useful indexes for pending/running jobs. Completed for queue claim lookup, worker lookup, correlation id, target status, and step status/sort order.
|
|
- Persist step and target output metadata. Completed with JSON metadata columns and SQL `ISJSON` constraints.
|
|
- Migration `20260708194227_AddQueueHardening` and SQL script `buildcheck/AddQueueHardening.sql` generated.
|
|
|
|
### 8. Web/API Migration
|
|
|
|
Status: In Progress
|
|
|
|
- Update API endpoints to expose template versions and deployment composition. Completed for deployment batch list/details, version-aware create, and queue payloads.
|
|
- Update Web to create DeploymentGroups with multiple Deployments. Completed for the rudimentary create flow: users select a template version, targets become target assignments, and legacy executions stay available for compatibility.
|
|
- Add GUI support for sealed parameters and sealed template blocks.
|
|
- Add preview mode based on `Resolve-DSCConfigurationData -SkipSecrets`.
|
|
- Add promotion flow from Test to QA/Prod.
|
|
|
|
### 8a. Interactive GUI Preview
|
|
|
|
Status: Deferred
|
|
|
|
- Keep current GUI changes rudimentary while backend work packages are implemented.
|
|
- Add an interactive parameter editor that can show effective target values.
|
|
- Add a deployment JSON preview for the composed deployment document.
|
|
- Add a Merge/Resolve preview action that displays the merged and resolved configuration data.
|
|
- Redesign deployment and deployment composition views after the backend model has stabilized.
|
|
|
|
### 9. Legacy Cleanup
|
|
|
|
Status: In Progress
|
|
|
|
- Remove legacy direct template references after migration.
|
|
- Remove old DTO fields that expose mutable `Template.JSONData`.
|
|
- Remove obsolete SMA Runbook/Event/Job API and data model. Completed.
|
|
- Keep read-only migration endpoints only if required.
|
|
- Archive or migrate old deployment records.
|
|
|
|
## First Implementation Slice
|
|
|
|
1. Finish `Docs/Architecture/DeploymentModel.md`. Completed.
|
|
2. Add `TemplateVersionModel` and EF migration. Completed.
|
|
3. Migrate existing `Templates.JSONData` into `TemplateVersions`. Completed.
|
|
4. Update Template API to return the current published version. Completed.
|
|
5. Add tests/build checks for the new model. Build and migration script checks completed; automated tests are still pending.
|
|
6. Add Contoso DemoData and reset/seed migration. Completed.
|
|
|
|
## Next Step
|
|
|
|
Continue work package 8 with sealed parameter/block visibility, a backend preview endpoint, and the later promotion flow. Keep rich interactive preview UX in Step 8a.
|