Microsoft.SelfService.Portal.Core.API/ToDo.md

SelfService Portal Restructure ToDo

This file tracks the planned restructure from the current single-template deployment model to a generic configuration composition platform for on-prem, hybrid, and cloud workloads.

Status Discipline

• Keep this file updated after every implementation step.
• Reflect completed work, deferred cleanup, migration status, and DemoData changes here before moving to the next package.
• Keep Context/DemoData.cs aligned with model/API changes so the API can be tested after every migration.

Goals

• Support reusable configuration building blocks for many services, such as Active Directory, SQL, SharePoint, Exchange, Teams, Azure, and M365.
• Allow shared catalog data, such as domains and environments, to be linked instead of duplicated.
• Compose one environment deployment from multiple deployments, for example AD, SQL, SharePoint, and cloud policy workloads.
• Keep deployments reproducible by referencing immutable template versions.
• Keep the worker independent from a single renderer so PowerShell DSC v2 and DSC v3 JSON can both be supported.

Work Packages

1. Architecture Baseline

Status: Completed

• Define the target domain terms: Environment, Domain, Service, Target, DeploymentGroup, Deployment, Template, TemplateVersion, TemplateSelection, DeploymentTarget, QueueJob, Artifact.
• Document the intended relationships.
• Decide how JsonData should be stored and validated.
• Define compatibility rules for the existing API and database.
• Output: Docs/Architecture/DeploymentModel.md

2. Template Versioning

Status: Completed

• Add TemplateVersionModel. Completed.
• Move mutable template document fields from TemplateModel into TemplateVersionModel. Completed for new writes while legacy fields remain for compatibility.
• Keep TemplateModel as the stable catalog entry. Completed.
• Add JsonHash, SchemaVersion, IsPublished, PublishedAt, and PublishedBy. Completed.
• Add unique constraints for TemplateId + Version. Completed.
• Migrate existing Templates.JSONData into an initial TemplateVersions row. Completed in migration.
• Keep API responses backward compatible at first. Completed.
• Add Contoso demo data for template versions. Completed.
• Add reset/seed migration for local development data. Completed and applied locally.
• Defer removal of legacy TemplateModel.Version and TemplateModel.JSONData until Web/Worker no longer depend on them.

3. JSON Document Contract

Status: Completed

• Define the canonical JSON document shape for templates and deployments. Completed in Docs/Architecture/JsonDocumentContract.md.
• Add schema versioning, for example schemaVersion. Completed for template documents and tolerant deployment overrides.
• Add server-side JSON validation. Completed through JsonDocuments/ConfigurationDocumentValidator.cs.
• Add database checks where supported, for example ISJSON(...). Completed for TemplateVersions.JsonData; other JSON columns remain validated in code until they are migrated.
• Add content hash calculation for version comparisons. Completed for TemplateVersionModel.JsonHash.
• Decide which document fields should be projected into queryable columns. Completed in the contract documentation.

4. Deployment Composition

Status: Completed

• Add DeploymentTemplateSelectionModel. Completed.
• Allow a deployment to reference multiple template versions with role and sort order. Completed through deployment batch composition endpoints.
• Add DeploymentParameterValueModel for explicit user overrides. Completed.
• Add target assignment model for deployment targets. Completed as DeploymentTargetAssignmentModel; generic TargetModel follows in Step 5.
• Keep old DeploymentGroup.TemplateId as legacy until Web and Worker are migrated. Completed for this slice.
• Update Web GUI to test deployment composition directly. Completed in DeploymentBatchDetailsPage.

5. Generic Target Model

Status: Completed

• Generalize beyond virtual machines. Completed with TargetModel, TargetController, TargetRepository, and target DTOs.
• Introduce target types such as VirtualMachine, Tenant, Subscription, ResourceGroup, User, Group, Site, and PolicyScope. Completed through TargetType.
• Link targets to environments/domains where applicable. Completed for optional domain links; environment targeting remains composition/worker-specific.
• Preserve VM-specific details in MetadataJson or a specialized extension table. Completed with generic MetadataJson; no extension table yet.
• Add provider scoping for on-prem, Azure, Microsoft 365, Teams, and related target providers. Completed through ProviderType.
• Add migration 20260707211957_AddGenericTargetModel; migration script generated at buildcheck/AddGenericTargetModel.sql.
• Update rudimentary Web GUI to use /targets and expose target type/provider fields. Completed.

6. Worker Pipeline Refactor

Status: Completed

• Load deployment composition from API/database. Completed in Worker DeploymentCompositionLoader.
• Convert JSON documents to ordered PowerShell hashtables for the existing Merge/Resolve modules. Completed as PowerShellDscV2Renderer, which writes ConfigurationData.psd1 with [ordered]@{...} output for the composed deployment document.
• Add a renderer abstraction. Completed through IDeploymentRenderer.
  • PowerShellDscV2Renderer. Completed.
  • DscV3JsonRenderer. Completed.
  • future cloud/action renderers. Extension point added.
• Store generated artifacts and logs. Completed as filesystem artifacts under configurable WorkerPipeline:ArtifactRoot; structured logs are written through ILogger.
• Replace the old dry-run provisioning provider path with DeploymentPipeline. Completed.
• Note: actual invocation of Merge-DSCConfigurationData / Resolve-DSCConfigurationData can now be added inside the DSC v2 renderer when the worker moves from render-only to execute mode.

7. Queue Hardening

Status: Completed

• Add worker claim/lock semantics based on LockedBy, LockedUntil, and a concurrency token. Completed with atomic ExecuteUpdateAsync claim logic in the worker and SQL rowversion on deployment jobs.
• Add CorrelationId, Priority, ScheduledAt, HeartbeatAt, and WorkerName. Completed in model, DTOs, API mapping, and Web TypeScript contracts.
• Add useful indexes for pending/running jobs. Completed for queue claim lookup, worker lookup, correlation id, target status, and step status/sort order.
• Persist step and target output metadata. Completed with JSON metadata columns and SQL ISJSON constraints.
• Migration 20260708194227_AddQueueHardening and SQL script buildcheck/AddQueueHardening.sql generated.

8. Web/API Migration

Status: In Progress

• Update API endpoints to expose template versions and deployment composition. Completed for deployment batch list/details, version-aware create, and queue payloads.
• Update Web to create DeploymentGroups with multiple Deployments. Completed for the rudimentary create flow: users select a template version, targets become target assignments, and legacy executions stay available for compatibility.
• Add GUI support for sealed parameters and sealed template blocks.
• Add preview mode based on Resolve-DSCConfigurationData -SkipSecrets.
• Add promotion flow from Test to QA/Prod.

8a. Interactive GUI Preview

Status: Deferred

• Keep current GUI changes rudimentary while backend work packages are implemented.
• Add an interactive parameter editor that can show effective target values.
• Add a deployment JSON preview for the composed deployment document.
• Add a Merge/Resolve preview action that displays the merged and resolved configuration data.
• Redesign deployment and deployment composition views after the backend model has stabilized.

9. Legacy Cleanup

Status: In Progress

• Remove legacy direct template references after migration.
• Remove old DTO fields that expose mutable Template.JSONData.
• Remove obsolete SMA Runbook/Event/Job API and data model. Completed.
• Keep read-only migration endpoints only if required.
• Archive or migrate old deployment records.

First Implementation Slice

1. Finish Docs/Architecture/DeploymentModel.md. Completed.
2. Add TemplateVersionModel and EF migration. Completed.
3. Migrate existing Templates.JSONData into TemplateVersions. Completed.
4. Update Template API to return the current published version. Completed.
5. Add tests/build checks for the new model. Build and migration script checks completed; automated tests are still pending.
6. Add Contoso DemoData and reset/seed migration. Completed.

Next Step

Continue work package 8 with sealed parameter/block visibility, a backend preview endpoint, and the later promotion flow. Keep rich interactive preview UX in Step 8a.