Microsoft.SelfService.Portal.Core.API/Docs/Architecture/DeploymentModel.md

# Deployment Model Architecture

## Purpose

The portal should become a generic configuration composition platform. It must support on-premises workloads such as Active Directory, SQL Server, SharePoint, and Exchange, as well as cloud workloads such as Teams, Azure, and M365.

The database should not know SharePoint-specific concepts such as farm accounts or service application pools. Those details belong inside versioned template documents. The database should know how templates, environments, domains, targets, deployments, jobs, and artifacts relate to each other.

## Core Concepts

The model separates reusable catalog data from concrete execution data:

```text
Catalog:
  Environment
  Domain
  Service
  Target
  Template
  TemplateVersion

Design:
  DeploymentGroup
  Deployment
  DeploymentTemplateSelection
  DeploymentParameterValue
  DeploymentTarget

Execution:
  QueueJob
  QueueJobStep
  QueueJobTarget
  DeploymentArtifact
```

### Environment

An environment is a logical target context, for example `Prod A`, `Test A`, `Prod B`, `Test B`, or `QA B`.

Suggested fields:

- `Id`
- `Key`
- `Name`
- `Stage`, for example `Development`, `Test`, `QA`, `Production`
- `HostingType`, for example `OnPrem`, `Azure`, `M365`, `Hybrid`
- `ProviderType`
- `TenantId`
- `SubscriptionId`
- `MetadataJson`

### Domain

A domain is reusable catalog data and configuration. A domain can be linked to multiple environments.

Example:

```text
Central-Management
  -> Test A
  -> Prod A
  -> QA B
```

The goal is to change the Central-Management domain configuration once and then decide which linked environments should consume the new version.

Suggested relationship:

```text
EnvironmentDomain
- EnvironmentId
- DomainId
- Role
- SortOrder
```

### Service

A service represents a workload family, for example:

- `active-directory`
- `sql-server`
- `sharepoint`
- `exchange`
- `teams`
- `azure-network`

Suggested fields:

- `Id`
- `Key`
- `Name`
- `Description`
- `WorkloadType`, for example `OnPrem`, `Azure`, `M365`, `Hybrid`
- `IconKey`
- `IsActive`

### Target

A target is anything a deployment can act on. It should not be limited to virtual machines.

Suggested target types:

- `VirtualMachine`
- `Tenant`
- `Subscription`
- `ResourceGroup`
- `User`
- `Group`
- `Site`
- `PolicyScope`

Suggested fields:

- `Id`
- `Name`
- `TargetType`
- `EnvironmentId`
- `DomainId`
- `ExternalId`
- `MetadataJson`

For current VM usage, `VirtualMachineModel` can either remain as a specialized target table during migration or be folded into a generic target model later.

## Templates

### Template

`Template` should be the stable catalog entry. It should not contain mutable deployment content.

Suggested fields:

- `Id`
- `Key`, for example `service.sharepoint.default`
- `Name`
- `TemplateType`, for example `Domain`, `Environment`, `Service`, `Stage`, `TargetSet`, `Policy`
- `WorkloadType`, for example `OnPrem`, `Azure`, `M365`, `Hybrid`
- `ServiceId`
- `TemplateCategoryId`
- `DeploymentRuleId`
- `IsActive`

`Template` is mutable catalog metadata. It can be renamed, categorized, activated, or deactivated without changing the immutable template document history.

### TemplateVersion

`TemplateVersion` should contain the immutable template document.

Suggested fields:

- `Id`
- `TemplateId`
- `Version`
- `JsonData`
- `JsonHash`
- `SchemaVersion`
- `IsPublished`
- `PublishedAt`
- `PublishedBy`
- `Created`
- `CreatedBy`

Recommended constraints:

- Unique: `TemplateId + Version`
- Optional unique filtered index: `TemplateId` where `IsPublished = true`, if only one current published version should exist.

Template version content is immutable after publication. If the document changes, create a new `TemplateVersion`.

## JsonData Strategy

Keep `JsonData` as text in SQL Server, but do not treat it as an unstructured dumping ground.
The canonical document shapes and validation rules are defined in `Docs/Architecture/JsonDocumentContract.md`.

Database storage:

```text
JsonData nvarchar(max)
JsonHash nvarchar(64)
SchemaVersion nvarchar(32)
```

EF model storage can remain a string:

```csharp
public string JsonData { get; private set; } = "{}";
public string JsonHash { get; private set; } = string.Empty;
public string SchemaVersion { get; set; } = "1.0";
```

API DTOs should expose real JSON instead of escaped JSON strings where practical:

```csharp
public JsonElement JsonData { get; set; }
```

The service layer should canonicalize JSON, validate it, calculate the hash, and store it as a string.

Recommended approach:

1. Store the canonical template/deployment document as JSON.
2. Version every template document.
3. Validate JSON before saving.
4. Store a content hash beside the document.
5. Store schema version beside the document.
6. Project frequently queried values into normal columns or helper tables.

Do not store binary .NET serialization, base64 payloads, or provider-specific object graphs. The stored document should remain plain JSON so it can be returned by the API, diffed, hashed, migrated, and rendered to DSC v3 documents later.

This gives us flexibility for DSC v2, DSC v3, and cloud workloads without turning every possible parameter/resource into relational tables.

### Why Keep JSON?

The configuration document is naturally hierarchical:

```text
Metadata
Parameters
Variables
Resources
```

Trying to fully normalize this would make the database rigid and painful for new services. SharePoint, Exchange, Teams, and Azure do not share the same resource shape.

### What Should Be Normalized?

Normalize stable catalog and workflow data:

- templates
- template versions
- environments
- domains
- services
- targets
- deployment groups
- deployments
- template selections
- parameter overrides
- jobs
- artifacts

Keep service-specific configuration inside `JsonData`.

### Optional Projection Tables

For GUI performance and validation, create projection tables derived from `JsonData`.

Example:

```text
TemplateParameters
- TemplateVersionId
- Name
- Type
- Required
- Sensitive
- Sealed
- DescriptionJson
- AllowedValuesJson
```

The source of truth stays `TemplateVersion.JsonData`. Projection tables can be rebuilt when a template version changes.

## Deployments

### DeploymentGroup

A deployment group represents a deployable environment composition, for example:

```text
Contoso Test A Full Farm
  - AD Deployment
  - SQL Deployment
  - SharePoint Deployment
```

Suggested fields:

- `Id`
- `EnvironmentId`
- `Name`
- `Status`
- `DeploymentRuleId`
- `Created`
- `CreatedBy`

In the current model, `DeploymentGroupModel` maps to the `DeploymentBatches` table and still has a direct `TemplateId`. In the target model this direct `TemplateId` becomes legacy. The selected templates move to `DeploymentTemplateSelection`.

### Deployment

A deployment is one deployable workload slice inside a group.

Examples:

- `AD`
- `SQL`
- `SharePoint`
- `Exchange`
- `Teams Policies`

Suggested fields:

- `Id`
- `DeploymentGroupId`
- `ServiceId`
- `Name`
- `SortOrder`
- `Status`
- `DeploymentDataJson`
- `MergedConfigurationDataJson`
- `ResolvedPreviewJson`
- `RendererTarget`, for example `PowerShellDscV2`, `DscV3Json`, `CloudApi`

In the current model, `DeploymentModel` maps to `DeploymentExecutions` and is keyed by `VirtualMachineId + DeploymentGroupId`. This is closer to a target execution than to a design-level deployment. During migration, keep it as compatibility data and introduce the new composition tables beside it.

### DeploymentTemplateSelection

A deployment selects multiple template versions.

Suggested fields:

- `Id`
- `DeploymentId`
- `TemplateVersionId`
- `TemplateRole`, for example `Domain`, `Environment`, `Service`, `Stage`, `TargetSet`, `Policy`
- `SortOrder`
- `Alias`

Example SharePoint deployment:

```text
Domain/Central-Management
Environment/Test A
Service/SharePoint/Contoso
Stage/Install
Targets/SharePointNodes
```

### DeploymentParameterValue

User overrides should be stored separately from the generated full deployment document.

Suggested fields:

- `Id`
- `DeploymentId`
- `DeploymentTemplateSelectionId`
- `Name`
- `ValueJson`
- `IsSecretReference`
- `IsOverride`

### DeploymentTarget

Suggested fields:

- `Id`
- `DeploymentId`
- `TargetId`
- `RoleKey`, for example `DomainController`, `SqlServer`, `SharePointServer`, `TeamsPolicyScope`
- `SortOrder`
- `NodeDataJson`

## Queue And Worker

The existing `QueueJobModel`, `QueueJobStepModel`, and `QueueJobTargetModel` are a good foundation.

Recommended additions:

- `CorrelationId`
- `Priority`
- `ScheduledAt`
- `HeartbeatAt`
- `WorkerName`
- `RowVersion`

Recommended indexes:

- `(Status, LockedUntil, Created)`
- `(Type, Status)`
- `(CorrelationId)`

The worker should claim jobs atomically and update `LockedBy`, `LockedUntil`, and `HeartbeatAt`.

The existing queue tables can remain. The first improvements should be additive so the current Web/API flow keeps working.

## Artifacts

Generated output should be stored as artifacts.

Suggested fields:

- `Id`
- `DeploymentId`
- `QueueJobId`
- `ArtifactType`, for example `MergedJson`, `ResolvedPreviewJson`, `PowerShellDataFile`, `Mof`, `DscV3Json`, `Log`
- `Path`
- `ContentHash`
- `ContentJson`
- `ContentText`
- `Created`

## Worker Pipeline

Current DSC v2 flow:

```text
Load template versions
Convert JSON to ordered PowerShell hashtables
Merge-DSCConfigurationData
Resolve-DSCConfigurationData
Render PowerShell DSC v2
Compile MOF
Store artifacts and status
```

Future DSC v3 flow:

```text
Load template versions
Merge/resolve
Render DSC v3 JSON/YAML document
dsc config test/set
Store artifacts and status
```

The worker should therefore have a renderer abstraction:

- `PowerShellDscV2Renderer`
- `DscV3JsonRenderer`
- future cloud/action renderers

## Migration Strategy

1. Add `TemplateVersions` while keeping `Templates.JSONData`.
2. Create one initial `TemplateVersion` per existing `Template`.
3. Update API reads to expose the current published version.
4. Add deployment composition tables next to the existing deployment batch/execution model.
5. Migrate Web and Worker to the new model.
6. Remove legacy fields only after the new flow is stable.

## Current Model Mapping

The current model maps to the target model as follows:

```text
TemplateModel
  -> Template
  -> TemplateVersion

TemplateCategoryModel
  -> TemplateCategory

ServiceModel
  -> Service

EnvironmentModel
  -> Environment

DomainModel
  -> Domain

EnvironmentDomainsModel
  -> EnvironmentDomain

VirtualMachineModel
  -> Target where TargetType = VirtualMachine

DeploymentGroupModel / DeploymentBatches
  -> DeploymentGroup

DeploymentModel / DeploymentExecutions
  -> legacy deployment target/execution record
  -> later replaced or complemented by Deployment + DeploymentTarget

QueueJobModel / DeploymentJobs
  -> QueueJob

QueueJobStepModel / DeploymentJobSteps
  -> QueueJobStep

QueueJobTargetModel / DeploymentJobTargets
  -> QueueJobTarget
```

## Compatibility Rules

During migration:

- Keep existing endpoints working.
- Keep `TemplateModel.JSONData` until API reads and writes use `TemplateVersion`.
- New code should prefer `TemplateVersion.JsonData`.
- Existing `DeploymentGroup.TemplateId` remains legacy and should be populated for old clients.
- New deployments should use `DeploymentTemplateSelection`.
- Do not remove old database columns until Web and Worker are migrated.
- Worker code should be written against the target composition model as soon as the new tables exist.

## Planned API Shape

Initial compatibility endpoint:

```text
GET /api/Template/{id}
```

should continue to return the existing DTO shape, but its `JsonData` should come from the current published `TemplateVersion` when available.

New endpoints:

```text
GET  /api/templates/{templateId}/versions
GET  /api/templates/{templateId}/versions/{versionId}
POST /api/templates/{templateId}/versions
POST /api/templates/{templateId}/versions/{versionId}/publish
```

Deployment composition endpoints later:

```text
GET  /api/deployment-groups/{id}/composition
POST /api/deployment-groups/{id}/deployments
POST /api/deployments/{id}/template-selections
POST /api/deployments/{id}/targets
POST /api/deployments/{id}/parameter-values
```

## First Database Change

The first concrete database change should be additive:

```text
TemplateVersions
- Id
- TemplateId
- Version
- JsonData
- JsonHash
- SchemaVersion
- IsPublished
- PublishedAt
- PublishedBy
- Modified
- ModifiedBy
- Created
- CreatedBy
```

`TemplateVersions.TemplateId` references `Templates.Id`.

For existing data, create one version per current template:

```text
Version = Templates.Version
JsonData = Templates.JSONData
IsPublished = true
SchemaVersion = '1.0'
```

After this step, new code can begin reading from `TemplateVersions` while old code can still read `Templates.JSONData`.