Microsoft.SelfService.Portal.Core.API/Docs/Architecture/DeploymentModel.md

Deployment Model Architecture

Purpose

The portal should become a generic configuration composition platform. It must support on-premises workloads such as Active Directory, SQL Server, SharePoint, and Exchange, as well as cloud workloads such as Teams, Azure, and M365.

The database should not know SharePoint-specific concepts such as farm accounts or service application pools. Those details belong inside versioned template documents. The database should know how templates, environments, domains, targets, deployments, jobs, and artifacts relate to each other.

Core Concepts

The model separates reusable catalog data from concrete execution data:

Catalog:
  Environment
  Domain
  Service
  Target
  Template
  TemplateVersion

Design:
  DeploymentGroup
  Deployment
  DeploymentTemplateSelection
  DeploymentParameterValue
  DeploymentTarget

Execution:
  QueueJob
  QueueJobStep
  QueueJobTarget
  DeploymentArtifact

Environment

An environment is a logical target context, for example Prod A, Test A, Prod B, Test B, or QA B.

Suggested fields:

• Id
• Key
• Name
• Stage, for example Development, Test, QA, Production
• HostingType, for example OnPrem, Azure, M365, Hybrid
• ProviderType
• TenantId
• SubscriptionId
• MetadataJson

Domain

A domain is reusable catalog data and configuration. A domain can be linked to multiple environments.

Example:

Central-Management
  -> Test A
  -> Prod A
  -> QA B

The goal is to change the Central-Management domain configuration once and then decide which linked environments should consume the new version.

Suggested relationship:

EnvironmentDomain
- EnvironmentId
- DomainId
- Role
- SortOrder

Service

A service represents a workload family, for example:

• active-directory
• sql-server
• sharepoint
• exchange
• teams
• azure-network

Suggested fields:

• Id
• Key
• Name
• Description
• WorkloadType, for example OnPrem, Azure, M365, Hybrid
• IconKey
• IsActive

Target

A target is anything a deployment can act on. It should not be limited to virtual machines.

Suggested target types:

• VirtualMachine
• Tenant
• Subscription
• ResourceGroup
• User
• Group
• Site
• PolicyScope

Suggested fields:

• Id
• Name
• TargetType
• EnvironmentId
• DomainId
• ExternalId
• MetadataJson

For current VM usage, VirtualMachineModel can either remain as a specialized target table during migration or be folded into a generic target model later.

Templates

Template

Template should be the stable catalog entry. It should not contain mutable deployment content.

Suggested fields:

• Id
• Key, for example service.sharepoint.default
• Name
• TemplateType, for example Domain, Environment, Service, Stage, TargetSet, Policy
• WorkloadType, for example OnPrem, Azure, M365, Hybrid
• ServiceId
• TemplateCategoryId
• DeploymentRuleId
• IsActive

Template is mutable catalog metadata. It can be renamed, categorized, activated, or deactivated without changing the immutable template document history.

TemplateVersion

TemplateVersion should contain the immutable template document.

Suggested fields:

• Id
• TemplateId
• Version
• JsonData
• JsonHash
• SchemaVersion
• IsPublished
• PublishedAt
• PublishedBy
• Created
• CreatedBy

Recommended constraints:

• Unique: TemplateId + Version
• Optional unique filtered index: TemplateId where IsPublished = true, if only one current published version should exist.

Template version content is immutable after publication. If the document changes, create a new TemplateVersion.

JsonData Strategy

Keep JsonData as text in SQL Server, but do not treat it as an unstructured dumping ground. The canonical document shapes and validation rules are defined in Docs/Architecture/JsonDocumentContract.md.

Database storage:

JsonData nvarchar(max)
JsonHash nvarchar(64)
SchemaVersion nvarchar(32)

EF model storage can remain a string:

public string JsonData { get; private set; } = "{}";
public string JsonHash { get; private set; } = string.Empty;
public string SchemaVersion { get; set; } = "1.0";

API DTOs should expose real JSON instead of escaped JSON strings where practical:

public JsonElement JsonData { get; set; }

The service layer should canonicalize JSON, validate it, calculate the hash, and store it as a string.

Recommended approach:

1. Store the canonical template/deployment document as JSON.
2. Version every template document.
3. Validate JSON before saving.
4. Store a content hash beside the document.
5. Store schema version beside the document.
6. Project frequently queried values into normal columns or helper tables.

Do not store binary .NET serialization, base64 payloads, or provider-specific object graphs. The stored document should remain plain JSON so it can be returned by the API, diffed, hashed, migrated, and rendered to DSC v3 documents later.

This gives us flexibility for DSC v2, DSC v3, and cloud workloads without turning every possible parameter/resource into relational tables.

Why Keep JSON?

The configuration document is naturally hierarchical:

Metadata
Parameters
Variables
Resources

Trying to fully normalize this would make the database rigid and painful for new services. SharePoint, Exchange, Teams, and Azure do not share the same resource shape.

What Should Be Normalized?

Normalize stable catalog and workflow data:

• templates
• template versions
• environments
• domains
• services
• targets
• deployment groups
• deployments
• template selections
• parameter overrides
• jobs
• artifacts

Keep service-specific configuration inside JsonData.

Optional Projection Tables

For GUI performance and validation, create projection tables derived from JsonData.

Example:

TemplateParameters
- TemplateVersionId
- Name
- Type
- Required
- Sensitive
- Sealed
- DescriptionJson
- AllowedValuesJson

The source of truth stays TemplateVersion.JsonData. Projection tables can be rebuilt when a template version changes.

Deployments

DeploymentGroup

A deployment group represents a deployable environment composition, for example:

Contoso Test A Full Farm
  - AD Deployment
  - SQL Deployment
  - SharePoint Deployment

Suggested fields:

• Id
• EnvironmentId
• Name
• Status
• DeploymentRuleId
• Created
• CreatedBy

In the current model, DeploymentGroupModel maps to the DeploymentBatches table and still has a direct TemplateId. In the target model this direct TemplateId becomes legacy. The selected templates move to DeploymentTemplateSelection.

Deployment

A deployment is one deployable workload slice inside a group.

Examples:

• AD
• SQL
• SharePoint
• Exchange
• Teams Policies

Suggested fields:

• Id
• DeploymentGroupId
• ServiceId
• Name
• SortOrder
• Status
• DeploymentDataJson
• MergedConfigurationDataJson
• ResolvedPreviewJson
• RendererTarget, for example PowerShellDscV2, DscV3Json, CloudApi

In the current model, DeploymentModel maps to DeploymentExecutions and is keyed by VirtualMachineId + DeploymentGroupId. This is closer to a target execution than to a design-level deployment. During migration, keep it as compatibility data and introduce the new composition tables beside it.

DeploymentTemplateSelection

A deployment selects multiple template versions.

Suggested fields:

• Id
• DeploymentId
• TemplateVersionId
• TemplateRole, for example Domain, Environment, Service, Stage, TargetSet, Policy
• SortOrder
• Alias

Example SharePoint deployment:

Domain/Central-Management
Environment/Test A
Service/SharePoint/Contoso
Stage/Install
Targets/SharePointNodes

DeploymentParameterValue

User overrides should be stored separately from the generated full deployment document.

Suggested fields:

• Id
• DeploymentId
• DeploymentTemplateSelectionId
• Name
• ValueJson
• IsSecretReference
• IsOverride

DeploymentTarget

Suggested fields:

• Id
• DeploymentId
• TargetId
• RoleKey, for example DomainController, SqlServer, SharePointServer, TeamsPolicyScope
• SortOrder
• NodeDataJson

Queue And Worker

The existing QueueJobModel, QueueJobStepModel, and QueueJobTargetModel are a good foundation.

Recommended additions:

• CorrelationId
• Priority
• ScheduledAt
• HeartbeatAt
• WorkerName
• RowVersion

Recommended indexes:

• (Status, LockedUntil, Created)
• (Type, Status)
• (CorrelationId)

The worker should claim jobs atomically and update LockedBy, LockedUntil, and HeartbeatAt.

The existing queue tables can remain. The first improvements should be additive so the current Web/API flow keeps working.

Artifacts

Generated output should be stored as artifacts.

Suggested fields:

• Id
• DeploymentId
• QueueJobId
• ArtifactType, for example MergedJson, ResolvedPreviewJson, PowerShellDataFile, Mof, DscV3Json, Log
• Path
• ContentHash
• ContentJson
• ContentText
• Created

Worker Pipeline

Current DSC v2 flow:

Load template versions
Convert JSON to ordered PowerShell hashtables
Merge-DSCConfigurationData
Resolve-DSCConfigurationData
Render PowerShell DSC v2
Compile MOF
Store artifacts and status

Future DSC v3 flow:

Load template versions
Merge/resolve
Render DSC v3 JSON/YAML document
dsc config test/set
Store artifacts and status

The worker should therefore have a renderer abstraction:

• PowerShellDscV2Renderer
• DscV3JsonRenderer
• future cloud/action renderers

Migration Strategy

1. Add TemplateVersions while keeping Templates.JSONData.
2. Create one initial TemplateVersion per existing Template.
3. Update API reads to expose the current published version.
4. Add deployment composition tables next to the existing deployment batch/execution model.
5. Migrate Web and Worker to the new model.
6. Remove legacy fields only after the new flow is stable.

Current Model Mapping

The current model maps to the target model as follows:

TemplateModel
  -> Template
  -> TemplateVersion

TemplateCategoryModel
  -> TemplateCategory

ServiceModel
  -> Service

EnvironmentModel
  -> Environment

DomainModel
  -> Domain

EnvironmentDomainsModel
  -> EnvironmentDomain

VirtualMachineModel
  -> Target where TargetType = VirtualMachine

DeploymentGroupModel / DeploymentBatches
  -> DeploymentGroup

DeploymentModel / DeploymentExecutions
  -> legacy deployment target/execution record
  -> later replaced or complemented by Deployment + DeploymentTarget

QueueJobModel / DeploymentJobs
  -> QueueJob

QueueJobStepModel / DeploymentJobSteps
  -> QueueJobStep

QueueJobTargetModel / DeploymentJobTargets
  -> QueueJobTarget

Compatibility Rules

During migration:

• Keep existing endpoints working.
• Keep TemplateModel.JSONData until API reads and writes use TemplateVersion.
• New code should prefer TemplateVersion.JsonData.
• Existing DeploymentGroup.TemplateId remains legacy and should be populated for old clients.
• New deployments should use DeploymentTemplateSelection.
• Do not remove old database columns until Web and Worker are migrated.
• Worker code should be written against the target composition model as soon as the new tables exist.

Planned API Shape

Initial compatibility endpoint:

GET /api/Template/{id}

should continue to return the existing DTO shape, but its JsonData should come from the current published TemplateVersion when available.

New endpoints:

GET  /api/templates/{templateId}/versions
GET  /api/templates/{templateId}/versions/{versionId}
POST /api/templates/{templateId}/versions
POST /api/templates/{templateId}/versions/{versionId}/publish

Deployment composition endpoints later:

GET  /api/deployment-groups/{id}/composition
POST /api/deployment-groups/{id}/deployments
POST /api/deployments/{id}/template-selections
POST /api/deployments/{id}/targets
POST /api/deployments/{id}/parameter-values

First Database Change

The first concrete database change should be additive:

TemplateVersions
- Id
- TemplateId
- Version
- JsonData
- JsonHash
- SchemaVersion
- IsPublished
- PublishedAt
- PublishedBy
- Modified
- ModifiedBy
- Created
- CreatedBy

TemplateVersions.TemplateId references Templates.Id.

For existing data, create one version per current template:

Version = Templates.Version
JsonData = Templates.JSONData
IsPublished = true
SchemaVersion = '1.0'

After this step, new code can begin reading from TemplateVersions while old code can still read Templates.JSONData.