Microsoft.SelfService.Portal.Core.API/Docs/Architecture/Process.md

# SelfService Portal Process

## Purpose

This document describes the current end-to-end process as understood for the SelfService Portal.
It is intentionally written as a business and platform flow, not as a low-level API reference.

The goal is to make visible how catalog data, reusable templates, deployment composition, queue jobs, and worker execution fit together.

## Big Picture

The portal is intended to become a generic configuration composition platform.
It should support on-premises workloads such as Active Directory, SQL Server, SharePoint, and Exchange, and also cloud workloads such as Teams, Azure, Microsoft 365, and Azure resources.

The important idea is:

```text
Reusable catalog data + immutable template versions + deployment-specific choices
  -> composed deployment document
  -> queue job
  -> worker renders and executes
```

The database should know generic concepts such as environments, domains, targets, services, templates, template versions, deployment groups, queue jobs, and artifacts.
Workload-specific details, for example SharePoint farm accounts or Teams policies, belong inside versioned template documents.

## Roles Of The Applications

### Microsoft.SelfService.Portal.Core.API

The API owns the central data model and exposes it to the GUI and worker.

Responsibilities:

- Store catalog data such as environments, domains, services, targets, templates, and template versions.
- Store deployment composition data such as selected template versions, parameter overrides, and target assignments.
- Validate JSON documents before storing them.
- Create deployment queue jobs.
- Expose queue job status, steps, target progress, errors, and metadata.

### Microsoft.SelfService.Portal.Web

The web frontend is the user-facing surface.

Responsibilities:

- Let users browse catalog data.
- Let users create and edit deployment groups.
- Let users select template versions, targets, parameters, and later preview effective configuration data.
- Submit deployment requests into the API queue.
- Show queue job state and execution progress.

The current GUI is intentionally rudimentary. The later target is a richer interactive deployment builder.

### Microsoft.SelfService.Portal.Core.Worker

The worker processes queued deployment jobs.

Responsibilities:

- Claim jobs safely from the queue.
- Load deployment composition from the API database.
- Render the deployment composition into an artifact.
- Support different renderers, currently DSC v2 PowerShell data files and DSC v3 JSON.
- Later execute or hand off the rendered artifacts.
- Persist target and step output metadata.

## Catalog Process

### 1. Environments Are Defined

An environment is a logical deployment context, for example:

```text
Prod A
Test A
QA B
Prod B
Test B
```

An environment describes where something belongs and what stage it represents.

Examples of environment metadata:

- Stage: Test, QA, Production
- Hosting type: OnPrem, Azure, M365, Hybrid
- Provider type
- Tenant or subscription reference

### 2. Domains Are Defined Once And Linked

A domain is reusable catalog and configuration data.
It can be linked to multiple environments.

Example:

```text
Central-Management Domain
  -> Test A
  -> QA A
  -> Prod A
```

The intent is that a domain configuration is not duplicated for every environment.
Instead, a domain baseline can be maintained once, then linked or promoted into the environments that should consume it.

Target process:

```text
Change Central-Management domain baseline
  -> validate in Test
  -> promote or reuse for Prod
```

Not desired:

```text
Maintain Central-Management Test separately
Maintain Central-Management Prod separately
Repeat every change manually
```

### 3. Services Describe Workload Families

A service represents a workload type, not a single deployment.

Examples:

- Active Directory
- SQL Server
- SharePoint
- Exchange
- Teams
- Azure Network
- Microsoft 365 Policies

Templates are categorized below services so users can find suitable building blocks.

### 4. Targets Are Generic

Targets are everything a deployment can act on.
They are not limited to virtual machines.

Examples:

- VirtualMachine
- Tenant
- Subscription
- ResourceGroup
- User
- Group
- Site
- PolicyScope

For on-premises workloads, targets are often servers.
For cloud workloads, targets may be tenants, subscriptions, groups, sites, or policy scopes.

## Template Process

### 1. A Template Is A Catalog Entry

A template is the stable entry shown in the portal.
It has metadata such as name, category, service, and description.

The template itself should not be the mutable source of deployment content in the long term.
It is the catalog shell.

### 2. TemplateVersions Are Immutable Building Blocks

The actual configuration document lives in `TemplateVersion.JsonData`.

Each template version contains a JSON document with this general shape:

```json
{
  "schemaVersion": "1.0",
  "templateType": "Service",
  "parameters": {},
  "variables": {},
  "resources": {}
}
```

The template version is the thing that should be selected in a deployment.
This makes deployments reproducible, because they point to an immutable version instead of a mutable template document.

### 3. Template Documents Stay Generic

Template documents can describe many workloads.

Examples:

- Environment defaults
- Domain defaults
- Landscape values
- Service definitions
- Stage-specific settings
- Target or role specific blocks

The document can contain parameters, variables, and resources.
The merge and resolve modules can later combine those pieces into effective configuration data.

## Deployment Design Process

### 1. User Creates A DeploymentGroup

A deployment group represents a deployable unit or one part of a larger environment rollout.

Example environment rollout:

```text
Deployment Group: Contoso Test Environment

  Deployment AD
    Targets: 2 domain controller servers
    Templates: Active Directory baseline, environment defaults, stage defaults

  Deployment SQL
    Targets: 1 SQL server
    Templates: SQL baseline, environment defaults, stage defaults

  Deployment SharePoint
    Targets: 6 SharePoint servers
    Templates: SharePoint baseline, environment defaults, landscape, stage defaults
```

Current implementation still uses `DeploymentGroup`/`DeploymentBatch` naming in places.
Conceptually this is the deployment design container.

### 2. User Selects Template Versions

The deployment group gets one or more `DeploymentTemplateSelections`.

Example:

```text
SortOrder 10: Environment Default 1.0
SortOrder 20: Environment Contoso Test 1.0
SortOrder 30: Service SharePoint 1.0
SortOrder 40: Stage Install 1.0
```

The order matters because the selected template versions are composed in order.
Later selections can extend or override earlier selections, unless a parameter or resource block is sealed.

Current state:

- The API can store multiple template selections.
- The GUI can create an initial selection from a selected template version.
- The details page can add more selections manually.

### 3. User Assigns Targets

The deployment group gets `DeploymentTargetAssignments`.

For server-based workloads, each target assignment usually points to a server.

Example:

```text
Target: CT-SHP-01
RoleKey: WebFrontEnd
NodeDataJson: { "nodeName": "CT-SHP-01" }

Target: CT-SHP-02
RoleKey: Application
NodeDataJson: { "nodeName": "CT-SHP-02" }
```

For cloud workloads, the target may be a tenant, policy scope, group, or site instead of a server.

Current state:

- The API stores target assignments.
- The GUI create flow turns selected targets into target assignments.
- Queue requests can use target assignments if explicit target IDs are not sent again.

### 4. User Adds Parameter Values

The deployment group can store `DeploymentParameterValues`.

Parameter values can be global or scoped to a specific template selection.

Examples:

```text
Global:
  DatabasePrefix = Contoso_Test

Scoped to SharePoint selection:
  FarmAccount = secret reference
```

The intention is that the GUI later shows user-facing parameter forms based on selected template versions.
The user edits parameter values, and the portal can preview the effective resolved result.

Current state:

- API stores parameter values.
- Values are JSON validated.
- Secret references can be marked.
- Rich parameter editor and preview are still pending.

## Deployment Request Process

### 1. User Starts A Deployment

When the user starts a deployment, the web frontend calls the API deployment request endpoint.

The request currently contains:

```text
DeploymentGroupId
TargetIds optional
JsonData optional deployment override
```

If `TargetIds` are omitted, the API can derive targets from the deployment group's target assignments.

Target direction:

```text
DeploymentGroup composition should be the source of truth.
The start request should eventually only reference the DeploymentGroup and optional runtime overrides.
```

### 2. API Validates The Request

The API validates:

- The deployment group exists.
- Target IDs exist or target assignments are present.
- Runtime JSON override is valid JSON.
- A deployment rule can be resolved from the deployment group or selected template metadata.

### 3. API Creates Or Updates Legacy DeploymentExecutions

Current compatibility behavior:

- The API still creates or updates `DeploymentExecutions`.
- These records are useful for existing UI views and migration compatibility.

Target direction:

- `DeploymentExecutions` should become either a read-only compatibility view or be replaced by queue job target/artifact state.
- The deployment composition and queue job should become the primary execution model.

### 4. API Creates A QueueJob

The API creates a `QueueJob`.

The queue job contains:

- Job type
- Status
- Correlation ID
- Priority
- Schedule and lock metadata
- Payload JSON
- Rule snapshot JSON
- Queue job targets
- Queue job steps

The payload includes the deployment group, selected template versions, target assignments, target IDs, runtime JSON, and metadata.

## Queue Process

### 1. Worker Claims A Job

The worker looks for pending jobs.

It claims a job atomically by setting:

```text
Status = Running
Attempts += 1
LockedBy
LockedUntil
HeartbeatAt
WorkerName
```

This prevents two workers from processing the same job at the same time.

### 2. Worker Processes Steps

Queue jobs can contain steps.

Examples:

- Approval
- Provision
- Validate
- Custom future step types

Approval steps can pause the job until a user approves or rejects them through the API.

### 3. Worker Processes Queue Targets

For each queue target, the worker loads the deployment composition:

```text
DeploymentGroup
  TemplateSelections
  ParameterValues
  TargetAssignments
  Target
  Environment and Domain context
```

The worker then renders artifacts for that target.

Current renderers:

- PowerShell DSC v2 data file renderer
- DSC v3 JSON renderer

### 4. Worker Writes Output Metadata

The worker writes target and step metadata back to the queue records.

Examples:

- Artifact paths
- Renderer name
- Finished timestamps
- Errors

## Composition And Rendering Process

### 1. Load Selected Template Versions

The worker loads all template selections in sort order.

Example:

```text
Environment Default
Environment Contoso Test
Landscape Test
Service SharePoint
Stage Install
```

### 2. Compose Documents

The selected template JSON documents are merged into one effective deployment document.

Conceptually:

```text
Parameters
Variables
Resources
Targets / AllNodes
Metadata
```

Current worker state:

- It can load the composition.
- It can render ordered PowerShell data files and JSON artifacts.

Target direction:

- DSC v2 renderer should call the existing `Merge-DSCConfigurationData` and `Resolve-DSCConfigurationData` modules.
- DSC v3 renderer can consume or emit JSON directly.

### 3. Resolve Parameters, Variables, And Secrets

The resolve module is responsible for resolving expressions and secrets.

Examples:

```text
[parameters('DatabasePrefix')]
[variables('ConfigDbName')]
[concat(parameters('DatabasePrefix'), '_Config')]
```

For previews, secrets should be skipped or replaced with dummy values.
For real deployments, secrets are resolved through the configured credential provider.

## Preview Process

This is not fully implemented yet, but the intended flow is:

```text
User edits deployment composition
  -> clicks Preview
  -> API builds effective deployment composition
  -> API or worker-style service merges selected template versions
  -> Resolve runs with SkipSecrets
  -> GUI shows effective parameters, variables, resources, and target data
```

The preview should help users see what will actually be deployed before a queue job is created.

This belongs mostly to Step 8 and Step 8a.

## Promotion Process

This is not fully implemented yet.

Target idea:

```text
Change shared domain or service baseline
  -> create new template version
  -> test in Test environment
  -> promote the same immutable version to QA or Prod deployment groups
```

Promotion should not mean copying large JSON blocks repeatedly.
It should mean reusing or advancing selected template versions in deployment groups.

## Current Implemented State

Implemented:

- Generic targets exist.
- Template versions exist and are versioned with hashes.
- Deployment groups can store template selections.
- Deployment groups can store parameter values.
- Deployment groups can store target assignments.
- Queue jobs have claim/lock/heartbeat metadata.
- Queue job targets and steps can persist output metadata.
- Worker can load deployment composition.
- Worker can render DSC v2-style PowerShell data files and DSC v3-style JSON artifacts.
- Web can create a rudimentary deployment group using a selected template version and selected targets.

Partially implemented:

- Legacy `DeploymentGroup.TemplateId` and `DeploymentExecution.JSONData` are still present for compatibility.
- API still mixes repository and direct `DataContext` logic in some controllers.
- Queue payloads include composition data, but the API contract is not yet fully cleaned up.

Pending:

- Dedicated deployment composition service.
- Transactional deployment group creation.
- Rich validation results instead of generic `false` / `500`.
- Preview endpoint based on merge and resolve.
- Sealed parameter and resource block visibility in API/GUI.
- Promotion flow.
- Final cleanup of legacy template JSON and deployment execution fields.

## Expected Future Clean Flow

The desired future flow should look like this:

```text
1. Admin maintains catalog:
   Environments, Domains, Services, Targets

2. Admin maintains templates:
   Template -> TemplateVersion -> Published immutable version

3. User creates deployment group:
   Select environment/context
   Select template versions
   Assign targets
   Set parameter values

4. User previews:
   Merge selected template versions
   Resolve parameters and variables
   Resolve secrets as dummy values
   Show effective output

5. User starts deployment:
   API creates QueueJob
   Worker claims QueueJob
   Worker renders artifacts
   Worker executes or hands off
   API reports status and metadata

6. User promotes:
   Reuse tested template versions in the next environment
   Avoid copying environment-specific JSON manually
```

## Main Understanding To Validate

The core understanding is:

```text
Templates describe reusable building blocks.
TemplateVersions make those building blocks immutable.
DeploymentGroups select and order those building blocks.
ParameterValues and TargetAssignments make the deployment concrete.
QueueJobs turn the deployment design into execution.
Workers render and execute without knowing SharePoint-specific database tables.
```

If this is correct, the next API work should focus on making this process stricter and more explicit, not on adding more legacy shortcuts.