# SelfService Portal Process ## Purpose This document describes the current end-to-end process as understood for the SelfService Portal. It is intentionally written as a business and platform flow, not as a low-level API reference. The goal is to make visible how catalog data, reusable templates, deployment composition, queue jobs, and worker execution fit together. ## Big Picture The portal is intended to become a generic configuration composition platform. It should support on-premises workloads such as Active Directory, SQL Server, SharePoint, and Exchange, and also cloud workloads such as Teams, Azure, Microsoft 365, and Azure resources. The important idea is: ```text Reusable catalog data + immutable template versions + deployment-specific choices -> composed deployment document -> queue job -> worker renders and executes ``` The database should know generic concepts such as environments, domains, targets, services, templates, template versions, deployment groups, queue jobs, and artifacts. Workload-specific details, for example SharePoint farm accounts or Teams policies, belong inside versioned template documents. ## Roles Of The Applications ### Microsoft.SelfService.Portal.Core.API The API owns the central data model and exposes it to the GUI and worker. Responsibilities: - Store catalog data such as environments, domains, services, targets, templates, and template versions. - Store deployment composition data such as selected template versions, parameter overrides, and target assignments. - Validate JSON documents before storing them. - Create deployment queue jobs. - Expose queue job status, steps, target progress, errors, and metadata. ### Microsoft.SelfService.Portal.Web The web frontend is the user-facing surface. Responsibilities: - Let users browse catalog data. - Let users create and edit deployment groups. - Let users select template versions, targets, parameters, and later preview effective configuration data. - Submit deployment requests into the API queue. - Show queue job state and execution progress. The current GUI is intentionally rudimentary. The later target is a richer interactive deployment builder. ### Microsoft.SelfService.Portal.Core.Worker The worker processes queued deployment jobs. Responsibilities: - Claim jobs safely from the queue. - Load deployment composition from the API database. - Render the deployment composition into an artifact. - Support different renderers, currently DSC v2 PowerShell data files and DSC v3 JSON. - Later execute or hand off the rendered artifacts. - Persist target and step output metadata. ## Catalog Process ### 1. Environments Are Defined An environment is a logical deployment context, for example: ```text Prod A Test A QA B Prod B Test B ``` An environment describes where something belongs and what stage it represents. Examples of environment metadata: - Stage: Test, QA, Production - Hosting type: OnPrem, Azure, M365, Hybrid - Provider type - Tenant or subscription reference ### 2. Domains Are Defined Once And Linked A domain is reusable catalog and configuration data. It can be linked to multiple environments. Example: ```text Central-Management Domain -> Test A -> QA A -> Prod A ``` The intent is that a domain configuration is not duplicated for every environment. Instead, a domain baseline can be maintained once, then linked or promoted into the environments that should consume it. Target process: ```text Change Central-Management domain baseline -> validate in Test -> promote or reuse for Prod ``` Not desired: ```text Maintain Central-Management Test separately Maintain Central-Management Prod separately Repeat every change manually ``` ### 3. Services Describe Workload Families A service represents a workload type, not a single deployment. Examples: - Active Directory - SQL Server - SharePoint - Exchange - Teams - Azure Network - Microsoft 365 Policies Templates are categorized below services so users can find suitable building blocks. ### 4. Targets Are Generic Targets are everything a deployment can act on. They are not limited to virtual machines. Examples: - VirtualMachine - Tenant - Subscription - ResourceGroup - User - Group - Site - PolicyScope For on-premises workloads, targets are often servers. For cloud workloads, targets may be tenants, subscriptions, groups, sites, or policy scopes. ## Template Process ### 1. A Template Is A Catalog Entry A template is the stable entry shown in the portal. It has metadata such as name, category, service, and description. The template itself should not be the mutable source of deployment content in the long term. It is the catalog shell. ### 2. TemplateVersions Are Immutable Building Blocks The actual configuration document lives in `TemplateVersion.JsonData`. Each template version contains a JSON document with this general shape: ```json { "schemaVersion": "1.0", "templateType": "Service", "parameters": {}, "variables": {}, "resources": {} } ``` The template version is the thing that should be selected in a deployment. This makes deployments reproducible, because they point to an immutable version instead of a mutable template document. ### 3. Template Documents Stay Generic Template documents can describe many workloads. Examples: - Environment defaults - Domain defaults - Landscape values - Service definitions - Stage-specific settings - Target or role specific blocks The document can contain parameters, variables, and resources. The merge and resolve modules can later combine those pieces into effective configuration data. ## Deployment Design Process ### 1. User Creates A DeploymentGroup A deployment group represents a deployable unit or one part of a larger environment rollout. Example environment rollout: ```text Deployment Group: Contoso Test Environment Deployment AD Targets: 2 domain controller servers Templates: Active Directory baseline, environment defaults, stage defaults Deployment SQL Targets: 1 SQL server Templates: SQL baseline, environment defaults, stage defaults Deployment SharePoint Targets: 6 SharePoint servers Templates: SharePoint baseline, environment defaults, landscape, stage defaults ``` Current implementation still uses `DeploymentGroup`/`DeploymentBatch` naming in places. Conceptually this is the deployment design container. ### 2. User Selects Template Versions The deployment group gets one or more `DeploymentTemplateSelections`. Example: ```text SortOrder 10: Environment Default 1.0 SortOrder 20: Environment Contoso Test 1.0 SortOrder 30: Service SharePoint 1.0 SortOrder 40: Stage Install 1.0 ``` The order matters because the selected template versions are composed in order. Later selections can extend or override earlier selections, unless a parameter or resource block is sealed. Current state: - The API can store multiple template selections. - The GUI can create an initial selection from a selected template version. - The details page can add more selections manually. ### 3. User Assigns Targets The deployment group gets `DeploymentTargetAssignments`. For server-based workloads, each target assignment usually points to a server. Example: ```text Target: CT-SHP-01 RoleKey: WebFrontEnd NodeDataJson: { "nodeName": "CT-SHP-01" } Target: CT-SHP-02 RoleKey: Application NodeDataJson: { "nodeName": "CT-SHP-02" } ``` For cloud workloads, the target may be a tenant, policy scope, group, or site instead of a server. Current state: - The API stores target assignments. - The GUI create flow turns selected targets into target assignments. - Queue requests can use target assignments if explicit target IDs are not sent again. ### 4. User Adds Parameter Values The deployment group can store `DeploymentParameterValues`. Parameter values can be global or scoped to a specific template selection. Examples: ```text Global: DatabasePrefix = Contoso_Test Scoped to SharePoint selection: FarmAccount = secret reference ``` The intention is that the GUI later shows user-facing parameter forms based on selected template versions. The user edits parameter values, and the portal can preview the effective resolved result. Current state: - API stores parameter values. - Values are JSON validated. - Secret references can be marked. - Rich parameter editor and preview are still pending. ## Deployment Request Process ### 1. User Starts A Deployment When the user starts a deployment, the web frontend calls the API deployment request endpoint. The request currently contains: ```text DeploymentGroupId TargetIds optional JsonData optional deployment override ``` If `TargetIds` are omitted, the API can derive targets from the deployment group's target assignments. Target direction: ```text DeploymentGroup composition should be the source of truth. The start request should eventually only reference the DeploymentGroup and optional runtime overrides. ``` ### 2. API Validates The Request The API validates: - The deployment group exists. - Target IDs exist or target assignments are present. - Runtime JSON override is valid JSON. - A deployment rule can be resolved from the deployment group or selected template metadata. ### 3. API Creates Or Updates Legacy DeploymentExecutions Current compatibility behavior: - The API still creates or updates `DeploymentExecutions`. - These records are useful for existing UI views and migration compatibility. Target direction: - `DeploymentExecutions` should become either a read-only compatibility view or be replaced by queue job target/artifact state. - The deployment composition and queue job should become the primary execution model. ### 4. API Creates A QueueJob The API creates a `QueueJob`. The queue job contains: - Job type - Status - Correlation ID - Priority - Schedule and lock metadata - Payload JSON - Rule snapshot JSON - Queue job targets - Queue job steps The payload includes the deployment group, selected template versions, target assignments, target IDs, runtime JSON, and metadata. ## Queue Process ### 1. Worker Claims A Job The worker looks for pending jobs. It claims a job atomically by setting: ```text Status = Running Attempts += 1 LockedBy LockedUntil HeartbeatAt WorkerName ``` This prevents two workers from processing the same job at the same time. ### 2. Worker Processes Steps Queue jobs can contain steps. Examples: - Approval - Provision - Validate - Custom future step types Approval steps can pause the job until a user approves or rejects them through the API. ### 3. Worker Processes Queue Targets For each queue target, the worker loads the deployment composition: ```text DeploymentGroup TemplateSelections ParameterValues TargetAssignments Target Environment and Domain context ``` The worker then renders artifacts for that target. Current renderers: - PowerShell DSC v2 data file renderer - DSC v3 JSON renderer ### 4. Worker Writes Output Metadata The worker writes target and step metadata back to the queue records. Examples: - Artifact paths - Renderer name - Finished timestamps - Errors ## Composition And Rendering Process ### 1. Load Selected Template Versions The worker loads all template selections in sort order. Example: ```text Environment Default Environment Contoso Test Landscape Test Service SharePoint Stage Install ``` ### 2. Compose Documents The selected template JSON documents are merged into one effective deployment document. Conceptually: ```text Parameters Variables Resources Targets / AllNodes Metadata ``` Current worker state: - It can load the composition. - It can render ordered PowerShell data files and JSON artifacts. Target direction: - DSC v2 renderer should call the existing `Merge-DSCConfigurationData` and `Resolve-DSCConfigurationData` modules. - DSC v3 renderer can consume or emit JSON directly. ### 3. Resolve Parameters, Variables, And Secrets The resolve module is responsible for resolving expressions and secrets. Examples: ```text [parameters('DatabasePrefix')] [variables('ConfigDbName')] [concat(parameters('DatabasePrefix'), '_Config')] ``` For previews, secrets should be skipped or replaced with dummy values. For real deployments, secrets are resolved through the configured credential provider. ## Preview Process This is not fully implemented yet, but the intended flow is: ```text User edits deployment composition -> clicks Preview -> API builds effective deployment composition -> API or worker-style service merges selected template versions -> Resolve runs with SkipSecrets -> GUI shows effective parameters, variables, resources, and target data ``` The preview should help users see what will actually be deployed before a queue job is created. This belongs mostly to Step 8 and Step 8a. ## Promotion Process This is not fully implemented yet. Target idea: ```text Change shared domain or service baseline -> create new template version -> test in Test environment -> promote the same immutable version to QA or Prod deployment groups ``` Promotion should not mean copying large JSON blocks repeatedly. It should mean reusing or advancing selected template versions in deployment groups. ## Current Implemented State Implemented: - Generic targets exist. - Template versions exist and are versioned with hashes. - Deployment groups can store template selections. - Deployment groups can store parameter values. - Deployment groups can store target assignments. - Queue jobs have claim/lock/heartbeat metadata. - Queue job targets and steps can persist output metadata. - Worker can load deployment composition. - Worker can render DSC v2-style PowerShell data files and DSC v3-style JSON artifacts. - Web can create a rudimentary deployment group using a selected template version and selected targets. Partially implemented: - Legacy `DeploymentGroup.TemplateId` and `DeploymentExecution.JSONData` are still present for compatibility. - API still mixes repository and direct `DataContext` logic in some controllers. - Queue payloads include composition data, but the API contract is not yet fully cleaned up. Pending: - Dedicated deployment composition service. - Transactional deployment group creation. - Rich validation results instead of generic `false` / `500`. - Preview endpoint based on merge and resolve. - Sealed parameter and resource block visibility in API/GUI. - Promotion flow. - Final cleanup of legacy template JSON and deployment execution fields. ## Expected Future Clean Flow The desired future flow should look like this: ```text 1. Admin maintains catalog: Environments, Domains, Services, Targets 2. Admin maintains templates: Template -> TemplateVersion -> Published immutable version 3. User creates deployment group: Select environment/context Select template versions Assign targets Set parameter values 4. User previews: Merge selected template versions Resolve parameters and variables Resolve secrets as dummy values Show effective output 5. User starts deployment: API creates QueueJob Worker claims QueueJob Worker renders artifacts Worker executes or hands off API reports status and metadata 6. User promotes: Reuse tested template versions in the next environment Avoid copying environment-specific JSON manually ``` ## Main Understanding To Validate The core understanding is: ```text Templates describe reusable building blocks. TemplateVersions make those building blocks immutable. DeploymentGroups select and order those building blocks. ParameterValues and TargetAssignments make the deployment concrete. QueueJobs turn the deployment design into execution. Workers render and execute without knowing SharePoint-specific database tables. ``` If this is correct, the next API work should focus on making this process stricter and more explicit, not on adding more legacy shortcuts.