DotNet-Projekte/Microsoft.SelfService.Portal.Core.API
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add queue hardening by modifying DeploymentJobTargets, DeploymentJobSteps, and DeploymentJobs tables

Browse Source 
- Drop existing indexes on DeploymentJobTargets and DeploymentJobSteps
- Alter Status column to nvarchar(450) in DeploymentJobTargets, DeploymentJobSteps, and DeploymentJobs
- Add new columns: Finished, OutputMetadataJson, Started to DeploymentJobTargets; ErrorMessage, Finished, OutputMetadataJson, Started to DeploymentJobSteps; CorrelationId, HeartbeatAt, Priority, RowVersion, ScheduledAt, WorkerName to DeploymentJobs
- Create new indexes for improved query performance
- Add constraints to ensure OutputMetadataJson is valid JSON
- Record migration in __EFMigrationsHistory
This commit is contained in:
Torsten Brendgen
2026-07-09 08:50:09 +02:00
parent 97238c28c9
commit 01c2a5df2e
45 changed files with 8198 additions and 75 deletions
Download Patch File Download Diff File Expand all files Collapse all files

597
Docs/Architecture/Process.md Normal file
View File

@@ -0,0 +1,597 @@
# SelfService Portal Process
## Purpose
This document describes the current end-to-end process as understood for the SelfService Portal.
It is intentionally written as a business and platform flow, not as a low-level API reference.
The goal is to make visible how catalog data, reusable templates, deployment composition, queue jobs, and worker execution fit together.
## Big Picture
The portal is intended to become a generic configuration composition platform.
It should support on-premises workloads such as Active Directory, SQL Server, SharePoint, and Exchange, and also cloud workloads such as Teams, Azure, Microsoft 365, and Azure resources.
The important idea is:
```text
Reusable catalog data + immutable template versions + deployment-specific choices
-> composed deployment document
-> queue job
-> worker renders and executes
```
The database should know generic concepts such as environments, domains, targets, services, templates, template versions, deployment groups, queue jobs, and artifacts.
Workload-specific details, for example SharePoint farm accounts or Teams policies, belong inside versioned template documents.
## Roles Of The Applications
### Microsoft.SelfService.Portal.Core.API
The API owns the central data model and exposes it to the GUI and worker.
Responsibilities:
- Store catalog data such as environments, domains, services, targets, templates, and template versions.
- Store deployment composition data such as selected template versions, parameter overrides, and target assignments.
- Validate JSON documents before storing them.
- Create deployment queue jobs.
- Expose queue job status, steps, target progress, errors, and metadata.
### Microsoft.SelfService.Portal.Web
The web frontend is the user-facing surface.
Responsibilities:
- Let users browse catalog data.
- Let users create and edit deployment groups.
- Let users select template versions, targets, parameters, and later preview effective configuration data.
- Submit deployment requests into the API queue.
- Show queue job state and execution progress.
The current GUI is intentionally rudimentary. The later target is a richer interactive deployment builder.
### Microsoft.SelfService.Portal.Core.Worker
The worker processes queued deployment jobs.
Responsibilities:
- Claim jobs safely from the queue.
- Load deployment composition from the API database.
- Render the deployment composition into an artifact.
- Support different renderers, currently DSC v2 PowerShell data files and DSC v3 JSON.
- Later execute or hand off the rendered artifacts.
- Persist target and step output metadata.
## Catalog Process
### 1. Environments Are Defined
An environment is a logical deployment context, for example:
```text
Prod A
Test A
QA B
Prod B
Test B
```
An environment describes where something belongs and what stage it represents.
Examples of environment metadata:
- Stage: Test, QA, Production
- Hosting type: OnPrem, Azure, M365, Hybrid
- Provider type
- Tenant or subscription reference
### 2. Domains Are Defined Once And Linked
A domain is reusable catalog and configuration data.
It can be linked to multiple environments.
Example:
```text
Central-Management Domain
-> Test A
-> QA A
-> Prod A
```
The intent is that a domain configuration is not duplicated for every environment.
Instead, a domain baseline can be maintained once, then linked or promoted into the environments that should consume it.
Target process:
```text
Change Central-Management domain baseline
-> validate in Test
-> promote or reuse for Prod
```
Not desired:
```text
Maintain Central-Management Test separately
Maintain Central-Management Prod separately
Repeat every change manually
```
### 3. Services Describe Workload Families
A service represents a workload type, not a single deployment.
Examples:
- Active Directory
- SQL Server
- SharePoint
- Exchange
- Teams
- Azure Network
- Microsoft 365 Policies
Templates are categorized below services so users can find suitable building blocks.
### 4. Targets Are Generic
Targets are everything a deployment can act on.
They are not limited to virtual machines.
Examples:
- VirtualMachine
- Tenant
- Subscription
- ResourceGroup
- User
- Group
- Site
- PolicyScope
For on-premises workloads, targets are often servers.
For cloud workloads, targets may be tenants, subscriptions, groups, sites, or policy scopes.
## Template Process
### 1. A Template Is A Catalog Entry
A template is the stable entry shown in the portal.
It has metadata such as name, category, service, and description.
The template itself should not be the mutable source of deployment content in the long term.
It is the catalog shell.
### 2. TemplateVersions Are Immutable Building Blocks
The actual configuration document lives in `TemplateVersion.JsonData`.
Each template version contains a JSON document with this general shape:
```json
{
"schemaVersion": "1.0",
"templateType": "Service",
"parameters": {},
"variables": {},
"resources": {}
}
```
The template version is the thing that should be selected in a deployment.
This makes deployments reproducible, because they point to an immutable version instead of a mutable template document.
### 3. Template Documents Stay Generic
Template documents can describe many workloads.
Examples:
- Environment defaults
- Domain defaults
- Landscape values
- Service definitions
- Stage-specific settings
- Target or role specific blocks
The document can contain parameters, variables, and resources.
The merge and resolve modules can later combine those pieces into effective configuration data.
## Deployment Design Process
### 1. User Creates A DeploymentGroup
A deployment group represents a deployable unit or one part of a larger environment rollout.
Example environment rollout:
```text
Deployment Group: Contoso Test Environment
Deployment AD
Targets: 2 domain controller servers
Templates: Active Directory baseline, environment defaults, stage defaults
Deployment SQL
Targets: 1 SQL server
Templates: SQL baseline, environment defaults, stage defaults
Deployment SharePoint
Targets: 6 SharePoint servers
Templates: SharePoint baseline, environment defaults, landscape, stage defaults
```
Current implementation still uses `DeploymentGroup`/`DeploymentBatch` naming in places.
Conceptually this is the deployment design container.
### 2. User Selects Template Versions
The deployment group gets one or more `DeploymentTemplateSelections`.
Example:
```text
SortOrder 10: Environment Default 1.0
SortOrder 20: Environment Contoso Test 1.0
SortOrder 30: Service SharePoint 1.0
SortOrder 40: Stage Install 1.0
```
The order matters because the selected template versions are composed in order.
Later selections can extend or override earlier selections, unless a parameter or resource block is sealed.
Current state:
- The API can store multiple template selections.
- The GUI can create an initial selection from a selected template version.
- The details page can add more selections manually.
### 3. User Assigns Targets
The deployment group gets `DeploymentTargetAssignments`.
For server-based workloads, each target assignment usually points to a server.
Example:
```text
Target: CT-SHP-01
RoleKey: WebFrontEnd
NodeDataJson: { "nodeName": "CT-SHP-01" }
Target: CT-SHP-02
RoleKey: Application
NodeDataJson: { "nodeName": "CT-SHP-02" }
```
For cloud workloads, the target may be a tenant, policy scope, group, or site instead of a server.
Current state:
- The API stores target assignments.
- The GUI create flow turns selected targets into target assignments.
- Queue requests can use target assignments if explicit target IDs are not sent again.
### 4. User Adds Parameter Values
The deployment group can store `DeploymentParameterValues`.
Parameter values can be global or scoped to a specific template selection.
Examples:
```text
Global:
DatabasePrefix = Contoso_Test
Scoped to SharePoint selection:
FarmAccount = secret reference
```
The intention is that the GUI later shows user-facing parameter forms based on selected template versions.
The user edits parameter values, and the portal can preview the effective resolved result.
Current state:
- API stores parameter values.
- Values are JSON validated.
- Secret references can be marked.
- Rich parameter editor and preview are still pending.
## Deployment Request Process
### 1. User Starts A Deployment
When the user starts a deployment, the web frontend calls the API deployment request endpoint.
The request currently contains:
```text
DeploymentGroupId
TargetIds optional
JsonData optional deployment override
```
If `TargetIds` are omitted, the API can derive targets from the deployment group's target assignments.
Target direction:
```text
DeploymentGroup composition should be the source of truth.
The start request should eventually only reference the DeploymentGroup and optional runtime overrides.
```
### 2. API Validates The Request
The API validates:
- The deployment group exists.
- Target IDs exist or target assignments are present.
- Runtime JSON override is valid JSON.
- A deployment rule can be resolved from the deployment group or selected template metadata.
### 3. API Creates Or Updates Legacy DeploymentExecutions
Current compatibility behavior:
- The API still creates or updates `DeploymentExecutions`.
- These records are useful for existing UI views and migration compatibility.
Target direction:
- `DeploymentExecutions` should become either a read-only compatibility view or be replaced by queue job target/artifact state.
- The deployment composition and queue job should become the primary execution model.
### 4. API Creates A QueueJob
The API creates a `QueueJob`.
The queue job contains:
- Job type
- Status
- Correlation ID
- Priority
- Schedule and lock metadata
- Payload JSON
- Rule snapshot JSON
- Queue job targets
- Queue job steps
The payload includes the deployment group, selected template versions, target assignments, target IDs, runtime JSON, and metadata.
## Queue Process
### 1. Worker Claims A Job
The worker looks for pending jobs.
It claims a job atomically by setting:
```text
Status = Running
Attempts += 1
LockedBy
LockedUntil
HeartbeatAt
WorkerName
```
This prevents two workers from processing the same job at the same time.
### 2. Worker Processes Steps
Queue jobs can contain steps.
Examples:
- Approval
- Provision
- Validate
- Custom future step types
Approval steps can pause the job until a user approves or rejects them through the API.
### 3. Worker Processes Queue Targets
For each queue target, the worker loads the deployment composition:
```text
DeploymentGroup
TemplateSelections
ParameterValues
TargetAssignments
Target
Environment and Domain context
```
The worker then renders artifacts for that target.
Current renderers:
- PowerShell DSC v2 data file renderer
- DSC v3 JSON renderer
### 4. Worker Writes Output Metadata
The worker writes target and step metadata back to the queue records.
Examples:
- Artifact paths
- Renderer name
- Finished timestamps
- Errors
## Composition And Rendering Process
### 1. Load Selected Template Versions
The worker loads all template selections in sort order.
Example:
```text
Environment Default
Environment Contoso Test
Landscape Test
Service SharePoint
Stage Install
```
### 2. Compose Documents
The selected template JSON documents are merged into one effective deployment document.
Conceptually:
```text
Parameters
Variables
Resources
Targets / AllNodes
Metadata
```
Current worker state:
- It can load the composition.
- It can render ordered PowerShell data files and JSON artifacts.
Target direction:
- DSC v2 renderer should call the existing `Merge-DSCConfigurationData` and `Resolve-DSCConfigurationData` modules.
- DSC v3 renderer can consume or emit JSON directly.
### 3. Resolve Parameters, Variables, And Secrets
The resolve module is responsible for resolving expressions and secrets.
Examples:
```text
[parameters('DatabasePrefix')]
[variables('ConfigDbName')]
[concat(parameters('DatabasePrefix'), '_Config')]
```
For previews, secrets should be skipped or replaced with dummy values.
For real deployments, secrets are resolved through the configured credential provider.
## Preview Process
This is not fully implemented yet, but the intended flow is:
```text
User edits deployment composition
-> clicks Preview
-> API builds effective deployment composition
-> API or worker-style service merges selected template versions
-> Resolve runs with SkipSecrets
-> GUI shows effective parameters, variables, resources, and target data
```
The preview should help users see what will actually be deployed before a queue job is created.
This belongs mostly to Step 8 and Step 8a.
## Promotion Process
This is not fully implemented yet.
Target idea:
```text
Change shared domain or service baseline
-> create new template version
-> test in Test environment
-> promote the same immutable version to QA or Prod deployment groups
```
Promotion should not mean copying large JSON blocks repeatedly.
It should mean reusing or advancing selected template versions in deployment groups.
## Current Implemented State
Implemented:
- Generic targets exist.
- Template versions exist and are versioned with hashes.
- Deployment groups can store template selections.
- Deployment groups can store parameter values.
- Deployment groups can store target assignments.
- Queue jobs have claim/lock/heartbeat metadata.
- Queue job targets and steps can persist output metadata.
- Worker can load deployment composition.
- Worker can render DSC v2-style PowerShell data files and DSC v3-style JSON artifacts.
- Web can create a rudimentary deployment group using a selected template version and selected targets.
Partially implemented:
- Legacy `DeploymentGroup.TemplateId` and `DeploymentExecution.JSONData` are still present for compatibility.
- API still mixes repository and direct `DataContext` logic in some controllers.
- Queue payloads include composition data, but the API contract is not yet fully cleaned up.
Pending:
- Dedicated deployment composition service.
- Transactional deployment group creation.
- Rich validation results instead of generic `false` / `500`.
- Preview endpoint based on merge and resolve.
- Sealed parameter and resource block visibility in API/GUI.
- Promotion flow.
- Final cleanup of legacy template JSON and deployment execution fields.
## Expected Future Clean Flow
The desired future flow should look like this:
```text
1. Admin maintains catalog:
Environments, Domains, Services, Targets
2. Admin maintains templates:
Template -> TemplateVersion -> Published immutable version
3. User creates deployment group:
Select environment/context
Select template versions
Assign targets
Set parameter values
4. User previews:
Merge selected template versions
Resolve parameters and variables
Resolve secrets as dummy values
Show effective output
5. User starts deployment:
API creates QueueJob
Worker claims QueueJob
Worker renders artifacts
Worker executes or hands off
API reports status and metadata
6. User promotes:
Reuse tested template versions in the next environment
Avoid copying environment-specific JSON manually
```
## Main Understanding To Validate
The core understanding is:
```text
Templates describe reusable building blocks.
TemplateVersions make those building blocks immutable.
DeploymentGroups select and order those building blocks.
ParameterValues and TargetAssignments make the deployment concrete.
QueueJobs turn the deployment design into execution.
Workers render and execute without knowing SharePoint-specific database tables.
```
If this is correct, the next API work should focus on making this process stricter and more explicit, not on adding more legacy shortcuts.