DC-UI: DataCollector User Interface Deep Dive

🖥️ DC-UI Overview & Capabilities

DC-UI is the web-based Experience Platform plugin that provides the complete interface for managing DataCollector. All configuration edits happen through either visual forms or direct YAML editing with validation. DC-UI manages configuration, triggers and monitors inventory runs, tests configurations before deploying, browses collected entities, analyzes past runs, and tracks administrative events.

What DC-UI Can Do

High-Level Areas (Navigation)

• Inventories – Real-time system status, trigger runs, system management
• Entities Explorer – Browse collected entities with server-side filtering
• Past Inventory Runs Analytics – Detailed event analysis per run
• Cross-Inventory View – Multi-run analysis across time
• Admin Events – Track operations, health checks, config changes
• Configuration – Systems, System Types, Strategies, Connectors, Schemas, Credentials

Access & Integration

DC-UI is loaded as an Experience Platform plugin. Access via the Experience shell at the configured route (typically /dc-ui). The plugin is located in experience/plugins/dc-ui/ and integrates with DataCollector's REST API for all operations.

Keyboard Shortcuts

DC-UI implements Blue Steel animations and keyboard shortcuts for power users:

• Ctrl+1 through Ctrl+4 – Navigate between main sections
• Ctrl+R – Refresh current data (e.g., Admin Events)
• Ctrl+S – Save (where applicable)
• Escape – Close modals or cancel actions

📦 Inventories Page

The Inventories page displays systems grouped by organization with real-time monitoring and full system management capabilities. It is the primary operational dashboard for data collection.

Grouped Layout, Streams as Sub-Rows

Systems are grouped by their group field (or Ungrouped when missing). Each group has a collapsible header row; group expansion is persisted in localStorage:

• Group header row — chevron / system count / aggregated stream-status counts / aggregated inventory-status counts / aggregated health success-failure tags / sum of entity counts
• Single-stream system — one row containing the system + its stream's controls inline; the system name is bold with no expand chevron
• Multi-stream system — a clickable parent row that toggles inline expansion (state persisted), shows aggregated status counts and the entity total, and exposes only system-level actions. Per-stream sub-rows render indented below when expanded, each with its own status, lifecycle, and per-stream actions.

Real-Time Status Polling

Status polls on the interval declared in the plugin manifest (sdk.config.polling.intervalMs — currently 2 seconds in the shipped manifest, but it's whatever the deployment sets). The page throws at construction time if the manifest doesn't supply it. Polling uses AbortController and an in-flight flag to prevent overlapping requests, and only the systems list refreshes on each tick. Health timeline is fetched once via getHealthTimeline(24, 'system_name') and refreshed every 5 minutes independently.

Stream-level status tags:

• enabled (success), disabled (warning), gone (default), error (error)
• A blue State Exists tag appears alongside when inventory_state_exists is true

Stream-level inventory status (live execution):

• RUNNING (green, play icon) — also surfaces the holding instance_id as a blue tag, a Started: timestamp line, and a View Diagram link to /dc-ui/inventory-diagram/:systemName/:streamName
• STOPPING (orange, stop icon)
• UNRESPONSIVE (red, "Unresponsive, expiring") — instance lease about to expire
• NOT_RUNNING (default)

Trigger Modes

DC has five engine-level trigger modes (day 26 covers the underlying execution-mode semantics). The Trigger Type modal presents them as four cards — Replay and Readonly Replay share one card with a Mode Switch (Writable publishes; Readonly previews only), since they share all other parameters:

Each mode allows selecting inventory_input source: from previous run, empty, custom JSON, or (when the strategy declares them) a named trigger profile via get_trigger_profiles().

Trigger Type Modal — choosing a mode

Clicking Trigger on a stream opens the Trigger Type modal. The header carries the system + stream context; the body offers four mode cards plus an optional Commands card:

• New Writable — production inventory: persists to the DB and publishes diffs
• Detached Readonly — in-memory inventory; no DB writes, no publishing
• Existing Readonly — attach to whatever inventory state is already in the database; results stay in memory
• Replay — re-publish stored entity data through the diff pipeline; no connector calls

The modal also shows a Commands card whenever the strategy publishes commands applicable to the current scope (system-level commands when no stream is selected; stream-level commands when a stream is selected; both for single-stream systems). Clicking the Commands card slides out a right panel listing each command as a small card with name, description, and stream label; clicking a command card runs it immediately and closes the modal — commands modify the stream's next_inventory_input via CAS without starting an inventory.

Trigger Inventory Modal — per-mode contents

Clicking a mode card opens the Trigger Inventory modal with mode-specific options:

Per-Stream vs Per-System Actions

Action sets are contextual — they appear on the row whose scope they apply to:

All actions gate through withAuth('dc-ui:system', 'manage', …); per-row loading replaces the row's tags with a spinner while an operation is pending. Click events on actions stopPropagation so they don't toggle the row's expansion.

Filtering, Grouping, Search

The filter row sits above the table; preferences are persisted in localStorage:

• Search — multi-term, AND, case-insensitive; tree filter — a system stays visible when every term matches the group, the system, or any of its stream names
• Existing only — Switch; filters out systems whose every stream is GONE
• Running only — Switch; keeps systems with any non-NOT_RUNNING stream

The breadcrumb shows a DC is live Badge (green) or DC is offline (red). Ctrl+R triggers a refresh.

Health Column & Entity Counts

The Health column renders a HealthCheckMiniGraph per system (not per stream sub-row), filtered by system name from the shared timeline data. Double-click a graph to navigate to Admin Events filtered to that system's health checks — the path from "this looks unhealthy" to "here's the failing event in detail."

The Entity Counts column lists each entity type with its total. A small bar-chart icon opens a Popover with an EntityCountsGrid (lazy-loaded granular counts per entity type, broken down by visibility and store dimensions). An arrow icon navigates to the Entity Explorer scoped to (systemName, entityType); the granular grid's filter clicks navigate to the explorer with pre-applied filters.

Next Inventory Input Column

The Next Inventory Input column shows the strategy-supplied display string when present (with eye-icon Popover for the full JSON), otherwise falls back to a JSON preview with truncation. An edit pencil opens the Edit Next Inventory Input modal: a JSON textarea preloaded with the current NII, an Empty shortcut button that resets to {}, Save / Cancel — and below the textarea, applicable commands as clickable command cards (clicking executes the command immediately and closes the modal).

Cleanup Modal

The Cleanup button is the only place that gates DC's destructive operations. The modal is radio-switched between two cleanup types — only one is enabled at a time, depending on the scope from which Cleanup was opened:

For the entity-purge variant, an optional Scope selector lazy-loads store names via getSystemEntityCounts and lets the user pick a single store (or leave empty for full purge). When a store is picked, an additional checkbox appears: Also delete entities left with no remaining store tracking.

Inventory Execution Flow

The following diagram illustrates the complete execution flow when an inventory is triggered, showing interactions between UI, engine, strategy, connectors, and destinations:

🔍 Entities Explorer

Entities Explorer browses entities collected and stored in PostgreSQL per-system with server-side filtering and pagination. Select a system to see available entity types with counts.

Selecting a Scope

Entity Explorer is scoped by system and entity type together — both must be chosen before the table renders:

• System selector — searchable dropdown ordered by total entity count (with-entities first, then alphabetical). Labels show the count when non-zero, e.g. prod-sap (1,234).
• Entity type selector — radio button group, each labeled with the entity type and a purple count Tag. Auto-selects the first entity type when none is chosen.

The URL reflects the current scope (/dc-ui/entities-explorer/:systemName/:entityType) and updates without a full navigation, so any view is bookmarkable. Per-entity-type arrow icons on the Inventories page deep-link straight here.

Granular Counts Grid

Once a scope is chosen, a granular counts grid appears under the selectors, summarizing the entity population across visibility and store dimensions (deleted, visible-in-store-X, no remaining store tracking, etc.). Clicking any cell drops you into the table pre-filtered to that slice — for example, "deleted entities", "entities visible only in store eid", or "orphan entities with no store tracking left".

Toolbar & Filters

The toolbar (visible after selection) drives every filter dimension. Aside from a scope-spanning Reset, Refresh (also Ctrl+R), Column Picker, and Export, the controls split into "always shown" and a "More Filters" panel:

The More Filters button shows a small Badge with the count of active advanced filters, and auto-expands when any of these were set from URL query params (store, is_deleted, has_alive_stores, has_visible_stores, has_invisible_stores, has_stores, store_visibility). Choosing or clearing a store automatically clears the per-store filters and sorts that no longer apply; setting has_stores=false automatically clears the selected store and per-store filters / sorts (orphan branch).

Fixed and Per-Store Columns

The table mixes a fixed set of entity-level columns with dynamic per-store columns built from the rows currently on screen.

• Fixed entity columns: Key1, Key2, Key3, Relation 1/2/3, Attribute 1/2/3, Updated On, Created On (default hidden), Last Confirmed, Is Deleted, Stores summary
• Per-store columns for each store name S seen in the current rows:
  • S.status (Visible / Invisible / Deleted, default hidden)
  • S.hash (default hidden)
  • S.store_updated_on
  • S.store_created_or_resurrected_on (default hidden)
  • S.transformed_data.<attr> per attribute key seen in attribute_transformed_data
  • S.hashes.<attr> per attribute key seen in attribute_hashes (default hidden)
  • S.ts.<attr> per attribute key seen in attribute_timestamps (default hidden)

Per-column filter dropdown: text input with Contains / Equals radio plus Reset / Search; pressing Enter or clicking Search applies it. Sorting is server-side, default updated_on desc; for store-prefixed columns the prefix is stripped before sending to the API.

Column Picker, Page Size, Persistence

The Column Picker Popover offers Show All / Keys Only preset buttons plus per-column checkboxes (the last visible column is disabled to prevent hiding everything). Page size options are 20 / 50 / 100 with default 50. All three preferences — hidden columns, page size, and the active store — are persisted per system+entityType in localStorage so your layout sticks across sessions.

Cell Rendering

The grid uses explicit, non-collapsing renderers so empty / missing data is unambiguous:

• null → grey "null" Tag
• undefined → grey "—" Tag
• empty string → grey "" Tag
• objects → <code> with the full JSON in a Tooltip

Entity Details Modal

Clicking the eye icon on a row (Actions column, fixed right) opens an 800px-wide tabbed modal:

• Overview — Descriptions grid for keys (Key1, Key2 / Key3 / relation keys only when present), Status (Active / Deleted), Last Confirmed, Created, Updated, plus a Stores summary table (per-store rows with status tag — green Visible / orange Invisible / red Deleted — hash 16-char preview, updated_on, created_or_resurrected_on). When the entity has no stores, the table reads "No stores (orphan entity)".
• Per-store tabs — one per store the entity has, with a status tag in the tab label. Body shows hash, updated_on, created_or_resurrected_on, plus a JsonViewer for attribute_transformed_data (when non-empty), a JsonViewer for attribute_hashes (when non-empty), and a Descriptions grid for attribute_timestamps.
• Raw JSON — the entire entity record dumped through JsonViewer.

Default active tab is the per-store tab matching the toolbar's current store selection (when that store exists on the entity), otherwise Overview.

Export

"All Data" re-fetches with pageSize=total using the same filters and sort. Filename pattern: {systemName}_{entityType}_{YYYY-MM-DD_HH-mm-ss}.{csv|json}.

📊 Past Inventory Runs Analytics

Lists all inventory runs with server-side pagination and filtering. Click any run to drill down into 12 detailed tabs with event-level analysis.

List-Level Filtering

• System, strategy, time range, trigger mode
• Execution statistics in columns: duration, step counts by outcome, connector calls, changes, diffs, instance

12 Drill-Down Tabs

Tab Features

Each tab provides:

• Filtering – Tab-specific filters
• Sorting – By any column
• Column choosers – Show/hide columns with localStorage persistence
• Query statistics – Rows read, bytes read, elapsed time for performance insight

Overview Tab Details

The Overview tab displays aggregated statistics: total duration, step count by outcome (SUCCESS, EXPECTED_FAILURE, FATAL_FAILURE, RETRYABLE_FAILURE), connector call count, changes count, diffs published, instance that ran it, trigger mode, strategy name. Metadata includes inventory_input summary and next_inventory_input if available.

Steps Tab Details

Shows execution timeline with columns: step name, step parameter (for generic execute steps), outcome, start time, duration, instance. Each step lifecycle event (started, finished) is captured. Filter by step name or outcome to find failures quickly.

Connector Calls Tab Details

Expandable rows show request params (connector name, method, endpoint, query), response status, duration, and raw response body. Useful for debugging API issues—verify request structure and inspect error responses.

Diffs Tab Details

The Diffs tab shows every change-decision the engine emitted during the run. Decisions split into two families based on whether they produce a diff:

Change-detection-only (no diff produced)

These tags are emitted by the ChangeProcessor (CP) and never reach the DiffPublisher. They explain why an entity was inspected but no diff was routed:

• Unchanged — same hash as last inventory; no work to do
• Deferred — purge-unseen scheduled, but min_absent_inventories threshold not yet reached
• NotFound — entity didn't exist in the previous state
• Invisible — entity exists but visibility rules (AttributeCheck / ForeignKeyCheck) suppress it
• InvisibleDataChanged — invisible entity's data changed (still suppressed)
• InvisibleUnchanged — invisible entity's data unchanged
• KeysChanged — composite key changed; treated as a new entity

Diff produced and routed through StoreDelta

These tags drive the Diff Publications tab — every diff in this family was sent to one or more destinations:

• Created, Updated, Deleted, Purged — standard lifecycle transitions
• BecameInvisible / BecameVisible — visibility-rule transitions (entity itself was already tracked)
• CascadeBecameInvisible / CascadeBecameVisible — backward-cascade transitions (visibility flipped because of a related entity)

Per-row context: store, destination, attribute_name, deletion_context, doc_id, doc_type_id, old/new hash 8-char preview, dynamic Data.* columns, and an eye icon that opens DetailsModal with the full record. See strategies-development/050-changes-processing-pipeline.md for the decision logic.

Performance Tab (Gantt Chart)

Visual timeline of step execution—see parallelism, bottlenecks, and overlap. Steps that ran in parallel appear as concurrent bars. Identify slow steps that block others.

Step Dispatch Chain

Execution flow showing how steps dispatched to each other (e.g., entry → execute:list_users → execute:list_groups). Complements the Steps tab by showing the logical flow rather than chronological order.

📈 Cross-Inventory View

Analyzes data across multiple inventory runs with 8 tabs. Each view has date range picker, specific search filters, and charts visualizing trends. All use server-side pagination and query statistics display.

Entity History Deep Dive

For a given entity key, Entity History shows every inventory run that touched that entity: when it was added, when it was updated (with hash changes), when it was deleted. Enables tracking drift or debugging why an entity disappeared.

Step Performance Deep Dive

Charts show trends over time: execution count per step, success rate (%), average duration. Identify degrading performance—e.g., a step that used to take 2s now takes 10s as data volume grew.

Instance Load Deep Dive

Stacked area chart: X-axis is time, Y-axis is active inventory count. Each DC instance has a band. Use to verify load balancing—inventories should distribute across instances rather than pile up on one.

Date Range Picker

All tabs support date range selection. Default may be last 24 hours or 7 days. Narrow range for focused investigation; widen for trend analysis.

📋 Admin Events

Tracks administrative operations across all DC instances. Filter by time range, category, action, instance, system, or user to investigate issues or track changes.

Event Categories Tracked

• Health check results – Automatic scheduled checks and manual testing
• Configuration changes – Full before/after YAML diffs for audit
• Inventory triggers – Who triggered what and when
• System management operations – Enable/disable/cleanup
• DC lifecycle events – Instance start/stop with config snapshots

Filter Preset Tiles

Quick-filter preset tiles for common investigations:

Details Modal

Clicking the eye icon on any event opens the Details modal — a tabbed view of the full record:

• Overview — primary fields grid (timestamp, category, action, system, success, duration, user, instance, error message)
• Details — additional fields + JSON viewers for nested data
• Changes — side-by-side YAML diff for config_change events (Before panel with red header, After panel with green header); theme-aware styling

URL Sync

Filters sync to URL using history.replaceState(). Page loads with filters from URL if present. Supports: category, action, past_hours, system_name, credential_pointer, success, has_user, instance_id, user_id.

Filter Dropdown Options

Dropdown options are loaded from distinct API calls: getDistinctCategories(), getDistinctActions(category), getDistinctInstances(), getDistinctUsers(), getDistinctSystemNames(), getDistinctCredentialPointers(). Category dropdown uses colored tags with icons. Active filters show blue border highlight. Clear Filters button appears when any non-default filter is set.

Past Hours Options

⚙️ Configuration Management

Systems

Instance configurations using strategy-specific visual designers. Each strategy has a dedicated designer with tab-based layout:

System and system-type edit pages are tab-organized; the exact tab list depends on whether the strategy is Universal (configured entirely in YAML) or one of the specific strategies (each with its own designer). The four canonical lists below are the source of truth — they reappear in the Quick Reference further down.

Universal System tabs

Specific-strategy System tabs

System Types

Template configurations with strategy-specific designers. The Universal System Type designer publishes a long tab set; specific-strategy designers stay narrower.

Universal System Type tabs

• General — strategy selector, monitoring, group; designer content
• Connection — CircuitBreakerEditor
• Settings — SettingsPanel (connector, schedule, execution, storage, health_check); embedded StoresEditor; embedded MonitoringEditor with level=systemType
• Variables — VariablesSchemaEditor (defines what systems must provide)
• Connector Calls — ConnectorCallsTable (named calls with params, produces, next_connector_calls, on_start)
• Before Finish — FinishProducesEditor (purge_unseen, purge_by_relation)
• Debugging — live test panel
• Flow — ReactFlow diagram (rendered conditionally when the parsed YAML has connector_calls)
• Used By (existing only) — systems referencing this system type
• YAML — raw editor

Specific-strategy System Type tabs

• General
• Connection
• Stores — StoresEditor with EmpowerID / Kafka / File destination toggles
• Settings
• Filters (LDAP Delta only) — strategy-specific filter editor inserted between Settings and Storage
• Storage — required StorageConfigEditor (allowUnset=false)
• Debugging (existing only) — page-level tab added by SystemTypeEditPage for non-Universal strategies; Universal gets its Debugging tab through getUniversalDesignerTabs instead
• Used By (existing only)
• YAML

Strategies & Connectors

Custom plugins support 5-second hot-reload—changes detected without restarting DC.

Schema Mappings

Credentials

Connector-specific designers:

• LDAP – Bind DN and password fields
• SCIM – OAuth2 client credentials or bearer tokens
• REST – Various auth types (bearer, basic, API key)

JSON tab shows raw structure with metadata:

• __connector_name – Identifies which connector
• __description – Human-readable context

Metadata stripped by DC before passing to connectors. Form and JSON tabs sync bidirectionally.

Create New System Flow

When creating a new system, you first select a system type. The system type determines which strategy runs and therefore which designer appears. Required fields come from the strategy's config schema. Universal strategy systems require variables matching the system type's variables_schema. Specific strategy systems require strategy-specific fields (e.g., base_url, tenant for SAP IAS).

Expression Types in Schema Mappings

The variable in scope inside a schema-mapping field is result — the raw item being transformed for that store. pair is a different variable scoped to the universal-strategy YAML; don't mix them up. Day 28 covers this in detail.

The Debugging tab on a schema page lets you test these expressions against captured connector-call context (see day 28's SchemaDebugDrawer section).

Credential Pointer URI Format

Systems reference credentials via URIs like openbao+kv2://secret/datacollector/system-name. The UI provides a credential picker that lists existing credentials; selecting one fills the URI. Credentials can be tested from the Credentials list page or from system designer Debugging tab.

📑 Tab-Based Designers Pattern

DC-UI uses a consistent Designer Tab Abstraction where strategy designers publish tabs that appear at the top level alongside General and YAML.

Designer Tab Interface

interface DesignerTab {
  key: string;
  label: React.ReactNode;
  children: React.ReactNode;
}

interface DesignerResult {
  content: React.ReactNode;      // Content for General tab
  additionalTabs: DesignerTab[];  // Extra tabs after General
}

Tab Order

1. General – Non-strategy-specific (selector, monitoring, group) + designer content
2. ...additionalTabs – Strategy-specific tabs from designer
3. YAML – Raw editor (always last)

General Tab Content

The General tab contains minimal shared content: strategy/system-type selector, monitoring overrides, group. Strategy-specific designer content (e.g., connection forms) renders within General.

Specialized Tabs

Strategies provide specialized tabs via additionalTabs. Example—Universal Strategy:

case 'builtin.strategies.universal.UniversalStrategy':
  return {
    content: <UniversalStrategyDesigner {...props} />,
    additionalTabs: getUniversalDesignerTabs({ ...props, yamlContent }),
  };

getUniversalDesignerTabs() returns the Universal-strategy-specific tabs that render between General and YAML: Connection, Settings, Variables, Connector Calls, Before Finish, Debugging, and Flow (conditional on parsed YAML having connector_calls). The page (SystemTypeEditPage) wraps these with the always-present General tab, the existing-only Used By tab, and the always-last YAML tab. Designers encapsulate strategy-specific logic (YAML parsing, flow diagrams) in their modules.

Adding Tabs to Custom Designers

To add tabs to any designer:

1. Create tab content components (e.g., MyCustomTab.tsx)
2. Export getMyStrategyDesignerTabs(props) returning array of DesignerTab
3. In edit page switch case: additionalTabs: getMyStrategyDesignerTabs(designerProps)

export function getMyStrategyDesignerTabs(props: MyDesignerProps): DesignerTab[] {
  return [
    { key: 'custom', label: 'Custom Tab', children: <MyCustomTab {...props} /> },
  ];
}

🔄 Universal System Designer

For systems using the universal strategy, the system designer generates form inputs dynamically from the system type's variables_schema.

Variables Schema

Defines what configuration variables each system must provide:

• Type – string, integer, number, boolean
• Required flag – Systems must provide
• Optional fallback – Default if not specified
• Description – Human-readable help

Common variables: base_url, users_per_page, tenant, connector-specific params.

VariablesValuesEditor

The VariablesValuesEditor component generates form controls based on variable type:

Dynamic Form Generation

Renders a table layout with columns: Variable (name, required indicator, description tooltip), Type (colored tag), Value (type-appropriate input), Default (fallback value). DC validates system.config.variables at load time using a Pydantic schema generated from variables_schema.

# variables_schema example
variables_schema:
  base_url:
    type: string
    required: true
    description: "API base URL"
  users_per_page:
    type: integer
    fallback: 100
  tenant:
    type: string
    required: true

VariableDefinition Structure

Each variable in variables_schema can have:

• type – string | integer | number | boolean (required)
• required – boolean, default false
• fallback – default value when not specified
• description – help text shown in UI with InfoCircle icon

VariablesValuesEditor Table Layout

The editor renders a table with columns: Variable (name + required asterisk + description tooltip), Type (colored tag: blue=string, green=integer, orange=number, purple=boolean), Value (type-appropriate input control), Default (fallback value or "-" if none). Empty schema shows "No variables defined in system type" message.

Validation Flow

At config load time, DC's universal strategy builds a Pydantic model from variables_schema using create_model(). System config's variables dict is validated against this model. Missing required variables or wrong types produce validation errors shown in Used In / Validation tabs.

🏷️ Entity Badges

A unified EntityBadge component system for consistent visual representation of the 6 core entity types across dc-ui.

Entity Type Configuration

Variants

• badge (default) – Light background (25% opacity) + 1px colored border
• tag – Solid colored background + white text (Ant Tag style)
• icon-only – Just the colored icon in a small box
• text – Icon + name, no background/border

Sizes

• small – 11px font, 1px 6px padding
• default – 12px font, 2px 8px padding
• large – 14px font, 4px 10px padding

Convenience Components

<SystemBadge name="prod-sap" />
<SystemTypeBadge name="sapias_specific" />
<StrategyBadge name="SapiasStrategy" />
<ConnectorBadge name="SCIM" />
<CredentialBadge name="sap-cred" />
<SchemaBadge name="account_mapping" />

renderEntityOption Helper

For dropdowns showing mixed entity types: renderEntityOption(type: EntityType, name: string): React.ReactNode renders a styled option with the appropriate badge. Used in Select components when choosing systems, system types, strategies, etc.

Navigation on Click

By default, EntityBadge is clickable and navigates to the edit page for that entity (e.g., /dc-ui/config/systems/prod-sap). Set clickable={false} for read-only display. Custom onClick overrides default navigation.

Short Names for Strategies/Connectors

Fully-qualified names like builtin.strategies.universal.UniversalStrategy are shortened for display (e.g., "UniversalStrategy"). Tooltip shows full name when truncated. EntityBadge handles this automatically.

CSS Overrides

EntityBadge injects CSS to override global card styles that force text color. Uses !important and specific selectors to ensure badge colors display correctly in list pages and dropdowns. Class names (eb-system, eb-systemType, etc.) allow theme customization.

❤️ Health Checks

System designers include health check testing that executes checks defined by the strategy. DC resolves credentials, calls the connector with health check params, and displays raw connector responses with timing and any errors.

Automatic Health Checks

• Run on configurable schedule (default 15 minutes)
• Results published to admin events
• Viewable as mini-graphs on Inventories page and Credentials list (24-hour success/failure timeline)

Manual Testing

• Test Health button on system / system-type edit pages — runs every health check the strategy declared for that system
• Test Credential button on credential edit pages — runs only the checks that use this credential pointer
• Test action on the Inventories table per-row menu — same as the system-page Test Health button, but reachable without navigating into the designer

All three open the HealthCheckResultsModal (covered in day 28), with per-result rows and per-row detail. Manual tests use the longer manual_timeout_seconds (vs the scheduler's auto_timeout_seconds) so a thorough validation isn't cut short.

Dual Timeout Configuration

Mini-Graphs

24-hour timeline per system or credential showing success (green) and failure (red) bars. Displayed on Inventories page and Credentials list.

Health Check Configuration (System Type)

In universal strategy system type designer, Settings tab includes health_check section:

• params – Connector-specific parameters (e.g., GET /Users for SCIM, empty search for LDAP)
• auto_timeout_seconds – Used by scheduler; keep low (e.g., 10–15) so checks don't block
• manual_timeout_seconds – Used by UI test button; can be higher (e.g., 30–60) for slow systems

Strategy Health Check Definition

Strategies implement get_health_check_details() returning list of HealthCheckDetails. Each includes connector_name, credential_pointer, params, auto_timeout_seconds, manual_timeout_seconds. A system may have multiple health checks (e.g., one per connector). Testing a credential pointer runs only checks that use that credential.

Health Check Execution Flow

Health checks can be triggered automatically by the scheduler or manually from the UI. The flow differs slightly between automatic and manual modes:

🔗 Used In Tabs

Configuration entities (system types, strategies, schemas, credentials) include Used In tabs showing which systems or other config items reference them.

Where Used In Appears

Validation Errors in Systems

System designers show Used In with error_source classification:

• SYSTEM – Errors fixable in this system config
• SYSTEM_TYPE – Errors require fixing referenced system type
• STRATEGY – Errors require fixing referenced strategy

🐛 Debugging Features & Workflows

The Test button in system, system type, and schema mapping designers runs a DETACHED_READONLY_INVENTORY with unsaved configuration changes, providing comprehensive debugging capabilities.

Complete Debugging Workflow

The following diagram shows the complete developer workflow for testing and debugging configuration changes:

Debug Event Streaming

• Real-time SSE – Events stream as inventory executes; no polling required
• No persistence – DETACHED_READONLY mode doesn't write to production database
• Unsaved changes – Test button uses current form state, not saved config
• use_db_entities toggle – Optionally reference existing entities for relation resolution

9 Debug Tabs

Universal Strategy Debug Badges

During streaming, debug badges appear throughout the designer showing evaluation counts at each expression field:

• Green badge – Number of times this expression was evaluated
• Click badge – Opens drawer showing all evaluations with input context and output
• Visual trail – Quickly identify which expressions ran and which didn't
• Zero badge – Expression never evaluated; indicates unreachable code path

Schema Mapping Debugging

Schema edit pages include a Debugging tab with specialized features:

1. System selector – Choose any system using this schema mapping
2. Run test – Triggers DETACHED_READONLY inventory for that system
3. Filtered connector calls – Shows only calls that produced entities for this schema
4. Schema debug drawer – Opens when you click a connector call row
5. Transformed data grid – Dynamic columns from transformed_data fields
6. Cell inspection – Click any cell to see the expression that generated that value
7. Expression re-evaluation – Edit expression and re-run with captured context
8. Reliability checks – Verifies expression results match between real run and debug re-evaluation

Expression Debug Features

The expression debugging system provides deep inspection capabilities:

• Captured context – Full ProduceItemContext from original evaluation
• Serialization safety – Tests if expressions work with JSON-deserialized context (important for API-driven re-evaluation)
• Side-by-side comparison – Original result vs re-evaluated result
• Reliability score – Percentage of expressions that match on re-evaluation
• Iteration over items – For list expressions, shows result for each item in source array

Download Debug Events

After a test run, download all captured events for offline analysis:

🔄 Common User Workflows

This section demonstrates common real-world workflows that combine multiple DC-UI features to accomplish typical administrative and development tasks.

Workflow 1: Triggering Inventory with Different Modes

Production Run (WRITABLE)

1. Navigate to Inventories page
2. Find target system
3. Click Trigger button
4. Select mode: New Writable
5. Choose inventory_input source:
   • From previous run – For delta sync continuation
   • Empty – For full sync
   • Custom JSON – For specific test scenarios
6. Click Start
7. Monitor real-time status (updates on the manifest's polling interval — typically every couple of seconds)
8. View state diagram for running inventory
9. After completion, check entity counts and mini-graph

Resume Interrupted Inventory (EXISTING_READONLY)

1. Navigate to Inventories page
2. Find system with incomplete inventory
3. Click Trigger → Existing Readonly
4. Review state to understand where it stopped
5. Navigate to Past Inventory Runs for detailed analysis

Test Configuration (DETACHED_READONLY)

1. Navigate to system designer
2. Make configuration changes (don't save yet)
3. Click Test button in Debugging tab
4. Toggle use_db_entities if relation resolution needed
5. Monitor debug event stream in real-time
6. Review 9 debug tabs for issues
7. If successful, save changes; if not, iterate

Workflow 2: Testing Configuration Changes Before Production

1. Open system designer – Navigate to /dc-ui/config/systems/:name
2. Navigate to Settings tab – Modify connector_timeout_ms or schedule
3. Switch to Debugging tab – Don't save changes yet
4. Click Test button – Runs DETACHED_READONLY with unsaved config
5. Monitor Overview tab – Check success/failure statistics
6. Review Connector Calls tab – Verify timeout behavior
7. Check Changes tab – Confirm entity creation
8. Download debug events – Save JSON for comparison
9. If successful: Save configuration and trigger production run
10. If failed: Review Logs tab, adjust config, test again

Workflow 3: Investigating Failed Inventories

1. Identify failure – Inventories page shows status or mini-graph red bars
2. Navigate to Past Inventory Runs – /dc-ui/analytics
3. Filter by system – Use system filter dropdown
4. Click failed run – Opens detailed view with 12 tabs
5. Check Overview tab – See failure summary and timing
6. Review Logs tab – Find error messages and stack traces
7. Examine Steps tab – Identify which step failed
8. Inspect Connector Calls tab – Check API responses and status codes
9. Diagnose root cause:
   • Timeout? Check connector_timeout_ms in Settings
   • Auth failure? Check Admin Events → Failed Health Checks, test credential
   • API error? Review raw connector response
   • Schema mapping issue? Navigate to schema debugging
10. Fix and retest – Update config, use Test button before production retry

Workflow 4: Debugging Schema Mapping Issues

1. Identify problem – Entities missing fields or wrong values
2. Navigate to Schema edit page – /dc-ui/config/schemas/:name
3. Open Debugging tab
4. Select system using this schema – From dropdown
5. Click Run – Starts test inventory
6. Wait for connector calls – Grid populates with schema-filtered calls
7. Click connector call row – Opens schema debug drawer
8. View transformed data grid – Dynamic columns from transformed_data
9. Click problematic cell – Loads expression for that field
10. Inspect captured context – See input data (raw response, variables, etc.)
11. Edit expression – Modify Jinja2 template
12. Re-evaluate – Test new expression with captured context
13. Check reliability – Verify result matches expected value
14. Iterate until correct – Adjust expression as needed
15. Apply to schema – Update Mapping tab with corrected expression
16. Save and retest – Verify fix with full test run

Workflow 5: Managing Credentials Rotation

1. Navigate to Credentials list – /dc-ui/config/credentials
2. Find credential to rotate – Search by name or filter
3. Check Used In tab – See which systems use this credential
4. Note mini-graph – 24-hour health check timeline
5. Update credential in OpenBao/Vault – External to DC-UI
6. Test credential – Click test button in credential designer
7. Review health check result – Verify new credential works
8. Check affected systems:
   • Navigate to Admin Events → Failed Health Checks preset
   • Filter by recent timeframe
   • Verify no new failures after rotation
9. Trigger test inventories – For critical systems using rotated credential
10. Monitor results – Past Inventory Runs for each system

Workflow 6: Monitoring System Health

1. Primary view: Inventories page – /dc-ui/systems
   • Mini-graphs show 24-hour health timeline per system
   • Green bars = successful checks, red bars = failures
2. Drill down for failures:
   • Navigate to Admin Events → /dc-ui/analytics/admin-events
   • Apply preset: Failed Health Checks
   • Review error details and timestamps
3. Test specific credential:
   • Navigate to credential edit page
   • Click Test button
   • Review raw connector response and timing
4. Adjust health check config if needed:
   • Navigate to system type designer → Settings tab
   • Update health_check section (params, timeouts)
   • Test changes before saving
5. Monitor trends:
   • Past Inventory Runs with date range filters
   • Cross-Inventory View for multi-system comparison

Workflow 7: Creating New System from Scratch

1. Choose connector and strategy – Review available connectors and strategies
2. Create or reuse system type:
   • If Universal Strategy: Create system type → Configure connector calls, variables, health check
   • If specific strategy: Select existing system type
3. Create credential:
   • Navigate to Credentials → Add
   • Fill credential schema (schema determines required fields)
   • Test credential via test button
4. Create system:
   • Navigate to Systems → Add
   • Select system type
   • Set credential pointer
   • Configure system-specific settings (host, overrides)
5. Test in debug mode:
   • Open system designer → Debugging tab
   • Click Test (DETACHED_READONLY)
   • Review all debug tabs for issues
6. Iterate on configuration:
   • Adjust connector params, schema mappings, variables
   • Test again until successful
7. Enable and schedule:
   • Save configuration
   • Enable system on Inventories page
   • Set schedule in system type Settings tab
   • Trigger first production run (WRITABLE)
8. Verify results:
   • Check entity counts on Inventories page
   • Navigate to Entities Explorer for data inspection
   • Review Past Inventory Runs for detailed execution

📚 Quick Reference Tables

DC-UI Route Structure

Universal Strategy System Type Tabs

Specific Strategy System Type Tabs

Universal System Tabs

Specific Strategy System Tabs

Admin Event Categories

Past Run Analytics Tab Summary

Trigger Mode Comparison

🔄 Universal Strategy Connector Calls

The Universal strategy defines its collection workflow entirely in YAML. The system-type designer's Connector Calls tab is a UI on top of this YAML. Understanding the schema is essential for configuration. Full reference: system-admins/03-universal-strategy-system-type.md.

connector_calls structure

Named connector call configurations. Each call specifies:

schema_mappings:
  eid: scim_user_mapping_for_eid
  kafka-feed: scim_user_mapping_for_kafka

produces configuration

Each produces rule specifies its action via a discriminator field whose value is the entity type — not via an action: key. The four discriminator fields:

• confirm: <entity_type> — upsert (calls dispatch_entity_confirmed_exact)
• delete: <entity_type> — soft-delete (calls dispatch_entity_deleted)
• purge_unseen: <entity_type> — purge entities not confirmed this inventory
• purge_by_relation: <entity_type> — unconditional purge scoped to a relation key

(The other four dispatch methods exist on the engine but are not exposed by the universal-strategy YAML — see day 26.)

Plus optional fields:

• timing — per_item (default; runs once per transformed item) or per_call (runs once after the call)
• mode — full, delta, or omit for "always"
• key1, relation1_key1, relation2_key1, relation3_key1 — entity-key expressions
• list — expression for sub-iteration (one entity per array element, available as child in expressions)
• transformed_data — override; defaults to pair.transformed (the multi-store dict) for per_item

next_connector_calls

• call — call name to dispatch to (the field is literally named call; extra="forbid" is set on the Pydantic model so other names will fail validation)
• timing — per_call (once after current call) or per_item (once per transformed item)
• when — boolean expression; skips dispatch if false (used for pagination "more pages?" check)
• mode — full / delta / omit for always
• list — expression for iteration (dispatch one call per array element)
• input — YAML dict of variables passed to the dispatched call

Expression contexts

# raw side — flat dict access:
key1: !expr pair.raw['id']

# transformed side — per-store, per-field:
key1: !expr pair.transformed['eid']['SystemIdentifier']

Flow Tab

The Flow tab uses ReactFlow to render a diagram of connector-call relationships. Entry step dispatches to on_start calls; arrows show next_connector_calls. Parses YAML from the system-type config; useful for visualizing complex workflows.

Example: SCIM user pagination

connector_calls:
  list_users:
    on_start: { mode: full }
    params:
      method: GET
      endpoint: "/Users"
      query_params:
        count: !expr system_var['users_per_page']
        startIndex: !expr input.get('startIndex', 0)
    schema_mappings:
      eid: scim_user_mapping
    produces:
      - confirm: User
        timing: per_item
        key1: !expr pair.raw['id']
    next_connector_calls:
      - call: list_users
        timing: per_call
        when: !expr response.correlations.raw_connector_response['totalResults'] > input.get('startIndex', 0) + len(response.correlations.raw_connector_response.get('Resources', []))
        input:
          startIndex: !expr input.get('startIndex', 0) + len(response.correlations.raw_connector_response.get('Resources', []))
    success_check: !expr response.status.value == 'success'

This defines pagination: list_users runs on full-sync start; produces User entities; when more pages exist, dispatches to itself with updated startIndex. Note the use of response.correlations.raw_connector_response to access the raw response in per_call contexts.

Example: Delta sync with since

For delta sync, since is available in params (an ISO timestamp). Many SCIM/LDAP APIs support filtering by meta.lastModified. Example: filter: "meta.lastModified gt {{ since.isoformat() }}" in params.

Before Finish (purge_unseen)

The finish.produces block typically includes purge_unseen to remove entities not seen during this sync. Mode filter mode: full skips during delta (delta doesn't list all entities). The internal _purge step checks step statistics first and cancels purge if any prior step ended with EXPECTED_FAILURE or FATAL_FAILURE — never delete entities when collection was incomplete.

📝 Schema Mapping Expression Reference

Schema mappings transform connector API responses into entity structures. Three expression types available.

Schema mapping expressions evaluate against result — the raw item being transformed. Schema-mapping expressions don't have access to pair; that's a universal-strategy per_item context. The three types are scoped to schema mapping fields.

Direct Field Paths

Bare field paths — dot notation extracts nested fields from the raw item. No result. prefix for Direct (the prefix only applies to Expression and String Template forms):

# Inside a schema mapping field, the Direct form is just the path:
id
displayName
name.givenName
emails.0.value
userName

Expression (Python)

Wrap Python in [[ ]] or use !expr YAML tag. Access result directly:

[[ result.get('email', '').lower() ]]
[[ result['id'] + '-' + result['tenantId'] ]]
[[ len(result.get('groups', [])) ]]

String Template (Jinja)

Use Jinja {{ }} for string concatenation and simple logic:

{{ result.id }}-{{ result.tenantId }}
{{ result.userName | default('unknown') }}

Schema Debug Drawer

When debugging a schema mapping, the debug drawer shows:

• Table of transformed pairs with dynamic columns from transformed_data
• Click a cell to load the field expression in an editor
• Re-evaluate with captured context (reliability check)
• Compare original vs re-evaluated result to verify expression

Connector Call Filtering

Schema debugging runs test inventory and filters connector calls by schema name. Only calls that used this schema appear—reduces noise when multiple schemas are in play.

!yaml Tag

For complex nested structures, use !yaml to embed YAML directly. Enables multi-line values or nested objects without escaping.

Reusable Schemas

Schema mappings are referenced by name. Multiple connector_calls can use the same schema. System types define schema_mapping names; schema files live in config. Used In tab on schema shows all systems/types that reference it.

jinja_guidelines.md

Reference doc for Jinja filters, tests, and built-ins. Covers default, length, join, and common patterns. DC uses Jinja 2.x compatible engine.

🔧 Troubleshooting Guide

Inventory Won't Start

• Check system is Enabled (not Disabled)
• Verify no other inventory is running for that stream (writable inventories are one-at-a-time per stream; multi-stream systems can have other streams running independently)
• Check Admin Events for recent config_change or management actions
• Verify credential_pointer resolves (test from Credentials page or system Debugging tab)

Inventory Fails with Connector Error

• Open Past Runs → select failed run → Connector Calls tab
• Expand the failing call to see request params and response
• Check health mini-graph—pattern of failures suggests credential or connectivity issue
• Test system from Debugging tab with same config
• Verify params expressions have correct system_var references

Entities Not Appearing in EmpowerID

• Check Diffs tab—are changes being produced? (add/update decisions)
• Check Diff Publications tab—are publishes succeeding or failing?
• Verify doc_type_ids mapping in diff_destinations.empowerid
• Verify resource_system_id is set at system level
• Check storage config—entity type must not be in disable_changes

Validation Errors with error_source

• SYSTEM – Fix in system config (e.g., missing required variable, invalid credential_pointer)
• SYSTEM_TYPE – Fix in system type (e.g., variables_schema, connector_calls)
• STRATEGY – Fix in strategy or connector Python (rare for config)

Health Check Failures

• Admin Events → Failed Health Checks preset
• Click event for full error message and timing
• Test credential or system manually from UI (uses manual_timeout_seconds)
• Check credential hasn't expired (OAuth tokens) or been rotated in vault

Slow Inventories

• Past Runs → Performance tab (Gantt chart) to find bottleneck steps
• Cross-Inventory → Step Performance for trends
• For universal strategies: consider increasing execution.max_parallelism in the system-type YAML. For custom strategies: review the throttle-group configuration (see day 26)
• Check pagination size—too small = many API calls; too large = memory/timeout

📡 Monitoring & Circuit Breaker

Configuration entities include monitoring and circuit_breaker settings. These appear in Connection or General tabs depending on designer.

Monitoring Configuration

Controls which events and fields are excluded from monitoring destinations (Kafka, local files):

• Exclude event types – e.g., template_evaluation to reduce volume
• Exclude fields – Hide sensitive data in events
• Per-destination rules – Different rules for Kafka vs files
• Contextual filters – decision_unchanged for change events, template_use_case_regex for template evaluations

Circuit Breaker

Protects external calls by opening after repeated failures:

• threshold – Number of failures within window to open
• window_seconds – Time window for counting failures
• test_timeout – How long before testing recovery (half-open state)

Only transient errors (429, connectivity, timeouts) trigger. Client errors (400) and server errors (500+) do not—they indicate request or remote issues.

Override Hierarchy

System type defines defaults; system can override. Monitoring and circuit_breaker follow this pattern. YAML tab shows merged result.

📤 Diff Destinations & Storage

Configurable at system type (defaults) and system (overrides) level. Define where entity changes publish and what gets persisted.

Stores and Destinations

Strategies declare stores (named state slots) via get_destination_config(), each with lists of routings to one or more destinations. A store can route to zero, one, or many destinations. The Stores tab in the system designer is the UI for this. Day 26 covers the underlying model.

DC-level overrides

DC's config.yaml can globally modify destination routing — these are operational toggles, not per-route fields:

• disabled — destination removed from store routings
• mirror_to_file — destination stays active, File destination added alongside
• divert_to_file — destination removed, File destination added instead
• mirror_all_stores_to_file — File destination added to every store

Output modes

Each Kafka or File route specifies output_mode: per_entity (one diff per entity with full transformed data) or per_attribute (one diff per changed attribute, with tombstones for removed attributes). EmpowerID is always per_entity. Switching output modes is safe at any time — store hashes are unaffected. The values are lowercase in YAML; the Python OutputMode enum class names are uppercase but the wire format is what you write.

Stores tab — system-type vs system level

The Stores tab uses the same StoresEditor component at both levels — it's the visible scope, not the editor, that changes:

• System type level — StoresEditor renders directly. Each store appears as a card with its entity types and three destination toggles: EmpowerID (configure doc_type_ids mapping entity_type → EmpowerID doc type, plus resource_system_id), Kafka (configure output_mode, optional topic, message formatters), and File (configure output_mode, formatters). Toggling a destination off removes that routing from the store.
• System level — the tab adds a three-mode inheritance radio at the top:
  • Inherit — the system uses the system type's stores verbatim. No editor renders.
  • Lightweight Override — narrow override applied across all inherited stores: edit only resource_system_id (for EmpowerID) and Kafka topic. The full store / destination shape stays inherited; this is the typical per-system override.
  • Full Replacement — drop into the same StoresEditor as the system-type tab and replace each store entirely. Use sparingly — it shadows the system type's defaults completely.

Storage tab — Override toggle

The Storage tab follows the same inherit-by-default pattern but with a simpler shape:

• System type level — StorageConfigEditor is required (allowUnset=false); a system type must declare storage.
• System level — an Override toggle appears at the top of the tab; off (default) inherits from the system type, on reveals the same StorageConfigEditor with allowUnset=true so the system can override storage for itself.

The Override-toggle pattern recurs across system-level designer tabs (e.g., monitoring, circuit breaker) — the system type provides defaults, the system overrides individual sections only when needed.

Storage Configuration

• disable_changes – Entity types to skip storing changes for (saves space; no history)
• enable_entities_transformed_data – Entity types to store full data in entities table (needed for Entity Explorer and reference queries)
• enable_changes_transformed_data – Entity types to store full data in changes table (debugging only; significantly increases DB size)

Must list all entity types produced in connector_calls and finish. Missing types cause validation errors.

Storage Best Practices

• Enable entities_transformed_data for entity types you query in Entity Explorer
• Enable changes_transformed_data only temporarily for debugging—disable after
• Use disable_changes for high-volume entity types where you only need current state

✅ DC-UI Best Practices

Configuration

• Use system types for shared defaults; override only instance-specific values in systems
• Define variables_schema in system type before creating systems—enables validation
• Use Used In tabs before deleting config—ensure nothing references it
• Test with Detached Readonly before deploying config changes to production

Operational

• Monitor health mini-graphs regularly—address patterns of failure early
• Use Admin Events preset tiles for common investigations
• Bookmark shareable URLs for frequent Entity Explorer views
• Export Past Run data when escalating to support—preserves context

Debugging

• Start with Overview tab, drill to specific tabs based on symptom
• Use schema debug drawer for expression issues—re-evaluate with real data
• Download debug events for offline analysis or test fixture creation
• Leverage debug badges in universal designer during streaming to trace evaluation

📖 Glossary

📌 Step Outcomes Reference

Steps finish with one of six outcomes. Visible in the Steps tab and Overview statistics. The central rule: only SUCCESS and EXPECTED_FAILURE persist a step's changes and dispatched-steps. Day 26 covers persistence rules and the dispatch_step_on_fatal_failure exception in detail.

The internal _purge step (in universal strategy) checks for EXPECTED_FAILURE or FATAL_FAILURE in prior steps. If any, purge is cancelled (entities not deleted) and the step finishes with EXPECTED_FAILURE — never delete entities when collection was incomplete.

📋 DataTable & Common Components

DC-UI uses consistent patterns for tables and lists across pages.

DataTable Features

• Column visibility – Show/hide columns via chooser; persisted in localStorage by table key
• Sorting – Server-side where applicable (Entities, Analytics); client-side for small datasets
• Pagination – Page size options (10, 20, 50, 100); total count when available
• Loading state – Spin indicator during fetch
• Empty state – Custom message when no data

DetailsModal

Reusable modal for viewing full record. Tabs: Overview, Details, Changes (for config diff). Theme-aware. Width 1200px for config diffs.

CredentialPointerInput

Input with credential picker. Lists existing credentials from vault. Selecting one fills URI. Supports create flow.

ErrorBoundary

Wraps routes and sections. Catches React errors, shows fallback with retry. Prevents white screen on component failure.

🛤️ Common DC-UI Workflows (Quick Reference)

Adding a New System (Universal Strategy)

Two paths get you here. The OnboardingWizard at /dc-ui/onboarding walks first-time users through the full sequence (System Type → Connection → Entity Mapping → Content Packs → First Inventory → First Scan); the manual flow below builds the same artifacts tab-by-tab. Either way produces an identical system.

1. Config → System Types → select/create system type with universal strategy, configure variables_schema, connector_calls, finish
2. Config → Systems → Create → select the system type
3. Connection tab: set credential_pointer (create credential in Config → Credentials first)
4. Variables tab: fill values for each variable in variables_schema
5. Stores tab: choose Inherit (use the system type's stores) or Lightweight Override (set resource_system_id for EmpowerID, or Kafka topic) or Full Replacement
6. Storage tab: leave the Override toggle off to inherit storage from the system type, or flip it on to set per-system disable_changes / enable_entities_transformed_data / enable_changes_transformed_data
7. Debugging tab: Test to verify config works
8. Save. System appears on Inventories page.

Investigating a Failed Run

1. Inventories or Past Runs → find failed system/run
2. Click run → Overview tab: note which steps failed
3. Steps tab: find failure point, note step name and outcome
4. Logs tab: filter ERROR, read error message
5. Connector Calls tab: expand failing call, inspect request/response
6. If credential: Admin Events → Failed Health Checks, test credential
7. Fix config, test with Detached Readonly before re-running

Debugging Schema Mapping

1. Config → Schemas → select schema
2. Debugging tab: select a system that uses this schema
3. Run test; events stream in panel
4. Connector Calls tab: filter by schema name
5. Open schema debug drawer: see transformed data grid
6. Click cell with wrong value → loads expression → edit → re-evaluate
7. Compare original vs re-evaluated to verify fix

Tracking Config Change

1. Admin Events → Config Changes preset (or filter category=config_change)
2. Find event for system/type/schema you care about
3. Click eye icon → Details modal → Changes tab
4. Side-by-side Before/After YAML shows exact diff
5. URL params preserved for sharing

Exporting Entity Data

1. Entities Explorer → select system and entity type
2. Apply filters if needed (date, keys, field values)
3. Sort by desired column
4. Export button → CSV or JSON
5. Copy URL to share filtered view

Rotating Credentials

1. Update credential in vault (OpenBao/Vault)
2. Config → Credentials → locate credential (or systems using it)
3. Test credential from list page or system Debugging tab
4. No system config change needed—credential_pointer stays the same. DC resolves the pointer on every connector call, so the rotation takes effect on the next connector call, not the next inventory.
5. Next inventory run uses new credentials

⏰ Schedule & Execution Configuration

Schedule (universal-strategy schedule block)

Execution (universal-strategy execution.max_parallelism)

The universal strategy exposes max_parallelism — concurrent steps allowed across one inventory (default 4). The engine internally translates this into a single throttle group with that limit. Custom strategies don't use this field — they declare per-group max_parallelism via get_throttle_group_config(), which is more flexible (different groups for different external resources). See day 26's Throttle Groups section.

Full vs Delta Sync

Entry step uses schedule to decide: if elapsed time since last full sync > full_sync_hours → full sync. Otherwise delta. Full sync runs all on_start calls with mode: full or no mode. Delta runs mode: delta or no mode. Strategies use the mode filter in produces and next_connector_calls to skip steps during delta.

🧩 More UI Patterns & Pages

A collection of cross-cutting UI surfaces and patterns that recur across DC-UI. They're brief on purpose — full mechanics live in system-admins/02-dc-ui.md.

Replay UI workflow

Replay is reached as one of the four mode cards in the Trigger Type modal. Picking it opens a modal with a Profile Select (loaded from getReplayProfiles — labels formatted name (store_name)), a profile description block, a Mode Switch (Readonly = monitoring only, Writable = publishes), and an optional Since ISO datetime that controls how far back to re-publish. OK is disabled until a profile is selected. See system-admins/04-replay.md for profile shape and the engine-side semantics.

Commands UI

Strategies declare commands via get_commands() (returning a list of CommandDefinition). Commands modify a stream's next_inventory_input via CAS without starting an inventory, and can run during active inventories. They surface in two places in DC-UI:

• The Trigger Type modal's Commands card (when the strategy publishes commands applicable to the current scope) — sliding panel of clickable command cards
• The Edit Next Inventory Input modal — applicable commands appear as command cards below the JSON textarea

A command card carries the command's name, description, and the stream(s) it applies to (a CommandDefinition.stream_name match against the active scope). Clicking a card calls executeCommand immediately.

Trigger profiles

Strategies expose preset trigger inputs via get_trigger_profiles(stream_name, …). The trigger modal renders these as Quick Trigger Profile tiles when the stream's trigger_profiles is non-empty; clicking a tile pre-populates the Custom JSON textarea, so an operator can launch a known-good run without crafting JSON from scratch.

Output modes per destination

Each Kafka or File store routing carries its own output_mode:

• per_entity — one diff per entity with full transformed data
• per_attribute — one diff per changed attribute, with tombstones for removed attributes

EmpowerID is always per_entity. Output mode drives whether the Diffs and Diff Publications tabs render entity-shaped or attribute-shaped diffs. Switching is safe at any time — store hashes are unaffected. (Values are lowercase in YAML.)

structured_output rendering

Strategies can attach arbitrary JSON to inventory and step finishes via finish_inventory(structured_output={...}) and finish_step(structured_output={...}). The Past Run Details and Debug panels render this data through a JsonViewer on the Overview tab (inventory-level) and within Step Finish events (step-level). Use it to surface strategy-computed metrics, summaries, or links the UI shouldn't have to compute itself.

OnboardingWizard

Reachable at /dc-ui/onboarding; gated by dc-ui:config manage. A six-step guided flow for first-time system creation: System Type → Connection → Entity Mapping → Content Packs → First Inventory → First Scan. The wizard composes the same designers and modals used elsewhere in DC-UI, so a system created here is identical to one built tab-by-tab.

InventoryDiagramViewer

Reachable from any running stream's View Diagram link; URL /dc-ui/inventory-diagram/:systemName/:streamName. A full-screen page (no top nav, no sidebar) that renders a live Mermaid diagram of the running inventory's state, refreshed on the manifest's polling interval. Supports mouse-drag pan and scroll-wheel zoom (clamped to [0.1, 5]); re-renders only when the diagram source string actually changes, so identical updates don't flicker. Error states: "No diagram available for this inventory" (404), "DataCollector is offline" (timeout), "Failed to render diagram - invalid syntax" (Mermaid render failure).

Immediate-editing pattern

DC-UI's drawers and modals (e.g., ConnectorCallDrawer, ProduceEditModal, NextCallEditModal) have no internal Save button. Every change flows to the parent state on each keystroke; the drawer's Close button just dismisses the UI. Saving happens once, at the page level, when the user hits Save in the page header.

The documented exception is ExpressionEditorDrawer: it saves on close because it validates locally and supports a sandbox mode (read-only contexts where the parent state shouldn't be mutated until the user explicitly applies their changes).

Edit-source tracking for YAML / JSON editors

Designer pages track which surface produced the latest edit:

• Designer edits immediately serialize to YAML — the YAML tab updates in real time
• YAML edits are kept verbatim — whitespace and comment changes register in hasChanges, so save remains enabled even for cosmetic edits
• Parse errors in the YAML tab set the Designer tab read-only with an inline warning until the YAML is valid again

Switching Designer → YAML keeps designer state intact. Switching YAML → Designer parses the YAML and may lose comments or unknown properties.

"+ New X" button pattern

Every config listing page (Systems, System Types, Schemas, Credentials, Connectors, Strategies) has a Create button in the breadcrumb. Clicking it navigates to the entity's edit page with ?create=true; the same designer renders, but in create mode (Save becomes Create; Delete is hidden until a record exists).

"Used By" tabs

Visible on system type, schema, credential, connector, and strategy edit pages. Each lists the systems referencing the entity, with badge-styled rows. The tab carries a count badge — secondary color when zero (safe to delete), accent color when non-zero (delete will break callers). Use this before any Delete action.

🔄 Plugin Hot-Reload

Custom strategies and connectors support 5-second hot-reload. DC watches plugin files; changes detected automatically. No restart required. Builtin plugins (builtin.strategies.*, builtin.connectors.*) are read-only. Custom plugins in config/strategies or config/connectors are editable. Validation tab shows loading errors. Used In tab shows references before delete.

🗂️ DC-UI Key Components & Structure

Pages

Designers

• UniversalStrategyDesigner – System type: General content + getUniversalDesignerTabs()
• UniversalSystemDesigner – System: CredentialPointerInput + VariablesValuesEditor
• FieldglassDesigner, IdmReferenceDataVdsDesigner, LdapDeltaDesigner, SapiasSpecificDesigner, SapldapvdsSystemDesigner – the five non-Universal specific-strategy designers
• ConnectorCallsDebugPanel – Shared debug panel for event streaming

Common Components

• EntityBadge – src/components/common/EntityBadge.tsx. ENTITY_CONFIGS, variants, sizes.
• DetailsModal – Tabbed modal (Overview, Details, Changes). Diff view for config.
• DataTable – Pagination, column chooser, sorting. localStorage persistence.
• HealthCheckMiniGraph – 24-hour success/failure bars.
• CredentialPointerInput – URI input with credential picker.
• VariablesValuesEditor – Form from variables_schema. designers/system/universal/.

Routes (routes.ts)

Centralized ROUTES constants: HOME, INVENTORIES, ENTITIES, ANALYTICS_RUNS, ANALYTICS_CROSS, ANALYTICS_ADMIN_EVENTS, CONFIG_SYSTEMS, CONFIG_SYSTEM, CONFIG_SYSTEM_TYPES, CONFIG_SYSTEM_TYPE, CONFIG_STRATEGIES, CONFIG_CONNECTORS, CONFIG_SCHEMAS, CONFIG_CREDENTIALS. buildRoute() for parameterized routes. navigateTo() utility.

❓ Frequently Asked Questions

Why can't I edit a system type?

Ensure you have an appropriate dc-ui role: app_user (view), app_admin (view + manage non-strategy resources), or platform_admin (full access including strategy management). In PDP mode, your PermissionGrant determines access; in scope mode, your token's datacollector:read / datacollector:write / datacollector:debug scopes do. Day 26 covers the auth model.

Why don't I see entity types in Entities Explorer?

Entity types only appear for systems that have run at least once and produced entities. Check storage config—entity types must be in enable_entities_transformed_data (or not in disable_changes) to be queryable. Run an inventory first.

Detached Readonly vs Existing Readonly?

Detached: in-memory, no DB, optional use_db_entities. For testing config changes. Existing: resumes from DB, read-only. For inspecting interrupted runs.

What is edit source tracking?

UI tracks whether last change came from Designer form or YAML editor. Designer mode serializes immediately. YAML mode preserves formatting. Switching Designer→YAML keeps designer state. Switching YAML→Designer parses YAML and may lose comments.

How do I add a new tab to a custom strategy designer?

Create DesignerTab objects (key, label, children). Export getMyStrategyDesignerTabs(props). In edit page switch, add additionalTabs: getMyStrategyDesignerTabs(props). Tabs render between General and YAML.

Why is health check failing?

Check credential validity (test manually). Verify params in health_check match connector expectations. Increase manual_timeout_seconds if system is slow. Review Admin Events for error message.

When to use Designer vs YAML tab?

Designer: guided forms, validation, most changes. YAML: preserve comments, bulk edits, advanced config, copy-paste from examples. Never mix—choose one mode per edit session.

How does Used In help before delete?

Used In shows systems/system types that reference the item. Deleting a schema used by 5 systems will break them. Fix or remove references first. Same for credentials, system types, strategies.

✨ Summary

DC-UI provides a complete, professional interface for DataCollector: real-time inventory monitoring with health mini-graphs, comprehensive entities explorer with server-side filtering and export, 12-tab past run analytics, 8-tab cross-inventory view, admin events with preset tiles and URL sync, strategy-specific designers for systems and system types, connector-specific credential designers, schema mapping with debugging drawer, unified entity badges, dual-timeout health checks, Used In tabs for dependency awareness, and the Designer Tab Abstraction pattern. Use this guide as your technical reference for all DC-UI capabilities.