DAY 24 WEEK 5: ADVANCED

NowConnect & Hybrid Connectivity

Master secure on-premises connectivity with outbound-only tunneling, identity-anchored sessions, WebSocket multiplexing, and PDP authorization for hybrid environments.

Every enterprise has on-premises systems: Active Directory, LDAP servers, databases, legacy applications. Cloud services need to reach them, but opening inbound firewall ports is a security nightmare. NowConnect solves this with an identity-anchored, outbound-only tunnel—your on-prem agent connects TO the cloud, not the other way around. No VPN sprawl, no firewall holes.

📋Learning Objectives

  • Understand the hybrid connectivity challenge and NowConnect's solution
  • Learn the outbound-only tunneling architecture
  • Understand identity-anchored sessions with JWT/JWKS
  • Configure connectors for LDAP, AD, databases, and REST APIs
  • Understand WebSocket multiplexing for multiple targets
  • Learn PDP authorization for connector access

🌐The Hybrid Connectivity Challenge

Organizations face a difficult choice: cloud benefits vs. on-premises security requirements.

Traditional Approach Problems
VPN Tunnels Complex, expensive, broad network access, hard to audit
Inbound Firewall Rules Security risk, attack surface, compliance violations
Data Replication Stale data, sync complexity, data sovereignty issues
Custom Relays Maintenance burden, no identity integration, no audit
💡 NowConnect Solution

An identity-anchored, vendor-agnostic tunnel that: (1) requires no inbound ports, (2) binds access to identity and policy, (3) works with any IdP (JWKS) and any PDP (AuthZEN-style).

🏗️NowConnect Architecture

flowchart LR subgraph CLOUD["☁️ Cloud Environment"] BFF["BFF / APIs"] CRUD["CRUDService"] IdP["IdP"] CH["Cloud Hub
/tunnel WSS
TCP listeners"] PDP["PDP"] end subgraph ONPREM["🏢 On-Premises"] NC["NowConnect
Agent"] AD["Active
Directory"] LDAP["LDAP
Server"] DB["PostgreSQL
Database"] REST["Internal
REST API"] end BFF --> CH CRUD --> CH IdP --> CH CH <-->|"WSS Tunnel
(Outbound Only)"| NC CH --> PDP NC --> AD NC --> LDAP NC --> DB NC --> REST style CH fill:#00d9ff22,stroke:#00d9ff style NC fill:#a855f722,stroke:#a855f7 style PDP fill:#f43f5e22,stroke:#f43f5e

Key Components

☁️ Cloud Hub

FastAPI service with /tunnel WebSocket endpoint, TCP listeners (389, 636, 1433...), health/metrics endpoints

🏢 Premise Agent

Lightweight process that opens outbound WSS to Cloud Hub, proxies TCP to local targets

🔑 IdP (JWKS)

Validates agent JWT tokens; any OIDC IdP works (Azure AD, Okta, Keycloak)

🛡️ PDP

Optional authorization for each OPEN(connector) request

🔗Identity-Anchored Session Flow

sequenceDiagram participant AG as Premise Agent participant CH as Cloud Hub participant IDP as IdP (JWKS) participant PDP as PDP participant TGT as Target (LDAP) participant CLI as Client Note over CH,AG: 1. Tunnel Establishment AG->>CH: WS Upgrade (Authorization: Bearer JWT) CH->>IDP: Fetch JWKS (cached) IDP-->>CH: JWKS keys CH->>CH: Validate JWT (aud, signature) CH-->>AG: 101 Switching Protocols AG->>CH: HELLO {agent_id, connectors, ver} CH->>CH: Reconcile agent_id with JWT claim CH-->>AG: HELLO_ACK Note over CLI,TGT: 2. Client Connection CLI->>CH: TCP connect to listener (port 389) CH->>PDP: Can agent open 'ldap-corp'? PDP-->>CH: Permit CH->>AG: OPEN {cid, connector: 'ldap-corp'} AG->>TGT: TCP connect host:port TGT-->>AG: Connected AG-->>CH: OPEN_ACK {cid} Note over CLI,TGT: 3. Data Relay CLI->>CH: DATA (LDAP query) CH->>AG: DATA {cid, seq, payload} AG->>TGT: Forward payload TGT-->>AG: Response AG-->>CH: DATA {cid, seq, payload} CH-->>CLI: Response

Protocol Messages

Message Direction Purpose
HELLO Agent → Hub Announce agent_id and available connectors
HELLO_ACK Hub → Agent Confirm registration
OPEN Hub → Agent Request connection to a connector
OPEN_ACK Agent → Hub Confirm connection established
DATA Both Relay TCP payload with sequence number
FIN Both Half-close signal (direction specified)

🔀WebSocket Multiplexing

A single WSS connection supports multiple concurrent channels to different targets, with full isolation between them.

flowchart TB subgraph TUNNEL["Single WSS Connection"] WSS["WebSocket"] end WSS --> CH1["Channel 1 (cid: 001)
LDAP Query"] WSS --> CH2["Channel 2 (cid: 002)
SQL Query"] WSS --> CH3["Channel 3 (cid: 003)
REST Call"] WSS --> CH4["Channel 4 (cid: 004)
LDAP Query"] CH1 --> AD["AD Server
ldap://ad.corp:389"] CH2 --> PG["PostgreSQL
pg.corp:5432"] CH3 --> API["Internal API
api.corp:8080"] CH4 --> AD style WSS fill:#a855f722,stroke:#a855f7

Benefits of Multiplexing

  • Single Firewall Egress: Only one outbound connection to manage
  • Resource Efficient: No connection per target; share one tunnel
  • Channel Isolation: Each channel has independent flow control
  • Automatic Reconnect: Agent reconnects on disconnect; channels resume

🛡️Security Controls

Control How It Works
JWT Authentication Agent presents JWT on WebSocket upgrade; Hub validates against IdP JWKS
Agent ID Reconciliation HELLO.agent_id must match JWT claim; prevents token reuse
PDP Authorization Optional check on each OPEN(connector); deny-fast on failure
Connector Scope Enforcement JWT can include allowed connectors in connectors or scp claim
TLS Transport WSS (TLS) from agent to cloud; application TLS (LDAPS) remains end-to-end
mTLS (Optional) Client certificate at ingress for enterprise hardened deployments
⚠️ Credential Flow

Credentials for on-prem targets never leave the datacenter. The NowConnect agent retrieves and uses credentials locally. Only identity tokens traverse the tunnel.

⚙️Connector Configuration

Agent Environment Variables

# Agent identity
NC_TOKEN_FILE=/run/secrets/agent-token    # JWT for authentication
NC_AGENT_ID=datacenter-west-01            # Must match JWT claim

# Cloud Hub connection
NC_HUB_URL=wss://nowconnect.self.empowernow.ai/tunnel

# Connector definitions
NC_CONNECTORS=ldap-corp,sql-erp,api-hr

# Per-connector targets
NC_TARGET_ldap_corp=ldap://ad.corp.internal:389
NC_TARGET_sql_erp=tcp://erp-db.corp.internal:1433
NC_TARGET_api_hr=tcp://hr-api.corp.internal:8080

# Optional: Corporate proxy and CA
NC_TRUST_ENV=true                         # Honor HTTP_PROXY, HTTPS_PROXY
NC_CA_BUNDLE=/etc/ssl/corp-ca.pem         # Trust corporate CA

# Health check
NC_HEALTH_HOST=0.0.0.0
NC_HEALTH_PORT=8080

Cloud Hub Configuration

# config/nowconnect.yaml
server:
  host: 0.0.0.0
  tunnel_port: 8443
  
  # TCP listeners exposed to cloud services
  listeners:
    - port: 389
      connector: ldap-corp
    - port: 636
      connector: ldaps-corp
    - port: 1433
      connector: sql-erp
    - port: 8080
      connector: api-hr

security:
  jwks_url: https://idp.self.empowernow.ai/.well-known/jwks.json
  audience: nowconnect
  require_connector_scopes: true   # JWT must include connector claims

pdp:
  url: https://pdp.self.empowernow.ai/access/v1/evaluation
  cache_ttl_sec: 60
  timeout_ms: 500
  fail_open: false                 # Deny if PDP unreachable

Supported Protocols

Protocol Typical Port Use Case
LDAP / LDAPS 389 / 636 Active Directory, OpenLDAP
SQL Server 1433 Database queries
PostgreSQL 5432 Database queries
MySQL 3306 Database queries
Oracle 1521 Database queries
HTTP/HTTPS 80 / 443 Internal REST APIs
SSH 22 Server management

📡Wire Protocol

NowConnect uses JSON frames over WebSocket text messages (binary header in V1.1).

Protocol Frames

// Agent → Hub: Initial handshake
{ "t": "HELLO", "agent_id": "premise-01", "connectors": ["addomain", "devdomain1"], "ver": "1.5" }

// Hub → Agent: Acknowledge registration
{ "t": "HELLO_ACK" }

// Hub → Agent: Open connection to a target
{ "t": "OPEN", "cid": "f47ac10b-58cc-4372-a567-0e02b2c3d479", "connector": "addomain" }

// Bidirectional: Data relay (base64-encoded)
{ "t": "DATA", "cid": "f47ac10b-...", "seq": 123, "b64": "TERBUCBxdWVyeS4uLg==" }

// Bidirectional: Half-close (c2a = client→agent, a2c = agent→client)
{ "t": "FIN", "cid": "f47ac10b-...", "dir": "c2a" }

// Error: Connection failed or broken
{ "t": "RST", "cid": "f47ac10b-...", "err": "connect_failed" }

Frame Fields

Field Description
t Frame type (HELLO, HELLO_ACK, OPEN, DATA, FIN, RST)
cid Channel ID (UUID) - identifies a logical connection
seq Sequence number for DATA frames (monotonic per cid)
b64 Base64-encoded payload bytes
dir FIN direction: c2a (client→agent) or a2c (agent→client)
err Error code: connect_failed, ws_broken, queue_overflow

Connection Lifecycle

stateDiagram-v2 [*] --> Opening: OPEN received Opening --> Active: Target connected Opening --> Closed: connect_failed Active --> HalfClosed_C2A: FIN c2a Active --> HalfClosed_A2C: FIN a2c Active --> Closed: RST HalfClosed_C2A --> Closed: FIN a2c HalfClosed_A2C --> Closed: FIN c2a Closed --> [*] note right of Active: DATA frames flow note right of Closed: Channel resources freed

🔄High Availability

For production deployments, NowConnect supports high availability with multiple Cloud Hub instances.

HA Components

Load Balancer

Distributes tunnel connections across Hub instances; sticky sessions per agent

Redis

Shares agent registry across Hubs; enables cross-hub routing

mTLS Mesh

Hub-to-Hub communication for cross-routing when agent connects to different Hub

Health Probes

/healthz and /readyz for Kubernetes liveness and readiness

Reconnection Behavior

  • Agent-side: Automatic reconnect with exponential backoff (2s, 4s, 8s...)
  • Channel resume: Open channels are re-established after reconnect
  • Graceful drain: Hub waits for active channels before shutdown
💡 DR/RTO Guidance

Target RTO ≤ 15 minutes for Cloud Hub recovery. Hubs are stateless (except channel state); agent reconnects establish fresh channels. Redis persistence optional for registry durability.

📊Observability

Key Metrics (Prometheus)

Metric Description
nowconnect_connections_active Current active tunnel connections
nowconnect_channels_active Current open channels across all tunnels
nowconnect_bytes_total Total bytes transferred (labels: direction, connector)
nowconnect_pdp_decisions_total PDP authorization decisions (labels: decision, connector)
nowconnect_errors_total Error count (labels: type: connect_failed, ws_broken, etc.)

Structured Logging

# Cloud Hub JSON log format
{
  "ts": "2026-01-21T14:30:00Z",
  "level": "info",
  "event": "channel_opened",
  "agent_id": "premise-01",
  "connector": "ldap-corp",
  "cid": "f47ac10b-...",
  "pdp_decision": "permit",
  "duration_ms": 45
}
⚠️ No Payload Logging

DATA frame payloads are never logged. Only metadata (agent_id, connector, cid, byte counts) appears in logs to protect sensitive on-prem data.

🔍Troubleshooting

Agent Can't Connect to Hub

Symptom: Agent logs show connection refused or timeout

Debug:

# Test outbound connectivity
curl -v https://nowconnect.self.empowernow.ai/healthz

# Check if proxy is needed
export HTTPS_PROXY=http://proxy.corp:8080
export NC_TRUST_ENV=true

# Verify CA bundle
openssl s_client -connect nowconnect.self.empowernow.ai:443 \
  -CAfile /etc/ssl/corp-ca.pem

JWT Validation Failed

Symptom: Hub rejects WebSocket upgrade with 401

Causes:

  • Token expired — check exp claim
  • Wrong audience — token aud must match Hub's security.audience
  • JWKS unreachable — Hub can't fetch IdP's JWKS

OPEN Rejected by PDP

Symptom: Connection refused with 403, logs show "PDP denied"

Debug:

# Check PDP evaluation
curl -X POST https://pdp.self.empowernow.ai/access/v1/evaluation \
  -H "Authorization: Bearer $AGENT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "subject": { "type": "agent", "id": "datacenter-west-01" },
    "action": { "name": "connect" },
    "resource": { "type": "connector", "id": "ldap-corp" },
    "context": { "reason_admin": true }
  }'

Target Unreachable

Symptom: OPEN succeeds but DATA fails, agent logs show connection refused

Fix: Verify target hostname:port is reachable from the agent's network. Check DNS resolution and firewall rules within the on-prem network.

Knowledge Check

1 Why is NowConnect "outbound-only"?

The Premise Agent initiates the WebSocket connection TO the Cloud Hub. No inbound firewall rules are needed on the corporate network. This eliminates the attack surface of open inbound ports while still allowing cloud services to reach on-prem targets.

2 What is "agent ID reconciliation"?

After the agent authenticates with its JWT, it sends a HELLO message containing its agent_id. The Cloud Hub verifies this matches the agent_id claim in the JWT. This prevents an attacker from using a stolen token with a different agent.

3 How does WebSocket multiplexing work?

A single WebSocket connection carries multiple logical channels, each identified by a cid (channel ID). Each channel is an independent TCP stream to a potentially different target. The Hub and Agent use DATA messages with sequence numbers to relay bytes for each channel independently.

4 When is PDP called during the connection flow?

PDP is called when the Cloud Hub receives an OPEN request (before forwarding to the agent). The Hub asks: "Can this agent_id connect to this connector_id?" Decisions are cached (configurable TTL). If PDP is unreachable and fail_open: false, the OPEN is denied.

5 Do on-prem credentials ever leave the datacenter?

No. The NowConnect agent retrieves and uses credentials locally when connecting to targets. Only identity tokens (JWTs) traverse the tunnel. Application-layer credentials (LDAP bind, database passwords) stay within the on-prem network.

📝Day 24 Checkpoint

  • Understand the hybrid connectivity challenge
  • Know NowConnect's outbound-only architecture
  • Understand identity-anchored sessions (JWT/JWKS)
  • Know the session flow (HELLO, OPEN, DATA, FIN)
  • Understand WebSocket multiplexing
  • Can configure connectors for various protocols
  • Understand PDP authorization for connector access