Change Data Capture
As changes are accepted to the Context Graph they are published externally via a Change Data Capture stream, as enabled via Configuration. Users of the product are not required to work with this Knowledge Graph, but may find it valuable for a variety of tasks from managing data lifecycle to auditing AI pipelines. Tranquil Data offers connectors that take this stream and make it available in services like Neo4j. This section covers the raw structure of the Change Graph structed output.
Introduction to Context#
At the core of Tranquil Data is a Knowledge Graph about data, associated entities, and the framework that makes up permissible use. Most of these details are purposefully hidden in this product edition to help streamline adoption. Becuase they're visible if you choose to work with the Change Graph, however, here is a quick introduction to the metadata model that forms.
Entity Context#
The foundation of the graph is Entity Context. This class of context is accepted via consensus and captured in a distributed ledger, with each change recoreded at a new logical timestamp called the Checkpoint. The ledger has provable integrity, and on re-start the product begins by replaying this data to ensure that context is starting in a known-correct state. In the Tranquil DataTM Trusted Flow Edition all Policy is pre-defined to support the UI components, and a Model is automatically managed as categories and resolvers are defined. With each change to the configuration, the Checkpoint is updated so all context state is known to be associated with an unambiguous version of business logic. There is a single Domain (namespace where policy evaluation happens) and Peer (logic host that represents the running server process).
Record Context#
Connected to Entity Context is Record Context. This class of context represents details about data at the record-version level, and connects records via the sessions that operated on those records. A Record might be (e.g.) a row in a SQL database, a document in a Document Databse, or a line in a CSV file. A session might be (e.g.) a SQL transaction, a batched HTTP operation, or a streamed file. Records might have identity that is mapped to the structure of an exported datastore, or may be "anonymous" as they flow through Tranquil Data from an unknown source. In either case, the context that forms references the associated peer and checkpoint, and includes user-defined properties from the record's content.
Subject Context#
The last category is Subject Context which supports an open-ended model for Subjects (e.g., people or organizations) and their Relationships, as well as Groups and their Members. This is not meant to replace a CRM, Directory, or other enterprise component. It's intended as a single place to capture the minimal knowledge needed to drive policy decision-making. In this product edition there are no APIs to directly define this structure. Intsead, there are APIs to onboard users, to define, record and modify consent, and to define purposes, third-party sharing, and contractual relationships. Intrernally, all are mapped to versioned Subjects and Groups, structured and connected to make evaluation efficient and transparent.
Transient State#
While not a part of the product Context Model in terms of history and recoverability, all transient operations like starting a new peer process or exporting a datastore are tracked and reported through the Change Graph. This makes it simple to connect the logical model that Tranquil Data defines to the physical infrastructure that you are supporting.
Change Data Graph Structure#
Changes are represented as additions to an append-only property graph. All changes, therefore, are represented as net-new nodes and/or edges added to an existing graph. These new elements may represent a new component in the graph, or an update to an existing component. Each captured change is supplied as a collection of these new graph elements.
ChangeCollection#
Changes are provided as an atomic collection, in-order, that represent the evolution of the Context Graph. Once a given collection is applied then it is safe to apply the next collection.
| Field Name | Type | Details |
|---|---|---|
| collectionType | CollectionType | The scope of the changes being reported |
| traceID | integer | The evaluation session associated with these changes (from the decision trace) |
| connectionID | integer | The connected user associated with these changes (from the audit trail) |
| checkpoint | integer | The logic moment when this change occurred |
| peerID | integer | The local peer where this change was accepted |
| addedNodes | array of ChangeNode | A (possibly absent) array of nodes representing new context state |
| addedRelations | array of ChangeRelation | A (possibly absent) array of relationships connecting existing and/or new context state |
The value of traceID will be non-zero if these changes are associated with an evaluation that was reported via Decision Trace. This identiier lets users connect changes in context to the rationale for allowing/redacting access to data.
ChangeNode#
A single Node represents either a new component in the Context Graph, like a new Record, or an update to an existing component, like a modified Subject. In the latter case, the collection will also include the appropriate ChangeRelation to explain what has changed.
| Field Name | Type | Details |
|---|---|---|
| identity | ChangeNodeIdentity | The unique version-identity of this node |
| properties | map of string to array of ChangeProperty | The (possibly empty) properties assocaited with this node |
| datastoreName | string | The datastore (if any) associated with this node |
| recordIdentity | array of string | If this node's identity is a Record type, then the associated datastore identity |
| policyName | string | If this node is govered by a specific policy, the name of that policy |
| evalID | integer | The evaluation action associated with these changes (from the decision trace) |
In the case that this node represents a new version of a previous component, the value of properties will be the absolute new set of properties for that component.
The value of recordIdentity is a name hierarchy that is meaningful for the type of store identified by datastoreName. For instance, if datastoreName identifies an exported SQL datastore, then the value of recordIdentity might include details like a schema, column name, or primary key value. The values provided here are inteneded to make it simple for an operator to map logic knowledge in the Context Graph to physical deployment across platforms.
ChangeNodeIdentity#
An Identity uniquely identifies a graph component at a specific version.
| Field Name | Type | Details |
|---|---|---|
| name | string | The unique name (within a given domain) for a component |
| domain | string | The domain where the component is uniquely identified |
| timestamp | integer | The moment when this identity was last created or updated |
| nodeType | NodeType | The type of node this identity represents |
ChangeProperty#
A single property value associated with a node in the change graph.
| Field Name | Type | Details |
|---|---|---|
| type | string | The original data type of the property value |
| string | string | The string representation of the property value |
ChangeRelation#
All nodes in the Context Graph are connected by one or more relationships. No two nodes are connected multiple times by the same kind of relationship. The direction of a relationship is meant to help guide typical flows through the graph, not to gate how the graph may be interpreted. For instance, when looking at a single Record Node you want to know the associated Session, so the direction will be from Record to Session. Once the graph forms, however, it may also be useful to query for all of the Records read within a given Session.
| Field Name | Type | Details |
|---|---|---|
| from | ChangeNodeIdentity | A node in the graph or the current change collection |
| to | ChangeNodeIdentity | A node in the graph or the currect change collection |
| relation | RelationType | A single relation to use between the two nodes |
Graph Data Identifiers#
While names and properties are open-ended and defined by end-users, much of the structure of the Context Graph forms using known types. Defined here are the standard collection, node, and relationship types used throughout a well-formed graph.
CollectionType#
Each collection provided through the Change Data Capture interface represents a set of changes for a particular scope of the graph.
| Value | Name | Details |
|---|---|---|
| 0 | Session | A collection of changes that happened within a single datastore session |
| 1 | Subject | A collection of changes to a single subject |
| 2 | Group | A collection of changes to a single group |
| 3 | Entity | A collection of entity changes, like changes to the peer or model |
| 4 | Datastore | A collection of updates due to the export or shutdown of a datastore |
| 5 | Instance | A collection of changes related to the host peer instance itself |
NodeType#
Each node added to the Change Graph represents a specific component type in context. These represent changes to the core categories of durable context, as well as changes to the transient runtime state of the service.
| Value | Name | Details |
|---|---|---|
| 0 | Session | A single session where record-level evaluation happens (Record Context) |
| 1 | Record | A single record-version associated with a known datastore (Record Context) |
| 2 | AnonymousRecord | A single record-version unassociated with any known datastore (Record Context) |
| 3 | Subject | A single subject and its properties (Subject Context) |
| 4 | Relationship | A single relationship, and its properties, associated with a single subject (Subject Context) |
| 5 | Group | A single group and its properties (Subject Context) |
| 6 | Member | A single member, and its properties, associated with a single group (Subject Context) |
| 7 | Peer | A single peer and its properties (Entity Context) |
| 8 | Policy | A single policy (Entity Context) |
| 9 | Model | A single model (Entity Context) |
| 10 | Domain | A single domain and its policy and/or model (Entity Context) |
| 11 | Datastore | A single exported datastore at a peer within a domain (Transient State) |
| 12 | PeerInstance | A single started or stopped process instance of a given peer (Transient State) |
Anonymous Records
An "anonymous" record is one that Tranquil Data is reading outside the context of an exported datastore, and therefore has no name or durable history in the context store. By default, these records are not stored or included in the Change Graph, but may prove helpful in tracking all records flowing through policy evaluation. To include this content in the Change Graph, set TRANQUIL_CHANGE_OUTPUT_ANONYMOUS_CONTEXT to true.
RelationType#
As nodes are added to the Change Graph realations explain how to interpret those nodes in-context.
| Value | Name | Details |
|---|---|---|
| 0 | Session | Relation to a session where another node was active |
| 1 | Create | Relation from a record to a session where the record was created |
| 2 | Read | Relation from a record to a session where the record was read |
| 3 | Update | Relation from a record to a session where the record was updated |
| 4 | Delete | Relation from a record to a session where the record was deleted or from an element of subject context to itself to mark deletion of that element |
| 5 | Descendant | Relation from a node to a new version of itself |
| 6 | Subject | Relation from a session to a subject referenced in that session |
| 7 | Related | Relation from a subject to one of its relationships |
| 8 | Group | Relation from a session to a group referenced in that session |
| 9 | Member | Relation from a subject to one of its members |
| 10 | Uses | Relation from a session to the datastore and domain where it was active, from a domain to the policy and model it uses, from a peer to the policy that it uses, or from a policy to any policies that it references |
| 11 | Entry | Relation from an entity node to the peer where it entered the service |
| 12 | DefinedBy | Relation from a datastore to the domain where it was exported or from a relationship to the policy that governs it |
| 13 | Retire | Relation from a peer to a new version of itself to mark that the peer was de-provisioned |
| 14 | ExportedBy | Relation from a datastore to the peer where the datastore was exported |
| 15 | InstanceOf | Relation from a peer instance to its associated peer |