ogrl-integration is the Python SDK for writing OGRL integrations as defined in RFC 0005. It implements the JSON-RPC protocol loop, a standard-library HTTP client with retries and error mapping, query-source executors over OpenAPI and AWS Smithy descriptions, an offline test kit with cassettes, and the ogrl-integration CLI that scaffolds, runs, conformance-tests, and packs an integration package. The runtime has no dependencies, so a packed integration runs from memory on a read-only filesystem.
#Concepts
An integration is a process the OGRL host starts once per operation. The host talks to it over stdin and stdout with JSON-RPC 2.0, one message per line (RFC 0005 section 7). Stdout carries protocol messages only; logs and tracebacks go to stderr.
An integration provides three kinds of operation:
| Operation | SDK hook | Sees credentials | Returns |
|---|---|---|---|
connection.validate | @app.validate | Yes | Vendor identity, granted scopes, warnings |
collect | @app.resource(name, version) | Yes | Items, streamed to the host; the host builds the snapshot |
check | @app.check(name, version) | No | pass, fail, or error with a summary and subjects |
A resource is a named, major-versioned stream of vendor records, described by a JSON Schema in the package. The host validates each item against that schema, strips undeclared properties, and stores the result as an RFC 0001 snapshot (section 7.6 of the object model). A resource is addressed as a source: <integration id>/<name>@<major>, such as dev.adversity.github/repositories@1. Tests read snapshots either with the rule runner (an expression over items, no integration code) or with the integration runner, which calls one of the integration's checks.
A connection is an operator's configured account at the vendor: non-secret config values plus credentials for one of the manifest's auth kinds (RFC 0005 section 5):
| Auth kind | What the process receives | SDK accessor |
|---|---|---|
fields | Each declared field as an environment variable | ctx.secret("NAME") |
oauth2 | A fresh access token in OGRL_ACCESS_TOKEN (bring-your-own OAuth app) | ctx.access_token |
aws_role | Short-lived AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN | ctx.aws_credentials |
#Install
The runtime needs Python 3.12 or later. The CLI's run, test, and pack commands read YAML and JSON Schema, so they need the cli extra (pyyaml, jsonschema):
pip install 'ogrl-integration[cli] @ git+https://github.com/adversitydotdev/ogrl-integration-sdk@v0.1.0a12'
#Project layout
ogrl-integration new produces this layout; first-party integrations such as adversitydotdev/ogrl-github follow it:
acme-directory/
manifest.yaml # id, version, runtime, config, auth, resources, checks, templates
directory_integration.py # the Integration and its operations
schemas/items.json # one JSON Schema per resource
tests/cassettes/items.json # recorded vendor HTTP
tests/test_directory_integration.py
Package code must not read files next to its modules; put data in code. pack excludes tests/, dist/, .venv/, and __pycache__/.
#Manifest
The manifest format is defined in RFC 0005 section 3. The fields the SDK and its conformance suite rely on:
| Field | Notes |
|---|---|
id | Reverse domain name (dev.acme.directory). Must equal Integration(id, ...). |
version | Semver. Must equal Integration(..., version). |
protocol | 1. |
runtime | kind: python, entrypoint: <module>:<app>.run. |
egress | Hosts the integration calls. Recorded, not yet enforced. |
config | Non-secret connection settings: name, label, optional pattern and required. Read with ctx.config. |
auth | At least one entry. fields entries list fields (name, label); their names are the environment variables ctx.secret() reads. |
resources | name, version (major), schema. Schemas must be object schemas declaring id and name. |
checks | name, version, sources (resource names), optional args. |
templates | Tests an organization can enable. A template may only read this integration's sources. |
openapi, smithy | Optional; enable query sources (below). |
#The app API
from ogrl_integration import Integration
app = Integration("dev.acme.directory", "0.1.0")
if __name__ == "__main__":
app.run()
app.run() serves requests until stdin closes. It answers initialize with {"protocol": 1, "id": ..., "version": ...} and any unknown method with an invalid_config error.
#Decorators
@app.validate:fn(ctx) -> Identity. Make the cheapest authenticated call that proves the credential works and returnctx.identity(identity, scopes=[...], warnings=[...]).@app.resource(name, version=1):fn(ctx)yields one mapping per item. Generators are the norm; items stream to the host asitemsnotifications in batches of at most 500 items or 512 KB, and are JSON-encoded withdefault=str. A single item over about 1 MB fails the collection.@app.check(name, version=1):fn(ctx, **snapshots) -> Outcome. Each source the host sends becomes one keyword argument holding that snapshot's item list; the name is the resource name with-replaced by_(branch-rulesbecomesbranch_rules).
#Context
| Member | Meaning |
|---|---|
ctx.config | The connection's config values. |
ctx.env | The process environment for validate and collect; empty for check. |
ctx.secret(name) | A credential from the environment; raises InvalidConfig when unset or empty. |
ctx.access_token | ctx.secret("OGRL_ACCESS_TOKEN"). |
ctx.aws_credentials | Dict of the three AWS_* variables. |
ctx.http | An Http client. Raises RuntimeError inside a check: checks evaluate snapshots and cannot call the vendor. |
ctx.args, ctx.params | The args and params sent with check, as strings. |
ctx.log(message) | Sends a log notification. |
ctx.progress(done, total=None) | Sends a progress notification. |
ctx.identity(...) | Builds the Identity that validate returns. |
#HTTP client
Http(transport=None, *, retries=3, timeout=30.0, sleep=time.sleep, headers=None) wraps urllib. Operations get one as ctx.http.
request(method, url, *, params=None, headers=None, json_body=None, data=None, content=None, throttled=None, error_detail=None), plusget(url, **kw)andpost(url, **kw).paramsentries set toNoneare dropped;json_bodyanddatasetContent-Type;contentsends raw bytes (for signed bodies).Accept: application/jsonand aUser-Agentare sent by default.- Retries. 429, 500, 502, 503, 504, a 403 with
X-RateLimit-Remaining: 0(GitHub), and any response yourthrottled(response)callback marks are retried up toretriestimes. The wait isRetry-After, else the reset time inX-Rate-Limit-ResetorX-RateLimit-Reset, else exponential backoff from 1 s, capped at 60 s. - Error mapping. After retries, 401 raises
AuthFailed, a rate limit raisesRateLimitedwithretry_after, 403 raisesPermissionDenied, and any other status of 400 or above raisesVendorError. The exception'sstatusholds the HTTP status; the message redacts the query string.error_detail(response)can append the vendor's error code and message. Network failures raiseVendorError. Responsehasstatus,headers,body,url,header(name)(case-insensitive),json(),text, andlinks(the RFC 8288Linkheader as{rel: url}).
paginate(url, *, params=None, headers=None, items=lambda r: r.json(), next_url=None, cursor=None, cursor_param="cursor", max_pages=10_000, error_detail=None) yields every item across pages. By default it follows Link: rel="next". Pass next_url to read the next URL from the body, or cursor to read a token that is sent back as cursor_param. items extracts the list from each page.
for user in ctx.http.paginate(
f"{base}/users",
headers={"Authorization": f"Bearer {ctx.secret('ACME_TOKEN')}"},
items=lambda r: r.json()["users"],
cursor=lambda r: r.json().get("next_cursor"),
cursor_param="after",
):
yield {"id": user["id"], "name": user["email"], "mfa": user["mfa_enabled"]}
For hand-written AWS calls, ogrl_integration.sigv4.sign(method, url, headers, body, *, access_key, secret_key, session_token, region, service) returns the headers with SigV4 Authorization, X-Amz-Date, and the session token added; send them with ctx.http.request(..., headers=signed, content=body).
#Outcomes and errors
Checks return one of:
from ogrl_integration import Subject, error, fail, pass_
pass_("42 admins use phishing-resistant MFA")
fail("2 admins lack MFA", [Subject.account("ana@acme.example", "00u1"), ...])
error("the users snapshot is empty")
Subject(kind, name, external_id=None) names a failing account or resource; Subject.account(...) and Subject.resource(...) set kind. Summaries are truncated to 2,048 characters. Returning anything other than an Outcome is an error.
Operations raise typed errors, which run() turns into JSON-RPC errors with the RFC 0005 data.kind:
| Exception | data |
|---|---|
AuthFailed(message) | {"kind": "auth_failed"} |
PermissionDenied(message, permission=None) | {"kind": "permission_denied", "permission": ...} |
RateLimited(message, retry_after=None) | {"kind": "rate_limited", "retry_after": ...} |
VendorError(message) | {"kind": "vendor_error"} |
InvalidConfig(message) | {"kind": "invalid_config"} |
| Any other exception | {"kind": "vendor_error"}, message Type: text, traceback on stderr |
All five derive from IntegrationError. Two cases end a collection early but keep what was yielded, returning complete: false with a reason: raising Incomplete(reason) from a resource, and RateLimited after at least one item. A RateLimited before any item is returned as an error so the host retries the operation. Re-raise PermissionDenied with permission= to tell the operator what to grant, as ogrl-github does for branch protection (administration:read).
#Query sources
Query sources let an organization define new sources over a vendor API without integration code. The integration bundles an API description in its manifest; the SDK validates each query again in the integration process and executes it. collect carries the definition (operation, params, items, fields, and for Smithy regions, plus optional for_each) in a query parameter. An integration with neither block answers invalid_config. When both are present, the Smithy executor is used.
#OpenAPI
Declared as in RFC 0006 section 4:
openapi:
description: openapi/acme.yaml # OpenAPI 3.x, JSON or YAML; local $refs only
base_url: "https://{domain}" # {placeholders} filled from config, which must be host names
pagination: link # link | next-url:<path> | cursor:<response path>:<param> | none
read_operations: [] # non-GET operations that only read
authorization: "Bearer {ACME_TOKEN} | Bearer {OGRL_ACCESS_TOKEN}"
authorization uses the first | alternative whose variables are all set in the environment. The executor refuses non-https base URLs (except localhost), non-read operations, and unsafe path parameters (RFC 0006 section 5), projects fields from each item ($parent.<path> reads the listed item under for_each), and stringifies id and name.
#Smithy
Declared as in RFC 0007 section 2: smithy.services.<name> with model (Smithy 2.0 JSON AST), endpoint ({region} makes it regional), optional signing_region and target_prefix, plus read_operations and deny_operations. Operations are named <service>.<Operation>. Supported protocols are awsJson1_0, awsJson1_1, restJson1, awsQuery, ec2Query, and restXml. Requests are signed with SigV4 from ctx.aws_credentials, so connections use aws_role. Pagination follows the model's @paginated trait. A query's regions is a list, a comma-separated string, or "*" for the connection's regions (or region) config; each item gains a region field, and $region can be projected. A failing region is skipped and marks the collection incomplete; if every region fails, the first error is raised. Which operations count as reads is defined in RFC 0007 section 3.
#Generated modules
The description cannot be read from disk at runtime, so the CLI compiles it into a Python module beside the entrypoint: _ogrl_openapi.py (settings and description, compressed) and _ogrl_smithy.py (models trimmed to queryable operations and the shapes they reach). run, test, and pack regenerate them; openapi build and smithy build do it explicitly. pack also ships the raw description and the trimmed models so the host can validate queries without importing package code.
#Testing
ogrl_integration.testing runs operations in process over the same JSON-RPC framing the host uses.
Harness(app).validate(config, secrets=None)returns theconnection.validateresult.Harness(app).collect(resource, version, config, secrets=None, query=None)returnsCollected(items, complete, reason, logs)and asserts thatcountequals the items streamed.Harness(app).check(name, version, snapshots, args=None, params=None)takes{source: [items]}, such as{"dev.acme.directory/users@1": [...]}, and returns the outcome dict.- An error answer raises
HarnessError, with.kindand the raw.error.
secrets replaces the whole environment for the call, so tests see exactly what the host would pass.
Cassette(path, mode="replay", transport=None) is an HTTP transport. Assign it to app.transport. In record mode it sends real requests and writes {"interactions": [...]} with method, url, status, headers, body, and body_sha256 for requests with a body. Request headers are never stored; Authorization, Cookie, Set-Cookie, X-API-Key, and Proxy-Authorization response headers are dropped; and query values for access_token, token, api_key, apikey, key, client_secret, and code become SCRUBBED. In replay mode it matches method, scrubbed URL, and body hash in recorded order, and raises CassetteMiss when nothing matches. Hand-written entries may omit body_sha256 to match any body; ogrl-github's cassettes are written by hand from GitHub's documented response shapes.
#CLI
ogrl-integration <command>:
| Command | What it does |
|---|---|
new DIR --id ID --name NAME | Scaffolds a package with one resource, one check, a cassette, and passing tests. |
run PKG validate | Runs connection.validate in process. |
run PKG collect NAME@MAJOR [--out FILE] [--query FILE] | Collects a resource, or a query source defined in a YAML file. Writes JSON Lines to --out or stdout and a count to stderr. |
run PKG check NAME@MAJOR --snapshot SOURCE=PATH [--arg K=V] | Runs a check. Snapshot files may be a snapshot document, a JSON list, or JSON Lines. |
test PKG | Runs the conformance suite and prints PASS/FAIL per finding. Exits 1 on any failure. |
pack PKG [--out dist] [--force] | Runs conformance, then writes <id>-<version>.tar.gz and prints its SHA-256. --force packs despite failures. |
openapi build PKG / openapi search PKG TEXT | Writes _ogrl_openapi.py; searches read operations by id, path, or summary. |
smithy build PKG / smithy search PKG TEXT [--limit N] | Writes _ogrl_smithy.py; searches queryable operations. |
smithy describe PKG SERVICE.Operation | Prints parameters, item fields, pagination, and why an operation is not queryable. |
smithy validate PKG QUERY.yaml | Validates a query definition and prints the derived item schema. |
run accepts --config KEY=VALUE (repeatable) and --record CASSETTE or --replay CASSETTE. Credentials come from the environment, limited to the manifest's fields names, OGRL_ACCESS_TOKEN, and the AWS_* variables.
Conformance checks that the manifest loads; id, version, protocol, and runtime.kind are valid; there is at least one auth kind; every resource schema is a valid object schema declaring id and name; checks read only declared resources and templates only this integration; the entrypoint imports and Integration(id, version) matches the manifest; the implemented resources and checks equal the manifest's; and every check survives empty snapshots. It then starts the entrypoint as the host would (working directory /, no HOME, PYTHONDONTWRITEBYTECODE=1) and verifies that stdout carries only JSON lines, initialize answers the manifest's id and version, unknown methods answer an error with a kind, and the process exits within 10 s of stdin closing.
Pack output is deterministic: entries are sorted with zeroed timestamps and owners, and the archive contains manifest.yaml, schemas/, every .py file, the OpenAPI description or trimmed Smithy models, any wheels/*.whl, and a pure-Python wheel of the SDK itself. Signing is not implemented. To install the archive, see Integrations.
#End-to-end example
manifest.yaml:
id: dev.acme.directory
name: Acme Directory
version: 0.1.0
protocol: 1
publisher: acme
description: Users from Acme Directory.
runtime:
kind: python
entrypoint: directory_integration:app.run
egress: [api.acme.example]
config:
- name: base_url
label: API base URL
pattern: "^https://"
auth:
- kind: fields
label: API token
fields:
- name: ACME_TOKEN
label: Read-only API token
resources:
- name: users
version: 1
schema: schemas/users.json
checks:
- name: all-mfa
version: 1
sources: [users]
templates: []
schemas/users.json:
{
"type": "object",
"required": ["id", "name"],
"properties": {
"id": {"type": "string"},
"name": {"type": "string"},
"mfa": {"type": "boolean"}
}
}
directory_integration.py:
from ogrl_integration import Integration, Subject, fail, pass_
app = Integration("dev.acme.directory", "0.1.0")
def _auth(ctx):
return {"Authorization": f"Bearer {ctx.secret('ACME_TOKEN')}"}
@app.validate
def validate(ctx):
me = ctx.http.get(f"{ctx.config['base_url']}/me", headers=_auth(ctx)).json()
return ctx.identity(f"{ctx.config['base_url']} as {me['name']}", scopes=me.get("scopes", []))
@app.resource("users", version=1)
def users(ctx):
for user in ctx.http.paginate(
f"{ctx.config['base_url']}/users",
headers=_auth(ctx),
items=lambda r: r.json()["users"],
next_url=lambda r: r.json().get("next"),
):
yield {"id": user["id"], "name": user["email"], "mfa": user.get("mfa", False)}
@app.check("all-mfa", version=1)
def all_mfa(ctx, users):
missing = [u for u in users if not u.get("mfa")]
if missing:
return fail(
f"{len(missing)} of {len(users)} users lack MFA",
[Subject.account(u["name"], u["id"]) for u in missing],
)
return pass_(f"{len(users)} users have MFA")
tests/test_directory_integration.py:
from pathlib import Path
from ogrl_integration.testing import Cassette, Harness
from directory_integration import app
CASSETTES = Path(__file__).parent / "cassettes"
def test_collects_users():
app.transport = Cassette(CASSETTES / "users.json")
result = Harness(app).collect("users", 1, {"base_url": "https://api.acme.example"}, {"ACME_TOKEN": "t"})
assert result.complete and result.items[0]["id"] == "u1"
def test_check_names_users_without_mfa():
bob = {"id": "u2", "name": "bob@acme.example", "mfa": False}
out = Harness(app).check("all-mfa", 1, {"dev.acme.directory/users@1": [bob]})
assert out["outcome"] == "fail"
assert out["subjects"] == [{"kind": "account", "name": "bob@acme.example", "external_id": "u2"}]
Record a cassette once against the real API, then work offline:
export ACME_TOKEN=...
ogrl-integration run . collect users@1 --config base_url=https://api.acme.example \
--record tests/cassettes/users.json --out users.jsonl
ogrl-integration run . check all-mfa@1 --snapshot dev.acme.directory/users@1=users.jsonl
PYTHONPATH=. pytest
ogrl-integration test .
ogrl-integration pack . --out dist/