An API can have an agreed OpenAPI design yet behave differently once implemented. Conformance testing exercises the API and checks its responses against that definition. It can catch invalid response schemas, unimplemented operations, and behavior absent from the published API reference. section 1.5.4
Choose where to make the checks:
- Controller-level schema tests give faster feedback from the API service but do not verify responses produced elsewhere in the request path. section 7.6
- End-to-end schema tests check responses as consumers receive them, including behavior introduced by gateways or other intermediaries. Maintaining the underlying end-to-end tests can cost more. section 7.6
- Generated tests can reduce test-code changes as the API evolves. Fuzzy testing can also send unexpected inputs to look for undocumented responses; a validating proxy can add schema checks to an existing end-to-end suite. section 1.5.4 section 7.8
Run generated checks with Portman
Point Portman at an OpenAPI file and a running API, enable its contract tests, then run the generated Postman collection with Newman. For the product-catalog example, `portman --cliOptionsFile portman/portman-cli.json` runs checks configured across API operations for successful status, response schema, content type, JSON body, and headers. The reported check marks show which assertions passed for a request; a failed assertion identifies a response that needs investigation against the definition. `portman --init` can create a starting configuration. section 7.4.2
Product Catalog API ❏ Products ↳ View a product's details GET http://localhost:8000/v1/catalog/ products/c05aed25-97cd-4cbc-b299-3796538eee9c [200 OK, 443B, 46ms] ✓ [GET]::/v1/catalog/products/:id - Status code is 2xx ✓ [GET]::/v1/catalog/products/:id - Schema is valid ✓ [GET]::/v1/catalog/products/:id - Content-Type is application/json ✓ [GET]::/v1/catalog/products/:id - Response has JSON Body Collection run completed.
Inspect a sample collection run to see the request, received status, individual conformance assertions, and completion message. section 7.4.2
Test the externally visible path when the API reference describes external behavior. A gateway can return its own authorization error, so generating an OpenAPI definition from service code alone cannot verify every response consumers see. Generation and schema testing can be combined: generation keeps artifacts aligned, while tests check observed behavior. Run conformance checks after deployment as part of the API delivery workflow. section 6.2.8 section 7.6 section 9.0
