Linting Documentation Code Samples

A well-known problem in developer documentation is code samples becoming stale as the codebase evolves. For my NPS Hikes project, I wanted to explore how to combat this problem with automated checks.

In addition to several new pre-commit hooks, most of this effort comes together in a docs-lint workflow.

Structural checks

Alongside checking code samples, in this workflow:

• markdownlint checks for structurally valid Markdown (blank lines, code fences, headings, and tables, etc).
• vale enforces semantic style rules.
• lychee tests that external links are valid.
• mkdocs runs a strict build of the docs website.

In terms of code samples though, my docs primarily have JSON, bash, and Python examples. I started with static analysis for structural errors.

• blacken-docs formats ```python blocks inside Markdown.
• ruff features in a script that extracts ```python blocks from Markdown into temporary .py files, and runs ruff check against those files.
• Another script implements the same pattern for checking the grammar of ```bash code blocks.
• A third does the same for JSON: extract ```json blocks from Markdown, sanitize partial examples, and check them with json.loads().

JSON validation

These tests handle syntax, but don’t directly tackle the problem of stale samples. For that, in the case of JSON in particular, I need to validate JSON samples against the actual OpenAPI schema.

The script validate_json_schema.py validates annotated JSON examples against Pydantic response-model schemas. Instead of a running service, it loads the FastAPI’s app’s OpenAPI schema at check time. Unlike the syntax check in check_json_schema.py, the validation step picks up on renamed or missing fields or incorrect types.

To do this, I map JSON blocks to ParksResponse and TrailsResponse models with an HTML comment.

<!-- response: GET /parks -->
```json
{
    "park_count": 3,
    ...
} ...    

Given the small number of blocks in my documentation, this was easy. The script:

1. Scans Markdown for <!-- response: METHOD /path --> followed by a ```json block.
2. Maps the endpoint to a Pydantic model via a lookup dict (GET /parks → ParksResponse).
3. Generates the JSON schema from the model with model_json_schema().
4. Patches the schema for partial doc examples:
   • Removes required constraints (doc examples intentionally omit fields).
   • Adds additionalProperties: false (catches renamed or bogus fields).
5. Sanitizes the doc JSON (same .../[...] handling as the syntax checker).
6. Validates with jsonschema.Draft202012Validator.

Future improvements

This approach is still fairly limited. For example, it doesn’t catch missing required fields. The required constraint is removed because doc examples are often intentionally partial (e.g., the Trail example omits geometry_type).

While there’s a lot more that could be done for more complex requirements, this approach largely meets the need for the scope of my documentation.