I’m familiar with using tools like vale, pre-commit hooks, and GitHub Actions to run automated tests on any files, but documentation in particular. In my NPS hiking project for example, I have a prose lint workflow that runs my vale configuration against files matching paths or file types signaling documentation.
Certain style rules, however, can’t be reduced to regular expressions. LLMs should be able to fill this gap. With this in mind, I’ve been looking into how to leverage AI for peer reviews of not only the code in my projects, but also the documentation. I experimented with a few approaches:
GitHub Copilot
My first thought was to try GitHub Copilot to review pull requests. While I had Claude make generic instruction files for both docs and code, this feature isn’t available on the free plan, and so I wasn’t able to actually try it.
CodeRabbit
CodeRabbit allows for free AI reviews of open source projects. I was able to create a .coderabbit.yaml file including instructions for specific paths, and request reviews of PRs.
Even with quite generic instructions, it was able to offer some constructive feedback on intentionally bad documentation.
Claude Code
I created a slash command for Claude Code to execute an open source skill for avoiding AI writing.
While the skill’s stated purpose is to avoid AI writing patterns, it has good advice for docs in general. Testing it out yielded a few small improvements on docs to which I had already given considerable attention. This commit includes an example of what it changed.
A local LLM
Finally, I wanted to see how well a local open source LLM could execute the same instructions given to Claude. For my NLQ endpoint, I already have an ollama configuration set up. I only needed a small Python script (instead of the slash command) to execute the same skill.
Instead of sending Claude:
/clean-ai-writing Rewrite @docs/getting-started.mdI can run:
python scripts/tools/review_writing.py docs/getting-started.md --rewriteThe initial results honestly weren’t too impressive. The local LLM doesn’t appear to have stuck to looking at just the one file I asked for. I’d want to see how far prompt engineering or other models could take me before ruling it out. That being said, as a free alternative, especially if Claude Code pricing changes, this is certainly an option to keep in mind as open source models improve.