Distribution
Current channels
GitHub Releases
Pre-built binaries via GoReleaser on every v* tag. Cross-compiled for linux/darwin/windows, amd64/arm64.
go install github.com/blackwell-systems/mcp-assert/cmd/mcp-assert@latest
GitHub Action
blackwell-systems/mcp-assert-action@v1: one line in any workflow. Downloads binary, runs assertions, uploads JUnit XML + badge. Listed on GitHub Marketplace.
- uses: blackwell-systems/mcp-assert-action@v1
with:
suite: evals/
How it stays in sync: The action is a composite action in a separate repo. It downloads the mcp-assert binary from GitHub Releases at runtime (default: latest). New mcp-assert releases are picked up automatically without touching the action repo.
When to update the action repo: Only when the action's own interface changes (new inputs, new outputs, different install logic). Not for new mcp-assert features or bug fixes.
Versioning: Users pin to @v1 (floating major tag). When updating the action repo, tag the specific release (v1.0.1) and force-update the v1 tag:
cd mcp-assert-action
git tag v1.0.1 && git push origin v1.0.1
git tag -fa v1 -m "Update v1 to v1.0.1" && git push origin v1 --force
Marketplace listing: Updates automatically when a new tag is pushed to the action repo. GitHub reads action.yml from the tagged commit.
Homebrew
brew install blackwell-systems/tap/mcp-assert
Formula auto-generated by GoReleaser on each release and pushed to blackwell-systems/homebrew-tap.
curl | sh (macOS / Linux)
curl -fsSL https://raw.githubusercontent.com/blackwell-systems/mcp-assert/main/install.sh | sh
Detects OS and architecture, downloads the matching binary from GitHub Releases, installs to /usr/local/bin. Set INSTALL_DIR to override.
npm
npx @blackwell-systems/mcp-assert
Uses the platform-specific optional dependency pattern (same as esbuild, turbo). npm resolves the correct binary for the user's OS/arch automatically. No Go toolchain required. Published via scripts/npm-publish.sh on each release tag.
PyPI
pip install mcp-assert
Platform-specific wheels containing the Go binary. Each wheel is tagged with the correct platform (e.g. macosx_11_0_arm64, manylinux2014_x86_64), so pip resolves the right one automatically. Built via scripts/pypi-build-wheels.sh and published with twine on each release tag.
Scoop (Windows)
scoop bucket add blackwell-systems https://github.com/blackwell-systems/scoop-bucket
scoop install mcp-assert
Manifest auto-generated by GoReleaser on each release and pushed to blackwell-systems/scoop-bucket.
pytest Plugin
pip install pytest-mcp-assert
pytest --mcp-suite evals/
Each YAML assertion becomes a pytest test item with pass/fail/skip semantics. Configurable via pyproject.toml (mcp_suite, mcp_fixture, mcp_timeout). The Go binary handles all MCP logic; the plugin is a 170-line bridge.
Source: pytest-plugin/ directory in this repo. Published to PyPI automatically by the pytest-plugin-publish job in release.yml on each v* tag.
"Works with mcp-assert" Badge
Static and dynamic (CI-verified) badge for MCP server READMEs. Every badge is a backlink. See the Badge guide.
Awesome MCP DevTools
PR submitted (punkpeye/awesome-mcp-devtools#144). Pending maintainer review. Listed under Testing Tools.
Planned
| Channel | Priority | Description |
|---|---|---|
| Nix flake | Low | Nix users are quality-focused and vocal. High signal in a niche community. |
| MCP registry integration | Low | Test badge on Glama and Smithery listings. |
How to release
A single git tag triggers the entire pipeline. Every channel is automated.
Step-by-step
-
Update CHANGELOG.md: move items from
[Unreleased]to a new version header (e.g.,## [0.5.0] - 2026-05-01). -
Tag and push:
bash git tag v0.5.0 git push origin v0.5.0 -
The
release.ymlworkflow does the rest (triggered by thev*tag):
| Job | What it does | Channel updated |
|---|---|---|
release (GoReleaser) |
Builds binaries for 6 platforms, creates GitHub Release, pushes Homebrew formula to homebrew-tap, pushes Scoop manifest to scoop-bucket | GitHub Releases, Homebrew, Scoop |
npm-publish |
Runs scripts/npm-publish.sh, downloads binaries from the Release, publishes platform packages to npm |
npm |
pypi-publish |
Runs scripts/pypi-publish.sh, builds platform wheels with Go binary inside, publishes to PyPI with twine |
PyPI |
pytest-plugin-publish |
Builds and publishes the pytest-mcp-assert plugin to PyPI with twine |
PyPI (pytest plugin) |
- Verify (after CI completes, ~5 minutes): ```bash # GitHub Release exists gh release view v0.5.0
# Homebrew updated brew update && brew info blackwell-systems/tap/mcp-assert
# npm updated npm view @blackwell-systems/mcp-assert version
# PyPI updated pip index versions mcp-assert ```
What is NOT automatic
| Channel | How to update |
|---|---|
| GitHub Action | Only update when the action's interface changes. See the GitHub Action section above. |
install.sh / install.ps1 |
These fetch latest from GitHub Releases. No update needed unless the script logic changes. |
| Badges | Static SVGs in assets/. Update manually if the badge design changes. |
| Awesome MCP DevTools listing | Manual PR to the upstream repo. |
Secrets required
| Secret | Where | Purpose |
|---|---|---|
GITHUB_TOKEN |
Automatic (GitHub provides) | GoReleaser, GitHub Release creation |
NPM_TOKEN |
Repository secret | npm publish |
PYPI_TOKEN |
Repository secret | twine upload to PyPI |
Rollback
If a release has a critical bug: delete the GitHub Release and tag, then publish a patch.
gh release delete v0.5.0 --yes
git push origin :refs/tags/v0.5.0
# Fix the bug, then tag v0.5.1
npm and PyPI don't support deletion (npm has a 72-hour unpublish window). Publish a patch version instead.
The scan-and-contribute flywheel
mcp-assert spreads through the bugs it finds, not through marketing. Every issue filed on a popular MCP server is passive promotion with built-in credibility.
Scan server -> Find bugs -> File issue -> Link mcp-assert -> Maintainers discover tool -> Adopt
Server suites shipped
| Server | Language | Assertions | Coverage | Notes |
|---|---|---|---|---|
agent-lsp + gopls |
Go | 63 | 100% (50/50 tools) | Internal dogfooding server |
agent-lsp skill protocols |
N/A | 20 | 20/20 skills | Trajectory assertions |
@modelcontextprotocol/server-filesystem |
TypeScript | 14 | 92% | Read, list, search, info, write, edit, create dir, move, directory tree, path traversal rejection |
@modelcontextprotocol/server-memory |
TypeScript | 9 | 100% (9/9 tools) | Clean scan |
mcp-server-time |
Python | 5 | 100% (2/2 tools) | Clean. UTC, named timezone, conversion, invalid timezone rejection. |
mcp-server-fetch |
Python | 3 | 100% (1/1 tool) | Clean. URL fetch, invalid URL, unreachable host. |
mcp-server-git |
Python | 11 | 92% (11/12 tools) | Clean. Status, log, branch, diff, show, invalid repo/ref rejection. |
mcp-server-sqlite |
Python | 9 | 100% (6/6 tools) | Clean scan |
mark3labs/mcp-go everything |
Go | 9 | 100% | stdio |
mark3labs/mcp-go everything |
Go | 5 | 100% | HTTP transport conformance |
mark3labs/mcp-go everything (prompts) |
Go | 4 | 100% | prompts/list, prompts/get |
mark3labs/mcp-go everything (resources) |
Go | 4 | 100% | resources/list, read, subscribe |
mark3labs/mcp-go everything (completion) |
Go | 3 | -- | completion/complete |
mark3labs/mcp-go everything (logging) |
Go | 2 | -- | logging/setLevel, message capture |
mark3labs/mcp-go typed_tools |
Go | 3 | 100% | Clean |
mark3labs/mcp-go structured |
Go | 6 | 100% | Clean |
mark3labs/mcp-go roots_server |
Go | 1 | 100% | Bidirectional roots/list |
mark3labs/mcp-go sampling_server |
Go | 3 | 100% | Bidirectional sampling |
mark3labs/mcp-go elicitation |
Go | 4 | 100% | Accept, decline, cancel, validation |
PrefectHQ/fastmcp testing_demo |
Python | 16 | 100% | All 3 MCP feature categories |
PrefectHQ/fastmcp testing_demo (SSE) |
Python | 11 | 100% | SSE transport verification |
github/github-mcp-server |
Go | 20 | -- | 17 read-only tools across 7 toolsets: context, repos, git, issues, pull requests, users, gists |
4t145/rmcp counter |
Rust | 14 | 100% (6/6 tools + resources + prompts) | First Rust server. Bug found: get_value mutates state. |
rust-mcp-stack/rust-mcp-filesystem |
Rust | 23 | 92% (22/24 tools) | Clean scan. Read, list, search, write, edit, zip/unzip, path traversal. |
haris-musa/excel-mcp-server |
Python | 15 | 52% (13/25 tools) | Clean. Workbook, sheets, data, formulas, charts, pivots, formatting. |
antvis/mcp-server-chart |
TypeScript | 25 | 93% (25/27 tools) | 9 bugs found (#291). Unhandled exceptions on default input. |
@modelcontextprotocol/server-everything |
TypeScript | 13 | 92% (12/13 tools) | Clean. Official Anthropic reference server. |
hashicorp/terraform-mcp-server |
Go | 5 | 56% (5/9 tools) | Clean. Provider, module, policy search. |
makenotion/notion-mcp-server |
TypeScript | 22 | 100% (22/22 tools) | Clean. Official Notion server. |
jamesward/hello-spring-mcp-server |
Kotlin | 3 | 100% (2/2 tools) | Clean. First JVM server. Spring AI MCP, HTTP transport. |
mongodb/mongodb-mcp-server |
TypeScript | 4 | -- | Clean. Knowledge search, error handling. |
microsoft/playwright-mcp |
TypeScript | 14 | 67% (14/21 tools) | Clean. Navigate, snapshot, screenshot, JS evaluate, console, network, resize, close, tabs, navigate back, press key, wait for element. |
openai/sample-deep-research-mcp |
Python | 4 | 100% (2/2 tools) | Clean. Search and fetch against static JSON. |
@google-cloud/storage-mcp |
TypeScript | 6 | 35% (6/17 tools) | Clean. Bucket metadata, object listing, IAM policy. |
grafana/mcp-grafana |
Go | 17 | 34% (17/50 tools) | 1 bug. 3 live-backend assertions use skip_unless_env. |
blazickjp/arxiv-mcp-server |
Python | 5 | 50% (5/10 tools) | 1 bug: isError flag not set on error content. |
awslabs/aws-documentation-mcp-server |
Python | 4 | 100% (4/4 tools) | Clean. Search, recommend, no-results handling. |
exa-labs/exa-mcp-server |
JavaScript | 2 | 100% (2/2 tools) | Clean. Proper 401 with API key guidance. |
onmyway133/git-mcp |
TypeScript | 14 | 39% (14/36 tools) | Clean. Status, log, branches, diff, show, reflog. |
| Total | 6 languages | 462 | 44 suites |
Target servers (next)
| Server | Language | Stars | Why target |
|---|---|---|---|
Rust MCP servers (rmcp SDK) |
Rust | -- | Underserved community, no existing testing tools |
| Java MCP servers (Spring, Quarkus) | Java | -- | Enterprise community, high signal |
| C# MCP servers (csharp-sdk) | C# | -- | .NET community, underserved |
Content (planned)
| Channel | Status | Description |
|---|---|---|
| r/MCP | Posted 2026-04-24 | Launch post: feature overview, scorecard, install methods |
| r/ClaudeCode | Posted 2026-04-24 | Adapted for Claude Code/Desktop users |
| Posted 2026-04-24 | Launch announcement to professional network | |
| Blog post: dogfooding | Material in docs/dogfooding-github-mcp.md |
"We tested the most popular MCP server and found 4 DX issues in our own tool" |
| MCP community Discord/forums | Not started | Post when awesome-mcp-devtools listing is merged |
| Hacker News | Not started | Ready to post |
Non-goals
- Paid tier. mcp-assert is free and open source. The value is ecosystem positioning, not revenue.
- SaaS dashboard. No hosted version. CI-native, single binary.
- LLM-as-judge features. Stay in our lane. Deterministic assertions only.