Skip to content

Commit 13bbd29

Browse files
koickoic
committed
Support running MCP conformance tests against the Ruby SDK
## Motivation To assess and track the Ruby SDK's compliance with the MCP specification, integrate `@modelcontextprotocol/conformance` as the standard evaluation harness. This makes it possible to measure conformance objectively, catch regressions early, and produce the kind of evaluation data needed for SDK tier assessments. ## Scope Only server conformance is included at this point. Client conformance requires protocol-level features that MCP::Client does not yet support (initialize handshake, notifications, ping, etc.). ## Usage Run conformance tests: ```console bundle exec rake conformance ``` Start the conformance server in the foreground for iterating on a single scenario without the per-run startup overhead: ```console bundle exec rake conformance_server ``` ## Default Rake Task (`bundle exec rake`) It is configured to run in order of lower execution cost: `rubocop`, `test`, and then `conformance`. ## CI `.github/workflows/conformance.yml` file has been added to run the conformance tests in CI. ## SDK Tier Report The MCP SDK Tier system (https://modelcontextprotocol.io/community/sdk-tiers) requires SDK maintainers to self-assess and report results to the SDK Working Group via https://github.com/modelcontextprotocol/modelcontextprotocol/issues. To generate a full tier assessment report, use the `/mcp-sdk-tier-audit` slash command from the https://github.com/modelcontextprotocol/conformance repository with the conformance server running: ```console # Terminal 1 (this repository): start the conformance server bundle exec rake conformance_server # Terminal 2 (conformance repository): run the tier audit skill as a slash command in Claude Code /mcp-sdk-tier-audit /path/to/modelcontextprotocol/ruby-sdk http://localhost:9292/mcp ``` The skill evaluates conformance pass rate, issue label taxonomy, triage metrics, documentation coverage, and policy compliance, then produces a markdown report suitable for tier advancement submissions.
1 parent 5f16abf commit 13bbd29

File tree

7 files changed

+835
-1
lines changed

7 files changed

+835
-1
lines changed

‎.github/workflows/conformance.yml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
name: Conformance Tests
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
workflow_dispatch:
8+
9+
concurrency:
10+
group: conformance-${{ github.ref }}
11+
cancel-in-progress: true
12+
13+
permissions:
14+
contents: read
15+
16+
jobs:
17+
server-conformance:
18+
runs-on: ubuntu-latest
19+
continue-on-error: true
20+
steps:
21+
- uses: actions/checkout@v6
22+
- uses: ruby/setup-ruby@v1
23+
with:
24+
ruby-version: '4.0' # Specify the latest supported Ruby version.
25+
bundler-cache: true
26+
- uses: actions/setup-node@v4
27+
with:
28+
node-version: '24' # Specify the latest Node.js version.
29+
- run: bundle exec rake conformance

‎README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1028,6 +1028,12 @@ The client provides a wrapper class for tools returned by the server:
10281028

10291029
This class provides easy access to tool properties like name, description, input schema, and output schema.
10301030

1031+
## Conformance Testing
1032+
1033+
The `conformance/` directory contains a test server and runner that validate the SDK against the MCP specification using [`@modelcontextprotocol/conformance`](https://github.com/modelcontextprotocol/conformance).
1034+
1035+
See [conformance/README.md](conformance/README.md) for usage instructions.
1036+
10311037
## Documentation
10321038

10331039
- [SDK API documentation](https://rubydoc.info/gems/mcp)

‎Rakefile‎

Lines changed: 40 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,4 +14,43 @@ require "rubocop/rake_task"
1414

1515
RuboCop::RakeTask.new
1616

17-
task default: [:test, :rubocop]
17+
task default: [:rubocop, :test, :conformance]
18+
19+
desc "Run MCP conformance tests (PORT, SCENARIO, SPEC_VERSION, VERBOSE)"
20+
task :conformance do |t|
21+
next unless npx_available?(t.name)
22+
23+
require_relative "conformance/runner"
24+
25+
options = {}
26+
options[:port] = Integer(ENV["PORT"]) if ENV["PORT"]
27+
options[:scenario] = ENV["SCENARIO"] if ENV["SCENARIO"]
28+
options[:spec_version] = ENV["SPEC_VERSION"] if ENV["SPEC_VERSION"]
29+
options[:verbose] = true if ENV["VERBOSE"]
30+
31+
Conformance::Runner.new(**options).run
32+
end
33+
34+
desc "List available conformance scenarios"
35+
task :conformance_list do |t|
36+
next unless npx_available?(t.name)
37+
38+
system("npx", "--yes", "@modelcontextprotocol/conformance", "list", "--server")
39+
end
40+
41+
desc "Start the conformance server (PORT)"
42+
task :conformance_server do
43+
require_relative "conformance/server"
44+
45+
options = {}
46+
options[:port] = Integer(ENV["PORT"]) if ENV["PORT"]
47+
48+
Conformance::Server.new(**options).start
49+
end
50+
51+
def npx_available?(task_name)
52+
return true if system("which", "npx", out: File::NULL, err: File::NULL)
53+
54+
warn("Skipping #{task_name}: npx is not installed. Install Node.js to run this task: https://nodejs.org/")
55+
false
56+
end

‎conformance/README.md‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
# MCP Conformance Tests
2+
3+
Validates the Ruby SDK's conformance to the MCP specification using [`@modelcontextprotocol/conformance`](https://github.com/modelcontextprotocol/conformance).
4+
5+
## Prerequisites
6+
7+
- Node.js (for `npx`)
8+
- `bundle install` completed
9+
10+
## Running the Tests
11+
12+
### Run all scenarios
13+
14+
```bash
15+
bundle exec rake conformance
16+
```
17+
18+
Starts the conformance server, runs all active scenarios against it, prints a pass/fail
19+
summary for each scenario, and exits with a non-zero status code if any unexpected failures
20+
are detected. Scenarios listed in `expected_failures.yml` are allowed to fail without
21+
affecting the exit code.
22+
23+
### Environment variables
24+
25+
| Variable | Description | Default |
26+
|----------------|--------------------------------------|---------|
27+
| `PORT` | Server port | `9292` |
28+
| `SCENARIO` | Run a single scenario by name | (all) |
29+
| `SPEC_VERSION` | Filter scenarios by spec version | (all) |
30+
| `VERBOSE` | Show raw JSON output when set | (off) |
31+
32+
```bash
33+
# Run a single scenario
34+
bundle exec rake conformance SCENARIO=ping
35+
36+
# Use a different port with verbose output
37+
bundle exec rake conformance PORT=3000 VERBOSE=1
38+
39+
# Start the server on a specific port
40+
bundle exec rake conformance_server PORT=3000
41+
```
42+
43+
### Start the server and test separately
44+
45+
```bash
46+
# Terminal 1: start the server
47+
bundle exec rake conformance_server
48+
49+
# Terminal 2: run all scenarios
50+
npx @modelcontextprotocol/conformance server --url http://localhost:9292/mcp
51+
52+
# Terminal 2: run a single scenario
53+
npx @modelcontextprotocol/conformance server --url http://localhost:9292/mcp --scenario ping
54+
```
55+
56+
Keeps the server alive between test runs, which avoids the startup overhead when iterating
57+
on a single scenario. Stop the server with Ctrl+C when done.
58+
59+
### List available scenarios
60+
61+
```bash
62+
bundle exec rake conformance_list
63+
```
64+
65+
Prints all scenario names that can be passed to `SCENARIO`.
66+
67+
## SDK Tier Report
68+
69+
The [MCP SDK Tier system](https://modelcontextprotocol.io/community/sdk-tiers) requires SDK
70+
maintainers to self-assess and report results to the SDK Working Group via
71+
[modelcontextprotocol/modelcontextprotocol issues](https://github.com/modelcontextprotocol/modelcontextprotocol/issues).
72+
73+
To generate a full tier assessment report, use the `/mcp-sdk-tier-audit` slash command from
74+
the [modelcontextprotocol/conformance](https://github.com/modelcontextprotocol/conformance)
75+
repository with the conformance server running:
76+
77+
```bash
78+
# Terminal 1 (this repository): start the conformance server
79+
bundle exec rake conformance_server
80+
81+
# Terminal 2 (conformance repository): run the tier audit skill as a slash command in Claude Code
82+
/mcp-sdk-tier-audit /path/to/modelcontextprotocol/ruby-sdk http://localhost:9292/mcp
83+
```
84+
85+
The skill evaluates conformance pass rate, issue label taxonomy, triage metrics, documentation
86+
coverage, and policy compliance, then produces a markdown report suitable for tier advancement
87+
submissions.
88+
89+
## File Structure
90+
91+
```
92+
conformance/
93+
server.rb # Conformance server (Rack + Puma, default port 9292)
94+
runner.rb # Starts the server, runs npx conformance, exits with result code
95+
expected_failures.yml # Baseline of known-failing scenarios
96+
README.md # This file
97+
```
98+
99+
## Known Limitations
100+
101+
Known-failing scenarios are registered in `conformance/expected_failures.yml`. They are allowed to
102+
fail without affecting the exit code and are tracked to catch regressions.
103+
These are shown in the output of `bundle exec rake conformance`.

‎conformance/expected_failures.yml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
server:
2+
# TODO: Server-to-client requests (sampling/createMessage, elicitation/create) are not implemented.
3+
# `Transport#send_request` does not exist in the current SDK.
4+
- tools-call-sampling
5+
- tools-call-elicitation
6+
- elicitation-sep1034-defaults
7+
- elicitation-sep1330-enums
8+
# TODO: The SDK does not extract `_meta.progressToken` from tool call requests or deliver `notifications/progress` to tools.
9+
- tools-call-with-progress

‎conformance/runner.rb‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
# frozen_string_literal: true
2+
3+
# Starts the conformance server and runs `npx @modelcontextprotocol/conformance` against it.
4+
require "English"
5+
require "net/http"
6+
require_relative "server"
7+
8+
module Conformance
9+
class Runner
10+
# Timeout for waiting for the Puma server to start.
11+
SERVER_START_TIMEOUT = 20
12+
SERVER_POLL_INTERVAL = 0.5
13+
SERVER_HEALTH_CHECK_RETRIES = (SERVER_START_TIMEOUT / SERVER_POLL_INTERVAL).to_i
14+
15+
def initialize(port: Server::DEFAULT_PORT, scenario: nil, spec_version: nil, verbose: false)
16+
@port = port
17+
@scenario = scenario
18+
@spec_version = spec_version
19+
@verbose = verbose
20+
end
21+
22+
def run
23+
command = build_command
24+
server_pid = start_server
25+
26+
run_conformance(command, server_pid: server_pid)
27+
end
28+
29+
private
30+
31+
def build_command
32+
expected_failures_yml = File.expand_path("expected_failures.yml", __dir__)
33+
34+
npx_command = ["npx", "--yes", "@modelcontextprotocol/conformance", "server", "--url", "http://localhost:#{@port}/mcp"]
35+
npx_command += ["--scenario", @scenario] if @scenario
36+
npx_command += ["--spec-version", @spec_version] if @spec_version
37+
npx_command += ["--verbose"] if @verbose
38+
npx_command += ["--expected-failures", expected_failures_yml]
39+
npx_command
40+
end
41+
42+
def start_server
43+
puts "Starting conformance server on port #{@port}..."
44+
45+
server_pid = fork do
46+
Conformance::Server.new(port: @port).start
47+
end
48+
49+
health_url = URI("http://localhost:#{@port}/health")
50+
ready = false
51+
SERVER_HEALTH_CHECK_RETRIES.times do
52+
begin
53+
response = Net::HTTP.get_response(health_url)
54+
if response.code == "200"
55+
ready = true
56+
break
57+
end
58+
rescue Errno::ECONNREFUSED, Errno::ECONNRESET, Net::ReadTimeout
59+
# not ready yet
60+
end
61+
sleep(SERVER_POLL_INTERVAL)
62+
end
63+
64+
unless ready
65+
warn("ERROR: Conformance server did not start within #{SERVER_START_TIMEOUT} seconds")
66+
terminate_server(server_pid)
67+
exit(1)
68+
end
69+
70+
puts "Server ready. Running conformance tests..."
71+
72+
server_pid
73+
end
74+
75+
def run_conformance(command, server_pid:)
76+
puts "Command: #{command.join(" ")}\n\n"
77+
78+
conformance_exit_code = nil
79+
begin
80+
system(*command)
81+
conformance_exit_code = $CHILD_STATUS.exitstatus
82+
ensure
83+
terminate_server(server_pid)
84+
end
85+
86+
exit(conformance_exit_code || 1)
87+
end
88+
89+
def terminate_server(pid)
90+
Process.kill("TERM", pid)
91+
rescue Errno::ESRCH
92+
# process already exited
93+
ensure
94+
begin
95+
Process.wait(pid)
96+
rescue Errno::ECHILD
97+
# already reaped
98+
end
99+
end
100+
end
101+
end

0 commit comments

Comments
 (0)