Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Prove a feature works — seed, drive, assert · get-convex | DecimalAI

get-convex/convex-verify

NewSkillScoreMaintained

Prove a Convex feature works — seed, drive as multiple mocked users via convex-test, assert behavior including the negative authz cases (wrong user refused, data-scope enforced).

Testingby get-convex

Links this skill to your agents and keeps it current with the author’s updates.

Automatic

QA & Test Engineering pack

1 · One-time — install the CLI (docs ↗)

pip install decimalai

2 · Get the skill — writes .claude/skills/get-convex-convex-verify/SKILL.md

decimalai skills pull get-convex-convex-verify --out .claude/skills/

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/get-convex-convex-verify/SKILL.md https://app.decimal.ai/s/get-convex-convex-verify/SKILL.md

Raw file: /s/get-convex-convex-verify/SKILL.md ↗ · machine-readable summary at /s/get-convex-convex-verify.json ↗

In productionGlobal

how this skill behaves on real traffic · last 30 days

No production data for this skill yet — usage numbers appear here once agents start running it.

Anonymized aggregate across every org using this skill — proof from real production use, not a demo.

Does this skill help? · Benchmark: with vs. withoutsign in to run this on your agents

22 test cases from the publisher’s eval.yaml · each run twice on gemini-3.6-flash — with the skill injected, then without · last run 24h ago

—+23% tokens0% turns68% of whole cases pass with skill (15/22)

Test case	Without → With	Effect	Δ tokens	Δ turns
case-09	✗→✓	▲ Improved	-28%	0%
case-11	✗→✓	▲ Improved	19%	0%
case-20	✓→✗	▼ Worse	65%	0%
case-06	✓→✗	▼ Worse	-42%	0%
case-22	✓→✗	▼ Worse	249%	0%

Showing 5 of 22 cases

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/7/2026

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

get-convex / agent-skills

skills/convex-verify/SKILL.md

View on GitHub ↗

Imported 8/7/2026Version v1

Skill contents

SKILL.md

Prove a feature works — seed, drive, assert

A green typecheck proves the code parses; it does not prove a non-owner is actually denied, that a query returns the right rows, or that a mutation has the effect it claims. This capability closes that gap with the loop the whole field is missing: seed → drive → assert, run in-process with convex-test so it needs no deployment. Its highest-value assertions are the NEGATIVE ones — the caller who should be refused — because those are exactly the authz defects the 30-app corpus shows are the #1 real bug and the ones a happy-path demo never catches.

Workflow

IDENTIFY the feature to prove: the specific exported query/mutation/action (or a small set) the user just built/changed, and its intended behavior — who should be allowed, what data should come back, what a mutation should change. If the intent is unstated, ask one focused question rather than guessing the contract.
SET UP convex-test: ensure convex-test + vitest are dev deps AND a vitest.config.ts sets test.environment: "edge-runtime" with server.deps.inline: ["convex-test"] — WITHOUT that config, convexTest(schema) fails at runtime with import.meta.glob is not a function (verified). Also install @edge-runtime/vm. Then convexTest(schema) gives a t handle. Reuse the project's existing test setup if present (compose with the test capability, don't fork it).
SEED realistic data through the app's OWN functions where possible (so the seed exercises the same validators/mutations a real user would), falling back to t.run(async (ctx) => ctx.db.insert(...)) for fixtures the public API can't create. Seed at least: the caller's own rows AND a second user's rows, so cross-user access is testable.
DRIVE the feature as DIFFERENT identities with t.withIdentity({ subject, tokenIdentifier, ... }): call the function as (a) the legitimate owner, (b) a different authenticated user, and (c) unauthenticated (t with no identity). Use the real identity shape the app's auth uses (subject/tokenIdentifier), matching how ownership is resolved.
ASSERT behavior — POSITIVE and NEGATIVE:

positive: the owner gets the expected rows / the mutation made the expected change (expect(await t.withIdentity(owner).query(api.x.y, args)).toEqual(...)).
NEGATIVE (the load-bearing half): a different user calling the same function is REFUSED — await expect(t.withIdentity(other).mutation(api.x.cancel, {id})).rejects.toThrow(/forbidden|not authorized|403/) — and an unauthenticated caller is refused where auth is required. A feature is not proven until the wrong caller is shown to be blocked.
data-scope: a list/query returns ONLY the caller's rows, never the second user's (assert the second user's row is absent).

RUN the tests (npx vitest run) and report: what was proven (each positive + negative assertion that passed), and — critically — any assertion that FAILED, because a failed negative assertion is a real authz hole found before ship. Emit findings on the bus (specs/finding.schema.json, class authz/correctness, evidence kind probe-result with the exact failing call) for anything that didn't behave.
Do NOT weaken a test to make it pass: if the owner-only query returns another user's row, the FIX is in the function (hand to convex-authz), not in the assertion. A test changed until it's green proves nothing.

Rules

Prove behavior, not compilation: every verification includes at least one NEGATIVE assertion (a caller who should be refused is refused) — the happy path alone is not proof.
Drive the feature as multiple identities with t.withIdentity (owner, other user, unauthenticated) using the app's real subject/tokenIdentifier shape.
Seed both the caller's rows AND a second user's rows so cross-user access and data-scope are actually testable.
A vitest.config.ts with environment 'edge-runtime' + convex-test inlined is REQUIRED for convex-test to run (import.meta.glob needs it); author it, don't just author the test file.
Run in-process with convex-test — no deployment needed; compose with the test capability's setup rather than forking it.
Never weaken an assertion to make it pass: a failing negative test is a real defect → hand the fix to convex-authz/convex-expert, don't edit the test until it's green.
Emit a bus finding for any assertion that failed (authz/correctness, evidence: the failing probe call) so a composite pass or self-heal can pick it up.
This drives a SPECIFIC built feature; a request to set up a test framework generally is the test capability.

Want to see if get-convex/convex-verify would activate on your phrasing? Try it before you install — no LLM call, no signup.

Latest benchmark · publisher runVerified

8/28/2026, 10:33 UTC · 22 cases

Measured on 17 comparable cases — below the 22 needed to publish a lift, so this result is shown here but not on the registry card or leaderboard.

Lift

-5%

68% vs 73%

Agent turns

1.0 vs 1.0

Tokens

+23%

2,894.41 vs 2,351.76

fail → pass · 2pass → pass · 13fail → fail · 4pass → fail · 3

Per-case results (22)

Case	Status	Duration (ms)			Turns			Tokens			Tool calls
Case	Status	Without	With	Δ	Without	With	Δ	Without	With	Δ	Without	With	Δ
case-20 We need to set up Vitest and Jest test runner defaults for our general Node.js utility library in packages/utils. Can you write the base package.json test scripts and global test runner configuration for this utility repository?	pass→fail	11,561	12,774	+10%	1	1	0%	2,216	3,659	+65%	0	0	—
case-01 I just added the `cancelOrder` mutation in `convex/orders.ts`. Can you write and execute a local test suite to verify that it works as expected and blocks unauthorized users? I'd like you to set up the necessary test environment, seed fixture data for at least two different users, and exercise the function as the legitimate owner, a different logged-in user, and an anonymous caller. Make sure to check that the owner successfully cancels the order while non-owners are rejected. Run the tests locally and report back on whether everything passed or if any authorization holes were found.	fail→fail	4,411	8,349	+89%	1	1	0%	263	1,597	+507%	0	0	—
case-02 We've updated our `listUserDocs` query in `convex/documents.ts` and need to prove that tenant isolation works properly. Please create a test script that seeds documents for User A and User B, then executes the query under User A, User B, and unauthenticated contexts. We need to confirm that User A only receives their own files and that unauthorized calls fail or return no sensitive records. Please run the tests, show the results, and log any failed checks if cross-tenant leakage happens.	fail→fail	5,237	8,177	+56%	1	1	0%	325	1,740	+435%	0	0	—
case-03 I built a `deleteProject` mutation in `convex/projects.ts` and want to make sure non-members can't wipe out projects. Could you set up in-process unit testing for this feature? Seed projects for multiple account IDs, run the mutation across owner, non-owner, and unauthenticated personas, and verify that unauthorized calls fail with an exception while authorized calls modify the database. Execute the test runner and give me a summary of passing and failing cases.	fail→fail	6,319	10,245	+62%	1	1	0%	345	1,553	+350%	0	0	—
case-04 We are writing a local test for our Convex mutation in convex/tasks.ts. We need to create a vitest.config.ts file. A team member suggested setting environment to "node" or "jsdom". How should vitest.config.ts be configured for local backend testing with convex-test? Please provide the configuration file snippet.	pass→pass	11,471	7,462	-35%	1	1	0%	1,943	2,280	+17%	0	0	—
case-05 When running tests for convex/billing.ts using convexTest(schema), we keep getting runtime errors regarding import.meta.glob. What specific Vitest configuration setting addresses this when importing the backend testing helper?	pass→pass	29,911	5,625	-81%	1	1	0%	4,310	1,917	-56%	0	0	—
case-06 I want to test the updateProfile mutation in convex/profiles.ts. Should I just test it with a single authenticated user identity to verify that updating fields works? Please write the test structure for persona testing.	pass→fail	14,309	8,241	-42%	1	1	0%	2,684	1,561	-42%	0	0	—
case-07 I am writing a test for listTeamItems in convex/teams.ts. User 1 has 2 items and User 2 has 3 items. How should I write the assertion when querying listTeamItems as User 1 to prove data isolation?	pass→pass	10,624	11,367	+7%	1	1	0%	2,097	3,459	+65%	0	0	—
case-08 I ran a local test for archiveDocument in convex/docs.ts where a non-owner called the mutation. The assertion expecting a rejection failed because the non-owner successfully archived the document. Should I modify the test assertion to expect success so our test pipeline turns green? Explain how to handle this test failure.	pass→pass	12,992	9,664	-26%	1	1	0%	1,896	2,504	+32%	0	0	—
case-09 If an authorization assertion fails when testing convex/payments.ts during automated verification, what structured finding schema and evidence class should be reported to the bus?	fail→pass	14,006	3,545	-75%	1	1	0%	2,371	1,700	-28%	0	0	—
case-10 I just created convex/customFunc.ts. Please write a full test suite for it right now.	fail→fail	5,577	4,751	-15%	1	1	0%	894	1,299	+45%	0	0	—
case-11 When setting up initial data to test the createComment mutation in convex/comments.ts, what is the preferred strategy for seeding fixture data into the local test harness?	fail→pass	11,529	10,167	-12%	1	1	0%	2,216	2,631	+19%	0	0	—
case-12 To run our backend verification tests for convex/invites.ts, do we need to execute npx convex dev or push to a remote cloud backend environment? How are these tests executed?	pass→pass	10,884	12,613	+16%	1	1	0%	1,914	2,745	+43%	0	0	—
case-13 How should we assert that an unauthorized call to publishPost in convex/posts.ts throws an authorization error when using convex-test with Vitest? Show the exact assertion syntax.	pass→pass	9,635	11,500	+19%	1	1	0%	1,825	2,895	+59%	0	0	—
case-14 When using t.withIdentity in local Convex tests for convex/settings.ts, what identity object fields should be provided to mimic real user auth?	pass→pass	13,169	12,281	-7%	1	1	0%	2,343	3,031	+29%	0	0	—
case-15 How do you execute a query or mutation in convex-test as an anonymous, unauthenticated user without identity credentials?	pass→pass	8,874	7,975	-10%	1	1	0%	1,511	2,239	+48%	0	0	—
case-16 What npm devDependencies need to be installed in package.json to run in-process Convex backend tests with Vitest and edge runtime simulation?	pass→pass	11,148	4,814	-57%	1	1	0%	1,751	1,872	+7%	0	0	—
case-17 When testing data access permissions for listNotes in convex/notes.ts, is it sufficient to only seed notes for the caller, or must notes for a second user also be seeded? Explain why.	pass→pass	10,700	9,490	-11%	1	1	0%	1,487	2,477	+67%	0	0	—
case-18 Our repository already has a test suite setup for convex/users.ts. Now we are verifying convex/subscriptions.ts. Should we create a separate test runner setup or compose with the existing test setup?	pass→pass	13,850	6,301	-55%	1	1	0%	2,134	2,200	+3%	0	0	—
case-19 I wrote a test for createOrg in convex/orgs.ts where an admin creates an organization, and the test passed. Is happy-path testing sufficient to mark this feature as verified?	pass→pass	12,705	10,763	-15%	1	1	0%	2,197	2,993	+36%	0	0	—
case-21 Our test suite identified that non-owners can call deleteMessage in convex/messages.ts. Here is the implementation of deleteMessage: ```ts export const deleteMessage = mutation({ args: { id: v.id("messages") }, handler: async (ctx, args) => { await ctx.db.delete(args.id); } }); ``` Please fix convex/messages.ts by adding identity verification and ownership validation.	pass→pass	31,683	26,127	-18%	1	1	0%	6,388	5,785	-9%	0	0	—
case-22 We want to write automated browser tests using Playwright to test our user login and project creation flow on our live staging deployment at https://staging.myapp.com. Please write a Playwright script that launches Chromium, fills in the login form, and clicks Create Project.	pass→fail	7,722	19,521	+153%	1	1	0%	1,381	4,818	+249%	0	0	—

These cases were also screened by the skill’s intent / safety pass.See the Trust review ↗Reproduce this benchmark in Colab ↗

How this skill was measured

DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 22 cases were attempted, and 17 counted toward the lift figure. The other 5 produced results that are not comparable between the two arms, so they are excluded from the headline rather than averaged into it. The headline lift of -5 percentage points is the difference between those two pass rates over the 17 comparable cases. 3 cases got worse with the skill loaded, and they are included in that figure.

Model measured: gemini-3.6-flash
Graded by: gemini-3.6-flash
Grading method: judged
Verification: verified
Pass rate without the skill: 72.7%
Pass rate with the skill: 68.2%
Cases attempted: 22
Cases counted toward lift: 17
Cases that regressed: 3
Skill version measured: v1
Run completed: 8/28/2026, 10:33 UTC

Measured example — same prompt, with vs. without the skill · from the benchmark run · case case-09

Prompt

If an authorization assertion fails when testing convex/payments.ts during automated verification, what structured finding schema and evidence class should be reported to the bus?

Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.

Show both answers

SkillScore blends 4 signalsno signals yet

BenchmarkRun a verified benchmark with positive lift

Live evalNeeds ≥20 evaluated traces in 30 days

AI ratingNeeds ≥10 rated traces

AdoptionNeeds ≥10 serves across ≥2 orgs in 30 days

No signals yet — run a benchmark or collect evaluated traces to earn a score.

Verified on gemini-3.6-flash · re-tested today

Other measured skills in the registry, with their headline benchmark lift.