---
name: shagghiesuperstar/free-sota-moa
source: https://app.decimal.ai/s/shagghiesuperstar-free-sota-moa@1/SKILL.md
source_sha256: 60be93f1c64a
---

# free-sota-moa

This skill installs and configures the **FreeLLMAPI + Hermes MoA** stack to
produce the best currently available LLM output from free-tier pooled quota.
It is designed to be repeatable at any future date: it queries the live model
catalog rather than hardcoding model names, so the selected models are always
the best currently available.

> **Scope caveat:** This stack operates over pooled free-tier provider quotas.
> Output quality is bounded by what is available and non-exhausted in your
> catalog at any given moment. "SOTA-seeking" means: best available diverse
> panel from your current free-tier pool, not guaranteed frontier-model
> performance. Provider limits, ToS, and model availability all apply.

## When to Use

- User says "set up free MoA", "configure MoA", "use the best free models",
  "set up FreeLLMAPI", or equivalent intent
- User wants to refresh their MoA preset after a model catalog update
- User needs to replicate this setup on a new machine

## Architecture Overview

```
[User / Hermes session]
        │
        ▼
  Hermes Agent  ←──── MoA preset "free-sota"
        │                    │
        │              ┌─────┴──────┐
        │         Reference   Reference
        │         model A     model B
        │         (no tools, (no tools,
        │          advisory)  advisory)
        │              └─────┬──────┘
        │                    ▼
        └──────────── Aggregator model
                       (full tool schema,
                        writes response,
                        emits tool calls)
                             │
                    FreeLLMAPI /v1 router
                     (localhost:3001)
                             │
          ┌──────┬───────────┼──────────┬──────┐
       Groq  OpenRouter  Gemini   DeepSeek  ...
       (free)  (free)    (free)    (free)
```

**Key principle:** Reference models provide heterogeneous analysis — no tool
schemas, cheap advisory calls. The aggregator synthesizes and acts — it has the
full Hermes tool schema and writes the real assistant response. Diversity of
model family in references (not raw count or rank) drives quality lift.
This matches Hermes MoA's documented agent loop exactly.

## Quick Reference

| Step | Command / Location |
|------|--------------------|
| Install FreeLLMAPI | `git clone` + `npm install` + `npm run dev` |
| FreeLLMAPI dashboard | http://localhost:5173 |
| FreeLLMAPI API endpoint | http://localhost:3001/v1 |
| Hermes provider wizard (use this first) | `hermes model` |
| Hermes config file | `~/.hermes/config.yaml` |
| List MoA presets | `hermes moa list` |
| Update a MoA preset interactively | `hermes moa configure free-sota` |
| Activate MoA for session | `/model free-sota --provider moa` |
| One-shot MoA call | `/moa <your prompt>` |
| Verify setup | `hermes doctor` |

---

## Premortem

Review failure modes before executing any phase.

**Failure: FreeLLMAPI port exposed to the internet**
- Indicator: `lsof -i :3001` shows bind on `0.0.0.0` AND the port is reachable outside your local network or Tailscale.
- Rollback: stop FreeLLMAPI, rebind to `127.0.0.1`, rotate FreeLLMAPI unified key from the dashboard, rotate any upstream provider keys that were accessible.

**Failure: FreeLLMAPI unified key or upstream provider keys committed to Git or logged**
- Indicator: `grep -rn "sk-" ~/.hermes/ .` finds a key in tracked files, or terminal scrollback/shell history exposes it.
- Rollback: rotate the key immediately from the dashboard, use `git filter-repo` if committed.

**Failure: `intelligence_rank` field absent from /v1/models response**
- Indicator: Phase 2 discovery script prints `?` in the Rank column or all ranks are 0.
- Rollback: skip automated ranking; use FreeLLMAPI dashboard → Models tab (sorted by rank) and manually copy the top model IDs instead.

**Failure: MoA references all from same model family**
- Indicator: Top 3 ranked models are all from the same provider (e.g., three DeepSeek variants).
- Rollback: override rule — pick the best model from each of the three highest-ranked *distinct* provider families, even if that means accepting lower individual model ranks.

**Failure: `config.yaml` env interpolation not supported by installed Hermes version**
- Indicator: `hermes doctor` fails with "unknown api_key" or literal `${…}` appears in error output.
- Rollback: remove interpolation syntax; use the `hermes model` wizard to register the provider (it stores the key correctly) or paste the literal key value.

---

## Procedure

### Phase 1 — Install and Run FreeLLMAPI

Perform this phase once per machine. Skip if FreeLLMAPI is already running and
reachable at `http://localhost:3001/v1`.

**1.1 Check prerequisites**

```bash
node --version   # must be v20.x or v22.x LTS
npm --version    # must be v10+
python3 --version  # must be 3.10+
```

If Node.js is not installed: https://nodejs.org/en/download or `nvm install --lts`.

**1.2 Clone and configure**

```bash
git clone https://github.com/tashfeenahmed/freellmapi.git
cd freellmapi
cp .env.example .env
```

Generate a 32-byte encryption key and insert it — no clipboard exposure:

```bash
ENCRYPTION_KEY=$(node -e "console.log(require('crypto').randomBytes(32).toString('hex'))")
sed -i.bak "s/^ENCRYPTION_KEY=.*/ENCRYPTION_KEY=$ENCRYPTION_KEY/" .env
unset ENCRYPTION_KEY   # remove from shell environment
echo "Encryption key written to .env (shell env cleared)"
```

> **Security:** Store the `.env` value in a secrets manager (1Password,
> Bitwarden, macOS Keychain) immediately. A lost key makes stored provider
> API keys undecryptable. In production, set `ENCRYPTION_KEY` as a system
> environment variable (e.g., via systemd `EnvironmentFile=`) rather than
> reading from `.env`.

**1.3 Install and start**

```bash
npm install
npm run dev
```

Expected: server on port 3001, dashboard (Vite) on port 5173.
Catalog sync runs on first boot — wait ~30–60 seconds for models to appear.

**1.4 Add provider API keys**

Open http://localhost:5173 → Keys tab. Add keys from at least three
different provider families to support reference diversity:

| Provider | Free key source | Why include |
|----------|----------------|-------------|
| OpenRouter | https://openrouter.ai → API Keys | Widest model family coverage |
| Groq | https://console.groq.com → API Keys | Fast inference, high RPM free tier |
| Google Gemini | https://aistudio.google.com → Get API key | Gemini family diversity |
| DeepSeek | https://platform.deepseek.com → API keys | Strong reasoning, distinct architecture |

All listed providers have free tiers with no payment method required, subject
to their own ToS and daily/monthly quota limits.

**1.5 Confirm catalog sync**

Dashboard → Models tab should populate within 60 seconds. If empty after
2 minutes: check terminal for sync errors and confirm at least one key shows
a green checkmark on the Keys tab.

**1.6 Record your unified API key (without shell exposure)**

Dashboard → Keys → copy the **Unified API key** value.

Store it in `~/.hermes/.env` directly — avoid `export` to prevent it
appearing in shell history or scrollback:

```bash
# Write directly to .env without echoing the key to terminal
read -rsp "Paste FreeLLMAPI unified key (input hidden): " FKEY
printf '\nFREELLMAPI_UNIFIED_KEY=%s\n' "$FKEY" >> ~/.hermes/.env
chmod 600 ~/.hermes/.env
unset FKEY
echo "Key stored in ~/.hermes/.env"
```

---

### Phase 2 — Discover the Best Current Models

Repeat this phase whenever you want to refresh the MoA preset. Never hardcode
model names — always query the live catalog.

**2.1 Query the catalog with automatic fallback**

The script handles two cases: FreeLLMAPI exposes ranking metadata
(`intelligence_rank`, `provider`), or it does not.

```bash
# Source the key if not already set in environment
[ -z "$FREELLMAPI_UNIFIED_KEY" ] && source ~/.hermes/.env

curl -s \
  -H "Authorization: Bearer $FREELLMAPI_UNIFIED_KEY" \
  http://localhost:3001/v1/models \
  | python3 - << 'PYEOF'
import json, sys

raw = json.load(sys.stdin)
models = raw.get('data', raw.get('models', []))

# Filter to chat-capable models only
exclude = ['embed', 'whisper', 'tts', 'image', 'vision-only', 'moderation']
chat = [
    m for m in models
    if m.get('object', 'model') in ('model', 'chat')
    and not any(t in m.get('id', '').lower() for t in exclude)
]

has_rank = any(m.get('intelligence_rank') for m in chat)
has_provider = any(m.get('provider') for m in chat)

if has_rank:
    chat.sort(key=lambda m: m.get('intelligence_rank', 0), reverse=True)
    print('[INFO] Sorted by intelligence_rank from API response.')
else:
    print('[WARN] intelligence_rank not present in /v1/models response.')
    print('[ACTION] Use FreeLLMAPI dashboard → Models tab to rank manually.')
    print('[ACTION] Copy exact model IDs and paste into Phase 3 template.')
    chat.sort(key=lambda m: m.get('id', ''))

print(f'\n{"#":<5} {"Provider":<22} {"Model ID"}')
print('-' * 75)
for i, m in enumerate(chat[:20], 1):
    provider = m.get('provider', m.get('owned_by', '?'))
    rank = f" [rank:{m['intelligence_rank']}]" if has_rank and m.get('intelligence_rank') else ''
    print(f"{i:<5} {provider:<22} {m['id']}{rank}")

if not has_provider:
    print('\n[WARN] Provider family field not present. Group by model name prefix manually.')
PYEOF
```

**2.2 Select models for your MoA preset**

From the output, apply these rules:

- **Aggregator**: single highest-ranked chat-capable model overall
- **Reference 1**: highest-ranked model from a *different* provider family than the aggregator
- **Reference 2**: highest-ranked model from a *third* provider family

If the script flagged missing fields: open the FreeLLMAPI dashboard → Models
tab (sorted by intelligence rank by default in the UI) and identify families
manually.

Note the exact Model ID strings — you need them verbatim in Phase 3.

---

### Phase 3 — Configure Hermes

**3.1 Register FreeLLMAPI as a Hermes custom provider**

Use the interactive wizard. This is the canonical, version-safe method:

```bash
hermes model
```

When prompted:
- Select: **Custom endpoint (self-hosted / VLLM / etc.)**
- API base URL: `http://localhost:3001/v1`
- API key: paste your FreeLLMAPI unified key (input is hidden)
- Model name: paste any valid chat model ID from Phase 2

Hermes writes this to `~/.hermes/config.yaml` and validates the connection.

> **Why wizard-first?** Hermes docs and the official providers page document
> literal API key values in `config.yaml`; env-var interpolation (`${VAR}`)
> in YAML is not explicitly confirmed in current Hermes documentation. The
> wizard handles key storage safely and is version-stable.

Verify after registration:

```bash
hermes doctor
```

Expected: green checkmark on the custom provider. If it fails, confirm
FreeLLMAPI is still running: `curl http://localhost:3001/v1/models` should
return JSON.

**3.2 Configure the MoA preset (interactive)**

```bash
hermes moa configure free-sota
```

This opens an interactive prompt. Supply the model IDs from Phase 2.

Alternatively, append directly to `~/.hermes/config.yaml` — replace all
`<placeholder>` values with exact model IDs from Phase 2:

```yaml
# Append to ~/.hermes/config.yaml
# Do not add this block more than once. If moa: already exists, merge the presets key.
moa:
  default_preset: free-sota
  presets:
    free-sota:
      reference_models:
        - provider: custom
          model: "<reference-model-id-1-from-different-family>"
        - provider: custom
          model: "<reference-model-id-2-from-third-family>"
      aggregator:
        provider: custom
        model: "<highest-ranked-model-id>"
      reference_temperature: 0.7
      aggregator_temperature: 0.3
      max_tokens: 4096
      enabled: true
```

Verify the preset registered:

```bash
hermes moa list
# Expected: free-sota appears in the list
```

**3.3 Set free-sota as the default MoA preset (optional)**

```bash
hermes moa configure   # update default preset
```

---

### Phase 4 — Activate and Use

**4.1 Session-wide MoA mode**

```
hermes
/model free-sota --provider moa
```

All turns in this session: aggregator writes the response, references advised
silently first. Normal agent loop (tool calls, follow-up iterations, goal
mode) all work through MoA as documented.

**4.2 One-shot MoA without changing active model**

```
/moa <your hard task here>
```

Hermes runs one turn through the default MoA preset, then restores your
previous model.

**4.3 Verify calls route through FreeLLMAPI**

FreeLLMAPI dashboard → Requests tab after a `/moa` call: you should see N+1
entries (one per reference + one aggregator) clustered by timestamp.

---

### Phase 5 — Ongoing Maintenance

| Trigger | Action |
|---------|--------|
| Monthly | Re-run Phase 2, compare to current preset, update if top-3 changed |
| New provider added to FreeLLMAPI | Re-run Phase 2 |
| Reference model failing often | Check Requests tab; swap to next-ranked model from same family |
| Daily budget exhausted for a provider | Rotate that model out of references temporarily |
| intelligence_rank no longer in API response | Use dashboard UI for ranking; update Phase 2 parser |

```bash
hermes moa configure free-sota   # interactive update of the preset
```

---

## Pitfalls

**FreeLLMAPI not reachable**: confirm it's running (`npm run dev` in the
freellmapi directory). In persistent deployments use `npm run build && npm run start`
or a process manager (PM2, systemd).

**Model ID not found**: IDs change when providers rename models. Re-run
Phase 2 and update the preset.

**Reference call failures**: expected when a provider quota is exhausted.
FreeLLMAPI reroutes silently; Hermes logs the failure in reference context
and continues. Not a bug.

**All references from same family**: see Premortem. Override by selecting
top model from each of three distinct provider families.

**MoA increases latency**: expected — reference calls run before the
aggregator. Use `/model <aggregator-model-id> --provider custom` directly
for latency-sensitive tasks.

**Recursive MoA**: blocked by Hermes design. Never set the aggregator to
another MoA preset.

---

## Verification Checklist

1. `hermes doctor` — green on custom provider ✓
2. `hermes moa list` — `free-sota` shown ✓
3. `/moa what is 2+2` — completes without error ✓
4. FreeLLMAPI Requests tab — shows N+1 calls per MoA turn ✓
5. Aggregator model ID in Requests log matches Phase 3 config ✓

---

## Credits

- **FreeLLMAPI** — Tashfeen Ahmed and contributors
  https://github.com/tashfeenahmed/freellmapi
- **Hermes Agent** — Nous Research and contributors
  https://github.com/NousResearch/hermes-agent
- **Mixture-of-Agents research** — Junlin Wang, Jue Wang, Ben Athiwaratkun,
  Ce Zhang, James Zou (Together AI / Stanford / U. Chicago / Duke)
  https://arxiv.org/abs/2406.04692
- **Agent Skills specification** — agentskills.io contributors
  https://agentskills.io/specification

## References

- FreeLLMAPI repository: https://github.com/tashfeenahmed/freellmapi
- Hermes Agent documentation: https://hermes-agent.nousresearch.com/docs
- Hermes MoA feature docs: https://hermes-agent.nousresearch.com/docs/user-guide/features/mixture-of-agents
- Hermes providers docs: https://hermes-agent.nousresearch.com/docs/integrations/providers
- Hermes skills authoring: https://hermes-agent.nousresearch.com/docs/developer-guide/creating-skills
- MoA paper: https://arxiv.org/abs/2406.04692