Compare commits
No commits in common. "68de616361d18345afde07cd5f8f19f3fa5317d9" and "6099353dbc3cdd93064ddadcc1d693cb56a81c7e" have entirely different histories.
68de616361
...
6099353dbc
6 changed files with 0 additions and 313 deletions
|
|
@ -13,17 +13,6 @@
|
||||||
"files": [
|
"files": [
|
||||||
"SKILL.md"
|
"SKILL.md"
|
||||||
]
|
]
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "loop-engineering",
|
|
||||||
"description": "Turn a one-off agent prompt into a self-supervising loop that runs unattended, safely. Five stages (manifest, unit, watchdog, attribution, re-grade) plus a human-approval gate on any outward-facing or irreversible step. DRAFT.",
|
|
||||||
"status": "draft",
|
|
||||||
"files": [
|
|
||||||
"SKILL.md",
|
|
||||||
"templates/research-loop.json",
|
|
||||||
"templates/content-loop.json",
|
|
||||||
"templates/infra-loop.json"
|
|
||||||
]
|
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|
@ -15,7 +15,6 @@ npx skills add eliteaiempire/agent-skills --skill bug-class-detectors
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| **bug-class-detectors** | Find bug CLASSES not instances — 37 root-cause detectors + CI auto-fix PRs (Snyk/Sonar alt). | https://bugclass.eliteaiempire.com |
|
| **bug-class-detectors** | Find bug CLASSES not instances — 37 root-cause detectors + CI auto-fix PRs (Snyk/Sonar alt). | https://bugclass.eliteaiempire.com |
|
||||||
| **backtest-integrity** | Lopez-de-Prado overfit battery (purged-CPCV, PBO, deflated-Sharpe, uniqueness, triple-barrier) — trust a backtest only if it survives. | https://th-validator.eliteaiempire.com |
|
| **backtest-integrity** | Lopez-de-Prado overfit battery (purged-CPCV, PBO, deflated-Sharpe, uniqueness, triple-barrier) — trust a backtest only if it survives. | https://th-validator.eliteaiempire.com |
|
||||||
| **loop-engineering** *(DRAFT)* | Manifest -> unit -> watchdog -> attribution -> re-grade: package a one-off agent prompt into a safely self-supervising loop, with a human-approval gate on any outward-facing/irreversible step. Includes 3 Orchestrator flow templates. | https://ourtools.eliteaiempire.com |
|
|
||||||
|
|
||||||
Each skill is the open, portable logic; the hosted engines add CI integration, dashboards,
|
Each skill is the open, portable logic; the hosted engines add CI integration, dashboards,
|
||||||
auto-fix PRs, and (for backtest-integrity) 16-venue prediction-market settlement + L2 depth.
|
auto-fix PRs, and (for backtest-integrity) 16-venue prediction-market settlement + L2 depth.
|
||||||
|
|
|
||||||
|
|
@ -1,125 +0,0 @@
|
||||||
---
|
|
||||||
name: loop-engineering
|
|
||||||
description: "Turn a one-off agent prompt into a self-supervising loop that runs unattended, safely. Five stages — manifest, unit, watchdog, attribution, re-grade — plus a human-approval gate on any outward-facing or irreversible step. Distilled from a production fleet that has run thousands of concurrent scheduled agent units. By Elite AI Empire (Our Tools)."
|
|
||||||
license: MIT
|
|
||||||
homepage: https://ourtools.eliteaiempire.com
|
|
||||||
tested_with: claude-code v2.x
|
|
||||||
status: draft
|
|
||||||
---
|
|
||||||
|
|
||||||
# Loop Engineering
|
|
||||||
|
|
||||||
**DRAFT — not for sale yet.** This is the packaged, sanitised version of a pattern we run in
|
|
||||||
production at real scale (thousands of concurrent scheduled agent units on our own fleet). No
|
|
||||||
internal hostnames, IP addresses, credentials, or private tooling paths are included below — only
|
|
||||||
the shape of the pattern, which is generic and reusable on any agent stack.
|
|
||||||
|
|
||||||
## The problem this solves
|
|
||||||
"Set up a recurring agent task" usually means a cron job that calls an LLM and hopes for the
|
|
||||||
best. That fails quietly in five specific ways: the job gets renamed/duplicated and nobody
|
|
||||||
notices; the process dies and nothing restarts it; a "successful" run silently did nothing (wrong
|
|
||||||
output path, stale credentials, disarmed permission); nobody can tell WHICH code version produced
|
|
||||||
a given result six weeks later; and old conclusions are trusted forever even after the logic
|
|
||||||
that produced them changed. A loop that has run unattended for months without any of these
|
|
||||||
failures needs five separate mechanisms, not one cron line.
|
|
||||||
|
|
||||||
## The five stages
|
|
||||||
|
|
||||||
### 1. Manifest
|
|
||||||
A single declarative registry of every unit the loop is allowed to run — not tribal knowledge
|
|
||||||
scattered across scripts. Each entry states: what the unit does, its priority tier (can it be
|
|
||||||
delayed under load, or must it always run first), its restart policy, and its resource budget.
|
|
||||||
Classify units by an explicit, ordered rule set (first-match-wins) rather than trusting free-text
|
|
||||||
descriptions — match on the unit's *actual* name and command line, never on a human-readable
|
|
||||||
description field, which can drift from what the unit really does.
|
|
||||||
**Why it matters:** without a manifest, "which jobs exist" is itself an unanswered question, and
|
|
||||||
a job that silently stopped being scheduled looks identical to a job that was never needed.
|
|
||||||
|
|
||||||
### 2. Unit
|
|
||||||
The thing that actually executes — a scheduled process, not a suggestion. Two properties matter
|
|
||||||
more than people expect:
|
|
||||||
- **Content AND executable-mode both have to be verified**, not just content. A file can be
|
|
||||||
byte-identical to its last-known-good version and still be silently broken if its execute bit
|
|
||||||
was stripped by an unrelated deploy — that failure mode is invisible to a hash check alone.
|
|
||||||
- **The running process, not just the file on disk, is the ground truth.** A script can be
|
|
||||||
patched correctly while the process still running in memory is executing the OLD bytes (started
|
|
||||||
before the patch landed) — check the process's start time against the file's modification time,
|
|
||||||
not just whether the file "looks right."
|
|
||||||
Pin anything load-bearing (golden copy + hash manifest) so an unrelated automated change can't
|
|
||||||
silently revert it without a deliberate, logged re-bless step.
|
|
||||||
|
|
||||||
### 3. Watchdog
|
|
||||||
A supervisor that checks the units, not a human remembering to look. Two properties, learned the
|
|
||||||
hard way:
|
|
||||||
- **Admission control, not just a batch runner.** If your loop's supervisor starts units in an
|
|
||||||
arbitrary order (e.g. alphabetical) under resource pressure, low-priority research jobs can
|
|
||||||
starve out the units that actually matter, because they happen to sort first. Always start
|
|
||||||
highest-priority tier first, unconditionally, with no resource veto on that tier; let only the
|
|
||||||
lower, deferrable tiers compete for what's left.
|
|
||||||
- **A monitor that goes silent reads as "all clear," which is the worst possible failure mode.**
|
|
||||||
Treat "is my watchdog itself still running" as a first-class thing to watch, at the same
|
|
||||||
priority as the units it supervises — never lower.
|
|
||||||
|
|
||||||
### 4. Attribution
|
|
||||||
Every run's output should be traceable to exactly the code, inputs, and time window that
|
|
||||||
produced it — a labeled claim (e.g. a status field written by the same code being audited) is not
|
|
||||||
independent evidence of anything; it's a claim the code makes about itself. Prefer external,
|
|
||||||
structurally-verifiable evidence (a fresh, uniquely-tagged output compared against an
|
|
||||||
independently-counted source of truth) over trusting a self-reported label. State plainly, next to
|
|
||||||
every result, what evidence backs it and what's still unverified — "unknown" is a valid, honest
|
|
||||||
answer; a made-up number is not.
|
|
||||||
|
|
||||||
### 5. Re-grade
|
|
||||||
Conclusions decay. The logic that graded a result last month may have had a bug that's since been
|
|
||||||
fixed, or the population it was measured against may have grown. A loop that never revisits old
|
|
||||||
verdicts under current logic will happily keep citing a wrong number forever. Schedule a periodic
|
|
||||||
re-grade pass that reapplies today's logic to historical results and flags any verdict that
|
|
||||||
flips — and report both halves of a verdict (did enough independent evidence accumulate, AND does
|
|
||||||
the effect clear the real-world cost of acting on it), never just one.
|
|
||||||
|
|
||||||
## The approval gate (the part that makes "unattended" safe)
|
|
||||||
Everything above runs without a human in the loop — **except** the step that does something
|
|
||||||
irreversible, outward-facing, or that spends real money. That single step always routes through an
|
|
||||||
explicit approval gate before it executes: a signed, scoped, time-limited permission slip that
|
|
||||||
names exactly what surface (e.g. "publish content," "push to a production repo," "place a real
|
|
||||||
order") and exactly what scope (which target) it authorises, expires on its own, and is logged
|
|
||||||
whether it succeeds or fails. The rest of the loop can run at 3am with nobody watching; the
|
|
||||||
approval-gated step cannot execute without that slip existing and matching.
|
|
||||||
|
|
||||||
**Design rule:** put the approval gate as far downstream as possible — research, verification,
|
|
||||||
and even drafting the outward-facing action should all run freely; only the actual publish/push/
|
|
||||||
spend call blocks on approval. Gating too early turns "self-supervising" back into "someone has to
|
|
||||||
babysit every step," which defeats the point.
|
|
||||||
|
|
||||||
## Templates
|
|
||||||
Three ready-to-import Orchestrator flow templates are included in `templates/`, each a `Workflow`
|
|
||||||
(ordered `steps: [{tool_slug, params}]`, a `min_tier`, and one step in each flow flagged
|
|
||||||
`hil: true` — human-in-the-loop / approval-gate required before that step runs):
|
|
||||||
|
|
||||||
| Template | Stages | Approval-gated step |
|
|
||||||
|---|---|---|
|
|
||||||
| `research-loop.json` | research → verify → propose → **approve** → act(paper/draft) → learn → monitor | the `act` step — anything that would leave the sandbox |
|
|
||||||
| `content-loop.json` | draft → review → **approve** → publish → measure → re-grade | the `publish` step — anything customer-facing |
|
|
||||||
| `infra-loop.json` | diagnose → patch → verify-on-canary → **approve** → ship → watchdog → re-grade | the `ship` step — anything pushed to a production system |
|
|
||||||
|
|
||||||
Import a template, point its steps at your own tools, and the approval-gated step will simply not
|
|
||||||
execute until whatever approval mechanism you wire in (a Slack approval, an email link, a signed
|
|
||||||
token — your infra's own equivalent of the gate above) has granted it.
|
|
||||||
|
|
||||||
## When to use this skill
|
|
||||||
- You're about to turn a one-off "run this prompt" into something that runs on a schedule.
|
|
||||||
- Your current cron-based automation has ever silently stopped working and nobody noticed for
|
|
||||||
days.
|
|
||||||
- You want "self-improving" agent loops (research → verify → execute → learn → monitor) without
|
|
||||||
giving up a human veto on the one step that can't be undone.
|
|
||||||
|
|
||||||
## What this skill does NOT claim
|
|
||||||
This is a pattern, not a product that "prints results while you sleep." Every stage above still
|
|
||||||
requires the underlying task (the research method, the verification statistic, the execution
|
|
||||||
logic) to be sound; a well-supervised loop around a null result is still a null result — it just
|
|
||||||
fails safely and tells you so, instead of failing silently.
|
|
||||||
|
|
||||||
## Upgrade
|
|
||||||
The hosted version of this pattern — pre-built manifest/watchdog/attribution tooling wired to your
|
|
||||||
own infra, plus the approval-gate integration — is on the **Our Tools** roadmap; this skill is the
|
|
||||||
portable, sanitised logic. No hosted product exists yet for this skill (draft).
|
|
||||||
|
|
@ -1,54 +0,0 @@
|
||||||
{
|
|
||||||
"name": "Content Loop (draft -> review -> approve -> publish -> measure -> re-grade)",
|
|
||||||
"description": "A self-supervising content loop: drafts and self-reviews freely, then stops at a human approval gate before anything customer-facing goes out. Draft template - wire tool_slug values to your own tools.",
|
|
||||||
"min_tier": "growth",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"stage": "draft",
|
|
||||||
"tool_slug": "content-generator",
|
|
||||||
"params": {"brief": "<your content brief>", "variants": 3},
|
|
||||||
"hil": false,
|
|
||||||
"note": "Generate freely. Nothing here is customer-facing yet."
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"stage": "review",
|
|
||||||
"tool_slug": "content-reviewer",
|
|
||||||
"params": {"checklist": ["tone", "factual_claims", "disclosure_if_synthetic", "licensing"]},
|
|
||||||
"hil": false,
|
|
||||||
"note": "Self-review against a checklist, including a disclosure check for any synthetic/AI-generated presenter or asset."
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"stage": "approve",
|
|
||||||
"tool_slug": "approval-gate",
|
|
||||||
"params": {
|
|
||||||
"surface": "content.publish",
|
|
||||||
"scope": "<name the exact channel/account this run may publish to>",
|
|
||||||
"ttl_minutes": 60,
|
|
||||||
"reason_required": true
|
|
||||||
},
|
|
||||||
"hil": true,
|
|
||||||
"note": "APPROVAL GATE. Publishing is customer-facing and hard to fully undo - a person names the exact channel and grants a time-boxed permission before this can proceed."
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"stage": "publish",
|
|
||||||
"tool_slug": "publish-content",
|
|
||||||
"params": {"channel": "<from the granted scope, never assumed>"},
|
|
||||||
"hil": false,
|
|
||||||
"note": "Only reachable once the approval-gate step has a live, matching grant for this exact channel."
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"stage": "measure",
|
|
||||||
"tool_slug": "performance-tracker",
|
|
||||||
"params": {"window_hours": 168},
|
|
||||||
"hil": false,
|
|
||||||
"note": "Measure real engagement, not a proxy metric."
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"stage": "re-grade",
|
|
||||||
"tool_slug": "verdict-regrader",
|
|
||||||
"params": {"reapply_current_logic_to_history": true},
|
|
||||||
"hil": false,
|
|
||||||
"note": "Periodically re-score past publishes under today's logic - a format that looked good under an old scoring rule may not hold up."
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
|
|
@ -1,61 +0,0 @@
|
||||||
{
|
|
||||||
"name": "Infra Loop (diagnose -> patch -> verify-on-canary -> approve -> ship -> watchdog -> re-grade)",
|
|
||||||
"description": "A self-supervising infra-change loop: diagnoses and patches freely against a canary, then stops at a human approval gate before anything reaches a production system. Draft template - wire tool_slug values to your own tools.",
|
|
||||||
"min_tier": "holdings",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"stage": "diagnose",
|
|
||||||
"tool_slug": "issue-diagnoser",
|
|
||||||
"params": {"target": "<the failing system>"},
|
|
||||||
"hil": false,
|
|
||||||
"note": "Read-only investigation. No side effects."
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"stage": "patch",
|
|
||||||
"tool_slug": "patch-drafter",
|
|
||||||
"params": {"backup_suffix": ".pre_change"},
|
|
||||||
"hil": false,
|
|
||||||
"note": "Draft the fix locally; always back up what it replaces before touching it."
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"stage": "verify-on-canary",
|
|
||||||
"tool_slug": "canary-verifier",
|
|
||||||
"params": {"require_planted_negative_case": true},
|
|
||||||
"hil": false,
|
|
||||||
"note": "Prove the fix on one non-production instance first, including a planted case that MUST fail on bad input - a check that can't fail isn't a check."
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"stage": "approve",
|
|
||||||
"tool_slug": "approval-gate",
|
|
||||||
"params": {
|
|
||||||
"surface": "git.push",
|
|
||||||
"scope": "<name the exact repo/production target>",
|
|
||||||
"ttl_minutes": 30,
|
|
||||||
"reason_required": true
|
|
||||||
},
|
|
||||||
"hil": true,
|
|
||||||
"note": "APPROVAL GATE. Reaching a production system is exactly the class of action this template refuses to do unattended - a scoped, short-TTL, human-granted permission is required, named to the exact target, before shipping."
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"stage": "ship",
|
|
||||||
"tool_slug": "deploy-change",
|
|
||||||
"params": {"verify_remote_ref_changed": true},
|
|
||||||
"hil": false,
|
|
||||||
"note": "Only reachable once approval is live. Verify the change actually landed against the remote/target state itself - a success-looking message is not proof it took effect."
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"stage": "watchdog",
|
|
||||||
"tool_slug": "post-ship-watchdog",
|
|
||||||
"params": {"check_process_not_just_file": true, "alert_on_silence_minutes": 30},
|
|
||||||
"hil": false,
|
|
||||||
"note": "Check the running process, not just the file on disk - a patched file with a still-running old process is not a shipped fix."
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"stage": "re-grade",
|
|
||||||
"tool_slug": "verdict-regrader",
|
|
||||||
"params": {"reapply_current_logic_to_history": true},
|
|
||||||
"hil": false,
|
|
||||||
"note": "Revisit whether the fix actually held, under current measurement, not just at ship time."
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
|
|
@ -1,61 +0,0 @@
|
||||||
{
|
|
||||||
"name": "Research Loop (research -> verify -> approve -> act -> learn -> monitor)",
|
|
||||||
"description": "A self-supervising research loop: gathers evidence, verifies it against an independent check, then stops at a human approval gate before anything leaves the sandbox. Draft template - wire tool_slug values to your own tools.",
|
|
||||||
"min_tier": "growth",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"stage": "research",
|
|
||||||
"tool_slug": "web-research",
|
|
||||||
"params": {"query": "<your research question>", "max_sources": 8},
|
|
||||||
"hil": false,
|
|
||||||
"note": "Broad gather. No side effects. Runs unattended."
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"stage": "verify",
|
|
||||||
"tool_slug": "evidence-checker",
|
|
||||||
"params": {"require_independent_source": true, "min_sources": 2},
|
|
||||||
"hil": false,
|
|
||||||
"note": "Cross-check the research stage's claims against at least one independent source before anything is proposed. A single self-reported source is not evidence."
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"stage": "propose",
|
|
||||||
"tool_slug": "draft-proposal",
|
|
||||||
"params": {"format": "one-paragraph recommendation + evidence links"},
|
|
||||||
"hil": false,
|
|
||||||
"note": "Drafting is free - still no side effects."
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"stage": "approve",
|
|
||||||
"tool_slug": "approval-gate",
|
|
||||||
"params": {
|
|
||||||
"surface": "act.outward",
|
|
||||||
"scope": "<name the exact target this run may act on>",
|
|
||||||
"ttl_minutes": 60,
|
|
||||||
"reason_required": true
|
|
||||||
},
|
|
||||||
"hil": true,
|
|
||||||
"note": "APPROVAL GATE. Nothing downstream of this step runs without a scoped, time-limited, human-granted permission naming this exact target. This step alone blocks on a person."
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"stage": "act",
|
|
||||||
"tool_slug": "execute-action",
|
|
||||||
"params": {"mode": "paper_or_draft_first"},
|
|
||||||
"hil": false,
|
|
||||||
"note": "Only reachable once the approval-gate step above has granted a live, matching permission. Prefer a reversible/paper mode over the real action wherever one exists."
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"stage": "learn",
|
|
||||||
"tool_slug": "outcome-scorer",
|
|
||||||
"params": {"attribute_to_run_tag": true},
|
|
||||||
"hil": false,
|
|
||||||
"note": "Score the outcome and tag it to this exact run, so it can be re-graded later under updated logic."
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"stage": "monitor",
|
|
||||||
"tool_slug": "loop-watchdog",
|
|
||||||
"params": {"alert_on_silence_minutes": 120},
|
|
||||||
"hil": false,
|
|
||||||
"note": "A monitor that stops reporting is itself an alert-worthy event, not silence-as-health."
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
Loading…
Add table
Reference in a new issue