# v0.24.0 Community Posts

*Output Quality Quantification (ADR-036)*

---

## Twitter/X Thread

**Tweet 1 (Main)**:
> LLM Council v0.24.0: Output Quality Quantification
>
> How do you know if a multi-model answer is reliable?
>
> Now every response includes 3 metrics:
> - CSS: Did reviewers agree?
> - DDI: Was deliberation thorough?
> - SAS: Is synthesis grounded?
>
> #LLM #AIEval #OpenSource

**Tweet 2**:
> The problem: Multi-model systems promise better answers through deliberation.
>
> But "trust the process" isn't enough. You need evidence.
>
> Was that unanimous agreement genuine consensus, or an echo chamber?
> Did the chairman just copy the winner, or synthesize multiple perspectives?

**Tweet 3**:
> Consensus Strength Score (CSS)
>
> Measures Stage 2 peer review agreement.
>
> - Winner Margin (40%)
> - Ordering Clarity (40%)
> - Non-Tie Factor (20%)
>
> CSS 0.85+ = strong consensus
> CSS <0.50 = significant disagreement

**Tweet 4**:
> Deliberation Depth Index (DDI)
>
> Measures thoroughness of deliberation.
>
> - Response Diversity (35%)
> - Review Coverage (35%)
> - Critique Richness (30%)
>
> DDI 0.70+ = deep deliberation
> DDI <0.40 = shallow (warning triggered)

**Tweet 5**:
> Synthesis Attribution Score (SAS)
>
> Measures grounding of final answer.
>
> - winner_alignment: How well synthesis matches top responses
> - hallucination_risk: 1 - max_source_alignment
> - grounded: bool (threshold: 0.6)
>
> Low SAS = chairman may have invented claims

**Tweet 6**:
> Quick decision guide:
>
> | CSS | SAS | Action |
> |-----|-----|--------|
> | High | High | Trust it |
> | High | Low | Check for hallucination |
> | Low | High | Healthy debate, surface dissent |
> | Low | Low | Retry with different models |

**Tweet 7**:
> Performance: <30ms overhead total (CSS <5ms, DDI <10ms, SAS <15ms)
>
> All calculations local, no extra API calls.
>
> pip install --upgrade llm-council-core
>
> Blog: [link]
> GitHub: github.com/amiable-dev/llm-council

---

## Reddit r/LocalLLaMA

**Title**: LLM Council v0.24.0: Quality metrics for multi-model deliberation

**Body**:

> **TL;DR**: New metrics tell you if your multi-model council reached genuine consensus or just rubber-stamped an answer. <30ms overhead, 44 tests, works offline with Jaccard similarity.
>
> ---
>
> Shipped a feature I've been thinking about for months: quality metrics that exploit multi-model deliberation to quantify answer reliability.
>
> **The Problem**
>
> When you get an answer from a multi-model system, how do you know if it's reliable? Did the models carefully debate, or just agree on the first thing they saw?
>
> **The Solution: Three Metrics**
>
> Every council response now includes:
>
> ```json
> {
>   "quality_metrics": {
>     "consensus_strength": 0.85,
>     "deliberation_depth": 0.72,
>     "synthesis_attribution": {
>       "winner_alignment": 0.78,
>       "hallucination_risk": 0.18,
>       "grounded": true
>     }
>   }
> }
> ```
>
> **1. Consensus Strength Score (CSS)**
>
> How much did Stage 2 reviewers agree on rankings?
>
> - 0.85+ = Strong consensus, high confidence
> - 0.50-0.84 = Moderate, consider minority views
> - <0.50 = Significant disagreement, use debate mode
>
> **2. Deliberation Depth Index (DDI)**
>
> How thorough was the deliberation?
>
> - Response diversity (35%): Were Stage 1 responses different?
> - Review coverage (35%): Did all models complete Stage 2?
> - Critique richness (30%): Were reviews substantive?
>
> **3. Synthesis Attribution Score (SAS)**
>
> Is the final answer grounded in the reviewed responses?
>
> - `grounded=True` if max alignment >= 0.6
> - `hallucination_risk` flags when chairman invents claims
>
> **Performance**
>
> <30ms total overhead. All calculations are local, no extra API calls.
>
> **Limitations (being upfront)**
>
> - Uses Jaccard similarity (lexical, not semantic) - Tier 2 will add embeddings
> - CSS can be high when models are confidently wrong together
> - DDI can be gamed by verbose but shallow responses
>
> These are indicators, not guarantees.
>
> **Decision Guide**
>
> | CSS | SAS | What to do |
> |-----|-----|------------|
> | High | High | Trust the answer |
> | High | Low | Check for hallucination |
> | Low | High | Surface minority views |
> | Low | Low | Retry |
>
> `pip install --upgrade llm-council-core`
>
> GitHub: https://github.com/amiable-dev/llm-council

---

## Reddit r/MachineLearning

**Title**: [P] Quantifying LLM output quality via multi-model deliberation

**Body**:

> We added quality metrics to LLM Council that exploit multi-model architectures to signal answer reliability.
>
> **Motivation**
>
> Single-model systems can only introspect their own outputs. Multi-model systems have additional signals:
>
> - Do the reviewers agree on which response is best?
> - How diverse were the initial responses?
> - Is the synthesis grounded in the reviewed content?
>
> **Metrics**
>
> **Consensus Strength Score (CSS)**
> ```
> CSS = (winner_margin * 0.4) + (ordering_clarity * 0.4) + (non_tie_factor * 0.2)
> ```
>
> Derived from aggregate rankings after Stage 2 peer review. High CSS = clear winner, low variance in rankings.
>
> **Deliberation Depth Index (DDI)**
> ```
> DDI = (diversity * 0.35) + (coverage * 0.35) + (richness * 0.30)
> ```
>
> Diversity = Jaccard dissimilarity of Stage 1 responses. Coverage = % models completing Stage 2. Richness = review length normalization.
>
> **Synthesis Attribution Score (SAS)**
> ```
> winner_alignment = jaccard(synthesis, top_ranked_responses)
> max_source_alignment = max(jaccard(synthesis, each_response))
> hallucination_risk = 1 - max_source_alignment
> grounded = max_source_alignment >= 0.6
> ```
>
> **Implementation Notes**
>
> - Phase 1 uses synchronous Jaccard similarity (no external deps, works offline)
> - Future phases will support configurable embedding providers
> - <30ms overhead total, calculations happen post-synthesis
>
> **Evaluation**
>
> Thresholds calibrated on ~200 council sessions with human quality ratings. They're starting points - tunable per domain.
>
> 44 unit tests covering edge cases.
>
> **Known Limitations**
>
> - Jaccard measures lexical, not semantic overlap
> - CSS can be high for echo chambers (confident but wrong)
> - These are indicators, not formal guarantees
>
> Open source: https://github.com/amiable-dev/llm-council
>
> Would appreciate feedback from anyone working on multi-model evaluation.

---

## Hacker News

**Title**: Show HN: Quality metrics for multi-model LLM deliberation

**Text**:

> GitHub: https://github.com/amiable-dev/llm-council
>
> When multiple LLMs deliberate on a question (generate -> peer review -> synthesize), you get signals that single-model systems can't provide.
>
> LLM Council v0.24.0 exposes three metrics that quantify answer reliability:
>
> **Consensus Strength Score (CSS)**
>
> How much did Stage 2 reviewers agree? Computed from aggregate rankings:
>
> - Winner Margin (40%): How far ahead is #1?
> - Ordering Clarity (40%): Consistent evaluations?
> - Non-Tie Factor (20%): Unique positions?
>
> CSS 0.85+ = strong consensus. CSS <0.50 = significant disagreement.
>
> **Deliberation Depth Index (DDI)**
>
> How thorough was deliberation?
>
> - Response Diversity (35%): Did models approach differently?
> - Review Coverage (35%): Did all models complete reviews?
> - Critique Richness (30%): Were reviews substantive?
>
> **Synthesis Attribution Score (SAS)**
>
> Is the synthesis grounded in sources?
>
> - Uses Jaccard similarity between synthesis and reviewed responses
> - `hallucination_risk` when chairman invents claims
> - `grounded=True` when max alignment >= 0.6
>
> **Decision Guide**
>
> High CSS + High SAS = trust it
> High CSS + Low SAS = check for hallucination
> Low CSS + High SAS = healthy debate, surface dissent
> Low CSS + Low SAS = retry
>
> **Overhead**: <30ms total, all local, no extra API calls.
>
> **Limitations**: Jaccard is lexical (not semantic), CSS can be high for echo chambers. These are indicators, not guarantees.
>
> Open source. 44 tests.
>
> GitHub: https://github.com/amiable-dev/llm-council
>
> Feedback welcome from anyone doing multi-model evaluation or automated code review.

---

## Cross-posting Notes

1. **Wait 24-48 hours between posts** to different platforms
2. **Engage with comments** - answer questions, take feedback
3. **HN timing**: Post between 6-9am PT for best visibility
4. **Reddit timing**: Post during US business hours

## Platform-specific Notes

| Platform | Tone | Focus |
|----------|------|-------|
| Twitter/X | Concise, visual tables | Quick overview, upgrade command |
| r/LocalLLaMA | Practical, friendly | How it helps their workflow, limitations |
| r/MachineLearning | Academic, technical | Formulas, methodology, evaluation |
| HN | Technical, concise | Problem + solution, decision guide |
