Xiaothink Python Module
Copyright 2026 Shi Jingqi

This product includes proprietary and innovative technologies developed by Xiaothink:

1. T17-History Retrieval Architecture (T17历史检索架构)
   - GRU3 History-Aware Sequence Modeling (GRU3历史感知序列建模)
     * GRU-based causal sequence processing with hidden state propagation
     * Maintains complete history of input embeddings for content-based retrieval
     * Top-k history attention: retrieves the k most relevant historical positions
     * Gated fusion of GRU output and history retrieval output via scalar sigmoid gate
     * Supports prefill (full prompt) and decode (token-by-token) dual-mode inference
   - Fixed Tensor History Cache (固定张量历史缓存)
     * Pre-allocated contiguous GPU tensor buffer replaces dynamic Python list
     * Circular pointer-based write mechanism for O(1) history updates
     * Enables efficient batched matrix ops for history attention
   - Squared ReLU FFN (平方ReLU前馈网络)
     * FFN with squared ReLU activation: f(x) = relu(W_k x)²
     * Receptance-gated output: output = sigmoid(W_r x) · relu(W_k x)² · W_v
     * Stronger nonlinearity per parameter compared to standard GELU/SwiGLU
   - Training-Compatible History Retrieval (训练兼容的历史检索)
     * History retrieval is active in both training and inference modes (controlled by `use_h`)
     * Training mode: per-batch Python list cache, causal-masked intra-sequence retrieval
     * Inference mode: persistent fixed-size tensor cache, cross-step retrieval across decode calls
   - Gradient Checkpoint Compatible (梯度检查点兼容)
     * All custom layers (GRU3Block, history_transform, FFN, o_proj) support activation checkpointing
     * Enables training on consumer GPUs with limited VRAM

2. Dual Vision Scheme (双视觉方案)
   - Native Vision Encoder (原生视觉编码器)
   - img_zip Image Compression to Feature Tokens (图像压缩转特征token技术)

3. MoF Hybrid Architecture (MoF混合架构)
   - Narrow-Deep Transformer Expert (窄深Transformer专家)
     * Focuses on recent context (128 tokens)
     * Utilizes linear attention for reduced computation
   - Wide-Shallow RNN Expert (宽浅RNN专家)
     * Handles long context (2048 tokens)
     * 8-layer GRU networks for long-term dependencies
   - Intelligent Routing Mechanism (智能路由机制)
     * Dynamic task allocation to optimal expert
   - Thought Space (思维空间)
     * Global context reasoning module

4. img_zip Technology (图像压缩转特征技术)
   - Advanced image/video compression based on deep feature extraction
   - Patch-based processing for large images (80x80 blocks)
   - High compression ratios with intelligent feature representation

5. Low-Resource Training Framework (低资源训练框架)
   - Efficient model training on consumer-grade hardware
   - Optimized for small datasets
   - Support for 4096-token context windows
   - Gradient accumulation with micro-batch splitting
   - Real-time dynamic learning rate adjustment based on validation loss
   - Weight-decoupled optimizer (param_groups for weight decay vs. no weight decay)
   - Gradient clipping by global norm for training stability

Portions of this software may incorporate third-party open source components,
whose licenses are available in the source code or accompanying documentation.
