Metadata-Version: 2.4
Name: speculators
Version: 0.1.0a9
Summary: A unified library for creating, representing, and storing speculative decoding algorithms for LLM serving such as in vLLM.
Author: Red Hat
License-Expression: Apache-2.0
Project-URL: homepage, https://github.com/neuralmagic/speculators
Project-URL: source, https://github.com/neuralmagic/speculators
Project-URL: issues, https://github.com/neuralmagic/speculators/issues
Keywords: speculative decoding,transformers,llm,inference,vllm,machine learning,deep learning,nlp,language models,serving,standard format,productization,decoding algorithms,inference server
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click
Requires-Dist: datasets
Requires-Dist: httpx[http2]
Requires-Dist: huggingface-hub
Requires-Dist: loguru
Requires-Dist: numpy
Requires-Dist: protobuf
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pydantic-settings>=2.0.0
Requires-Dist: pyyaml>=6.0.0
Requires-Dist: safetensors
Requires-Dist: torch
Requires-Dist: transformers
Provides-Extra: dev
Requires-Dist: build>=1.0.0; extra == "dev"
Requires-Dist: setuptools>=61.0; extra == "dev"
Requires-Dist: setuptools-git-versioning<3,>=2.0; extra == "dev"
Requires-Dist: pre-commit~=3.5.0; extra == "dev"
Requires-Dist: sphinx~=7.1.2; extra == "dev"
Requires-Dist: tox~=4.16.0; extra == "dev"
Requires-Dist: pytest~=8.2.2; extra == "dev"
Requires-Dist: pytest-asyncio~=0.23.8; extra == "dev"
Requires-Dist: pytest-cov~=5.0.0; extra == "dev"
Requires-Dist: pytest-mock~=3.14.0; extra == "dev"
Requires-Dist: pytest-rerunfailures~=14.0; extra == "dev"
Requires-Dist: respx~=0.22.0; extra == "dev"
Requires-Dist: mypy~=1.15.0; extra == "dev"
Requires-Dist: ruff~=0.11.7; extra == "dev"
Requires-Dist: mdformat~=0.7.17; extra == "dev"
Requires-Dist: mdformat-footnote~=0.1.1; extra == "dev"
Requires-Dist: mdformat-frontmatter~=2.0.8; extra == "dev"
Requires-Dist: mdformat-gfm~=0.3.6; extra == "dev"
Requires-Dist: types-click~=7.1.8; extra == "dev"
Requires-Dist: types-PyYAML~=6.0.1; extra == "dev"
Requires-Dist: types-requests~=2.32.0; extra == "dev"
Requires-Dist: types-toml; extra == "dev"
Requires-Dist: mkdocs-linkcheck~=1.0.6; extra == "dev"
Dynamic: license-file

<p align="center">
  <img alt="Speculators Logo" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAfQAAACWCAYAAAAonXpvAAAAAXNSR0IArs4c6QAAHzdJREFUeF7tnQfYHkW5hp+ACAKCQhQFpCkgoYMQhNBFQAEJSBMEQSA0KZdiOR4R9eixICC9KSgQKRIBIYAgLaHXUEITpEsvgihIOedhs9c3O/9+3+7X8v+Z/57rykXItzs7c8/sPjPvvPPOiKUOfecdkSAAAQhAAAIQmKEJjEDQZ+j2o/AQgAAEIACBdwkg6HQECEAAAhCAQAIEEPQEGpEqQAACEIAABBB0+gAEIAABCEAgAQIIegKNSBUgAAEIQAACCDp9AAIQgAAEIJAAAQQ9gUakChCAAAQgAAEEnT4AAQhAAAIQSIAAgp5AI1IFCEAAAhCAAIJOH4AABCAAAQgkQABBT6ARqQIEIAABCEAAQacPQAACEIAABBIggKAn0IhUAQIQgAAEIICg0wcgAAEIQAACCRBA0BNoRKoAAQhAAAIQQNDpAxCAAAQgAIEECCDoCTQiVYAABCAAAQgg6PQBCEAAAhCAQAIEEPQEGpEqQAACEIAABBB0+gAEIAABCEAgAQIIegKNSBUgAAEIQAACCDp9AAIQgAAEIJAAAQQ9gUakChCAAAQgAAEEnT4AAQhAAAIQSIAAgp5AI1IFCEAAAhCAAIJOH4AABCAAAQgkQABBT6ARqQIEIAABCEAAQacPQAACEIAABBIggKAn0IhUAQIQgAAEIICg0wcgAAEIQAACCRBA0BNoRKoAAQhAAAIQQNDpAxCAAAQgAIEECCDoCTQiVYAABCAAAQgg6PQBCEAAAhCAQAIEEPQEGpEqQAACEIAABBB0+gAEIAABCEAgAQIIegKNSBUgAAEIQAACCDp9AAIQgAAEIJAAAQQ9gUakChCAAAQgAAEEnT4AAQhAAAIQSIAAgp5AI1IFCEAAAhCAAIJOH4AABCAAAQgkQABBT6ARqQIEIAABCEAAQacPQAACEIAABBIggKAn0IhUAQIQgAAEIICg0wcgAAEIQAACCRBA0BNoRKoAAQhAAAIQQNDpAxCAAAQgAIEECCDoCTQiVYAABCAAAQgg6PQBCEAAAhCAQAIEEPQEGpEqQAACEIAABBB0+gAEIAABCEAgAQIIegKNSBUgAAEIQAACCDp9AAIQgAAEIJAAAQQ9gUakChCAAAQgAAEEnT4AAQhAAAIQSIAAgp5AI1IFCEAAAhCAAII+RPvA4vNKXxglLT5S+tjc0sg5pFlnlv71H+m516Qn/yE98Lx0yi3Ss/8copUYRsX6+LzSPO9rVPiWJ6S33xlGAGbAqn5gtuz9ytOrb0j3PDMDVoQiQ2AaAQR9iHWFuWaTDtlYGrNIvYJZM256XPrR5dKDz9e7h6t6T+DmfaTZZ2nku9sE6ZpHev8ccuwdgZ9uJG22VCM/D8CWObx3+ZMTBKY3AQR9ehNv8bztV5C+vbY080ydFer8e6RvX9zZvdzVHYFb9pHeFwj67hOkydNR0NdeVJrzvVkd3nxHuuT+7uozHO7+2UbSpgj6cGjqYVNHBH2INPVyH5F+v500osvyHDZZOvGmLjPh9rYJDLag331Ase+se6L09KttV2NY3YCgD6vmHhaVRdCHQDPP+h5p8jhpjmkzrLxI/3lLOvMO6frHpLufll55XVpxfmmFj0rLzy+tsfDAAYBN8HueK139tyFQsWFUBAR9xmtsBH3GazNK3JoAgj4Eesh315Vsbg/TrU9KXz1Hev3N5gX0rP7ELaT3z1q85vnXpDWPHwIVG0ZFQNBnvMZG0Ge8NqPECPqQ7wMX7CQtNk+jmC//WxpznPRWDS/p98wkTRonzT1b436ce6Z/kyPo0595t09E0LslyP1DjQAz9CHQIlP2lWaZuVGQo6+Tjr6+fsG2WFr6n88Wrx97qnTfc/Xz4MruCCDo3fEbjLsR9MGgzjP7SQBB7yfdGnnPN6d0xW7FC70GflUba+Azj5Du2L+4nn7oZOmkyDlu81GSt8U5vfq6NOHu7O8LzCX91zrSqPmymb7zs0PV1GekKx+Szp1aoyLRJbY4rLOYtMqC0pIjpTlnld5+W3rhX9L1j2YDFi8NtJNcrh1XklZbSFpipPShObK77Wtgq8bDL0k/u6r5XmJfv/uqjSc+/rL021urS7DXatIHp+0xf/Nt6ZeTJP83TJ0I+uiPSeNWlRaYO9vDbl8K18X7oV94Tbr5CenI66R//Lu8jF6mWeSD2W/xks0F92ZMnLz74c6nyvPw/R4QrjS/tOg8DUuP+8ejL0tTn5aOvaHawc75hGW47K/SDY9lz3Sb+RkLzp31rf+9UjrrzoHl2XUV6TOfkD76fsl7xL3bw34jT/xDuvdZ6YoHpb88WN1eda+YXoLeK8Zxvdxf/I55h4P9at47c9Z3HntZOu6GzO/GyTESFpwr+/u/32y0SytObi/76Cz14aw9POF46+2sH/z1+awdzrmrLmmum14EEPTpRbrJc+adPTOZh+k3N0uHTGqvYKdu3fi4+04L5hlTinnEntCjDpN+sqFkoW+VHnpB2uFM6aUmwhLeO9MI6ZefkzZcorr8Nz4u7XNe9hGqSgt9QDp9G8m8qpIHInufN1CEvOfYe4/z5I/bSkdW5Sbdtb/keuWpzIO8HUH/7OLS99arVxevuliMv3J29jEO0417N7aqtarFH+6SDrp04BU7ryx9Y63qnRVewjnm+uxPs7TbKtIBYxq/TnpYOvAiafw2xeUkX+EBwpHXNq7deEnpoPWKy0bNnuNBgvf4xwOq6lYceMX0EPReMg5rYBE/YtOiZS+uod/bL50pnbuD9JH3N35d7lfN+fn98nvm960qPfJS1hYeGJOGBgEEfQi0Q2xy94d781OlR1/qbeFiQb/8QWm9j9d7xhtvZSLZKliKZ/r+GHx4znp5+irnu/2ZjdlE2Z1jl5Z++Jn29uc/86q03knFaG1DQdA9i7UTZLvJkQE/d0rGK0+dCrpncidsIa26YHul8Ex5m/GZlSVOsaBf96j0iXkbVpTw+lDQN/mk9PON2yuHLQ/bnSE9/GJ798VX91PQ+8E4L/+31pZ2Wqle3c3KA7LcwuS7mgm6vwWHfb71ICF+qvP+1TVsla3XGv2/CkHvP+PKJ8ROcb7Bptcz7shMZy+WfEArMy25IBb0+BKHlfUfb5+zOS9O9rhf5ejy0b1NxlfuLtlJL0421b3+VjGSWniNRWrjk6W/vzLw3pUXkGx9iJNnri++ln2svEwQ+iDk19rsu++fGncOtqA7kpxFOJztu3Sug9vYSxC+xksDZfw9ANvn/EZ9rt5dmnfaskNZ/ILcp9LhgX9xdeO+c7bPTKllH2eHEZ5lJumDs5fP3O96Wtp6/MB7Y0Fv1T9zQbcJ/uKdB/LwEoPDG5uTQx67P8b1s1l5w9908hY07umnoPeDsUv+nXWkL69YXm/3o7hvlV1ZJujLfkQ6c7vyfF/7T7YE5IG6ByplaavxrQfl3bUUd9clgKDXJdXH6/Zfo7i2Gz/quX9K3sY28T7JH/VOzY3NBN2zvwMuLK6zrrWo5A9e6D3vcnnd7Hsl5tuTvyh5TThMniWP+2PDOS9f8/NsO95qN/lhafc/DoR82Vel+aet/+W/2pz7rYuKSwBbLZuZseMBxehjsnVYp8EW9LLtibZ42PIRzrxdVi9Z/GLjYn38UR3TZDti3cAyn15I+vWWRc7uT3udJ7kN8mSOGy6eLVHEkQsPnChdeF8xj1aCbqHxbPq+ZzOh/v2U7P8PXl/aerlGPr7ua+dLVzxUzNtnGRw/trik5CvcX8Iyt/uK9kvQ+8XY/i/XjCu2hwdtF96b+SV4UDjbe6S1F5O+v37mh1BX0C/aWVo4MrPb12W/Cxrvj/NyW/xwg4HvugeCa5/Qbgtwfa8JIOi9JtphfmWCWJZVPjOd+mwWPGbiveUm0LJ7ywTd61+eHZdtkbPo/nmXgVvi/OKGDm1Lzyed/aXiE699RNp1QjkMm/8u+krDQc9XWVRWOKJoIndYTn90w+RBzTcmlufrAYU5hukHf8mC8zgNtqDHszavB+/8h+YdZv2PS0du1vjdbb/c4eVtVVfQ7YBpR8w82RK07RnNHQnt1Gim4czPS0KrHVMchDQTdPeTL59Vbh6Py3L67dKPr2jO49avZYKVp2aDy7qvYL8EvV+Mj95MWjdYInN/2Pf8ckdBD8hs2Vr+owNpxDN0+3QcvknxOjsg7h1Yg+JczthOchyMMNmaeETgG1G3HbiudwQQ9N6x7DqnP2wvjSoxhVZl7I/mxfdnL1M+Gy27p0zQvS7bai1yzMLZemuY4g+vzaahE41NdP7gt7Ik2EQ/aY+iKfWHlxcd+a7bsziYcL6jj269P99mQ5sP8+QZsB13hoKgOxrgPIFTn4XOp7K1SnGbrX5suXNiHUG3L8KPo+2NMfOyssQDIV9jK03o5Vwm6F6+sYWkWT+4fd+iCdc7DrxLoVmKD1PxDHKXc6rejua/90PQ+8XYg7DLdyu+L97J8M2LmtfP5vGb9h64HBUL+jV7FNfYveSx2rGtuXqAd5WXfIL+bKdZ90/S4BFA0AePfemT7Zhy4FqZQLYb191r1UddLx1/Q3ml4o++TaBjT6sGcP2exdl0+CH1NqQ79y/mEc6KW+Uei69Pi9v0d9kdnol5RhYmz7Sdd6u07fKZx3SeQlPgYM/Q915NWvJDWcncVl7maJVsIblhr+IV3Qi6owp6K1KePDtf8ch6x7zG4nvW/1s9Dg7aokzQY9GP6+q6hUsvXnaw012z+AlesrHjZdi2rQawVT27H4LeL8ZbLysd/JlGjTw7X/nIgTsf4jqXLfOEgm6mt0XvWd1dNvutLo0b3XgiAa2qelz/f0fQ+8+4oyf4RbMHsP8sM9/AOO+tMm0mqLGgn3CjdPg11cWLTcWhQ1KZM024bt0qdwvc3p9uXPHPNzKnO6cyZziLfdURsTY12rs6T15XzA8pGWxBryadXeG1UgvvgWsWtxv5t24EPV4n/fMD0v4X1CtVvCRknw5vZcxTLOgWnKUPa533b7bMYgqEyfd5oHnyLdnacD/PlO+HoPeLsQepHqzmyf4p65xY3XZeR792z+J1oaDH75n5r3KUZGtYVSrL2xEuy3ZBVOXF770hgKD3hmPfc/HRmA664UASfglb7cf2S2lz7q2ROTcW9LqeqTbT2pSYJ7/snzoq+z973NrzNkz3PFMPh71mw3qEI3xvy/H2nDx5Nrn8EfXybXbVUBJ0DzzWXCT7s9i8WfAOL0P4CNZWnsrdCLrNr+EBQD+5UjrttnpM7Qj1xWWaC0os6HXMr+suJh39hebPdz9++pUswM75U3t/HG0/BL1fjOMBVdUad0g1fu9DQfc+eVsE82SLhwfkdVMco8EDRA8USYNDAEEfHO5dP9WzuG2Xy/ajhntM84y9Z3iDXxcfE7/Yqx5dL6hLmbkvn33VCUxTt7LhrC4eRNRZ16t6zlARdEer23N0+da0qjp0I+hx+9shL4/mVvXc2Jpi87idGPMUC3pZ/yt7Rty3WpXDyxRT/i4des3AwWpV+ct+74eg94uxnVO9zS9PdZaf8mvjoEehoMd+Ce1uB4wdFeta/TppL+6pJoCgVzMa8lfEo2wX2I5IfnHDVBYprk7lvIXtuM0bV4bCO37bLOxkr5Kj1zmdtrW00gKNXJ96JQsU003qVNDrOJzViRTnELi/3aroj9CqPnYq84w9TJ0Kuve224kpTO2cmR6vl8ZR9joVdJfHuxP+e90sRGnd1Gq3Q908ei3o/WQcBxGKo+21qnMrQY/fXw+YHLSnbop9K466rnVEwbr5cl1nBBD0zrj15C47lF361eJe4x3P7iwClredePtJmBx4wyPuPNX1mI4rF5vVPVNadtpg4djNs1jSYeo0FKTX0HMnvXh9tRdHwnYi6GVOQ52EfrXHsQPB5LH0Q14eIDmil9f6n3g5m4V6S6Kdw+q2WdWgo6weW57efLta3Acczc3+HHmKB1jdCHqepx3etls+25q1kOO+lwQpCstV13mr2cvaa0HvJ+MrdytGYKza4hfWOZ5FhzP02InvgeelL0xzTK3zkYv7nWMqxHEE6uTDNb0hgKD3hmPHucRrUHW2EZU9rCyYRewcF798X58oXRQFCCnLO3bICdfQ45lbbIrtFEwcEasX+XYi6D5kxpH8wtSJoPvwmx2iCF+egXublmOtN3P+6pWgu/xxiOHv/ln647QDeqra6bRtsgNc8nTbk1nI3jz1QtDjMthC43MGvHUyjEWeX1dmhaqqR/h7rwW9n4zjfd8+vMmHOFUl+2pM2a+4YyYUdK+f28KXJzuRrnFcVa7Z7/b5mLxH8do1j2//0KV6T+OqOgQQ9DqU+niNPVDDiE7teB6HxfI6uveThslRt8LTqWJxcMSuH11eXTkHqLAjXp5u/7v0pWlmOW+zOyoIfuJrvLYaRz4re4qdv8JQkv5A53uWy4LKeIuVw8+2Sht8Qjrkc40r/vZiFhffKRb0OoOEPUZL+65efGIngh6H93U9Njq59SlmXsqwSTRMnZrcnYctBA6lmicf3uMBZJ0Uz/I8EPCAIE/tCnrc9nZ6LAtulOdvp8GTtpQWnXa6XP7vZtjpmQf9EPR+MY4tJHVn0mVHK4eC/vklpV8E74utb3Y+rbO7IH436uxsqNPXuKZzAgh65+x6cme8F9uC5hPA2g3v6lChPlghTPH2sVjQ66yXebBxzZ7Nj2YtG0g4bri3HVWl2Nxn73ibgZ3sAGRHoDDVcbg5ZavioSM+0c0nlTmVMapyDCyLyd2JoMdrjR5oecDVKsWhUX1tN4Ie97Vwm2CrcpRZf2IP+XYFPZ4ZNgv9G5bLS1S37VtcourUouV8+yHo/WJcti3QW8Sqznko67+hoHuZw8t+YbLVqM6xwrGjXrse8lXfB35vnwCC3j6znt7hYBH29A2T9/judFbrGUtciMt3LZoly7Z5xYLuUXgcxjXON54Z+PfYrObAFOFhInVebDtA/WnH4tNih5rYmcfrzJ9uEYnKszh/nMJtX9+/TDp72tnbZSFqWy07fPJD0oQdBjZ3J4Iem7uronw5sI4jg8XxuLsR9FhEXTNHGnNZWiWfoLdiYG5331r5qOKgs11Bj09Zq+sjEfeJbrZJ9UPQ+8XYyw92FA2Tl8vcf5ulMYtIJ4wd+GscKS7um3Xawme8T/xKMW9vgfRAjzR4BBD0wWP/7pO9DuXYz/FpYT6QxfvE86AozYrpGbJN4l7rDVOZSa4s9KvDvm72u3KLgANZ+MCTMGJdmViXedn/6R7pWxeXl9pltlCGMcXLAlrsuJL07WAvunMLQ7mGuXv2NnHn7PCIPDlPWztyM7338ttbOEw+4c3Og7FFxB7LHnCUObF1IujxgMvt6nzKkgcm5+xQfrhGXUE/+DLprGkDmfwZXk/1Pulw8OVlB8cs8JnrZaks0liZQ1a7gl4WBa8qslyZQNVd3imrWz8EvZ+MLaAW0rB/H9LEGuYY/D6Ep+z0w1jQyw6HarXP3e+Gwz2HOzA8yHNQqDpLbYP8yU368Qj6EGjeZmdC+yW56fHsY+vgGv4zxyzSKh/LDkbwwQueOZWFiB176sAQms1OW/MpXj+/Wjr/ngyGo7/tsnJmoo5TvC6f/x4fSOF/v/+5bI12ypOZtcGzzU2WyiKgxQOYeE02zzeOM+1/dySxX9+cnTz3rpVhMekH6w8U37LDT+J46s7Pou7ZxdUPS3PPKm02Sho7qvm50J0IumdKFqQwuR4+DtV7tp08uPO65DbLNX+2B0keLMUpnmXZnG6Lx1OvSjc82oj/HgfscT5m6IiB502VHCrXyTNCxzkIPdv97x74OKhQ/OFuV9CdV7ze7H/zUsTXLyzmb2vFuFWl3UcX+7oHvWt1ccJXLOh5/Tr5JHiQk8eh7xdjW7XO33Hg+37HU9m74CiKPitgy2Wyd7hZ6OhY0G3Run4vyQPeQv98TvrpldLNj2fvrwfi3klzwBoD37W6/jidsOWe+gQQ9Pqs+nqlD0CxN28vktevw/Ov8zyrzkOvera9sQ8qOTrV93mAYU/csuSZsp1tymYLvt7b3D7b5GxrO935xLF249rbPO+lgXjmvddq0j5BuNmqOpf93omgN/sYO3+Lo+tXdqZ7M56HTZZOuqnx66RxzaMHxu1WJqR5Thb3ESOa8262/7kTQbfXvL3ny5KZuA3tNGkrSdz+70ZDPDM7VrjTVCboneYV74vvB2OXrRdlLjsPvewAnpyFWb/T4qx179ZY/bhqh9VO2XJffQIIen1Wfb3SYufoaJ7BtiteecEsmj4wIzwFKyx0LOjeS1y2Haisog+9IG3y29YIvB/d++FDk24VNH+0bfLPZ4Zl1zvwiM/DDj3iW+Vry8YWpzeP+17mbNYsP68vr7pgcQ9wJ4Lu/O0t7xl43WQh9hJHuK0ovzc+mexrq2fR58pSLOgWSAe4caCbdlKr40o7EXQ/u8xHo06ZPJg5dHKdK5tf0wtxzHOPBb0fjPNn+fwDt3WrEMH5tW57e7qH1y5zeLkX+1bLZgcbVe3/D4k6wJDf305jT3TXgtwdE0DQh1if8PrpYZsMPGu4VTHt6Tp+SjZja7WtKxZ0O8T9eMPWlgHPcB3Ao84hLi6jzXbeXhSflRyX36P6CXdLh0yqN7K3Ofr32xXXyOM8Pbu02dhm/qrtbXZesmm02UfRe+1tQrVD3SW7FJ9bttc2juS1Q0ksfZd3+xWkb67VejbugZbPfPfs08Jw4U4DZ9+xoNuH4JjNs9jwcWpmWXEMgdiMXdbPbL4/6LLWMQvi4EP2zfDRvHWS7zWTOkJiM7uXHa57tE7Ora+J49N3k2OzJaNeMg7L5+/EEZtmBxGVDaC9jPPzq6RL/1oMTlR1Ipp3l9gnJ/RvacZl8iPZ8kg3J951w5x7BxJA0Idor/DsafWFpaU+nDnC+AW2WP7rTclr3v7oOwqcZwZeZ6+TYkHPT0byy2sz9OIjpZGzS6+/JXlG7rX78bfXi/ceP98mZs/Y7ZyzxMjM3P5uuV/N1vtyz/M65Q6vMYM1F80GIfPPlZny7QB499PZGrhjvreT7PluEfQ+e4v7vc9KXpO89IF6e3HbeVZ+rR3C7PDnQc/CH8ieawuFmds3ID6fPj95Lw/s8uAL0rl3l59qZYelTy2YrcfbouFrfdxtM2clr4u6/raCjJov+5B7MGTLicvhmOG9EM8qTrPPIm20hLTB4pJ3F3jd3OLjsrjPuCw3PiadO7Uqp6H3e78Zuz95a6Hb23vy3Y9efSPj4AGhjz/Ok3k6nkNVcgwEn/hnHx2/y87bwu3Jg4PauF9UbZmrega/954Agt57pkM2x2aCPmQLTMEgAIGuCNiM/oPgHPU6p+B19UBuHlQCCPqg4p++D0fQpy9vngaBXhHwMcPe+hgmn3vg2XirFB+7eu0j0q4TelUq8hlqBBD0odYifSwPgt5HuGQNgT4TuHO/op9B2bbMsAjxkbf+zdskveRFSpMAgp5mu5bWCkEfRo1NVZMjcO6XM3+UPHk72Sm3SL+cVPT3eNdHY0XJ3vBhqrt+nhy4YVQhBH0YNTaCPowam6omR8Anz/1kw4HVssOjvf+9hczOs2EEt/DqAy6ULrk/OSxUKCCAoA+j7oCgD6PGpqpJEjhgjOQ9/+0me6X7OGVS2gQQ9LTbt1A7BH0YNTZVTZbAuNGZqHurX1XyrP07lzAzr+KUyu8IeiotWaMeDqQy92zZhQ4Yc9wN/dtrXaM4XAIBCHRBwDHbHWvf0R7zuAMWcO8Xd+S2027HAa4LvDPkrQj6DNlsFBoCEIAABCBQJICg0yMgAAEIQAACCRBA0BNoRKoAAQhAAAIQQNDpAxCAAAQgAIEECCDoCTQiVYAABCAAAQgg6PQBCEAAAhCAQAIEEPQEGpEqQAACEIAABBB0+gAEIAABCEAgAQIIegKNSBUgAAEIQAACCDp9AAIQgAAEIJAAAQQ9gUakChCAAAQgAAEEnT4AAQhAAAIQSIAAgp5AI1IFCEAAAhCAAIJOH4AABCAAAQgkQABBT6ARqQIEIAABCEAAQacPQAACEIAABBIggKAn0IhUAQIQgAAEIICg0wcgAAEIQAACCRBA0BNoRKoAAQhAAAIQQNDpAxCAAAQgAIEECCDoCTQiVYAABCAAAQgg6PQBCEAAAhCAQAIEEPQEGpEqQAACEIAABBB0+gAEIAABCEAgAQIIegKNSBUgAAEIQAACCDp9AAIQgAAEIJAAAQQ9gUakChCAAAQgAAEEnT4AAQhAAAIQSIAAgp5AI1IFCEAAAhCAAIJOH4AABCAAAQgkQABBT6ARqQIEIAABCEAAQacPQAACEIAABBIggKAn0IhUAQIQgAAEIICg0wcgAAEIQAACCRBA0BNoRKoAAQhAAAIQQNDpAxCAAAQgAIEECCDoCTQiVYAABCAAAQgg6PQBCEAAAhCAQAIEEPQEGpEqQAACEIAABBB0+gAEIAABCEAgAQIIegKNSBUgAAEIQAACCDp9AAIQgAAEIJAAAQQ9gUakChCAAAQgAAEEnT4AAQhAAAIQSIAAgp5AI1IFCEAAAhCAAIJOH4AABCAAAQgkQABBT6ARqQIEIAABCEAAQacPQAACEIAABBIggKAn0IhUAQIQgAAEIICg0wcgAAEIQAACCRBA0BNoRKoAAQhAAAIQQNDpAxCAAAQgAIEECCDoCTQiVYAABCAAAQj8H8wWJFmimzKYAAAAAElFTkSuQmCC" width="55%">
</p>

<h3 align="center">
A Unified Library for Speculative Decoding Algorithms for LLMs
</h3>

[![GitHub Release](https://img.shields.io/github/release/neuralmagic/speculators.svg?label=Version)](https://github.com/neuralmagic/speculators/releases) [![Documentation](https://img.shields.io/badge/Documentation-8A2BE2?logo=read-the-docs&logoColor=%23ffffff&color=%231BC070)](https://github.com/neuralmagic/speculators/tree/main/docs) [![License](https://img.shields.io/github/license/neuralmagic/speculators.svg)](https://github.com/neuralmagic/speculators/blob/main/LICENSE) [![Python Versions](https://img.shields.io/badge/Python-3.9--3.13-orange)](https://pypi.python.org/pypi/speculators)

## Overview

**Speculators** is a unified library for creating, representing, and storing speculative decoding algorithms for large language model (LLM) serving, such as in vLLM. It provides a standard format and tools to facilitate the productization of decoding algorithms for inference servers.

### Key Features

- **Speculative Decoding:** Simplify the creation and representation of speculative decoding algorithms for LLMs.
- **Standardized Format:** Ensure compatibility and ease of use with a standardized format for decoding algorithms.
- **Integration Ready:** Designed to integrate seamlessly with LLM inference servers like vLLM.
- **Productization Tools:** Streamline the process of deploying decoding algorithms in production environments.

## Getting Started

### Installation

Before installing, ensure you have the following prerequisites:

- OS: Linux or MacOS
- Python: 3.9 or higher

The latest Speculators release can be installed using pip:

```bash
pip install speculators
```

Or from source code using pip:

```bash
pip install git+https://github.com/neuralmagic/speculators.git
```

### Quick Start

Coming soon

## Resources

### Documentation

Coming soon

### Releases

Visit our [GitHub Releases page](https://github.com/neuralmagic/speculators/releases) and review the release notes to stay updated with the latest releases.

### License

Speculators is licensed under the [Apache License 2.0](https://github.com/neuralmagic/speculators/blob/main/LICENSE).

### Cite

If you find Speculators helpful in your research or projects, please consider citing it:

```bibtex
@misc{speculators2025,
  title={Speculators: A Unified Library for Speculative Decoding Algorithms in LLM Serving},
  author={Red Hat},
  year={2025},
  howpublished={\url{https://github.com/neuralmagic/speculators}},
}
```
