LLMs present an opportunity to create automated assistants that can help users navigate complex tasks. However, existing approaches have limitations in: (i) handling conditional logic, (ii) integrating knowledge sources, and (iii) consistently following instructions. Researchers and industry professionals often employ ad-hoc pipelines to construct conversational agents. These pipelines aim to maintain context, address failure cases, and minimize hallucinations, yet frequently fail to achieve these objectives.
We present Genie - a programmable framework for creating conversational agents that are designed to handle complex user interactions and knowledge queries. Unlike LLMs, Genie provides reliable grounded responses, with controllable agent policies through its expressive specification, Genie Worksheet. In contrast to dialog trees, it is resilient to diverse user queries, helpful with knowledge sources, and offers ease of programming policies through its declarative paradigm.
Features | Pure LLMs | Dialog Trees | GenieWorksheets |
---|---|---|---|
Handles unexpected queries | ✅ | ❌ | ✅ |
Consistent output | ❌ | ✅ | ✅ |
Knowledge integration | ✅ | ❌ | ✅ |
Easy maintenance | ❌ | ❌ | ✅ |
Natural conversations | ✅ | ❌ | ✅ |
Control over responses | ❌ | ✅ | ✅ |
Complex logic support | ❌ | ❌ | ✅ |
Low hallucination risk | ❌ | ✅ | ✅ |
Handles interruptions | ✅ | ❌ | ✅ |
Programmable behaviors | ❌ | ✅ | ✅ |
Dynamic field dependencies | ❌ | ❌ | ✅ |
Development speed | ✅ | ❌ | ✅ |
We perform a real user study with 62 users across three domains. We choose three diverse applications with varying complexities. Compared to the 3 StarV2 domains, these real-world applications are much more complicated, as shown in Table 5 in the Appendix.
@article{genieworksheets,
title={Coding Reliable LLM-based Integrated Task and Knowledge Agents with GenieWorksheets},
author={Joshi, Harshit and Liu, Shicheng and Chen, James and Weigle, Robert and Lam, Monica S},
journal={arXiv preprint arXiv:2407.05674},
year={2024}
}