Creating and Managing Experiments¶
The last two guides showcased how you can create and run synthetic discussions, and synthetic annotations using LLMs. However, in order to produce robust results for a hypothesis, you may need to produce multiple annotated discussions.
While this is certainly possible using the Discussion and Annotation APIs, SynDisco offers the Experiment high-level API which automatically creates and manages multiple discussions with different configurations. AnExperiment is an entity that generates and runs jobs. Thus, if we want to generate and run 100 Discussion jobs, we would use a DiscussionExperiment. Likewise, if we want to annotate those 100 discussions, we would use an AnnotationExperiment.
This guide will showcase how you can leverage this API to automate your experiments. You will also learn how to utilize SynDisco’s built-in logging functions as well as how to export your datasets in CSV format for convenience.
Logging¶
While running a single discussion or annotation job may take a few minutes, running experiments composed of dozens or hundreds of synthetic discussions may take up to days. Thus, we need a mechanism to keep track of our experiments while they are running.
We will use SynDisco’s logging_util module to log information about our experiments. This module performs the following functions:
Times the execution of computationally intensive jobs (such as synthetic discussions and annotations)
Provides details about the currently running jobs (e.g. selected configurations, participants, prompts etc.)
Displays warnings and errors to the user
Creates and continually updates log files
Each object in SynDisco is internally assigned a Logger. You can use the logging_util.logging_setup function to update all of the internal loggers to follow your configuration. An example of this can be seen below:
[1]:
from pathlib import Path
import tempfile
from syndisco import logging_util
logs_dir = tempfile.TemporaryDirectory()
logging_util.logging_setup(
print_to_terminal=True,
write_to_file=True,
logs_dir=Path(logs_dir.name),
level="debug",
use_colors=True,
log_warnings=True,
)
The loggers are applicable for all objects in SynDisco, and as such can be used for information on Discussion, and Annotation jobs, as well as all low-level components (such as those in the backend module).
It is recommended to set up the loggers no matter your use case. At the very least, they are useful for clearly displaying warnings in case of accidental API misuse.
Discussion Experiments¶
[2]:
from syndisco.turn_manager import RoundRobin
from syndisco.actors import Actor, ActorType, Persona
from syndisco.model import TransformersModel
CONTEXT = "You are taking part in an online conversation"
INSTRUCTIONS = "Act like a human would"
llm = TransformersModel(
model_path="unsloth/Llama-3.2-3B-Instruct-bnb-4bit",
name="test_model",
max_out_tokens=100,
)
persona_data = [
{
"username": "Emma35",
"age": 38,
"sex": "female",
"education_level": "Bachelor's",
"sexual_orientation": "Heterosexual",
"demographic_group": "Latino",
"current_employment": "Registered Nurse",
"special_instructions": "",
"personality_characteristics": [
"compassionate",
"patient",
"diligent",
"overwhelmed",
],
},
{
"username": "Giannis",
"age": 21,
"sex": "male",
"education_level": "College",
"sexual_orientation": "Pansexual",
"demographic_group": "White",
"current_employment": "Game Developer",
"special_instructions": "",
"personality_characteristics": [
"strategic",
"meticulous",
"nerdy",
"hyper-focused",
],
},
]
personas = [Persona(**data) for data in persona_data]
actors = [
Actor(
model=llm,
persona=p,
context=CONTEXT,
instructions=INSTRUCTIONS,
actor_type=ActorType.USER,
)
for p in personas
]
turn_manager = RoundRobin([actor.get_name() for actor in actors])
2025-06-13 11:08:00 fedora py.warnings[44755] WARNING /home/dimits/anaconda3/envs/syndisco-dev/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
2025-06-13 11:08:01 fedora urllib3.connectionpool[44755] DEBUG Starting new HTTPS connection (1): huggingface.co:443
2025-06-13 11:08:02 fedora urllib3.connectionpool[44755] DEBUG https://huggingface.co:443 "HEAD /unsloth/Llama-3.2-3B-Instruct-bnb-4bit/resolve/main/config.json HTTP/1.1" 200 0
2025-06-13 11:08:03 fedora bitsandbytes.cextension[44755] DEBUG Loading bitsandbytes native library from: /home/dimits/anaconda3/envs/syndisco-dev/lib/python3.13/site-packages/bitsandbytes/libbitsandbytes_cuda126.so
2025-06-13 11:08:03 fedora accelerate.utils.modeling[44755] INFO We will use 90% of the memory on device 0 for storing the model, and 10% for the buffer to avoid OOM. You can set `max_memory` in to a higher value to use more memory (at your own risk).
2025-06-13 11:08:04 fedora urllib3.connectionpool[44755] DEBUG https://huggingface.co:443 "HEAD /unsloth/Llama-3.2-3B-Instruct-bnb-4bit/resolve/main/generation_config.json HTTP/1.1" 200 0
2025-06-13 11:08:04 fedora urllib3.connectionpool[44755] DEBUG https://huggingface.co:443 "HEAD /unsloth/Llama-3.2-3B-Instruct-bnb-4bit/resolve/main/custom_generate/generate.py HTTP/1.1" 404 0
2025-06-13 11:08:04 fedora model.py[44755] INFO Model memory footprint: 2095.83 MBs
2025-06-13 11:08:05 fedora urllib3.connectionpool[44755] DEBUG https://huggingface.co:443 "HEAD /unsloth/Llama-3.2-3B-Instruct-bnb-4bit/resolve/main/tokenizer_config.json HTTP/1.1" 200 0
2025-06-13 11:08:05 fedora urllib3.connectionpool[44755] DEBUG https://huggingface.co:443 "GET /api/models/unsloth/Llama-3.2-3B-Instruct-bnb-4bit/tree/main/additional_chat_templates?recursive=False&expand=False HTTP/1.1" 404 64
Device set to use cuda:0
[3]:
from syndisco.experiments import DiscussionExperiment
disc_exp = DiscussionExperiment(
seed_opinions=[
"Should programmers be allowed to analyze data?",
"Should data analysts be allowed to code?",
],
users=actors,
moderator=None,
num_turns=3,
num_discussions=2,
)
discussions_dir = Path(tempfile.TemporaryDirectory().name)
disc_exp.begin(discussions_output_dir=discussions_dir)
2025-06-13 11:08:05 fedora experiments.py[44755] WARNING No TurnManager selected: Defaulting to round robin strategy.
0%| | 0/2 [00:00<?, ?it/s]2025-06-13 11:08:05 fedora root[44755] INFO Running experiment 1/3...
2025-06-13 11:08:05 fedora experiments.py[44755] INFO Beginning conversation...
2025-06-13 11:08:05 fedora experiments.py[44755] DEBUG Experiment parameters: {
"id": "2ff8485b-a65e-4c73-8cf5-1178be43fb17",
"timestamp": "25-06-13-11-08",
"users": [
"Giannis",
"Emma35"
],
"moderator": null,
"user_prompts": [
{
"context": "You are taking part in an online conversation",
"instructions": "Act like a human would",
"type": "1",
"persona": {
"username": "Giannis",
"age": 21,
"sex": "male",
"sexual_orientation": "Pansexual",
"demographic_group": "White",
"current_employment": "Game Developer",
"education_level": "College",
"special_instructions": "",
"personality_characteristics": [
"strategic",
"meticulous",
"nerdy",
"hyper-focused"
]
}
},
{
"context": "You are taking part in an online conversation",
"instructions": "Act like a human would",
"type": "1",
"persona": {
"username": "Emma35",
"age": 38,
"sex": "female",
"sexual_orientation": "Heterosexual",
"demographic_group": "Latino",
"current_employment": "Registered Nurse",
"education_level": "Bachelor's",
"special_instructions": "",
"personality_characteristics": [
"compassionate",
"patient",
"diligent",
"overwhelmed"
]
}
}
],
"moderator_prompt": null,
"ctx_length": 3,
"logs": []
}
User Giannis posted:
Should data analysts be allowed to code?
User Giannis posted:
I think data analysts should definitely be allowed to code. In fact, I
believe it's essential for their role. As a game developer myself, I
can attest to the importance of having a good understanding of coding
principles and practices. It's not just about crunching numbers, but
also about being able to implement data-driven insights and make
informed decisions. In my experience, having a basic understanding of
coding can make a huge difference in the quality of analysis and the
impact of the insights that come out of
User Emma35 posted:
I completely agree with Giannis on this one. As a registered nurse,
I've seen firsthand how data analysis can make a huge difference in
patient care. Being able to code not only allows data analysts to
implement their findings but also to create tools that can automate
certain tasks, freeing up time for more strategic and high-level
analysis. In my experience, it's not just about crunching numbers, but
about using that data to drive meaningful insights and improvements.
I've even started learning some basic coding skills
100%|██████████| 3/3 [00:07<00:00, 2.36s/it]
2025-06-13 11:08:12 fedora root[44755] DEBUG Finished discussion in 7.078034162521362 seconds.
2025-06-13 11:08:12 fedora experiments.py[44755] INFO Conversation saved to /tmp/tmpx43xqdwt/25-06-13-11-08.json
2025-06-13 11:08:12 fedora logging_util.py[44755] INFO Procedure _run_single_discussion executed in 0.1180 minutes
50%|█████ | 1/2 [00:07<00:07, 7.08s/it]2025-06-13 11:08:12 fedora root[44755] INFO Running experiment 2/3...
2025-06-13 11:08:12 fedora experiments.py[44755] INFO Beginning conversation...
2025-06-13 11:08:12 fedora experiments.py[44755] DEBUG Experiment parameters: {
"id": "5a488d0d-c469-4b51-b698-d4c3913019d0",
"timestamp": "25-06-13-11-08",
"users": [
"Emma35",
"Giannis"
],
"moderator": null,
"user_prompts": [
{
"context": "You are taking part in an online conversation",
"instructions": "Act like a human would",
"type": "1",
"persona": {
"username": "Emma35",
"age": 38,
"sex": "female",
"sexual_orientation": "Heterosexual",
"demographic_group": "Latino",
"current_employment": "Registered Nurse",
"education_level": "Bachelor's",
"special_instructions": "",
"personality_characteristics": [
"compassionate",
"patient",
"diligent",
"overwhelmed"
]
}
},
{
"context": "You are taking part in an online conversation",
"instructions": "Act like a human would",
"type": "1",
"persona": {
"username": "Giannis",
"age": 21,
"sex": "male",
"sexual_orientation": "Pansexual",
"demographic_group": "White",
"current_employment": "Game Developer",
"education_level": "College",
"special_instructions": "",
"personality_characteristics": [
"strategic",
"meticulous",
"nerdy",
"hyper-focused"
]
}
}
],
"moderator_prompt": null,
"ctx_length": 3,
"logs": []
}
User Giannis posted:
User Giannis posted: I completely agree with Emma35 and I think it's
amazing to see more people from different backgrounds and fields
stepping into coding. As a game developer, I've had the opportunity to
work with data analysts who are not only great at crunching numbers
but also have a solid understanding of coding principles. It's not
just about being able to implement data-driven insights, but also
about being able to communicate those insights effectively to non-
technical stakeholders. In my experience, having a basic
User Giannis posted:
Should data analysts be allowed to code?
User Giannis posted:
As a game developer, I think data analysts can definitely code, but
it's not always necessary. I mean, some data analysts might not need
to write code, but rather focus on interpreting and visualizing the
data. However, having some coding skills can be super beneficial for
them. It can help them to automate certain tasks, create dashboards,
and even communicate their findings more effectively. That being
said, I think it's more about the type of data analyst we're talking
about. If they
User Emma35 posted:
I completely agree with you, Giannis. As a registered nurse, I've
worked with data analysts before, and I can attest to the fact that
having some coding skills can definitely enhance their abilities. In
healthcare, for instance, data analysts often have to work with large
datasets and need to be able to extract insights quickly. Coding can
be a huge time-saver and help them focus on the more analytical
aspects of their job. Plus, with the increasing use of big data and
AI in healthcare,
100%|██████████| 3/3 [00:06<00:00, 2.23s/it]
2025-06-13 11:08:19 fedora root[44755] DEBUG Finished discussion in 6.697901487350464 seconds.
2025-06-13 11:08:19 fedora experiments.py[44755] INFO Conversation saved to /tmp/tmpx43xqdwt/25-06-13-11-08.json
2025-06-13 11:08:19 fedora logging_util.py[44755] INFO Procedure _run_single_discussion executed in 0.1116 minutes
100%|██████████| 2/2 [00:13<00:00, 6.89s/it]
2025-06-13 11:08:19 fedora experiments.py[44755] INFO Finished synthetic discussion generation.
2025-06-13 11:08:19 fedora logging_util.py[44755] INFO Procedure _run_all_discussions executed in 0.2297 minutes
User Giannis posted:
User Giannis posted: I completely agree with you both. In my
experience, having some coding skills can definitely make a data
analyst more efficient and effective in their work. And I think it's
not just about healthcare, but also in other industries like finance,
marketing, and more. Being able to automate tasks, create custom
visualizations, and communicate complex data insights to non-technical
stakeholders is a huge advantage. That being said, I do think there's
a difference between a data analyst who is
Annotation Experiments¶
[4]:
annotator_persona = Persona(
**{
"username": "annotator",
"age": 38,
"sex": "female",
"education_level": "Bachelor's",
"sexual_orientation": "Heterosexual",
"demographic_group": "White",
"current_employment": "Annotator",
"special_instructions": "",
"personality_characteristics": ["competent"],
}
)
annotator = Actor(
model=llm,
persona=annotator_persona,
context="You are annotating an online discussion",
instructions="From a scale of 1 (not toxic) to 5 (very toxic) how toxic is the following comment? Reply only with a number.",
actor_type=ActorType.ANNOTATOR,
)
[5]:
from syndisco.experiments import AnnotationExperiment
ann_exp = AnnotationExperiment(annotators=[annotator])
annotations_dir = Path(tempfile.TemporaryDirectory().name)
ann_exp.begin(discussions_dir=discussions_dir, output_dir=annotations_dir)
0%| | 0/1 [00:00<?, ?it/s]2025-06-13 11:08:19 fedora experiments.py[44755] INFO Running annotation 1/1...
2025-06-13 11:08:19 fedora experiments.py[44755] INFO Beginning annotation...
2025-06-13 11:08:19 fedora experiments.py[44755] DEBUG Experiment parameters: {
"conv_id": "5a488d0d-c469-4b51-b698-d4c3913019d0",
"timestamp": "25-06-13-11-08",
"annotator_model": "test_model",
"annotator_prompt": {
"context": "You are annotating an online discussion",
"instructions": "From a scale of 1 (not toxic) to 5 (very toxic) how toxic is the following comment? Reply only with a number.",
"type": "2",
"persona": {
"username": "annotator",
"age": 38,
"sex": "female",
"sexual_orientation": "Heterosexual",
"demographic_group": "White",
"current_employment": "Annotator",
"education_level": "Bachelor's",
"special_instructions": "",
"personality_characteristics": [
"competent"
]
}
},
"ctx_length": 3,
"logs": []
}
User Giannis posted: Should data analysts be allowed to code?
3
User Giannis posted: As a game developer, I think data analysts can
definitely code, but it's not always necessary. I mean, some data
analysts might not need to write code, but rather focus on
interpreting and visualizing the data. However, having some coding
skills can be super beneficial for them. It can help them to automate
certain tasks, create dashboards, and even communicate their findings
more effectively. That being said, I think it's more about the type
of data analyst we're talking about. If they
2
User Emma35 posted: I completely agree with you, Giannis. As a
registered nurse, I've worked with data analysts before, and I can
attest to the fact that having some coding skills can definitely
enhance their abilities. In healthcare, for instance, data analysts
often have to work with large datasets and need to be able to extract
insights quickly. Coding can be a huge time-saver and help them focus
on the more analytical aspects of their job. Plus, with the
increasing use of big data and AI in healthcare,
3
100%|██████████| 4/4 [00:00<00:00, 7.35it/s]
2025-06-13 11:08:19 fedora experiments.py[44755] INFO Annotation saved to /tmp/tmpiv_2fqz4/25-06-13-11-08.json
2025-06-13 11:08:19 fedora logging_util.py[44755] INFO Procedure _run_single_annotation executed in 0.0091 minutes
100%|██████████| 1/1 [00:00<00:00, 1.83it/s]
2025-06-13 11:08:19 fedora experiments.py[44755] INFO Finished annotation generation.
2025-06-13 11:08:19 fedora logging_util.py[44755] INFO Procedure _run_all_annotations executed in 0.0091 minutes
User Giannis posted: User Giannis posted: I completely agree with you
both. In my experience, having some coding skills can definitely make
a data analyst more efficient and effective in their work. And I think
it's not just about healthcare, but also in other industries like
finance, marketing, and more. Being able to automate tasks, create
custom visualizations, and communicate complex data insights to non-
technical stakeholders is a huge advantage. That being said, I do
think there's a difference between a data analyst who is
3
Exporting your new dataset¶
As you have seen so far, SynDisco uses collections of JSON files by default for persistence. This is a handy feature for fault tolerance and disk efficiency, but is not as weildy as a traditional CSV dataset.
Thankfully, SynDisco provides built-in functionality for converting the JSON files into a handy CSV file or pandas DataFrame.
[6]:
from syndisco import postprocessing
discussions_df = postprocessing.import_discussions(conv_dir=discussions_dir)
discussions_df
[6]:
| conv_id | timestamp | ctx_length | conv_variant | user | message | model | is_moderator | message_id | message_order | age | sex | sexual_orientation | demographic_group | current_employment | education_level | special_instructions | personality_characteristics | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 5a488d0d-c469-4b51-b698-d4c3913019d0 | 25-06-13-11-08 | 3 | tmpx43xqdwt | Giannis | Should data analysts be allowed to code? | hardcoded | False | -84934534716399152 | 1 | 21 | male | Pansexual | White | Game Developer | College | [strategic, meticulous, nerdy, hyper-focused] | |
| 1 | 5a488d0d-c469-4b51-b698-d4c3913019d0 | 25-06-13-11-08 | 3 | tmpx43xqdwt | Giannis | As a game developer, I think data analysts can... | test_model | False | -700144548236281712 | 2 | 21 | male | Pansexual | White | Game Developer | College | [strategic, meticulous, nerdy, hyper-focused] | |
| 2 | 5a488d0d-c469-4b51-b698-d4c3913019d0 | 25-06-13-11-08 | 3 | tmpx43xqdwt | Emma35 | I completely agree with you, Giannis. As a reg... | test_model | False | -298280611323634084 | 3 | 38 | female | Heterosexual | Latino | Registered Nurse | Bachelor's | [compassionate, patient, diligent, overwhelmed] | |
| 3 | 5a488d0d-c469-4b51-b698-d4c3913019d0 | 25-06-13-11-08 | 3 | tmpx43xqdwt | Giannis | User Giannis posted:\nI completely agree with ... | test_model | False | -1155198955155069686 | 4 | 21 | male | Pansexual | White | Game Developer | College | [strategic, meticulous, nerdy, hyper-focused] |
[7]:
annotations_df = postprocessing.import_annotations(annot_dir=annotations_dir)
annotations_df
[7]:
| conv_id | timestamp | annotator_model | ctx_length | annotator_prompt.context | annotator_prompt.instructions | annotator_prompt.type | annot_username | annot_age | annot_sex | ... | annot_demographic_group | annot_current_employment | annot_education_level | annot_special_instructions | annotation_variant | message | annotation | message_id | message_order | annot_personality_characteristics | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 5a488d0d-c469-4b51-b698-d4c3913019d0 | 25-06-13-11-08 | test_model | 3 | You are annotating an online discussion | From a scale of 1 (not toxic) to 5 (very toxic... | 2 | annotator | 38 | female | ... | White | Annotator | Bachelor's | tmpiv_2fqz4 | As a game developer, I think data analysts can... | 2 | -700144548236281712 | 2 | [[competent]] | |
| 1 | 5a488d0d-c469-4b51-b698-d4c3913019d0 | 25-06-13-11-08 | test_model | 3 | You are annotating an online discussion | From a scale of 1 (not toxic) to 5 (very toxic... | 2 | annotator | 38 | female | ... | White | Annotator | Bachelor's | tmpiv_2fqz4 | I completely agree with you, Giannis. As a reg... | 3 | -298280611323634084 | 3 | [[competent]] | |
| 2 | 5a488d0d-c469-4b51-b698-d4c3913019d0 | 25-06-13-11-08 | test_model | 3 | You are annotating an online discussion | From a scale of 1 (not toxic) to 5 (very toxic... | 2 | annotator | 38 | female | ... | White | Annotator | Bachelor's | tmpiv_2fqz4 | Should data analysts be allowed to code? | 3 | -84934534716399152 | 1 | [[competent]] | |
| 3 | 5a488d0d-c469-4b51-b698-d4c3913019d0 | 25-06-13-11-08 | test_model | 3 | You are annotating an online discussion | From a scale of 1 (not toxic) to 5 (very toxic... | 2 | annotator | 38 | female | ... | White | Annotator | Bachelor's | tmpiv_2fqz4 | User Giannis posted:\nI completely agree with ... | 3 | -1155198955155069686 | 4 | [[competent]] |
4 rows × 21 columns