/Users/rajsiba/miniconda3/envs/fmcore/lib/python3.11/site-packages/langchain/__init__.py:30: UserWarning: Importing Anthropic from langchain root module is no longer supported. Please use langchain_community.llms.Anthropic instead.
  warnings.warn(
/Users/rajsiba/miniconda3/envs/fmcore/lib/python3.11/site-packages/langchain/__init__.py:30: UserWarning: Importing HuggingFaceHub from langchain root module is no longer supported. Please use langchain_community.llms.HuggingFaceHub instead.
  warnings.warn(
/Users/rajsiba/miniconda3/envs/fmcore/lib/python3.11/site-packages/langchain/__init__.py:30: UserWarning: Importing LLMChain from langchain root module is no longer supported. Please use langchain.chains.LLMChain instead.
  warnings.warn(
/Users/rajsiba/miniconda3/envs/fmcore/lib/python3.11/site-packages/langchain/__init__.py:30: UserWarning: Importing OpenAI from langchain root module is no longer supported. Please use langchain_community.llms.OpenAI instead.
  warnings.warn(
/Users/rajsiba/miniconda3/envs/fmcore/lib/python3.11/site-packages/langchain/__init__.py:30: UserWarning: Importing PromptTemplate from langchain root module is no longer supported. Please use langchain_core.prompts.PromptTemplate instead.
  warnings.warn(
/Users/rajsiba/miniconda3/envs/fmcore/lib/python3.11/site-packages/langchain/__init__.py:30: UserWarning: Importing Anthropic from langchain root module is no longer supported. Please use langchain_community.llms.Anthropic instead.
  warnings.warn(
/Users/rajsiba/miniconda3/envs/fmcore/lib/python3.11/site-packages/langchain/__init__.py:30: UserWarning: Importing HuggingFaceHub from langchain root module is no longer supported. Please use langchain_community.llms.HuggingFaceHub instead.
  warnings.warn(
/Users/rajsiba/miniconda3/envs/fmcore/lib/python3.11/site-packages/langchain/__init__.py:30: UserWarning: Importing LLMChain from langchain root module is no longer supported. Please use langchain.chains.LLMChain instead.
  warnings.warn(
/Users/rajsiba/miniconda3/envs/fmcore/lib/python3.11/site-packages/langchain/chat_models/__init__.py:33: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.chat_models import ChatOpenAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/Users/rajsiba/miniconda3/envs/fmcore/lib/python3.11/site-packages/langchain/chat_models/__init__.py:33: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.chat_models import ChatOpenAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/Volumes/workplace/fmcore/fmcore/src/fmcore/prompt_tuner/evaluator/base_evaluator.py:13: GenericBeforeBaseModelWarning: Classes should inherit from `BaseModel` before generic classes (e.g. `typing.Generic[T]`) for pydantic generics to work properly.
  class BaseEvaluator(Generic[I, O], MutableTyped, Registry, ABC):
/Volumes/workplace/fmcore/fmcore/src/fmcore/prompt_tuner/evaluator/llm_as_a_judge_boolean/llm_as_a_judge_boolean_evaluator.py:17: GenericBeforeBaseModelWarning: Classes should inherit from `BaseModel` before generic classes (e.g. `typing.Generic[T]`) for pydantic generics to work properly.
  class LLMAsJudgeBooleanEvaluator(BaseEvaluator[Dict, bool]):
/Volumes/workplace/fmcore/fmcore/src/fmcore/prompt_tuner/evaluator/classification/classification_evaluator.py:9: GenericBeforeBaseModelWarning: Classes should inherit from `BaseModel` before generic classes (e.g. `typing.Generic[T]`) for pydantic generics to work properly.
  class ClassificationEvaluator(BaseEvaluator[Dict, bool]):
Running Prompt Tuner with Classification
Running Prompt Tuner with Boolean as judge
{'task_type': 'TEXT_GENERATION', 'dataset_config': {'inputs': {'TRAIN': {'name': None, 'path': 's3://iml-development-us-east-1/starfish/data/train/HEADPHONES_earpiece_shape.parquet', 'storage': S3, 'format': PARQUET, 'contents': None, 'file_glob': None, 'data_schema': None}, 'TEST': {'name': None, 'path': 's3://iml-development-us-east-1/starfish/data/test/HEADPHONES_earpiece_shape.parquet', 'storage': S3, 'format': PARQUET, 'contents': None, 'file_glob': None, 'data_schema': None}}, 'output': {'name': 'results', 'path': 's3://iml-development-us-east-1/starfish/output/HEADPHONES_earpiece_shape.parquet', 'storage': S3, 'format': PARQUET, 'contents': None, 'file_glob': None, 'data_schema': None}}, 'prompt_config': {'prompt': "ROLE: You are a Catalog Expert. You analyze product information and you are trying your best to infer missing attribute values.\n\nAnalyze the provided Amazon product information in JSON format, detailed above, to determine the value in English of a specific attribute.\n\nYour task is to thoroughly examine the product details. If the attribute's value is clearly inferable from the provided information, make an accurate prediction. \nIn scenarios where the value cannot be deduced, indicate this with '[NO]' for Not Obtainable. \nIf the attribute does not pertain to the product, use '[NA]' for Not Applicable. \nEnsure your prediction is compatible with the attribute's data type, such as predicting 'True' or 'False' for Boolean attributes, or an integer for Integer attributes. Avoid using scientific notation for any prediction.\n        \nFocus your analysis on this specific attribute: \nattribute name: headphones.earpiece_shape", 'input_fields': [{'name': 'asin', 'description': 'This field represent the unique identifier for a product', 'field_type': 'string'}, {'name': 'product_type', 'description': 'This fields represent the type of product', 'field_type': 'string'}, {'name': 'attribute', 'description': 'This field represents the attribute to be extracted', 'field_type': 'string'}, {'name': 'asin_info', 'description': 'This field represent the information related to product', 'field_type': 'string'}, {'name': 'attribute_instructions', 'description': 'This field represent the additional information related to product like possible values for attribute_value', 'field_type': 'string'}], 'output_fields': [{'name': 'attribute_value', 'description': 'This field represent the value extracted for the given attribute_name from the product', 'field_type': 'string'}]}, 'framework': 'DSPY', 'optimizer_config': {'evaluator_config': {'evaluator_type': 'LLM_AS_A_JUDGE_BOOLEAN', 'evaluator_params': {'llm_config': {'provider_type': 'BEDROCK', 'model_id': 'anthropic.claude-3-5-sonnet-20240620-v1:0', 'model_params': {'temperature': 0.5, 'max_tokens': 1024, 'top_p': 0.5}, 'provider_params_list': [{'retries': {'max_retries': 50, 'backoff_factor': 1.0, 'jitter': 1.0, 'retryable_exceptions': ['InvalidSignatureException', 'ThrottlingException', 'ModelTimeoutException', 'ServiceUnavailableException', 'ModelNotReadyException', 'ServiceQuotaExceededException', 'ModelErrorException', 'EndpointConnectionError']}, 'rate_limit': {'max_rate': 50, 'time_period': 60}, 'role_arn': 'arn:aws:iam::863518436859:role/ModelFactoryBedrockAccessRole', 'region': 'us-east-1'}, {'retries': {'max_retries': 50, 'backoff_factor': 1.0, 'jitter': 1.0, 'retryable_exceptions': ['InvalidSignatureException', 'ThrottlingException', 'ModelTimeoutException', 'ServiceUnavailableException', 'ModelNotReadyException', 'ServiceQuotaExceededException', 'ModelErrorException', 'EndpointConnectionError']}, 'rate_limit': {'max_rate': 50, 'time_period': 60}, 'role_arn': 'arn:aws:iam::615299746603:role/ModelFactoryBedrockAccessRole', 'region': 'us-east-1'}, {'retries': {'max_retries': 50, 'backoff_factor': 1.0, 'jitter': 1.0, 'retryable_exceptions': ['InvalidSignatureException', 'ThrottlingException', 'ModelTimeoutException', 'ServiceUnavailableException', 'ModelNotReadyException', 'ServiceQuotaExceededException', 'ModelErrorException', 'EndpointConnectionError']}, 'rate_limit': {'max_rate': 50, 'time_period': 60}, 'role_arn': 'arn:aws:iam::615299746603:role/ModelFactoryBedrockAccessRole', 'region': 'us-west-2'}, {'retries': {'max_retries': 50, 'backoff_factor': 1.0, 'jitter': 1.0, 'retryable_exceptions': ['InvalidSignatureException', 'ThrottlingException', 'ModelTimeoutException', 'ServiceUnavailableException', 'ModelNotReadyException', 'ServiceQuotaExceededException', 'ModelErrorException', 'EndpointConnectionError']}, 'rate_limit': {'max_rate': 50, 'time_period': 60}, 'role_arn': 'arn:aws:iam::710271919393:role/ModelFactoryBedrockAccessRole', 'region': 'us-east-1'}, {'retries': {'max_retries': 50, 'backoff_factor': 1.0, 'jitter': 1.0, 'retryable_exceptions': ['InvalidSignatureException', 'ThrottlingException', 'ModelTimeoutException', 'ServiceUnavailableException', 'ModelNotReadyException', 'ServiceQuotaExceededException', 'ModelErrorException', 'EndpointConnectionError']}, 'rate_limit': {'max_rate': 50, 'time_period': 60}, 'role_arn': 'arn:aws:iam::710271919393:role/ModelFactoryBedrockAccessRole', 'region': 'us-west-2'}, {'retries': {'max_retries': 50, 'backoff_factor': 1.0, 'jitter': 1.0, 'retryable_exceptions': ['InvalidSignatureException', 'ThrottlingException', 'ModelTimeoutException', 'ServiceUnavailableException', 'ModelNotReadyException', 'ServiceQuotaExceededException', 'ModelErrorException', 'EndpointConnectionError']}, 'rate_limit': {'max_rate': 50, 'time_period': 60}, 'role_arn': 'arn:aws:iam::872515274170:role/ModelFactoryBedrockAccessRole', 'region': 'us-east-1'}, {'retries': {'max_retries': 50, 'backoff_factor': 1.0, 'jitter': 1.0, 'retryable_exceptions': ['InvalidSignatureException', 'ThrottlingException', 'ModelTimeoutException', 'ServiceUnavailableException', 'ModelNotReadyException', 'ServiceQuotaExceededException', 'ModelErrorException', 'EndpointConnectionError']}, 'rate_limit': {'max_rate': 50, 'time_period': 60}, 'role_arn': 'arn:aws:iam::872515274170:role/ModelFactoryBedrockAccessRole', 'region': 'us-west-2'}]}, 'prompt': 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nYou are an auditor for Amazon. Your task is to verify the \'earpiece_shape\' of a product in the Amazon catalog. \nYou will be given the Amazon product data and a test value of \'earpiece_shape\'.\nFirst, you need to deduce the earpiece_shape of the product from the Amazon product data.\nThen, you need to compare the earpiece_shape you deduced from the Amazon product data to the test value of earpiece_shape. \nYou need to predict if the test value of \'earpiece_shape\' is \'Correct\', \'Incorrect\', or \'Unknown\' based on the Amazon product data.\nYou also need to give the reason for your prediction.\n\n\n\n\n\n\n### Rules:\nTo ensure accurate predictions, follow these rules in sequence and think step by step before responding.\n\n1. If you cannot deduce the value of the \'earpiece_shape\' from the given product data, predict \'Unknown\'.\n2. If the test value is aligns with the deduced value, predict \'Correct\'.\n3. If the test value is less informative than the deduced value, predict \'Correct\'.\n4. If the test value is more informative than the deduced value, predict \'Correct\'.\n5. Predict \'Incorrect\' if the test value contradicts the deduced value.\n\n\n### Additional information:\nHere is some additional information about \'earpiece_shape\' to help you make highly accurate classifications.\nEarpiece shape refers to the design and form factor of the part of headphones that fits into or around the outer ear, affecting comfort, fit, and aesthetics.\n\n\n### Amazon product data:\nGiven below is the Amazon product data.\n \n{{input}}\n\n### Test value:\nNow verify the test value of the attribute \'earpiece_shape\': \'{{output.attribute_value}}\'.\n\n\n### Output format:\nRemember to make only one overall prediction. Feel free to ignore irrelevant information and only pay close attention to relevant information in product data.\nPlease output the results in the following JSON format. The JSON should not have anything else except the reason and the prediction.\n{\n    "reason": "evidence for the prediction", \n    "prediction": "Correct/Incorrect/Unknown"\n} \nDo not output anything except the JSON. Always begin your output with {.\n\n\n### Response:', 'criteria': "prediction == 'Correct'"}}, 'teacher_config': {'provider_type': 'BEDROCK', 'model_id': 'anthropic.claude-3-5-sonnet-20240620-v1:0', 'model_params': {'temperature': 0.5, 'max_tokens': 1024, 'top_p': 0.5}, 'provider_params_list': [{'retries': {'max_retries': 50, 'backoff_factor': 1.0, 'jitter': 1.0, 'retryable_exceptions': ['InvalidSignatureException', 'ThrottlingException', 'ModelTimeoutException', 'ServiceUnavailableException', 'ModelNotReadyException', 'ServiceQuotaExceededException', 'ModelErrorException', 'EndpointConnectionError']}, 'rate_limit': {'max_rate': 50, 'time_period': 60}, 'role_arn': 'arn:aws:iam::863518436859:role/ModelFactoryBedrockAccessRole', 'region': 'us-east-1'}, {'retries': {'max_retries': 50, 'backoff_factor': 1.0, 'jitter': 1.0, 'retryable_exceptions': ['InvalidSignatureException', 'ThrottlingException', 'ModelTimeoutException', 'ServiceUnavailableException', 'ModelNotReadyException', 'ServiceQuotaExceededException', 'ModelErrorException', 'EndpointConnectionError']}, 'rate_limit': {'max_rate': 50, 'time_period': 60}, 'role_arn': 'arn:aws:iam::615299746603:role/ModelFactoryBedrockAccessRole', 'region': 'us-east-1'}, {'retries': {'max_retries': 50, 'backoff_factor': 1.0, 'jitter': 1.0, 'retryable_exceptions': ['InvalidSignatureException', 'ThrottlingException', 'ModelTimeoutException', 'ServiceUnavailableException', 'ModelNotReadyException', 'ServiceQuotaExceededException', 'ModelErrorException', 'EndpointConnectionError']}, 'rate_limit': {'max_rate': 50, 'time_period': 60}, 'role_arn': 'arn:aws:iam::615299746603:role/ModelFactoryBedrockAccessRole', 'region': 'us-west-2'}, {'retries': {'max_retries': 50, 'backoff_factor': 1.0, 'jitter': 1.0, 'retryable_exceptions': ['InvalidSignatureException', 'ThrottlingException', 'ModelTimeoutException', 'ServiceUnavailableException', 'ModelNotReadyException', 'ServiceQuotaExceededException', 'ModelErrorException', 'EndpointConnectionError']}, 'rate_limit': {'max_rate': 50, 'time_period': 60}, 'role_arn': 'arn:aws:iam::710271919393:role/ModelFactoryBedrockAccessRole', 'region': 'us-east-1'}, {'retries': {'max_retries': 50, 'backoff_factor': 1.0, 'jitter': 1.0, 'retryable_exceptions': ['InvalidSignatureException', 'ThrottlingException', 'ModelTimeoutException', 'ServiceUnavailableException', 'ModelNotReadyException', 'ServiceQuotaExceededException', 'ModelErrorException', 'EndpointConnectionError']}, 'rate_limit': {'max_rate': 50, 'time_period': 60}, 'role_arn': 'arn:aws:iam::710271919393:role/ModelFactoryBedrockAccessRole', 'region': 'us-west-2'}, {'retries': {'max_retries': 50, 'backoff_factor': 1.0, 'jitter': 1.0, 'retryable_exceptions': ['InvalidSignatureException', 'ThrottlingException', 'ModelTimeoutException', 'ServiceUnavailableException', 'ModelNotReadyException', 'ServiceQuotaExceededException', 'ModelErrorException', 'EndpointConnectionError']}, 'rate_limit': {'max_rate': 50, 'time_period': 60}, 'role_arn': 'arn:aws:iam::872515274170:role/ModelFactoryBedrockAccessRole', 'region': 'us-east-1'}, {'retries': {'max_retries': 50, 'backoff_factor': 1.0, 'jitter': 1.0, 'retryable_exceptions': ['InvalidSignatureException', 'ThrottlingException', 'ModelTimeoutException', 'ServiceUnavailableException', 'ModelNotReadyException', 'ServiceQuotaExceededException', 'ModelErrorException', 'EndpointConnectionError']}, 'rate_limit': {'max_rate': 50, 'time_period': 60}, 'role_arn': 'arn:aws:iam::872515274170:role/ModelFactoryBedrockAccessRole', 'region': 'us-west-2'}]}, 'student_config': {'provider_type': 'LAMBDA', 'model_id': 'mistralai/Mistral-Nemo-Instruct-2407', 'model_params': {'temperature': 1.0, 'max_tokens': 1024, 'top_p': 0.9}, 'provider_params': {'retries': {'max_retries': 3, 'backoff_factor': 1.0, 'jitter': 1.0, 'retryable_exceptions': ['InvalidSignatureException', 'ThrottlingException', 'ModelTimeoutException', 'ServiceUnavailableException', 'ModelNotReadyException', 'ServiceQuotaExceededException', 'ModelErrorException', 'EndpointConnectionError']}, 'rate_limit': {'max_rate': 10000, 'time_period': 60}, 'role_arn': None, 'region': 'us-west-2', 'function_arn': 'arn:aws:lambda:us-west-2:136238946932:function:MistralNemo'}}, 'optimizer_type': 'MIPRO_V2', 'optimizer_params': {'optimizer_metric': 'ACCURACY', 'num_candidates': 10, 'max_errors': 10, 'minibatch': False, 'auto': None}}}
2025/04/16 17:18:10 INFO dspy.teleprompt.mipro_optimizer_v2: 
==> STEP 1: BOOTSTRAP FEWSHOT EXAMPLES <==
2025/04/16 17:18:10 INFO dspy.teleprompt.mipro_optimizer_v2: These will be used as few-shot example candidates for our program and for creating instructions.

2025/04/16 17:18:10 INFO dspy.teleprompt.mipro_optimizer_v2: Bootstrapping N=10 sets of demonstrations...
Bootstrapping set 1/10
Bootstrapping set 2/10
Bootstrapping set 3/10

  0%|          | 0/349 [00:00<?, ?it/s]
  0%|          | 1/349 [00:07<43:47,  7.55s/it]/Users/rajsiba/miniconda3/envs/fmcore/lib/python3.11/site-packages/aiolimiter/leakybucket.py:102: RuntimeWarning: This AsyncLimiter instance is being re-used across loops. Please create a new limiter per event loop as re-use can lead to undefined behaviour.
  _warn_reuse()

  1%|          | 2/349 [00:13<37:04,  6.41s/it]/Users/rajsiba/miniconda3/envs/fmcore/lib/python3.11/site-packages/aiolimiter/leakybucket.py:102: RuntimeWarning: This AsyncLimiter instance is being re-used across loops. Please create a new limiter per event loop as re-use can lead to undefined behaviour.
  _warn_reuse()

  1%|          | 3/349 [00:20<38:45,  6.72s/it]/Users/rajsiba/miniconda3/envs/fmcore/lib/python3.11/site-packages/aiolimiter/leakybucket.py:102: RuntimeWarning: This AsyncLimiter instance is being re-used across loops. Please create a new limiter per event loop as re-use can lead to undefined behaviour.
  _warn_reuse()
