diff --git a/llm_core/README b/llm_core/README new file mode 100644 index 000000000..13b35e322 --- /dev/null +++ b/llm_core/README @@ -0,0 +1,214 @@ +## Content + +- [Overview](#overview) +- [Key Features](#key-features) +- [Core Components](#core-components) + - [1. Configuration Files](#1-configuration-files) + - [2. `ModelConfig` and Type System](#2-modelconfig-and-type-system) + - [3. Loaders](#3-loaders) + - [4. Utils](#4-utils) + - [5. Core Logic](#5-core-logic) + - [6. Callbacks](#6-callbacks) +- [Usage](#usage) + - [1. Configuration](#1-configuration) + - [2. Using LLMs in Modules](#2-using-llms-in-modules) +- [Extending with New Providers](#extending-with-new-providers) + +## Overview + +The `llm_core` module provides a robust and extensible system for configuring and managing Large Language Models (LLMs) within the Athena framework. It allows for defining task-specific LLM models, configure their capabilities, and seamlessly integrate new LLM providers. This README provides a comprehensive guide to understanding and utilizing the `llm_core` module's features. + +## Key Features + +1. **Granular LLM Model Selection for Tasks:** + * Define different LLM models for distinct tasks within each module (e.g., `module_modeling_llm`, `module_programming_llm`). +2. **Flexible and Comprehensive LLM Model Configuration:** + * Support a diverse range of LLM models with varying capabilities by configuring model settings (e.g., `temperature`, `top_p`) and capability flags (e.g., `supports_function_calling`, `supports_structured_output`) through YAML files. +3. **Preserved Dynamic Configuration Overrides:** + * Retain the ability to dynamically override LLM configurations via `x-` headers in API requests (used in the Athena playground). +4. **Multi-Provider Support:** + * Easily add support for new LLM providers by extending the `ModelConfig` class. +5. **Version-Controlled Configuration:** + * LLM configurations are managed through YAML files under version control, ensuring consistent deployments and eliminating environment-specific discrepancies. + +## Core Components + +### 1. Configuration Files + +The `llm_core` module uses two types of YAML files to manage LLM configurations: + +* **`llm_capabilities.yml` (llm\_core level):** + * Defines the core capabilities and default settings for different LLM models. + * Specifies model-specific overrides to default settings. + * Resides at the top level of the `llm_core` directory, shared across all modules. + * Example: + + ```yaml + defaults: + temperature: 0.7 + top_p: 1.0 + supports_function_calling: true + supports_structured_output: true + + models: + openai_o1-mini: + max_completion_tokens: 8000 + temperature: 1.0 + top_p: 1.0 + n: 1 + presence_penalty: 0.0 + frequency_penalty: 0.0 + supports_system_messages: false + supports_function_calling: false + supports_structured_output: false + ``` + +* **`llm_config.yml` (Module-specific):** + * Defines the concrete LLM models to be used for different tasks within a specific module (e.g., `module_modeling_llm`). + * Located at the root level of each module. + * Example (in `module_modeling_llm/llm_config.yml`): + + ```yaml + models: + base_model: "azure_openai_gpt-4o" + mini_model: "openai_o1-mini" + ``` + +### 2. `ModelConfig` and Type System + +The `ModelConfig` class serves as an abstraction layer for different LLM providers. It defines a common interface for interacting with LLMs and explicitly declares their capabilities. + +* **`model_config.py` (llm\_core/models):** + * Contains the abstract `ModelConfig` class. + * Defines abstract methods like: + * `get_model()`: Returns an instance of the configured LLM. + * `supports_system_messages()`: Indicates if the model supports system messages. + * `supports_function_calling()`: Indicates if the model supports function calling. + * `supports_structured_output()`: Indicates if the model supports structured output. + +* **`providers/` (llm\_core/models/providers):** + * Contains provider-specific implementations of `ModelConfig`. + * **`openai_model_config.py`:** + * Implements `ModelConfig` for OpenAI models (including Azure OpenAI). + * `get_model()` constructs `ChatOpenAI` or `AzureChatOpenAI` objects based on YAML configurations. + * Implements capability flag methods based on the model's features. + * Uses the `get_model_capabilities` function from `llm_capabilities_loader.py` to merge default and model-specific settings. + * Adding new providers: + * Create a new subclass of `ModelConfig` in the `providers` directory. + * Implement the abstract methods to define the provider's capabilities and model instantiation logic. + +### 3. Loaders + +* **`llm_capabilities_loader.py` (llm\_core/loaders):** + * Loads the `llm_capabilities.yml` file. + * Provides the `get_model_capabilities(model_key)` function to retrieve the merged capabilities (defaults and overrides) for a given model. + +* **`llm_config_loader.py` (llm\_core/loaders):** + * Loads the module-specific `llm_config.yml` file. + * Uses the `create_config_for_model` function to create `ModelConfig` instances based on the model names specified in the YAML file. + * Caches the loaded `LLMConfig` to avoid reloading on subsequent calls. + * `get_llm_config(path: Optional[str] = None) -> LLMConfig` is the public function for accessing the cached `LLMConfig` instance. If not loaded yet, loads from YAML and materializes it. + +* **`openai_loader.py` (llm\_core/loaders):** + * Discovers available OpenAI and Azure OpenAI models using environment variables (`OPENAI_API_KEY`, `AZURE_OPENAI_API_KEY`, etc.). + * Creates an `OpenAIModel` Enum for referencing discovered models. + +### 4. Utils + +* **`model_factory.py` (llm\_core/utils):** + * Provides the `create_config_for_model(model_name)` function. + * Determines the provider (OpenAI, Azure, etc.) based on the `model_name` prefix. + * Creates the appropriate `ModelConfig` subclass instance (e.g., `OpenAIModelConfig`). + * Handles unknown providers by raising a `ValueError`. + +* **`append_format_instructions.py` (llm\_core/utils):** + * Appends format instructions to the chat prompts + +* **`llm_utils.py` (llm\_core/utils):** + * Provides utility functions for: + * Calculating the number of tokens in a string or prompt (`num_tokens_from_string`, `num_tokens_from_prompt`). + * Checking prompt length and omitting features if necessary (`check_prompt_length_and_omit_features_if_necessary`). + * Removing system messages if the model doesn't support them (`remove_system_message`). + +### 5. Core Logic + +* **`predict_and_parse.py` (llm\_core/core):** + * Provides the `predict_and_parse` function for making LLM predictions and parsing the output using a Pydantic model. + * Handles models with and without native structured output/function calling support. + * Adds appropriate tags to the LLM run based on the experiment environment. + +### 6. Callbacks + +* **`callbacks.py` (llm\_core/models):** + * Provides the `UsageHandler` callback to track and emit LLM usage metadata (input/output tokens, cost). + +## Usage + +### 1. Configuration + +1. **Define LLM Capabilities (`llm_capabilities.yml`):** + + * Specify default settings and capability flags for different LLM models. + * Add model-specific overrides as needed. +2. **Define Task-Specific Models (`llm_config.yml`):** + + * In each module directory (e.g., `module_modeling_llm`), create an `llm_config.yml` file. + * Specify the `base_model`, `mini_model`, `fast_reasoning_model`, and `long_reasoning_model` to use for different tasks within that module. +3. **Set Environment Variables:** + + * Define necessary environment variables for your LLM providers (e.g., `OPENAI_API_KEY`, `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT`). + * Use the `.env.example` file as a template. + +### 2. Using LLMs in Modules + +1. **Import `get_llm_config`:** + + ```python + from llm_core.loaders.llm_config_loader import get_llm_config + ``` + +2. **Load the LLM Configuration:** + + ```python + llm_config = get_llm_config() # Loads the llm_config.yml from the current module's directory + ``` + +3. **Access `ModelConfig` Instances:** + + ```python + base_model_config = llm_config.models.base_model_config + mini_model_config = llm_config.models.mini_model_config + ``` + +4. **Use `ModelConfig` to Get LLM Instances and Check Capabilities:** + + ```python + base_model = base_model_config.get_model() + + if base_model_config.supports_function_calling(): + # Use function calling features + ``` + +## Extending with New Providers + +1. **Create a new `ModelConfig` subclass:** + + * In `llm_core/models/providers`, create a new Python file (e.g., `my_provider_model_config.py`). + * Define a new class that inherits from `llm_core.models.model_config.ModelConfig`. + * Implement the abstract methods: + * `get_model()`: Instantiate and return your provider's LLM object. + * `supports_system_messages()`: Return `True` if the provider supports system messages, `False` otherwise. + * `supports_function_calling()`: Return `True` if the provider supports function calling, `False` otherwise. + * `supports_structured_output()`: Return `True` if the provider supports structured output, `False` otherwise. +2. **Update `model_factory.py`:** + + * Add a new `elif` block to `find_provider_for_model` to recognize your provider's model name prefix. + * Add a corresponding `elif` block to `create_config_for_model` to instantiate your new `ModelConfig` subclass. +3. **Define Capabilities in `llm_capabilities.yml`:** + + * Add entries for your provider's models, specifying their default settings and capabilities. +4. **Use Your New Provider:** + + * In your module's `llm_config.yml`, specify the model name using your provider's prefix. + * The `llm_core` system will automatically use your new `ModelConfig` implementation. + diff --git a/llm_core/llm_capabilities.yml b/llm_core/llm_capabilities.yml new file mode 100644 index 000000000..33e9569b5 --- /dev/null +++ b/llm_core/llm_capabilities.yml @@ -0,0 +1,18 @@ +defaults: + temperature: 0.7 + top_p: 1.0 + + supports_function_calling: true + supports_structured_output: true + +models: + openai_o1-mini: + max_completion_tokens: 8000 + temperature: 1.0 + top_p: 1.0 + n: 1 + presence_penalty: 0.0 + frequency_penalty: 0.0 + supports_system_messages: false + supports_function_calling: false + supports_structured_output: false \ No newline at end of file diff --git a/llm_core/llm_core/core/predict_and_parse.py b/llm_core/llm_core/core/predict_and_parse.py new file mode 100644 index 000000000..cd1e82bc6 --- /dev/null +++ b/llm_core/llm_core/core/predict_and_parse.py @@ -0,0 +1,58 @@ +from typing import Optional, Type, TypeVar, List +from langchain_core.language_models import BaseLanguageModel +from langchain_core.prompts import ChatPromptTemplate +from langchain_core.pydantic_v1 import BaseModel, ValidationError +from langchain_core.runnables import RunnableSequence +from langchain_core.output_parsers import PydanticOutputParser +from athena import get_experiment_environment +from llm_core.utils.append_format_instructions import append_format_instructions +from llm_core.utils.llm_utils import remove_system_message +from llm_core.models.model_config import ModelConfig + +T = TypeVar("T", bound=BaseModel) + +async def predict_and_parse( + model: ModelConfig, + chat_prompt: ChatPromptTemplate, + prompt_input: dict, + pydantic_object: Type[T], + tags: Optional[List[str]], + ) -> Optional[T]: + """ + Predicts an LLM completion using the model and parses the output using the provided Pydantic model + """ + + # Remove system messages if the model does not support them + if not model.supports_system_messages(): + chat_prompt = remove_system_message(chat_prompt) + + llm_model = model.get_model() + + # Add tags + experiment = get_experiment_environment() + tags = tags or [] + if experiment.experiment_id is not None: + tags.append(f"experiment-{experiment.experiment_id}") + if experiment.module_configuration_id is not None: + tags.append(f"module-configuration-{experiment.module_configuration_id}") + if experiment.run_id is not None: + tags.append(f"run-{experiment.run_id}") + + # Currently structured output and function calling both expect the expected json to be in the prompt input + chat_prompt = append_format_instructions(chat_prompt, pydantic_object) + + # Run the model and parse the output + if model.supports_structured_output(): + structured_output_llm = llm_model.with_structured_output(pydantic_object, method = "json_mode") + elif model.supports_function_calling(): + structured_output_llm = llm_model.with_structured_output(pydantic_object) + else: + structured_output_llm = RunnableSequence(llm_model, PydanticOutputParser(pydantic_object=pydantic_object)) + + chain = RunnableSequence(chat_prompt, structured_output_llm) + + try: + return await chain.ainvoke(prompt_input, config={"tags": tags}, debug=True) + except ValidationError as e: + raise ValueError(f"Could not parse output: {e}") from e + diff --git a/llm_core/llm_core/loaders/llm_capabilities_loader.py b/llm_core/llm_core/loaders/llm_capabilities_loader.py new file mode 100644 index 000000000..138fb7350 --- /dev/null +++ b/llm_core/llm_core/loaders/llm_capabilities_loader.py @@ -0,0 +1,28 @@ +import os +import yaml +from pathlib import Path +from typing import Dict + + +CONFIG_PATH = Path(__file__).resolve().parents[2] / "llm_capabilities.yml" +if os.path.exists(CONFIG_PATH): + with open(CONFIG_PATH, "r", encoding="utf-8") as f: + _yaml_config = yaml.safe_load(f) +else: + _yaml_config = {} + +DEFAULTS = _yaml_config.get("defaults", {}) +MODEL_OVERRIDES = _yaml_config.get("models", {}) + + +def get_model_capabilities(model_key: str) -> Dict: + """ + Return the merged dictionary of defaults and overrides + for the given model key. + """ + caps = dict(DEFAULTS) # start with a copy of the defaults + if model_key in MODEL_OVERRIDES: + # override with model-specific entries + for k, v in MODEL_OVERRIDES[model_key].items(): + caps[k] = v + return caps \ No newline at end of file diff --git a/llm_core/llm_core/loaders/llm_config_loader.py b/llm_core/llm_core/loaders/llm_config_loader.py new file mode 100644 index 000000000..f4044d946 --- /dev/null +++ b/llm_core/llm_core/loaders/llm_config_loader.py @@ -0,0 +1,64 @@ +from llm_core.models.llm_config import LLMConfig, LLMConfigModel, RawLLMConfig +from llm_core.utils.model_factory import create_config_for_model +import yaml +from pathlib import Path +from typing import Dict, Optional + +_state: Dict[str, Optional[LLMConfig]] = {"llm_config": None} + +def _load_raw_llm_config(path: Optional[str] = None) -> RawLLMConfig: + """ + Loads the LLM configuration from a YAML file and returns a validated LLMConfig. + Raises pydantic.ValidationError if something is incorrect in the YAML. + """ + # By default, we assume 'llm_config.yml' is in the same directory + if path is None: + path = "llm_config.yml" + + config_path = Path(path).resolve() + if not config_path.exists(): + raise FileNotFoundError(f"LLM config file not found at: {config_path}") + + with config_path.open("r", encoding="utf-8") as f: + raw_data = yaml.safe_load(f) + + # Validate and parse the Pydantic model + return RawLLMConfig(**raw_data) + +def _materialize_llm_config(raw_config: RawLLMConfig) -> LLMConfig: + base_model = raw_config.models.base_model + if not base_model: + raise ValueError("Missing required 'base_model' in models") + + models_obj = LLMConfigModel( + base_model_config=create_config_for_model(base_model), + mini_model_config=( + create_config_for_model(raw_config.models.mini_model) + if raw_config.models.mini_model + else None + ), + fast_reasoning_model_config=( + create_config_for_model(raw_config.models.fast_reasoning_model) + if raw_config.models.fast_reasoning_model + else None + ), + long_reasoning_model_config=( + create_config_for_model(raw_config.models.long_reasoning_model) + if raw_config.models.long_reasoning_model + else None + ), + ) + + return LLMConfig(models=models_obj) + +def get_llm_config(path: Optional[str] = None) -> LLMConfig: + """ + Public function: returns a cached LLMConfig instance. + If not loaded yet, loads from YAML and materializes it. + """ + # Here we read/write _state["llm_config"] without using 'global'. + if _state["llm_config"] is None: + raw_config = _load_raw_llm_config(path) + _state["llm_config"] = _materialize_llm_config(raw_config) + + return _state["llm_config"] \ No newline at end of file diff --git a/llm_core/llm_core/loaders/model_loaders/azure_loader.py b/llm_core/llm_core/loaders/model_loaders/azure_loader.py new file mode 100644 index 000000000..42b2b20e3 --- /dev/null +++ b/llm_core/llm_core/loaders/model_loaders/azure_loader.py @@ -0,0 +1,60 @@ +from enum import Enum +import os +import requests +from typing import Dict, List + +from langchain.base_language import BaseLanguageModel +from langchain_openai import AzureChatOpenAI + +from athena.logger import logger + + +# Discover available models from Azure +AZURE_OPENAI_PREFIX = "azure_openai_" +azure_available = bool(os.environ.get("AZURE_OPENAI_API_KEY")) + +azure_available_models: Dict[str, BaseLanguageModel] = {} + +# Azure OpenAI +if azure_available: + def _get_azure_openai_deployments() -> List[str]: + base_url = f"{os.environ.get('AZURE_OPENAI_ENDPOINT')}/openai" + headers = {"api-key": os.environ["AZURE_OPENAI_API_KEY"]} + + models_response = requests.get( + f"{base_url}/models?api-version=2023-03-15-preview", + headers=headers, timeout=30 + ) + models_data = models_response.json()["data"] + + deployments_response = requests.get( + f"{base_url}/deployments?api-version=2023-03-15-preview", + headers=headers, timeout=30 + ) + deployments_data = deployments_response.json()["data"] + + # If deployment["model"] is substring of model["id"], we consider it a chat completion model + chat_completion_models = ",".join( + m["id"] for m in models_data if m["capabilities"]["chat_completion"] + ) + return [ + d["id"] + for d in deployments_data + if d["model"] in chat_completion_models + ] + + for deployment in _get_azure_openai_deployments(): + azure_available_models[AZURE_OPENAI_PREFIX + deployment] = AzureChatOpenAI(azure_deployment=deployment) + + +if azure_available_models: + logger.info("Available azure models: %s", ", ".join(azure_available_models.keys())) +else: + logger.warning("No azure models discovered.") + +# Enum for referencing the discovered models +if azure_available_models: + AzureModel = Enum('AzureModel', {name: name for name in azure_available_models}) # type: ignore +else: + class AzureModel(str, Enum): + pass \ No newline at end of file diff --git a/llm_core/llm_core/loaders/model_loaders/openai_loader.py b/llm_core/llm_core/loaders/model_loaders/openai_loader.py new file mode 100644 index 000000000..31802af4e --- /dev/null +++ b/llm_core/llm_core/loaders/model_loaders/openai_loader.py @@ -0,0 +1,36 @@ +from enum import Enum +import os +import openai +from typing import Dict + +from langchain.base_language import BaseLanguageModel +from langchain_openai import ChatOpenAI + +from athena.logger import logger + + +# Discover available models from OpenAI +OPENAI_PREFIX = "openai_" +openai_available = bool(os.environ.get("OPENAI_API_KEY")) + +openai_available_models: Dict[str, BaseLanguageModel] = {} + +# OpenAI +if openai_available: + openai.api_type = "openai" + for model in openai.models.list(): + if "gpt" in model.id or "o1" in model.id: + openai_available_models[OPENAI_PREFIX + model.id] = ChatOpenAI(model=model.id) + + +if openai_available_models: + logger.info("Available openai models: %s", ", ".join(openai_available_models.keys())) +else: + logger.warning("No openai models discovered.") + +# Enum for referencing the discovered models +if openai_available_models: + OpenAIModel = Enum('OpenAIModel', {name: name for name in openai_available_models}) # type: ignore +else: + class OpenAIModel(str, Enum): + pass \ No newline at end of file diff --git a/llm_core/llm_core/models/__init__.py b/llm_core/llm_core/models/__init__.py index c5cdc3d97..a546bf0b1 100644 --- a/llm_core/llm_core/models/__init__.py +++ b/llm_core/llm_core/models/__init__.py @@ -1,39 +1,7 @@ -import os -from typing import Type, Union, List, Optional -from langchain.base_language import BaseLanguageModel +from typing import Union -from llm_core.models.model_config import ModelConfig +from llm_core.models.providers.azure_model_config import AzureModelConfig +from llm_core.models.providers.openai_model_config import OpenAIModelConfig -DefaultModelConfig: Type[ModelConfig] -MiniModelConfig: ModelConfig -default_model_name = os.environ.get("LLM_DEFAULT_MODEL") -evaluation_model_name = os.environ.get("LLM_EVALUATION_MODEL") - -# Model used during evaluation for judging the output (should be a more powerful model) -evaluation_model: Optional[BaseLanguageModel] = None - -types: List[Type[ModelConfig]] = [] -try: - import llm_core.models.openai as openai_config - types.append(openai_config.OpenAIModelConfig) - if default_model_name in openai_config.available_models: - DefaultModelConfig = openai_config.OpenAIModelConfig - if evaluation_model_name in openai_config.available_models: - evaluation_model = openai_config.available_models[evaluation_model_name] -except AttributeError: - pass - -if not types: - raise EnvironmentError( - "No model configurations available, please set up at least one provider in the environment variables.") - -if 'DefaultModelConfig' not in globals(): - DefaultModelConfig = types[0] - -type0 = types[0] -if len(types) == 1: - ModelConfigType = type0 -else: - type1 = types[1] - ModelConfigType = Union[type0, type1] # type: ignore \ No newline at end of file +ModelConfigType = Union[OpenAIModelConfig, AzureModelConfig] \ No newline at end of file diff --git a/llm_core/llm_core/models/llm_config.py b/llm_core/llm_core/models/llm_config.py new file mode 100644 index 000000000..192588dfc --- /dev/null +++ b/llm_core/llm_core/models/llm_config.py @@ -0,0 +1,21 @@ +from pydantic import BaseModel +from typing import Optional +from llm_core.models.model_config import ModelConfig + +class RawModelsSection(BaseModel): + base_model: Optional[str] + mini_model: Optional[str] + fast_reasoning_model: Optional[str] + long_reasoning_model: Optional[str] + +class RawLLMConfig(BaseModel): + models: RawModelsSection + +class LLMConfigModel(BaseModel): + base_model_config: ModelConfig + mini_model_config: Optional[ModelConfig] + fast_reasoning_model_config: Optional[ModelConfig] + long_reasoning_model_config: Optional[ModelConfig] + +class LLMConfig(BaseModel): + models: LLMConfigModel \ No newline at end of file diff --git a/llm_core/llm_core/models/model_config.py b/llm_core/llm_core/models/model_config.py index f433ab587..574818a19 100644 --- a/llm_core/llm_core/models/model_config.py +++ b/llm_core/llm_core/models/model_config.py @@ -4,7 +4,32 @@ class ModelConfig(BaseModel, ABC): - + """ + Abstract base class representing a general model configuration. + """ + @abstractmethod def get_model(self) -> BaseLanguageModel: pass + + @abstractmethod + def supports_system_messages(self) -> bool: + """ + Indicates whether this model supports system prompt messages + (for example, the 'system' role, in gpt4o) + """ + pass + + @abstractmethod + def supports_function_calling(self) -> bool: + """ + Indicates whether this model supports function-calling + """ + pass + + @abstractmethod + def supports_structured_output(self) -> bool: + """ + Indicates whether this model supports producing structured output + """ + pass \ No newline at end of file diff --git a/llm_core/llm_core/models/openai.py b/llm_core/llm_core/models/openai.py deleted file mode 100644 index 7bcc0f11f..000000000 --- a/llm_core/llm_core/models/openai.py +++ /dev/null @@ -1,145 +0,0 @@ -import os -import openai -import requests - -from typing import Dict, List -from enum import Enum -from pydantic import Field, validator, PositiveInt -from langchain.base_language import BaseLanguageModel -from langchain_openai import AzureChatOpenAI, ChatOpenAI - -from athena.logger import logger -from .model_config import ModelConfig -from .callbacks import UsageHandler - - -OPENAI_PREFIX = "openai_" -AZURE_OPENAI_PREFIX = "azure_openai_" -openai_available = bool(os.environ.get("OPENAI_API_KEY")) -azure_openai_available = bool(os.environ.get("AZURE_OPENAI_API_KEY")) - -available_models: Dict[str, BaseLanguageModel] = {} - -# Load Non-Azure OpenAI models -if openai_available: - openai.api_type = "openai" - for model in openai.models.list(): - if "gpt" in model.id: - available_models[OPENAI_PREFIX + model.id] = ChatOpenAI(model=model.id) - -# Load Azure OpenAI models -if azure_openai_available: - def _get_azure_openai_deployments() -> List[str]: - # If this breaks in the future we have to use azure-mgmt-cognitiveservices which needs 6 additional environment variables - base_url = f"{os.environ.get('AZURE_OPENAI_ENDPOINT')}/openai" - headers = { - "api-key": os.environ["AZURE_OPENAI_API_KEY"] - } - - models_response = requests.get(f"{base_url}/models?api-version=2023-03-15-preview", headers=headers, timeout=30) - models_data = models_response.json()["data"] - deployments_response = requests.get(f"{base_url}/deployments?api-version=2023-03-15-preview", headers=headers, - timeout=30) - deployments_data = deployments_response.json()["data"] - - # Check if deployment["model"] is a substring of model["id"], i.e. "gpt-4o" is substring "gpt-4o-2024-05-13" - chat_completion_models = ",".join(model["id"] for model in models_data if model["capabilities"]["chat_completion"]) - return [deployment["id"] for deployment in deployments_data if deployment["model"] in chat_completion_models] - - for deployment in _get_azure_openai_deployments(): - available_models[AZURE_OPENAI_PREFIX + deployment] = AzureChatOpenAI(azure_deployment=deployment) - -if available_models: - logger.info("Available openai models: %s", ", ".join(available_models.keys())) - - OpenAIModel = Enum('OpenAIModel', {name: name for name in available_models}) # type: ignore - default_model_name = None - if "LLM_DEFAULT_MODEL" in os.environ and os.environ["LLM_DEFAULT_MODEL"] in available_models: - default_model_name = os.environ["LLM_DEFAULT_MODEL"] - if default_model_name not in available_models: - default_model_name = list(available_models.keys())[0] - - default_openai_model = OpenAIModel[default_model_name] # type: ignore - - # Long descriptions will be displayed in the playground UI and are copied from the OpenAI docs - class OpenAIModelConfig(ModelConfig): - """OpenAI LLM configuration.""" - - model_name: OpenAIModel = Field(default=default_openai_model, # type: ignore - description="The name of the model to use.") - max_tokens: PositiveInt = Field(4000, description="""\ -The maximum number of [tokens](https://platform.openai.com/tokenizer) to generate in the chat completion. - -The total length of input tokens and generated tokens is limited by the model's context length. \ -[Example Python code](https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb) for counting tokens.\ -""") - - temperature: float = Field(default=0.0, ge=0, le=2, description="""\ -What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, \ -while lower values like 0.2 will make it more focused and deterministic. - -We generally recommend altering this or `top_p` but not both.\ -""") - - top_p: float = Field(default=1, ge=0, le=1, description="""\ -An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. \ -So 0.1 means only the tokens comprising the top 10% probability mass are considered. - -We generally recommend altering this or `temperature` but not both.\ -""") - - presence_penalty: float = Field(default=0, ge=-2, le=2, description="""\ -Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far, \ -increasing the model's likelihood to talk about new topics. - -[See more information about frequency and presence penalties.](https://platform.openai.com/docs/api-reference/parameter-details)\ -""") - - frequency_penalty: float = Field(default=0, ge=-2, le=2, description="""\ -Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far, \ -decreasing the model's likelihood to repeat the same line verbatim. - -[See more information about frequency and presence penalties.](https://platform.openai.com/docs/api-reference/parameter-details)\ -""") - - @validator('max_tokens') - def max_tokens_must_be_positive(cls, v): - """ - Validate that max_tokens is a positive integer. - """ - if v <= 0: - raise ValueError('max_tokens must be a positive integer') - return v - - def get_model(self) -> BaseLanguageModel: - """Get the model from the configuration. - - Returns: - BaseLanguageModel: The model. - """ - model = available_models[self.model_name.value] - kwargs = model.__dict__ - secrets = {secret: getattr(model, secret) for secret in model.lc_secrets.keys()} - kwargs.update(secrets) - - model_kwargs = kwargs.get("model_kwargs", {}) - for attr, value in self.dict().items(): - if attr == "model_name": - # Skip model_name - continue - if hasattr(model, attr): - # If the model has the attribute, add it to kwargs - kwargs[attr] = value - else: - # Otherwise, add it to model_kwargs (necessary for chat models) - model_kwargs[attr] = value - kwargs["model_kwargs"] = model_kwargs - kwargs["callbacks"] = [UsageHandler()] - - # Initialize a copy of the model using the config - model = model.__class__(**kwargs) - return model - - - class Config: - title = 'OpenAI' \ No newline at end of file diff --git a/llm_core/llm_core/models/providers/azure_model_config.py b/llm_core/llm_core/models/providers/azure_model_config.py new file mode 100644 index 000000000..f0337749d --- /dev/null +++ b/llm_core/llm_core/models/providers/azure_model_config.py @@ -0,0 +1,137 @@ +from llm_core.loaders.model_loaders.openai_loader import OpenAIModel +from llm_core.loaders.llm_capabilities_loader import get_model_capabilities +from pydantic import Field, PrivateAttr, validator, PositiveInt +from langchain.base_language import BaseLanguageModel +from llm_core.loaders.model_loaders.azure_loader import AzureModel, azure_available_models + +from ..model_config import ModelConfig +from ..usage_handler import UsageHandler + +class AzureModelConfig(ModelConfig): + """OpenAI LLM configuration.""" + + model_name: AzureModel = Field( + description="The name of the model to use (e.g., openai_gpt-3.5-turbo)." + ) + + max_tokens: PositiveInt = Field( + 4000, + description="An upper bound for the number of tokens that can be generated for a completion, including visible output tokens and reasoning tokens." + ) + + temperature: float = Field( + default=0.0, ge=0, le=2, + description="""\ +What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, +while lower values like 0.2 will make it more focused and deterministic. + +We generally recommend altering this or `top_p` but not both.\ +""" + ) + + top_p: float = Field( + default=1, ge=0, le=1, + description="""\ +An alternative to sampling with temperature, called nucleus sampling, where the model considers the results +of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability +mass are considered. + +We generally recommend altering this or `temperature` but not both.\ +""" + ) + + presence_penalty: float = Field( + default=0, ge=-2, le=2, + description="""\ +Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far, +increasing the model's likelihood to talk about new topics. + +[See more information about frequency and presence penalties.](https://platform.openai.com/docs/api-reference/parameter-details)\ +""" + ) + + frequency_penalty: float = Field( + default=0, ge=-2, le=2, + description="""\ +Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far, +decreasing the model's likelihood to repeat the same line verbatim. + +[See more information about frequency and presence penalties.](https://platform.openai.com/docs/api-reference/parameter-details)\ +""" + ) + + # Private boolean flags + _supports_function_calling: bool = PrivateAttr(default=True) + _supports_structured_output: bool = PrivateAttr(default=True) + _supports_system_messages: bool = PrivateAttr(default=True) + + # Initialization: Merge YAML capabilities + def __init__(self, **data): + super().__init__(**data) + raw_key = self.model_name.value if hasattr(self.model_name, "value") else self.model_name + caps = get_model_capabilities(raw_key) + + fields_set_by_user = set(data.keys()) + + # If user didn't pass a certain field, override from YAML + if "temperature" not in fields_set_by_user and "temperature" in caps: + self.temperature = float(caps["temperature"]) + if "top_p" not in fields_set_by_user and "top_p" in caps: + self.top_p = float(caps["top_p"]) + if "presence_penalty" not in fields_set_by_user and "presence_penalty" in caps: + self.presence_penalty = float(caps["presence_penalty"]) + if "frequency_penalty" not in fields_set_by_user and "frequency_penalty" in caps: + self.frequency_penalty = float(caps["frequency_penalty"]) + if "max_tokens" not in fields_set_by_user and "max_tokens" in caps: + self.max_tokens = caps["max_tokens"] + + # set booleans from caps + self._supports_function_calling = bool(caps.get("supports_function_calling", True)) + self._supports_structured_output = bool(caps.get("supports_structured_output", True)) + self._supports_system_messages = bool(caps.get("supports_system_messages", True)) + + def supports_system_messages(self) -> bool: + return self._supports_system_messages + + def supports_function_calling(self) -> bool: + return self._supports_function_calling + + def supports_structured_output(self) -> bool: + return self._supports_structured_output + + @validator('max_tokens') + def max_tokens_must_be_positive(cls, v): + if v <= 0: + raise ValueError('max_tokens must be a positive integer') + return v + + def get_model(self) -> BaseLanguageModel: + """ + Create the ChatOpenAI/AzureChatOpenAI object. + Only add parameters that the model actually supports. + """ + model = azure_available_models[self.model_name.value] + kwargs = model.__dict__.copy() + + secrets = {secret: getattr(model, secret) for secret in model.lc_secrets.keys()} + kwargs.update(secrets) + + model_kwargs = kwargs.get("model_kwargs", {}) + + for attr, value in self.dict().items(): + if attr == "model_name": + continue + if hasattr(model, attr): + kwargs[attr] = value + else: + model_kwargs[attr] = value + + + kwargs["model_kwargs"] = model_kwargs + kwargs["callbacks"] = [UsageHandler()] + + new_model = model.__class__(**kwargs) + return new_model + + class Config: + title = 'Azure' \ No newline at end of file diff --git a/llm_core/llm_core/models/providers/openai_model_config.py b/llm_core/llm_core/models/providers/openai_model_config.py new file mode 100644 index 000000000..43d6f6372 --- /dev/null +++ b/llm_core/llm_core/models/providers/openai_model_config.py @@ -0,0 +1,137 @@ +from llm_core.loaders.model_loaders.openai_loader import OpenAIModel +from llm_core.loaders.llm_capabilities_loader import get_model_capabilities +from pydantic import Field, PrivateAttr, validator, PositiveInt +from langchain.base_language import BaseLanguageModel +from llm_core.loaders.model_loaders.openai_loader import openai_available_models + +from ..model_config import ModelConfig +from ..usage_handler import UsageHandler + +class OpenAIModelConfig(ModelConfig): + """OpenAI LLM configuration.""" + + model_name: OpenAIModel = Field( + description="The name of the model to use (e.g., openai_gpt-3.5-turbo)." + ) + + max_completion_tokens: PositiveInt = Field( + 4000, + description="An upper bound for the number of tokens that can be generated for a completion, including visible output tokens and reasoning tokens." + ) + + temperature: float = Field( + default=0.0, ge=0, le=2, + description="""\ +What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, +while lower values like 0.2 will make it more focused and deterministic. + +We generally recommend altering this or `top_p` but not both.\ +""" + ) + + top_p: float = Field( + default=1, ge=0, le=1, + description="""\ +An alternative to sampling with temperature, called nucleus sampling, where the model considers the results +of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability +mass are considered. + +We generally recommend altering this or `temperature` but not both.\ +""" + ) + + presence_penalty: float = Field( + default=0, ge=-2, le=2, + description="""\ +Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far, +increasing the model's likelihood to talk about new topics. + +[See more information about frequency and presence penalties.](https://platform.openai.com/docs/api-reference/parameter-details)\ +""" + ) + + frequency_penalty: float = Field( + default=0, ge=-2, le=2, + description="""\ +Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far, +decreasing the model's likelihood to repeat the same line verbatim. + +[See more information about frequency and presence penalties.](https://platform.openai.com/docs/api-reference/parameter-details)\ +""" + ) + + # Private boolean flags + _supports_function_calling: bool = PrivateAttr(default=True) + _supports_structured_output: bool = PrivateAttr(default=True) + _supports_system_messages: bool = PrivateAttr(default=True) + + # Initialization: Merge YAML capabilities + def __init__(self, **data): + super().__init__(**data) + raw_key = self.model_name.value if hasattr(self.model_name, "value") else self.model_name + caps = get_model_capabilities(raw_key) + + fields_set_by_user = set(data.keys()) + + # If user didn't pass a certain field, override from YAML + if "temperature" not in fields_set_by_user and "temperature" in caps: + self.temperature = float(caps["temperature"]) + if "top_p" not in fields_set_by_user and "top_p" in caps: + self.top_p = float(caps["top_p"]) + if "presence_penalty" not in fields_set_by_user and "presence_penalty" in caps: + self.presence_penalty = float(caps["presence_penalty"]) + if "frequency_penalty" not in fields_set_by_user and "frequency_penalty" in caps: + self.frequency_penalty = float(caps["frequency_penalty"]) + if "max_completion_tokens" not in fields_set_by_user and "max_completion_tokens" in caps: + self.max_completion_tokens = caps["max_completion_tokens"] + + # set booleans from caps + self._supports_function_calling = bool(caps.get("supports_function_calling", True)) + self._supports_structured_output = bool(caps.get("supports_structured_output", True)) + self._supports_system_messages = bool(caps.get("supports_system_messages", True)) + + def supports_system_messages(self) -> bool: + return self._supports_system_messages + + def supports_function_calling(self) -> bool: + return self._supports_function_calling + + def supports_structured_output(self) -> bool: + return self._supports_structured_output + + @validator('max_completion_tokens') + def max_completion_tokens_must_be_positive(cls, v): + if v <= 0: + raise ValueError('max_completion_tokens must be a positive integer') + return v + + def get_model(self) -> BaseLanguageModel: + """ + Create the ChatOpenAI/AzureChatOpenAI object. + Only add parameters that the model actually supports. + """ + model = openai_available_models[self.model_name.value] + kwargs = model.__dict__.copy() + + secrets = {secret: getattr(model, secret) for secret in model.lc_secrets.keys()} + kwargs.update(secrets) + + model_kwargs = kwargs.get("model_kwargs", {}) + + for attr, value in self.dict().items(): + if attr == "model_name": + continue + if hasattr(model, attr): + kwargs[attr] = value + else: + model_kwargs[attr] = value + + + kwargs["model_kwargs"] = model_kwargs + kwargs["callbacks"] = [UsageHandler()] + + new_model = model.__class__(**kwargs) + return new_model + + class Config: + title = 'OpenAI' \ No newline at end of file diff --git a/llm_core/llm_core/models/callbacks.py b/llm_core/llm_core/models/usage_handler.py similarity index 100% rename from llm_core/llm_core/models/callbacks.py rename to llm_core/llm_core/models/usage_handler.py diff --git a/llm_core/llm_core/utils/append_format_instructions.py b/llm_core/llm_core/utils/append_format_instructions.py new file mode 100644 index 000000000..8cf44bec8 --- /dev/null +++ b/llm_core/llm_core/utils/append_format_instructions.py @@ -0,0 +1,42 @@ +from typing import Type, TypeVar +from langchain.prompts import ( + ChatPromptTemplate, + SystemMessagePromptTemplate, + HumanMessagePromptTemplate, +) +from langchain.output_parsers import PydanticOutputParser +from pydantic import BaseModel + +T = TypeVar("T", bound=BaseModel) + +def append_format_instructions( + chat_prompt: ChatPromptTemplate, + pydantic_object: Type[T] +) -> ChatPromptTemplate: + """ + Appends format instructions to the first message of an existing chat prompt + """ + output_parser = PydanticOutputParser(pydantic_object=pydantic_object) + format_instructions = output_parser.get_format_instructions() + + messages = chat_prompt.messages + if not messages: + raise ValueError("Cannot append format instructions to an empty chat prompt") + + # Append format instructions to the first message + first_message = messages[0] + if isinstance(first_message, SystemMessagePromptTemplate): + new_first_message = SystemMessagePromptTemplate.from_template( + first_message.prompt.template + "\n\n{format_instructions}" # type: ignore[attr-defined] + ) + elif isinstance(first_message, HumanMessagePromptTemplate): + new_first_message = HumanMessagePromptTemplate.from_template( + first_message.prompt.template + "\n\n{format_instructions}" # type: ignore[attr-defined] + ) + else: + raise ValueError(f"Unsupported message type for appending format instructions: {type(first_message)}") + + new_first_message.prompt.partial_variables = {"format_instructions": format_instructions} # type: ignore[attr-defined] + new_first_message.prompt.input_variables.remove("format_instructions") # type: ignore[attr-defined] + + return ChatPromptTemplate.from_messages([new_first_message] + messages[1:]) diff --git a/llm_core/llm_core/utils/llm_utils.py b/llm_core/llm_core/utils/llm_utils.py index 0bfeb047d..227d18bf0 100644 --- a/llm_core/llm_core/utils/llm_utils.py +++ b/llm_core/llm_core/utils/llm_utils.py @@ -1,19 +1,15 @@ -from typing import Type, TypeVar, List +from typing import List, TypeVar from pydantic import BaseModel import tiktoken -from langchain_openai import AzureChatOpenAI, ChatOpenAI -from langchain.base_language import BaseLanguageModel from langchain.prompts import ( ChatPromptTemplate, SystemMessagePromptTemplate, HumanMessagePromptTemplate, ) -from langchain.output_parsers import PydanticOutputParser from athena import emit_meta T = TypeVar("T", bound=BaseModel) - def num_tokens_from_string(string: str) -> int: """Returns the number of tokens in a text string.""" encoding = tiktoken.get_encoding("cl100k_base") @@ -26,11 +22,13 @@ def num_tokens_from_prompt(chat_prompt: ChatPromptTemplate, prompt_input: dict) return num_tokens_from_string(chat_prompt.format(**prompt_input)) -def check_prompt_length_and_omit_features_if_necessary(prompt: ChatPromptTemplate, - prompt_input: dict, - max_input_tokens: int, - omittable_features: List[str], - debug: bool): +def check_prompt_length_and_omit_features_if_necessary( + prompt: ChatPromptTemplate, + prompt_input: dict, + max_input_tokens: int, + omittable_features: List[str], + debug: bool, +): """Check if the input is too long and omit features if necessary. Note: Omitted features will be replaced with "omitted" in the prompt @@ -43,7 +41,7 @@ def check_prompt_length_and_omit_features_if_necessary(prompt: ChatPromptTemplat debug (bool): Debug flag Returns: - (dict, bool): Tuple of (prompt_input, should_run) where prompt_input is the input with omitted features and + (dict, bool): Tuple of (prompt_input, should_run) where prompt_input is the input with omitted features and should_run is True if the model should run, False otherwise """ if num_tokens_from_prompt(prompt, prompt_input) <= max_input_tokens: @@ -66,45 +64,29 @@ def check_prompt_length_and_omit_features_if_necessary(prompt: ChatPromptTemplat return prompt_input, False -def supports_function_calling(model: BaseLanguageModel): - """Returns True if the model supports function calling, False otherwise - - Args: - model (BaseLanguageModel): The model to check +def get_chat_prompt( + system_message: str, + human_message: str, +) -> ChatPromptTemplate: + system_message_prompt = SystemMessagePromptTemplate.from_template(system_message) + human_message_prompt = HumanMessagePromptTemplate.from_template(human_message) + return ChatPromptTemplate.from_messages( + [system_message_prompt, human_message_prompt] + ) - Returns: - boolean: True if the model supports function calling, False otherwise +def remove_system_message(chat_prompt: ChatPromptTemplate) -> ChatPromptTemplate: """ - return isinstance(model, ChatOpenAI) or isinstance(model, AzureChatOpenAI) - - -def get_chat_prompt_with_formatting_instructions( - model: BaseLanguageModel, - system_message: str, - human_message: str, - pydantic_object: Type[T] - ) -> ChatPromptTemplate: - """Returns a ChatPromptTemplate with formatting instructions (if necessary) - - Note: Does nothing if the model supports function calling - - Args: - model (BaseLanguageModel): The model to check if it supports function calling - system_message (str): System message - human_message (str): Human message - pydantic_object (Type[T]): Model to parse the output - - Returns: - ChatPromptTemplate: ChatPromptTemplate with formatting instructions (if necessary) + Create a NEW ChatPromptTemplate with "system" messages replaced as "human" if the model does not support system messages. """ - if supports_function_calling(model): - system_message_prompt = SystemMessagePromptTemplate.from_template(system_message) - human_message_prompt = HumanMessagePromptTemplate.from_template(human_message) - return ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt]) - - output_parser = PydanticOutputParser(pydantic_object=pydantic_object) - system_message_prompt = SystemMessagePromptTemplate.from_template(system_message + "\n{format_instructions}") - system_message_prompt.prompt.partial_variables = {"format_instructions": output_parser.get_format_instructions()} - system_message_prompt.prompt.input_variables.remove("format_instructions") - human_message_prompt = HumanMessagePromptTemplate.from_template(human_message + "\n\nJSON response following the provided schema:") - return ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt]) \ No newline at end of file + new_prompt_templates = [] + for prompt_msg in chat_prompt.messages: + if isinstance(prompt_msg, SystemMessagePromptTemplate): + # Convert it to a human prompt, preserving content + new_prompt_templates.append( + HumanMessagePromptTemplate.from_template( + "[System message]: " + prompt_msg.prompt.template # type: ignore + ) + ) + else: + new_prompt_templates.append(prompt_msg) + return ChatPromptTemplate.from_messages(new_prompt_templates) diff --git a/llm_core/llm_core/utils/model_factory.py b/llm_core/llm_core/utils/model_factory.py new file mode 100644 index 000000000..ea6787efe --- /dev/null +++ b/llm_core/llm_core/utils/model_factory.py @@ -0,0 +1,35 @@ + +from typing import Optional +from llm_core.models.model_config import ModelConfig +from llm_core.models.providers.openai_model_config import OpenAIModelConfig +from llm_core.models.providers.azure_model_config import AzureModelConfig + +def find_provider_for_model(model_name: str) -> Optional[str]: + """ + Determine which provider a model_name belongs to based on its prefix or a naming pattern. + """ + model_name_lower = model_name.lower() + + if model_name_lower.startswith("openai_"): + return "openai" + elif model_name_lower.startswith("azure_openai_"): + return "azure_openai" + elif model_name_lower.startswith("replicate_"): + return "replicate" + else: + return None + +def create_config_for_model(model_name: str) -> ModelConfig: + """ + Create the appropriate ModelConfig subclass instance (OpenAI, Azure, etc.) + depending on the provider indicated by `model_name`. + """ + provider = find_provider_for_model(model_name) + if provider == "openai": + return OpenAIModelConfig(model_name=model_name) + elif provider == "azure_openai": + return AzureModelConfig(model_name=model_name) + elif provider == "ollama": + raise NotImplementedError("ollama support is not implemented yet.") + else: + raise ValueError(f"Unknown provider for model name '{model_name}'") \ No newline at end of file diff --git a/llm_core/llm_core/utils/predict_and_parse.py b/llm_core/llm_core/utils/predict_and_parse.py deleted file mode 100644 index e72bb3e91..000000000 --- a/llm_core/llm_core/utils/predict_and_parse.py +++ /dev/null @@ -1,66 +0,0 @@ -from typing import Optional, Type, TypeVar, List -from langchain_core.language_models import BaseLanguageModel -from langchain_core.prompts import ChatPromptTemplate -from langchain_core.pydantic_v1 import BaseModel, ValidationError -from langchain_core.runnables import RunnableSequence -from athena import get_experiment_environment - -T = TypeVar("T", bound=BaseModel) - -async def predict_and_parse( - model: BaseLanguageModel, - chat_prompt: ChatPromptTemplate, - prompt_input: dict, - pydantic_object: Type[T], - tags: Optional[List[str]], - use_function_calling: bool = False - ) -> Optional[T]: - """Predicts an LLM completion using the model and parses the output using the provided Pydantic model - - Args: - model (BaseLanguageModel): The model to predict with - chat_prompt (ChatPromptTemplate): Prompt to use - prompt_input (dict): Input parameters to use for the prompt - pydantic_object (Type[T]): Pydantic model to parse the output - tags (Optional[List[str]]: List of tags to tag the prediction with - - Returns: - Optional[T]: Parsed output, or None if it could not be parsed - """ - experiment = get_experiment_environment() - - tags = tags or [] - if experiment.experiment_id is not None: - tags.append(f"experiment-{experiment.experiment_id}") - if experiment.module_configuration_id is not None: - tags.append(f"module-configuration-{experiment.module_configuration_id}") - if experiment.run_id is not None: - tags.append(f"run-{experiment.run_id}") - - - if (use_function_calling): - structured_output_llm = model.with_structured_output(pydantic_object) - chain = chat_prompt | structured_output_llm - - try: - result = await chain.ainvoke(prompt_input, config={"tags": tags}) - - if isinstance(result, pydantic_object): - return result - else: - raise ValueError("Parsed output does not match the expected Pydantic model.") - - except ValidationError as e: - raise ValueError(f"Could not parse output: {e}") from e - - else: - structured_output_llm = model.with_structured_output(pydantic_object, method = "json_mode") - chain = RunnableSequence( - chat_prompt, - structured_output_llm - ) - try: - return await chain.ainvoke(prompt_input, config={"tags": tags}) - except ValidationError as e: - raise ValueError(f"Could not parse output: {e}") from e - diff --git a/llm_core/poetry.lock b/llm_core/poetry.lock index 8339acaaf..f26d5dd9c 100644 --- a/llm_core/poetry.lock +++ b/llm_core/poetry.lock @@ -1,4 +1,4 @@ -# This file is automatically @generated by Poetry 1.8.3 and should not be changed by hand. +# This file is automatically @generated by Poetry 1.8.5 and should not be changed by hand. [[package]] name = "aiohappyeyeballs" @@ -1260,13 +1260,13 @@ files = [ [[package]] name = "openai" -version = "1.51.1" +version = "1.58.1" description = "The official Python library for the openai API" optional = false -python-versions = ">=3.7.1" +python-versions = ">=3.8" files = [ - {file = "openai-1.51.1-py3-none-any.whl", hash = "sha256:035ba637bef7523282b5b8d9f2f5fdc0bb5bc18d52af2bfc7f64e4a7b0a169fb"}, - {file = "openai-1.51.1.tar.gz", hash = "sha256:a4908d68e0a1f4bcb45cbaf273c5fbdc3a4fa6239bb75128b58b94f7d5411563"}, + {file = "openai-1.58.1-py3-none-any.whl", hash = "sha256:e2910b1170a6b7f88ef491ac3a42c387f08bd3db533411f7ee391d166571d63c"}, + {file = "openai-1.58.1.tar.gz", hash = "sha256:f5a035fd01e141fc743f4b0e02c41ca49be8fab0866d3b67f5f29b4f4d3c0973"}, ] [package.dependencies] @@ -1281,6 +1281,7 @@ typing-extensions = ">=4.11,<5" [package.extras] datalib = ["numpy (>=1)", "pandas (>=1.2.3)", "pandas-stubs (>=1.1.0.11)"] +realtime = ["websockets (>=13,<15)"] [[package]] name = "orjson" @@ -2480,4 +2481,4 @@ propcache = ">=0.2.0" [metadata] lock-version = "2.0" python-versions = "3.11.*" -content-hash = "0d77a7b3cc17adfc22591419da73e887a3a0a4acf8657a122a9d056f97456ce6" +content-hash = "c0f287638fd6bcfb443d162e213c1a08e2c31be3368a69d92202842a10d9f582" diff --git a/llm_core/pyproject.toml b/llm_core/pyproject.toml index 359949dca..23b6b5609 100644 --- a/llm_core/pyproject.toml +++ b/llm_core/pyproject.toml @@ -14,7 +14,7 @@ langchain = "0.2.15" langchain-community="0.2.15" langchain-openai = "0.1.23" nltk = "3.9.1" -openai = "1.51.1" +openai = "1.58.1" python-dotenv = "1.0.0" tiktoken = "0.7.0" diff --git a/modules/modeling/module_modeling_llm/llm_config.yml b/modules/modeling/module_modeling_llm/llm_config.yml new file mode 100644 index 000000000..738f70f42 --- /dev/null +++ b/modules/modeling/module_modeling_llm/llm_config.yml @@ -0,0 +1,5 @@ +models: + base_model: azure_openai_gpt-4o + mini_model: azure_openai_gpt-4o + fast_reasoning_model: azure_openai_gpt-4o + long_reasoning_model: azure_openai_gpt-4o \ No newline at end of file diff --git a/modules/modeling/module_modeling_llm/module_modeling_llm/config.py b/modules/modeling/module_modeling_llm/module_modeling_llm/config.py index c7f0f4ad0..48e084295 100644 --- a/modules/modeling/module_modeling_llm/module_modeling_llm/config.py +++ b/modules/modeling/module_modeling_llm/module_modeling_llm/config.py @@ -1,13 +1,17 @@ +from llm_core.loaders.llm_config_loader import get_llm_config +from llm_core.models import ModelConfigType from pydantic import BaseModel, Field from athena import config_schema_provider -from llm_core.models import ModelConfigType, DefaultModelConfig +from llm_core.models.model_config import ModelConfig from module_modeling_llm.prompts import ( graded_feedback_prompt, filter_feedback_prompt, structured_grading_instructions_prompt ) +llm_config = get_llm_config() + class GenerateSuggestionsPrompt(BaseModel): """ Features available: **{problem_statement}**, **{example_solution}**, **{grading_instructions}**, **{max_points}**, @@ -44,7 +48,10 @@ class GenerateSuggestionsPrompt(BaseModel): class BasicApproachConfig(BaseModel): """This approach uses a LLM with a single prompt to generate feedback in a single step.""" max_input_tokens: int = Field(default=5000, description="Maximum number of tokens in the input prompt.") - model: ModelConfigType = Field(default=DefaultModelConfig()) # type: ignore + generate_feedback: ModelConfigType = Field(default=llm_config.models.base_model_config) + filter_feedback: ModelConfigType = Field(default=llm_config.models.base_model_config) + review_feedback: ModelConfigType = Field(default=llm_config.models.fast_reasoning_model_config) + generate_grading_instructions: ModelConfigType = Field(default=llm_config.models.base_model_config) generate_suggestions_prompt: GenerateSuggestionsPrompt = Field(default=GenerateSuggestionsPrompt()) @config_schema_provider diff --git a/modules/modeling/module_modeling_llm/module_modeling_llm/core/filter_feedback.py b/modules/modeling/module_modeling_llm/module_modeling_llm/core/filter_feedback.py index efe2b1871..54021102b 100644 --- a/modules/modeling/module_modeling_llm/module_modeling_llm/core/filter_feedback.py +++ b/modules/modeling/module_modeling_llm/module_modeling_llm/core/filter_feedback.py @@ -3,7 +3,7 @@ from athena import emit_meta from module_modeling_llm.config import BasicApproachConfig -from llm_core.utils.predict_and_parse import predict_and_parse +from llm_core.core.predict_and_parse import predict_and_parse from module_modeling_llm.models.assessment_model import AssessmentModel from module_modeling_llm.models.exercise_model import ExerciseModel from module_modeling_llm.prompts.filter_feedback_prompt import FilterFeedbackInputs @@ -24,11 +24,10 @@ async def filter_feedback( prompt_inputs = FilterFeedbackInputs( original_feedback=original_feedback.json(), - feedback_output_format=PydanticOutputParser(pydantic_object=AssessmentModel).get_format_instructions() ) feedback_result = await predict_and_parse( - model=config.model.get_model(), # type: ignore[attr-defined] + model=config.filter_feedback, chat_prompt=chat_prompt, prompt_input=prompt_inputs.dict(), pydantic_object=AssessmentModel, diff --git a/modules/modeling/module_modeling_llm/module_modeling_llm/core/generate_suggestions.py b/modules/modeling/module_modeling_llm/module_modeling_llm/core/generate_suggestions.py index 61f051348..000958c3d 100644 --- a/modules/modeling/module_modeling_llm/module_modeling_llm/core/generate_suggestions.py +++ b/modules/modeling/module_modeling_llm/module_modeling_llm/core/generate_suggestions.py @@ -6,7 +6,7 @@ from module_modeling_llm.config import BasicApproachConfig from module_modeling_llm.models.assessment_model import AssessmentModel from module_modeling_llm.prompts.apollon_format_description import apollon_format_description -from llm_core.utils.predict_and_parse import predict_and_parse +from llm_core.core.predict_and_parse import predict_and_parse from module_modeling_llm.prompts.graded_feedback_prompt import GradedFeedbackInputs from module_modeling_llm.models.exercise_model import ExerciseModel @@ -34,7 +34,6 @@ async def generate_suggestions( submission_uml_type=exercise_model.submission_uml_type, example_solution=exercise_model.transformed_example_solution, uml_diagram_format=apollon_format_description, - feedback_output_format=PydanticOutputParser(pydantic_object=AssessmentModel).get_format_instructions() ) chat_prompt = ChatPromptTemplate.from_messages([ @@ -42,7 +41,7 @@ async def generate_suggestions( ("human", config.generate_suggestions_prompt.graded_feedback_human_message)]) feedback_result = await predict_and_parse( - model=config.model.get_model(), # type: ignore[attr-defined] + model=config.generate_feedback, chat_prompt=chat_prompt, prompt_input=prompt_inputs.dict(), pydantic_object=AssessmentModel, diff --git a/modules/modeling/module_modeling_llm/module_modeling_llm/core/get_structured_grading_instructions.py b/modules/modeling/module_modeling_llm/module_modeling_llm/core/get_structured_grading_instructions.py index 75b7ded47..8965aaca3 100644 --- a/modules/modeling/module_modeling_llm/module_modeling_llm/core/get_structured_grading_instructions.py +++ b/modules/modeling/module_modeling_llm/module_modeling_llm/core/get_structured_grading_instructions.py @@ -8,7 +8,7 @@ from langchain_core.prompts import ChatPromptTemplate from athena.schemas import GradingCriterion, StructuredGradingCriterion -from llm_core.utils.predict_and_parse import predict_and_parse +from llm_core.core.predict_and_parse import predict_and_parse from module_modeling_llm.config import BasicApproachConfig from module_modeling_llm.models.exercise_model import ExerciseModel from module_modeling_llm.prompts.structured_grading_instructions_prompt import StructuredGradingInstructionsInputs @@ -42,11 +42,10 @@ async def get_structured_grading_instructions( grading_instructions=grading_instructions or "No grading instructions.", submission_uml_type=exercise_model.submission_uml_type, example_solution=exercise_model.transformed_example_solution or "No example solution.", - structured_instructions_output_format=PydanticOutputParser(pydantic_object=StructuredGradingCriterion).get_format_instructions() ) grading_instruction_result = await predict_and_parse( - model=config.model.get_model(), # type: ignore[attr-defined] + model=config.generate_grading_instructions, chat_prompt=chat_prompt, prompt_input=prompt_inputs.dict(), pydantic_object=StructuredGradingCriterion, diff --git a/modules/modeling/module_modeling_llm/module_modeling_llm/prompts/filter_feedback_prompt.py b/modules/modeling/module_modeling_llm/module_modeling_llm/prompts/filter_feedback_prompt.py index ef048ebe2..f14258528 100644 --- a/modules/modeling/module_modeling_llm/module_modeling_llm/prompts/filter_feedback_prompt.py +++ b/modules/modeling/module_modeling_llm/module_modeling_llm/prompts/filter_feedback_prompt.py @@ -3,7 +3,6 @@ class FilterFeedbackInputs(BaseModel): original_feedback: str - feedback_output_format: str filter_feedback_system_message = """ @@ -45,9 +44,6 @@ class FilterFeedbackInputs(BaseModel): "description": "Class 'Car' is correctly defined." Remember, the goal is to guide the student towards improvement without providing a complete solution or grading information in the feedback. Also keep the original structure of the feedback. Just modify the title and description values. - - -{feedback_output_format} """ filter_feedback_human_message = """ diff --git a/modules/modeling/module_modeling_llm/module_modeling_llm/prompts/graded_feedback_prompt.py b/modules/modeling/module_modeling_llm/module_modeling_llm/prompts/graded_feedback_prompt.py index 71662c84b..60d2d28eb 100644 --- a/modules/modeling/module_modeling_llm/module_modeling_llm/prompts/graded_feedback_prompt.py +++ b/modules/modeling/module_modeling_llm/module_modeling_llm/prompts/graded_feedback_prompt.py @@ -11,7 +11,6 @@ class GradedFeedbackInputs(BaseModel): submission_uml_type: str example_solution: Optional[str] = None uml_diagram_format: str - feedback_output_format: str graded_feedback_system_message = """ @@ -56,9 +55,6 @@ class GradedFeedbackInputs(BaseModel): The submission uses the following UML diagram format: {uml_diagram_format} - Note: Don't mention elements that have no name, by there articial name: e.g. ##A or ##B, instad just say e.g. the task in ... is missing ... - - -{feedback_output_format} """ graded_feedback_human_message = """ diff --git a/modules/modeling/module_modeling_llm/module_modeling_llm/prompts/structured_grading_instructions_prompt.py b/modules/modeling/module_modeling_llm/module_modeling_llm/prompts/structured_grading_instructions_prompt.py index 9db3e6bef..55ebc103f 100644 --- a/modules/modeling/module_modeling_llm/module_modeling_llm/prompts/structured_grading_instructions_prompt.py +++ b/modules/modeling/module_modeling_llm/module_modeling_llm/prompts/structured_grading_instructions_prompt.py @@ -8,15 +8,11 @@ class StructuredGradingInstructionsInputs(BaseModel): grading_instructions: str submission_uml_type: str example_solution: str - structured_instructions_output_format: str structured_grading_instructions_system_message = """You are an AI tutor for {submission_uml_type} modeling exercise assessment at a prestigious university. Create a structured grading instruction based on the given grading instructions (important), the solution diagram (if present) and the problem statement. The structured grading instruction should be highly detailed to ensure consistent grading across different tutors. - - -{structured_instructions_output_format} """ structured_grading_instructions_human_message = """ diff --git a/modules/modeling/module_modeling_llm/poetry.lock b/modules/modeling/module_modeling_llm/poetry.lock index ef71273fd..be2c5543f 100644 --- a/modules/modeling/module_modeling_llm/poetry.lock +++ b/modules/modeling/module_modeling_llm/poetry.lock @@ -1,4 +1,4 @@ -# This file is automatically @generated by Poetry 1.8.3 and should not be changed by hand. +# This file is automatically @generated by Poetry 1.8.5 and should not be changed by hand. [[package]] name = "aiohappyeyeballs" @@ -996,7 +996,7 @@ langchain-community = "0.2.15" langchain-openai = "0.1.23" langsmith = ">=0.1.0,<0.2.0" nltk = "3.9.1" -openai = "1.51.1" +openai = "1.58.1" python-dotenv = "1.0.0" tiktoken = "0.7.0" @@ -1270,13 +1270,13 @@ files = [ [[package]] name = "openai" -version = "1.51.1" +version = "1.58.1" description = "The official Python library for the openai API" optional = false -python-versions = ">=3.7.1" +python-versions = ">=3.8" files = [ - {file = "openai-1.51.1-py3-none-any.whl", hash = "sha256:035ba637bef7523282b5b8d9f2f5fdc0bb5bc18d52af2bfc7f64e4a7b0a169fb"}, - {file = "openai-1.51.1.tar.gz", hash = "sha256:a4908d68e0a1f4bcb45cbaf273c5fbdc3a4fa6239bb75128b58b94f7d5411563"}, + {file = "openai-1.58.1-py3-none-any.whl", hash = "sha256:e2910b1170a6b7f88ef491ac3a42c387f08bd3db533411f7ee391d166571d63c"}, + {file = "openai-1.58.1.tar.gz", hash = "sha256:f5a035fd01e141fc743f4b0e02c41ca49be8fab0866d3b67f5f29b4f4d3c0973"}, ] [package.dependencies] @@ -1291,6 +1291,7 @@ typing-extensions = ">=4.11,<5" [package.extras] datalib = ["numpy (>=1)", "pandas (>=1.2.3)", "pandas-stubs (>=1.1.0.11)"] +realtime = ["websockets (>=13,<15)"] [[package]] name = "orjson" diff --git a/modules/programming/module_programming_llm/llm_config.yml b/modules/programming/module_programming_llm/llm_config.yml new file mode 100644 index 000000000..25842669a --- /dev/null +++ b/modules/programming/module_programming_llm/llm_config.yml @@ -0,0 +1,5 @@ +models: + base_model: openai_gpt-4o + mini_model: openai_gpt-4o-mini + fast_reasoning_model: openai_o1-mini + long_reasoning_model: openai_o1-preview \ No newline at end of file diff --git a/modules/programming/module_programming_llm/module_programming_llm/config.py b/modules/programming/module_programming_llm/module_programming_llm/config.py index 6a632208e..28f355b10 100644 --- a/modules/programming/module_programming_llm/module_programming_llm/config.py +++ b/modules/programming/module_programming_llm/module_programming_llm/config.py @@ -1,9 +1,10 @@ from abc import ABC +from llm_core.loaders.llm_config_loader import get_llm_config from pydantic import BaseModel, Field +from llm_core.models import ModelConfigType from athena import config_schema_provider -from llm_core.models import ModelConfigType, DefaultModelConfig from module_programming_llm.prompts.generate_graded_suggestions_by_file import ( system_message as generate_graded_suggestions_by_file_system_message, human_message as generate_graded_suggestions_by_file_human_message, @@ -29,6 +30,8 @@ human_message as summarize_submission_by_file_human_message, ) +llm_config = get_llm_config() + class SplitProblemStatementsBasePrompt(BaseModel): """Base class for splitting problem statements into file-based ones, providing a structured approach for processing statements.""" @@ -107,7 +110,7 @@ class BasicApproachConfig(BaseModel): """Defines a basic configuration for processing submissions, incorporating problem statement splitting, feedback generation, and file summarization.""" max_input_tokens: int = Field(default=3000, description="Maximum number of tokens in the input prompt.") - model: ModelConfigType = Field(default=DefaultModelConfig()) # type: ignore + model: ModelConfigType = Field(default=llm_config.models.base_model_config) max_number_of_files: int = Field(default=25, description="Maximum number of files. If exceeded, it will prioritize the most important ones.") split_problem_statement_by_file_prompt: SplitProblemStatementsBasePrompt = Field(description="To be defined in " "subclasses.") generate_suggestions_by_file_prompt: SplitProblemStatementsBasePrompt = Field(description="To be defined in " "subclasses.") diff --git a/modules/programming/module_programming_llm/module_programming_llm/generate_graded_suggestions_by_file.py b/modules/programming/module_programming_llm/module_programming_llm/generate_graded_suggestions_by_file.py index eff86889c..0a6f905b2 100644 --- a/modules/programming/module_programming_llm/module_programming_llm/generate_graded_suggestions_by_file.py +++ b/modules/programming/module_programming_llm/module_programming_llm/generate_graded_suggestions_by_file.py @@ -15,10 +15,10 @@ ) from llm_core.utils.llm_utils import ( check_prompt_length_and_omit_features_if_necessary, - get_chat_prompt_with_formatting_instructions, + get_chat_prompt, num_tokens_from_string, ) -from llm_core.utils.predict_and_parse import predict_and_parse +from llm_core.core.predict_and_parse import predict_and_parse from module_programming_llm.helpers.utils import ( get_diff, @@ -64,13 +64,10 @@ async def generate_suggestions_by_file( config: GradedBasicApproachConfig, debug: bool, ) -> List[Feedback]: - model = config.model.get_model() # type: ignore[attr-defined] - chat_prompt = get_chat_prompt_with_formatting_instructions( - model=model, + chat_prompt = get_chat_prompt( system_message=config.generate_suggestions_by_file_prompt.system_message, human_message=config.generate_suggestions_by_file_prompt.human_message, - pydantic_object=AssessmentModel, ) # Get split problem statement and grading instructions by file (if necessary) @@ -264,11 +261,10 @@ async def generate_suggestions_by_file( results: List[Optional[AssessmentModel]] = await asyncio.gather( *[ predict_and_parse( - model=model, + model=config.model, chat_prompt=chat_prompt, prompt_input=prompt_input, pydantic_object=AssessmentModel, - use_function_calling=True, tags=[ f"exercise-{exercise.id}", f"submission-{submission.id}", diff --git a/modules/programming/module_programming_llm/module_programming_llm/generate_non_graded_suggestions_by_file.py b/modules/programming/module_programming_llm/module_programming_llm/generate_non_graded_suggestions_by_file.py index 4e965c438..68a606a37 100644 --- a/modules/programming/module_programming_llm/module_programming_llm/generate_non_graded_suggestions_by_file.py +++ b/modules/programming/module_programming_llm/module_programming_llm/generate_non_graded_suggestions_by_file.py @@ -13,10 +13,10 @@ ) from llm_core.utils.llm_utils import ( check_prompt_length_and_omit_features_if_necessary, - get_chat_prompt_with_formatting_instructions, + get_chat_prompt, num_tokens_from_string, ) -from llm_core.utils.predict_and_parse import predict_and_parse +from llm_core.core.predict_and_parse import predict_and_parse from module_programming_llm.helpers.utils import ( get_diff, @@ -58,13 +58,10 @@ async def generate_suggestions_by_file( config: NonGradedBasicApproachConfig, debug: bool, ) -> List[Feedback]: - model = config.model.get_model() # type: ignore[attr-defined] - chat_prompt = get_chat_prompt_with_formatting_instructions( - model=model, + chat_prompt = get_chat_prompt( system_message=config.generate_suggestions_by_file_prompt.system_message, human_message=config.generate_suggestions_by_file_prompt.human_message, - pydantic_object=ImprovementModel, ) prompt_inputs: List[dict] = [] @@ -215,11 +212,10 @@ async def generate_suggestions_by_file( results: List[Optional[ImprovementModel]] = await asyncio.gather( *[ predict_and_parse( - model=model, + model=config.model, chat_prompt=chat_prompt, prompt_input=prompt_input, pydantic_object=ImprovementModel, - use_function_calling=True, tags=[ f"exercise-{exercise.id}", f"submission-{submission.id}", diff --git a/modules/programming/module_programming_llm/module_programming_llm/generate_summary_by_file.py b/modules/programming/module_programming_llm/module_programming_llm/generate_summary_by_file.py index 6f50af4f0..f2a9c6fb7 100644 --- a/modules/programming/module_programming_llm/module_programming_llm/generate_summary_by_file.py +++ b/modules/programming/module_programming_llm/module_programming_llm/generate_summary_by_file.py @@ -10,10 +10,10 @@ from module_programming_llm.config import GradedBasicApproachConfig, BasicApproachConfig from llm_core.utils.llm_utils import ( - get_chat_prompt_with_formatting_instructions, + get_chat_prompt, num_tokens_from_prompt, ) -from llm_core.utils.predict_and_parse import predict_and_parse +from llm_core.core.predict_and_parse import predict_and_parse from module_programming_llm.helpers.utils import ( get_diff, @@ -70,8 +70,6 @@ async def generate_summary_by_file( if "summary" not in prompt.input_variables: return None - model = config.model.get_model() # type: ignore[attr-defined] - template_repo = exercise.get_template_repository() submission_repo = submission.get_repository() @@ -88,11 +86,9 @@ async def generate_summary_by_file( submission_repo, file_filter=lambda file_path: file_path in changed_files_from_template_to_submission, ) - chat_prompt = get_chat_prompt_with_formatting_instructions( - model=model, + chat_prompt = get_chat_prompt( system_message=config.generate_file_summary_prompt.system_message, human_message=config.generate_file_summary_prompt.human_message, - pydantic_object=SolutionSummary, ) prompt_inputs = [] @@ -118,11 +114,10 @@ async def generate_summary_by_file( results: List[Optional[FileDescription]] = await asyncio.gather( *[ predict_and_parse( - model=model, + model=config.model, chat_prompt=chat_prompt, prompt_input=prompt_input, pydantic_object=FileDescription, - use_function_calling=True, tags=[ f"exercise-{exercise.id}", f"submission-{submission.id}", diff --git a/modules/programming/module_programming_llm/module_programming_llm/split_grading_instructions_by_file.py b/modules/programming/module_programming_llm/module_programming_llm/split_grading_instructions_by_file.py index 9c3bb4a7e..d27cd6860 100644 --- a/modules/programming/module_programming_llm/module_programming_llm/split_grading_instructions_by_file.py +++ b/modules/programming/module_programming_llm/module_programming_llm/split_grading_instructions_by_file.py @@ -9,11 +9,11 @@ from module_programming_llm.config import GradedBasicApproachConfig from llm_core.utils.llm_utils import ( - get_chat_prompt_with_formatting_instructions, + get_chat_prompt, num_tokens_from_string, num_tokens_from_prompt, ) -from llm_core.utils.predict_and_parse import predict_and_parse +from llm_core.core.predict_and_parse import predict_and_parse from module_programming_llm.helpers.utils import format_grading_instructions, get_diff @@ -59,9 +59,7 @@ async def split_grading_instructions_by_file( # Return None if the grading instructions are not in the prompt if "grading_instructions" not in prompt.input_variables: return None - - model = config.model.get_model() # type: ignore[attr-defined] - + template_repo = exercise.get_template_repository() solution_repo = exercise.get_solution_repository() submission_repo = submission.get_repository() @@ -74,11 +72,9 @@ async def split_grading_instructions_by_file( src_repo=template_repo, dst_repo=submission_repo, file_path=None, name_only=True ).split("\n") - chat_prompt = get_chat_prompt_with_formatting_instructions( - model=model, + chat_prompt = get_chat_prompt( system_message=config.split_grading_instructions_by_file_prompt.system_message, human_message=config.split_grading_instructions_by_file_prompt.human_message, - pydantic_object=SplitGradingInstructions, ) prompt_input = { @@ -96,11 +92,10 @@ async def split_grading_instructions_by_file( return None split_grading_instructions = await predict_and_parse( - model=model, + model=config.model, chat_prompt=chat_prompt, prompt_input=prompt_input, pydantic_object=SplitGradingInstructions, - use_function_calling=True, tags=[ f"exercise-{exercise.id}", f"submission-{submission.id}", diff --git a/modules/programming/module_programming_llm/module_programming_llm/split_problem_statement_by_file.py b/modules/programming/module_programming_llm/module_programming_llm/split_problem_statement_by_file.py index 26718fb51..cb2e886cf 100644 --- a/modules/programming/module_programming_llm/module_programming_llm/split_problem_statement_by_file.py +++ b/modules/programming/module_programming_llm/module_programming_llm/split_problem_statement_by_file.py @@ -9,11 +9,11 @@ from module_programming_llm.config import GradedBasicApproachConfig, BasicApproachConfig from llm_core.utils.llm_utils import ( - get_chat_prompt_with_formatting_instructions, + get_chat_prompt, num_tokens_from_string, num_tokens_from_prompt, ) -from llm_core.utils.predict_and_parse import predict_and_parse +from llm_core.core.predict_and_parse import predict_and_parse from module_programming_llm.helpers.utils import get_diff @@ -69,11 +69,9 @@ async def split_problem_statement_by_file( name_only=True ).split("\n") - chat_prompt = get_chat_prompt_with_formatting_instructions( - model=model, + chat_prompt = get_chat_prompt( system_message=config.split_problem_statement_by_file_prompt.system_message, human_message=config.split_problem_statement_by_file_prompt.human_message, - pydantic_object=SplitProblemStatement ) prompt_input = { @@ -98,11 +96,10 @@ async def split_problem_statement_by_file( return None split_problem_statement = await predict_and_parse( - model=model, + model=config.model, chat_prompt=chat_prompt, prompt_input=prompt_input, pydantic_object=SplitProblemStatement, - use_function_calling=True, tags=[ f"exercise-{exercise.id}", f"submission-{submission.id}", diff --git a/modules/text/module_text_llm/llm_config.yml b/modules/text/module_text_llm/llm_config.yml new file mode 100644 index 000000000..25842669a --- /dev/null +++ b/modules/text/module_text_llm/llm_config.yml @@ -0,0 +1,5 @@ +models: + base_model: openai_gpt-4o + mini_model: openai_gpt-4o-mini + fast_reasoning_model: openai_o1-mini + long_reasoning_model: openai_o1-preview \ No newline at end of file diff --git a/modules/text/module_text_llm/module_text_llm/approach_config.py b/modules/text/module_text_llm/module_text_llm/approach_config.py index 71aab1ab8..96c951218 100644 --- a/modules/text/module_text_llm/module_text_llm/approach_config.py +++ b/modules/text/module_text_llm/module_text_llm/approach_config.py @@ -1,11 +1,14 @@ +from llm_core.loaders.llm_config_loader import get_llm_config from pydantic import BaseModel, Field -from llm_core.models import ModelConfigType, DefaultModelConfig +from llm_core.models import ModelConfigType from abc import ABC, abstractmethod from athena.text import Exercise, Submission + +llm_config = get_llm_config() class ApproachConfig(BaseModel, ABC): max_input_tokens: int = Field(default=3000, description="Maximum number of tokens in the input prompt.") - model: ModelConfigType = Field(default=DefaultModelConfig()) + model: ModelConfigType = Field(default=llm_config.models.base_model_config) type: str = Field(..., description="The type of approach config") @abstractmethod diff --git a/modules/text/module_text_llm/module_text_llm/basic_approach/generate_suggestions.py b/modules/text/module_text_llm/module_text_llm/basic_approach/generate_suggestions.py index b5b4180d2..c22b59640 100644 --- a/modules/text/module_text_llm/module_text_llm/basic_approach/generate_suggestions.py +++ b/modules/text/module_text_llm/module_text_llm/basic_approach/generate_suggestions.py @@ -5,11 +5,11 @@ from athena.text import Exercise, Submission, Feedback from athena.logger import logger from llm_core.utils.llm_utils import ( - get_chat_prompt_with_formatting_instructions, + get_chat_prompt, check_prompt_length_and_omit_features_if_necessary, num_tokens_from_prompt, ) -from llm_core.utils.predict_and_parse import predict_and_parse +from llm_core.core.predict_and_parse import predict_and_parse # from module_text_llm.config import BasicApproachConfig from module_text_llm.helpers.utils import add_sentence_numbers, get_index_range_from_line_range, format_grading_instructions @@ -26,11 +26,9 @@ async def generate_suggestions(exercise: Exercise, submission: Submission, confi "submission": add_sentence_numbers(submission.text) } - chat_prompt = get_chat_prompt_with_formatting_instructions( - model=model, + chat_prompt = get_chat_prompt( system_message=config.generate_suggestions_prompt.system_message, human_message=config.generate_suggestions_prompt.human_message, - pydantic_object=AssessmentModel ) # Check if the prompt is too long and omit features if necessary (in order of importance) @@ -52,7 +50,7 @@ async def generate_suggestions(exercise: Exercise, submission: Submission, confi return [] result = await predict_and_parse( - model=model, + model=config.model, chat_prompt=chat_prompt, prompt_input=prompt_input, pydantic_object=AssessmentModel, @@ -60,7 +58,6 @@ async def generate_suggestions(exercise: Exercise, submission: Submission, confi f"exercise-{exercise.id}", f"submission-{submission.id}", ], - use_function_calling=True ) if debug: diff --git a/modules/text/module_text_llm/module_text_llm/chain_of_thought_approach/generate_suggestions.py b/modules/text/module_text_llm/module_text_llm/chain_of_thought_approach/generate_suggestions.py index e54462593..2b2fbcf1c 100644 --- a/modules/text/module_text_llm/module_text_llm/chain_of_thought_approach/generate_suggestions.py +++ b/modules/text/module_text_llm/module_text_llm/chain_of_thought_approach/generate_suggestions.py @@ -6,11 +6,11 @@ from module_text_llm.approach_config import ApproachConfig from llm_core.utils.llm_utils import ( - get_chat_prompt_with_formatting_instructions, - check_prompt_length_and_omit_features_if_necessary, + check_prompt_length_and_omit_features_if_necessary, + get_chat_prompt, num_tokens_from_prompt, ) -from llm_core.utils.predict_and_parse import predict_and_parse +from llm_core.core.predict_and_parse import predict_and_parse from module_text_llm.helpers.utils import add_sentence_numbers, get_index_range_from_line_range, format_grading_instructions from module_text_llm.chain_of_thought_approach.prompt_thinking import InitialAssessmentModel @@ -29,11 +29,9 @@ async def generate_suggestions(exercise: Exercise, submission: Submission, confi "submission": add_sentence_numbers(submission.text) } - chat_prompt = get_chat_prompt_with_formatting_instructions( - model=model, + chat_prompt = get_chat_prompt( system_message=config.thinking_prompt.system_message, human_message=config.thinking_prompt.human_message, - pydantic_object=InitialAssessmentModel ) # Check if the prompt is too long and omit features if necessary (in order of importance) @@ -55,15 +53,14 @@ async def generate_suggestions(exercise: Exercise, submission: Submission, confi return [] initial_result = await predict_and_parse( - model=model, + model=config.model, chat_prompt=chat_prompt, prompt_input=prompt_input, pydantic_object=InitialAssessmentModel, tags=[ f"exercise-{exercise.id}", f"submission-{submission.id}", - ], - use_function_calling=True + ] ) second_prompt_input = { @@ -72,24 +69,22 @@ async def generate_suggestions(exercise: Exercise, submission: Submission, confi } - second_chat_prompt = get_chat_prompt_with_formatting_instructions( - model=model, + second_chat_prompt = get_chat_prompt( system_message=config.generate_suggestions_prompt.second_system_message, human_message=config.generate_suggestions_prompt.answer_message, - pydantic_object=AssessmentModel) + ) result = await predict_and_parse( - model=model, + model=config.model, chat_prompt=second_chat_prompt, prompt_input=second_prompt_input, pydantic_object=AssessmentModel, tags=[ f"exercise-{exercise.id}", f"submission-{submission.id}", - ], - use_function_calling=True + ] ) - + if debug: emit_meta("generate_suggestions", { "prompt": chat_prompt.format(**prompt_input), diff --git a/modules/text/module_text_llm/module_text_llm/generate_evaluation.py b/modules/text/module_text_llm/module_text_llm/generate_evaluation.py index 50473dd3a..ff154ed85 100644 --- a/modules/text/module_text_llm/module_text_llm/generate_evaluation.py +++ b/modules/text/module_text_llm/module_text_llm/generate_evaluation.py @@ -1,16 +1,15 @@ from typing import List, Sequence, Dict, Literal +from llm_core.loaders.llm_config_loader import get_llm_config from pydantic import BaseModel, Field import json from athena.text import Exercise, Submission, Feedback from athena.logger import logger - -from llm_core.models import evaluation_model from llm_core.utils.llm_utils import ( - get_chat_prompt_with_formatting_instructions, + get_chat_prompt, check_prompt_length_and_omit_features_if_necessary, ) -from llm_core.utils.predict_and_parse import predict_and_parse +from llm_core.core.predict_and_parse import predict_and_parse from module_text_llm.helpers.utils import add_sentence_numbers, get_line_range_from_index_range from module_text_llm.prompts.generate_evaluation import system_message, human_message @@ -33,9 +32,6 @@ async def generate_evaluation( predicted_feedbacks: List[Feedback] ) -> Dict[int, dict]: - if evaluation_model is None: - raise EnvironmentError("No evaluation model available, please set up LLM_EVALUATION_MODEL correctly" - "by setting it to one of the available models logged during startup.") max_input_tokens = 3000 def feedback_to_dict(feedback: Feedback): @@ -56,11 +52,9 @@ def feedback_to_dict(feedback: Feedback): "predicted_feedbacks": json.dumps([feedback_to_dict(feedback) for feedback in predicted_feedbacks]), } - chat_prompt = get_chat_prompt_with_formatting_instructions( - model=evaluation_model, + chat_prompt = get_chat_prompt( system_message=system_message, human_message=human_message, - pydantic_object=Evaluation ) # Check if the prompt is too long and omit features if necessary (in order of importance) @@ -78,7 +72,7 @@ def feedback_to_dict(feedback: Feedback): return {} result = await predict_and_parse( - model=evaluation_model, + model=get_llm_config().models.base_model_config, chat_prompt=chat_prompt, prompt_input=prompt_input, pydantic_object=Evaluation,