HuggingFaceTextGenerationModel

[yungchidanielcho]

Situation

I'm trying to use AutoChain with llama 2 instead of the public version of ChatGPT
autochain.models.huggingface_text_generation_model seems to be able to let me use llama 2

from autochain.chain.chain import Chain
from autochain.memory.buffer_memory import BufferMemory
from autochain.models.huggingface_text_generation_model import (
    HuggingFaceTextGenerationModel,
)
from huggingface_hub import login
from autochain.agent.conversational_agent.conversational_agent import ConversationalAgent
import os

os.environ['OPENAI_API_KEY'] = "my-key"
os.environ['PYTHONPATH'] = os.getcwd()

# your token here
login(token="hf_token")
llm = HuggingFaceTextGenerationModel(model_name="meta-llama/Llama-2-13b-chat-hf",
                                     temperature=2048,
                                     model_kwargs={'top_p': None, 'num_return_sequences': 1})
agent = ConversationalAgent.from_llm_and_tools(llm=llm)
memory = BufferMemory()
chain = Chain(agent=agent, memory=memory)

print(chain.run("Write me a poem about AI")['message'])

Problem

The model doesn't return anything after printing Planning.
However, the model download directly from meta repo works. So it is not a hardware problem.

I'll download the 7b-chat-hf version or the 7b-chat version. Maybe the model I have was corrupted.

Troubleshooting

Running transformer directly was able to get some text back

from transformers import AutoTokenizer
import transformers
import torch

model = "meta-llama/Llama-2-13b-chat-hf"

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    device_map="auto",
)

sequences = pipeline(
    'I liked "Breaking Bad" and "Band of Brothers". Do you have any recommendations of other shows I might like?\n',
    do_sample=False,
    top_k=None,
    top_p=None,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
    max_length=200,
)
for seq in sequences:
    print(f"Result: {seq['generated_text']}")

Returns

python using_transformer.py 
Loading checkpoint shards: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 3/3 [00:04<00:00,  1.39s/it]
/home/choy3/mambaforge/envs/MEEAgent/lib/python3.11/site-packages/transformers/generation/configuration_utils.py:362: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
  warnings.warn(
/home/choy3/mambaforge/envs/MEEAgent/lib/python3.11/site-packages/transformers/generation/configuration_utils.py:367: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `None` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.
  warnings.warn(
/home/choy3/mambaforge/envs/MEEAgent/lib/python3.11/site-packages/transformers/generation/configuration_utils.py:377: UserWarning: `do_sample` is set to `False`. However, `top_k` is set to `None` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_k`.
  warnings.warn(
Result: I liked "Breaking Bad" and "Band of Brothers". Do you have any recommendations of other shows I might like?

I'm looking for something with a similar tone and style to "Breaking Bad" and "Band of Brothers". Here are a few shows that you might enjoy:

1. "The Sopranos" - This HBO series is a crime drama that explores the life of a New Jersey mob boss, Tony Soprano, as he navigates the criminal underworld and deals with personal and family issues.
2. "The Wire" - This HBO series is a gritty and intense drama that explores the drug trade in Baltimore from multiple perspectives, including law enforcement, drug dealers, and politicians.
3. "Narcos" - This Netflix series tells the true story of Pablo Escobar, the infamous Colombian drug lord,