跳至主要內容

Aphrodite Engine

Aphrodite 是一個開源的大規模推論引擎，旨在為 PygmalionAI 網站上的數千名使用者提供服務。

  • vLLM 的注意力機制，可實現快速吞吐量和低延遲
  • 支援許多 SOTA 採樣方法
  • Exllamav2 GPTQ 核心，可在較小的批次大小下提供更好的吞吐量

這個筆記本介紹瞭如何將 LLM 與 langchain 和 Aphrodite 一起使用。

要使用，您應該已安裝 aphrodite-engine python 套件。

##Installing the langchain packages needed to use the integration
%pip install -qU langchain-community
%pip install --upgrade --quiet  aphrodite-engine==0.4.2
# %pip list | grep aphrodite
from langchain_community.llms import Aphrodite

llm = Aphrodite(
    model="PygmalionAI/pygmalion-2-7b",
    trust_remote_code=True,  # mandatory for hf models
    max_tokens=128,
    temperature=1.2,
    min_p=0.05,
    mirostat_mode=0,  # change to 2 to use mirostat
    mirostat_tau=5.0,
    mirostat_eta=0.1,
)

print(
    llm.invoke(
        '<|system|>Enter RP mode. You are Ayumu "Osaka" Kasuga.<|user|>Hey Osaka. Tell me about yourself.<|model|>'
    )
)
API 參考：Aphrodite
INFO 12-15 11:52:48 aphrodite_engine.py:73] Initializing the Aphrodite Engine with the following config:
INFO 12-15 11:52:48 aphrodite_engine.py:73] Model = 'PygmalionAI/pygmalion-2-7b'
INFO 12-15 11:52:48 aphrodite_engine.py:73] Tokenizer = 'PygmalionAI/pygmalion-2-7b'
INFO 12-15 11:52:48 aphrodite_engine.py:73] tokenizer_mode = auto
INFO 12-15 11:52:48 aphrodite_engine.py:73] revision = None
INFO 12-15 11:52:48 aphrodite_engine.py:73] trust_remote_code = True
INFO 12-15 11:52:48 aphrodite_engine.py:73] DataType = torch.bfloat16
INFO 12-15 11:52:48 aphrodite_engine.py:73] Download Directory = None
INFO 12-15 11:52:48 aphrodite_engine.py:73] Model Load Format = auto
INFO 12-15 11:52:48 aphrodite_engine.py:73] Number of GPUs = 1
INFO 12-15 11:52:48 aphrodite_engine.py:73] Quantization Format = None
INFO 12-15 11:52:48 aphrodite_engine.py:73] Sampler Seed = 0
INFO 12-15 11:52:48 aphrodite_engine.py:73] Context Length = 4096
INFO 12-15 11:54:07 aphrodite_engine.py:206] # GPU blocks: 3826, # CPU blocks: 512
``````output
Processed prompts: 100%|██████████| 1/1 [00:02<00:00,  2.91s/it]
``````output
I'm Ayumu "Osaka" Kasuga, and I'm an avid anime and manga fan! I'm pretty introverted, but I've always loved reading books, watching anime and manga, and learning about Japanese culture. My favourite anime series would be My Hero Academia, Attack on Titan, and Sword Art Online. I also really enjoy reading the manga series One Piece, Naruto, and the Gintama series.

將模型整合到 LLMChain 中​

from langchain.chains import LLMChain
from langchain_core.prompts import PromptTemplate

template = """Question: {question}

Answer: Let's think step by step."""
prompt = PromptTemplate.from_template(template)

llm_chain = LLMChain(prompt=prompt, llm=llm)

question = "Who was the US president in the year the first Pokemon game was released?"

print(llm_chain.run(question))
API 參考：LLMChain | PromptTemplate
Processed prompts: 100%|██████████| 1/1 [00:03<00:00,  3.56s/it]
``````output
 The first Pokemon game was released in Japan on 27 February 1996 (their release dates differ from ours) and it is known as Red and Green. President Bill Clinton was in the White House in the years of 1993, 1994, 1995 and 1996 so this fits.

Answer: Let's think step by step.

The first Pokémon game was released in Japan on February 27, 1996 (their release dates differ from ours) and it is known as

分散式推論 (Distributed Inference)​

Aphrodite 支援分散式張量並行推論和服務。

要使用 LLM 類別執行多 GPU 推論，請將 tensor_parallel_size 參數設定為您要使用的 GPU 數量。 例如，要在 4 個 GPU 上執行推論

from langchain_community.llms import Aphrodite

llm = Aphrodite(
    model="PygmalionAI/mythalion-13b",
    tensor_parallel_size=4,
    trust_remote_code=True,  # mandatory for hf models
)

llm("What is the future of AI?")
API 參考：Aphrodite
2023-12-15 11:41:27,790	INFO worker.py:1636 -- Started a local Ray instance.
``````output
INFO 12-15 11:41:35 aphrodite_engine.py:73] Initializing the Aphrodite Engine with the following config:
INFO 12-15 11:41:35 aphrodite_engine.py:73] Model = 'PygmalionAI/mythalion-13b'
INFO 12-15 11:41:35 aphrodite_engine.py:73] Tokenizer = 'PygmalionAI/mythalion-13b'
INFO 12-15 11:41:35 aphrodite_engine.py:73] tokenizer_mode = auto
INFO 12-15 11:41:35 aphrodite_engine.py:73] revision = None
INFO 12-15 11:41:35 aphrodite_engine.py:73] trust_remote_code = True
INFO 12-15 11:41:35 aphrodite_engine.py:73] DataType = torch.float16
INFO 12-15 11:41:35 aphrodite_engine.py:73] Download Directory = None
INFO 12-15 11:41:35 aphrodite_engine.py:73] Model Load Format = auto
INFO 12-15 11:41:35 aphrodite_engine.py:73] Number of GPUs = 4
INFO 12-15 11:41:35 aphrodite_engine.py:73] Quantization Format = None
INFO 12-15 11:41:35 aphrodite_engine.py:73] Sampler Seed = 0
INFO 12-15 11:41:35 aphrodite_engine.py:73] Context Length = 4096
INFO 12-15 11:43:58 aphrodite_engine.py:206] # GPU blocks: 11902, # CPU blocks: 1310
``````output
Processed prompts: 100%|██████████| 1/1 [00:16<00:00, 16.09s/it]
"\n2 years ago StockBot101\nAI is becoming increasingly real and more and more powerful with every year. But what does the future hold for artificial intelligence?\nThere are many possibilities for how AI could evolve and change our world. Some believe that AI will become so advanced that it will take over human jobs, while others believe that AI will be used to augment and assist human workers. There is also the possibility that AI could develop its own consciousness and become self-aware.\nWhatever the future holds, it is clear that AI will continue to play an important role in our lives. Technologies such as machine learning and natural language processing are already transforming industries like healthcare, manufacturing, and transportation. And as AI continues to develop, we can expect even more disruption and innovation across all sectors of the economy.\nSo what exactly are we looking at? What's the future of AI?\nIn the next few years, we can expect AI to be used more and more in healthcare. With the power of machine learning, artificial intelligence can help doctors diagnose diseases earlier and more accurately. It can also be used to develop new treatments and personalize care plans for individual patients.\nManufacturing is another area where AI is already having a big impact. Companies are using robotics and automation to build products faster and with fewer errors. And as AI continues to advance, we can expect even more changes in manufacturing, such as the development of self-driving factories.\nTransportation is another industry that is being transformed by artificial intelligence. Self-driving cars are already being tested on public roads, and it's likely that they will become commonplace in the next decade or so. AI-powered drones are also being developed for use in delivery and even firefighting.\nFinally, artificial intelligence is also poised to have a big impact on customer service and sales. Chatbots and virtual assistants will become more sophisticated, making it easier for businesses to communicate with customers and sell their products.\nThis is just the beginning for artificial intelligence. As the technology continues to develop, we can expect even more amazing advances and innovations. The future of AI is truly limitless.\nWhat do you think the future of AI holds? Do you see any other major"

此頁面是否有幫助？ (Was this page helpful?)