LLMonitor

LLMonitor 是一個開源的可觀察性平台，提供成本和使用量分析、使用者追蹤、追蹤和評估工具。

設定

在 llmonitor.com 上建立帳戶，然後複製新應用程式的 tracking id。

取得 tracking id 後，執行以下命令將其設定為環境變數

export LLMONITOR_APP_ID="..."

如果您不想設定環境變數，可以在初始化回呼處理常式時直接傳遞金鑰

from langchain_community.callbacks.llmonitor_callback import LLMonitorCallbackHandler

handler = LLMonitorCallbackHandler(app_id="...")

API 參考：LLMonitorCallbackHandler

搭配 LLM/聊天模型使用

from langchain_openai import OpenAI
from langchain_openai import ChatOpenAI

handler = LLMonitorCallbackHandler()

llm = OpenAI(
    callbacks=[handler],
)

chat = ChatOpenAI(callbacks=[handler])

llm("Tell me a joke")

API 參考：OpenAI | ChatOpenAI

搭配鏈和代理程式使用

請務必將回呼處理常式傳遞給 run 方法，以便正確追蹤所有相關的鏈和 llm 呼叫。

也建議在 metadata 中傳遞 agent_name，以便能夠在儀表板中區分代理程式。

範例

from langchain_openai import ChatOpenAI
from langchain_community.callbacks.llmonitor_callback import LLMonitorCallbackHandler
from langchain_core.messages import SystemMessage, HumanMessage
from langchain.agents import OpenAIFunctionsAgent, AgentExecutor, tool

llm = ChatOpenAI(temperature=0)

handler = LLMonitorCallbackHandler()

@tool
def get_word_length(word: str) -> int:
    """Returns the length of a word."""
    return len(word)

tools = [get_word_length]

prompt = OpenAIFunctionsAgent.create_prompt(
    system_message=SystemMessage(
        content="You are very powerful assistant, but bad at calculating lengths of words."
    )
)

agent = OpenAIFunctionsAgent(llm=llm, tools=tools, prompt=prompt, verbose=True)
agent_executor = AgentExecutor(
    agent=agent, tools=tools, verbose=True, metadata={"agent_name": "WordCount"}  # <- recommended, assign a custom name
)
agent_executor.run("how many letters in the word educa?", callbacks=[handler])

API 參考：ChatOpenAI | LLMonitorCallbackHandler | SystemMessage | HumanMessage | OpenAIFunctionsAgent | AgentExecutor | tool

另一個範例

from langchain.agents import load_tools, initialize_agent, AgentType
from langchain_openai import OpenAI
from langchain_community.callbacks.llmonitor_callback import LLMonitorCallbackHandler


handler = LLMonitorCallbackHandler()

llm = OpenAI(temperature=0)
tools = load_tools(["serpapi", "llm-math"], llm=llm)
agent = initialize_agent(tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, metadata={ "agent_name": "GirlfriendAgeFinder" })  # <- recommended, assign a custom name

agent.run(
    "Who is Leo DiCaprio's girlfriend? What is her current age raised to the 0.43 power?",
    callbacks=[handler],
)

API 參考：load_tools | initialize_agent | AgentType | OpenAI | LLMonitorCallbackHandler

使用者追蹤

使用者追蹤可讓您識別使用者、追蹤其成本、對話等等。

from langchain_community.callbacks.llmonitor_callback import LLMonitorCallbackHandler, identify

with identify("user-123"):
    llm.invoke("Tell me a joke")

with identify("user-456", user_props={"email": "user456@test.com"}):
    agent.run("Who is Leo DiCaprio's girlfriend?")

API 參考：LLMonitorCallbackHandler | identify

支援

如有任何關於整合的問題或疑慮，您可以透過 Discord 或 電子郵件 聯繫 LLMonitor 團隊。