跳到主要內容

StarRocks

StarRocks 是一個高效能的分析型資料庫。StarRocks 是一個新世代的次秒級 MPP 資料庫，適用於完整的分析情境，包含多維度分析、即時分析和 Ad-hoc 查詢。

通常 StarRocks 被歸類為 OLAP，並且在 ClickBench — 一個分析型資料庫管理系統的基準測試 中展現了卓越的效能。由於它具有超快速的向量化執行引擎，因此也可以用作快速的向量資料庫 (vectordb)。

在這裡，我們將展示如何使用 StarRocks Vector Store。

設定​

%pip install --upgrade --quiet  pymysql langchain-community

一開始設定 update_vectordb = False。如果沒有文件更新，那麼我們就不需要重建文件的嵌入 (embeddings)。

from langchain.chains import RetrievalQA
from langchain_community.document_loaders import (
    DirectoryLoader,
    UnstructuredMarkdownLoader,
)
from langchain_community.vectorstores import StarRocks
from langchain_community.vectorstores.starrocks import StarRocksSettings
from langchain_openai import OpenAI, OpenAIEmbeddings
from langchain_text_splitters import TokenTextSplitter

update_vectordb = False
/Users/dirlt/utils/py3env/lib/python3.9/site-packages/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.7) or chardet (5.1.0)/charset_normalizer (2.0.9) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "

載入文件並將它們分割成 tokens​

載入 docs 目錄下的所有 markdown 檔案。

對於 starrocks 文件，您可以從 https://github.com/StarRocks/starrocks 克隆 repo，其中包含一個 docs 目錄。

loader = DirectoryLoader(
    "./docs", glob="**/*.md", loader_cls=UnstructuredMarkdownLoader
)
documents = loader.load()

將文件分割成 tokens，並設定 update_vectordb = True，因為有新的文件/tokens。

# load text splitter and split docs into snippets of text
text_splitter = TokenTextSplitter(chunk_size=400, chunk_overlap=50)
split_docs = text_splitter.split_documents(documents)

# tell vectordb to update text embeddings
update_vectordb = True
split_docs[-20]
Document(page_content='Compile StarRocks with Docker\n\nThis topic describes how to compile StarRocks using Docker.\n\nOverview\n\nStarRocks provides development environment images for both Ubuntu 22.04 and CentOS 7.9. With the image, you can launch a Docker container and compile StarRocks in the container.\n\nStarRocks version and DEV ENV image\n\nDifferent branches of StarRocks correspond to different development environment images provided on StarRocks Docker Hub.\n\nFor Ubuntu 22.04:\n\n| Branch name | Image name              |\n  | --------------- | ----------------------------------- |\n  | main            | starrocks/dev-env-ubuntu:latest     |\n  | branch-3.0      | starrocks/dev-env-ubuntu:3.0-latest |\n  | branch-2.5      | starrocks/dev-env-ubuntu:2.5-latest |\n\nFor CentOS 7.9:\n\n| Branch name | Image name                       |\n  | --------------- | ------------------------------------ |\n  | main            | starrocks/dev-env-centos7:latest     |\n  | branch-3.0      | starrocks/dev-env-centos7:3.0-latest |\n  | branch-2.5      | starrocks/dev-env-centos7:2.5-latest |\n\nPrerequisites\n\nBefore compiling StarRocks, make sure the following requirements are satisfied:\n\nHardware\n\n', metadata={'source': 'docs/developers/build-starrocks/Build_in_docker.md'})
print("# docs  = %d, # splits = %d" % (len(documents), len(split_docs)))
# docs  = 657, # splits = 2802

建立 vectordb 實例​

使用 StarRocks 作為 vectordb​

def gen_starrocks(update_vectordb, embeddings, settings):
    if update_vectordb:
        docsearch = StarRocks.from_documents(split_docs, embeddings, config=settings)
    else:
        docsearch = StarRocks(embeddings, settings)
    return docsearch

將 tokens 轉換為 embeddings 並將它們放入 vectordb 中​

在這裡，我們使用 StarRocks 作為 vectordb，您可以透過 StarRocksSettings 配置 StarRocks 實例。

配置 StarRocks 實例非常像配置 mysql 實例。您需要指定

  1. host/port
  2. username (預設：'root')
  3. password (預設：'')
  4. database (預設：'default')
  5. table (預設：'langchain')
embeddings = OpenAIEmbeddings()

# configure starrocks settings(host/port/user/pw/db)
settings = StarRocksSettings()
settings.port = 41003
settings.host = "127.0.0.1"
settings.username = "root"
settings.password = ""
settings.database = "zya"
docsearch = gen_starrocks(update_vectordb, embeddings, settings)

print(docsearch)

update_vectordb = False
Inserting data...: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2802/2802 [02:26<00:00, 19.11it/s]
``````output
zya.langchain @ 127.0.0.1:41003

username: root

Table Schema:
----------------------------------------------------------------------------
|name                    |type                    |key                     |
----------------------------------------------------------------------------
|id                      |varchar(65533)          |true                    |
|document                |varchar(65533)          |false                   |
|embedding               |array<float>            |false                   |
|metadata                |varchar(65533)          |false                   |
----------------------------------------------------------------------------

建立 QA 並向它提問​

llm = OpenAI()
qa = RetrievalQA.from_chain_type(
    llm=llm, chain_type="stuff", retriever=docsearch.as_retriever()
)
query = "is profile enabled by default? if not, how to enable profile?"
resp = qa.run(query)
print(resp)
 No, profile is not enabled by default. To enable profile, set the variable `enable_profile` to `true` using the command `set enable_profile = true;`

這個頁面有幫助嗎？