Amazon Document DB
Amazon DocumentDB（與 MongoDB 相容） 可讓您輕鬆地在雲端設定、操作和擴展與 MongoDB 相容的資料庫。使用 Amazon DocumentDB，您可以執行與 MongoDB 相同的應用程式碼，並使用相同的驅動程式和工具。Amazon DocumentDB 的向量搜尋結合了基於 JSON 的文件資料庫的彈性和豐富的查詢能力，以及向量搜尋的強大功能。
本筆記本向您展示如何使用 Amazon Document DB 向量搜尋 將文件儲存在集合中、建立索引，並使用近似最近鄰演算法（如「cosine」、「euclidean」和「dotProduct」）執行向量搜尋查詢。預設情況下，DocumentDB 會建立分層可導航小世界 (HNSW) 索引。若要瞭解其他支援的向量索引類型，請參閱上面連結的文件。
若要使用 DocumentDB，您必須先部署叢集。請參閱開發人員指南以取得更多詳細資訊。
免費註冊，立即開始使用。
!pip install pymongo
import getpass
# DocumentDB connection string
# i.e., "mongodb://{username}:{pass}@{cluster_endpoint}:{port}/?{params}"
CONNECTION_STRING = getpass.getpass("DocumentDB Cluster URI:")
INDEX_NAME = "izzy-test-index"
NAMESPACE = "izzy_test_db.izzy_test_collection"
DB_NAME, COLLECTION_NAME = NAMESPACE.split(".")
我們想要使用 OpenAIEmbeddings，因此我們需要設定我們的 OpenAI 環境變數。
import getpass
import os
# Set up the OpenAI Environment Variables
if "OPENAI_API_KEY" not in os.environ:
os.environ["OPENAI_API_KEY"] = getpass.getpass("OpenAI API Key:")
os.environ["OPENAI_EMBEDDINGS_DEPLOYMENT"] = (
"smart-agent-embedding-ada" # the deployment name for the embedding model
)
os.environ["OPENAI_EMBEDDINGS_MODEL_NAME"] = "text-embedding-ada-002" # the model name
現在，我們會將文件載入集合中、建立索引，然後對索引執行查詢。
如果您對某些參數有疑問，請參閱文件
from langchain.vectorstores.documentdb import (
DocumentDBSimilarityType,
DocumentDBVectorSearch,
)
from langchain_community.document_loaders import TextLoader
from langchain_openai import OpenAIEmbeddings
from langchain_text_splitters import CharacterTextSplitter
SOURCE_FILE_NAME = "../../how_to/state_of_the_union.txt"
loader = TextLoader(SOURCE_FILE_NAME)
documents = loader.load()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
docs = text_splitter.split_documents(documents)
# OpenAI Settings
model_deployment = os.getenv(
"OPENAI_EMBEDDINGS_DEPLOYMENT", "smart-agent-embedding-ada"
)
model_name = os.getenv("OPENAI_EMBEDDINGS_MODEL_NAME", "text-embedding-ada-002")
openai_embeddings: OpenAIEmbeddings = OpenAIEmbeddings(
deployment=model_deployment, model=model_name
)
from pymongo import MongoClient
INDEX_NAME = "izzy-test-index-2"
NAMESPACE = "izzy_test_db.izzy_test_collection"
DB_NAME, COLLECTION_NAME = NAMESPACE.split(".")
client: MongoClient = MongoClient(CONNECTION_STRING)
collection = client[DB_NAME][COLLECTION_NAME]
model_deployment = os.getenv(
"OPENAI_EMBEDDINGS_DEPLOYMENT", "smart-agent-embedding-ada"
)
model_name = os.getenv("OPENAI_EMBEDDINGS_MODEL_NAME", "text-embedding-ada-002")
vectorstore = DocumentDBVectorSearch.from_documents(
documents=docs,
embedding=openai_embeddings,
collection=collection,
index_name=INDEX_NAME,
)
# number of dimensions used by model above
dimensions = 1536
# specify similarity algorithm, valid options are:
# cosine (COS), euclidean (EUC), dotProduct (DOT)
similarity_algorithm = DocumentDBSimilarityType.COS
vectorstore.create_index(dimensions, similarity_algorithm)
{ 'createdCollectionAutomatically' : false,
'numIndexesBefore' : 1,
'numIndexesAfter' : 2,
'ok' : 1,
'operationTime' : Timestamp(1703656982, 1)}
# perform a similarity search between the embedding of the query and the embeddings of the documents
query = "What did the President say about Ketanji Brown Jackson"
docs = vectorstore.similarity_search(query)
print(docs[0].page_content)
Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections.
Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.
One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.
And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.
載入文件並建立索引後，您現在可以直接實例化向量儲存並針對索引執行查詢
vectorstore = DocumentDBVectorSearch.from_connection_string(
connection_string=CONNECTION_STRING,
namespace=NAMESPACE,
embedding=openai_embeddings,
index_name=INDEX_NAME,
)
# perform a similarity search between a query and the ingested documents
query = "What did the president say about Ketanji Brown Jackson"
docs = vectorstore.similarity_search(query)
print(docs[0].page_content)
Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections.
Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.
One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.
And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.
# perform a similarity search between a query and the ingested documents
query = "Which stats did the President share about the U.S. economy"
docs = vectorstore.similarity_search(query)
print(docs[0].page_content)
And unlike the $2 Trillion tax cut passed in the previous administration that benefitted the top 1% of Americans, the American Rescue Plan helped working people—and left no one behind.
And it worked. It created jobs. Lots of jobs.
In fact—our economy created over 6.5 Million new jobs just last year, more jobs created in one year
than ever before in the history of America.
Our economy grew at a rate of 5.7% last year, the strongest growth in nearly 40 years, the first step in bringing fundamental change to an economy that hasn’t worked for the working people of this nation for too long.
For the past 40 years we were told that if we gave tax breaks to those at the very top, the benefits would trickle down to everyone else.
But that trickle-down theory led to weaker economic growth, lower wages, bigger deficits, and the widest gap between those at the top and everyone else in nearly a century.
問答
qa_retriever = vectorstore.as_retriever(
search_type="similarity",
search_kwargs={"k": 25},
)
from langchain_core.prompts import PromptTemplate
prompt_template = """Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.
{context}
Question: {question}
"""
PROMPT = PromptTemplate(
template=prompt_template, input_variables=["context", "question"]
)
API 參考：PromptTemplate
from langchain.chains import RetrievalQA
from langchain_openai import OpenAI
qa = RetrievalQA.from_chain_type(
llm=OpenAI(),
chain_type="stuff",
retriever=qa_retriever,
return_source_documents=True,
chain_type_kwargs={"prompt": PROMPT},
)
docs = qa({"query": "gpt-4 compute requirements"})
print(docs["result"])
print(docs["source_documents"])
API 參考：RetrievalQA | OpenAI