scrapfly

ScrapFly

ScrapFly 是一個網頁抓取 API，具備無頭瀏覽器功能、代理伺服器與反 Bot 繞過機制。它允許將網頁資料擷取為可存取的 LLM Markdown 或文字。

安裝

使用 pip 安裝 ScrapFly Python SDK 與必要的 Langchain 套件

pip install scrapfly-sdk langchain langchain-community

用法

from langchain_community.document_loaders import ScrapflyLoader

scrapfly_loader = ScrapflyLoader(
    ["https://web-scraping.dev/products"],
    api_key="Your ScrapFly API key",  # Get your API key from https://www.scrapfly.io/
    continue_on_failure=True,  # Ignore unprocessable web pages and log their exceptions
)

# Load documents from URLs as markdown
documents = scrapfly_loader.load()
print(documents)

API 參考文件：ScrapflyLoader

ScrapflyLoader 也允許傳遞 ScrapeConfig 物件，以自訂抓取請求。請參閱文件以了解完整功能詳情及其 API 參數：https://scrapfly.io/docs/scrape-api/getting-started

from langchain_community.document_loaders import ScrapflyLoader

scrapfly_scrape_config = {
    "asp": True,  # Bypass scraping blocking and antibot solutions, like Cloudflare
    "render_js": True,  # Enable JavaScript rendering with a cloud headless browser
    "proxy_pool": "public_residential_pool",  # Select a proxy pool (datacenter or residnetial)
    "country": "us",  # Select a proxy location
    "auto_scroll": True,  # Auto scroll the page
    "js": "",  # Execute custom JavaScript code by the headless browser
}

scrapfly_loader = ScrapflyLoader(
    ["https://web-scraping.dev/products"],
    api_key="Your ScrapFly API key",  # Get your API key from https://www.scrapfly.io/
    continue_on_failure=True,  # Ignore unprocessable web pages and log their exceptions
    scrape_config=scrapfly_scrape_config,  # Pass the scrape_config object
    scrape_format="markdown",  # The scrape result format, either `markdown`(default) or `text`
)

# Load documents from URLs as markdown
documents = scrapfly_loader.load()
print(documents)

API 參考文件：ScrapflyLoader

相關連結

• 文件載入器 概念指南
• 文件載入器 操作指南