> ## Documentation Index > Fetch the complete documentation index at: https://mintlify.com/gnosis/prediction-market-agent/llms.txt > Use this file to discover all available pages before exploring further. # Web Scraping API > API reference for web scraping functions and tools ## Overview The web scraping module provides multiple functions for extracting content from URLs with different levels of processing. All functions are located in `prediction_market_agent.tools.web_scrape`. ## Functions ### web\_scrape (Markdown) Extracts web content and converts it to clean markdown format with automatic caching and retry logic. **Location:** `prediction_market_agent.tools.web_scrape.markdown` The URL to scrape Request timeout in seconds Markdown-formatted text content, or `None` if the request fails or content is non-HTML **Features:** * Cached for 1 day using `@db_cache` decorator * Automatic retry with 3 attempts and 1-second delays * Removes scripts, styles, images, and non-content elements * User-agent spoofing to avoid bot detection * Returns `None` for non-HTML content ```python Basic Usage theme={null} from prediction_market_agent.tools.web_scrape.markdown import web_scrape markdown_content = web_scrape( url="https://example.com/article", timeout=10 ) if markdown_content: print(f"Scraped {len(markdown_content)} characters") ``` ```python Agent Example theme={null} from prediction_market_agent.tools.web_scrape.markdown import web_scrape from prediction_market_agent_tooling.tools.google_utils import search_google_serper # Search and scrape top results google_results = search_google_serper("prediction markets") contents = [ scraped[:10000] for url in google_results[:5] if (scraped := web_scrape(url)) ] ``` The function uses `tenacity` for retry logic and `markdownify` for HTML-to-markdown conversion: ```python theme={null} @tenacity.retry( stop=tenacity.stop_after_attempt(3), wait=tenacity.wait_fixed(1), reraise=True ) def fetch_html(url: str, timeout: int) -> Response: headers = { "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:107.0) Gecko/20100101 Firefox/107.0" } response = requests.get(url, headers=headers, timeout=timeout) return response ``` Elements removed during scraping: * `