> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/gnosis/prediction-market-agent/llms.txt
> Use this file to discover all available pages before exploring further.
# Web Scraping
> Powerful web scraping utilities for extracting and processing web content
## Overview
The Gnosis Prediction Market Agent provides multiple web scraping tools for extracting content from URLs. These tools offer different levels of processing, from raw text extraction to structured summaries, making them suitable for various use cases in market research and prediction analysis.
## Available Tools
### Basic Web Scraping
The basic web scraping tool extracts text from a URL and optionally summarizes it using GPT if the content exceeds 10,000 characters.
```python basic_summary.py theme={null}
from prediction_market_agent.tools.web_scrape.basic_summary import web_scrape
# Scrape and auto-summarize if needed
result = web_scrape(
objective="Extract information about prediction markets",
url="https://example.com/article"
)
```
The `web_scrape` function uses:
* **BeautifulSoup** for HTML parsing
* **LangChain's map-reduce chain** for intelligent summarization
* **Recursive text splitting** with 10,000 character chunks and 500 character overlap
* **OpenAI GPT** for generating objective-focused summaries
```python theme={null}
def web_scrape(objective: str, url: str) -> str:
response = requests.get(url)
response.raise_for_status()
soup = bs4.BeautifulSoup(response.content, "html.parser")
text: str = soup.get_text()
if len(text) > 10000:
text = _summary(objective, text)
return text
```
### Markdown Web Scraping
This tool converts HTML content to clean markdown format with automatic retry logic and caching.
```python markdown.py theme={null}
from prediction_market_agent.tools.web_scrape.markdown import web_scrape
# Scrape and convert to markdown
markdown_content = web_scrape(
url="https://example.com/page",
timeout=10
)
```
This function is cached for 1 day using `db_cache` and includes automatic retries with exponential backoff.
* **Automatic retry**: Up to 3 attempts with 1-second delays
* **Database caching**: Results cached for 24 hours
* **Clean extraction**: Removes scripts, styles, images, and other non-content elements
* **Markdown conversion**: Uses `markdownify` for clean text formatting
* **User-agent spoofing**: Mimics Firefox browser to avoid bot detection
```python theme={null}
@tenacity.retry(
stop=tenacity.stop_after_attempt(3),
wait=tenacity.wait_fixed(1),
reraise=True
)
def fetch_html(url: str, timeout: int) -> Response:
headers = {
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:107.0) Gecko/20100101 Firefox/107.0"
}
response = requests.get(url, headers=headers, timeout=timeout)
return response
```
The tool removes unnecessary elements:
* `