GET /v1/robots-txtrobots.txt parser
Web Intel lf-web-intel
Reads a website's robots.txt file and reports which parts it lets crawlers visit. Fetches and parses a domain's robots.txt file, returning disallowed paths, allowed paths, crawl-delay, and sitemaps.
Query Parameters
| Parameter | Type | Required | Description |
|---|---|---|---|
url | string | Yes | URL or domain to fetch robots.txt |
Response Schema
success (boolean): Operation success
url (string): robots.txt URL
exists (boolean): Whether file exists
rules (object): Parsed rules by user-agent
sitemaps (array): Referenced sitemaps
cURL Example
curl 'https://lf-web-intel.laminarforge.dev/v1/robots-txt?url=https://example.com' -H 'Authorization: Bearer lf_YourApiKey'
Python Example
import requests
url = 'https://lf-web-intel.laminarforge.dev/v1/robots-txt'
headers = {'Authorization': 'Bearer lf_YourApiKey'}
params = {'url': 'https://example.com'}
response = requests.get(url, headers=headers, params=params)
print(response.json())