Using the API
Features
API endpoints features
We provide several powerful API endpoints for web scraping and data extraction. Below is a comprehensive overview of our API features.
API Overview
| Endpoint | Description | Concurrency Limit | Response Type |
|---|---|---|---|
/scrape | Web content collection | 20 | Synchronous |
/extract | Data extraction | 20 | Synchronous |
/extractTask/batch | Batch extraction tasks | 3000 per batch | Asynchronous |
/extractTask/info | Task status monitoring | N/A | Synchronous |
Detailed Features
Scrape API
/scrape Endpoint
A powerful web scraping solution designed for collecting web page content.
| Feature | Description |
|---|---|
| Output Formats | Base64, UTF-8 markdown |
| Response Type | Synchronous |
| Concurrency | Up to 20 requests |
| Rate Limiting | Yes |
| Error Handling | Automatic retry |
Usage Examples:
Extract API
/extract Endpoint
Specialized in extracting structured data from specific platforms.
| Platform | Data Types | Format |
|---|---|---|
| Amazon | Product details, Reviews, Pricing | JSON |
| Google Maps | Business info, Reviews, Hours | JSON |
| Google Search | Search results, Snippets | JSON |
Batch Operations
/extractTask/batch Endpoint
| Feature | Description |
|---|---|
| Batch Size | Up to 1000 URLs per batch |
| Notification | Webhook support |
| Status Tracking | Real-time via /extractTask/info |
| Result Format | JSON |
Batch Processing Workflow:
- Submit batch task
- Receive task ID
- Get results through webhook or active polling
Task Management
/extractTask/info Endpoint
| Feature | Value |
|---|---|
| Task Status | Running/Completed/Failed/Cancelled |
| Progress Tracking | Percentage complete |
| Request Info | Request ID, timestamp |
| Task Details | Task ID, type, parameters |
| Result Data | Extracted data |
| Timing Info | Processing and completion timestamps |
| Error Info | Error code, message, stacktrace |
Response Example:
Best Practices
Limitations
| Limitation | Description |
|---|---|
| JavaScript Rendering | Does not support websites with dynamic JavaScript rendering |
Rate Limiting
| API | Rate Limit | Burst Limit |
|---|---|---|
| Scrape | 20 req/sec | 30 req/sec |
| Extract | 20 req/sec | 30 req/sec |
| Batch | 5 batches/min | 10 batches/min |