Skip to content

Commit 463b7a1

Browse files
nickscamaranickscamara
authored
Update README.md
1 parent f5f8d3a commit 463b7a1

File tree

1 file changed

+103
-101
lines changed

1 file changed

+103
-101
lines changed

‎README.md‎

Lines changed: 103 additions & 101 deletions
Original file line numberDiff line numberDiff line change
@@ -134,64 +134,86 @@ firecrawl https://example.com --exclude-tags nav,aside,.ad
134134

135135
---
136136

137-
### `crawl` - Crawl an entire website
138137

139-
Crawl multiple pages from a website.
138+
### `search` - Search the web
139+
140+
Search the web and optionally scrape content from search results.
140141

141142
```bash
142-
# Start a crawl (returns job ID)
143-
firecrawl crawl https://example.com
143+
# Basic search
144+
firecrawl search "firecrawl web scraping"
144145

145-
# Wait for crawl to complete
146-
firecrawl crawl https://example.com --wait
146+
# Limit results
147+
firecrawl search "AI news" --limit 10
147148

148-
# With progress indicator
149-
firecrawl crawl https://example.com --wait --progress
149+
# Search news sources
150+
firecrawl search "tech startups" --sources news
150151

151-
# Check crawl status
152-
firecrawl crawl <job-id>
152+
# Search images
153+
firecrawl search "landscape photography" --sources images
153154

154-
# Limit pages
155-
firecrawl crawl https://example.com --limit 100 --max-depth 3
155+
# Multiple sources
156+
firecrawl search "machine learning" --sources web,news,images
157+
158+
# Filter by category (GitHub, research papers, PDFs)
159+
firecrawl search "web scraping python" --categories github
160+
firecrawl search "transformer architecture" --categories research
161+
firecrawl search "machine learning" --categories github,research
162+
163+
# Time-based search
164+
firecrawl search "AI announcements" --tbs qdr:d # Past day
165+
firecrawl search "tech news" --tbs qdr:w # Past week
166+
167+
# Location-based search
168+
firecrawl search "restaurants" --location "San Francisco,California,United States"
169+
firecrawl search "local news" --country DE
170+
171+
# Search and scrape results
172+
firecrawl search "firecrawl tutorials" --scrape
173+
firecrawl search "API documentation" --scrape --scrape-formats markdown,links
174+
175+
# Output as pretty JSON
176+
firecrawl search "web scraping"
156177
```
157178

158-
#### Crawl Options
179+
#### Search Options
159180

160-
| Option | Description |
161-
| --------------------------- | ---------------------------------------- |
162-
| `--wait` | Wait for crawl to complete |
163-
| `--progress` | Show progress while waiting |
164-
| `--limit <n>` | Maximum pages to crawl |
165-
| `--max-depth <n>` | Maximum crawl depth |
166-
| `--include-paths <paths>` | Only crawl matching paths |
167-
| `--exclude-paths <paths>` | Skip matching paths |
168-
| `--sitemap <mode>` | `include`, `skip`, or `only` |
169-
| `--allow-subdomains` | Include subdomains |
170-
| `--allow-external-links` | Follow external links |
171-
| `--crawl-entire-domain` | Crawl entire domain |
172-
| `--ignore-query-parameters` | Treat URLs with different params as same |
173-
| `--delay <ms>` | Delay between requests |
174-
| `--max-concurrency <n>` | Max concurrent requests |
175-
| `--timeout <seconds>` | Timeout when waiting |
176-
| `--poll-interval <seconds>` | Status check interval |
181+
| Option | Description |
182+
| ---------------------------- | ------------------------------------------------------------------------------------------- |
183+
| `--limit <n>` | Maximum results (default: 5, max: 100) |
184+
| `--sources <sources>` | Comma-separated: `web`, `images`, `news` (default: web) |
185+
| `--categories <categories>` | Comma-separated: `github`, `research`, `pdf` |
186+
| `--tbs <value>` | Time filter: `qdr:h` (hour), `qdr:d` (day), `qdr:w` (week), `qdr:m` (month), `qdr:y` (year) |
187+
| `--location <location>` | Geo-targeting (e.g., "Germany", "San Francisco,California,United States") |
188+
| `--country <code>` | ISO country code (default: US) |
189+
| `--timeout <ms>` | Timeout in milliseconds (default: 60000) |
190+
| `--ignore-invalid-urls` | Exclude URLs invalid for other Firecrawl endpoints |
191+
| `--scrape` | Enable scraping of search results |
192+
| `--scrape-formats <formats>` | Scrape formats when `--scrape` enabled (default: markdown) |
193+
| `--only-main-content` | Include only main content when scraping (default: true) |
194+
| `-o, --output <path>` | Save to file |
195+
| `--json` | Output as compact JSON (use `-p` for pretty JSON) |
177196

178197
#### Examples
179198

180199
```bash
181-
# Crawl blog section only
182-
firecrawl crawl https://example.com --include-paths /blog,/posts
200+
# Research a topic with recent results
201+
firecrawl search "React Server Components" --tbs qdr:m --limit 10
183202

184-
# Exclude admin pages
185-
firecrawl crawl https://example.com --exclude-paths /admin,/login
203+
# Find GitHub repositories
204+
firecrawl search "web scraping library" --categories github --limit 20
186205

187-
# Crawl with rate limiting
188-
firecrawl crawl https://example.com --delay 1000 --max-concurrency 2
206+
# Search and get full content
207+
firecrawl search "firecrawl documentation" --scrape --scrape-formats markdown -p -o results.json
189208

190-
# Deep crawl with high limit
191-
firecrawl crawl https://example.com --limit 1000 --max-depth 10 --wait --progress
209+
# Find research papers
210+
firecrawl search "large language models" --categories research -p
192211

193-
# Save results
194-
firecrawl crawl https://example.com --wait -o crawl-results.json --pretty
212+
# Search with location targeting
213+
firecrawl search "best coffee shops" --location "Berlin,Germany" --country DE
214+
215+
# Get news from the past week
216+
firecrawl search "AI startups funding" --sources news --tbs qdr:w --limit 15
195217
```
196218

197219
---
@@ -245,89 +267,69 @@ firecrawl map https://example.com --include-subdomains --limit 1000
245267

246268
---
247269

248-
### `search` - Search the web
270+
### `crawl` - Crawl an entire website
249271

250-
Search the web and optionally scrape content from search results.
272+
Crawl multiple pages from a website.
251273

252274
```bash
253-
# Basic search
254-
firecrawl search "firecrawl web scraping"
255-
256-
# Limit results
257-
firecrawl search "AI news" --limit 10
258-
259-
# Search news sources
260-
firecrawl search "tech startups" --sources news
261-
262-
# Search images
263-
firecrawl search "landscape photography" --sources images
264-
265-
# Multiple sources
266-
firecrawl search "machine learning" --sources web,news,images
267-
268-
# Filter by category (GitHub, research papers, PDFs)
269-
firecrawl search "web scraping python" --categories github
270-
firecrawl search "transformer architecture" --categories research
271-
firecrawl search "machine learning" --categories github,research
275+
# Start a crawl (returns job ID)
276+
firecrawl crawl https://example.com
272277

273-
# Time-based search
274-
firecrawl search "AI announcements" --tbs qdr:d # Past day
275-
firecrawl search "tech news" --tbs qdr:w # Past week
278+
# Wait for crawl to complete
279+
firecrawl crawl https://example.com --wait
276280

277-
# Location-based search
278-
firecrawl search "restaurants" --location "San Francisco,California,United States"
279-
firecrawl search "local news" --country DE
281+
# With progress indicator
282+
firecrawl crawl https://example.com --wait --progress
280283

281-
# Search and scrape results
282-
firecrawl search "firecrawl tutorials" --scrape
283-
firecrawl search "API documentation" --scrape --scrape-formats markdown,links
284+
# Check crawl status
285+
firecrawl crawl <job-id>
284286

285-
# Output as pretty JSON
286-
firecrawl search "web scraping"
287+
# Limit pages
288+
firecrawl crawl https://example.com --limit 100 --max-depth 3
287289
```
288290

289-
#### Search Options
291+
#### Crawl Options
290292

291-
| Option | Description |
292-
| ---------------------------- | ------------------------------------------------------------------------------------------- |
293-
| `--limit <n>` | Maximum results (default: 5, max: 100) |
294-
| `--sources <sources>` | Comma-separated: `web`, `images`, `news` (default: web) |
295-
| `--categories <categories>` | Comma-separated: `github`, `research`, `pdf` |
296-
| `--tbs <value>` | Time filter: `qdr:h` (hour), `qdr:d` (day), `qdr:w` (week), `qdr:m` (month), `qdr:y` (year) |
297-
| `--location <location>` | Geo-targeting (e.g., "Germany", "San Francisco,California,United States") |
298-
| `--country <code>` | ISO country code (default: US) |
299-
| `--timeout <ms>` | Timeout in milliseconds (default: 60000) |
300-
| `--ignore-invalid-urls` | Exclude URLs invalid for other Firecrawl endpoints |
301-
| `--scrape` | Enable scraping of search results |
302-
| `--scrape-formats <formats>` | Scrape formats when `--scrape` enabled (default: markdown) |
303-
| `--only-main-content` | Include only main content when scraping (default: true) |
304-
| `-o, --output <path>` | Save to file |
305-
| `--json` | Output as compact JSON (use `-p` for pretty JSON) |
293+
| Option | Description |
294+
| --------------------------- | ---------------------------------------- |
295+
| `--wait` | Wait for crawl to complete |
296+
| `--progress` | Show progress while waiting |
297+
| `--limit <n>` | Maximum pages to crawl |
298+
| `--max-depth <n>` | Maximum crawl depth |
299+
| `--include-paths <paths>` | Only crawl matching paths |
300+
| `--exclude-paths <paths>` | Skip matching paths |
301+
| `--sitemap <mode>` | `include`, `skip`, or `only` |
302+
| `--allow-subdomains` | Include subdomains |
303+
| `--allow-external-links` | Follow external links |
304+
| `--crawl-entire-domain` | Crawl entire domain |
305+
| `--ignore-query-parameters` | Treat URLs with different params as same |
306+
| `--delay <ms>` | Delay between requests |
307+
| `--max-concurrency <n>` | Max concurrent requests |
308+
| `--timeout <seconds>` | Timeout when waiting |
309+
| `--poll-interval <seconds>` | Status check interval |
306310

307311
#### Examples
308312

309313
```bash
310-
# Research a topic with recent results
311-
firecrawl search "React Server Components" --tbs qdr:m --limit 10
312-
313-
# Find GitHub repositories
314-
firecrawl search "web scraping library" --categories github --limit 20
314+
# Crawl blog section only
315+
firecrawl crawl https://example.com --include-paths /blog,/posts
315316

316-
# Search and get full content
317-
firecrawl search "firecrawl documentation" --scrape --scrape-formats markdown -p -o results.json
317+
# Exclude admin pages
318+
firecrawl crawl https://example.com --exclude-paths /admin,/login
318319

319-
# Find research papers
320-
firecrawl search "large language models" --categories research -p
320+
# Crawl with rate limiting
321+
firecrawl crawl https://example.com --delay 1000 --max-concurrency 2
321322

322-
# Search with location targeting
323-
firecrawl search "best coffee shops" --location "Berlin,Germany" --country DE
323+
# Deep crawl with high limit
324+
firecrawl crawl https://example.com --limit 1000 --max-depth 10 --wait --progress
324325

325-
# Get news from the past week
326-
firecrawl search "AI startups funding" --sources news --tbs qdr:w --limit 15
326+
# Save results
327+
firecrawl crawl https://example.com --wait -o crawl-results.json --pretty
327328
```
328329

329330
---
330331

332+
331333
### `credit-usage` - Check your credits
332334

333335
```bash

0 commit comments

Comments
 (0)