Browser Automation Prototype

A practical, production-ready browser automation system for common web tasks: shopping, booking, form-filling, and data extraction.

🎯 Features

• Shopping Automation: Extract product prices, availability, add to cart
• Booking Automation: Check availability, fill reservation forms
• Form Fill: Auto-fill web forms with structured data
• Data Extraction: Extract structured data (tables, lists, metadata)
• Full Logging: Every action logged with timestamps
• Screenshots: Visual confirmation at key steps
• Error Handling: Graceful failures with detailed error reporting
• Git-First: Designed for version control workflow

📋 Requirements

• Node.js 18 or higher
• npm (comes with Node.js)
• Playwright (auto-installed with npm install)

🚀 Quick Start

1. Installation

cd automation-scripts
npm install

This will install Playwright and its dependencies. First run may take a minute as it downloads the Chromium browser.

2. Run Your First Automation

# Extract data from a webpage
./automate.sh extract --url="https://example.com" --all

📖 Usage Guide

Wrapper Script (Recommended)

The automate.sh wrapper provides a unified interface:

./automate.sh <task-type> [options]

Task Types

1. Shopping Automation

Extract product information and optionally add to cart.

# Extract product info
./automate.sh shopping --url="https://example.com/product"

# Extract and add to cart
./automate.sh shopping --url="https://example.com/product" --add-to-cart

What it does:

• Navigates to product page
• Extracts title, price, availability
• Screenshots the page
• Optionally clicks "Add to Cart"
• Saves results to JSON

Direct usage:

node templates/shopping.js --url="https://example.com/product" --add-to-cart

2. Booking Automation

Check availability and fill booking forms.

./automate.sh booking \
  --url="https://booking-site.com" \
  --date="2026-02-15" \
  --time="19:00" \
  --party-size=2

What it does:

• Navigates to booking page
• Fills date, time, party size fields
• Clicks search/submit button
• Checks availability status
• Screenshots before/after

Direct usage:

node templates/booking.js --url="..." --date="2026-02-15" --time="19:00" --party-size=2

3. Form Fill Automation

Auto-fill web forms with structured data.

./automate.sh form \
  --url="https://example.com/contact" \
  --data='{"name":"John Doe","email":"john@example.com","message":"Hello"}' \
  --submit

What it does:

• Navigates to form page
• Intelligently matches fields by name/id/placeholder
• Fills text, select, checkbox, radio fields
• Optionally submits the form
• Captures confirmation messages

Direct usage:

node templates/form-fill.js \
  --url="..." \
  --data='{"field":"value"}' \
  --submit

Field Matching: The form filler automatically matches fields using:

• Exact name/id match
• Partial name/id match (case-insensitive)
• Placeholder text
• Data attributes

4. Data Extraction

Extract structured data from any webpage.

# Extract everything (metadata, tables, lists, headings)
./automate.sh extract --url="https://example.com" --all

# Extract tables only
./automate.sh extract --url="https://example.com" --table

# Extract with custom selectors
./automate.sh extract \
  --url="https://example.com" \
  --selectors='{"title":"h1","price":".price","items":".product-list li"}'

What it extracts:

• Metadata: title, description, Open Graph tags
• Tables: Converts HTML tables to structured JSON
• Lists: Ordered and unordered lists
• Headings: H1, H2, H3 elements
• Links: First 50 links on the page
• Custom: Any CSS selector

Direct usage:

node templates/data-extract.js --url="..." --all
node templates/data-extract.js --url="..." --selectors='{"key":"selector"}'

📂 Project Structure

automation-scripts/
├── automate.sh           # Wrapper script (unified interface)
├── package.json          # Node.js dependencies
├── README.md            # This file
├── templates/           # Automation templates
│   ├── shopping.js      # Shopping automation
│   ├── booking.js       # Booking automation
│   ├── form-fill.js     # Form filling
│   └── data-extract.js  # Data extraction
├── logs/                # Execution logs and results
│   ├── shopping.log
│   ├── booking.log
│   ├── form-fill.log
│   ├── data-extract.log
│   └── *-result-*.json  # JSON results
└── screenshots/         # Automation screenshots
    └── *.png

🔧 Configuration

Environment Variables

For sensitive data (API keys, credentials), use environment variables:

# Example: if a site needs authentication
export SITE_USERNAME="your-username"
export SITE_PASSWORD="your-password"

# Reference in templates with process.env.SITE_USERNAME

Customizing Templates

All templates are in templates/ and can be modified:

1. Copy a template to create a new automation
2. Modify selectors for specific sites
3. Add custom logic for site-specific behavior
4. Keep generic templates untouched for reuse

🛡️ Safety & Best Practices

⚠️ Never Hardcode Credentials

// ❌ DON'T DO THIS
const password = "my-secret-password";

// ✅ DO THIS
const password = process.env.PASSWORD;

Screenshots for Verification

Every template takes screenshots at key steps:

• Before action
• After action
• On error

Check screenshots/ to verify automation behavior.

Logging Everything

All actions are logged with timestamps:

[2026-02-07T10:30:00.000Z] Starting shopping automation for: https://example.com
[2026-02-07T10:30:02.000Z] Navigating to product page...
[2026-02-07T10:30:05.000Z] Extracting product information...

Check logs/*.log for detailed execution history.

Timeouts & Error Handling

All templates include:

• 30-second navigation timeout
• Graceful failure on missing elements
• Error screenshots
• Structured error reporting

Rate Limiting

Be respectful of websites:

• Don't run automations in tight loops
• Add delays between requests if scraping multiple pages
• Check site's robots.txt and terms of service

🐛 Troubleshooting

"Playwright not installed"

cd automation-scripts
npm install

"Could not find field for: X"

The form filler couldn't match a field. Solutions:

1. Check the page HTML to find the actual field name/id
2. Use custom selectors in the data extraction template
3. Modify the template to add site-specific selectors

"Navigation timeout"

The page took too long to load:

1. Check your internet connection
2. Try increasing timeout in the template
3. Some sites block automated browsers (see Stealth Mode below)

Automation detected / blocked

Some sites detect automation. To improve stealth:

1. Add delays: await page.waitForTimeout(2000)
2. Use residential proxies (not included in prototype)
3. Randomize user agents
4. Consider playwright-stealth plugin

🎓 Advanced Usage

Chaining Automations

# Extract data, then use it to fill a form
PRODUCT_DATA=$(./automate.sh extract --url="..." --selectors='{"price":".price"}')
PRICE=$(echo $PRODUCT_DATA | jq -r '.data.custom.price')

./automate.sh form --url="..." --data="{\"price\":\"$PRICE\"}"

Running in CI/CD

Templates work in headless mode by default (no GUI):

# GitHub Actions example
- name: Run automation
  run: |
    cd automation-scripts
    npm install
    ./automate.sh extract --url="https://example.com" --all

Custom Templates

Create new templates by copying existing ones:

cp templates/shopping.js templates/my-custom-automation.js
# Edit my-custom-automation.js
# Run with: node templates/my-custom-automation.js

📊 Output Formats

JSON Results

All automations output JSON to stdout (last line):

{
  "success": true,
  "data": {
    "title": "Product Name",
    "price": "$29.99",
    "availability": "In Stock"
  }
}

Parse with jq:

./automate.sh shopping --url="..." | jq -r '.data.price'

Log Files

Detailed logs in logs/:

• shopping.log, booking.log, etc.: execution logs
• *-result-*.json: structured results with timestamps

🤝 Contributing

This is a prototype. To extend:

1. Add new templates in templates/
2. Update automate.sh to route new task types
3. Document in this README
4. Commit with descriptive messages

📝 License

This is a prototype for practical automation. Use responsibly and ethically. Always respect website terms of service and robots.txt.

🔗 Resources

• Playwright Documentation
• CSS Selectors Reference
• Web Scraping Best Practices

---

Built with ❤️ for practical automation needs.