Skip to content

INSTRUCTIONS.md

Latest commit

 

History

History
388 lines (277 loc) · 8.69 KB

INSTRUCTIONS.md

File metadata and controls

388 lines (277 loc) · 8.69 KB
Raw

Setup and Usage Instructions

Table of Contents

  1. Prerequisites
  2. Setup and Usage Instructions
  3. Available Commands
  4. Running Scripts
  5. Troubleshooting
  6. Best Practices
  7. Extending the Application
  8. External Resources

Version

1.0.0

Last Updated

June 2026

Prerequisites

System Requirements

  • Node.js: Version 16 or higher
  • MongoDB: Version 4.4 or higher
  • Operating System: Windows, macOS, or Linux
  • Snowflake Account: Active account with metadata access permissions

Setup and Usage Instructions

1. Install Dependencies

npm install

Setup Instructions

  1. Open the project in your IDE (VSCode recommended)
  2. Install dependencies: 
    npm install
  3. Configure environment variables (see Configuration section)
  4. Ensure MongoDB is running locally or update connection string
  5. Ensure Snowflake credentials are valid

Configuration

MongoDB Configuration

Update the MongoDB connection in src/config/constants.config.ts:

MONGODB: {
  CONNECTION_URL: 'mongodb://127.0.0.1:27017/test',
}

For production, use environment variables instead of hardcoded values.

Snowflake Configuration

SECURITY WARNING: The current implementation has hardcoded credentials in src/config/constants.config.ts. Before deploying or committing:

  1. Move credentials to environment variables
  2. Update config/env.json with placeholder values
  3. Load from environment in config/env.js
  4. Reference from CONSTANTS.DATA.SNOWFLAKE using env vars

Current configuration structure:

SNOWFLAKE: {
  ACCESS_URL: 'https://your-account.snowflakecomputing.com',
  USERNAME: 'your-username',
  PASSWORD: 'your-password',
  ACCOUNT: 'YOUR_ACCOUNT_ID',
  DATABASE: 'YOUR_DATABASE',
  ROLE: 'YOUR_ROLE',
  TABLE: 'YOUR_SCHEMA',
}

Server Configuration

Server settings in src/config/constants.config.ts:

SERVER: {
  PORT: '8080',
  EXPRESS_LIMIT: '5mb',
}

Running the Application

Development Mode (with hot reload)

npm run dev

Production Build and Start

npm run start

This will:

  1. Compile TypeScript to JavaScript
  2. Start the server on port 8080 (or configured port)
  3. Connect to MongoDB
  4. Connect to Snowflake
  5. Initialize Swagger UI at /api-docs

Kill Running Node Processes (Windows)

npm run kill

API Endpoints

Swagger Documentation

Once the server is running, access the interactive API documentation:

http://localhost:8080/api-docs

Available Endpoints

1. Integration Sync

POST /api/metadata/integration/sync

Fetches all table metadata from Snowflake and syncs to MongoDB.

Response:

{
  "status": "success"
}

What it does:

  • Clears existing metadata in MongoDB
  • Queries Snowflake for all tables in the configured schema
  • Fetches column information for each table
  • Stores the complete metadata in MongoDB

2. Get Tables with Pagination

GET /api/metadata/tables?page=1&limit=10

Retrieves table metadata from MongoDB with pagination.

Query Parameters:

  • page (required): Page number (default: 1)
  • limit (required): Items per page (default: 10)

Response:

{
  "data": [
    {
      "_id": "...",
      "created_on": "2021-11-10T05:04:44.572Z",
      "name": "TABLE_NAME",
      "database_name": "SNOWFLAKE_SAMPLE_DATA",
      "schema_name": "SCHEMA_NAME",
      "kind": "TABLE",
      "columns": [
        {
          "name": "COLUMN_NAME",
          "type": "TEXT"
        }
      ],
      "rows": 1000,
      "bytes": 50000
    }
  ],
  "page": 1,
  "limit": 10,
  "totalRecords": 24
}

Development Workflow

1. Initial Setup

npm install
npm run dev

2. Sync Snowflake Metadata

Use Swagger UI or curl:

curl -X POST http://localhost:8080/api/metadata/integration/sync

3. Query Tables

curl http://localhost:8080/api/metadata/tables?page=1&limit=10

Code Quality

Linting

npm run lint

Format Check

npm run prettier-check

Format Fix

npm run prettier-fix

Project Structure

snowflake/
├── src/
│   ├── config/              # Configuration constants
│   ├── controllers/         # Express route controllers
│   ├── custom/              # Custom error classes
│   ├── helpers/             # Express helper utilities
│   ├── middlewares/         # Express middlewares (validation, logging, errors)
│   ├── models/              # Business logic models
│   ├── routes/              # API route definitions
│   │   └── public/          # Public routes (no auth)
│   ├── schemas/             # Mongoose schemas
│   ├── services/            # External service integrations
│   │   ├── logger.service.ts
│   │   ├── mongodb.service.ts
│   │   ├── snowflake.service.ts
│   │   └── swagger.service.ts
│   ├── utils/               # Utility functions
│   ├── validations/         # Joi validation schemas
│   └── app.ts               # Express app entry point
├── config/                  # Environment configuration
│   ├── env.js
│   └── env.json
└── dist/                    # Compiled JavaScript output

Error Handling

The application uses a custom error handling system:

  • CustomError class in src/custom/error.custom.ts
  • Centralized error middleware in src/middlewares/errors.middleware.ts
  • All errors return consistent JSON responses

Logging

Winston logger configured in src/services/logger.service.ts:

  • All requests logged via src/middlewares/logs.middleware.ts
  • Service initialization logs
  • Error logs with stack traces

Security Considerations

Before deploying to production:

  1. Remove hardcoded credentials from src/config/constants.config.ts
  2. Use environment variables for all sensitive data
  3. Enable authentication on API endpoints
  4. Use HTTPS instead of HTTP
  5. Configure CORS properly (currently set to *)
  6. Enable rate limiting to prevent abuse
  7. Validate all inputs (already implemented with Joi)
  8. Use MongoDB connection with authentication
  9. Set proper Snowflake role permissions (least privilege)

Available Commands

Development Commands

# Start development server with hot-reload
npm run dev

# Run linter
npm run lint

# Format code
npm run prettier-fix

# Check formatting
npm run prettier-check

Running Scripts

# Compile and start production server
npm run start

# Kill running node processes (Windows)
npm run kill

Install Dependencies

To install all required dependencies for the project:

npm install

Troubleshooting

MongoDB Connection Failed

  • Ensure MongoDB is running locally (mongod)
  • Verify the connection string in src/config/constants.config.ts
  • Check for firewall blocks on port 27017

Snowflake Connection Failed

  • Verify credentials and account URL
  • Ensure the user has appropriate role and database access
  • Check network connectivity to Snowflake endpoints

Port Already in Use

  • Use npm run kill to stop existing processes
  • Or change the port in src/config/constants.config.ts

Best Practices

  • Security: Never commit sensitive credentials to the repository. Use environment variables.
  • Validation: Always use Joi schemas to validate incoming request data.
  • Logging: Use the structured Winston logger for all service-level operations.
  • Error Handling: Utilize the CustomError class for consistent API responses.

Extending the Application

Adding New API Routes

  1. Define the route in src/routes/public/
  2. Register the route in src/routes/index.public.routes.ts
  3. Implement the controller logic in src/controllers/

Creating New Services

  1. Create a new service file in src/services/
  2. Implement the integration or business logic
  3. Use the service in controllers or models

External Resources

Author