diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md new file mode 100644 index 00000000..1b73be5b --- /dev/null +++ b/DEPLOYMENT.md @@ -0,0 +1,356 @@ +# CMC Sales Deployment Guide + +## Overview + +This guide covers deploying the CMC Sales application to a Debian 12 VM at `cmc.springupsoftware.com` with both staging and production environments. + +## Architecture + +- **Production**: `https://cmc.springupsoftware.com` +- **Staging**: `https://staging.cmc.springupsoftware.com` +- **Components**: CakePHP legacy app, Go modern app, MariaDB, Nginx reverse proxy +- **SSL**: Let's Encrypt certificates +- **Authentication**: Basic auth for both environments + +## Prerequisites + +### Server Setup (Debian 12) + +```bash +# Update system +sudo apt update && sudo apt upgrade -y + +# Install Docker and Docker Compose +sudo apt install -y docker.io docker-compose-plugin +sudo systemctl enable docker +sudo systemctl start docker + +# Add user to docker group +sudo usermod -aG docker $USER +# Log out and back in + +# Create backup directory +sudo mkdir -p /var/backups/cmc-sales +sudo chown $USER:$USER /var/backups/cmc-sales +``` + +### DNS Configuration + +Ensure these DNS records point to your server: +- `cmc.springupsoftware.com` → Server IP +- `staging.cmc.springupsoftware.com` → Server IP + +## Initial Deployment + +### 1. Clone Repository + +```bash +cd /opt +sudo git clone cmc-sales +sudo chown -R $USER:$USER cmc-sales +cd cmc-sales +``` + +### 2. Environment Configuration + +```bash +# Copy environment files +cp .env.staging go-app/.env.staging +cp .env.production go-app/.env.production + +# Edit with actual passwords +nano .env.staging +nano .env.production + +# Create credential directories +mkdir -p credentials/staging credentials/production +``` + +### 3. Gmail OAuth Setup + +For each environment (staging/production): + +1. Go to [Google Cloud Console](https://console.cloud.google.com) +2. Create/select project +3. Enable Gmail API +4. Create OAuth 2.0 credentials +5. Download `credentials.json` +6. Place in appropriate directory: + - Staging: `credentials/staging/credentials.json` + - Production: `credentials/production/credentials.json` + +Generate tokens (run on local machine first): +```bash +cd go-app +go run cmd/auth/main.go +# Follow OAuth flow +# Copy token.json to appropriate credential directory +``` + +### 4. SSL Certificates + +```bash +# Start proxy services (includes Lego container) +docker compose -f docker-compose.proxy.yml up -d + +# Setup SSL certificates using Lego +./scripts/setup-lego-certs.sh admin@springupsoftware.com + +# Verify certificates +./scripts/lego-list-certs.sh +``` + +### 5. Database Initialization + +```bash +# Start database containers first +docker compose -f docker-compose.staging.yml up -d db-staging +docker compose -f docker-compose.production.yml up -d db-production + +# Wait for databases to be ready +sleep 30 + +# Restore production database (if you have a backup) +./scripts/restore-db.sh production /path/to/backup.sql.gz + +# Or initialize empty database and run migrations +# (implementation specific) +``` + +## Deployment Commands + +### Starting Services + +```bash +# Start staging environment +docker compose -f docker-compose.staging.yml up -d + +# Start production environment +docker compose -f docker-compose.production.yml up -d + +# Start reverse proxy (after both environments are running) +docker compose -f docker-compose.proxy.yml up -d +``` + +### Updating Applications + +```bash +# Pull latest code +git pull origin main + +# Rebuild and restart staging +docker compose -f docker-compose.staging.yml down +docker compose -f docker-compose.staging.yml build --no-cache +docker compose -f docker-compose.staging.yml up -d + +# Test staging thoroughly, then update production +docker compose -f docker-compose.production.yml down +docker compose -f docker-compose.production.yml build --no-cache +docker compose -f docker-compose.production.yml up -d +``` + +## Monitoring and Maintenance + +### Health Checks + +```bash +# Check all containers +docker ps + +# Check logs +docker compose -f docker-compose.production.yml logs -f + +# Check application health +curl https://cmc.springupsoftware.com/health +curl https://staging.cmc.springupsoftware.com/health +``` + +### Database Backups + +```bash +# Manual backup +./scripts/backup-db.sh production +./scripts/backup-db.sh staging + +# Set up automated backups (cron) +sudo crontab -e +# Add: 0 2 * * * /opt/cmc-sales/scripts/backup-db.sh production +# Add: 0 3 * * * /opt/cmc-sales/scripts/backup-db.sh staging +``` + +### Log Management + +```bash +# View nginx logs +sudo tail -f /var/log/nginx/access.log +sudo tail -f /var/log/nginx/error.log + +# View application logs +docker compose -f docker-compose.production.yml logs -f cmc-go-production +docker compose -f docker-compose.staging.yml logs -f cmc-go-staging +``` + +### SSL Certificate Renewal + +```bash +# Manual renewal +./scripts/lego-renew-cert.sh all + +# Renew specific domain +./scripts/lego-renew-cert.sh cmc.springupsoftware.com + +# Set up auto-renewal (cron) +sudo crontab -e +# Add: 0 2 * * * /opt/cmc-sales/scripts/lego-renew-cert.sh all +``` + +## Security Considerations + +### Basic Authentication + +Update passwords in `userpasswd` file: +```bash +# Generate new password hash +sudo apt install apache2-utils +htpasswd -c userpasswd username + +# Restart nginx containers +docker compose -f docker-compose.proxy.yml restart nginx-proxy +``` + +### Database Security + +- Use strong passwords in environment files +- Database containers are not exposed externally in production +- Regular backups with encryption at rest + +### Network Security + +- All traffic encrypted with SSL/TLS +- Rate limiting configured in nginx +- Security headers enabled +- Docker networks isolate environments + +## Troubleshooting + +### Common Issues + +1. **Containers won't start** + ```bash + # Check logs + docker compose logs container-name + + # Check system resources + df -h + free -h + ``` + +2. **SSL issues** + ```bash + # Check certificate status + ./scripts/lego-list-certs.sh + + # Test SSL configuration + curl -I https://cmc.springupsoftware.com + + # Manually renew certificates + ./scripts/lego-renew-cert.sh all + ``` + +3. **Database connection issues** + ```bash + # Test database connectivity + docker exec -it cmc-db-production mysql -u cmc -p + ``` + +4. **Gmail API issues** + ```bash + # Check credentials are mounted + docker exec -it cmc-go-production ls -la /root/credentials/ + + # Check logs for OAuth errors + docker compose logs cmc-go-production | grep -i gmail + ``` + +### Emergency Procedures + +1. **Quick rollback** + ```bash + # Stop current containers + docker compose -f docker-compose.production.yml down + + # Restore from backup + ./scripts/restore-db.sh production /var/backups/cmc-sales/latest_backup.sql.gz + + # Start previous version + git checkout previous-commit + docker compose -f docker-compose.production.yml up -d + ``` + +2. **Database corruption** + ```bash + # Stop application + docker compose -f docker-compose.production.yml stop cmc-go-production cmc-php-production + + # Restore from backup + ./scripts/restore-db.sh production /var/backups/cmc-sales/backup_production_YYYYMMDD-HHMMSS.sql.gz + + # Restart application + docker compose -f docker-compose.production.yml start cmc-go-production cmc-php-production + ``` + +## File Structure + +``` +/opt/cmc-sales/ +├── docker-compose.staging.yml +├── docker-compose.production.yml +├── docker-compose.proxy.yml +├── conf/ +│ ├── nginx-staging.conf +│ ├── nginx-production.conf +│ └── nginx-proxy.conf +├── credentials/ +│ ├── staging/ +│ │ ├── credentials.json +│ │ └── token.json +│ └── production/ +│ ├── credentials.json +│ └── token.json +├── scripts/ +│ ├── backup-db.sh +│ ├── restore-db.sh +│ ├── lego-obtain-cert.sh +│ ├── lego-renew-cert.sh +│ ├── lego-list-certs.sh +│ └── setup-lego-certs.sh +└── .env files +``` + +## Performance Tuning + +### Resource Limits + +Resource limits are configured in the Docker Compose files: +- Production: 2 CPU cores, 2-4GB RAM per service +- Staging: More relaxed limits for testing + +### Database Optimization + +```sql +-- Monitor slow queries +SHOW VARIABLES LIKE 'slow_query_log'; +SET GLOBAL slow_query_log = 'ON'; +SET GLOBAL long_query_time = 2; + +-- Check database performance +SHOW PROCESSLIST; +SHOW ENGINE INNODB STATUS; +``` + +### Nginx Optimization + +- Gzip compression enabled +- Static file caching +- Connection keep-alive +- Rate limiting configured \ No newline at end of file diff --git a/Dockerfile.go b/Dockerfile.go index 2e59e2ec..57c7d546 100644 --- a/Dockerfile.go +++ b/Dockerfile.go @@ -40,6 +40,13 @@ COPY --from=builder /app/server . COPY go-app/templates ./templates COPY go-app/static ./static +# Copy Gmail OAuth credentials if they exist +# Note: These files contain sensitive data and should not be committed to version control +# Consider using Docker secrets or environment variables for production +# Using conditional copy pattern to handle missing files gracefully +COPY go-app/credentials.jso[n] ./ +COPY go-app/token.jso[n] ./ + # Copy .env file if needed COPY go-app/.env.example .env diff --git a/Dockerfile.go.production b/Dockerfile.go.production new file mode 100644 index 00000000..adb9f510 --- /dev/null +++ b/Dockerfile.go.production @@ -0,0 +1,63 @@ +# Build stage +FROM golang:1.23-alpine AS builder + +# Install build dependencies +RUN apk add --no-cache git + +# Set working directory +WORKDIR /app + +# Copy go mod files +COPY go-app/go.mod go-app/go.sum ./ + +# Download dependencies +RUN go mod download + +# Copy source code +COPY go-app/ . + +# Install sqlc (compatible with Go 1.23+) +RUN go install github.com/sqlc-dev/sqlc/cmd/sqlc@latest + +# Generate sqlc code +RUN sqlc generate + +# Build the application with production optimizations +RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags="-w -s" -tags production -o server cmd/server/main.go + +# Runtime stage - minimal image for production +FROM alpine:latest + +# Install only essential runtime dependencies +RUN apk --no-cache add ca-certificates && \ + addgroup -g 1001 -S appgroup && \ + adduser -u 1001 -S appuser -G appgroup + +WORKDIR /app + +# Copy the binary from builder +COPY --from=builder /app/server . + +# Copy templates and static files +COPY go-app/templates ./templates +COPY go-app/static ./static + +# Copy production environment file +COPY go-app/.env.production .env + +# Create credentials directory with proper permissions +RUN mkdir -p ./credentials && \ + chown -R appuser:appgroup /app + +# Switch to non-root user +USER appuser + +# Expose port +EXPOSE 8080 + +# Health check +HEALTHCHECK --interval=60s --timeout=10s --start-period=10s --retries=3 \ + CMD wget --no-verbose --tries=1 --spider http://localhost:8080/api/v1/health || exit 1 + +# Run the application +CMD ["./server"] \ No newline at end of file diff --git a/Dockerfile.go.staging b/Dockerfile.go.staging new file mode 100644 index 00000000..6c814aec --- /dev/null +++ b/Dockerfile.go.staging @@ -0,0 +1,57 @@ +# Build stage +FROM golang:1.23-alpine AS builder + +# Install build dependencies +RUN apk add --no-cache git + +# Set working directory +WORKDIR /app + +# Copy go mod files +COPY go-app/go.mod go-app/go.sum ./ + +# Download dependencies +RUN go mod download + +# Copy source code +COPY go-app/ . + +# Install sqlc (compatible with Go 1.23+) +RUN go install github.com/sqlc-dev/sqlc/cmd/sqlc@latest + +# Generate sqlc code +RUN sqlc generate + +# Build the application with staging tags +RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -tags staging -o server cmd/server/main.go + +# Runtime stage +FROM alpine:latest + +# Install runtime dependencies and debugging tools for staging +RUN apk --no-cache add ca-certificates curl net-tools + +WORKDIR /root/ + +# Copy the binary from builder +COPY --from=builder /app/server . + +# Copy templates and static files +COPY go-app/templates ./templates +COPY go-app/static ./static + +# Copy staging environment file +COPY go-app/.env.staging .env + +# Create credentials directory +RUN mkdir -p ./credentials + +# Expose port +EXPOSE 8080 + +# Health check +HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \ + CMD curl -f http://localhost:8080/api/v1/health || exit 1 + +# Run the application +CMD ["./server"] \ No newline at end of file diff --git a/TESTING_DOCKER.md b/TESTING_DOCKER.md new file mode 100644 index 00000000..17256fcb --- /dev/null +++ b/TESTING_DOCKER.md @@ -0,0 +1,394 @@ +# Running CMC Django Tests in Docker + +This guide explains how to run the comprehensive CMC Django test suite using Docker for consistent, isolated testing. + +## Quick Start + +```bash +# 1. Setup test environment (one-time) +./run-tests-docker.sh setup + +# 2. Run all tests +./run-tests-docker.sh run + +# 3. Run tests with coverage +./run-tests-docker.sh coverage +``` + +## Test Environment Overview + +The Docker test environment includes: +- **Isolated test database** (MariaDB on port 3307) +- **Django test container** with all dependencies +- **Coverage reporting** with HTML and XML output +- **PDF generation testing** with WeasyPrint/ReportLab +- **Parallel test execution** support + +## Available Commands + +### Setup and Management + +```bash +# Build containers and setup test database +./run-tests-docker.sh setup + +# Clean up all test containers and data +./run-tests-docker.sh clean + +# View test container logs +./run-tests-docker.sh logs + +# Open shell in test container +./run-tests-docker.sh shell +``` + +### Running Tests + +```bash +# Run all tests +./run-tests-docker.sh run + +# Run specific test suites +./run-tests-docker.sh run models # Model tests only +./run-tests-docker.sh run services # Service layer tests +./run-tests-docker.sh run auth # Authentication tests +./run-tests-docker.sh run views # View and URL tests +./run-tests-docker.sh run pdf # PDF generation tests +./run-tests-docker.sh run integration # Integration tests + +# Run quick tests (models + services) +./run-tests-docker.sh quick + +# Run tests with coverage reporting +./run-tests-docker.sh coverage +``` + +## Advanced Test Options + +### Using Docker Compose Directly + +```bash +# Run specific test with custom options +docker-compose -f docker-compose.test.yml run --rm cmc-django-test \ + python cmcsales/manage.py test cmc.tests.test_models --verbosity=2 --keepdb + +# Run tests with coverage +docker-compose -f docker-compose.test.yml run --rm cmc-django-test \ + coverage run --source='.' cmcsales/manage.py test cmc.tests + +# Generate coverage report +docker-compose -f docker-compose.test.yml run --rm cmc-django-test \ + coverage report --show-missing +``` + +### Using the Test Script Directly + +```bash +# Inside the container, you can use the test script with advanced options +docker-compose -f docker-compose.test.yml run --rm cmc-django-test \ + /app/scripts/run-tests.sh --coverage --keepdb --failfast models + +# Script options: +# -c, --coverage Enable coverage reporting +# -k, --keepdb Keep test database between runs +# -p, --parallel NUM Run tests in parallel +# -f, --failfast Stop on first failure +# -v, --verbosity NUM Verbosity level 0-3 +``` + +## Test Suite Structure + +### 1. Model Tests (`test_models.py`) +Tests all Django models including: +- Customer, Enquiry, Job, Document models +- Model validation and constraints +- Relationships and cascade behavior +- Financial calculations + +```bash +./run-tests-docker.sh run models +``` + +### 2. Service Tests (`test_services.py`) +Tests business logic layer: +- Number generation service +- Financial calculation service +- Document service workflows +- Validation service + +```bash +./run-tests-docker.sh run services +``` + +### 3. Authentication Tests (`test_authentication.py`) +Tests authentication system: +- Multiple authentication backends +- Permission decorators and middleware +- User management workflows +- Security features + +```bash +./run-tests-docker.sh run auth +``` + +### 4. View Tests (`test_views.py`) +Tests web interface: +- CRUD operations for all entities +- AJAX endpoints +- Permission enforcement +- URL routing + +```bash +./run-tests-docker.sh run views +``` + +### 5. PDF Tests (`test_pdf.py`) +Tests PDF generation: +- WeasyPrint and ReportLab engines +- Template rendering +- Document formatting +- Security and performance + +```bash +./run-tests-docker.sh run pdf +``` + +### 6. Integration Tests (`test_integration.py`) +Tests complete workflows: +- End-to-end business processes +- Multi-user collaboration +- System integration scenarios +- Performance and security + +```bash +./run-tests-docker.sh run integration +``` + +## Test Reports and Coverage + +### Coverage Reports + +After running tests with coverage, reports are available in: +- **HTML Report**: `./coverage-reports/html/index.html` +- **XML Report**: `./coverage-reports/coverage.xml` +- **Console**: Displayed after test run + +```bash +# Run tests with coverage +./run-tests-docker.sh coverage + +# View HTML report +open coverage-reports/html/index.html +``` + +### Test Artifacts + +Test outputs are saved to: +- **Test Reports**: `./test-reports/` +- **Coverage Reports**: `./coverage-reports/` +- **Logs**: `./logs/` +- **PDF Test Files**: `./test-reports/pdf/` + +## Configuration + +### Environment Variables + +The test environment uses these key variables: + +```yaml +# Database configuration +DATABASE_HOST: test-db +DATABASE_NAME: test_cmc +DATABASE_USER: test_cmc +DATABASE_PASSWORD: testPassword123 + +# Django settings +DJANGO_SETTINGS_MODULE: cmcsales.settings +TESTING: 1 +DEBUG: 0 + +# PDF generation +PDF_GENERATION_ENGINE: weasyprint +PDF_SAVE_DIRECTORY: /app/test-reports/pdf +``` + +### Test Database + +- **Isolated database** separate from development/production +- **Runs on port 3307** to avoid conflicts +- **Optimized for testing** with reduced buffer sizes +- **Automatically reset** between test runs (unless `--keepdb` used) + +## Performance Optimization + +### Parallel Test Execution + +```bash +# Run tests in parallel (faster execution) +docker-compose -f docker-compose.test.yml run --rm cmc-django-test \ + /app/scripts/run-tests.sh --parallel=4 all +``` + +### Keeping Test Database + +```bash +# Keep database between runs for faster subsequent tests +./run-tests-docker.sh run models --keepdb +``` + +### Quick Test Suite + +```bash +# Run only essential tests for rapid feedback +./run-tests-docker.sh quick +``` + +## Continuous Integration + +### GitHub Actions Example + +```yaml +name: Test CMC Django +on: [push, pull_request] + +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + + - name: Setup test environment + run: ./run-tests-docker.sh setup + + - name: Run tests with coverage + run: ./run-tests-docker.sh coverage + + - name: Upload coverage reports + uses: codecov/codecov-action@v3 + with: + file: ./coverage-reports/coverage.xml +``` + +## Troubleshooting + +### Database Connection Issues + +```bash +# Check database status +docker-compose -f docker-compose.test.yml ps + +# View database logs +docker-compose -f docker-compose.test.yml logs test-db + +# Restart database +docker-compose -f docker-compose.test.yml restart test-db +``` + +### Test Failures + +```bash +# Run with maximum verbosity for debugging +docker-compose -f docker-compose.test.yml run --rm cmc-django-test \ + python cmcsales/manage.py test cmc.tests.test_models --verbosity=3 + +# Use failfast to stop on first error +./run-tests-docker.sh run models --failfast + +# Open shell to investigate +./run-tests-docker.sh shell +``` + +### Permission Issues + +```bash +# Fix file permissions +sudo chown -R $USER:$USER test-reports coverage-reports logs + +# Check Docker permissions +docker-compose -f docker-compose.test.yml run --rm cmc-django-test whoami +``` + +### Memory Issues + +```bash +# Run tests with reduced parallel workers +docker-compose -f docker-compose.test.yml run --rm cmc-django-test \ + /app/scripts/run-tests.sh --parallel=1 all + +# Monitor resource usage +docker stats +``` + +## Development Workflow + +### Recommended Testing Workflow + +1. **Initial Setup** (one-time): + ```bash + ./run-tests-docker.sh setup + ``` + +2. **During Development** (fast feedback): + ```bash + ./run-tests-docker.sh quick --keepdb + ``` + +3. **Before Commit** (comprehensive): + ```bash + ./run-tests-docker.sh coverage + ``` + +4. **Debugging Issues**: + ```bash + ./run-tests-docker.sh shell + # Inside container: + python cmcsales/manage.py test cmc.tests.test_models.CustomerModelTest.test_customer_creation --verbosity=3 + ``` + +### Adding New Tests + +1. Create test file in appropriate module +2. Follow existing test patterns and base classes +3. Test locally: + ```bash + ./run-tests-docker.sh run models --keepdb + ``` +4. Run full suite before committing: + ```bash + ./run-tests-docker.sh coverage + ``` + +## Integration with IDE + +### PyCharm/IntelliJ + +Configure remote interpreter using Docker: +1. Go to Settings → Project → Python Interpreter +2. Add Docker Compose interpreter +3. Use `docker-compose.test.yml` configuration +4. Set service to `cmc-django-test` + +### VS Code + +Use Dev Containers extension: +1. Create `.devcontainer/devcontainer.json` +2. Configure to use test Docker environment +3. Run tests directly in integrated terminal + +## Best Practices + +1. **Always run tests in Docker** for consistency +2. **Use `--keepdb` during development** for speed +3. **Run coverage reports before commits** +4. **Clean up regularly** to free disk space +5. **Monitor test performance** and optimize slow tests +6. **Use parallel execution** for large test suites +7. **Keep test data realistic** but minimal +8. **Test error conditions** as well as happy paths + +## Resources + +- **Django Testing Documentation**: https://docs.djangoproject.com/en/5.1/topics/testing/ +- **Coverage.py Documentation**: https://coverage.readthedocs.io/ +- **Docker Compose Reference**: https://docs.docker.com/compose/ +- **CMC Test Suite Documentation**: See individual test modules for detailed information \ No newline at end of file diff --git a/conf/nginx-production.conf b/conf/nginx-production.conf new file mode 100644 index 00000000..bd9829b3 --- /dev/null +++ b/conf/nginx-production.conf @@ -0,0 +1,152 @@ +# Production environment configuration +upstream cmc_php_production { + server cmc-php-production:80; + keepalive 32; +} + +upstream cmc_go_production { + server cmc-go-production:8080; + keepalive 32; +} + +# Rate limiting +limit_req_zone $binary_remote_addr zone=login:10m rate=5r/m; +limit_req_zone $binary_remote_addr zone=api:10m rate=30r/m; + +server { + server_name cmc.springupsoftware.com; + + # Basic auth for production + auth_basic_user_file /etc/nginx/userpasswd; + auth_basic "CMC Sales - Restricted Access"; + + # Security headers + add_header X-Frame-Options DENY; + add_header X-Content-Type-Options nosniff; + add_header X-XSS-Protection "1; mode=block"; + add_header Referrer-Policy "strict-origin-when-cross-origin"; + add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; + + # Hide server information + server_tokens off; + + # Request size limits + client_max_body_size 50M; + client_body_timeout 30s; + client_header_timeout 30s; + + # Compression + gzip on; + gzip_vary on; + gzip_min_length 1024; + gzip_types text/plain text/css text/xml text/javascript application/javascript application/xml+rss application/json; + + # CakePHP legacy app routes + location / { + limit_req zone=api burst=10 nodelay; + + proxy_pass http://cmc_php_production; + proxy_read_timeout 300s; + proxy_connect_timeout 10s; + proxy_send_timeout 30s; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + + # Buffer settings for better performance + proxy_buffering on; + proxy_buffer_size 128k; + proxy_buffers 4 256k; + proxy_busy_buffers_size 256k; + } + + # Go API routes + location /api/ { + limit_req zone=api burst=20 nodelay; + + proxy_pass http://cmc_go_production; + proxy_read_timeout 300s; + proxy_connect_timeout 10s; + proxy_send_timeout 30s; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + + # Buffer settings for better performance + proxy_buffering on; + proxy_buffer_size 128k; + proxy_buffers 4 256k; + proxy_busy_buffers_size 256k; + } + + # Go page routes for emails + location ~ ^/(emails|customers|products|purchase-orders|enquiries|documents) { + limit_req zone=api burst=15 nodelay; + + proxy_pass http://cmc_go_production; + proxy_read_timeout 300s; + proxy_connect_timeout 10s; + proxy_send_timeout 30s; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + + # Buffer settings for better performance + proxy_buffering on; + proxy_buffer_size 128k; + proxy_buffers 4 256k; + proxy_busy_buffers_size 256k; + } + + # Static files from Go app with aggressive caching + location /static/ { + proxy_pass http://cmc_go_production; + proxy_cache_valid 200 24h; + add_header Cache-Control "public, max-age=86400"; + expires 1d; + } + + # PDF files with caching + location /pdf/ { + proxy_pass http://cmc_go_production; + proxy_cache_valid 200 1h; + add_header Cache-Control "public, max-age=3600"; + expires 1h; + } + + # Health check endpoints (no rate limiting) + location /health { + proxy_pass http://cmc_go_production/api/v1/health; + access_log off; + } + + # Block common attack patterns + location ~ /\. { + deny all; + access_log off; + log_not_found off; + } + + location ~ ~$ { + deny all; + access_log off; + log_not_found off; + } + + # Error pages + error_page 502 503 504 /50x.html; + location = /50x.html { + root /usr/share/nginx/html; + } + + # Custom error page for rate limiting + error_page 429 /429.html; + location = /429.html { + root /usr/share/nginx/html; + } + + listen 80; +} \ No newline at end of file diff --git a/conf/nginx-proxy.conf b/conf/nginx-proxy.conf new file mode 100644 index 00000000..e1652854 --- /dev/null +++ b/conf/nginx-proxy.conf @@ -0,0 +1,137 @@ +user nginx; +worker_processes auto; +error_log /var/log/nginx/error.log warn; +pid /var/run/nginx.pid; + +events { + worker_connections 1024; + use epoll; + multi_accept on; +} + +http { + include /etc/nginx/mime.types; + default_type application/octet-stream; + + # Logging + log_format main '$remote_addr - $remote_user [$time_local] "$request" ' + '$status $body_bytes_sent "$http_referer" ' + '"$http_user_agent" "$http_x_forwarded_for"'; + + access_log /var/log/nginx/access.log main; + + # Performance + sendfile on; + tcp_nopush on; + tcp_nodelay on; + keepalive_timeout 65; + types_hash_max_size 2048; + server_tokens off; + + # Gzip + gzip on; + gzip_vary on; + gzip_min_length 1024; + gzip_types text/plain text/css text/xml text/javascript application/javascript application/xml+rss application/json; + + # Rate limiting + limit_req_zone $binary_remote_addr zone=global:10m rate=10r/s; + + # SSL configuration + ssl_protocols TLSv1.2 TLSv1.3; + ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384; + ssl_prefer_server_ciphers off; + ssl_session_cache shared:SSL:10m; + ssl_session_timeout 10m; + + # Upstream servers + upstream cmc_staging { + server nginx-staging:80; + keepalive 32; + } + + upstream cmc_production { + server nginx-production:80; + keepalive 32; + } + + # Redirect HTTP to HTTPS + server { + listen 80; + server_name cmc.springupsoftware.com staging.cmc.springupsoftware.com; + + # ACME challenge for Lego + location /.well-known/acme-challenge/ { + root /var/www/acme-challenge; + try_files $uri =404; + } + + # Redirect all other traffic to HTTPS + location / { + return 301 https://$server_name$request_uri; + } + } + + # Production HTTPS + server { + listen 443 ssl http2; + server_name cmc.springupsoftware.com; + + ssl_certificate /etc/ssl/certs/cmc.springupsoftware.com.crt; + ssl_certificate_key /etc/ssl/certs/cmc.springupsoftware.com.key; + + # Security headers + add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; + add_header X-Frame-Options DENY; + add_header X-Content-Type-Options nosniff; + add_header X-XSS-Protection "1; mode=block"; + + # Rate limiting + limit_req zone=global burst=20 nodelay; + + location / { + proxy_pass http://cmc_production; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + + # Buffer settings + proxy_buffering on; + proxy_buffer_size 128k; + proxy_buffers 4 256k; + proxy_busy_buffers_size 256k; + } + } + + # Staging HTTPS + server { + listen 443 ssl http2; + server_name staging.cmc.springupsoftware.com; + + ssl_certificate /etc/ssl/certs/staging.cmc.springupsoftware.com.crt; + ssl_certificate_key /etc/ssl/certs/staging.cmc.springupsoftware.com.key; + + # Security headers (less strict for staging) + add_header X-Frame-Options DENY; + add_header X-Content-Type-Options nosniff; + add_header X-Environment "STAGING"; + + # Rate limiting (more lenient for staging) + limit_req zone=global burst=50 nodelay; + + location / { + proxy_pass http://cmc_staging; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + + # Buffer settings + proxy_buffering on; + proxy_buffer_size 128k; + proxy_buffers 4 256k; + proxy_busy_buffers_size 256k; + } + } +} \ No newline at end of file diff --git a/conf/nginx-staging.conf b/conf/nginx-staging.conf new file mode 100644 index 00000000..ac7ed8d8 --- /dev/null +++ b/conf/nginx-staging.conf @@ -0,0 +1,89 @@ +# Staging environment configuration +upstream cmc_php_staging { + server cmc-php-staging:80; +} + +upstream cmc_go_staging { + server cmc-go-staging:8080; +} + +server { + server_name staging.cmc.springupsoftware.com; + + # Basic auth for staging + auth_basic_user_file /etc/nginx/userpasswd; + auth_basic "CMC Sales Staging - Restricted Access"; + + # Security headers + add_header X-Frame-Options DENY; + add_header X-Content-Type-Options nosniff; + add_header X-XSS-Protection "1; mode=block"; + add_header Referrer-Policy "strict-origin-when-cross-origin"; + + # Staging banner + add_header X-Environment "STAGING"; + + # CakePHP legacy app routes + location / { + proxy_pass http://cmc_php_staging; + proxy_read_timeout 300s; + proxy_connect_timeout 10s; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Environment "staging"; + } + + # Go API routes + location /api/ { + proxy_pass http://cmc_go_staging; + proxy_read_timeout 300s; + proxy_connect_timeout 10s; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Environment "staging"; + } + + # Go page routes for emails + location ~ ^/(emails|customers|products|purchase-orders|enquiries|documents) { + proxy_pass http://cmc_go_staging; + proxy_read_timeout 300s; + proxy_connect_timeout 10s; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Environment "staging"; + } + + # Static files from Go app + location /static/ { + proxy_pass http://cmc_go_staging; + proxy_cache_valid 200 1h; + add_header Cache-Control "public, max-age=3600"; + } + + # PDF files + location /pdf/ { + proxy_pass http://cmc_go_staging; + proxy_cache_valid 200 1h; + add_header Cache-Control "public, max-age=3600"; + } + + # Health check endpoints + location /health { + proxy_pass http://cmc_go_staging/api/v1/health; + access_log off; + } + + # Error pages + error_page 502 503 504 /50x.html; + location = /50x.html { + root /usr/share/nginx/html; + } + + listen 80; +} \ No newline at end of file diff --git a/docker-compose.production.yml b/docker-compose.production.yml new file mode 100644 index 00000000..376e5bb8 --- /dev/null +++ b/docker-compose.production.yml @@ -0,0 +1,110 @@ +version: '3.8' + +services: + nginx-production: + image: nginx:latest + hostname: nginx-production + container_name: cmc-nginx-production + ports: + - "8080:80" # Internal port for production + volumes: + - ./conf/nginx-production.conf:/etc/nginx/conf.d/cmc-production.conf + - ./userpasswd:/etc/nginx/userpasswd:ro + depends_on: + - cmc-php-production + restart: unless-stopped + networks: + - cmc-production-network + environment: + - NGINX_ENVSUBST_TEMPLATE_SUFFIX=.template + + cmc-php-production: + build: + context: . + dockerfile: Dockerfile + platform: linux/amd64 + container_name: cmc-php-production + depends_on: + - db-production + volumes: + - production_pdf_data:/var/www/cmc-sales/app/webroot/pdf + - production_attachments_data:/var/www/cmc-sales/app/webroot/attachments_files + networks: + - cmc-production-network + restart: unless-stopped + environment: + - APP_ENV=production + # Remove development features + deploy: + resources: + limits: + cpus: '2.0' + memory: 2G + reservations: + cpus: '0.5' + memory: 512M + + db-production: + image: mariadb:latest + container_name: cmc-db-production + environment: + MYSQL_ROOT_PASSWORD: ${DB_ROOT_PASSWORD_PRODUCTION} + MYSQL_DATABASE: cmc + MYSQL_USER: cmc + MYSQL_PASSWORD: ${DB_PASSWORD_PRODUCTION} + volumes: + - production_db_data:/var/lib/mysql + - ./backups:/backups:ro # Backup restore directory + networks: + - cmc-production-network + restart: unless-stopped + # No external port exposure for security + deploy: + resources: + limits: + cpus: '2.0' + memory: 4G + reservations: + cpus: '0.5' + memory: 1G + + cmc-go-production: + build: + context: . + dockerfile: Dockerfile.go.production + container_name: cmc-go-production + environment: + DB_HOST: db-production + DB_PORT: 3306 + DB_USER: cmc + DB_PASSWORD: ${DB_PASSWORD_PRODUCTION} + DB_NAME: cmc + PORT: 8080 + APP_ENV: production + depends_on: + db-production: + condition: service_started + # No external port exposure - only through nginx + volumes: + - production_pdf_data:/root/webroot/pdf + - ./credentials/production:/root/credentials:ro # Production Gmail credentials + networks: + - cmc-production-network + restart: unless-stopped + deploy: + resources: + limits: + cpus: '2.0' + memory: 2G + reservations: + cpus: '0.5' + memory: 512M + +volumes: + production_db_data: + production_pdf_data: + production_attachments_data: + +networks: + cmc-production-network: + driver: bridge \ No newline at end of file diff --git a/docker-compose.proxy.yml b/docker-compose.proxy.yml new file mode 100644 index 00000000..616270ab --- /dev/null +++ b/docker-compose.proxy.yml @@ -0,0 +1,68 @@ +# Main reverse proxy for both staging and production +version: '3.8' + +services: + nginx-proxy: + image: nginx:latest + container_name: cmc-nginx-proxy + ports: + - "80:80" + - "443:443" + volumes: + - ./conf/nginx-proxy.conf:/etc/nginx/nginx.conf + - lego_certificates:/etc/ssl/certs:ro + - lego_acme_challenge:/var/www/acme-challenge:ro + restart: unless-stopped + depends_on: + - nginx-staging + - nginx-production + networks: + - proxy-network + - cmc-staging-network + - cmc-production-network + + lego: + image: goacme/lego:latest + container_name: cmc-lego + volumes: + - lego_certificates:/data/certificates + - lego_accounts:/data/accounts + - lego_acme_challenge:/data/acme-challenge + - ./scripts:/scripts:ro + environment: + - LEGO_DISABLE_CNAME=true + command: sleep infinity + restart: unless-stopped + networks: + - proxy-network + + # Import staging services + nginx-staging: + extends: + file: docker-compose.staging.yml + service: nginx-staging + networks: + - proxy-network + - cmc-staging-network + + # Import production services + nginx-production: + extends: + file: docker-compose.production.yml + service: nginx-production + networks: + - proxy-network + - cmc-production-network + +volumes: + lego_certificates: + lego_accounts: + lego_acme_challenge: + +networks: + proxy-network: + driver: bridge + cmc-staging-network: + external: true + cmc-production-network: + external: true \ No newline at end of file diff --git a/docker-compose.staging.yml b/docker-compose.staging.yml new file mode 100644 index 00000000..cde10cad --- /dev/null +++ b/docker-compose.staging.yml @@ -0,0 +1,86 @@ +version: '3.8' + +services: + nginx-staging: + image: nginx:latest + hostname: nginx-staging + container_name: cmc-nginx-staging + ports: + - "8081:80" # Internal port for staging + volumes: + - ./conf/nginx-staging.conf:/etc/nginx/conf.d/cmc-staging.conf + - ./userpasswd:/etc/nginx/userpasswd:ro + depends_on: + - cmc-php-staging + restart: unless-stopped + networks: + - cmc-staging-network + environment: + - NGINX_ENVSUBST_TEMPLATE_SUFFIX=.template + + cmc-php-staging: + build: + context: . + dockerfile: Dockerfile + platform: linux/amd64 + container_name: cmc-php-staging + depends_on: + - db-staging + volumes: + - staging_pdf_data:/var/www/cmc-sales/app/webroot/pdf + - staging_attachments_data:/var/www/cmc-sales/app/webroot/attachments_files + networks: + - cmc-staging-network + restart: unless-stopped + environment: + - APP_ENV=staging + + db-staging: + image: mariadb:latest + container_name: cmc-db-staging + environment: + MYSQL_ROOT_PASSWORD: ${DB_ROOT_PASSWORD_STAGING} + MYSQL_DATABASE: cmc_staging + MYSQL_USER: cmc_staging + MYSQL_PASSWORD: ${DB_PASSWORD_STAGING} + volumes: + - staging_db_data:/var/lib/mysql + networks: + - cmc-staging-network + restart: unless-stopped + ports: + - "3307:3306" # Different port for staging DB access + + cmc-go-staging: + build: + context: . + dockerfile: Dockerfile.go.staging + container_name: cmc-go-staging + environment: + DB_HOST: db-staging + DB_PORT: 3306 + DB_USER: cmc_staging + DB_PASSWORD: ${DB_PASSWORD_STAGING} + DB_NAME: cmc_staging + PORT: 8080 + APP_ENV: staging + depends_on: + db-staging: + condition: service_started + ports: + - "8082:8080" # Direct access for testing + volumes: + - staging_pdf_data:/root/webroot/pdf + - ./credentials/staging:/root/credentials:ro # Staging Gmail credentials + networks: + - cmc-staging-network + restart: unless-stopped + +volumes: + staging_db_data: + staging_pdf_data: + staging_attachments_data: + +networks: + cmc-staging-network: + driver: bridge \ No newline at end of file diff --git a/go-app/.gitignore b/go-app/.gitignore index 56c712b8..ea2b3066 100644 --- a/go-app/.gitignore +++ b/go-app/.gitignore @@ -30,4 +30,11 @@ vendor/ # OS specific files .DS_Store -Thumbs.db \ No newline at end of file +Thumbs.db + +# Goose database migration config +goose.env + +# Gmail OAuth credentials - NEVER commit these! +credentials.json +token.json diff --git a/go-app/MIGRATIONS.md b/go-app/MIGRATIONS.md new file mode 100644 index 00000000..012636f7 --- /dev/null +++ b/go-app/MIGRATIONS.md @@ -0,0 +1,70 @@ +# Database Migrations with Goose + +This document explains how to use goose for database migrations in the CMC Sales Go application. + +## Setup + +1. **Install goose**: + ```bash + make install + ``` + +2. **Configure database connection**: + ```bash + cp goose.env.example goose.env + # Edit goose.env with your database credentials + ``` + +## Migration Commands + +### Run Migrations +```bash +# Run all pending migrations +make migrate + +# Check migration status +make migrate-status +``` + +### Rollback Migrations +```bash +# Rollback the last migration +make migrate-down +``` + +### Create New Migrations +```bash +# Create a new migration file +make migrate-create name=add_new_feature +``` + +## Migration Structure + +Migrations are stored in `sql/migrations/` and follow this naming convention: +- `001_add_gmail_fields.sql` +- `002_add_new_feature.sql` + +Each migration file contains: +```sql +-- +goose Up +-- Your upgrade SQL here + +-- +goose Down +-- Your rollback SQL here +``` + +## Configuration Files + +- `goose.env` - Database connection settings (gitignored) +- `goose.env.example` - Template for goose.env + +## Current Migrations + +1. **001_add_gmail_fields.sql** - Adds Gmail integration fields to emails and email_attachments tables + +## Tips + +- Always test migrations on a backup database first +- Use `make migrate-status` to check current state +- Migrations are atomic - if one fails, none are applied +- Each migration should be reversible with a corresponding Down section \ No newline at end of file diff --git a/go-app/Makefile b/go-app/Makefile index 37e17ecd..66a3dea9 100644 --- a/go-app/Makefile +++ b/go-app/Makefile @@ -11,6 +11,7 @@ install: ## Install dependencies go env -w GOPRIVATE=code.springupsoftware.com go mod download go install github.com/sqlc-dev/sqlc/cmd/sqlc@latest + go install github.com/pressly/goose/v3/cmd/goose@latest .PHONY: sqlc sqlc: ## Generate Go code from SQL queries @@ -76,4 +77,48 @@ dbshell-root: ## Connect to MariaDB as root user exit 1; \ else \ docker compose exec -e MYSQL_PWD="$$DB_ROOT_PASSWORD" db mariadb -u root; \ + fi + +.PHONY: migrate +migrate: ## Run database migrations + @echo "Running database migrations..." + @if [ -f goose.env ]; then \ + export $$(cat goose.env | xargs) && goose up; \ + else \ + echo "Error: goose.env file not found"; \ + exit 1; \ + fi + +.PHONY: migrate-down +migrate-down: ## Rollback last migration + @echo "Rolling back last migration..." + @if [ -f goose.env ]; then \ + export $$(cat goose.env | xargs) && goose down; \ + else \ + echo "Error: goose.env file not found"; \ + exit 1; \ + fi + +.PHONY: migrate-status +migrate-status: ## Show migration status + @echo "Migration status:" + @if [ -f goose.env ]; then \ + export $$(cat goose.env | xargs) && goose status; \ + else \ + echo "Error: goose.env file not found"; \ + exit 1; \ + fi + +.PHONY: migrate-create +migrate-create: ## Create a new migration file (use: make migrate-create name=add_new_table) + @if [ -z "$(name)" ]; then \ + echo "Error: Please provide a migration name. Usage: make migrate-create name=add_new_table"; \ + exit 1; \ + fi + @echo "Creating new migration: $(name)" + @if [ -f goose.env ]; then \ + export $$(cat goose.env | xargs) && goose create $(name) sql; \ + else \ + echo "Error: goose.env file not found"; \ + exit 1; \ fi \ No newline at end of file diff --git a/go-app/cmd/server/main.go b/go-app/cmd/server/main.go index 954dc444..bc361765 100644 --- a/go-app/cmd/server/main.go +++ b/go-app/cmd/server/main.go @@ -58,12 +58,13 @@ func main() { purchaseOrderHandler := handlers.NewPurchaseOrderHandler(queries) enquiryHandler := handlers.NewEnquiryHandler(queries) documentHandler := handlers.NewDocumentHandler(queries) - pageHandler := handlers.NewPageHandler(queries, tmpl) + pageHandler := handlers.NewPageHandler(queries, tmpl, database) addressHandler := handlers.NewAddressHandler(queries) attachmentHandler := handlers.NewAttachmentHandler(queries) countryHandler := handlers.NewCountryHandler(queries) statusHandler := handlers.NewStatusHandler(queries) lineItemHandler := handlers.NewLineItemHandler(queries) + emailHandler := handlers.NewEmailHandler(queries, database) // Setup routes r := mux.NewRouter() @@ -161,6 +162,14 @@ func main() { api.HandleFunc("/line-items/{id}", lineItemHandler.Delete).Methods("DELETE") api.HandleFunc("/line-items/document/{documentID}/table", lineItemHandler.GetTable).Methods("GET") + // Email routes + api.HandleFunc("/emails", emailHandler.List).Methods("GET") + api.HandleFunc("/emails/{id}", emailHandler.Get).Methods("GET") + api.HandleFunc("/emails/{id}/content", emailHandler.StreamContent).Methods("GET") + api.HandleFunc("/emails/{id}/attachments", emailHandler.ListAttachments).Methods("GET") + api.HandleFunc("/emails/{id}/attachments/{attachmentId}/stream", emailHandler.StreamAttachment).Methods("GET") + api.HandleFunc("/emails/search", emailHandler.Search).Methods("GET") + // Health check api.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusOK) @@ -208,6 +217,12 @@ func main() { r.HandleFunc("/documents/{id}", pageHandler.DocumentsShow).Methods("GET") r.HandleFunc("/documents/pdf/{id}", documentHandler.GeneratePDF).Methods("GET") + // Email pages + r.HandleFunc("/emails", pageHandler.EmailsIndex).Methods("GET") + r.HandleFunc("/emails/search", pageHandler.EmailsSearch).Methods("GET") + r.HandleFunc("/emails/{id}", pageHandler.EmailsShow).Methods("GET") + r.HandleFunc("/emails/{id}/attachments", pageHandler.EmailsAttachments).Methods("GET") + // Address routes (matching CakePHP) r.HandleFunc("/addresses", addressHandler.List).Methods("GET") r.HandleFunc("/addresses/view/{id}", addressHandler.Get).Methods("GET") diff --git a/go-app/cmd/vault/README.md b/go-app/cmd/vault/README.md index 908312c5..e7125816 100644 --- a/go-app/cmd/vault/README.md +++ b/go-app/cmd/vault/README.md @@ -1,28 +1,23 @@ -# Vault Email Processor +# Vault Email Processor - Smart Proxy -This is a Go rewrite of the PHP vault.php script that processes emails for the CMC Sales system. +This is a Go rewrite of the PHP vault.php script that processes emails for the CMC Sales system. It now supports three modes: local file processing, Gmail indexing, and HTTP streaming proxy. -## Key Changes from PHP Version +## Key Features -1. **No ripmime dependency**: Uses the enmime Go library for MIME parsing instead of external ripmime binary -2. **Better error handling**: Proper error handling and database transactions -3. **Type safety**: Strongly typed Go structures -4. **Modern email parsing**: Uses enmime for robust email parsing +1. **Gmail Integration**: Index Gmail emails without downloading +2. **Smart Proxy**: Stream email content on-demand without storing to disk +3. **No ripmime dependency**: Uses the enmime Go library for MIME parsing +4. **Better error handling**: Proper error handling and database transactions +5. **Type safety**: Strongly typed Go structures +6. **Modern email parsing**: Uses enmime for robust email parsing -## Features +## Operating Modes -- Processes emails from vault directory -- Parses email headers and extracts recipients -- Identifies and creates users as needed -- Extracts attachments and email body parts -- Matches document identifiers in subjects (enquiries, invoices, POs, jobs) -- Saves emails with all associations to database -- Moves processed emails to processed directory - -## Usage +### 1. Local Mode (Original functionality) +Processes emails from local filesystem directories. ```bash -go run cmd/vault/main.go \ +./vault --mode=local \ --emaildir=/var/www/emails \ --vaultdir=/var/www/vaultmsgs/new \ --processeddir=/var/www/vaultmsgs/cur \ @@ -32,6 +27,83 @@ go run cmd/vault/main.go \ --dbname=cmc ``` +### 2. Gmail Index Mode +Indexes Gmail emails without downloading content. Creates database references only. + +```bash +./vault --mode=index \ + --gmail-query="is:unread" \ + --credentials=credentials.json \ + --token=token.json \ + --dbhost=127.0.0.1 \ + --dbuser=cmc \ + --dbpass="xVRQI&cA?7AU=hqJ!%au" \ + --dbname=cmc +``` + +### 3. HTTP Server Mode +Runs an HTTP server that streams Gmail content on-demand. + +```bash +./vault --mode=serve \ + --port=8080 \ + --credentials=credentials.json \ + --token=token.json \ + --dbhost=127.0.0.1 \ + --dbuser=cmc \ + --dbpass="xVRQI&cA?7AU=hqJ!%au" \ + --dbname=cmc +``` + +## Gmail Setup + +1. Enable Gmail API in Google Cloud Console +2. Create OAuth 2.0 credentials +3. Download credentials as `credentials.json` +4. Run vault in any Gmail mode - it will prompt for authorization +5. Token will be saved as `token.json` for future use + +## API Endpoints (Server Mode) + +- `GET /api/emails` - List indexed emails (metadata only) +- `GET /api/emails/:id` - Get email metadata +- `GET /api/emails/:id/content` - Stream email HTML/text from Gmail +- `GET /api/emails/:id/attachments` - List attachment metadata +- `GET /api/emails/:id/attachments/:attachmentId` - Stream attachment from Gmail +- `GET /api/emails/:id/raw` - Stream raw email (for email clients) + +## Database Schema Changes + +Required migrations for Gmail support: + +```sql +ALTER TABLE emails + ADD COLUMN gmail_message_id VARCHAR(255) UNIQUE, + ADD COLUMN gmail_thread_id VARCHAR(255), + ADD COLUMN is_downloaded BOOLEAN DEFAULT FALSE, + ADD COLUMN raw_headers TEXT; + +CREATE INDEX idx_gmail_message_id ON emails(gmail_message_id); + +ALTER TABLE email_attachments + ADD COLUMN gmail_attachment_id VARCHAR(255), + ADD COLUMN gmail_message_id VARCHAR(255); +``` + +## Architecture + +### Smart Proxy Benefits +- **No Disk Storage**: Emails/attachments streamed directly from Gmail +- **Low Storage Footprint**: Only metadata stored in database +- **Fresh Content**: Always serves latest version from Gmail +- **Scalable**: No file management overhead +- **On-Demand**: Content fetched only when requested + +### Processing Flow +1. **Index Mode**: Scans Gmail, stores metadata, creates associations +2. **Server Mode**: Receives HTTP requests, fetches from Gmail, streams to client +3. **Local Mode**: Original file-based processing (backwards compatible) + ## Build ```bash @@ -41,17 +113,28 @@ go build -o vault cmd/vault/main.go ## Dependencies - github.com/jhillyerd/enmime - MIME email parsing -- github.com/google/uuid - UUID generation for unique filenames +- github.com/google/uuid - UUID generation - github.com/go-sql-driver/mysql - MySQL driver +- github.com/gorilla/mux - HTTP router +- golang.org/x/oauth2 - OAuth2 support +- google.golang.org/api/gmail/v1 - Gmail API client ## Database Tables Used -- emails - Main email records +- emails - Main email records with Gmail metadata - email_recipients - To/CC recipients -- email_attachments - File attachments +- email_attachments - Attachment metadata (no file storage) - emails_enquiries - Email to enquiry associations - emails_invoices - Email to invoice associations - emails_purchase_orders - Email to PO associations - emails_jobs - Email to job associations - users - System users -- enquiries, invoices, purchase_orders, jobs - For identifier matching \ No newline at end of file +- enquiries, invoices, purchase_orders, jobs - For identifier matching + +## Gmail Query Examples + +- `is:unread` - Unread emails +- `newer_than:1d` - Emails from last 24 hours +- `from:customer@example.com` - From specific sender +- `subject:invoice` - Subject contains "invoice" +- `has:attachment` - Emails with attachments \ No newline at end of file diff --git a/go-app/cmd/vault/main.go b/go-app/cmd/vault/main.go index 45d1cfa9..8917a204 100644 --- a/go-app/cmd/vault/main.go +++ b/go-app/cmd/vault/main.go @@ -2,11 +2,15 @@ package main import ( "bytes" + "context" "database/sql" + "encoding/base64" + "encoding/json" "flag" "fmt" "io/ioutil" "log" + "net/http" "os" "path/filepath" "regexp" @@ -15,39 +19,82 @@ import ( _ "github.com/go-sql-driver/mysql" "github.com/google/uuid" + "github.com/gorilla/mux" "github.com/jhillyerd/enmime" + "golang.org/x/oauth2" + "golang.org/x/oauth2/google" + "google.golang.org/api/gmail/v1" + "google.golang.org/api/option" ) type Config struct { - EmailDir string - VaultDir string - ProcessedDir string - DBHost string - DBUser string - DBPassword string - DBName string + Mode string + EmailDir string + VaultDir string + ProcessedDir string + DBHost string + DBUser string + DBPassword string + DBName string + Port string + CredentialsFile string + TokenFile string + GmailQuery string +} + +type VaultService struct { + db *sql.DB + config Config + gmailService *gmail.Service + indexer *GmailIndexer + processor *EmailProcessor + server *HTTPServer } type EmailProcessor struct { - db *sql.DB - config Config - enquiryMap map[string]int - invoiceMap map[string]int - poMap map[string]int - userMap map[string]int - jobMap map[string]int + db *sql.DB + config Config + enquiryMap map[string]int + invoiceMap map[string]int + poMap map[string]int + userMap map[string]int + jobMap map[string]int } -type Attachment struct { - Type string - Name string - Filename string - Size int64 - IsMessageBody int +type GmailIndexer struct { + db *sql.DB + gmailService *gmail.Service + processor *EmailProcessor +} + +type HTTPServer struct { + db *sql.DB + gmailService *gmail.Service + processor *EmailProcessor +} + +type EmailMetadata struct { + Subject string + From []string + To []string + CC []string + Date time.Time + GmailMessageID string + GmailThreadID string + AttachmentCount int + Attachments []AttachmentMeta +} + +type AttachmentMeta struct { + Filename string + ContentType string + Size int + GmailAttachmentID string } func main() { var config Config + flag.StringVar(&config.Mode, "mode", "serve", "Mode: index, serve, or local") flag.StringVar(&config.EmailDir, "emaildir", "/var/www/emails", "Email storage directory") flag.StringVar(&config.VaultDir, "vaultdir", "/var/www/vaultmsgs/new", "Vault messages directory") flag.StringVar(&config.ProcessedDir, "processeddir", "/var/www/vaultmsgs/cur", "Processed messages directory") @@ -55,17 +102,23 @@ func main() { flag.StringVar(&config.DBUser, "dbuser", "cmc", "Database user") flag.StringVar(&config.DBPassword, "dbpass", "xVRQI&cA?7AU=hqJ!%au", "Database password") flag.StringVar(&config.DBName, "dbname", "cmc", "Database name") + flag.StringVar(&config.Port, "port", "8080", "HTTP server port") + flag.StringVar(&config.CredentialsFile, "credentials", "credentials.json", "Gmail credentials file") + flag.StringVar(&config.TokenFile, "token", "token.json", "Gmail token file") + flag.StringVar(&config.GmailQuery, "gmail-query", "is:unread", "Gmail search query") flag.Parse() + // Connect to database dsn := fmt.Sprintf("%s:%s@tcp(%s)/%s?parseTime=true", config.DBUser, config.DBPassword, config.DBHost, config.DBName) - + db, err := sql.Open("mysql", dsn) if err != nil { log.Fatal("Failed to connect to database:", err) } defer db.Close() + // Create processor processor := &EmailProcessor{ db: db, config: config, @@ -75,11 +128,623 @@ func main() { log.Fatal("Failed to load maps:", err) } - if err := processor.processEmails(); err != nil { - log.Fatal("Failed to process emails:", err) + // Create vault service + service := &VaultService{ + db: db, + config: config, + processor: processor, + } + + switch config.Mode { + case "index": + // Initialize Gmail service + gmailService, err := getGmailService(config.CredentialsFile, config.TokenFile) + if err != nil { + log.Fatal("Failed to get Gmail service:", err) + } + service.gmailService = gmailService + + // Create and run indexer + service.indexer = &GmailIndexer{ + db: db, + gmailService: gmailService, + processor: processor, + } + + if err := service.indexer.IndexEmails(config.GmailQuery); err != nil { + log.Fatal("Failed to index emails:", err) + } + + case "serve": + // Initialize Gmail service + gmailService, err := getGmailService(config.CredentialsFile, config.TokenFile) + if err != nil { + log.Fatal("Failed to get Gmail service:", err) + } + service.gmailService = gmailService + + // Create and start HTTP server + service.server = &HTTPServer{ + db: db, + gmailService: gmailService, + processor: processor, + } + + log.Printf("Starting HTTP server on port %s", config.Port) + service.server.Start(config.Port) + + case "local": + // Original file-based processing + if err := processor.processEmails(); err != nil { + log.Fatal("Failed to process emails:", err) + } + + default: + log.Fatal("Invalid mode. Use: index, serve, or local") } } +// Gmail OAuth2 functions +func getGmailService(credentialsFile, tokenFile string) (*gmail.Service, error) { + ctx := context.Background() + + b, err := ioutil.ReadFile(credentialsFile) + if err != nil { + return nil, fmt.Errorf("unable to read client secret file: %v", err) + } + + config, err := google.ConfigFromJSON(b, gmail.GmailReadonlyScope) + if err != nil { + return nil, fmt.Errorf("unable to parse client secret file to config: %v", err) + } + + client := getClient(config, tokenFile) + srv, err := gmail.NewService(ctx, option.WithHTTPClient(client)) + if err != nil { + return nil, fmt.Errorf("unable to retrieve Gmail client: %v", err) + } + + return srv, nil +} + +func getClient(config *oauth2.Config, tokFile string) *http.Client { + tok, err := tokenFromFile(tokFile) + if err != nil { + tok = getTokenFromWeb(config) + saveToken(tokFile, tok) + } + return config.Client(context.Background(), tok) +} + +func getTokenFromWeb(config *oauth2.Config) *oauth2.Token { + authURL := config.AuthCodeURL("state-token", oauth2.AccessTypeOffline) + fmt.Printf("Go to the following link in your browser then type the authorization code: \n%v\n", authURL) + + var authCode string + if _, err := fmt.Scan(&authCode); err != nil { + log.Fatalf("Unable to read authorization code: %v", err) + } + + tok, err := config.Exchange(context.TODO(), authCode) + if err != nil { + log.Fatalf("Unable to retrieve token from web: %v", err) + } + return tok +} + +func tokenFromFile(file string) (*oauth2.Token, error) { + f, err := os.Open(file) + if err != nil { + return nil, err + } + defer f.Close() + tok := &oauth2.Token{} + err = json.NewDecoder(f).Decode(tok) + return tok, err +} + +func saveToken(path string, token *oauth2.Token) { + fmt.Printf("Saving credential file to: %s\n", path) + f, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0600) + if err != nil { + log.Fatalf("Unable to cache oauth token: %v", err) + } + defer f.Close() + json.NewEncoder(f).Encode(token) +} + +// GmailIndexer methods +func (g *GmailIndexer) IndexEmails(query string) error { + user := "me" + var pageToken string + + for { + // List messages with query + call := g.gmailService.Users.Messages.List(user).Q(query).MaxResults(500) + if pageToken != "" { + call = call.PageToken(pageToken) + } + + response, err := call.Do() + if err != nil { + return fmt.Errorf("unable to retrieve messages: %v", err) + } + + // Process each message + for _, msg := range response.Messages { + if err := g.indexMessage(msg.Id); err != nil { + log.Printf("Error indexing message %s: %v", msg.Id, err) + continue + } + } + + // Check for more pages + pageToken = response.NextPageToken + if pageToken == "" { + break + } + + log.Printf("Processed %d messages, continuing with next page...", len(response.Messages)) + } + + return nil +} + +func (g *GmailIndexer) indexMessage(messageID string) error { + // Get message metadata only + message, err := g.gmailService.Users.Messages.Get("me", messageID). + Format("METADATA"). + MetadataHeaders("From", "To", "Cc", "Subject", "Date"). + Do() + if err != nil { + return err + } + + // Extract headers + headers := make(map[string]string) + for _, header := range message.Payload.Headers { + headers[header.Name] = header.Value + } + + // Parse email addresses + toRecipients := g.processor.parseEnmimeAddresses(headers["To"]) + fromRecipients := g.processor.parseEnmimeAddresses(headers["From"]) + ccRecipients := g.processor.parseEnmimeAddresses(headers["Cc"]) + + // Check if we should save this email + saveThis := false + fromKnownUser := false + + for _, email := range toRecipients { + if g.processor.userExists(email) { + saveThis = true + } + } + + for _, email := range fromRecipients { + if g.processor.userExists(email) { + saveThis = true + fromKnownUser = true + } + } + + for _, email := range ccRecipients { + if g.processor.userExists(email) { + saveThis = true + } + } + + subject := headers["Subject"] + if subject == "" { + return nil // Skip emails without subject + } + + // Check for identifiers in subject + foundEnquiries := g.processor.checkIdentifier(subject, g.processor.enquiryMap, "enquiry") + foundInvoices := g.processor.checkIdentifier(subject, g.processor.invoiceMap, "invoice") + foundPOs := g.processor.checkIdentifier(subject, g.processor.poMap, "purchaseorder") + foundJobs := g.processor.checkIdentifier(subject, g.processor.jobMap, "job") + + foundIdent := len(foundEnquiries) > 0 || len(foundInvoices) > 0 || + len(foundPOs) > 0 || len(foundJobs) > 0 + + if fromKnownUser || saveThis || foundIdent { + // Parse date + unixTime := time.Now().Unix() + if dateStr := headers["Date"]; dateStr != "" { + if t, err := time.Parse(time.RFC1123Z, dateStr); err == nil { + unixTime = t.Unix() + } + } + + // Get recipient user IDs + recipientIDs := make(map[string][]int) + recipientIDs["to"] = g.processor.getUserIDs(toRecipients) + recipientIDs["from"] = g.processor.getUserIDs(fromRecipients) + recipientIDs["cc"] = g.processor.getUserIDs(ccRecipients) + + if len(recipientIDs["from"]) == 0 { + return nil // Skip if no from recipient + } + + // Marshal headers for storage + headerJSON, _ := json.Marshal(headers) + + // Count attachments (from message metadata) + attachmentCount := 0 + if message.Payload != nil { + attachmentCount = countAttachments(message.Payload) + } + + fmt.Printf("Indexing message: %s - Subject: %s\n", messageID, subject) + + // Save to database + if err := g.saveEmailMetadata(messageID, message.ThreadId, subject, string(headerJSON), + unixTime, recipientIDs, attachmentCount, message.Payload, + foundEnquiries, foundInvoices, foundPOs, foundJobs); err != nil { + return err + } + } + + return nil +} + +func countAttachments(payload *gmail.MessagePart) int { + count := 0 + if payload.Body != nil && payload.Body.AttachmentId != "" { + count++ + } + for _, part := range payload.Parts { + count += countAttachments(part) + } + return count +} + +func (g *GmailIndexer) saveEmailMetadata(gmailMessageID, threadID, subject, headers string, + unixTime int64, recipientIDs map[string][]int, attachmentCount int, + payload *gmail.MessagePart, foundEnquiries, foundInvoices, foundPOs, foundJobs []int) error { + + tx, err := g.db.Begin() + if err != nil { + return err + } + defer tx.Rollback() + + // Insert email + result, err := tx.Exec( + `INSERT INTO emails (user_id, udate, created, subject, gmail_message_id, + gmail_thread_id, raw_headers, is_downloaded, email_attachment_count) + VALUES (?, ?, NOW(), ?, ?, ?, ?, FALSE, ?)`, + recipientIDs["from"][0], unixTime, subject, gmailMessageID, + threadID, headers, attachmentCount) + + if err != nil { + return err + } + + emailID, err := result.LastInsertId() + if err != nil { + return err + } + + // Insert recipients + for recipType, userIDs := range recipientIDs { + for _, userID := range userIDs { + if recipType == "from" { + continue // From is already stored in emails.user_id + } + _, err = tx.Exec( + "INSERT INTO email_recipients (email_id, user_id, type) VALUES (?, ?, ?)", + emailID, userID, recipType) + if err != nil { + return err + } + } + } + + // Index attachment metadata + if payload != nil { + if err := g.indexAttachments(tx, emailID, gmailMessageID, payload); err != nil { + return err + } + } + + // Insert associations + for _, jobID := range foundJobs { + _, err = tx.Exec("INSERT INTO emails_jobs (email_id, job_id) VALUES (?, ?)", emailID, jobID) + if err != nil { + return err + } + } + + for _, poID := range foundPOs { + _, err = tx.Exec("INSERT INTO emails_purchase_orders (email_id, purchase_order_id) VALUES (?, ?)", emailID, poID) + if err != nil { + return err + } + } + + for _, enqID := range foundEnquiries { + _, err = tx.Exec("INSERT INTO emails_enquiries (email_id, enquiry_id) VALUES (?, ?)", emailID, enqID) + if err != nil { + return err + } + } + + for _, invID := range foundInvoices { + _, err = tx.Exec("INSERT INTO emails_invoices (email_id, invoice_id) VALUES (?, ?)", emailID, invID) + if err != nil { + return err + } + } + + return tx.Commit() +} + +func (g *GmailIndexer) indexAttachments(tx *sql.Tx, emailID int64, gmailMessageID string, part *gmail.MessagePart) error { + // Check if this part is an attachment + if part.Body != nil && part.Body.AttachmentId != "" { + filename := part.Filename + if filename == "" { + filename = "attachment" + } + + _, err := tx.Exec( + `INSERT INTO email_attachments + (email_id, gmail_attachment_id, gmail_message_id, type, size, filename, is_message_body, created) + VALUES (?, ?, ?, ?, ?, ?, 0, NOW())`, + emailID, part.Body.AttachmentId, gmailMessageID, part.MimeType, part.Body.Size, filename) + if err != nil { + return err + } + } + + // Process sub-parts + for _, subPart := range part.Parts { + if err := g.indexAttachments(tx, emailID, gmailMessageID, subPart); err != nil { + return err + } + } + + return nil +} + +// HTTPServer methods +func (s *HTTPServer) Start(port string) { + router := mux.NewRouter() + + // API routes + router.HandleFunc("/api/emails", s.ListEmails).Methods("GET") + router.HandleFunc("/api/emails/{id:[0-9]+}", s.GetEmail).Methods("GET") + router.HandleFunc("/api/emails/{id:[0-9]+}/content", s.StreamEmailContent).Methods("GET") + router.HandleFunc("/api/emails/{id:[0-9]+}/attachments", s.ListAttachments).Methods("GET") + router.HandleFunc("/api/emails/{id:[0-9]+}/attachments/{attachmentId:[0-9]+}", s.StreamAttachment).Methods("GET") + router.HandleFunc("/api/emails/{id:[0-9]+}/raw", s.StreamRawEmail).Methods("GET") + + log.Fatal(http.ListenAndServe(":"+port, router)) +} + +func (s *HTTPServer) ListEmails(w http.ResponseWriter, r *http.Request) { + // TODO: Add pagination + rows, err := s.db.Query(` + SELECT id, subject, user_id, created, gmail_message_id, email_attachment_count + FROM emails + ORDER BY created DESC + LIMIT 100`) + if err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } + defer rows.Close() + + var emails []map[string]interface{} + for rows.Next() { + var id, userID, attachmentCount int + var subject, gmailMessageID string + var created time.Time + + if err := rows.Scan(&id, &subject, &userID, &created, &gmailMessageID, &attachmentCount); err != nil { + continue + } + + emails = append(emails, map[string]interface{}{ + "id": id, + "subject": subject, + "user_id": userID, + "created": created, + "gmail_message_id": gmailMessageID, + "attachment_count": attachmentCount, + }) + } + + w.Header().Set("Content-Type", "application/json") + json.NewEncoder(w).Encode(emails) +} + +func (s *HTTPServer) GetEmail(w http.ResponseWriter, r *http.Request) { + vars := mux.Vars(r) + emailID := vars["id"] + + var gmailMessageID, subject, rawHeaders string + var created time.Time + + err := s.db.QueryRow(` + SELECT gmail_message_id, subject, created, raw_headers + FROM emails WHERE id = ?`, emailID). + Scan(&gmailMessageID, &subject, &created, &rawHeaders) + + if err != nil { + http.Error(w, "Email not found", http.StatusNotFound) + return + } + + response := map[string]interface{}{ + "id": emailID, + "gmail_message_id": gmailMessageID, + "subject": subject, + "created": created, + "headers": rawHeaders, + } + + w.Header().Set("Content-Type", "application/json") + json.NewEncoder(w).Encode(response) +} + +func (s *HTTPServer) StreamEmailContent(w http.ResponseWriter, r *http.Request) { + vars := mux.Vars(r) + emailID := vars["id"] + + // Get Gmail message ID from database + var gmailMessageID string + err := s.db.QueryRow("SELECT gmail_message_id FROM emails WHERE id = ?", emailID). + Scan(&gmailMessageID) + if err != nil { + http.Error(w, "Email not found", http.StatusNotFound) + return + } + + // Fetch from Gmail + message, err := s.gmailService.Users.Messages.Get("me", gmailMessageID). + Format("RAW").Do() + if err != nil { + http.Error(w, "Failed to fetch email from Gmail", http.StatusInternalServerError) + return + } + + // Decode message + rawEmail, err := base64.URLEncoding.DecodeString(message.Raw) + if err != nil { + http.Error(w, "Failed to decode email", http.StatusInternalServerError) + return + } + + // Parse with enmime + env, err := enmime.ReadEnvelope(bytes.NewReader(rawEmail)) + if err != nil { + http.Error(w, "Failed to parse email", http.StatusInternalServerError) + return + } + + // Stream HTML or Text directly to client + if env.HTML != "" { + w.Header().Set("Content-Type", "text/html; charset=utf-8") + w.Write([]byte(env.HTML)) + } else if env.Text != "" { + w.Header().Set("Content-Type", "text/plain; charset=utf-8") + w.Write([]byte(env.Text)) + } else { + http.Error(w, "No content found in email", http.StatusNotFound) + } +} + +func (s *HTTPServer) ListAttachments(w http.ResponseWriter, r *http.Request) { + vars := mux.Vars(r) + emailID := vars["id"] + + rows, err := s.db.Query(` + SELECT id, filename, type, size, gmail_attachment_id + FROM email_attachments + WHERE email_id = ?`, emailID) + if err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } + defer rows.Close() + + var attachments []map[string]interface{} + for rows.Next() { + var id, size int + var filename, contentType, gmailAttachmentID string + + if err := rows.Scan(&id, &filename, &contentType, &size, &gmailAttachmentID); err != nil { + continue + } + + attachments = append(attachments, map[string]interface{}{ + "id": id, + "filename": filename, + "content_type": contentType, + "size": size, + "gmail_attachment_id": gmailAttachmentID, + }) + } + + w.Header().Set("Content-Type", "application/json") + json.NewEncoder(w).Encode(attachments) +} + +func (s *HTTPServer) StreamAttachment(w http.ResponseWriter, r *http.Request) { + vars := mux.Vars(r) + attachmentID := vars["attachmentId"] + + // Get attachment info from database + var gmailMessageID, gmailAttachmentID, filename, contentType string + err := s.db.QueryRow(` + SELECT gmail_message_id, gmail_attachment_id, filename, type + FROM email_attachments + WHERE id = ?`, attachmentID). + Scan(&gmailMessageID, &gmailAttachmentID, &filename, &contentType) + + if err != nil { + http.Error(w, "Attachment not found", http.StatusNotFound) + return + } + + // Fetch from Gmail + attachment, err := s.gmailService.Users.Messages.Attachments. + Get("me", gmailMessageID, gmailAttachmentID).Do() + if err != nil { + http.Error(w, "Failed to fetch attachment from Gmail", http.StatusInternalServerError) + return + } + + // Decode base64 + data, err := base64.URLEncoding.DecodeString(attachment.Data) + if err != nil { + http.Error(w, "Failed to decode attachment", http.StatusInternalServerError) + return + } + + // Set headers and stream + w.Header().Set("Content-Type", contentType) + w.Header().Set("Content-Disposition", fmt.Sprintf("inline; filename=\"%s\"", filename)) + w.Header().Set("Content-Length", fmt.Sprintf("%d", len(data))) + w.Write(data) +} + +func (s *HTTPServer) StreamRawEmail(w http.ResponseWriter, r *http.Request) { + vars := mux.Vars(r) + emailID := vars["id"] + + // Get Gmail message ID + var gmailMessageID string + err := s.db.QueryRow("SELECT gmail_message_id FROM emails WHERE id = ?", emailID). + Scan(&gmailMessageID) + if err != nil { + http.Error(w, "Email not found", http.StatusNotFound) + return + } + + // Fetch from Gmail + message, err := s.gmailService.Users.Messages.Get("me", gmailMessageID). + Format("RAW").Do() + if err != nil { + http.Error(w, "Failed to fetch email from Gmail", http.StatusInternalServerError) + return + } + + // Decode and stream + rawEmail, err := base64.URLEncoding.DecodeString(message.Raw) + if err != nil { + http.Error(w, "Failed to decode email", http.StatusInternalServerError) + return + } + + w.Header().Set("Content-Type", "message/rfc822") + w.Write(rawEmail) +} + +// Original EmailProcessor methods (kept for local mode and shared logic) func (p *EmailProcessor) loadMaps() error { p.enquiryMap = make(map[string]int) p.invoiceMap = make(map[string]int) @@ -243,7 +908,7 @@ func (p *EmailProcessor) processEmail(filename string) error { foundPOs := p.checkIdentifier(subject, p.poMap, "purchaseorder") foundJobs := p.checkIdentifier(subject, p.jobMap, "job") - foundIdent := len(foundEnquiries) > 0 || len(foundInvoices) > 0 || + foundIdent := len(foundEnquiries) > 0 || len(foundInvoices) > 0 || len(foundPOs) > 0 || len(foundJobs) > 0 if fromKnownUser || saveThis || foundIdent { @@ -327,11 +992,14 @@ func (p *EmailProcessor) getUserIDs(emails []string) []int { func (p *EmailProcessor) createUser(email string) int { fmt.Printf("Making a new User for: '%s'\n", email) - + result, err := p.db.Exec( - "INSERT INTO users (type, email, by_vault, created, modified) VALUES (?, ?, ?, NOW(), NOW())", - "contact", strings.ToLower(email), 1) - + `INSERT INTO users (principle_id, customer_id, type, access_level, username, password, + first_name, last_name, email, job_title, phone, mobile, fax, phone_extension, + direct_phone, notes, by_vault, blacklisted, enabled, primary_contact) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + 0, 0, "contact", "none", "", "", "", "", strings.ToLower(email), "", "", "", "", "", "", "", 1, 0, 1, 0) + if err != nil { fmt.Printf("Serious Error: Unable to create user for email '%s': %v\n", email, err) return 0 @@ -407,7 +1075,7 @@ func (p *EmailProcessor) extractEnmimeAttachments(env *enmime.Envelope, relative fileName := "texthtml" newFileName := uuid + "-" + fileName filePath := filepath.Join(outputDir, newFileName) - + if err := ioutil.WriteFile(filePath, htmlData, 0644); err == nil { att := Attachment{ Type: "text/html", @@ -428,7 +1096,7 @@ func (p *EmailProcessor) extractEnmimeAttachments(env *enmime.Envelope, relative fileName := "textplain" newFileName := uuid + "-" + fileName filePath := filepath.Join(outputDir, newFileName) - + if err := ioutil.WriteFile(filePath, textData, 0644); err == nil { att := Attachment{ Type: "text/plain", @@ -449,10 +1117,10 @@ func (p *EmailProcessor) extractEnmimeAttachments(env *enmime.Envelope, relative if fileName == "" { fileName = "attachment" } - + newFileName := uuid + "-" + fileName filePath := filepath.Join(outputDir, newFileName) - + if err := ioutil.WriteFile(filePath, part.Content, 0644); err != nil { log.Printf("Failed to save attachment %s: %v", fileName, err) continue @@ -485,10 +1153,10 @@ func (p *EmailProcessor) extractEnmimeAttachments(env *enmime.Envelope, relative if fileName == "" { fileName = "inline" } - + newFileName := uuid + "-" + fileName filePath := filepath.Join(outputDir, newFileName) - + if err := ioutil.WriteFile(filePath, part.Content, 0644); err != nil { log.Printf("Failed to save inline part %s: %v", fileName, err) continue @@ -525,7 +1193,7 @@ func (p *EmailProcessor) extractEnmimeAttachments(env *enmime.Envelope, relative return attachments } -func (p *EmailProcessor) saveEmail(filename, subject string, unixTime int64, +func (p *EmailProcessor) saveEmail(filename, subject string, unixTime int64, recipientIDs map[string][]int, attachments []Attachment, foundEnquiries, foundInvoices, foundPOs, foundJobs []int) error { @@ -537,9 +1205,9 @@ func (p *EmailProcessor) saveEmail(filename, subject string, unixTime int64, // Insert email result, err := tx.Exec( - "INSERT INTO emails (user_id, udate, created, subject, filename) VALUES (?, ?, NOW(), ?, ?)", - recipientIDs["from"][0], unixTime, subject, filename) - + "INSERT INTO emails (user_id, udate, created, subject) VALUES (?, ?, NOW(), ?)", + recipientIDs["from"][0], unixTime, subject) + if err != nil { return err } @@ -621,4 +1289,12 @@ func (p *EmailProcessor) moveEmail(filename string) error { } return nil -} \ No newline at end of file +} + +type Attachment struct { + Type string + Name string + Filename string + Size int64 + IsMessageBody int +} diff --git a/go-app/go.mod b/go-app/go.mod index 982ff1aa..b5904ead 100644 --- a/go-app/go.mod +++ b/go-app/go.mod @@ -1,6 +1,8 @@ module code.springupsoftware.com/cmc/cmc-sales -go 1.23 +go 1.23.0 + +toolchain go1.24.4 require ( github.com/go-sql-driver/mysql v1.7.1 @@ -9,17 +11,38 @@ require ( github.com/jhillyerd/enmime v1.3.0 github.com/joho/godotenv v1.5.1 github.com/jung-kurt/gofpdf v1.16.2 + golang.org/x/oauth2 v0.30.0 + google.golang.org/api v0.244.0 ) require ( + cloud.google.com/go/auth v0.16.3 // indirect + cloud.google.com/go/auth/oauth2adapt v0.2.8 // indirect + cloud.google.com/go/compute/metadata v0.7.0 // indirect github.com/cention-sany/utf7 v0.0.0-20170124080048-26cad61bd60a // indirect + github.com/felixge/httpsnoop v1.0.4 // indirect + github.com/go-logr/logr v1.4.3 // indirect + github.com/go-logr/stdr v1.2.2 // indirect github.com/gogs/chardet v0.0.0-20211120154057-b7413eaefb8f // indirect + github.com/google/s2a-go v0.1.9 // indirect + github.com/googleapis/enterprise-certificate-proxy v0.3.6 // indirect + github.com/googleapis/gax-go/v2 v2.15.0 // indirect github.com/jaytaylor/html2text v0.0.0-20230321000545-74c2419ad056 // indirect github.com/mattn/go-runewidth v0.0.15 // indirect github.com/olekukonko/tablewriter v0.0.5 // indirect github.com/pkg/errors v0.9.1 // indirect github.com/rivo/uniseg v0.4.4 // indirect github.com/ssor/bom v0.0.0-20170718123548-6386211fdfcf // indirect - golang.org/x/net v0.23.0 // indirect - golang.org/x/text v0.14.0 // indirect + go.opentelemetry.io/auto/sdk v1.1.0 // indirect + go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0 // indirect + go.opentelemetry.io/otel v1.36.0 // indirect + go.opentelemetry.io/otel/metric v1.36.0 // indirect + go.opentelemetry.io/otel/trace v1.36.0 // indirect + golang.org/x/crypto v0.40.0 // indirect + golang.org/x/net v0.42.0 // indirect + golang.org/x/sys v0.34.0 // indirect + golang.org/x/text v0.27.0 // indirect + google.golang.org/genproto/googleapis/rpc v0.0.0-20250728155136-f173205681a0 // indirect + google.golang.org/grpc v1.74.2 // indirect + google.golang.org/protobuf v1.36.6 // indirect ) diff --git a/go-app/go.sum b/go-app/go.sum index a7a25f89..8366941a 100644 --- a/go-app/go.sum +++ b/go-app/go.sum @@ -1,13 +1,40 @@ +cloud.google.com/go/auth v0.16.3 h1:kabzoQ9/bobUmnseYnBO6qQG7q4a/CffFRlJSxv2wCc= +cloud.google.com/go/auth v0.16.3/go.mod h1:NucRGjaXfzP1ltpcQ7On/VTZ0H4kWB5Jy+Y9Dnm76fA= +cloud.google.com/go/auth/oauth2adapt v0.2.8 h1:keo8NaayQZ6wimpNSmW5OPc283g65QNIiLpZnkHRbnc= +cloud.google.com/go/auth/oauth2adapt v0.2.8/go.mod h1:XQ9y31RkqZCcwJWNSx2Xvric3RrU88hAYYbjDWYDL+c= +cloud.google.com/go/compute/metadata v0.7.0 h1:PBWF+iiAerVNe8UCHxdOt6eHLVc3ydFeOCw78U8ytSU= +cloud.google.com/go/compute/metadata v0.7.0/go.mod h1:j5MvL9PprKL39t166CoB1uVHfQMs4tFQZZcKwksXUjo= github.com/boombuler/barcode v1.0.0/go.mod h1:paBWMcWSl3LHKBqUq+rly7CNSldXjb2rDl3JlRe0mD8= github.com/cention-sany/utf7 v0.0.0-20170124080048-26cad61bd60a h1:MISbI8sU/PSK/ztvmWKFcI7UGb5/HQT7B+i3a2myKgI= github.com/cention-sany/utf7 v0.0.0-20170124080048-26cad61bd60a/go.mod h1:2GxOXOlEPAMFPfp014mK1SWq8G8BN8o7/dfYqJrVGn8= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg= +github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U= +github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= +github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI= +github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= +github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= +github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= github.com/go-sql-driver/mysql v1.7.1 h1:lUIinVbN1DY0xBg0eMOzmmtGoHwWBbvnWubQUrtU8EI= github.com/go-sql-driver/mysql v1.7.1/go.mod h1:OXbVy3sEdcQ2Doequ6Z5BW6fXNQTmx+9S1MCJN5yJMI= +github.com/go-test/deep v1.1.0 h1:WOcxcdHcvdgThNXjw0t76K42FXTU7HpNQWHpA2HHNlg= +github.com/go-test/deep v1.1.0/go.mod h1:5C2ZWiW0ErCdrYzpqxLbTX7MG14M9iiw8DgHncVwcsE= github.com/gogs/chardet v0.0.0-20211120154057-b7413eaefb8f h1:3BSP1Tbs2djlpprl7wCLuiqMaUh5SJkkzI2gDs+FgLs= github.com/gogs/chardet v0.0.0-20211120154057-b7413eaefb8f/go.mod h1:Pcatq5tYkCW2Q6yrR2VRHlbHpZ/R4/7qyL1TCF7vl14= +github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek= +github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps= +github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= +github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= +github.com/google/s2a-go v0.1.9 h1:LGD7gtMgezd8a/Xak7mEWL0PjoTQFvpRudN895yqKW0= +github.com/google/s2a-go v0.1.9/go.mod h1:YA0Ei2ZQL3acow2O62kdp9UlnvMmU7kA6Eutn0dXayM= github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/googleapis/enterprise-certificate-proxy v0.3.6 h1:GW/XbdyBFQ8Qe+YAmFU9uHLo7OnF5tL52HFAgMmyrf4= +github.com/googleapis/enterprise-certificate-proxy v0.3.6/go.mod h1:MkHOF77EYAE7qfSuSS9PU6g4Nt4e11cnsDUowfwewLA= +github.com/googleapis/gax-go/v2 v2.15.0 h1:SyjDc1mGgZU5LncH8gimWo9lW1DtIfPibOG81vgd/bo= +github.com/googleapis/gax-go/v2 v2.15.0/go.mod h1:zVVkkxAQHa1RQpg9z2AUCMnKhi0Qld9rcmyfL1OZhoc= github.com/gorilla/mux v1.8.1 h1:TuBL49tXwgrFYWhqrNgrUNEY92u81SPhu7sTdzQEiWY= github.com/gorilla/mux v1.8.1/go.mod h1:AKf9I4AEqPTmMytcMc0KkNouC66V3BtZ4qD5fmWSiMQ= github.com/jaytaylor/html2text v0.0.0-20230321000545-74c2419ad056 h1:iCHtR9CQyktQ5+f3dMVZfwD2KWJUgm7M0gdL9NGr8KA= @@ -28,6 +55,7 @@ github.com/phpdave11/gofpdi v1.0.7/go.mod h1:vBmVV0Do6hSBHC8uKUQ71JGW+ZGQq74llk/ github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc= github.com/rivo/uniseg v0.4.4 h1:8TfxU8dW6PdqD27gjM8MVNuicgxIjxpm4K7x4jp8sis= @@ -36,9 +64,47 @@ github.com/ruudk/golang-pdf417 v0.0.0-20181029194003-1af4ab5afa58/go.mod h1:6lfF github.com/ssor/bom v0.0.0-20170718123548-6386211fdfcf h1:pvbZ0lM0XWPBqUKqFU8cmavspvIl9nulOYwdy6IFRRo= github.com/ssor/bom v0.0.0-20170718123548-6386211fdfcf/go.mod h1:RJID2RhlZKId02nZ62WenDCkgHFerpIOmW0iT7GKmXM= github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= +github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA= +github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY= +go.opentelemetry.io/auto/sdk v1.1.0 h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA= +go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A= +go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0 h1:F7Jx+6hwnZ41NSFTO5q4LYDtJRXBf2PD0rNBkeB/lus= +go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0/go.mod h1:UHB22Z8QsdRDrnAtX4PntOl36ajSxcdUMt1sF7Y6E7Q= +go.opentelemetry.io/otel v1.36.0 h1:UumtzIklRBY6cI/lllNZlALOF5nNIzJVb16APdvgTXg= +go.opentelemetry.io/otel v1.36.0/go.mod h1:/TcFMXYjyRNh8khOAO9ybYkqaDBb/70aVwkNML4pP8E= +go.opentelemetry.io/otel/metric v1.36.0 h1:MoWPKVhQvJ+eeXWHFBOPoBOi20jh6Iq2CcCREuTYufE= +go.opentelemetry.io/otel/metric v1.36.0/go.mod h1:zC7Ks+yeyJt4xig9DEw9kuUFe5C3zLbVjV2PzT6qzbs= +go.opentelemetry.io/otel/sdk v1.36.0 h1:b6SYIuLRs88ztox4EyrvRti80uXIFy+Sqzoh9kFULbs= +go.opentelemetry.io/otel/sdk v1.36.0/go.mod h1:+lC+mTgD+MUWfjJubi2vvXWcVxyr9rmlshZni72pXeY= +go.opentelemetry.io/otel/sdk/metric v1.36.0 h1:r0ntwwGosWGaa0CrSt8cuNuTcccMXERFwHX4dThiPis= +go.opentelemetry.io/otel/sdk/metric v1.36.0/go.mod h1:qTNOhFDfKRwX0yXOqJYegL5WRaW376QbB7P4Pb0qva4= +go.opentelemetry.io/otel/trace v1.36.0 h1:ahxWNuqZjpdiFAyrIoQ4GIiAIhxAunQR6MUoKrsNd4w= +go.opentelemetry.io/otel/trace v1.36.0/go.mod h1:gQ+OnDZzrybY4k4seLzPAWNwVBBVlF2szhehOBB/tGA= +golang.org/x/crypto v0.40.0 h1:r4x+VvoG5Fm+eJcxMaY8CQM7Lb0l1lsmjGBQ6s8BfKM= +golang.org/x/crypto v0.40.0/go.mod h1:Qr1vMER5WyS2dfPHAlsOj01wgLbsyWtFn/aY+5+ZdxY= golang.org/x/image v0.0.0-20190910094157-69e4b8554b2a/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0= -golang.org/x/net v0.23.0 h1:7EYJ93RZ9vYSZAIb2x3lnuvqO5zneoD6IvWjuhfxjTs= -golang.org/x/net v0.23.0/go.mod h1:JKghWKKOSdJwpW2GEx0Ja7fmaKnMsbu+MWVZTokSYmg= +golang.org/x/net v0.42.0 h1:jzkYrhi3YQWD6MLBJcsklgQsoAcw89EcZbJw8Z614hs= +golang.org/x/net v0.42.0/go.mod h1:FF1RA5d3u7nAYA4z2TkclSCKh68eSXtiFwcWQpPXdt8= +golang.org/x/oauth2 v0.30.0 h1:dnDm7JmhM45NNpd8FDDeLhK6FwqbOf4MLCM9zb1BOHI= +golang.org/x/oauth2 v0.30.0/go.mod h1:B++QgG3ZKulg6sRPGD/mqlHQs5rB3Ml9erfeDY7xKlU= +golang.org/x/sync v0.16.0 h1:ycBJEhp9p4vXvUZNszeOq0kGTPghopOL8q0fq3vstxw= +golang.org/x/sync v0.16.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA= +golang.org/x/sys v0.34.0 h1:H5Y5sJ2L2JRdyv7ROF1he/lPdvFsd0mJHFw2ThKHxLA= +golang.org/x/sys v0.34.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= -golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ= -golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= +golang.org/x/text v0.27.0 h1:4fGWRpyh641NLlecmyl4LOe6yDdfaYNrGb2zdfo4JV4= +golang.org/x/text v0.27.0/go.mod h1:1D28KMCvyooCX9hBiosv5Tz/+YLxj0j7XhWjpSUF7CU= +google.golang.org/api v0.244.0 h1:lpkP8wVibSKr++NCD36XzTk/IzeKJ3klj7vbj+XU5pE= +google.golang.org/api v0.244.0/go.mod h1:dMVhVcylamkirHdzEBAIQWUCgqY885ivNeZYd7VAVr8= +google.golang.org/genproto v0.0.0-20250603155806-513f23925822 h1:rHWScKit0gvAPuOnu87KpaYtjK5zBMLcULh7gxkCXu4= +google.golang.org/genproto v0.0.0-20250603155806-513f23925822/go.mod h1:HubltRL7rMh0LfnQPkMH4NPDFEWp0jw3vixw7jEM53s= +google.golang.org/genproto/googleapis/api v0.0.0-20250603155806-513f23925822 h1:oWVWY3NzT7KJppx2UKhKmzPq4SRe0LdCijVRwvGeikY= +google.golang.org/genproto/googleapis/api v0.0.0-20250603155806-513f23925822/go.mod h1:h3c4v36UTKzUiuaOKQ6gr3S+0hovBtUrXzTG/i3+XEc= +google.golang.org/genproto/googleapis/rpc v0.0.0-20250728155136-f173205681a0 h1:MAKi5q709QWfnkkpNQ0M12hYJ1+e8qYVDyowc4U1XZM= +google.golang.org/genproto/googleapis/rpc v0.0.0-20250728155136-f173205681a0/go.mod h1:qQ0YXyHHx3XkvlzUtpXDkS29lDSafHMZBAZDc03LQ3A= +google.golang.org/grpc v1.74.2 h1:WoosgB65DlWVC9FqI82dGsZhWFNBSLjQ84bjROOpMu4= +google.golang.org/grpc v1.74.2/go.mod h1:CtQ+BGjaAIXHs/5YS3i473GqwBBa1zGQNevxdeBEXrM= +google.golang.org/protobuf v1.36.6 h1:z1NpPI8ku2WgiWnf+t9wTPsn6eP1L7ksHUlkfLvd9xY= +google.golang.org/protobuf v1.36.6/go.mod h1:jduwjTPXsFjZGTmRluh+L6NjiWu7pchiJ2/5YcXBHnY= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/go-app/goose.env.example b/go-app/goose.env.example new file mode 100644 index 00000000..c40a0103 --- /dev/null +++ b/go-app/goose.env.example @@ -0,0 +1,3 @@ +GOOSE_DRIVER=mysql +GOOSE_DBSTRING=username:password@tcp(localhost:3306)/database?parseTime=true +GOOSE_MIGRATION_DIR=sql/migrations \ No newline at end of file diff --git a/go-app/internal/cmc/handlers/emails.go b/go-app/internal/cmc/handlers/emails.go new file mode 100644 index 00000000..b34caae1 --- /dev/null +++ b/go-app/internal/cmc/handlers/emails.go @@ -0,0 +1,870 @@ +package handlers + +import ( + "bytes" + "context" + "database/sql" + "encoding/base64" + "encoding/json" + "fmt" + "io/ioutil" + "net/http" + "os" + "strconv" + "strings" + "time" + + "code.springupsoftware.com/cmc/cmc-sales/internal/cmc/db" + "github.com/gorilla/mux" + "github.com/jhillyerd/enmime" + "golang.org/x/oauth2" + "golang.org/x/oauth2/google" + "google.golang.org/api/gmail/v1" + "google.golang.org/api/option" +) + +type EmailHandler struct { + queries *db.Queries + db *sql.DB + gmailService *gmail.Service +} + +type EmailResponse struct { + ID int32 `json:"id"` + Subject string `json:"subject"` + UserID int32 `json:"user_id"` + Created time.Time `json:"created"` + GmailMessageID *string `json:"gmail_message_id,omitempty"` + AttachmentCount int32 `json:"attachment_count"` + IsDownloaded *bool `json:"is_downloaded,omitempty"` +} + +type EmailDetailResponse struct { + ID int32 `json:"id"` + Subject string `json:"subject"` + UserID int32 `json:"user_id"` + Created time.Time `json:"created"` + GmailMessageID *string `json:"gmail_message_id,omitempty"` + GmailThreadID *string `json:"gmail_thread_id,omitempty"` + RawHeaders *string `json:"raw_headers,omitempty"` + IsDownloaded *bool `json:"is_downloaded,omitempty"` + Enquiries []int32 `json:"enquiries,omitempty"` + Invoices []int32 `json:"invoices,omitempty"` + PurchaseOrders []int32 `json:"purchase_orders,omitempty"` + Jobs []int32 `json:"jobs,omitempty"` +} + +type EmailAttachmentResponse struct { + ID int32 `json:"id"` + Name string `json:"name"` + Type string `json:"type"` + Size int32 `json:"size"` + Filename string `json:"filename"` + IsMessageBody bool `json:"is_message_body"` + GmailAttachmentID *string `json:"gmail_attachment_id,omitempty"` + Created time.Time `json:"created"` +} + +func NewEmailHandler(queries *db.Queries, database *sql.DB) *EmailHandler { + // Try to initialize Gmail service + gmailService, err := getGmailService("credentials.json", "token.json") + if err != nil { + // Log the error but continue without Gmail service + fmt.Printf("Warning: Gmail service not available: %v\n", err) + } + + return &EmailHandler{ + queries: queries, + db: database, + gmailService: gmailService, + } +} + +// List emails with pagination and filtering +func (h *EmailHandler) List(w http.ResponseWriter, r *http.Request) { + // Parse query parameters + limitStr := r.URL.Query().Get("limit") + offsetStr := r.URL.Query().Get("offset") + search := r.URL.Query().Get("search") + userID := r.URL.Query().Get("user_id") + + // Set defaults + limit := 50 + offset := 0 + + if limitStr != "" { + if l, err := strconv.Atoi(limitStr); err == nil && l > 0 && l <= 100 { + limit = l + } + } + + if offsetStr != "" { + if o, err := strconv.Atoi(offsetStr); err == nil && o >= 0 { + offset = o + } + } + + // Build query + query := ` + SELECT e.id, e.subject, e.user_id, e.created, e.gmail_message_id, e.email_attachment_count, e.is_downloaded + FROM emails e` + + var args []interface{} + var conditions []string + + if search != "" { + conditions = append(conditions, "e.subject LIKE ?") + args = append(args, "%"+search+"%") + } + + if userID != "" { + conditions = append(conditions, "e.user_id = ?") + args = append(args, userID) + } + + if len(conditions) > 0 { + query += " WHERE " + joinConditions(conditions, " AND ") + } + + query += " ORDER BY e.id DESC LIMIT ? OFFSET ?" + args = append(args, limit, offset) + + rows, err := h.db.Query(query, args...) + if err != nil { + http.Error(w, fmt.Sprintf("Database error: %v", err), http.StatusInternalServerError) + return + } + defer rows.Close() + + var emails []EmailResponse + for rows.Next() { + var email EmailResponse + var gmailMessageID sql.NullString + var isDownloaded sql.NullBool + + err := rows.Scan( + &email.ID, + &email.Subject, + &email.UserID, + &email.Created, + &gmailMessageID, + &email.AttachmentCount, + &isDownloaded, + ) + if err != nil { + continue + } + + if gmailMessageID.Valid { + email.GmailMessageID = &gmailMessageID.String + } + if isDownloaded.Valid { + email.IsDownloaded = &isDownloaded.Bool + } + + emails = append(emails, email) + } + + w.Header().Set("Content-Type", "application/json") + json.NewEncoder(w).Encode(emails) +} + +// Get a specific email with details +func (h *EmailHandler) Get(w http.ResponseWriter, r *http.Request) { + vars := mux.Vars(r) + emailID, err := strconv.Atoi(vars["id"]) + if err != nil { + http.Error(w, "Invalid email ID", http.StatusBadRequest) + return + } + + // Get email details + query := ` + SELECT e.id, e.subject, e.user_id, e.created, e.gmail_message_id, + e.gmail_thread_id, e.raw_headers, e.is_downloaded + FROM emails e + WHERE e.id = ?` + + var email EmailDetailResponse + var gmailMessageID, gmailThreadID, rawHeaders sql.NullString + var isDownloaded sql.NullBool + + err = h.db.QueryRow(query, emailID).Scan( + &email.ID, + &email.Subject, + &email.UserID, + &email.Created, + &gmailMessageID, + &gmailThreadID, + &rawHeaders, + &isDownloaded, + ) + + if err != nil { + if err == sql.ErrNoRows { + http.Error(w, "Email not found", http.StatusNotFound) + } else { + http.Error(w, fmt.Sprintf("Database error: %v", err), http.StatusInternalServerError) + } + return + } + + if gmailMessageID.Valid { + email.GmailMessageID = &gmailMessageID.String + } + if gmailThreadID.Valid { + email.GmailThreadID = &gmailThreadID.String + } + if rawHeaders.Valid { + email.RawHeaders = &rawHeaders.String + } + if isDownloaded.Valid { + email.IsDownloaded = &isDownloaded.Bool + } + + // Get associated enquiries + enquiryRows, err := h.db.Query("SELECT enquiry_id FROM emails_enquiries WHERE email_id = ?", emailID) + if err == nil { + defer enquiryRows.Close() + for enquiryRows.Next() { + var enquiryID int32 + if enquiryRows.Scan(&enquiryID) == nil { + email.Enquiries = append(email.Enquiries, enquiryID) + } + } + } + + // Get associated invoices + invoiceRows, err := h.db.Query("SELECT invoice_id FROM emails_invoices WHERE email_id = ?", emailID) + if err == nil { + defer invoiceRows.Close() + for invoiceRows.Next() { + var invoiceID int32 + if invoiceRows.Scan(&invoiceID) == nil { + email.Invoices = append(email.Invoices, invoiceID) + } + } + } + + // Get associated purchase orders + poRows, err := h.db.Query("SELECT purchase_order_id FROM emails_purchase_orders WHERE email_id = ?", emailID) + if err == nil { + defer poRows.Close() + for poRows.Next() { + var poID int32 + if poRows.Scan(&poID) == nil { + email.PurchaseOrders = append(email.PurchaseOrders, poID) + } + } + } + + // Get associated jobs + jobRows, err := h.db.Query("SELECT job_id FROM emails_jobs WHERE email_id = ?", emailID) + if err == nil { + defer jobRows.Close() + for jobRows.Next() { + var jobID int32 + if jobRows.Scan(&jobID) == nil { + email.Jobs = append(email.Jobs, jobID) + } + } + } + + w.Header().Set("Content-Type", "application/json") + json.NewEncoder(w).Encode(email) +} + +// List attachments for an email +func (h *EmailHandler) ListAttachments(w http.ResponseWriter, r *http.Request) { + vars := mux.Vars(r) + emailID, err := strconv.Atoi(vars["id"]) + if err != nil { + http.Error(w, "Invalid email ID", http.StatusBadRequest) + return + } + + // First check if attachments are already in database + query := ` + SELECT id, name, type, size, filename, is_message_body, gmail_attachment_id, created + FROM email_attachments + WHERE email_id = ? + ORDER BY is_message_body DESC, created ASC` + + rows, err := h.db.Query(query, emailID) + if err != nil { + http.Error(w, fmt.Sprintf("Database error: %v", err), http.StatusInternalServerError) + return + } + defer rows.Close() + + var attachments []EmailAttachmentResponse + hasStoredAttachments := false + + for rows.Next() { + hasStoredAttachments = true + var attachment EmailAttachmentResponse + var gmailAttachmentID sql.NullString + + err := rows.Scan( + &attachment.ID, + &attachment.Name, + &attachment.Type, + &attachment.Size, + &attachment.Filename, + &attachment.IsMessageBody, + &gmailAttachmentID, + &attachment.Created, + ) + if err != nil { + continue + } + + if gmailAttachmentID.Valid { + attachment.GmailAttachmentID = &gmailAttachmentID.String + } + + attachments = append(attachments, attachment) + } + + // If no stored attachments and this is a Gmail email, try to fetch from Gmail + if !hasStoredAttachments && h.gmailService != nil { + // Get Gmail message ID + var gmailMessageID sql.NullString + err := h.db.QueryRow("SELECT gmail_message_id FROM emails WHERE id = ?", emailID).Scan(&gmailMessageID) + + if err == nil && gmailMessageID.Valid { + // Fetch message metadata from Gmail + message, err := h.gmailService.Users.Messages.Get("me", gmailMessageID.String). + Format("FULL").Do() + + if err == nil && message.Payload != nil { + // Extract attachment info from Gmail message + attachmentIndex := int32(1) + h.extractGmailAttachments(message.Payload, &attachments, &attachmentIndex) + } + } + } + + // Check if this is an HTMX request + if r.Header.Get("HX-Request") == "true" { + // Return HTML for HTMX + w.Header().Set("Content-Type", "text/html; charset=utf-8") + + if len(attachments) == 0 { + // No attachments found + html := `
+

No attachments found for this email.

+
` + w.Write([]byte(html)) + return + } + + // Build HTML table for attachments + var htmlBuilder strings.Builder + htmlBuilder.WriteString(`
+

Attachments

+
+ + + + + + + + + + `) + + for _, att := range attachments { + icon := `` + if att.IsMessageBody { + icon = `` + } + + downloadURL := fmt.Sprintf("/api/v1/emails/%d/attachments/%d", emailID, att.ID) + if att.GmailAttachmentID != nil { + downloadURL = fmt.Sprintf("/api/v1/emails/%d/attachments/%d/stream", emailID, att.ID) + } + + htmlBuilder.WriteString(fmt.Sprintf(` + + + + + + `, icon, att.Name, att.Type, att.Size, downloadURL)) + } + + htmlBuilder.WriteString(` + +
NameTypeSizeActions
+ %s + %s + %s%d bytes + + + Download + +
+
+
`) + + w.Write([]byte(htmlBuilder.String())) + return + } + + // Return JSON for API requests + w.Header().Set("Content-Type", "application/json") + json.NewEncoder(w).Encode(attachments) +} + +// Helper function to extract attachment info from Gmail message parts +func (h *EmailHandler) extractGmailAttachments(part *gmail.MessagePart, attachments *[]EmailAttachmentResponse, index *int32) { + // Check if this part is an attachment + // Some attachments may not have filenames or may be inline + if part.Body != nil && part.Body.AttachmentId != "" { + filename := part.Filename + if filename == "" { + // Try to generate a filename from content type + switch part.MimeType { + case "application/pdf": + filename = "attachment.pdf" + case "image/png": + filename = "image.png" + case "image/jpeg": + filename = "image.jpg" + case "text/plain": + filename = "text.txt" + default: + filename = "attachment" + } + } + + attachment := EmailAttachmentResponse{ + ID: *index, + Name: filename, + Type: part.MimeType, + Size: int32(part.Body.Size), + Filename: filename, + IsMessageBody: false, + GmailAttachmentID: &part.Body.AttachmentId, + Created: time.Now(), // Use current time as placeholder + } + *attachments = append(*attachments, attachment) + *index++ + } + + // Process sub-parts + for _, subPart := range part.Parts { + h.extractGmailAttachments(subPart, attachments, index) + } +} + +// Search emails +func (h *EmailHandler) Search(w http.ResponseWriter, r *http.Request) { + query := r.URL.Query().Get("q") + if query == "" { + http.Error(w, "Search query is required", http.StatusBadRequest) + return + } + + // Parse optional parameters + limitStr := r.URL.Query().Get("limit") + limit := 20 + + if limitStr != "" { + if l, err := strconv.Atoi(limitStr); err == nil && l > 0 && l <= 100 { + limit = l + } + } + + // Search in subjects and headers + sqlQuery := ` + SELECT e.id, e.subject, e.user_id, e.created, e.gmail_message_id, e.email_attachment_count, e.is_downloaded + FROM emails e + WHERE e.subject LIKE ? OR e.raw_headers LIKE ? + ORDER BY e.id DESC + LIMIT ?` + + searchTerm := "%" + query + "%" + rows, err := h.db.Query(sqlQuery, searchTerm, searchTerm, limit) + if err != nil { + http.Error(w, fmt.Sprintf("Database error: %v", err), http.StatusInternalServerError) + return + } + defer rows.Close() + + var emails []EmailResponse + for rows.Next() { + var email EmailResponse + var gmailMessageID sql.NullString + var isDownloaded sql.NullBool + + err := rows.Scan( + &email.ID, + &email.Subject, + &email.UserID, + &email.Created, + &gmailMessageID, + &email.AttachmentCount, + &isDownloaded, + ) + if err != nil { + continue + } + + if gmailMessageID.Valid { + email.GmailMessageID = &gmailMessageID.String + } + if isDownloaded.Valid { + email.IsDownloaded = &isDownloaded.Bool + } + + emails = append(emails, email) + } + + w.Header().Set("Content-Type", "application/json") + json.NewEncoder(w).Encode(emails) +} + +// Stream email content from Gmail +func (h *EmailHandler) StreamContent(w http.ResponseWriter, r *http.Request) { + vars := mux.Vars(r) + emailID, err := strconv.Atoi(vars["id"]) + if err != nil { + http.Error(w, "Invalid email ID", http.StatusBadRequest) + return + } + + // Get email details to check if it's a Gmail email + query := ` + SELECT e.gmail_message_id, e.subject, e.created, e.user_id + FROM emails e + WHERE e.id = ?` + + var gmailMessageID sql.NullString + var subject string + var created time.Time + var userID int32 + + err = h.db.QueryRow(query, emailID).Scan(&gmailMessageID, &subject, &created, &userID) + if err != nil { + if err == sql.ErrNoRows { + http.Error(w, "Email not found", http.StatusNotFound) + } else { + http.Error(w, fmt.Sprintf("Database error: %v", err), http.StatusInternalServerError) + } + return + } + + if !gmailMessageID.Valid { + w.Header().Set("Content-Type", "text/html; charset=utf-8") + html := ` +
+ Local Email
+ This email is not from Gmail and does not have stored content available for display. +
` + w.Write([]byte(html)) + return + } + + // Check for stored message body content in attachments + attachmentQuery := ` + SELECT id, name, type, size + FROM email_attachments + WHERE email_id = ? AND is_message_body = 1 + ORDER BY created ASC` + + attachmentRows, err := h.db.Query(attachmentQuery, emailID) + if err == nil { + defer attachmentRows.Close() + if attachmentRows.Next() { + var attachmentID int32 + var name, attachmentType string + var size int32 + + if attachmentRows.Scan(&attachmentID, &name, &attachmentType, &size) == nil { + // Found stored message body - would normally read the content from file storage + w.Header().Set("Content-Type", "text/html; charset=utf-8") + html := fmt.Sprintf(` +
+
+ Stored Email Content
+ Message body is stored locally as attachment: %s (%s, %d bytes) +
+
+

Content would be loaded from local storage here.

+

Attachment ID: %d

+
+
`, name, attachmentType, size, attachmentID) + w.Write([]byte(html)) + return + } + } + } + + // Try to fetch from Gmail if service is available + if h.gmailService != nil { + // Fetch from Gmail + message, err := h.gmailService.Users.Messages.Get("me", gmailMessageID.String). + Format("RAW").Do() + if err != nil { + w.Header().Set("Content-Type", "text/html; charset=utf-8") + html := fmt.Sprintf(` +
+ Gmail API Error
+ Failed to fetch email from Gmail: %v +
`, err) + w.Write([]byte(html)) + return + } + + // Decode message + rawEmail, err := base64.URLEncoding.DecodeString(message.Raw) + if err != nil { + w.Header().Set("Content-Type", "text/html; charset=utf-8") + html := fmt.Sprintf(` +
+ Decode Error
+ Failed to decode email: %v +
`, err) + w.Write([]byte(html)) + return + } + + // Parse with enmime + env, err := enmime.ReadEnvelope(bytes.NewReader(rawEmail)) + if err != nil { + w.Header().Set("Content-Type", "text/html; charset=utf-8") + html := fmt.Sprintf(` +
+ Parse Error
+ Failed to parse email: %v +
`, err) + w.Write([]byte(html)) + return + } + + // Stream HTML or Text directly to client + if env.HTML != "" { + w.Header().Set("Content-Type", "text/html; charset=utf-8") + w.Write([]byte(env.HTML)) + } else if env.Text != "" { + // Convert plain text to HTML for better display + w.Header().Set("Content-Type", "text/html; charset=utf-8") + html := fmt.Sprintf(` +
+
+ Plain Text Email
+ This email contains only plain text content. +
+
+ 
%s
+
+
`, env.Text) + w.Write([]byte(html)) + } else { + w.Header().Set("Content-Type", "text/html; charset=utf-8") + html := ` +
+ No Content
+ No HTML or text content found in this email. +
` + w.Write([]byte(html)) + } + return + } + + // No Gmail service available - show error + w.Header().Set("Content-Type", "text/html; charset=utf-8") + html := fmt.Sprintf(` +
+
+ Gmail Service Unavailable
+ Subject: %s
+ Date: %s
+ Gmail Message ID: %s +
+ +
+
+

Integration Status

+

Gmail service is not available. To enable email content display:

+
    +
  1. Ensure credentials.json and token.json files are present
    2. +
  2. Configure Gmail API OAuth2 authentication
    3. +
  3. Restart the application
    4. +
+

+ Gmail Message ID: %s +

+
+
+
`, + subject, + created.Format("2006-01-02 15:04:05"), + gmailMessageID.String, + gmailMessageID.String) + + w.Write([]byte(html)) +} + +// Stream attachment from Gmail +func (h *EmailHandler) StreamAttachment(w http.ResponseWriter, r *http.Request) { + vars := mux.Vars(r) + emailID, err := strconv.Atoi(vars["id"]) + if err != nil { + http.Error(w, "Invalid email ID", http.StatusBadRequest) + return + } + + attachmentID := vars["attachmentId"] + + // Get email's Gmail message ID + var gmailMessageID sql.NullString + err = h.db.QueryRow("SELECT gmail_message_id FROM emails WHERE id = ?", emailID).Scan(&gmailMessageID) + if err != nil || !gmailMessageID.Valid { + http.Error(w, "Email not found or not a Gmail email", http.StatusNotFound) + return + } + + if h.gmailService == nil { + http.Error(w, "Gmail service not available", http.StatusServiceUnavailable) + return + } + + // For dynamic attachments, we need to fetch the message and find the attachment + message, err := h.gmailService.Users.Messages.Get("me", gmailMessageID.String). + Format("FULL").Do() + if err != nil { + http.Error(w, "Failed to fetch email from Gmail", http.StatusInternalServerError) + return + } + + // Find the attachment by index + var targetAttachment *gmail.MessagePart + attachmentIndex := 1 + findAttachment(message.Payload, attachmentID, &attachmentIndex, &targetAttachment) + + if targetAttachment == nil || targetAttachment.Body == nil || targetAttachment.Body.AttachmentId == "" { + http.Error(w, "Attachment not found", http.StatusNotFound) + return + } + + // Fetch attachment data from Gmail + attachment, err := h.gmailService.Users.Messages.Attachments. + Get("me", gmailMessageID.String, targetAttachment.Body.AttachmentId).Do() + if err != nil { + http.Error(w, "Failed to fetch attachment from Gmail", http.StatusInternalServerError) + return + } + + // Decode base64 + data, err := base64.URLEncoding.DecodeString(attachment.Data) + if err != nil { + http.Error(w, "Failed to decode attachment", http.StatusInternalServerError) + return + } + + // Set headers and stream + filename := targetAttachment.Filename + if filename == "" { + // Generate filename from content type (same logic as extractGmailAttachments) + switch targetAttachment.MimeType { + case "application/pdf": + filename = "attachment.pdf" + case "image/png": + filename = "image.png" + case "image/jpeg": + filename = "image.jpg" + case "text/plain": + filename = "text.txt" + default: + filename = "attachment" + } + } + + w.Header().Set("Content-Type", targetAttachment.MimeType) + w.Header().Set("Content-Disposition", fmt.Sprintf("attachment; filename=\"%s\"", filename)) + w.Header().Set("Content-Length", fmt.Sprintf("%d", len(data))) + w.Write(data) +} + +// Helper function to find attachment by index +func findAttachment(part *gmail.MessagePart, targetID string, currentIndex *int, result **gmail.MessagePart) { + // Check if this part is an attachment (same logic as extractGmailAttachments) + if part.Body != nil && part.Body.AttachmentId != "" { + fmt.Printf("Checking attachment %d (looking for %s): %s\n", *currentIndex, targetID, part.Filename) + if strconv.Itoa(*currentIndex) == targetID { + fmt.Printf("Found matching attachment!\n") + *result = part + return + } + *currentIndex++ + } + + for _, subPart := range part.Parts { + findAttachment(subPart, targetID, currentIndex, result) + if *result != nil { + return + } + } +} + +// Helper function to join conditions +func joinConditions(conditions []string, separator string) string { + if len(conditions) == 0 { + return "" + } + if len(conditions) == 1 { + return conditions[0] + } + + result := conditions[0] + for i := 1; i < len(conditions); i++ { + result += separator + conditions[i] + } + return result +} + +// Gmail OAuth2 functions +func getGmailService(credentialsFile, tokenFile string) (*gmail.Service, error) { + ctx := context.Background() + + b, err := ioutil.ReadFile(credentialsFile) + if err != nil { + return nil, fmt.Errorf("unable to read client secret file: %v", err) + } + + config, err := google.ConfigFromJSON(b, gmail.GmailReadonlyScope) + if err != nil { + return nil, fmt.Errorf("unable to parse client secret file to config: %v", err) + } + + client := getClient(config, tokenFile) + srv, err := gmail.NewService(ctx, option.WithHTTPClient(client)) + if err != nil { + return nil, fmt.Errorf("unable to retrieve Gmail client: %v", err) + } + + return srv, nil +} + +func getClient(config *oauth2.Config, tokFile string) *http.Client { + tok, err := tokenFromFile(tokFile) + if err != nil { + return nil + } + return config.Client(context.Background(), tok) +} + +func tokenFromFile(file string) (*oauth2.Token, error) { + f, err := os.Open(file) + if err != nil { + return nil, err + } + defer f.Close() + tok := &oauth2.Token{} + err = json.NewDecoder(f).Decode(tok) + return tok, err +} \ No newline at end of file diff --git a/go-app/internal/cmc/handlers/pages.go b/go-app/internal/cmc/handlers/pages.go index 3b79fc4a..e97ca415 100644 --- a/go-app/internal/cmc/handlers/pages.go +++ b/go-app/internal/cmc/handlers/pages.go @@ -1,9 +1,11 @@ package handlers import ( + "database/sql" "log" "net/http" "strconv" + "time" "code.springupsoftware.com/cmc/cmc-sales/internal/cmc/db" "code.springupsoftware.com/cmc/cmc-sales/internal/cmc/templates" @@ -13,12 +15,14 @@ import ( type PageHandler struct { queries *db.Queries tmpl *templates.TemplateManager + db *sql.DB } -func NewPageHandler(queries *db.Queries, tmpl *templates.TemplateManager) *PageHandler { +func NewPageHandler(queries *db.Queries, tmpl *templates.TemplateManager, database *sql.DB) *PageHandler { return &PageHandler{ queries: queries, tmpl: tmpl, + db: database, } } @@ -776,3 +780,404 @@ func (h *PageHandler) DocumentsView(w http.ResponseWriter, r *http.Request) { http.Error(w, err.Error(), http.StatusInternalServerError) } } + +// Email page handlers +func (h *PageHandler) EmailsIndex(w http.ResponseWriter, r *http.Request) { + page := 1 + if p := r.URL.Query().Get("page"); p != "" { + if val, err := strconv.Atoi(p); err == nil && val > 0 { + page = val + } + } + + limit := 30 + offset := (page - 1) * limit + search := r.URL.Query().Get("search") + filter := r.URL.Query().Get("filter") + + // Build SQL query based on filters + query := ` + SELECT e.id, e.subject, e.user_id, e.created, e.gmail_message_id, + e.email_attachment_count, e.is_downloaded, + u.email as user_email, u.first_name, u.last_name + FROM emails e + LEFT JOIN users u ON e.user_id = u.id` + + var args []interface{} + var conditions []string + + // Apply search filter + if search != "" { + conditions = append(conditions, "(e.subject LIKE ? OR e.raw_headers LIKE ?)") + searchTerm := "%" + search + "%" + args = append(args, searchTerm, searchTerm) + } + + // Apply type filter + switch filter { + case "downloaded": + conditions = append(conditions, "e.is_downloaded = 1") + case "gmail": + conditions = append(conditions, "e.gmail_message_id IS NOT NULL") + case "unassociated": + conditions = append(conditions, `NOT EXISTS ( + SELECT 1 FROM emails_enquiries WHERE email_id = e.id + UNION SELECT 1 FROM emails_invoices WHERE email_id = e.id + UNION SELECT 1 FROM emails_purchase_orders WHERE email_id = e.id + UNION SELECT 1 FROM emails_jobs WHERE email_id = e.id + )`) + } + + if len(conditions) > 0 { + query += " WHERE " + joinConditions(conditions, " AND ") + } + + query += " ORDER BY e.id DESC LIMIT ? OFFSET ?" + args = append(args, limit+1, offset) // Get one extra to check if there are more + + // Execute the query to get emails + rows, err := h.db.Query(query, args...) + if err != nil { + log.Printf("Error querying emails: %v", err) + http.Error(w, "Database error", http.StatusInternalServerError) + return + } + defer rows.Close() + + type EmailWithUser struct { + ID int32 `json:"id"` + Subject string `json:"subject"` + UserID int32 `json:"user_id"` + Created time.Time `json:"created"` + GmailMessageID *string `json:"gmail_message_id"` + AttachmentCount int32 `json:"attachment_count"` + IsDownloaded *bool `json:"is_downloaded"` + UserEmail *string `json:"user_email"` + FirstName *string `json:"first_name"` + LastName *string `json:"last_name"` + } + + var emails []EmailWithUser + for rows.Next() { + var email EmailWithUser + var gmailMessageID, userEmail, firstName, lastName sql.NullString + var isDownloaded sql.NullBool + + err := rows.Scan( + &email.ID, + &email.Subject, + &email.UserID, + &email.Created, + &gmailMessageID, + &email.AttachmentCount, + &isDownloaded, + &userEmail, + &firstName, + &lastName, + ) + if err != nil { + log.Printf("Error scanning email row: %v", err) + continue + } + + if gmailMessageID.Valid { + email.GmailMessageID = &gmailMessageID.String + } + if isDownloaded.Valid { + email.IsDownloaded = &isDownloaded.Bool + } + if userEmail.Valid { + email.UserEmail = &userEmail.String + } + if firstName.Valid { + email.FirstName = &firstName.String + } + if lastName.Valid { + email.LastName = &lastName.String + } + + emails = append(emails, email) + } + + hasMore := len(emails) > limit + if hasMore { + emails = emails[:limit] + } + + data := map[string]interface{}{ + "Emails": emails, + "Page": page, + "PrevPage": page - 1, + "NextPage": page + 1, + "HasMore": hasMore, + "TotalPages": ((len(emails) + limit - 1) / limit), + } + + // Check if this is an HTMX request + if r.Header.Get("HX-Request") == "true" { + if err := h.tmpl.RenderPartial(w, "emails/table.html", "email-table", data); err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + } + return + } + + if err := h.tmpl.Render(w, "emails/index.html", data); err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + } +} + +func (h *PageHandler) EmailsShow(w http.ResponseWriter, r *http.Request) { + vars := mux.Vars(r) + id, err := strconv.Atoi(vars["id"]) + if err != nil { + http.Error(w, "Invalid email ID", http.StatusBadRequest) + return + } + + // Get email details from database + emailQuery := ` + SELECT e.id, e.subject, e.user_id, e.created, e.gmail_message_id, + e.gmail_thread_id, e.raw_headers, e.is_downloaded, + u.email as user_email, u.first_name, u.last_name + FROM emails e + LEFT JOIN users u ON e.user_id = u.id + WHERE e.id = ?` + + var email struct { + ID int32 `json:"id"` + Subject string `json:"subject"` + UserID int32 `json:"user_id"` + Created time.Time `json:"created"` + GmailMessageID *string `json:"gmail_message_id"` + GmailThreadID *string `json:"gmail_thread_id"` + RawHeaders *string `json:"raw_headers"` + IsDownloaded *bool `json:"is_downloaded"` + User *struct { + Email string `json:"email"` + FirstName string `json:"first_name"` + LastName string `json:"last_name"` + } `json:"user"` + Enquiries []int32 `json:"enquiries"` + Invoices []int32 `json:"invoices"` + PurchaseOrders []int32 `json:"purchase_orders"` + Jobs []int32 `json:"jobs"` + } + + var gmailMessageID, gmailThreadID, rawHeaders sql.NullString + var isDownloaded sql.NullBool + var userEmail, firstName, lastName sql.NullString + + err = h.db.QueryRow(emailQuery, id).Scan( + &email.ID, + &email.Subject, + &email.UserID, + &email.Created, + &gmailMessageID, + &gmailThreadID, + &rawHeaders, + &isDownloaded, + &userEmail, + &firstName, + &lastName, + ) + + if err != nil { + if err == sql.ErrNoRows { + http.Error(w, "Email not found", http.StatusNotFound) + } else { + log.Printf("Error fetching email %d: %v", id, err) + http.Error(w, "Database error", http.StatusInternalServerError) + } + return + } + + // Set nullable fields + if gmailMessageID.Valid { + email.GmailMessageID = &gmailMessageID.String + } + if gmailThreadID.Valid { + email.GmailThreadID = &gmailThreadID.String + } + if rawHeaders.Valid { + email.RawHeaders = &rawHeaders.String + } + if isDownloaded.Valid { + email.IsDownloaded = &isDownloaded.Bool + } + + // Set user info if available + if userEmail.Valid { + email.User = &struct { + Email string `json:"email"` + FirstName string `json:"first_name"` + LastName string `json:"last_name"` + }{ + Email: userEmail.String, + FirstName: firstName.String, + LastName: lastName.String, + } + } + + // Get email attachments + attachmentQuery := ` + SELECT id, name, type, size, filename, is_message_body, gmail_attachment_id, created + FROM email_attachments + WHERE email_id = ? + ORDER BY is_message_body DESC, created ASC` + + attachmentRows, err := h.db.Query(attachmentQuery, id) + if err != nil { + log.Printf("Error fetching attachments for email %d: %v", id, err) + } + + type EmailAttachment struct { + ID int32 `json:"id"` + Name string `json:"name"` + Type string `json:"type"` + Size int32 `json:"size"` + Filename string `json:"filename"` + IsMessageBody bool `json:"is_message_body"` + GmailAttachmentID *string `json:"gmail_attachment_id"` + Created time.Time `json:"created"` + } + + var attachments []EmailAttachment + hasStoredAttachments := false + + if attachmentRows != nil { + defer attachmentRows.Close() + for attachmentRows.Next() { + hasStoredAttachments = true + var attachment EmailAttachment + var gmailAttachmentID sql.NullString + + err := attachmentRows.Scan( + &attachment.ID, + &attachment.Name, + &attachment.Type, + &attachment.Size, + &attachment.Filename, + &attachment.IsMessageBody, + &gmailAttachmentID, + &attachment.Created, + ) + if err != nil { + log.Printf("Error scanning attachment: %v", err) + continue + } + + if gmailAttachmentID.Valid { + attachment.GmailAttachmentID = &gmailAttachmentID.String + } + + attachments = append(attachments, attachment) + } + } + + // If no stored attachments and this is a Gmail email, show a notice + if !hasStoredAttachments && email.GmailMessageID != nil { + // For the page view, we'll just show a notice that attachments can be fetched + // The actual fetching will happen via the API endpoint when needed + log.Printf("Email %d is a Gmail email without indexed attachments", id) + } + + // Get associated records (simplified queries for now) + // Enquiries + enquiryRows, err := h.db.Query("SELECT enquiry_id FROM emails_enquiries WHERE email_id = ?", id) + if err == nil { + defer enquiryRows.Close() + for enquiryRows.Next() { + var enquiryID int32 + if enquiryRows.Scan(&enquiryID) == nil { + email.Enquiries = append(email.Enquiries, enquiryID) + } + } + } + + // Invoices + invoiceRows, err := h.db.Query("SELECT invoice_id FROM emails_invoices WHERE email_id = ?", id) + if err == nil { + defer invoiceRows.Close() + for invoiceRows.Next() { + var invoiceID int32 + if invoiceRows.Scan(&invoiceID) == nil { + email.Invoices = append(email.Invoices, invoiceID) + } + } + } + + // Purchase Orders + poRows, err := h.db.Query("SELECT purchase_order_id FROM emails_purchase_orders WHERE email_id = ?", id) + if err == nil { + defer poRows.Close() + for poRows.Next() { + var poID int32 + if poRows.Scan(&poID) == nil { + email.PurchaseOrders = append(email.PurchaseOrders, poID) + } + } + } + + // Jobs + jobRows, err := h.db.Query("SELECT job_id FROM emails_jobs WHERE email_id = ?", id) + if err == nil { + defer jobRows.Close() + for jobRows.Next() { + var jobID int32 + if jobRows.Scan(&jobID) == nil { + email.Jobs = append(email.Jobs, jobID) + } + } + } + + data := map[string]interface{}{ + "Email": email, + "Attachments": attachments, + } + + if err := h.tmpl.Render(w, "emails/show.html", data); err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + } +} + +func (h *PageHandler) EmailsSearch(w http.ResponseWriter, r *http.Request) { + _ = r.URL.Query().Get("search") // TODO: Implement search functionality + + // Empty result for now - would need proper implementation + emails := []interface{}{} + + data := map[string]interface{}{ + "Emails": emails, + "Page": 1, + "PrevPage": 0, + "NextPage": 2, + "HasMore": false, + "TotalPages": 1, + } + + w.Header().Set("Content-Type", "text/html") + if err := h.tmpl.RenderPartial(w, "emails/table.html", "email-table", data); err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + } +} + +func (h *PageHandler) EmailsAttachments(w http.ResponseWriter, r *http.Request) { + vars := mux.Vars(r) + _, err := strconv.Atoi(vars["id"]) + if err != nil { + http.Error(w, "Invalid email ID", http.StatusBadRequest) + return + } + + // Empty attachments for now - would need proper implementation + attachments := []interface{}{} + + data := map[string]interface{}{ + "Attachments": attachments, + } + + w.Header().Set("Content-Type", "text/html") + if err := h.tmpl.RenderPartial(w, "emails/attachments.html", "email-attachments", data); err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + } +} diff --git a/go-app/internal/cmc/templates/templates.go b/go-app/internal/cmc/templates/templates.go index ba9071e5..18f18e40 100644 --- a/go-app/internal/cmc/templates/templates.go +++ b/go-app/internal/cmc/templates/templates.go @@ -54,6 +54,10 @@ func NewTemplateManager(templatesDir string) (*TemplateManager, error) { "enquiries/show.html", "enquiries/form.html", "enquiries/table.html", + "emails/index.html", + "emails/show.html", + "emails/table.html", + "emails/attachments.html", "documents/index.html", "documents/show.html", "documents/table.html", diff --git a/go-app/sql/migrations/001_add_gmail_fields.sql b/go-app/sql/migrations/001_add_gmail_fields.sql new file mode 100644 index 00000000..9fd351b6 --- /dev/null +++ b/go-app/sql/migrations/001_add_gmail_fields.sql @@ -0,0 +1,52 @@ +-- +goose Up +-- Add Gmail-specific fields to emails table +ALTER TABLE emails + ADD COLUMN gmail_message_id VARCHAR(255) UNIQUE AFTER id, + ADD COLUMN gmail_thread_id VARCHAR(255) AFTER gmail_message_id, + ADD COLUMN is_downloaded BOOLEAN DEFAULT FALSE AFTER email_attachment_count, + ADD COLUMN raw_headers TEXT AFTER subject; + +-- +goose StatementBegin +CREATE INDEX idx_gmail_message_id ON emails(gmail_message_id); +-- +goose StatementEnd + +-- +goose StatementBegin +CREATE INDEX idx_gmail_thread_id ON emails(gmail_thread_id); +-- +goose StatementEnd + +-- Add Gmail-specific fields to email_attachments +ALTER TABLE email_attachments + ADD COLUMN gmail_attachment_id VARCHAR(255) AFTER email_id, + ADD COLUMN gmail_message_id VARCHAR(255) AFTER gmail_attachment_id, + ADD COLUMN content_id VARCHAR(255) AFTER filename; + +-- +goose StatementBegin +CREATE INDEX idx_gmail_attachment_id ON email_attachments(gmail_attachment_id); +-- +goose StatementEnd + +-- +goose Down +-- Remove indexes +-- +goose StatementBegin +DROP INDEX idx_gmail_attachment_id ON email_attachments; +-- +goose StatementEnd + +-- +goose StatementBegin +DROP INDEX idx_gmail_thread_id ON emails; +-- +goose StatementEnd + +-- +goose StatementBegin +DROP INDEX idx_gmail_message_id ON emails; +-- +goose StatementEnd + +-- Remove columns from email_attachments +ALTER TABLE email_attachments + DROP COLUMN content_id, + DROP COLUMN gmail_message_id, + DROP COLUMN gmail_attachment_id; + +-- Remove columns from emails +ALTER TABLE emails + DROP COLUMN raw_headers, + DROP COLUMN is_downloaded, + DROP COLUMN gmail_thread_id, + DROP COLUMN gmail_message_id; \ No newline at end of file diff --git a/go-app/static/css/style.css b/go-app/static/css/style.css index d5d7e6d7..7c622bba 100644 --- a/go-app/static/css/style.css +++ b/go-app/static/css/style.css @@ -106,4 +106,20 @@ body { .input.is-danger:focus { border-color: #ff3860; box-shadow: 0 0 0 0.125em rgba(255,56,96,.25); +} + +/* Simple CSS loader */ +.loader { + border: 2px solid #f3f3f3; + border-top: 2px solid #3273dc; + border-radius: 50%; + width: 16px; + height: 16px; + animation: spin 1s linear infinite; + display: inline-block; +} + +@keyframes spin { + 0% { transform: rotate(0deg); } + 100% { transform: rotate(360deg); } } \ No newline at end of file diff --git a/go-app/templates/emails/attachments.html b/go-app/templates/emails/attachments.html new file mode 100644 index 00000000..24086346 --- /dev/null +++ b/go-app/templates/emails/attachments.html @@ -0,0 +1,67 @@ +{{define "email-attachments"}} +
+ + + + + + + + + + + + {{range .}} + + + + + + + + {{else}} + + + + {{end}} + +
NameTypeSizeDateActions
+ {{if .IsMessageBody}} + + + + {{.Name}} + Body + {{else}} + + + + {{.Name}} + {{end}} + + {{.Type}} + {{.Size}} bytes + {{.Created}} + + {{if .GmailAttachmentID}} + + + + + Stream + + {{else}} + + + + + Download + + {{end}} +
+

No attachments found

+
+
+{{end}} \ No newline at end of file diff --git a/go-app/templates/emails/index.html b/go-app/templates/emails/index.html new file mode 100644 index 00000000..d3ba1bab --- /dev/null +++ b/go-app/templates/emails/index.html @@ -0,0 +1,82 @@ +{{define "title"}}Emails - CMC Sales{{end}} + +{{define "content"}} +
+
+
+

Emails

+
+
+
+
+
+
+
+ +
+
+
+ +
+
+
+
+
+ + +
+
+
+ + + + +
+
+ +
+
+
+ + +
+ {{template "email-table" .}} +
+{{end}} + +{{define "scripts"}} + +{{end}} \ No newline at end of file diff --git a/go-app/templates/emails/show.html b/go-app/templates/emails/show.html new file mode 100644 index 00000000..41a67278 --- /dev/null +++ b/go-app/templates/emails/show.html @@ -0,0 +1,293 @@ +{{define "title"}}Email {{.Email.ID}} - CMC Sales{{end}} + +{{define "content"}} +
+
+
+ +
+
+
+
+
+ {{if .Email.GmailMessageID}} + + + + + View in Gmail + + {{end}} + + + + + Back to Emails + +
+
+
+
+ +
+
+ +
+

+ {{if .Email.Subject}}{{.Email.Subject}}{{else}}(No Subject){{end}} +

+ +
+
+
+ From: + {{if .Email.User}} + {{.Email.User.Email}} ({{.Email.User.FirstName}} {{.Email.User.LastName}}) + {{else}} + Unknown sender + {{end}} +
+
+ Date: {{.Email.Created.Format "2006-01-02 15:04:05"}} +
+
+ Type: + {{if .Email.GmailMessageID}} + + + + + Gmail + + {{if and .Email.IsDownloaded (not .Email.IsDownloaded)}} + Remote + {{else}} + Downloaded + {{end}} + {{else}} + Local Email + {{end}} +
+
+ Attachments: + {{if gt (len .Attachments) 0}} + {{len .Attachments}} files + {{else}} + None + {{end}} +
+ {{if .Email.GmailMessageID}} +
+ Gmail Message ID: + {{.Email.GmailMessageID}} +
+ {{end}} + {{if .Email.GmailThreadID}} +
+ Gmail Thread ID: + {{.Email.GmailThreadID}} +
+ {{end}} +
+
+
+ + + {{if .Email.GmailMessageID}} +
+

Email Content

+
+
+
+ + Loading email content... +
+
+
+
+ {{end}} + + + {{if gt (len .Attachments) 0}} +
+

Attachments

+
+ + + + + + + + + + + + {{range .Attachments}} + + + + + + + + {{end}} + +
NameTypeSizeDateActions
+ {{if .IsMessageBody}} + + + + {{else}} + + + + {{end}} + {{.Name}} + + {{.Type}} + {{.Size}} bytes + {{.Created.Format "2006-01-02 15:04"}} + + {{if .GmailAttachmentID}} + + + + + Download + + {{else}} + + + + + Download + + {{end}} +
+
+
+ {{else if .Email.GmailMessageID}} + +
+

Attachments

+
+
+
+ + Checking for Gmail attachments... +
+
+
+
+ {{end}} +
+ +
+ +
+

Associated Records

+ + {{if .Email.Enquiries}} +
+ +
+ {{range .Email.Enquiries}} + + ENQ-{{.}} + + {{end}} +
+
+ {{end}} + + {{if .Email.Invoices}} +
+ +
+ {{range .Email.Invoices}} + + INV-{{.}} + + {{end}} +
+
+ {{end}} + + {{if .Email.PurchaseOrders}} +
+ +
+ {{range .Email.PurchaseOrders}} + + PO-{{.}} + + {{end}} +
+
+ {{end}} + + {{if .Email.Jobs}} +
+ +
+ {{range .Email.Jobs}} + + JOB-{{.}} + + {{end}} +
+
+ {{end}} + + {{if and (not .Email.Enquiries) (not .Email.Invoices) (not .Email.PurchaseOrders) (not .Email.Jobs)}} +
+

No associations found

+

This email is not associated with any enquiries, invoices, purchase orders, or jobs.

+
+ {{end}} +
+ + +
+

Quick Actions

+
+ + + {{if .Email.GmailMessageID}} + + {{end}} +
+
+
+
+{{end}} \ No newline at end of file diff --git a/go-app/templates/emails/table.html b/go-app/templates/emails/table.html new file mode 100644 index 00000000..a0798ce3 --- /dev/null +++ b/go-app/templates/emails/table.html @@ -0,0 +1,143 @@ +{{define "email-table"}} +
+ + + + + + + + + + + + + + + {{range .Emails}} + + + + + + + + + + + {{else}} + + + + {{end}} + +
IDSubjectFromDateAttachmentsTypeAssociatedActions
{{.ID}} + + {{if .Subject}}{{.Subject}}{{else}}(No Subject){{end}} + + + {{if .UserEmail}} + {{.UserEmail}} + {{else}} + Unknown + {{end}} + + + {{.Created.Format "2006-01-02 15:04"}} + + + {{if gt .AttachmentCount 0}} + + + + + {{.AttachmentCount}} + + {{else}} + None + {{end}} + + {{if .GmailMessageID}} + + + + + Gmail + + {{if and .IsDownloaded (not .IsDownloaded)}} + + + + + Remote + + {{end}} + {{else}} + + + + + Local + + {{end}} + + Associations TBD + +
+ + + + + + {{if gt .AttachmentCount 0}} + + {{end}} + {{if .GmailMessageID}} + + + + + + {{end}} +
+
+

No emails found

+
+
+ + +{{if .Emails}} + +{{end}} + + + +{{end}} \ No newline at end of file diff --git a/go-app/templates/layouts/base.html b/go-app/templates/layouts/base.html index a9e75312..022bc5ad 100644 --- a/go-app/templates/layouts/base.html +++ b/go-app/templates/layouts/base.html @@ -61,6 +61,11 @@ Enquiries + + + + Emails +