Merge branch 'master' into prod
This commit is contained in:
Karl Cordes
commit
b21ab4a7bf
344
DEPLOYMENT-CADDY.md
Normal file
344
DEPLOYMENT-CADDY.md
Normal file
|
|
@ -0,0 +1,344 @@
|
|||
# CMC Sales Deployment Guide with Caddy
|
||||
|
||||
## Overview
|
||||
|
||||
This guide covers deploying the CMC Sales application to a Debian 12 VM using Caddy as the reverse proxy with automatic HTTPS.
|
||||
|
||||
## Architecture
|
||||
|
||||
- **Production**: `https://cmc.springupsoftware.com`
|
||||
- **Staging**: `https://staging.cmc.springupsoftware.com`
|
||||
- **Reverse Proxy**: Caddy (running on host)
|
||||
- **Applications**: Docker containers
|
||||
- CakePHP legacy app
|
||||
- Go modern app
|
||||
- MariaDB database
|
||||
- **SSL**: Automatic via Caddy (Let's Encrypt)
|
||||
- **Authentication**: Basic auth configured in Caddy
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### 1. Server Setup (Debian 12)
|
||||
|
||||
```bash
|
||||
# Update system
|
||||
sudo apt update && sudo apt upgrade -y
|
||||
|
||||
# Install Docker
|
||||
sudo apt install -y docker.io docker-compose-plugin
|
||||
sudo systemctl enable docker
|
||||
sudo systemctl start docker
|
||||
|
||||
# Add user to docker group
|
||||
sudo usermod -aG docker $USER
|
||||
# Log out and back in
|
||||
|
||||
# Create directories
|
||||
sudo mkdir -p /var/backups/cmc-sales
|
||||
sudo chown $USER:$USER /var/backups/cmc-sales
|
||||
```
|
||||
|
||||
### 2. Install Caddy
|
||||
|
||||
```bash
|
||||
# Run the installation script
|
||||
sudo ./scripts/install-caddy.sh
|
||||
|
||||
# Or manually install
|
||||
sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
|
||||
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
|
||||
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
|
||||
sudo apt update
|
||||
sudo apt install caddy
|
||||
```
|
||||
|
||||
### 3. DNS Configuration
|
||||
|
||||
Ensure DNS records point to your server:
|
||||
- `cmc.springupsoftware.com` → Server IP
|
||||
- `staging.cmc.springupsoftware.com` → Server IP
|
||||
|
||||
## Initial Deployment
|
||||
|
||||
### 1. Clone Repository
|
||||
|
||||
```bash
|
||||
cd /home/cmc
|
||||
git clone git@code.springupsoftware.com:cmc/cmc-sales.git cmc-sales
|
||||
sudo chown -R $USER:$USER cmc-sales
|
||||
cd cmc-sales
|
||||
```
|
||||
|
||||
### 2. Environment Configuration
|
||||
|
||||
```bash
|
||||
# Copy environment files
|
||||
cp .env.staging go-app/.env.staging
|
||||
cp .env.production go-app/.env.production
|
||||
|
||||
# Edit with actual passwords
|
||||
nano .env.staging
|
||||
nano .env.production
|
||||
|
||||
# Create credential directories
|
||||
mkdir -p credentials/staging credentials/production
|
||||
```
|
||||
|
||||
### 3. Setup Basic Authentication
|
||||
|
||||
```bash
|
||||
# Generate password hashes for Caddy
|
||||
./scripts/setup-caddy-auth.sh
|
||||
|
||||
# Or manually
|
||||
caddy hash-password
|
||||
# Copy the hash and update Caddyfile
|
||||
```
|
||||
|
||||
### 4. Configure Caddy
|
||||
|
||||
```bash
|
||||
# Copy Caddyfile
|
||||
sudo cp Caddyfile /etc/caddy/Caddyfile
|
||||
|
||||
# Edit to update passwords and email
|
||||
sudo nano /etc/caddy/Caddyfile
|
||||
|
||||
# Validate configuration
|
||||
caddy validate --config /etc/caddy/Caddyfile
|
||||
|
||||
# Start Caddy
|
||||
sudo systemctl start caddy
|
||||
sudo systemctl status caddy
|
||||
```
|
||||
|
||||
### 5. Gmail OAuth Setup
|
||||
|
||||
Same as before - set up OAuth credentials for each environment:
|
||||
- Staging: `credentials/staging/credentials.json`
|
||||
- Production: `credentials/production/credentials.json`
|
||||
|
||||
### 6. Database Initialization
|
||||
|
||||
```bash
|
||||
# Start database containers
|
||||
docker compose -f docker-compose.caddy-staging.yml up -d db-staging
|
||||
docker compose -f docker-compose.caddy-production.yml up -d db-production
|
||||
|
||||
# Wait for databases
|
||||
sleep 30
|
||||
|
||||
# Restore production database (if you have a backup)
|
||||
./scripts/restore-db.sh production /path/to/backup.sql.gz
|
||||
```
|
||||
|
||||
## Deployment Commands
|
||||
|
||||
### Starting Services
|
||||
|
||||
```bash
|
||||
# Start staging environment
|
||||
docker compose -f docker-compose.caddy-staging.yml up -d
|
||||
|
||||
# Start production environment
|
||||
docker compose -f docker-compose.caddy-production.yml up -d
|
||||
|
||||
# Reload Caddy configuration
|
||||
sudo systemctl reload caddy
|
||||
```
|
||||
|
||||
### Updating Applications
|
||||
|
||||
```bash
|
||||
# Pull latest code
|
||||
git pull origin main
|
||||
|
||||
# Update staging
|
||||
docker compose -f docker-compose.caddy-staging.yml down
|
||||
docker compose -f docker-compose.caddy-staging.yml build --no-cache
|
||||
docker compose -f docker-compose.caddy-staging.yml up -d
|
||||
|
||||
# Test staging, then update production
|
||||
docker compose -f docker-compose.caddy-production.yml down
|
||||
docker compose -f docker-compose.caddy-production.yml build --no-cache
|
||||
docker compose -f docker-compose.caddy-production.yml up -d
|
||||
```
|
||||
|
||||
## Caddy Management
|
||||
|
||||
### Configuration
|
||||
|
||||
```bash
|
||||
# Edit Caddyfile
|
||||
sudo nano /etc/caddy/Caddyfile
|
||||
|
||||
# Validate configuration
|
||||
caddy validate --config /etc/caddy/Caddyfile
|
||||
|
||||
# Reload configuration (zero downtime)
|
||||
sudo systemctl reload caddy
|
||||
```
|
||||
|
||||
### Monitoring
|
||||
|
||||
```bash
|
||||
# Check Caddy status
|
||||
sudo systemctl status caddy
|
||||
|
||||
# View Caddy logs
|
||||
sudo journalctl -u caddy -f
|
||||
|
||||
# View access logs
|
||||
sudo tail -f /var/log/caddy/cmc-production.log
|
||||
sudo tail -f /var/log/caddy/cmc-staging.log
|
||||
```
|
||||
|
||||
### SSL Certificates
|
||||
|
||||
Caddy handles SSL automatically! To check certificates:
|
||||
|
||||
```bash
|
||||
# List certificates
|
||||
sudo ls -la /var/lib/caddy/.local/share/caddy/certificates/
|
||||
|
||||
# Force certificate renewal (rarely needed)
|
||||
sudo systemctl stop caddy
|
||||
sudo rm -rf /var/lib/caddy/.local/share/caddy/certificates/*
|
||||
sudo systemctl start caddy
|
||||
```
|
||||
|
||||
## Container Port Mapping
|
||||
|
||||
| Service | Container Port | Host Port | Access |
|
||||
|---------|---------------|-----------|---------|
|
||||
| cmc-php-staging | 80 | 8091 | localhost only |
|
||||
| cmc-go-staging | 8080 | 8092 | localhost only |
|
||||
| cmc-db-staging | 3306 | 3307 | localhost only |
|
||||
| cmc-php-production | 80 | 8093 | localhost only |
|
||||
| cmc-go-production | 8080 | 8094 | localhost only |
|
||||
| cmc-db-production | 3306 | - | internal only |
|
||||
|
||||
## Monitoring and Maintenance
|
||||
|
||||
### Health Checks
|
||||
|
||||
```bash
|
||||
# Check all containers
|
||||
docker ps
|
||||
|
||||
# Check application health
|
||||
curl -I https://cmc.springupsoftware.com
|
||||
curl -I https://staging.cmc.springupsoftware.com
|
||||
|
||||
# Internal health checks (from server)
|
||||
curl http://localhost:8094/api/v1/health # Production Go
|
||||
curl http://localhost:8092/api/v1/health # Staging Go
|
||||
```
|
||||
|
||||
### Database Backups
|
||||
|
||||
Same backup scripts work:
|
||||
|
||||
```bash
|
||||
# Manual backup
|
||||
./scripts/backup-db.sh production
|
||||
./scripts/backup-db.sh staging
|
||||
|
||||
# Automated backups
|
||||
sudo crontab -e
|
||||
# Add:
|
||||
# 0 2 * * * /home/cmc/cmc-sales/scripts/backup-db.sh production
|
||||
# 0 3 * * * /home/cmc/cmc-sales/scripts/backup-db.sh staging
|
||||
```
|
||||
|
||||
## Security Benefits with Caddy
|
||||
|
||||
1. **Automatic HTTPS**: No manual certificate management
|
||||
2. **Modern TLS**: Always up-to-date TLS configuration
|
||||
3. **OCSP Stapling**: Enabled by default
|
||||
4. **Security Headers**: Easy to configure
|
||||
5. **Rate Limiting**: Built-in support
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Caddy Issues
|
||||
|
||||
```bash
|
||||
# Check Caddy configuration
|
||||
caddy validate --config /etc/caddy/Caddyfile
|
||||
|
||||
# Check Caddy service
|
||||
sudo systemctl status caddy
|
||||
sudo journalctl -u caddy -n 100
|
||||
|
||||
# Test reverse proxy
|
||||
curl -v http://localhost:8094/api/v1/health
|
||||
```
|
||||
|
||||
### Container Issues
|
||||
|
||||
```bash
|
||||
# Check container logs
|
||||
docker compose -f docker-compose.caddy-production.yml logs -f
|
||||
|
||||
# Restart specific service
|
||||
docker compose -f docker-compose.caddy-production.yml restart cmc-go-production
|
||||
```
|
||||
|
||||
### SSL Issues
|
||||
|
||||
```bash
|
||||
# Caddy automatically handles SSL, but if issues arise:
|
||||
# 1. Check DNS is resolving correctly
|
||||
dig cmc.springupsoftware.com
|
||||
|
||||
# 2. Check Caddy can reach Let's Encrypt
|
||||
sudo journalctl -u caddy | grep -i acme
|
||||
|
||||
# 3. Ensure ports 80 and 443 are open
|
||||
sudo ufw status
|
||||
```
|
||||
|
||||
## Advantages of Caddy Setup
|
||||
|
||||
1. **Simpler Configuration**: Caddyfile is more readable than nginx
|
||||
2. **Automatic HTTPS**: No certbot or lego needed
|
||||
3. **Zero-Downtime Reloads**: Config changes without dropping connections
|
||||
4. **Better Performance**: More efficient than nginx for this use case
|
||||
5. **Native Rate Limiting**: Built-in without additional modules
|
||||
6. **Automatic Certificate Renewal**: No cron jobs needed
|
||||
|
||||
## Migration from Nginx
|
||||
|
||||
If migrating from the nginx setup:
|
||||
|
||||
1. Stop nginx containers: `docker compose -f docker-compose.proxy.yml down`
|
||||
2. Install and configure Caddy
|
||||
3. Start new containers with caddy compose files
|
||||
4. Update DNS if needed
|
||||
5. Monitor logs during transition
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
/home/cmc/cmc-sales/
|
||||
├── docker-compose.caddy-staging.yml
|
||||
├── docker-compose.caddy-production.yml
|
||||
├── Caddyfile
|
||||
├── credentials/
|
||||
│ ├── staging/
|
||||
│ └── production/
|
||||
├── scripts/
|
||||
│ ├── backup-db.sh
|
||||
│ ├── restore-db.sh
|
||||
│ ├── install-caddy.sh
|
||||
│ └── setup-caddy-auth.sh
|
||||
└── .env files
|
||||
|
||||
/etc/caddy/
|
||||
└── Caddyfile (deployed config)
|
||||
|
||||
/var/log/caddy/
|
||||
├── cmc-production.log
|
||||
└── cmc-staging.log
|
||||
```
|
||||
362
DEPLOYMENT.md
Normal file
362
DEPLOYMENT.md
Normal file
|
|
@ -0,0 +1,362 @@
|
|||
# CMC Sales Deployment Guide
|
||||
|
||||
## Overview
|
||||
|
||||
This guide covers deploying the CMC Sales application to a Debian 12 VM at `cmc.springupsoftware.com` with both staging and production environments.
|
||||
|
||||
## Architecture
|
||||
|
||||
- **Production**: `https://cmc.springupsoftware.com`
|
||||
- **Staging**: `https://staging.cmc.springupsoftware.com`
|
||||
- **Components**: CakePHP legacy app, Go modern app, MariaDB, Nginx reverse proxy
|
||||
- **SSL**: Let's Encrypt certificates
|
||||
- **Authentication**: Basic auth for both environments
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Server Setup (Debian 12)
|
||||
|
||||
```bash
|
||||
# Update system
|
||||
sudo apt update && sudo apt upgrade -y
|
||||
|
||||
# Install Docker and Docker Compose
|
||||
sudo apt install -y docker.io docker-compose-plugin
|
||||
sudo systemctl enable docker
|
||||
sudo systemctl start docker
|
||||
|
||||
# Add user to docker group
|
||||
sudo usermod -aG docker $USER
|
||||
# Log out and back in
|
||||
|
||||
# Create backup directory
|
||||
sudo mkdir -p /var/backups/cmc-sales
|
||||
sudo chown $USER:$USER /var/backups/cmc-sales
|
||||
```
|
||||
|
||||
### DNS Configuration
|
||||
|
||||
Ensure these DNS records point to your server:
|
||||
- `cmc.springupsoftware.com` → Server IP
|
||||
- `staging.cmc.springupsoftware.com` → Server IP
|
||||
|
||||
## Initial Deployment
|
||||
|
||||
### 1. Clone Repository
|
||||
|
||||
```bash
|
||||
cd /home/cmc
|
||||
git clone git@code.springupsoftware.com:cmc/cmc-sales.git cmc-sales
|
||||
sudo chown -R $USER:$USER cmc-sales
|
||||
cd cmc-sales
|
||||
```
|
||||
|
||||
### 2. Environment Configuration
|
||||
|
||||
```bash
|
||||
# Copy environment files
|
||||
cp .env.staging go-app/.env.staging
|
||||
cp .env.production go-app/.env.production
|
||||
|
||||
# Edit with actual passwords -- up to this.
|
||||
nano .env.staging
|
||||
nano .env.production
|
||||
|
||||
# Create credential directories
|
||||
mkdir -p credentials/staging credentials/production
|
||||
```
|
||||
|
||||
### 3. Gmail OAuth Setup
|
||||
|
||||
For each environment (staging/production):
|
||||
|
||||
1. Go to [Google Cloud Console](https://console.cloud.google.com)
|
||||
2. Create/select project
|
||||
3. Enable Gmail API
|
||||
4. Create OAuth 2.0 credentials
|
||||
5. Download `credentials.json`
|
||||
6. Place in appropriate directory:
|
||||
- Staging: `credentials/staging/credentials.json`
|
||||
- Production: `credentials/production/credentials.json`
|
||||
|
||||
Generate tokens (run on local machine first):
|
||||
```bash
|
||||
cd go-app
|
||||
go run cmd/auth/main.go
|
||||
# Follow OAuth flow
|
||||
# Copy token.json to appropriate credential directory
|
||||
```
|
||||
|
||||
### 4. SSL Certificates
|
||||
|
||||
```bash
|
||||
# Start proxy services (includes Lego container)
|
||||
docker compose -f docker-compose.proxy.yml up -d
|
||||
|
||||
# Setup SSL certificates using Lego
|
||||
./scripts/setup-lego-certs.sh accounts@springupsoftware.com
|
||||
|
||||
# Verify certificates
|
||||
./scripts/lego-list-certs.sh
|
||||
```
|
||||
|
||||
### 5. Database Initialization
|
||||
|
||||
```bash
|
||||
# Start database containers first
|
||||
docker compose -f docker-compose.staging.yml up -d db-staging
|
||||
docker compose -f docker-compose.production.yml up -d db-production
|
||||
|
||||
# Wait for databases to be ready
|
||||
sleep 30
|
||||
|
||||
# Restore production database (if you have a backup)
|
||||
./scripts/restore-db.sh production /path/to/backup.sql.gz
|
||||
|
||||
# Or initialize empty database and run migrations
|
||||
# (implementation specific)
|
||||
```
|
||||
|
||||
## Deployment Commands
|
||||
|
||||
### Starting Services
|
||||
|
||||
```bash
|
||||
# Start staging environment
|
||||
docker compose -f docker-compose.staging.yml up -d
|
||||
|
||||
# Start production environment
|
||||
docker compose -f docker-compose.production.yml up -d
|
||||
|
||||
# Wait for services to be ready
|
||||
sleep 10
|
||||
|
||||
# Start reverse proxy (after both environments are running)
|
||||
docker compose -f docker-compose.proxy.yml up -d
|
||||
|
||||
# Or use the make command for full stack deployment
|
||||
make full-stack
|
||||
```
|
||||
|
||||
### Updating Applications
|
||||
|
||||
```bash
|
||||
# Pull latest code
|
||||
git pull origin main
|
||||
|
||||
# Rebuild and restart staging
|
||||
docker compose -f docker-compose.staging.yml down
|
||||
docker compose -f docker-compose.staging.yml build --no-cache
|
||||
docker compose -f docker-compose.staging.yml up -d
|
||||
|
||||
# Test staging thoroughly, then update production
|
||||
docker compose -f docker-compose.production.yml down
|
||||
docker compose -f docker-compose.production.yml build --no-cache
|
||||
docker compose -f docker-compose.production.yml up -d
|
||||
```
|
||||
|
||||
## Monitoring and Maintenance
|
||||
|
||||
### Health Checks
|
||||
|
||||
```bash
|
||||
# Check all containers
|
||||
docker ps
|
||||
|
||||
# Check logs
|
||||
docker compose -f docker-compose.production.yml logs -f
|
||||
|
||||
# Check application health
|
||||
curl https://cmc.springupsoftware.com/health
|
||||
curl https://staging.cmc.springupsoftware.com/health
|
||||
```
|
||||
|
||||
### Database Backups
|
||||
|
||||
```bash
|
||||
# Manual backup
|
||||
./scripts/backup-db.sh production
|
||||
./scripts/backup-db.sh staging
|
||||
|
||||
# Set up automated backups (cron)
|
||||
sudo crontab -e
|
||||
# Add: 0 2 * * * /opt/cmc-sales/scripts/backup-db.sh production
|
||||
# Add: 0 3 * * * /opt/cmc-sales/scripts/backup-db.sh staging
|
||||
```
|
||||
|
||||
### Log Management
|
||||
|
||||
```bash
|
||||
# View nginx logs
|
||||
sudo tail -f /var/log/nginx/access.log
|
||||
sudo tail -f /var/log/nginx/error.log
|
||||
|
||||
# View application logs
|
||||
docker compose -f docker-compose.production.yml logs -f cmc-go-production
|
||||
docker compose -f docker-compose.staging.yml logs -f cmc-go-staging
|
||||
```
|
||||
|
||||
### SSL Certificate Renewal
|
||||
|
||||
```bash
|
||||
# Manual renewal
|
||||
./scripts/lego-renew-cert.sh all
|
||||
|
||||
# Renew specific domain
|
||||
./scripts/lego-renew-cert.sh cmc.springupsoftware.com
|
||||
|
||||
# Set up auto-renewal (cron)
|
||||
sudo crontab -e
|
||||
# Add: 0 2 * * * /opt/cmc-sales/scripts/lego-renew-cert.sh all
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Basic Authentication
|
||||
|
||||
Update passwords in `userpasswd` file:
|
||||
```bash
|
||||
# Generate new password hash
|
||||
sudo apt install apache2-utils
|
||||
htpasswd -c userpasswd username
|
||||
|
||||
# Restart nginx containers
|
||||
docker compose -f docker-compose.proxy.yml restart nginx-proxy
|
||||
```
|
||||
|
||||
### Database Security
|
||||
|
||||
- Use strong passwords in environment files
|
||||
- Database containers are not exposed externally in production
|
||||
- Regular backups with encryption at rest
|
||||
|
||||
### Network Security
|
||||
|
||||
- All traffic encrypted with SSL/TLS
|
||||
- Rate limiting configured in nginx
|
||||
- Security headers enabled
|
||||
- Docker networks isolate environments
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Containers won't start**
|
||||
```bash
|
||||
# Check logs
|
||||
docker compose logs container-name
|
||||
|
||||
# Check system resources
|
||||
df -h
|
||||
free -h
|
||||
```
|
||||
|
||||
2. **SSL issues**
|
||||
```bash
|
||||
# Check certificate status
|
||||
./scripts/lego-list-certs.sh
|
||||
|
||||
# Test SSL configuration
|
||||
curl -I https://cmc.springupsoftware.com
|
||||
|
||||
# Manually renew certificates
|
||||
./scripts/lego-renew-cert.sh all
|
||||
```
|
||||
|
||||
3. **Database connection issues**
|
||||
```bash
|
||||
# Test database connectivity
|
||||
docker exec -it cmc-db-production mysql -u cmc -p
|
||||
```
|
||||
|
||||
4. **Gmail API issues**
|
||||
```bash
|
||||
# Check credentials are mounted
|
||||
docker exec -it cmc-go-production ls -la /root/credentials/
|
||||
|
||||
# Check logs for OAuth errors
|
||||
docker compose logs cmc-go-production | grep -i gmail
|
||||
```
|
||||
|
||||
### Emergency Procedures
|
||||
|
||||
1. **Quick rollback**
|
||||
```bash
|
||||
# Stop current containers
|
||||
docker compose -f docker-compose.production.yml down
|
||||
|
||||
# Restore from backup
|
||||
./scripts/restore-db.sh production /var/backups/cmc-sales/latest_backup.sql.gz
|
||||
|
||||
# Start previous version
|
||||
git checkout previous-commit
|
||||
docker compose -f docker-compose.production.yml up -d
|
||||
```
|
||||
|
||||
2. **Database corruption**
|
||||
```bash
|
||||
# Stop application
|
||||
docker compose -f docker-compose.production.yml stop cmc-go-production cmc-php-production
|
||||
|
||||
# Restore from backup
|
||||
./scripts/restore-db.sh production /var/backups/cmc-sales/backup_production_YYYYMMDD-HHMMSS.sql.gz
|
||||
|
||||
# Restart application
|
||||
docker compose -f docker-compose.production.yml start cmc-go-production cmc-php-production
|
||||
```
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
/opt/cmc-sales/
|
||||
├── docker-compose.staging.yml
|
||||
├── docker-compose.production.yml
|
||||
├── docker-compose.proxy.yml
|
||||
├── conf/
|
||||
│ ├── nginx-staging.conf
|
||||
│ ├── nginx-production.conf
|
||||
│ └── nginx-proxy.conf
|
||||
├── credentials/
|
||||
│ ├── staging/
|
||||
│ │ ├── credentials.json
|
||||
│ │ └── token.json
|
||||
│ └── production/
|
||||
│ ├── credentials.json
|
||||
│ └── token.json
|
||||
├── scripts/
|
||||
│ ├── backup-db.sh
|
||||
│ ├── restore-db.sh
|
||||
│ ├── lego-obtain-cert.sh
|
||||
│ ├── lego-renew-cert.sh
|
||||
│ ├── lego-list-certs.sh
|
||||
│ └── setup-lego-certs.sh
|
||||
└── .env files
|
||||
```
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### Resource Limits
|
||||
|
||||
Resource limits are configured in the Docker Compose files:
|
||||
- Production: 2 CPU cores, 2-4GB RAM per service
|
||||
- Staging: More relaxed limits for testing
|
||||
|
||||
### Database Optimization
|
||||
|
||||
```sql
|
||||
-- Monitor slow queries
|
||||
SHOW VARIABLES LIKE 'slow_query_log';
|
||||
SET GLOBAL slow_query_log = 'ON';
|
||||
SET GLOBAL long_query_time = 2;
|
||||
|
||||
-- Check database performance
|
||||
SHOW PROCESSLIST;
|
||||
SHOW ENGINE INNODB STATUS;
|
||||
```
|
||||
|
||||
### Nginx Optimization
|
||||
|
||||
- Gzip compression enabled
|
||||
- Static file caching
|
||||
- Connection keep-alive
|
||||
- Rate limiting configured
|
||||
65
Dockerfile
Normal file
65
Dockerfile
Normal file
|
|
@ -0,0 +1,65 @@
|
|||
FROM ghcr.io/kzrl/ubuntu:lucid
|
||||
|
||||
# Set environment variables.
|
||||
ENV HOME /root
|
||||
|
||||
# Define working directory.
|
||||
WORKDIR /root
|
||||
|
||||
RUN sed -i 's/archive/old-releases/' /etc/apt/sources.list
|
||||
|
||||
|
||||
RUN apt-get update
|
||||
RUN apt-get -y upgrade
|
||||
|
||||
# Install apache, PHP, and supplimentary programs. curl and lynx-cur are for debugging the container.
|
||||
RUN DEBIAN_FRONTEND=noninteractive apt-get -y install apache2 libapache2-mod-php5 php5-mysql php5-gd php-pear php-apc php5-curl php5-imap
|
||||
|
||||
# Enable apache mods.
|
||||
#RUN php5enmod openssl
|
||||
RUN a2enmod php5
|
||||
RUN a2enmod rewrite
|
||||
RUN a2enmod headers
|
||||
|
||||
|
||||
# Update the PHP.ini file, enable <? ?> tags and quieten logging.
|
||||
# RUN sed -i "s/short_open_tag = Off/short_open_tag = On/" /etc/php5/apache2/php.ini
|
||||
#RUN sed -i "s/error_reporting = .*$/error_reporting = E_ERROR | E_WARNING | E_PARSE/" /etc/php5/apache2/php.ini
|
||||
|
||||
ADD conf/php.ini /etc/php5/apache2/php.ini
|
||||
|
||||
# Manually set up the apache environment variables
|
||||
ENV APACHE_RUN_USER www-data
|
||||
ENV APACHE_RUN_GROUP www-data
|
||||
ENV APACHE_LOG_DIR /var/log/apache2
|
||||
ENV APACHE_LOCK_DIR /var/lock/apache2
|
||||
ENV APACHE_PID_FILE /var/run/apache2.pid
|
||||
|
||||
ARG COMMIT
|
||||
ENV COMMIT_SHA=${COMMIT}
|
||||
|
||||
EXPOSE 80
|
||||
|
||||
# Update the default apache site with the config we created.
|
||||
ADD conf/apache-vhost.conf /etc/apache2/sites-available/cmc-sales
|
||||
ADD conf/ripmime /bin/ripmime
|
||||
|
||||
RUN chmod +x /bin/ripmime
|
||||
RUN a2dissite 000-default
|
||||
RUN a2ensite cmc-sales
|
||||
|
||||
RUN mkdir -p /var/www/cmc-sales/app/tmp/logs
|
||||
RUN chmod -R 755 /var/www/cmc-sales/app/tmp
|
||||
|
||||
# Copy site into place.
|
||||
ADD . /var/www/cmc-sales
|
||||
RUN chmod +x /var/www/cmc-sales/run_vault.sh
|
||||
RUN chmod +x /var/www/cmc-sales/run_update_invoices.sh
|
||||
|
||||
|
||||
# Ensure Apache error/access logs go to Docker stdout/stderr
|
||||
RUN ln -sf /dev/stdout /var/log/apache2/access.log && \
|
||||
ln -sf /dev/stderr /var/log/apache2/error.log
|
||||
|
||||
# By default, simply start apache.
|
||||
CMD /usr/sbin/apache2ctl -D FOREGROUND
|
||||
63
Dockerfile.go.production
Normal file
63
Dockerfile.go.production
Normal file
|
|
@ -0,0 +1,63 @@
|
|||
# Build stage
|
||||
FROM golang:1.23-alpine AS builder
|
||||
|
||||
# Install build dependencies
|
||||
RUN apk add --no-cache git
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Copy go mod files
|
||||
COPY go-app/go.mod go-app/go.sum ./
|
||||
|
||||
# Download dependencies
|
||||
RUN go mod download
|
||||
|
||||
# Copy source code
|
||||
COPY go-app/ .
|
||||
|
||||
# Install sqlc (compatible with Go 1.23+)
|
||||
RUN go install github.com/sqlc-dev/sqlc/cmd/sqlc@latest
|
||||
|
||||
# Generate sqlc code
|
||||
RUN sqlc generate
|
||||
|
||||
# Build the application with production optimizations
|
||||
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags="-w -s" -tags production -o server cmd/server/main.go
|
||||
|
||||
# Runtime stage - minimal image for production
|
||||
FROM alpine:latest
|
||||
|
||||
# Install only essential runtime dependencies
|
||||
RUN apk --no-cache add ca-certificates && \
|
||||
addgroup -g 1001 -S appgroup && \
|
||||
adduser -u 1001 -S appuser -G appgroup
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Copy the binary from builder
|
||||
COPY --from=builder /app/server .
|
||||
|
||||
# Copy templates and static files
|
||||
COPY go-app/templates ./templates
|
||||
COPY go-app/static ./static
|
||||
|
||||
# Copy production environment file
|
||||
COPY go-app/.env.production .env
|
||||
|
||||
# Create credentials directory with proper permissions
|
||||
RUN mkdir -p ./credentials && \
|
||||
chown -R appuser:appgroup /app
|
||||
|
||||
# Switch to non-root user
|
||||
USER appuser
|
||||
|
||||
# Expose port
|
||||
EXPOSE 8080
|
||||
|
||||
# Health check
|
||||
HEALTHCHECK --interval=60s --timeout=10s --start-period=10s --retries=3 \
|
||||
CMD wget --no-verbose --tries=1 --spider http://localhost:8080/api/v1/health || exit 1
|
||||
|
||||
# Run the application
|
||||
CMD ["./server"]
|
||||
57
Dockerfile.go.staging
Normal file
57
Dockerfile.go.staging
Normal file
|
|
@ -0,0 +1,57 @@
|
|||
# Build stage
|
||||
FROM golang:1.23-alpine AS builder
|
||||
|
||||
# Install build dependencies
|
||||
RUN apk add --no-cache git
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Copy go mod files
|
||||
COPY go-app/go.mod go-app/go.sum ./
|
||||
|
||||
# Download dependencies
|
||||
RUN go mod download
|
||||
|
||||
# Copy source code
|
||||
COPY go-app/ .
|
||||
|
||||
# Install sqlc (compatible with Go 1.23+)
|
||||
RUN go install github.com/sqlc-dev/sqlc/cmd/sqlc@latest
|
||||
|
||||
# Generate sqlc code
|
||||
RUN sqlc generate
|
||||
|
||||
# Build the application with staging tags
|
||||
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -tags staging -o server cmd/server/main.go
|
||||
|
||||
# Runtime stage
|
||||
FROM alpine:latest
|
||||
|
||||
# Install runtime dependencies and debugging tools for staging
|
||||
RUN apk --no-cache add ca-certificates curl net-tools
|
||||
|
||||
WORKDIR /root/
|
||||
|
||||
# Copy the binary from builder
|
||||
COPY --from=builder /app/server .
|
||||
|
||||
# Copy templates and static files
|
||||
COPY go-app/templates ./templates
|
||||
COPY go-app/static ./static
|
||||
|
||||
# Copy staging environment file
|
||||
COPY go-app/.env.staging .env
|
||||
|
||||
# Create credentials directory
|
||||
RUN mkdir -p ./credentials
|
||||
|
||||
# Expose port
|
||||
EXPOSE 8080
|
||||
|
||||
# Health check
|
||||
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
|
||||
CMD curl -f http://localhost:8080/api/v1/health || exit 1
|
||||
|
||||
# Run the application
|
||||
CMD ["./server"]
|
||||
75
Dockerfile.ubuntu-php
Normal file
75
Dockerfile.ubuntu-php
Normal file
|
|
@ -0,0 +1,75 @@
|
|||
# Simple working PHP setup using Ubuntu
|
||||
FROM ubuntu:20.04
|
||||
|
||||
# Prevent interactive prompts during package installation
|
||||
ENV DEBIAN_FRONTEND=noninteractive
|
||||
ENV TZ=Australia/Sydney
|
||||
|
||||
# Install Apache, PHP and required extensions
|
||||
RUN apt-get update && apt-get install -y \
|
||||
apache2 \
|
||||
libapache2-mod-php7.4 \
|
||||
php7.4 \
|
||||
php7.4-mysql \
|
||||
php7.4-gd \
|
||||
php7.4-curl \
|
||||
php7.4-mbstring \
|
||||
php7.4-xml \
|
||||
php7.4-zip \
|
||||
php7.4-imap \
|
||||
php7.4-intl \
|
||||
php7.4-bcmath \
|
||||
curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Enable Apache modules
|
||||
RUN a2enmod rewrite headers php7.4
|
||||
|
||||
# Configure PHP for CakePHP
|
||||
RUN { \
|
||||
echo 'error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT & ~E_NOTICE & ~E_WARNING'; \
|
||||
echo 'display_errors = On'; \
|
||||
echo 'display_startup_errors = On'; \
|
||||
echo 'log_errors = On'; \
|
||||
echo 'max_execution_time = 300'; \
|
||||
echo 'memory_limit = 256M'; \
|
||||
echo 'post_max_size = 50M'; \
|
||||
echo 'upload_max_filesize = 50M'; \
|
||||
} > /etc/php/7.4/apache2/conf.d/99-cakephp.ini
|
||||
|
||||
# Set up Apache virtual host
|
||||
RUN echo '<VirtualHost *:80>\n\
|
||||
ServerName localhost\n\
|
||||
DocumentRoot /var/www/cmc-sales/app/webroot\n\
|
||||
<Directory /var/www/cmc-sales/app/webroot>\n\
|
||||
Options FollowSymLinks\n\
|
||||
AllowOverride All\n\
|
||||
Require all granted\n\
|
||||
</Directory>\n\
|
||||
ErrorLog ${APACHE_LOG_DIR}/error.log\n\
|
||||
CustomLog ${APACHE_LOG_DIR}/access.log combined\n\
|
||||
</VirtualHost>' > /etc/apache2/sites-available/000-default.conf
|
||||
|
||||
# Create app directory structure
|
||||
RUN mkdir -p /var/www/cmc-sales/app/tmp/{cache,logs,sessions} \
|
||||
&& mkdir -p /var/www/cmc-sales/app/webroot/{pdf,attachments_files}
|
||||
|
||||
# Set permissions
|
||||
RUN chown -R www-data:www-data /var/www/cmc-sales \
|
||||
&& chmod -R 777 /var/www/cmc-sales/app/tmp
|
||||
|
||||
# Copy ripmime if it exists
|
||||
# COPY conf/ripmime* /usr/local/bin/ || true
|
||||
# RUN chmod +x /usr/local/bin/ripmime* 2>/dev/null || true
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /var/www/cmc-sales
|
||||
|
||||
# Copy application (will be overridden by volume mount)
|
||||
COPY app/ /var/www/cmc-sales/app/
|
||||
|
||||
# Expose port 80
|
||||
EXPOSE 80
|
||||
|
||||
# Start Apache in foreground
|
||||
CMD ["apache2ctl", "-D", "FOREGROUND"]
|
||||
190
Makefile
Normal file
190
Makefile
Normal file
|
|
@ -0,0 +1,190 @@
|
|||
# CMC Sales Deployment Makefile for Caddy setup
|
||||
|
||||
.PHONY: help staging production backup-staging backup-production restart-staging restart-production status logs clean caddy-reload caddy-logs
|
||||
|
||||
# Default target
|
||||
help:
|
||||
@echo "CMC Sales Deployment Commands (Caddy version)"
|
||||
@echo ""
|
||||
@echo "Environments:"
|
||||
@echo " staging Start staging environment"
|
||||
@echo " staging-down Stop staging environment"
|
||||
@echo " staging-logs Show staging logs"
|
||||
@echo " restart-staging Rebuild and restart staging"
|
||||
@echo ""
|
||||
@echo " production Start production environment"
|
||||
@echo " production-down Stop production environment"
|
||||
@echo " production-logs Show production logs"
|
||||
@echo " restart-production Rebuild and restart production"
|
||||
@echo ""
|
||||
@echo "Database:"
|
||||
@echo " backup-staging Backup staging database"
|
||||
@echo " backup-production Backup production database"
|
||||
@echo ""
|
||||
@echo "Caddy:"
|
||||
@echo " caddy-status Show Caddy service status"
|
||||
@echo " caddy-reload Reload Caddy configuration"
|
||||
@echo " caddy-logs Show Caddy logs"
|
||||
@echo " caddy-validate Validate Caddyfile"
|
||||
@echo ""
|
||||
@echo "Utility:"
|
||||
@echo " status Show all container status"
|
||||
@echo " clean Stop and remove all containers"
|
||||
@echo " setup-auth Setup basic authentication"
|
||||
|
||||
# Staging environment
|
||||
staging:
|
||||
@echo "Starting staging environment..."
|
||||
docker compose -f docker-compose.caddy-staging-ubuntu.yml up -d
|
||||
@echo "Staging environment started"
|
||||
@echo "Access at: https://staging.cmc.springupsoftware.com"
|
||||
|
||||
staging-down:
|
||||
docker compose -f docker-compose.caddy-staging-ubuntu.yml down
|
||||
|
||||
staging-logs:
|
||||
docker compose -f docker-compose.caddy-staging-ubuntu.yml logs -f
|
||||
|
||||
restart-staging:
|
||||
@echo "Restarting staging environment..."
|
||||
docker compose -f docker-compose.caddy-staging-ubuntu.yml down
|
||||
docker compose -f docker-compose.caddy-staging-ubuntu.yml build --no-cache
|
||||
docker compose -f docker-compose.caddy-staging-ubuntu.yml up -d
|
||||
@echo "Staging environment restarted"
|
||||
|
||||
# Production environment
|
||||
production:
|
||||
@echo "Starting production environment..."
|
||||
docker compose -f docker-compose.caddy-production.yml up -d
|
||||
@echo "Production environment started"
|
||||
@echo "Access at: https://cmc.springupsoftware.com"
|
||||
|
||||
production-down:
|
||||
@echo "WARNING: This will stop the production environment!"
|
||||
@read -p "Are you sure? (yes/no): " confirm && [ "$$confirm" = "yes" ]
|
||||
docker compose -f docker-compose.caddy-production.yml down
|
||||
|
||||
production-logs:
|
||||
docker compose -f docker-compose.caddy-production.yml logs -f
|
||||
|
||||
restart-production:
|
||||
@echo "WARNING: This will restart the production environment!"
|
||||
@read -p "Are you sure? (yes/no): " confirm && [ "$$confirm" = "yes" ]
|
||||
docker compose -f docker-compose.caddy-production.yml down
|
||||
docker compose -f docker-compose.caddy-production.yml build --no-cache
|
||||
docker compose -f docker-compose.caddy-production.yml up -d
|
||||
@echo "Production environment restarted"
|
||||
|
||||
# Database backups
|
||||
backup-staging:
|
||||
@echo "Creating staging database backup..."
|
||||
./scripts/backup-db.sh staging
|
||||
|
||||
backup-production:
|
||||
@echo "Creating production database backup..."
|
||||
./scripts/backup-db.sh production
|
||||
|
||||
# Caddy management
|
||||
caddy-status:
|
||||
@echo "=== Caddy Status ==="
|
||||
sudo systemctl status caddy --no-pager
|
||||
|
||||
caddy-reload:
|
||||
@echo "Reloading Caddy configuration..."
|
||||
sudo systemctl reload caddy
|
||||
@echo "Caddy reloaded successfully"
|
||||
|
||||
caddy-logs:
|
||||
@echo "=== Caddy Logs ==="
|
||||
sudo journalctl -u caddy -f
|
||||
|
||||
caddy-validate:
|
||||
@echo "Validating Caddyfile..."
|
||||
caddy validate --config Caddyfile
|
||||
@echo "Caddyfile is valid"
|
||||
|
||||
# Setup authentication
|
||||
setup-auth:
|
||||
@echo "Setting up basic authentication..."
|
||||
./scripts/setup-caddy-auth.sh
|
||||
|
||||
# System status
|
||||
status:
|
||||
@echo "=== Container Status ==="
|
||||
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
|
||||
@echo ""
|
||||
@echo "=== Caddy Status ==="
|
||||
sudo systemctl is-active caddy || true
|
||||
@echo ""
|
||||
@echo "=== Port Usage ==="
|
||||
sudo netstat -tlnp | grep -E ":(80|443|8091|8092|8093|8094|3306|3307) " || true
|
||||
|
||||
# Logs
|
||||
logs:
|
||||
@echo "Which logs? [staging/production/caddy]"
|
||||
@read env; \
|
||||
case $$env in \
|
||||
staging) docker compose -f docker-compose.caddy-staging-ubuntu.yml logs -f ;; \
|
||||
production) docker compose -f docker-compose.caddy-production.yml logs -f ;; \
|
||||
caddy) sudo journalctl -u caddy -f ;; \
|
||||
*) echo "Invalid option" ;; \
|
||||
esac
|
||||
|
||||
# Cleanup
|
||||
clean:
|
||||
@echo "WARNING: This will stop and remove ALL CMC containers!"
|
||||
@read -p "Are you sure? (yes/no): " confirm && [ "$$confirm" = "yes" ]
|
||||
docker compose -f docker-compose.caddy-staging-ubuntu.yml down --volumes --remove-orphans
|
||||
docker compose -f docker-compose.caddy-production.yml down --volumes --remove-orphans
|
||||
docker system prune -f
|
||||
@echo "Cleanup completed"
|
||||
|
||||
# Health checks
|
||||
health:
|
||||
@echo "=== Health Checks ==="
|
||||
@echo "Staging:"
|
||||
@curl -s -o /dev/null -w " HTTPS %{http_code}: https://staging.cmc.springupsoftware.com/health\n" https://staging.cmc.springupsoftware.com/health -u admin:password || echo " Staging not accessible (update with correct auth)"
|
||||
@curl -s -o /dev/null -w " Internal %{http_code}: http://localhost:8092/api/v1/health\n" http://localhost:8092/api/v1/health || echo " Staging Go not accessible"
|
||||
@echo "Production:"
|
||||
@curl -s -o /dev/null -w " HTTPS %{http_code}: https://cmc.springupsoftware.com/health\n" https://cmc.springupsoftware.com/health -u admin:password || echo " Production not accessible (update with correct auth)"
|
||||
@curl -s -o /dev/null -w " Internal %{http_code}: http://localhost:8094/api/v1/health\n" http://localhost:8094/api/v1/health || echo " Production Go not accessible"
|
||||
|
||||
# Deploy to staging
|
||||
deploy-staging:
|
||||
@echo "Deploying to staging..."
|
||||
git pull origin main
|
||||
$(MAKE) restart-staging
|
||||
@echo "Staging deployment complete"
|
||||
|
||||
# Deploy to production
|
||||
deploy-production:
|
||||
@echo "WARNING: This will deploy to PRODUCTION!"
|
||||
@echo "Make sure you have tested thoroughly in staging first."
|
||||
@read -p "Are you sure you want to deploy to production? (yes/no): " confirm && [ "$$confirm" = "yes" ]
|
||||
git pull origin main
|
||||
$(MAKE) backup-production
|
||||
$(MAKE) restart-production
|
||||
@echo "Production deployment complete"
|
||||
|
||||
# First-time setup
|
||||
initial-setup:
|
||||
@echo "Running initial setup..."
|
||||
@echo "1. Installing Caddy..."
|
||||
sudo ./scripts/install-caddy.sh
|
||||
@echo ""
|
||||
@echo "2. Setting up authentication..."
|
||||
./scripts/setup-caddy-auth.sh
|
||||
@echo ""
|
||||
@echo "3. Copying Caddyfile..."
|
||||
sudo cp Caddyfile /etc/caddy/Caddyfile
|
||||
@echo ""
|
||||
@echo "4. Starting Caddy..."
|
||||
sudo systemctl start caddy
|
||||
@echo ""
|
||||
@echo "5. Starting containers..."
|
||||
$(MAKE) staging
|
||||
$(MAKE) production
|
||||
@echo ""
|
||||
@echo "Initial setup complete!"
|
||||
@echo "Access staging at: https://staging.cmc.springupsoftware.com"
|
||||
@echo "Access production at: https://cmc.springupsoftware.com"
|
||||
394
TESTING_DOCKER.md
Normal file
394
TESTING_DOCKER.md
Normal file
|
|
@ -0,0 +1,394 @@
|
|||
# Running CMC Django Tests in Docker
|
||||
|
||||
This guide explains how to run the comprehensive CMC Django test suite using Docker for consistent, isolated testing.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# 1. Setup test environment (one-time)
|
||||
./run-tests-docker.sh setup
|
||||
|
||||
# 2. Run all tests
|
||||
./run-tests-docker.sh run
|
||||
|
||||
# 3. Run tests with coverage
|
||||
./run-tests-docker.sh coverage
|
||||
```
|
||||
|
||||
## Test Environment Overview
|
||||
|
||||
The Docker test environment includes:
|
||||
- **Isolated test database** (MariaDB on port 3307)
|
||||
- **Django test container** with all dependencies
|
||||
- **Coverage reporting** with HTML and XML output
|
||||
- **PDF generation testing** with WeasyPrint/ReportLab
|
||||
- **Parallel test execution** support
|
||||
|
||||
## Available Commands
|
||||
|
||||
### Setup and Management
|
||||
|
||||
```bash
|
||||
# Build containers and setup test database
|
||||
./run-tests-docker.sh setup
|
||||
|
||||
# Clean up all test containers and data
|
||||
./run-tests-docker.sh clean
|
||||
|
||||
# View test container logs
|
||||
./run-tests-docker.sh logs
|
||||
|
||||
# Open shell in test container
|
||||
./run-tests-docker.sh shell
|
||||
```
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
./run-tests-docker.sh run
|
||||
|
||||
# Run specific test suites
|
||||
./run-tests-docker.sh run models # Model tests only
|
||||
./run-tests-docker.sh run services # Service layer tests
|
||||
./run-tests-docker.sh run auth # Authentication tests
|
||||
./run-tests-docker.sh run views # View and URL tests
|
||||
./run-tests-docker.sh run pdf # PDF generation tests
|
||||
./run-tests-docker.sh run integration # Integration tests
|
||||
|
||||
# Run quick tests (models + services)
|
||||
./run-tests-docker.sh quick
|
||||
|
||||
# Run tests with coverage reporting
|
||||
./run-tests-docker.sh coverage
|
||||
```
|
||||
|
||||
## Advanced Test Options
|
||||
|
||||
### Using Docker Compose Directly
|
||||
|
||||
```bash
|
||||
# Run specific test with custom options
|
||||
docker-compose -f docker-compose.test.yml run --rm cmc-django-test \
|
||||
python cmcsales/manage.py test cmc.tests.test_models --verbosity=2 --keepdb
|
||||
|
||||
# Run tests with coverage
|
||||
docker-compose -f docker-compose.test.yml run --rm cmc-django-test \
|
||||
coverage run --source='.' cmcsales/manage.py test cmc.tests
|
||||
|
||||
# Generate coverage report
|
||||
docker-compose -f docker-compose.test.yml run --rm cmc-django-test \
|
||||
coverage report --show-missing
|
||||
```
|
||||
|
||||
### Using the Test Script Directly
|
||||
|
||||
```bash
|
||||
# Inside the container, you can use the test script with advanced options
|
||||
docker-compose -f docker-compose.test.yml run --rm cmc-django-test \
|
||||
/app/scripts/run-tests.sh --coverage --keepdb --failfast models
|
||||
|
||||
# Script options:
|
||||
# -c, --coverage Enable coverage reporting
|
||||
# -k, --keepdb Keep test database between runs
|
||||
# -p, --parallel NUM Run tests in parallel
|
||||
# -f, --failfast Stop on first failure
|
||||
# -v, --verbosity NUM Verbosity level 0-3
|
||||
```
|
||||
|
||||
## Test Suite Structure
|
||||
|
||||
### 1. Model Tests (`test_models.py`)
|
||||
Tests all Django models including:
|
||||
- Customer, Enquiry, Job, Document models
|
||||
- Model validation and constraints
|
||||
- Relationships and cascade behavior
|
||||
- Financial calculations
|
||||
|
||||
```bash
|
||||
./run-tests-docker.sh run models
|
||||
```
|
||||
|
||||
### 2. Service Tests (`test_services.py`)
|
||||
Tests business logic layer:
|
||||
- Number generation service
|
||||
- Financial calculation service
|
||||
- Document service workflows
|
||||
- Validation service
|
||||
|
||||
```bash
|
||||
./run-tests-docker.sh run services
|
||||
```
|
||||
|
||||
### 3. Authentication Tests (`test_authentication.py`)
|
||||
Tests authentication system:
|
||||
- Multiple authentication backends
|
||||
- Permission decorators and middleware
|
||||
- User management workflows
|
||||
- Security features
|
||||
|
||||
```bash
|
||||
./run-tests-docker.sh run auth
|
||||
```
|
||||
|
||||
### 4. View Tests (`test_views.py`)
|
||||
Tests web interface:
|
||||
- CRUD operations for all entities
|
||||
- AJAX endpoints
|
||||
- Permission enforcement
|
||||
- URL routing
|
||||
|
||||
```bash
|
||||
./run-tests-docker.sh run views
|
||||
```
|
||||
|
||||
### 5. PDF Tests (`test_pdf.py`)
|
||||
Tests PDF generation:
|
||||
- WeasyPrint and ReportLab engines
|
||||
- Template rendering
|
||||
- Document formatting
|
||||
- Security and performance
|
||||
|
||||
```bash
|
||||
./run-tests-docker.sh run pdf
|
||||
```
|
||||
|
||||
### 6. Integration Tests (`test_integration.py`)
|
||||
Tests complete workflows:
|
||||
- End-to-end business processes
|
||||
- Multi-user collaboration
|
||||
- System integration scenarios
|
||||
- Performance and security
|
||||
|
||||
```bash
|
||||
./run-tests-docker.sh run integration
|
||||
```
|
||||
|
||||
## Test Reports and Coverage
|
||||
|
||||
### Coverage Reports
|
||||
|
||||
After running tests with coverage, reports are available in:
|
||||
- **HTML Report**: `./coverage-reports/html/index.html`
|
||||
- **XML Report**: `./coverage-reports/coverage.xml`
|
||||
- **Console**: Displayed after test run
|
||||
|
||||
```bash
|
||||
# Run tests with coverage
|
||||
./run-tests-docker.sh coverage
|
||||
|
||||
# View HTML report
|
||||
open coverage-reports/html/index.html
|
||||
```
|
||||
|
||||
### Test Artifacts
|
||||
|
||||
Test outputs are saved to:
|
||||
- **Test Reports**: `./test-reports/`
|
||||
- **Coverage Reports**: `./coverage-reports/`
|
||||
- **Logs**: `./logs/`
|
||||
- **PDF Test Files**: `./test-reports/pdf/`
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
The test environment uses these key variables:
|
||||
|
||||
```yaml
|
||||
# Database configuration
|
||||
DATABASE_HOST: test-db
|
||||
DATABASE_NAME: test_cmc
|
||||
DATABASE_USER: test_cmc
|
||||
DATABASE_PASSWORD: testPassword123
|
||||
|
||||
# Django settings
|
||||
DJANGO_SETTINGS_MODULE: cmcsales.settings
|
||||
TESTING: 1
|
||||
DEBUG: 0
|
||||
|
||||
# PDF generation
|
||||
PDF_GENERATION_ENGINE: weasyprint
|
||||
PDF_SAVE_DIRECTORY: /app/test-reports/pdf
|
||||
```
|
||||
|
||||
### Test Database
|
||||
|
||||
- **Isolated database** separate from development/production
|
||||
- **Runs on port 3307** to avoid conflicts
|
||||
- **Optimized for testing** with reduced buffer sizes
|
||||
- **Automatically reset** between test runs (unless `--keepdb` used)
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
### Parallel Test Execution
|
||||
|
||||
```bash
|
||||
# Run tests in parallel (faster execution)
|
||||
docker-compose -f docker-compose.test.yml run --rm cmc-django-test \
|
||||
/app/scripts/run-tests.sh --parallel=4 all
|
||||
```
|
||||
|
||||
### Keeping Test Database
|
||||
|
||||
```bash
|
||||
# Keep database between runs for faster subsequent tests
|
||||
./run-tests-docker.sh run models --keepdb
|
||||
```
|
||||
|
||||
### Quick Test Suite
|
||||
|
||||
```bash
|
||||
# Run only essential tests for rapid feedback
|
||||
./run-tests-docker.sh quick
|
||||
```
|
||||
|
||||
## Continuous Integration
|
||||
|
||||
### GitHub Actions Example
|
||||
|
||||
```yaml
|
||||
name: Test CMC Django
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
|
||||
- name: Setup test environment
|
||||
run: ./run-tests-docker.sh setup
|
||||
|
||||
- name: Run tests with coverage
|
||||
run: ./run-tests-docker.sh coverage
|
||||
|
||||
- name: Upload coverage reports
|
||||
uses: codecov/codecov-action@v3
|
||||
with:
|
||||
file: ./coverage-reports/coverage.xml
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Database Connection Issues
|
||||
|
||||
```bash
|
||||
# Check database status
|
||||
docker-compose -f docker-compose.test.yml ps
|
||||
|
||||
# View database logs
|
||||
docker-compose -f docker-compose.test.yml logs test-db
|
||||
|
||||
# Restart database
|
||||
docker-compose -f docker-compose.test.yml restart test-db
|
||||
```
|
||||
|
||||
### Test Failures
|
||||
|
||||
```bash
|
||||
# Run with maximum verbosity for debugging
|
||||
docker-compose -f docker-compose.test.yml run --rm cmc-django-test \
|
||||
python cmcsales/manage.py test cmc.tests.test_models --verbosity=3
|
||||
|
||||
# Use failfast to stop on first error
|
||||
./run-tests-docker.sh run models --failfast
|
||||
|
||||
# Open shell to investigate
|
||||
./run-tests-docker.sh shell
|
||||
```
|
||||
|
||||
### Permission Issues
|
||||
|
||||
```bash
|
||||
# Fix file permissions
|
||||
sudo chown -R $USER:$USER test-reports coverage-reports logs
|
||||
|
||||
# Check Docker permissions
|
||||
docker-compose -f docker-compose.test.yml run --rm cmc-django-test whoami
|
||||
```
|
||||
|
||||
### Memory Issues
|
||||
|
||||
```bash
|
||||
# Run tests with reduced parallel workers
|
||||
docker-compose -f docker-compose.test.yml run --rm cmc-django-test \
|
||||
/app/scripts/run-tests.sh --parallel=1 all
|
||||
|
||||
# Monitor resource usage
|
||||
docker stats
|
||||
```
|
||||
|
||||
## Development Workflow
|
||||
|
||||
### Recommended Testing Workflow
|
||||
|
||||
1. **Initial Setup** (one-time):
|
||||
```bash
|
||||
./run-tests-docker.sh setup
|
||||
```
|
||||
|
||||
2. **During Development** (fast feedback):
|
||||
```bash
|
||||
./run-tests-docker.sh quick --keepdb
|
||||
```
|
||||
|
||||
3. **Before Commit** (comprehensive):
|
||||
```bash
|
||||
./run-tests-docker.sh coverage
|
||||
```
|
||||
|
||||
4. **Debugging Issues**:
|
||||
```bash
|
||||
./run-tests-docker.sh shell
|
||||
# Inside container:
|
||||
python cmcsales/manage.py test cmc.tests.test_models.CustomerModelTest.test_customer_creation --verbosity=3
|
||||
```
|
||||
|
||||
### Adding New Tests
|
||||
|
||||
1. Create test file in appropriate module
|
||||
2. Follow existing test patterns and base classes
|
||||
3. Test locally:
|
||||
```bash
|
||||
./run-tests-docker.sh run models --keepdb
|
||||
```
|
||||
4. Run full suite before committing:
|
||||
```bash
|
||||
./run-tests-docker.sh coverage
|
||||
```
|
||||
|
||||
## Integration with IDE
|
||||
|
||||
### PyCharm/IntelliJ
|
||||
|
||||
Configure remote interpreter using Docker:
|
||||
1. Go to Settings → Project → Python Interpreter
|
||||
2. Add Docker Compose interpreter
|
||||
3. Use `docker-compose.test.yml` configuration
|
||||
4. Set service to `cmc-django-test`
|
||||
|
||||
### VS Code
|
||||
|
||||
Use Dev Containers extension:
|
||||
1. Create `.devcontainer/devcontainer.json`
|
||||
2. Configure to use test Docker environment
|
||||
3. Run tests directly in integrated terminal
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Always run tests in Docker** for consistency
|
||||
2. **Use `--keepdb` during development** for speed
|
||||
3. **Run coverage reports before commits**
|
||||
4. **Clean up regularly** to free disk space
|
||||
5. **Monitor test performance** and optimize slow tests
|
||||
6. **Use parallel execution** for large test suites
|
||||
7. **Keep test data realistic** but minimal
|
||||
8. **Test error conditions** as well as happy paths
|
||||
|
||||
## Resources
|
||||
|
||||
- **Django Testing Documentation**: https://docs.djangoproject.com/en/5.1/topics/testing/
|
||||
- **Coverage.py Documentation**: https://coverage.readthedocs.io/
|
||||
- **Docker Compose Reference**: https://docs.docker.com/compose/
|
||||
- **CMC Test Suite Documentation**: See individual test modules for detailed information
|
||||
|
|
@ -43,4 +43,6 @@
|
|||
*
|
||||
*/
|
||||
//EOF
|
||||
|
||||
require_once(dirname(__FILE__) . '/php7_compat.php');
|
||||
?>
|
||||
|
|
@ -168,6 +168,28 @@ Configure::write('Security.salt', 'uiPxR3MzVXAID5zucbxLdxP4TX33buPoCWZr4JfroGoaE
|
|||
Configure::write('Acl.classname', 'DbAcl');
|
||||
Configure::write('Acl.database', 'default');
|
||||
|
||||
/**
|
||||
* Tailscale Authentication Configuration
|
||||
*
|
||||
* Enable Tailscale HTTP header authentication support
|
||||
* When enabled, the system will check for Tailscale authentication headers
|
||||
* before falling back to HTTP Basic Auth
|
||||
*/
|
||||
Configure::write('Tailscale.enabled', true);
|
||||
|
||||
/**
|
||||
* Auto-create users from Tailscale authentication
|
||||
* When enabled, users authenticated via Tailscale headers will be
|
||||
* automatically created if they don't exist in the database
|
||||
*/
|
||||
Configure::write('Tailscale.autoCreateUsers', false);
|
||||
|
||||
/**
|
||||
* Default access level for auto-created Tailscale users
|
||||
* Options: 'user', 'manager', 'admin'
|
||||
*/
|
||||
Configure::write('Tailscale.defaultAccessLevel', 'user');
|
||||
|
||||
|
||||
|
||||
|
||||
|
|
|
|||
96
app/config/php7_compat.php
Normal file
96
app/config/php7_compat.php
Normal file
|
|
@ -0,0 +1,96 @@
|
|||
<?php
|
||||
/**
|
||||
* PHP 7 Compatibility layer for CakePHP 1.2.5
|
||||
* Include this file in your bootstrap.php
|
||||
*/
|
||||
|
||||
// Fix for deprecated $HTTP_RAW_POST_DATA
|
||||
if (!isset($HTTP_RAW_POST_DATA)) {
|
||||
$HTTP_RAW_POST_DATA = file_get_contents('php://input');
|
||||
}
|
||||
|
||||
// Replace deprecated ereg functions
|
||||
if (!function_exists('ereg')) {
|
||||
function ereg($pattern, $string, &$regs = array()) {
|
||||
return preg_match('~' . $pattern . '~', $string, $regs);
|
||||
}
|
||||
}
|
||||
|
||||
if (!function_exists('eregi')) {
|
||||
function eregi($pattern, $string, &$regs = array()) {
|
||||
return preg_match('~' . $pattern . '~i', $string, $regs);
|
||||
}
|
||||
}
|
||||
|
||||
if (!function_exists('ereg_replace')) {
|
||||
function ereg_replace($pattern, $replacement, $string) {
|
||||
return preg_replace('~' . $pattern . '~', $replacement, $string);
|
||||
}
|
||||
}
|
||||
|
||||
if (!function_exists('eregi_replace')) {
|
||||
function eregi_replace($pattern, $replacement, $string) {
|
||||
return preg_replace('~' . $pattern . '~i', $replacement, $string);
|
||||
}
|
||||
}
|
||||
|
||||
if (!function_exists('split')) {
|
||||
function split($pattern, $string, $limit = -1) {
|
||||
return preg_split('~' . $pattern . '~', $string, $limit);
|
||||
}
|
||||
}
|
||||
|
||||
// Fix for mysql_* functions if needed
|
||||
if (!function_exists('mysql_connect')) {
|
||||
function mysql_connect($server, $username, $password, $new_link = false, $client_flags = 0) {
|
||||
return mysqli_connect($server, $username, $password);
|
||||
}
|
||||
|
||||
function mysql_select_db($database_name, $link_identifier = null) {
|
||||
return mysqli_select_db($link_identifier, $database_name);
|
||||
}
|
||||
|
||||
function mysql_query($query, $link_identifier = null) {
|
||||
return mysqli_query($link_identifier, $query);
|
||||
}
|
||||
|
||||
function mysql_fetch_array($result, $result_type = MYSQLI_BOTH) {
|
||||
return mysqli_fetch_array($result, $result_type);
|
||||
}
|
||||
|
||||
function mysql_fetch_assoc($result) {
|
||||
return mysqli_fetch_assoc($result);
|
||||
}
|
||||
|
||||
function mysql_fetch_row($result) {
|
||||
return mysqli_fetch_row($result);
|
||||
}
|
||||
|
||||
function mysql_num_rows($result) {
|
||||
return mysqli_num_rows($result);
|
||||
}
|
||||
|
||||
function mysql_affected_rows($link_identifier = null) {
|
||||
return mysqli_affected_rows($link_identifier);
|
||||
}
|
||||
|
||||
function mysql_insert_id($link_identifier = null) {
|
||||
return mysqli_insert_id($link_identifier);
|
||||
}
|
||||
|
||||
function mysql_close($link_identifier = null) {
|
||||
return mysqli_close($link_identifier);
|
||||
}
|
||||
|
||||
function mysql_error($link_identifier = null) {
|
||||
return mysqli_error($link_identifier);
|
||||
}
|
||||
|
||||
function mysql_errno($link_identifier = null) {
|
||||
return mysqli_errno($link_identifier);
|
||||
}
|
||||
|
||||
function mysql_real_escape_string($unescaped_string, $link_identifier = null) {
|
||||
return mysqli_real_escape_string($link_identifier, $unescaped_string);
|
||||
}
|
||||
}
|
||||
|
|
@ -8,10 +8,115 @@ class AppController extends Controller {
|
|||
|
||||
var $uses = array('User');
|
||||
var $helpers = array('Javascript', 'Time', 'Html', 'Form');
|
||||
|
||||
// Define public actions that don't require authentication
|
||||
var $allowedActions = array();
|
||||
|
||||
function beforeFilter() {
|
||||
|
||||
// Find the user that matches the HTTP basic auth user
|
||||
$user = $this->User->find('first', array('recursive' => 0, 'conditions' => array('User.username'=>$_SERVER["PHP_AUTH_USER"])));
|
||||
// Check if current action is allowed without authentication
|
||||
if (in_array($this->action, $this->allowedActions)) {
|
||||
error_log('[AUTH_BYPASS] Action ' . $this->action . ' allowed without authentication');
|
||||
return;
|
||||
}
|
||||
|
||||
$user = null;
|
||||
|
||||
// Check if Tailscale authentication is enabled
|
||||
if (Configure::read('Tailscale.enabled')) {
|
||||
error_log('[WEBAUTH] Checking web authentication headers');
|
||||
error_log('X-Webauth-User: ' . (isset($_SERVER['HTTP_X_WEBAUTH_USER']) ? $_SERVER['HTTP_X_WEBAUTH_USER'] : 'not set'));
|
||||
error_log('X-Webauth-Name: ' . (isset($_SERVER['HTTP_X_WEBAUTH_NAME']) ? $_SERVER['HTTP_X_WEBAUTH_NAME'] : 'not set'));
|
||||
// Check for web authentication headers
|
||||
$tailscaleLogin = isset($_SERVER['HTTP_X_WEBAUTH_USER']) ? $_SERVER['HTTP_X_WEBAUTH_USER'] : null;
|
||||
$tailscaleName = isset($_SERVER['HTTP_X_WEBAUTH_NAME']) ? $_SERVER['HTTP_X_WEBAUTH_NAME'] : null;
|
||||
|
||||
if ($tailscaleLogin) {
|
||||
// Log web authentication attempt
|
||||
error_log('[WEBAUTH] Attempting authentication for: ' . $tailscaleLogin);
|
||||
|
||||
// Try to find user by email address from web auth header
|
||||
$user = $this->User->find('first', array(
|
||||
'recursive' => 0,
|
||||
'conditions' => array('User.email' => $tailscaleLogin)
|
||||
));
|
||||
|
||||
// If user not found and auto-creation is enabled, create a new user
|
||||
if (!$user && Configure::read('Tailscale.autoCreateUsers')) {
|
||||
// Parse the name
|
||||
$firstName = '';
|
||||
$lastName = '';
|
||||
if ($tailscaleName) {
|
||||
$nameParts = explode(' ', $tailscaleName);
|
||||
$firstName = $nameParts[0];
|
||||
if (count($nameParts) > 1) {
|
||||
array_shift($nameParts);
|
||||
$lastName = implode(' ', $nameParts);
|
||||
}
|
||||
}
|
||||
|
||||
$userData = array(
|
||||
'User' => array(
|
||||
'email' => $tailscaleLogin,
|
||||
'username' => $tailscaleLogin,
|
||||
'first_name' => $firstName,
|
||||
'last_name' => $lastName,
|
||||
'type' => 'user',
|
||||
'access_level' => Configure::read('Tailscale.defaultAccessLevel'),
|
||||
'enabled' => 1,
|
||||
'by_vault' => 0
|
||||
)
|
||||
);
|
||||
$this->User->create();
|
||||
if ($this->User->save($userData)) {
|
||||
$user = $this->User->find('first', array(
|
||||
'recursive' => 0,
|
||||
'conditions' => array('User.id' => $this->User->id)
|
||||
));
|
||||
error_log('[WEBAUTH] Created new user: ' . $tailscaleLogin);
|
||||
} else {
|
||||
error_log('[WEBAUTH] Failed to create user: ' . $tailscaleLogin);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Fall back to HTTP basic auth if no Tailscale auth or user not found
|
||||
if (!$user && isset($_SERVER["PHP_AUTH_USER"])) {
|
||||
error_log('[BASIC_AUTH] Attempting authentication for: ' . $_SERVER["PHP_AUTH_USER"]);
|
||||
$user = $this->User->find('first', array(
|
||||
'recursive' => 0,
|
||||
'conditions' => array('User.username' => $_SERVER["PHP_AUTH_USER"])
|
||||
));
|
||||
}
|
||||
|
||||
if ($user) {
|
||||
error_log('[AUTH_SUCCESS] User authenticated: ' . $user['User']['email']);
|
||||
} else {
|
||||
error_log('[AUTH_FAILED] No valid authentication found');
|
||||
|
||||
// Check if we have any authentication attempt (Web Auth or Basic Auth)
|
||||
$hasAuthAttempt = (Configure::read('Tailscale.enabled') && isset($_SERVER['HTTP_X_WEBAUTH_USER'])) ||
|
||||
isset($_SERVER["PHP_AUTH_USER"]);
|
||||
|
||||
// If there was an authentication attempt but it failed, return 401
|
||||
if ($hasAuthAttempt) {
|
||||
header('HTTP/1.1 401 Unauthorized');
|
||||
header('Content-Type: text/plain');
|
||||
echo "Authentication failed. Invalid credentials or user not found.";
|
||||
error_log('[AUTH_FAILED] Returning 401 Unauthorized');
|
||||
exit();
|
||||
}
|
||||
|
||||
// If no authentication headers at all, request authentication
|
||||
header('WWW-Authenticate: Basic realm="CMC Sales System"');
|
||||
header('HTTP/1.1 401 Unauthorized');
|
||||
header('Content-Type: text/plain');
|
||||
echo "Authentication required. Please provide valid credentials.";
|
||||
error_log('[AUTH_FAILED] No authentication headers, requesting authentication');
|
||||
exit();
|
||||
}
|
||||
|
||||
$this->set("currentuser", $user);
|
||||
|
||||
if($this->RequestHandler->isAjax()) {
|
||||
|
|
@ -52,7 +157,29 @@ class AppController extends Controller {
|
|||
* @return array - the currently logged in user.
|
||||
*/
|
||||
function getCurrentUser() {
|
||||
$user = $this->User->find('first', array('recursive' => 0, 'conditions' => array('User.username'=>$_SERVER["PHP_AUTH_USER"])));
|
||||
$user = null;
|
||||
|
||||
// Check if Tailscale authentication is enabled
|
||||
if (Configure::read('Tailscale.enabled')) {
|
||||
$tailscaleLogin = isset($_SERVER['HTTP_X_WEBAUTH_USER']) ? $_SERVER['HTTP_X_WEBAUTH_USER'] : null;
|
||||
|
||||
if ($tailscaleLogin) {
|
||||
// Try to find user by email address from web auth header
|
||||
$user = $this->User->find('first', array(
|
||||
'recursive' => 0,
|
||||
'conditions' => array('User.email' => $tailscaleLogin)
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
// Fall back to HTTP basic auth if no Tailscale auth or user not found
|
||||
if (!$user && isset($_SERVER["PHP_AUTH_USER"])) {
|
||||
$user = $this->User->find('first', array(
|
||||
'recursive' => 0,
|
||||
'conditions' => array('User.username' => $_SERVER["PHP_AUTH_USER"])
|
||||
));
|
||||
}
|
||||
|
||||
return $user;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -1,10 +1,11 @@
|
|||
<VirtualHost *:80>
|
||||
ServerName localhost
|
||||
DocumentRoot /var/www/cmc-sales/app/webroot
|
||||
DirectoryIndex index.php
|
||||
|
||||
<Directory /var/www/cmc-sales/app/webroot>
|
||||
AllowOverride All
|
||||
Require all granted
|
||||
</Directory>
|
||||
# Send Apache logs to stdout/stderr for Docker
|
||||
ErrorLog /dev/stderr
|
||||
CustomLog /dev/stdout combined
|
||||
|
||||
# Ensure PHP errors are also logged
|
||||
php_flag log_errors on
|
||||
php_value error_log /dev/stderr
|
||||
</VirtualHost>
|
||||
152
conf/nginx-production.conf
Normal file
152
conf/nginx-production.conf
Normal file
|
|
@ -0,0 +1,152 @@
|
|||
# Production environment configuration
|
||||
upstream cmc_php_production {
|
||||
server cmc-php-production:80;
|
||||
keepalive 32;
|
||||
}
|
||||
|
||||
upstream cmc_go_production {
|
||||
server cmc-go-production:8080;
|
||||
keepalive 32;
|
||||
}
|
||||
|
||||
# Rate limiting
|
||||
limit_req_zone $binary_remote_addr zone=login:10m rate=5r/m;
|
||||
limit_req_zone $binary_remote_addr zone=api:10m rate=30r/m;
|
||||
|
||||
server {
|
||||
server_name cmc.springupsoftware.com;
|
||||
|
||||
# Basic auth for production
|
||||
auth_basic_user_file /etc/nginx/userpasswd;
|
||||
auth_basic "CMC Sales - Restricted Access";
|
||||
|
||||
# Security headers
|
||||
add_header X-Frame-Options DENY;
|
||||
add_header X-Content-Type-Options nosniff;
|
||||
add_header X-XSS-Protection "1; mode=block";
|
||||
add_header Referrer-Policy "strict-origin-when-cross-origin";
|
||||
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
|
||||
|
||||
# Hide server information
|
||||
server_tokens off;
|
||||
|
||||
# Request size limits
|
||||
client_max_body_size 50M;
|
||||
client_body_timeout 30s;
|
||||
client_header_timeout 30s;
|
||||
|
||||
# Compression
|
||||
gzip on;
|
||||
gzip_vary on;
|
||||
gzip_min_length 1024;
|
||||
gzip_types text/plain text/css text/xml text/javascript application/javascript application/xml+rss application/json;
|
||||
|
||||
# CakePHP legacy app routes
|
||||
location / {
|
||||
limit_req zone=api burst=10 nodelay;
|
||||
|
||||
proxy_pass http://cmc_php_production;
|
||||
proxy_read_timeout 300s;
|
||||
proxy_connect_timeout 10s;
|
||||
proxy_send_timeout 30s;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
# Buffer settings for better performance
|
||||
proxy_buffering on;
|
||||
proxy_buffer_size 128k;
|
||||
proxy_buffers 4 256k;
|
||||
proxy_busy_buffers_size 256k;
|
||||
}
|
||||
|
||||
# Go API routes
|
||||
location /api/ {
|
||||
limit_req zone=api burst=20 nodelay;
|
||||
|
||||
proxy_pass http://cmc_go_production;
|
||||
proxy_read_timeout 300s;
|
||||
proxy_connect_timeout 10s;
|
||||
proxy_send_timeout 30s;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
# Buffer settings for better performance
|
||||
proxy_buffering on;
|
||||
proxy_buffer_size 128k;
|
||||
proxy_buffers 4 256k;
|
||||
proxy_busy_buffers_size 256k;
|
||||
}
|
||||
|
||||
# Go page routes for emails
|
||||
location ~ ^/(emails|customers|products|purchase-orders|enquiries|documents) {
|
||||
limit_req zone=api burst=15 nodelay;
|
||||
|
||||
proxy_pass http://cmc_go_production;
|
||||
proxy_read_timeout 300s;
|
||||
proxy_connect_timeout 10s;
|
||||
proxy_send_timeout 30s;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
# Buffer settings for better performance
|
||||
proxy_buffering on;
|
||||
proxy_buffer_size 128k;
|
||||
proxy_buffers 4 256k;
|
||||
proxy_busy_buffers_size 256k;
|
||||
}
|
||||
|
||||
# Static files from Go app with aggressive caching
|
||||
location /static/ {
|
||||
proxy_pass http://cmc_go_production;
|
||||
proxy_cache_valid 200 24h;
|
||||
add_header Cache-Control "public, max-age=86400";
|
||||
expires 1d;
|
||||
}
|
||||
|
||||
# PDF files with caching
|
||||
location /pdf/ {
|
||||
proxy_pass http://cmc_go_production;
|
||||
proxy_cache_valid 200 1h;
|
||||
add_header Cache-Control "public, max-age=3600";
|
||||
expires 1h;
|
||||
}
|
||||
|
||||
# Health check endpoints (no rate limiting)
|
||||
location /health {
|
||||
proxy_pass http://cmc_go_production/api/v1/health;
|
||||
access_log off;
|
||||
}
|
||||
|
||||
# Block common attack patterns
|
||||
location ~ /\. {
|
||||
deny all;
|
||||
access_log off;
|
||||
log_not_found off;
|
||||
}
|
||||
|
||||
location ~ ~$ {
|
||||
deny all;
|
||||
access_log off;
|
||||
log_not_found off;
|
||||
}
|
||||
|
||||
# Error pages
|
||||
error_page 502 503 504 /50x.html;
|
||||
location = /50x.html {
|
||||
root /usr/share/nginx/html;
|
||||
}
|
||||
|
||||
# Custom error page for rate limiting
|
||||
error_page 429 /429.html;
|
||||
location = /429.html {
|
||||
root /usr/share/nginx/html;
|
||||
}
|
||||
|
||||
listen 80;
|
||||
}
|
||||
137
conf/nginx-proxy.conf
Normal file
137
conf/nginx-proxy.conf
Normal file
|
|
@ -0,0 +1,137 @@
|
|||
user nginx;
|
||||
worker_processes auto;
|
||||
error_log /var/log/nginx/error.log warn;
|
||||
pid /var/run/nginx.pid;
|
||||
|
||||
events {
|
||||
worker_connections 1024;
|
||||
use epoll;
|
||||
multi_accept on;
|
||||
}
|
||||
|
||||
http {
|
||||
include /etc/nginx/mime.types;
|
||||
default_type application/octet-stream;
|
||||
|
||||
# Logging
|
||||
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
|
||||
'$status $body_bytes_sent "$http_referer" '
|
||||
'"$http_user_agent" "$http_x_forwarded_for"';
|
||||
|
||||
access_log /var/log/nginx/access.log main;
|
||||
|
||||
# Performance
|
||||
sendfile on;
|
||||
tcp_nopush on;
|
||||
tcp_nodelay on;
|
||||
keepalive_timeout 65;
|
||||
types_hash_max_size 2048;
|
||||
server_tokens off;
|
||||
|
||||
# Gzip
|
||||
gzip on;
|
||||
gzip_vary on;
|
||||
gzip_min_length 1024;
|
||||
gzip_types text/plain text/css text/xml text/javascript application/javascript application/xml+rss application/json;
|
||||
|
||||
# Rate limiting
|
||||
limit_req_zone $binary_remote_addr zone=global:10m rate=10r/s;
|
||||
|
||||
# SSL configuration
|
||||
ssl_protocols TLSv1.2 TLSv1.3;
|
||||
ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384;
|
||||
ssl_prefer_server_ciphers off;
|
||||
ssl_session_cache shared:SSL:10m;
|
||||
ssl_session_timeout 10m;
|
||||
|
||||
# Upstream servers
|
||||
upstream cmc_staging {
|
||||
server nginx-staging:80;
|
||||
keepalive 32;
|
||||
}
|
||||
|
||||
upstream cmc_production {
|
||||
server nginx-production:80;
|
||||
keepalive 32;
|
||||
}
|
||||
|
||||
# Redirect HTTP to HTTPS
|
||||
server {
|
||||
listen 80;
|
||||
server_name cmc.springupsoftware.com staging.cmc.springupsoftware.com;
|
||||
|
||||
# ACME challenge for Lego
|
||||
location /.well-known/acme-challenge/ {
|
||||
root /var/www/acme-challenge;
|
||||
try_files $uri =404;
|
||||
}
|
||||
|
||||
# Redirect all other traffic to HTTPS
|
||||
location / {
|
||||
return 301 https://$server_name$request_uri;
|
||||
}
|
||||
}
|
||||
|
||||
# Production HTTPS
|
||||
server {
|
||||
listen 443 ssl http2;
|
||||
server_name cmc.springupsoftware.com;
|
||||
|
||||
ssl_certificate /etc/ssl/certs/cmc.springupsoftware.com.crt;
|
||||
ssl_certificate_key /etc/ssl/certs/cmc.springupsoftware.com.key;
|
||||
|
||||
# Security headers
|
||||
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
|
||||
add_header X-Frame-Options DENY;
|
||||
add_header X-Content-Type-Options nosniff;
|
||||
add_header X-XSS-Protection "1; mode=block";
|
||||
|
||||
# Rate limiting
|
||||
limit_req zone=global burst=20 nodelay;
|
||||
|
||||
location / {
|
||||
proxy_pass http://cmc_production;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
# Buffer settings
|
||||
proxy_buffering on;
|
||||
proxy_buffer_size 128k;
|
||||
proxy_buffers 4 256k;
|
||||
proxy_busy_buffers_size 256k;
|
||||
}
|
||||
}
|
||||
|
||||
# Staging HTTPS
|
||||
server {
|
||||
listen 443 ssl http2;
|
||||
server_name staging.cmc.springupsoftware.com;
|
||||
|
||||
ssl_certificate /etc/ssl/certs/staging.cmc.springupsoftware.com.crt;
|
||||
ssl_certificate_key /etc/ssl/certs/staging.cmc.springupsoftware.com.key;
|
||||
|
||||
# Security headers (less strict for staging)
|
||||
add_header X-Frame-Options DENY;
|
||||
add_header X-Content-Type-Options nosniff;
|
||||
add_header X-Environment "STAGING";
|
||||
|
||||
# Rate limiting (more lenient for staging)
|
||||
limit_req zone=global burst=50 nodelay;
|
||||
|
||||
location / {
|
||||
proxy_pass http://cmc_staging;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
# Buffer settings
|
||||
proxy_buffering on;
|
||||
proxy_buffer_size 128k;
|
||||
proxy_buffers 4 256k;
|
||||
proxy_busy_buffers_size 256k;
|
||||
}
|
||||
}
|
||||
}
|
||||
89
conf/nginx-staging.conf
Normal file
89
conf/nginx-staging.conf
Normal file
|
|
@ -0,0 +1,89 @@
|
|||
# Staging environment configuration
|
||||
upstream cmc_php_staging {
|
||||
server cmc-php-staging:80;
|
||||
}
|
||||
|
||||
upstream cmc_go_staging {
|
||||
server cmc-go-staging:8080;
|
||||
}
|
||||
|
||||
server {
|
||||
server_name staging.cmc.springupsoftware.com;
|
||||
|
||||
# Basic auth for staging
|
||||
auth_basic_user_file /etc/nginx/userpasswd;
|
||||
auth_basic "CMC Sales Staging - Restricted Access";
|
||||
|
||||
# Security headers
|
||||
add_header X-Frame-Options DENY;
|
||||
add_header X-Content-Type-Options nosniff;
|
||||
add_header X-XSS-Protection "1; mode=block";
|
||||
add_header Referrer-Policy "strict-origin-when-cross-origin";
|
||||
|
||||
# Staging banner
|
||||
add_header X-Environment "STAGING";
|
||||
|
||||
# CakePHP legacy app routes
|
||||
location / {
|
||||
proxy_pass http://cmc_php_staging;
|
||||
proxy_read_timeout 300s;
|
||||
proxy_connect_timeout 10s;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header X-Environment "staging";
|
||||
}
|
||||
|
||||
# Go API routes
|
||||
location /api/ {
|
||||
proxy_pass http://cmc_go_staging;
|
||||
proxy_read_timeout 300s;
|
||||
proxy_connect_timeout 10s;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header X-Environment "staging";
|
||||
}
|
||||
|
||||
# Go page routes for emails
|
||||
location ~ ^/(emails|customers|products|purchase-orders|enquiries|documents) {
|
||||
proxy_pass http://cmc_go_staging;
|
||||
proxy_read_timeout 300s;
|
||||
proxy_connect_timeout 10s;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header X-Environment "staging";
|
||||
}
|
||||
|
||||
# Static files from Go app
|
||||
location /static/ {
|
||||
proxy_pass http://cmc_go_staging;
|
||||
proxy_cache_valid 200 1h;
|
||||
add_header Cache-Control "public, max-age=3600";
|
||||
}
|
||||
|
||||
# PDF files
|
||||
location /pdf/ {
|
||||
proxy_pass http://cmc_go_staging;
|
||||
proxy_cache_valid 200 1h;
|
||||
add_header Cache-Control "public, max-age=3600";
|
||||
}
|
||||
|
||||
# Health check endpoints
|
||||
location /health {
|
||||
proxy_pass http://cmc_go_staging/api/v1/health;
|
||||
access_log off;
|
||||
}
|
||||
|
||||
# Error pages
|
||||
error_page 502 503 504 /50x.html;
|
||||
location = /50x.html {
|
||||
root /usr/share/nginx/html;
|
||||
}
|
||||
|
||||
listen 80;
|
||||
}
|
||||
|
|
@ -633,7 +633,8 @@ html_errors = Off
|
|||
; empty.
|
||||
; http://php.net/error-log
|
||||
; Example:
|
||||
error_log = /var/log/php_errors.log
|
||||
; For Docker: Send errors to stderr so they appear in docker logs
|
||||
error_log = /dev/stderr
|
||||
; Log errors to syslog (Event Log on NT, not valid in Windows 95).
|
||||
;error_log = syslog
|
||||
|
||||
|
|
|
|||
203
deploy-production.sh
Executable file
203
deploy-production.sh
Executable file
|
|
@ -0,0 +1,203 @@
|
|||
#!/bin/bash
|
||||
|
||||
# Production Deployment Script for CMC Sales
|
||||
# This script deploys the application to sales.cmctechnologies.com.au
|
||||
# Based on .gitlab-ci.yml deployment steps
|
||||
|
||||
set -e # Exit on error
|
||||
|
||||
# Color codes for output
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Configuration
|
||||
PRODUCTION_HOST="cmc@sales.cmctechnologies.com.au"
|
||||
PRODUCTION_DIR="/home/cmc/cmc-sales"
|
||||
CURRENT_BRANCH=$(git branch --show-current)
|
||||
|
||||
echo -e "${GREEN}========================================${NC}"
|
||||
echo -e "${GREEN}CMC Sales Production Deployment Script${NC}"
|
||||
echo -e "${GREEN}========================================${NC}"
|
||||
echo ""
|
||||
|
||||
# Check if we're on master branch
|
||||
if [ "$CURRENT_BRANCH" != "master" ]; then
|
||||
echo -e "${YELLOW}Warning: You are not on the master branch.${NC}"
|
||||
echo -e "${YELLOW}Current branch: $CURRENT_BRANCH${NC}"
|
||||
read -p "Do you want to continue deployment from $CURRENT_BRANCH? (y/N): " confirm
|
||||
if [ "$confirm" != "y" ] && [ "$confirm" != "Y" ]; then
|
||||
echo -e "${RED}Deployment cancelled.${NC}"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# Check for uncommitted changes
|
||||
if ! git diff-index --quiet HEAD --; then
|
||||
echo -e "${YELLOW}Warning: You have uncommitted changes.${NC}"
|
||||
git status --short
|
||||
read -p "Do you want to continue? (y/N): " confirm
|
||||
if [ "$confirm" != "y" ] && [ "$confirm" != "Y" ]; then
|
||||
echo -e "${RED}Deployment cancelled.${NC}"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# Get latest commit hash for build arg
|
||||
COMMIT_HASH=$(git rev-parse --short HEAD)
|
||||
echo -e "${GREEN}Deploying commit: $COMMIT_HASH${NC}"
|
||||
echo ""
|
||||
|
||||
# Push latest changes to origin
|
||||
echo -e "${YELLOW}Step 1: Pushing latest changes to origin...${NC}"
|
||||
git push origin $CURRENT_BRANCH
|
||||
if [ $? -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ Changes pushed successfully${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Failed to push changes${NC}"
|
||||
exit 1
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# SSH to production and execute deployment
|
||||
echo -e "${YELLOW}Step 2: Connecting to production server...${NC}"
|
||||
echo -e "${YELLOW}Executing deployment on $PRODUCTION_HOST${NC}"
|
||||
echo ""
|
||||
|
||||
ssh -T $PRODUCTION_HOST << 'ENDSSH'
|
||||
set -e
|
||||
|
||||
# Color codes for remote output
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m'
|
||||
|
||||
echo -e "${GREEN}Connected to production server${NC}"
|
||||
echo ""
|
||||
|
||||
# Navigate to project directory
|
||||
echo -e "${YELLOW}Step 3: Navigating to project directory...${NC}"
|
||||
cd /home/cmc/cmc-sales
|
||||
pwd
|
||||
echo ""
|
||||
|
||||
# Pull latest changes
|
||||
echo -e "${YELLOW}Step 4: Pulling latest changes from Git...${NC}"
|
||||
git pull origin master
|
||||
if [ $? -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ Git pull successful${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Git pull failed${NC}"
|
||||
exit 1
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Get commit hash for build
|
||||
COMMIT_HASH=$(git rev-parse --short HEAD)
|
||||
echo -e "${GREEN}Building from commit: $COMMIT_HASH${NC}"
|
||||
|
||||
# Copy userpasswd file
|
||||
echo -e "${YELLOW}Step 5: Copying userpasswd file...${NC}"
|
||||
cp /home/cmc/cmc-sales/userpasswd /home/cmc/userpasswd
|
||||
if [ $? -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ userpasswd file copied${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Failed to copy userpasswd file${NC}"
|
||||
exit 1
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Build Docker image
|
||||
echo -e "${YELLOW}Step 6: Building Docker image...${NC}"
|
||||
docker build --build-arg=COMMIT=$COMMIT_HASH . -t "cmc:latest"
|
||||
if [ $? -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ Docker image built successfully${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Docker build failed${NC}"
|
||||
exit 1
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Stop existing container
|
||||
echo -e "${YELLOW}Step 7: Stopping existing container...${NC}"
|
||||
export ID=$(docker ps -q --filter ancestor=cmc:latest)
|
||||
if [ ! -z "$ID" ]; then
|
||||
docker kill $ID
|
||||
echo -e "${GREEN}✓ Existing container stopped${NC}"
|
||||
sleep 1
|
||||
else
|
||||
echo -e "${YELLOW}No existing container found${NC}"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Run new container
|
||||
echo -e "${YELLOW}Step 8: Starting new container...${NC}"
|
||||
docker run -d --restart always -p 127.0.0.1:8888:80 \
|
||||
--mount type=bind,source=/mnt/vault/pdf,target=/var/www/cmc-sales/app/webroot/pdf \
|
||||
--mount type=bind,source=/mnt/vault/attachments_files,target=/var/www/cmc-sales/app/webroot/attachments_files \
|
||||
--mount type=bind,source=/mnt/vault/emails,target=/var/www/emails \
|
||||
--mount type=bind,source=/mnt/vault/vaultmsgs,target=/var/www/vaultmsgs \
|
||||
cmc:latest
|
||||
|
||||
if [ $? -eq 0 ]; then
|
||||
echo -e "${GREEN}✓ New container started successfully${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Failed to start new container${NC}"
|
||||
exit 1
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Verify container is running
|
||||
echo -e "${YELLOW}Step 9: Verifying deployment...${NC}"
|
||||
sleep 2
|
||||
NEW_ID=$(docker ps -q --filter ancestor=cmc:latest)
|
||||
if [ ! -z "$NEW_ID" ]; then
|
||||
echo -e "${GREEN}✓ Container is running with ID: $NEW_ID${NC}"
|
||||
docker ps --filter ancestor=cmc:latest --format "table {{.ID}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"
|
||||
else
|
||||
echo -e "${RED}✗ Container is not running!${NC}"
|
||||
exit 1
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Show recent logs
|
||||
echo -e "${YELLOW}Step 10: Recent container logs:${NC}"
|
||||
docker logs --tail 20 $NEW_ID
|
||||
echo ""
|
||||
|
||||
echo -e "${GREEN}========================================${NC}"
|
||||
echo -e "${GREEN}✓ Deployment completed successfully!${NC}"
|
||||
echo -e "${GREEN}========================================${NC}"
|
||||
|
||||
ENDSSH
|
||||
|
||||
if [ $? -eq 0 ]; then
|
||||
echo ""
|
||||
echo -e "${GREEN}========================================${NC}"
|
||||
echo -e "${GREEN}✓ Production deployment successful!${NC}"
|
||||
echo -e "${GREEN}========================================${NC}"
|
||||
echo ""
|
||||
echo -e "${GREEN}Application is running at:${NC}"
|
||||
echo -e "${GREEN} Internal: http://127.0.0.1:8888${NC}"
|
||||
echo -e "${GREEN} External: https://sales.cmctechnologies.com.au${NC}"
|
||||
echo ""
|
||||
echo -e "${YELLOW}To view live logs:${NC}"
|
||||
echo " ssh $PRODUCTION_HOST 'docker logs -f \$(docker ps -q --filter ancestor=cmc:latest)'"
|
||||
echo ""
|
||||
echo -e "${YELLOW}To rollback if needed:${NC}"
|
||||
echo " ssh $PRODUCTION_HOST 'docker run -d --restart always -p 127.0.0.1:8888:80 [previous-image-id]'"
|
||||
else
|
||||
echo ""
|
||||
echo -e "${RED}========================================${NC}"
|
||||
echo -e "${RED}✗ Deployment failed!${NC}"
|
||||
echo -e "${RED}========================================${NC}"
|
||||
echo ""
|
||||
echo -e "${YELLOW}To check the status:${NC}"
|
||||
echo " ssh $PRODUCTION_HOST 'docker ps -a'"
|
||||
echo ""
|
||||
echo -e "${YELLOW}To view logs:${NC}"
|
||||
echo " ssh $PRODUCTION_HOST 'docker logs \$(docker ps -aq | head -1)'"
|
||||
exit 1
|
||||
fi
|
||||
85
docker-compose.caddy-production.yml
Normal file
85
docker-compose.caddy-production.yml
Normal file
|
|
@ -0,0 +1,85 @@
|
|||
version: '3.8'
|
||||
|
||||
services:
|
||||
cmc-php-production:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
platform: linux/amd64
|
||||
container_name: cmc-php-production
|
||||
depends_on:
|
||||
- db-production
|
||||
ports:
|
||||
- "127.0.0.1:8093:80" # Only accessible from localhost
|
||||
volumes:
|
||||
- production_pdf_data:/var/www/cmc-sales/app/webroot/pdf
|
||||
- production_attachments_data:/var/www/cmc-sales/app/webroot/attachments_files
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- APP_ENV=production
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '2.0'
|
||||
memory: 2G
|
||||
reservations:
|
||||
cpus: '0.5'
|
||||
memory: 512M
|
||||
|
||||
db-production:
|
||||
image: mariadb:latest
|
||||
container_name: cmc-db-production
|
||||
environment:
|
||||
MYSQL_ROOT_PASSWORD: ${DB_ROOT_PASSWORD_PRODUCTION}
|
||||
MYSQL_DATABASE: cmc
|
||||
MYSQL_USER: cmc
|
||||
MYSQL_PASSWORD: ${DB_PASSWORD_PRODUCTION}
|
||||
volumes:
|
||||
- production_db_data:/var/lib/mysql
|
||||
- ./backups:/backups:ro
|
||||
restart: unless-stopped
|
||||
# No external port exposure for security
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '2.0'
|
||||
memory: 4G
|
||||
reservations:
|
||||
cpus: '0.5'
|
||||
memory: 1G
|
||||
|
||||
cmc-go-production:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile.go.production
|
||||
container_name: cmc-go-production
|
||||
environment:
|
||||
DB_HOST: db-production
|
||||
DB_PORT: 3306
|
||||
DB_USER: cmc
|
||||
DB_PASSWORD: ${DB_PASSWORD_PRODUCTION}
|
||||
DB_NAME: cmc
|
||||
PORT: 8080
|
||||
APP_ENV: production
|
||||
depends_on:
|
||||
db-production:
|
||||
condition: service_started
|
||||
ports:
|
||||
- "127.0.0.1:8094:8080" # Only accessible from localhost
|
||||
volumes:
|
||||
- production_pdf_data:/root/webroot/pdf
|
||||
- ./credentials/production:/root/credentials:ro
|
||||
restart: unless-stopped
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '2.0'
|
||||
memory: 2G
|
||||
reservations:
|
||||
cpus: '0.5'
|
||||
memory: 512M
|
||||
|
||||
volumes:
|
||||
production_db_data:
|
||||
production_pdf_data:
|
||||
production_attachments_data:
|
||||
65
docker-compose.caddy-staging-ubuntu.yml
Normal file
65
docker-compose.caddy-staging-ubuntu.yml
Normal file
|
|
@ -0,0 +1,65 @@
|
|||
version: '3.8'
|
||||
|
||||
services:
|
||||
cmc-php-staging:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile.ubuntu-php
|
||||
platform: linux/amd64
|
||||
container_name: cmc-php-staging
|
||||
depends_on:
|
||||
- db-staging
|
||||
ports:
|
||||
- "127.0.0.1:8091:80"
|
||||
volumes:
|
||||
- ./app:/var/www/cmc-sales/app
|
||||
- staging_pdf_data:/var/www/cmc-sales/app/webroot/pdf
|
||||
- staging_attachments_data:/var/www/cmc-sales/app/webroot/attachments_files
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- APP_ENV=staging
|
||||
- DB_HOST=db-staging
|
||||
- DB_NAME=cmc_staging
|
||||
- DB_USER=cmc_staging
|
||||
- DB_PASSWORD=${DB_PASSWORD_STAGING:-staging_password}
|
||||
|
||||
db-staging:
|
||||
image: mariadb:10.11
|
||||
container_name: cmc-db-staging
|
||||
environment:
|
||||
MYSQL_ROOT_PASSWORD: ${DB_ROOT_PASSWORD_STAGING:-root_password}
|
||||
MYSQL_DATABASE: cmc_staging
|
||||
MYSQL_USER: cmc_staging
|
||||
MYSQL_PASSWORD: ${DB_PASSWORD_STAGING:-staging_password}
|
||||
volumes:
|
||||
- staging_db_data:/var/lib/mysql
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:3307:3306"
|
||||
|
||||
cmc-go-staging:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile.go.staging
|
||||
container_name: cmc-go-staging
|
||||
environment:
|
||||
DB_HOST: db-staging
|
||||
DB_PORT: 3306
|
||||
DB_USER: cmc_staging
|
||||
DB_PASSWORD: ${DB_PASSWORD_STAGING:-staging_password}
|
||||
DB_NAME: cmc_staging
|
||||
PORT: 8080
|
||||
APP_ENV: staging
|
||||
depends_on:
|
||||
- db-staging
|
||||
ports:
|
||||
- "127.0.0.1:8092:8080"
|
||||
volumes:
|
||||
- staging_pdf_data:/root/webroot/pdf
|
||||
- ./credentials/staging:/root/credentials:ro
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
staging_db_data:
|
||||
staging_pdf_data:
|
||||
staging_attachments_data:
|
||||
61
docker-compose.caddy-staging.yml
Normal file
61
docker-compose.caddy-staging.yml
Normal file
|
|
@ -0,0 +1,61 @@
|
|||
version: '3.8'
|
||||
|
||||
services:
|
||||
cmc-php-staging:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
platform: linux/amd64
|
||||
container_name: cmc-php-staging
|
||||
depends_on:
|
||||
- db-staging
|
||||
ports:
|
||||
- "127.0.0.1:8091:80" # Only accessible from localhost
|
||||
volumes:
|
||||
- staging_pdf_data:/var/www/cmc-sales/app/webroot/pdf
|
||||
- staging_attachments_data:/var/www/cmc-sales/app/webroot/attachments_files
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- APP_ENV=staging
|
||||
|
||||
db-staging:
|
||||
image: mariadb:latest
|
||||
container_name: cmc-db-staging
|
||||
environment:
|
||||
MYSQL_ROOT_PASSWORD: ${DB_ROOT_PASSWORD_STAGING}
|
||||
MYSQL_DATABASE: cmc_staging
|
||||
MYSQL_USER: cmc_staging
|
||||
MYSQL_PASSWORD: ${DB_PASSWORD_STAGING}
|
||||
volumes:
|
||||
- staging_db_data:/var/lib/mysql
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:3307:3306" # Only accessible from localhost
|
||||
|
||||
cmc-go-staging:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile.go.staging
|
||||
container_name: cmc-go-staging
|
||||
environment:
|
||||
DB_HOST: db-staging
|
||||
DB_PORT: 3306
|
||||
DB_USER: cmc_staging
|
||||
DB_PASSWORD: ${DB_PASSWORD_STAGING}
|
||||
DB_NAME: cmc_staging
|
||||
PORT: 8080
|
||||
APP_ENV: staging
|
||||
depends_on:
|
||||
db-staging:
|
||||
condition: service_started
|
||||
ports:
|
||||
- "127.0.0.1:8092:8080" # Only accessible from localhost
|
||||
volumes:
|
||||
- staging_pdf_data:/root/webroot/pdf
|
||||
- ./credentials/staging:/root/credentials:ro
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
staging_db_data:
|
||||
staging_pdf_data:
|
||||
staging_attachments_data:
|
||||
105
docker-compose.production.yml
Normal file
105
docker-compose.production.yml
Normal file
|
|
@ -0,0 +1,105 @@
|
|||
version: '3.8'
|
||||
|
||||
services:
|
||||
cmc-php-production:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
platform: linux/amd64
|
||||
container_name: cmc-php-production
|
||||
depends_on:
|
||||
- db-production
|
||||
volumes:
|
||||
- production_pdf_data:/var/www/cmc-sales/app/webroot/pdf
|
||||
- production_attachments_data:/var/www/cmc-sales/app/webroot/attachments_files
|
||||
network_mode: bridge
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- APP_ENV=production
|
||||
# Remove development features
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '2.0'
|
||||
memory: 2G
|
||||
reservations:
|
||||
cpus: '0.5'
|
||||
memory: 512M
|
||||
|
||||
nginx-production:
|
||||
image: nginx:latest
|
||||
hostname: nginx-production
|
||||
container_name: cmc-nginx-production
|
||||
ports:
|
||||
- "8080:80" # Internal port for production
|
||||
volumes:
|
||||
- ./conf/nginx-production.conf:/etc/nginx/conf.d/cmc-production.conf
|
||||
- ./userpasswd:/etc/nginx/userpasswd:ro
|
||||
depends_on:
|
||||
- cmc-php-production
|
||||
restart: unless-stopped
|
||||
network_mode: bridge
|
||||
environment:
|
||||
- NGINX_ENVSUBST_TEMPLATE_SUFFIX=.template
|
||||
|
||||
|
||||
|
||||
db-production:
|
||||
image: mariadb:latest
|
||||
container_name: cmc-db-production
|
||||
environment:
|
||||
MYSQL_ROOT_PASSWORD: ${DB_ROOT_PASSWORD_PRODUCTION}
|
||||
MYSQL_DATABASE: cmc
|
||||
MYSQL_USER: cmc
|
||||
MYSQL_PASSWORD: ${DB_PASSWORD_PRODUCTION}
|
||||
volumes:
|
||||
- production_db_data:/var/lib/mysql
|
||||
- ./backups:/backups:ro # Backup restore directory
|
||||
network_mode: bridge
|
||||
restart: unless-stopped
|
||||
# No external port exposure for security
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '2.0'
|
||||
memory: 4G
|
||||
reservations:
|
||||
cpus: '0.5'
|
||||
memory: 1G
|
||||
|
||||
cmc-go-production:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile.go.production
|
||||
container_name: cmc-go-production
|
||||
environment:
|
||||
DB_HOST: db-production
|
||||
DB_PORT: 3306
|
||||
DB_USER: cmc
|
||||
DB_PASSWORD: ${DB_PASSWORD_PRODUCTION}
|
||||
DB_NAME: cmc
|
||||
PORT: 8080
|
||||
APP_ENV: production
|
||||
depends_on:
|
||||
db-production:
|
||||
condition: service_started
|
||||
# No external port exposure - only through nginx
|
||||
volumes:
|
||||
- production_pdf_data:/root/webroot/pdf
|
||||
- ./credentials/production:/root/credentials:ro # Production Gmail credentials
|
||||
network_mode: bridge
|
||||
restart: unless-stopped
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '2.0'
|
||||
memory: 2G
|
||||
reservations:
|
||||
cpus: '0.5'
|
||||
memory: 512M
|
||||
|
||||
volumes:
|
||||
production_db_data:
|
||||
production_pdf_data:
|
||||
production_attachments_data:
|
||||
|
||||
68
docker-compose.proxy.yml
Normal file
68
docker-compose.proxy.yml
Normal file
|
|
@ -0,0 +1,68 @@
|
|||
# Main reverse proxy for both staging and production
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
nginx-proxy:
|
||||
image: nginx:latest
|
||||
container_name: cmc-nginx-proxy
|
||||
ports:
|
||||
- "80:80"
|
||||
- "443:443"
|
||||
volumes:
|
||||
- ./conf/nginx-proxy.conf:/etc/nginx/nginx.conf
|
||||
- lego_certificates:/etc/ssl/certs:ro
|
||||
- lego_acme_challenge:/var/www/acme-challenge:ro
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
- nginx-staging
|
||||
- nginx-production
|
||||
networks:
|
||||
- proxy-network
|
||||
- cmc-staging-network
|
||||
- cmc-production-network
|
||||
|
||||
lego:
|
||||
image: goacme/lego:latest
|
||||
container_name: cmc-lego
|
||||
volumes:
|
||||
- lego_certificates:/data/certificates
|
||||
- lego_accounts:/data/accounts
|
||||
- lego_acme_challenge:/data/acme-challenge
|
||||
- ./scripts:/scripts:ro
|
||||
environment:
|
||||
- LEGO_DISABLE_CNAME=true
|
||||
command: sleep infinity
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- proxy-network
|
||||
|
||||
# Import staging services
|
||||
nginx-staging:
|
||||
extends:
|
||||
file: docker-compose.staging.yml
|
||||
service: nginx-staging
|
||||
networks:
|
||||
- proxy-network
|
||||
- cmc-staging-network
|
||||
|
||||
# Import production services
|
||||
nginx-production:
|
||||
extends:
|
||||
file: docker-compose.production.yml
|
||||
service: nginx-production
|
||||
networks:
|
||||
- proxy-network
|
||||
- cmc-production-network
|
||||
|
||||
volumes:
|
||||
lego_certificates:
|
||||
lego_accounts:
|
||||
lego_acme_challenge:
|
||||
|
||||
networks:
|
||||
proxy-network:
|
||||
driver: bridge
|
||||
cmc-staging-network:
|
||||
external: true
|
||||
cmc-production-network:
|
||||
external: true
|
||||
80
docker-compose.staging.yml
Normal file
80
docker-compose.staging.yml
Normal file
|
|
@ -0,0 +1,80 @@
|
|||
version: '3.8'
|
||||
|
||||
services:
|
||||
|
||||
cmc-php-staging:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
platform: linux/amd64
|
||||
container_name: cmc-php-staging
|
||||
depends_on:
|
||||
- db-staging
|
||||
volumes:
|
||||
- staging_pdf_data:/var/www/cmc-sales/app/webroot/pdf
|
||||
- staging_attachments_data:/var/www/cmc-sales/app/webroot/attachments_files
|
||||
network_mode: bridge
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- APP_ENV=staging
|
||||
|
||||
nginx-staging:
|
||||
image: nginx:latest
|
||||
hostname: nginx-staging
|
||||
container_name: cmc-nginx-staging
|
||||
ports:
|
||||
- "8081:80" # Internal port for staging
|
||||
volumes:
|
||||
- ./conf/nginx-staging.conf:/etc/nginx/conf.d/cmc-staging.conf
|
||||
- ./userpasswd:/etc/nginx/userpasswd:ro
|
||||
depends_on:
|
||||
- cmc-php-staging
|
||||
restart: unless-stopped
|
||||
network_mode: bridge
|
||||
environment:
|
||||
- NGINX_ENVSUBST_TEMPLATE_SUFFIX=.template
|
||||
|
||||
db-staging:
|
||||
image: mariadb:latest
|
||||
container_name: cmc-db-staging
|
||||
environment:
|
||||
MYSQL_ROOT_PASSWORD: ${DB_ROOT_PASSWORD_STAGING}
|
||||
MYSQL_DATABASE: cmc_staging
|
||||
MYSQL_USER: cmc_staging
|
||||
MYSQL_PASSWORD: ${DB_PASSWORD_STAGING}
|
||||
volumes:
|
||||
- staging_db_data:/var/lib/mysql
|
||||
network_mode: bridge
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "3307:3306" # Different port for staging DB access
|
||||
|
||||
cmc-go-staging:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile.go.staging
|
||||
container_name: cmc-go-staging
|
||||
environment:
|
||||
DB_HOST: db-staging
|
||||
DB_PORT: 3306
|
||||
DB_USER: cmc_staging
|
||||
DB_PASSWORD: ${DB_PASSWORD_STAGING}
|
||||
DB_NAME: cmc_staging
|
||||
PORT: 8080
|
||||
APP_ENV: staging
|
||||
depends_on:
|
||||
db-staging:
|
||||
condition: service_started
|
||||
ports:
|
||||
- "8082:8080" # Direct access for testing
|
||||
volumes:
|
||||
- staging_pdf_data:/root/webroot/pdf
|
||||
- ./credentials/staging:/root/credentials:ro # Staging Gmail credentials
|
||||
network_mode: bridge
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
staging_db_data:
|
||||
staging_pdf_data:
|
||||
staging_attachments_data:
|
||||
|
||||
7
go-app/.gitignore
vendored
7
go-app/.gitignore
vendored
|
|
@ -31,3 +31,10 @@ vendor/
|
|||
# OS specific files
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# Goose database migration config
|
||||
goose.env
|
||||
|
||||
# Gmail OAuth credentials - NEVER commit these!
|
||||
credentials.json
|
||||
token.json
|
||||
|
|
|
|||
70
go-app/MIGRATIONS.md
Normal file
70
go-app/MIGRATIONS.md
Normal file
|
|
@ -0,0 +1,70 @@
|
|||
# Database Migrations with Goose
|
||||
|
||||
This document explains how to use goose for database migrations in the CMC Sales Go application.
|
||||
|
||||
## Setup
|
||||
|
||||
1. **Install goose**:
|
||||
```bash
|
||||
make install
|
||||
```
|
||||
|
||||
2. **Configure database connection**:
|
||||
```bash
|
||||
cp goose.env.example goose.env
|
||||
# Edit goose.env with your database credentials
|
||||
```
|
||||
|
||||
## Migration Commands
|
||||
|
||||
### Run Migrations
|
||||
```bash
|
||||
# Run all pending migrations
|
||||
make migrate
|
||||
|
||||
# Check migration status
|
||||
make migrate-status
|
||||
```
|
||||
|
||||
### Rollback Migrations
|
||||
```bash
|
||||
# Rollback the last migration
|
||||
make migrate-down
|
||||
```
|
||||
|
||||
### Create New Migrations
|
||||
```bash
|
||||
# Create a new migration file
|
||||
make migrate-create name=add_new_feature
|
||||
```
|
||||
|
||||
## Migration Structure
|
||||
|
||||
Migrations are stored in `sql/migrations/` and follow this naming convention:
|
||||
- `001_add_gmail_fields.sql`
|
||||
- `002_add_new_feature.sql`
|
||||
|
||||
Each migration file contains:
|
||||
```sql
|
||||
-- +goose Up
|
||||
-- Your upgrade SQL here
|
||||
|
||||
-- +goose Down
|
||||
-- Your rollback SQL here
|
||||
```
|
||||
|
||||
## Configuration Files
|
||||
|
||||
- `goose.env` - Database connection settings (gitignored)
|
||||
- `goose.env.example` - Template for goose.env
|
||||
|
||||
## Current Migrations
|
||||
|
||||
1. **001_add_gmail_fields.sql** - Adds Gmail integration fields to emails and email_attachments tables
|
||||
|
||||
## Tips
|
||||
|
||||
- Always test migrations on a backup database first
|
||||
- Use `make migrate-status` to check current state
|
||||
- Migrations are atomic - if one fails, none are applied
|
||||
- Each migration should be reversible with a corresponding Down section
|
||||
|
|
@ -11,6 +11,7 @@ install: ## Install dependencies
|
|||
go env -w GOPRIVATE=code.springupsoftware.com
|
||||
go mod download
|
||||
go install github.com/sqlc-dev/sqlc/cmd/sqlc@latest
|
||||
go install github.com/pressly/goose/v3/cmd/goose@latest
|
||||
|
||||
.PHONY: sqlc
|
||||
sqlc: ## Generate Go code from SQL queries
|
||||
|
|
@ -19,11 +20,24 @@ sqlc: ## Generate Go code from SQL queries
|
|||
.PHONY: build
|
||||
build: sqlc ## Build the application
|
||||
go build -o bin/server cmd/server/main.go
|
||||
go build -o bin/vault cmd/vault/main.go
|
||||
|
||||
.PHONY: build-server
|
||||
build-server: sqlc ## Build only the server
|
||||
go build -o bin/server cmd/server/main.go
|
||||
|
||||
.PHONY: build-vault
|
||||
build-vault: ## Build only the vault command
|
||||
go build -o bin/vault cmd/vault/main.go
|
||||
|
||||
.PHONY: run
|
||||
run: ## Run the application
|
||||
go run cmd/server/main.go
|
||||
|
||||
.PHONY: run-vault
|
||||
run-vault: ## Run the vault command
|
||||
go run cmd/vault/main.go
|
||||
|
||||
.PHONY: dev
|
||||
dev: sqlc ## Run the application with hot reload (requires air)
|
||||
air
|
||||
|
|
@ -64,3 +78,47 @@ dbshell-root: ## Connect to MariaDB as root user
|
|||
else \
|
||||
docker compose exec -e MYSQL_PWD="$$DB_ROOT_PASSWORD" db mariadb -u root; \
|
||||
fi
|
||||
|
||||
.PHONY: migrate
|
||||
migrate: ## Run database migrations
|
||||
@echo "Running database migrations..."
|
||||
@if [ -f goose.env ]; then \
|
||||
export $$(cat goose.env | xargs) && goose up; \
|
||||
else \
|
||||
echo "Error: goose.env file not found"; \
|
||||
exit 1; \
|
||||
fi
|
||||
|
||||
.PHONY: migrate-down
|
||||
migrate-down: ## Rollback last migration
|
||||
@echo "Rolling back last migration..."
|
||||
@if [ -f goose.env ]; then \
|
||||
export $$(cat goose.env | xargs) && goose down; \
|
||||
else \
|
||||
echo "Error: goose.env file not found"; \
|
||||
exit 1; \
|
||||
fi
|
||||
|
||||
.PHONY: migrate-status
|
||||
migrate-status: ## Show migration status
|
||||
@echo "Migration status:"
|
||||
@if [ -f goose.env ]; then \
|
||||
export $$(cat goose.env | xargs) && goose status; \
|
||||
else \
|
||||
echo "Error: goose.env file not found"; \
|
||||
exit 1; \
|
||||
fi
|
||||
|
||||
.PHONY: migrate-create
|
||||
migrate-create: ## Create a new migration file (use: make migrate-create name=add_new_table)
|
||||
@if [ -z "$(name)" ]; then \
|
||||
echo "Error: Please provide a migration name. Usage: make migrate-create name=add_new_table"; \
|
||||
exit 1; \
|
||||
fi
|
||||
@echo "Creating new migration: $(name)"
|
||||
@if [ -f goose.env ]; then \
|
||||
export $$(cat goose.env | xargs) && goose create $(name) sql; \
|
||||
else \
|
||||
echo "Error: goose.env file not found"; \
|
||||
exit 1; \
|
||||
fi
|
||||
140
go-app/cmd/vault/README.md
Normal file
140
go-app/cmd/vault/README.md
Normal file
|
|
@ -0,0 +1,140 @@
|
|||
# Vault Email Processor - Smart Proxy
|
||||
|
||||
This is a Go rewrite of the PHP vault.php script that processes emails for the CMC Sales system. It now supports three modes: local file processing, Gmail indexing, and HTTP streaming proxy.
|
||||
|
||||
## Key Features
|
||||
|
||||
1. **Gmail Integration**: Index Gmail emails without downloading
|
||||
2. **Smart Proxy**: Stream email content on-demand without storing to disk
|
||||
3. **No ripmime dependency**: Uses the enmime Go library for MIME parsing
|
||||
4. **Better error handling**: Proper error handling and database transactions
|
||||
5. **Type safety**: Strongly typed Go structures
|
||||
6. **Modern email parsing**: Uses enmime for robust email parsing
|
||||
|
||||
## Operating Modes
|
||||
|
||||
### 1. Local Mode (Original functionality)
|
||||
Processes emails from local filesystem directories.
|
||||
|
||||
```bash
|
||||
./vault --mode=local \
|
||||
--emaildir=/var/www/emails \
|
||||
--vaultdir=/var/www/vaultmsgs/new \
|
||||
--processeddir=/var/www/vaultmsgs/cur \
|
||||
--dbhost=127.0.0.1 \
|
||||
--dbuser=cmc \
|
||||
--dbpass="xVRQI&cA?7AU=hqJ!%au" \
|
||||
--dbname=cmc
|
||||
```
|
||||
|
||||
### 2. Gmail Index Mode
|
||||
Indexes Gmail emails without downloading content. Creates database references only.
|
||||
|
||||
```bash
|
||||
./vault --mode=index \
|
||||
--gmail-query="is:unread" \
|
||||
--credentials=credentials.json \
|
||||
--token=token.json \
|
||||
--dbhost=127.0.0.1 \
|
||||
--dbuser=cmc \
|
||||
--dbpass="xVRQI&cA?7AU=hqJ!%au" \
|
||||
--dbname=cmc
|
||||
```
|
||||
|
||||
### 3. HTTP Server Mode
|
||||
Runs an HTTP server that streams Gmail content on-demand.
|
||||
|
||||
```bash
|
||||
./vault --mode=serve \
|
||||
--port=8080 \
|
||||
--credentials=credentials.json \
|
||||
--token=token.json \
|
||||
--dbhost=127.0.0.1 \
|
||||
--dbuser=cmc \
|
||||
--dbpass="xVRQI&cA?7AU=hqJ!%au" \
|
||||
--dbname=cmc
|
||||
```
|
||||
|
||||
## Gmail Setup
|
||||
|
||||
1. Enable Gmail API in Google Cloud Console
|
||||
2. Create OAuth 2.0 credentials
|
||||
3. Download credentials as `credentials.json`
|
||||
4. Run vault in any Gmail mode - it will prompt for authorization
|
||||
5. Token will be saved as `token.json` for future use
|
||||
|
||||
## API Endpoints (Server Mode)
|
||||
|
||||
- `GET /api/emails` - List indexed emails (metadata only)
|
||||
- `GET /api/emails/:id` - Get email metadata
|
||||
- `GET /api/emails/:id/content` - Stream email HTML/text from Gmail
|
||||
- `GET /api/emails/:id/attachments` - List attachment metadata
|
||||
- `GET /api/emails/:id/attachments/:attachmentId` - Stream attachment from Gmail
|
||||
- `GET /api/emails/:id/raw` - Stream raw email (for email clients)
|
||||
|
||||
## Database Schema Changes
|
||||
|
||||
Required migrations for Gmail support:
|
||||
|
||||
```sql
|
||||
ALTER TABLE emails
|
||||
ADD COLUMN gmail_message_id VARCHAR(255) UNIQUE,
|
||||
ADD COLUMN gmail_thread_id VARCHAR(255),
|
||||
ADD COLUMN is_downloaded BOOLEAN DEFAULT FALSE,
|
||||
ADD COLUMN raw_headers TEXT;
|
||||
|
||||
CREATE INDEX idx_gmail_message_id ON emails(gmail_message_id);
|
||||
|
||||
ALTER TABLE email_attachments
|
||||
ADD COLUMN gmail_attachment_id VARCHAR(255),
|
||||
ADD COLUMN gmail_message_id VARCHAR(255);
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
### Smart Proxy Benefits
|
||||
- **No Disk Storage**: Emails/attachments streamed directly from Gmail
|
||||
- **Low Storage Footprint**: Only metadata stored in database
|
||||
- **Fresh Content**: Always serves latest version from Gmail
|
||||
- **Scalable**: No file management overhead
|
||||
- **On-Demand**: Content fetched only when requested
|
||||
|
||||
### Processing Flow
|
||||
1. **Index Mode**: Scans Gmail, stores metadata, creates associations
|
||||
2. **Server Mode**: Receives HTTP requests, fetches from Gmail, streams to client
|
||||
3. **Local Mode**: Original file-based processing (backwards compatible)
|
||||
|
||||
## Build
|
||||
|
||||
```bash
|
||||
go build -o vault cmd/vault/main.go
|
||||
```
|
||||
|
||||
## Dependencies
|
||||
|
||||
- github.com/jhillyerd/enmime - MIME email parsing
|
||||
- github.com/google/uuid - UUID generation
|
||||
- github.com/go-sql-driver/mysql - MySQL driver
|
||||
- github.com/gorilla/mux - HTTP router
|
||||
- golang.org/x/oauth2 - OAuth2 support
|
||||
- google.golang.org/api/gmail/v1 - Gmail API client
|
||||
|
||||
## Database Tables Used
|
||||
|
||||
- emails - Main email records with Gmail metadata
|
||||
- email_recipients - To/CC recipients
|
||||
- email_attachments - Attachment metadata (no file storage)
|
||||
- emails_enquiries - Email to enquiry associations
|
||||
- emails_invoices - Email to invoice associations
|
||||
- emails_purchase_orders - Email to PO associations
|
||||
- emails_jobs - Email to job associations
|
||||
- users - System users
|
||||
- enquiries, invoices, purchase_orders, jobs - For identifier matching
|
||||
|
||||
## Gmail Query Examples
|
||||
|
||||
- `is:unread` - Unread emails
|
||||
- `newer_than:1d` - Emails from last 24 hours
|
||||
- `from:customer@example.com` - From specific sender
|
||||
- `subject:invoice` - Subject contains "invoice"
|
||||
- `has:attachment` - Emails with attachments
|
||||
1300
go-app/cmd/vault/main.go
Normal file
1300
go-app/cmd/vault/main.go
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -6,7 +6,9 @@ toolchain go1.24.3
|
|||
|
||||
require (
|
||||
github.com/go-sql-driver/mysql v1.7.1
|
||||
github.com/google/uuid v1.6.0
|
||||
github.com/gorilla/mux v1.8.1
|
||||
github.com/jhillyerd/enmime v1.3.0
|
||||
github.com/joho/godotenv v1.5.1
|
||||
github.com/jung-kurt/gofpdf v1.16.2
|
||||
golang.org/x/text v0.27.0
|
||||
|
|
|
|||
|
|
@ -1,3 +1,9 @@
|
|||
cloud.google.com/go/auth v0.16.3 h1:kabzoQ9/bobUmnseYnBO6qQG7q4a/CffFRlJSxv2wCc=
|
||||
cloud.google.com/go/auth v0.16.3/go.mod h1:NucRGjaXfzP1ltpcQ7On/VTZ0H4kWB5Jy+Y9Dnm76fA=
|
||||
cloud.google.com/go/auth/oauth2adapt v0.2.8 h1:keo8NaayQZ6wimpNSmW5OPc283g65QNIiLpZnkHRbnc=
|
||||
cloud.google.com/go/auth/oauth2adapt v0.2.8/go.mod h1:XQ9y31RkqZCcwJWNSx2Xvric3RrU88hAYYbjDWYDL+c=
|
||||
cloud.google.com/go/compute/metadata v0.7.0 h1:PBWF+iiAerVNe8UCHxdOt6eHLVc3ydFeOCw78U8ytSU=
|
||||
cloud.google.com/go/compute/metadata v0.7.0/go.mod h1:j5MvL9PprKL39t166CoB1uVHfQMs4tFQZZcKwksXUjo=
|
||||
github.com/boombuler/barcode v1.0.0/go.mod h1:paBWMcWSl3LHKBqUq+rly7CNSldXjb2rDl3JlRe0mD8=
|
||||
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
|
|
@ -10,6 +16,10 @@ github.com/google/uuid v1.4.0 h1:MtMxsa51/r9yyhkyLsVeVt0B+BGQZzpQiTQ4eHZ8bc4=
|
|||
github.com/google/uuid v1.4.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/gorilla/mux v1.8.1 h1:TuBL49tXwgrFYWhqrNgrUNEY92u81SPhu7sTdzQEiWY=
|
||||
github.com/gorilla/mux v1.8.1/go.mod h1:AKf9I4AEqPTmMytcMc0KkNouC66V3BtZ4qD5fmWSiMQ=
|
||||
github.com/jaytaylor/html2text v0.0.0-20230321000545-74c2419ad056 h1:iCHtR9CQyktQ5+f3dMVZfwD2KWJUgm7M0gdL9NGr8KA=
|
||||
github.com/jaytaylor/html2text v0.0.0-20230321000545-74c2419ad056/go.mod h1:CVKlgaMiht+LXvHG173ujK6JUhZXKb2u/BQtjPDIvyk=
|
||||
github.com/jhillyerd/enmime v1.3.0 h1:LV5kzfLidiOr8qRGIpYYmUZCnhrPbcFAnAFUnWn99rw=
|
||||
github.com/jhillyerd/enmime v1.3.0/go.mod h1:6c6jg5HdRRV2FtvVL69LjiX1M8oE0xDX9VEhV3oy4gs=
|
||||
github.com/joho/godotenv v1.5.1 h1:7eLL/+HRGLY0ldzfGMeQkb7vMd0as4CfYvUVzLqw0N0=
|
||||
github.com/joho/godotenv v1.5.1/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4=
|
||||
github.com/jung-kurt/gofpdf v1.0.0/go.mod h1:7Id9E/uU8ce6rXgefFLlgrJj/GYY22cpxn+r32jIOes=
|
||||
|
|
@ -24,6 +34,9 @@ github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
|
|||
github.com/phpdave11/gofpdi v1.0.7/go.mod h1:vBmVV0Do6hSBHC8uKUQ71JGW+ZGQq74llk/7bXwjDoI=
|
||||
github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=
|
||||
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
|
||||
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
|
||||
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/robfig/cron/v3 v3.0.1 h1:WdRxkvbJztn8LMz/QEvLN5sBU+xKpSqwwUO1Pjr4qDs=
|
||||
github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro=
|
||||
|
|
@ -41,6 +54,14 @@ github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o
|
|||
go.uber.org/atomic v1.9.0 h1:ECmE8Bn/WFTYwEW/bpKD3M8VtR/zQVbavAoalC1PYyE=
|
||||
go.uber.org/atomic v1.9.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
|
||||
golang.org/x/image v0.0.0-20190910094157-69e4b8554b2a/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
|
||||
golang.org/x/net v0.42.0 h1:jzkYrhi3YQWD6MLBJcsklgQsoAcw89EcZbJw8Z614hs=
|
||||
golang.org/x/net v0.42.0/go.mod h1:FF1RA5d3u7nAYA4z2TkclSCKh68eSXtiFwcWQpPXdt8=
|
||||
golang.org/x/oauth2 v0.30.0 h1:dnDm7JmhM45NNpd8FDDeLhK6FwqbOf4MLCM9zb1BOHI=
|
||||
golang.org/x/oauth2 v0.30.0/go.mod h1:B++QgG3ZKulg6sRPGD/mqlHQs5rB3Ml9erfeDY7xKlU=
|
||||
golang.org/x/sync v0.16.0 h1:ycBJEhp9p4vXvUZNszeOq0kGTPghopOL8q0fq3vstxw=
|
||||
golang.org/x/sync v0.16.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
|
||||
golang.org/x/sys v0.34.0 h1:H5Y5sJ2L2JRdyv7ROF1he/lPdvFsd0mJHFw2ThKHxLA=
|
||||
golang.org/x/sys v0.34.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
|
||||
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||
golang.org/x/text v0.27.0 h1:4fGWRpyh641NLlecmyl4LOe6yDdfaYNrGb2zdfo4JV4=
|
||||
golang.org/x/text v0.27.0/go.mod h1:1D28KMCvyooCX9hBiosv5Tz/+YLxj0j7XhWjpSUF7CU=
|
||||
|
|
|
|||
3
go-app/goose.env.example
Normal file
3
go-app/goose.env.example
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
GOOSE_DRIVER=mysql
|
||||
GOOSE_DBSTRING=username:password@tcp(localhost:3306)/database?parseTime=true
|
||||
GOOSE_MIGRATION_DIR=sql/migrations
|
||||
870
go-app/internal/cmc/handlers/emails.go
Normal file
870
go-app/internal/cmc/handlers/emails.go
Normal file
|
|
@ -0,0 +1,870 @@
|
|||
package handlers
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io/ioutil"
|
||||
"net/http"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"code.springupsoftware.com/cmc/cmc-sales/internal/cmc/db"
|
||||
"github.com/gorilla/mux"
|
||||
"github.com/jhillyerd/enmime"
|
||||
"golang.org/x/oauth2"
|
||||
"golang.org/x/oauth2/google"
|
||||
"google.golang.org/api/gmail/v1"
|
||||
"google.golang.org/api/option"
|
||||
)
|
||||
|
||||
type EmailHandler struct {
|
||||
queries *db.Queries
|
||||
db *sql.DB
|
||||
gmailService *gmail.Service
|
||||
}
|
||||
|
||||
type EmailResponse struct {
|
||||
ID int32 `json:"id"`
|
||||
Subject string `json:"subject"`
|
||||
UserID int32 `json:"user_id"`
|
||||
Created time.Time `json:"created"`
|
||||
GmailMessageID *string `json:"gmail_message_id,omitempty"`
|
||||
AttachmentCount int32 `json:"attachment_count"`
|
||||
IsDownloaded *bool `json:"is_downloaded,omitempty"`
|
||||
}
|
||||
|
||||
type EmailDetailResponse struct {
|
||||
ID int32 `json:"id"`
|
||||
Subject string `json:"subject"`
|
||||
UserID int32 `json:"user_id"`
|
||||
Created time.Time `json:"created"`
|
||||
GmailMessageID *string `json:"gmail_message_id,omitempty"`
|
||||
GmailThreadID *string `json:"gmail_thread_id,omitempty"`
|
||||
RawHeaders *string `json:"raw_headers,omitempty"`
|
||||
IsDownloaded *bool `json:"is_downloaded,omitempty"`
|
||||
Enquiries []int32 `json:"enquiries,omitempty"`
|
||||
Invoices []int32 `json:"invoices,omitempty"`
|
||||
PurchaseOrders []int32 `json:"purchase_orders,omitempty"`
|
||||
Jobs []int32 `json:"jobs,omitempty"`
|
||||
}
|
||||
|
||||
type EmailAttachmentResponse struct {
|
||||
ID int32 `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Size int32 `json:"size"`
|
||||
Filename string `json:"filename"`
|
||||
IsMessageBody bool `json:"is_message_body"`
|
||||
GmailAttachmentID *string `json:"gmail_attachment_id,omitempty"`
|
||||
Created time.Time `json:"created"`
|
||||
}
|
||||
|
||||
func NewEmailHandler(queries *db.Queries, database *sql.DB) *EmailHandler {
|
||||
// Try to initialize Gmail service
|
||||
gmailService, err := getGmailService("credentials.json", "token.json")
|
||||
if err != nil {
|
||||
// Log the error but continue without Gmail service
|
||||
fmt.Printf("Warning: Gmail service not available: %v\n", err)
|
||||
}
|
||||
|
||||
return &EmailHandler{
|
||||
queries: queries,
|
||||
db: database,
|
||||
gmailService: gmailService,
|
||||
}
|
||||
}
|
||||
|
||||
// List emails with pagination and filtering
|
||||
func (h *EmailHandler) List(w http.ResponseWriter, r *http.Request) {
|
||||
// Parse query parameters
|
||||
limitStr := r.URL.Query().Get("limit")
|
||||
offsetStr := r.URL.Query().Get("offset")
|
||||
search := r.URL.Query().Get("search")
|
||||
userID := r.URL.Query().Get("user_id")
|
||||
|
||||
// Set defaults
|
||||
limit := 50
|
||||
offset := 0
|
||||
|
||||
if limitStr != "" {
|
||||
if l, err := strconv.Atoi(limitStr); err == nil && l > 0 && l <= 100 {
|
||||
limit = l
|
||||
}
|
||||
}
|
||||
|
||||
if offsetStr != "" {
|
||||
if o, err := strconv.Atoi(offsetStr); err == nil && o >= 0 {
|
||||
offset = o
|
||||
}
|
||||
}
|
||||
|
||||
// Build query
|
||||
query := `
|
||||
SELECT e.id, e.subject, e.user_id, e.created, e.gmail_message_id, e.email_attachment_count, e.is_downloaded
|
||||
FROM emails e`
|
||||
|
||||
var args []interface{}
|
||||
var conditions []string
|
||||
|
||||
if search != "" {
|
||||
conditions = append(conditions, "e.subject LIKE ?")
|
||||
args = append(args, "%"+search+"%")
|
||||
}
|
||||
|
||||
if userID != "" {
|
||||
conditions = append(conditions, "e.user_id = ?")
|
||||
args = append(args, userID)
|
||||
}
|
||||
|
||||
if len(conditions) > 0 {
|
||||
query += " WHERE " + joinConditions(conditions, " AND ")
|
||||
}
|
||||
|
||||
query += " ORDER BY e.id DESC LIMIT ? OFFSET ?"
|
||||
args = append(args, limit, offset)
|
||||
|
||||
rows, err := h.db.Query(query, args...)
|
||||
if err != nil {
|
||||
http.Error(w, fmt.Sprintf("Database error: %v", err), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var emails []EmailResponse
|
||||
for rows.Next() {
|
||||
var email EmailResponse
|
||||
var gmailMessageID sql.NullString
|
||||
var isDownloaded sql.NullBool
|
||||
|
||||
err := rows.Scan(
|
||||
&email.ID,
|
||||
&email.Subject,
|
||||
&email.UserID,
|
||||
&email.Created,
|
||||
&gmailMessageID,
|
||||
&email.AttachmentCount,
|
||||
&isDownloaded,
|
||||
)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
if gmailMessageID.Valid {
|
||||
email.GmailMessageID = &gmailMessageID.String
|
||||
}
|
||||
if isDownloaded.Valid {
|
||||
email.IsDownloaded = &isDownloaded.Bool
|
||||
}
|
||||
|
||||
emails = append(emails, email)
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(emails)
|
||||
}
|
||||
|
||||
// Get a specific email with details
|
||||
func (h *EmailHandler) Get(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
emailID, err := strconv.Atoi(vars["id"])
|
||||
if err != nil {
|
||||
http.Error(w, "Invalid email ID", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
// Get email details
|
||||
query := `
|
||||
SELECT e.id, e.subject, e.user_id, e.created, e.gmail_message_id,
|
||||
e.gmail_thread_id, e.raw_headers, e.is_downloaded
|
||||
FROM emails e
|
||||
WHERE e.id = ?`
|
||||
|
||||
var email EmailDetailResponse
|
||||
var gmailMessageID, gmailThreadID, rawHeaders sql.NullString
|
||||
var isDownloaded sql.NullBool
|
||||
|
||||
err = h.db.QueryRow(query, emailID).Scan(
|
||||
&email.ID,
|
||||
&email.Subject,
|
||||
&email.UserID,
|
||||
&email.Created,
|
||||
&gmailMessageID,
|
||||
&gmailThreadID,
|
||||
&rawHeaders,
|
||||
&isDownloaded,
|
||||
)
|
||||
|
||||
if err != nil {
|
||||
if err == sql.ErrNoRows {
|
||||
http.Error(w, "Email not found", http.StatusNotFound)
|
||||
} else {
|
||||
http.Error(w, fmt.Sprintf("Database error: %v", err), http.StatusInternalServerError)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
if gmailMessageID.Valid {
|
||||
email.GmailMessageID = &gmailMessageID.String
|
||||
}
|
||||
if gmailThreadID.Valid {
|
||||
email.GmailThreadID = &gmailThreadID.String
|
||||
}
|
||||
if rawHeaders.Valid {
|
||||
email.RawHeaders = &rawHeaders.String
|
||||
}
|
||||
if isDownloaded.Valid {
|
||||
email.IsDownloaded = &isDownloaded.Bool
|
||||
}
|
||||
|
||||
// Get associated enquiries
|
||||
enquiryRows, err := h.db.Query("SELECT enquiry_id FROM emails_enquiries WHERE email_id = ?", emailID)
|
||||
if err == nil {
|
||||
defer enquiryRows.Close()
|
||||
for enquiryRows.Next() {
|
||||
var enquiryID int32
|
||||
if enquiryRows.Scan(&enquiryID) == nil {
|
||||
email.Enquiries = append(email.Enquiries, enquiryID)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Get associated invoices
|
||||
invoiceRows, err := h.db.Query("SELECT invoice_id FROM emails_invoices WHERE email_id = ?", emailID)
|
||||
if err == nil {
|
||||
defer invoiceRows.Close()
|
||||
for invoiceRows.Next() {
|
||||
var invoiceID int32
|
||||
if invoiceRows.Scan(&invoiceID) == nil {
|
||||
email.Invoices = append(email.Invoices, invoiceID)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Get associated purchase orders
|
||||
poRows, err := h.db.Query("SELECT purchase_order_id FROM emails_purchase_orders WHERE email_id = ?", emailID)
|
||||
if err == nil {
|
||||
defer poRows.Close()
|
||||
for poRows.Next() {
|
||||
var poID int32
|
||||
if poRows.Scan(&poID) == nil {
|
||||
email.PurchaseOrders = append(email.PurchaseOrders, poID)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Get associated jobs
|
||||
jobRows, err := h.db.Query("SELECT job_id FROM emails_jobs WHERE email_id = ?", emailID)
|
||||
if err == nil {
|
||||
defer jobRows.Close()
|
||||
for jobRows.Next() {
|
||||
var jobID int32
|
||||
if jobRows.Scan(&jobID) == nil {
|
||||
email.Jobs = append(email.Jobs, jobID)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(email)
|
||||
}
|
||||
|
||||
// List attachments for an email
|
||||
func (h *EmailHandler) ListAttachments(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
emailID, err := strconv.Atoi(vars["id"])
|
||||
if err != nil {
|
||||
http.Error(w, "Invalid email ID", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
// First check if attachments are already in database
|
||||
query := `
|
||||
SELECT id, name, type, size, filename, is_message_body, gmail_attachment_id, created
|
||||
FROM email_attachments
|
||||
WHERE email_id = ?
|
||||
ORDER BY is_message_body DESC, created ASC`
|
||||
|
||||
rows, err := h.db.Query(query, emailID)
|
||||
if err != nil {
|
||||
http.Error(w, fmt.Sprintf("Database error: %v", err), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var attachments []EmailAttachmentResponse
|
||||
hasStoredAttachments := false
|
||||
|
||||
for rows.Next() {
|
||||
hasStoredAttachments = true
|
||||
var attachment EmailAttachmentResponse
|
||||
var gmailAttachmentID sql.NullString
|
||||
|
||||
err := rows.Scan(
|
||||
&attachment.ID,
|
||||
&attachment.Name,
|
||||
&attachment.Type,
|
||||
&attachment.Size,
|
||||
&attachment.Filename,
|
||||
&attachment.IsMessageBody,
|
||||
&gmailAttachmentID,
|
||||
&attachment.Created,
|
||||
)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
if gmailAttachmentID.Valid {
|
||||
attachment.GmailAttachmentID = &gmailAttachmentID.String
|
||||
}
|
||||
|
||||
attachments = append(attachments, attachment)
|
||||
}
|
||||
|
||||
// If no stored attachments and this is a Gmail email, try to fetch from Gmail
|
||||
if !hasStoredAttachments && h.gmailService != nil {
|
||||
// Get Gmail message ID
|
||||
var gmailMessageID sql.NullString
|
||||
err := h.db.QueryRow("SELECT gmail_message_id FROM emails WHERE id = ?", emailID).Scan(&gmailMessageID)
|
||||
|
||||
if err == nil && gmailMessageID.Valid {
|
||||
// Fetch message metadata from Gmail
|
||||
message, err := h.gmailService.Users.Messages.Get("me", gmailMessageID.String).
|
||||
Format("FULL").Do()
|
||||
|
||||
if err == nil && message.Payload != nil {
|
||||
// Extract attachment info from Gmail message
|
||||
attachmentIndex := int32(1)
|
||||
h.extractGmailAttachments(message.Payload, &attachments, &attachmentIndex)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Check if this is an HTMX request
|
||||
if r.Header.Get("HX-Request") == "true" {
|
||||
// Return HTML for HTMX
|
||||
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||
|
||||
if len(attachments) == 0 {
|
||||
// No attachments found
|
||||
html := `<div class="notification is-light">
|
||||
<p class="has-text-grey">No attachments found for this email.</p>
|
||||
</div>`
|
||||
w.Write([]byte(html))
|
||||
return
|
||||
}
|
||||
|
||||
// Build HTML table for attachments
|
||||
var htmlBuilder strings.Builder
|
||||
htmlBuilder.WriteString(`<div class="box">
|
||||
<h3 class="title is-5">Attachments</h3>
|
||||
<div class="table-container">
|
||||
<table class="table is-fullwidth">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Name</th>
|
||||
<th>Type</th>
|
||||
<th>Size</th>
|
||||
<th>Actions</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>`)
|
||||
|
||||
for _, att := range attachments {
|
||||
icon := `<i class="fas fa-paperclip"></i>`
|
||||
if att.IsMessageBody {
|
||||
icon = `<i class="fas fa-envelope"></i>`
|
||||
}
|
||||
|
||||
downloadURL := fmt.Sprintf("/api/v1/emails/%d/attachments/%d", emailID, att.ID)
|
||||
if att.GmailAttachmentID != nil {
|
||||
downloadURL = fmt.Sprintf("/api/v1/emails/%d/attachments/%d/stream", emailID, att.ID)
|
||||
}
|
||||
|
||||
htmlBuilder.WriteString(fmt.Sprintf(`
|
||||
<tr>
|
||||
<td>
|
||||
<span class="icon has-text-grey">%s</span>
|
||||
%s
|
||||
</td>
|
||||
<td><span class="tag is-light">%s</span></td>
|
||||
<td>%d bytes</td>
|
||||
<td>
|
||||
<a href="%s" target="_blank" class="button is-small is-info">
|
||||
<span class="icon"><i class="fas fa-download"></i></span>
|
||||
<span>Download</span>
|
||||
</a>
|
||||
</td>
|
||||
</tr>`, icon, att.Name, att.Type, att.Size, downloadURL))
|
||||
}
|
||||
|
||||
htmlBuilder.WriteString(`
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>`)
|
||||
|
||||
w.Write([]byte(htmlBuilder.String()))
|
||||
return
|
||||
}
|
||||
|
||||
// Return JSON for API requests
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(attachments)
|
||||
}
|
||||
|
||||
// Helper function to extract attachment info from Gmail message parts
|
||||
func (h *EmailHandler) extractGmailAttachments(part *gmail.MessagePart, attachments *[]EmailAttachmentResponse, index *int32) {
|
||||
// Check if this part is an attachment
|
||||
// Some attachments may not have filenames or may be inline
|
||||
if part.Body != nil && part.Body.AttachmentId != "" {
|
||||
filename := part.Filename
|
||||
if filename == "" {
|
||||
// Try to generate a filename from content type
|
||||
switch part.MimeType {
|
||||
case "application/pdf":
|
||||
filename = "attachment.pdf"
|
||||
case "image/png":
|
||||
filename = "image.png"
|
||||
case "image/jpeg":
|
||||
filename = "image.jpg"
|
||||
case "text/plain":
|
||||
filename = "text.txt"
|
||||
default:
|
||||
filename = "attachment"
|
||||
}
|
||||
}
|
||||
|
||||
attachment := EmailAttachmentResponse{
|
||||
ID: *index,
|
||||
Name: filename,
|
||||
Type: part.MimeType,
|
||||
Size: int32(part.Body.Size),
|
||||
Filename: filename,
|
||||
IsMessageBody: false,
|
||||
GmailAttachmentID: &part.Body.AttachmentId,
|
||||
Created: time.Now(), // Use current time as placeholder
|
||||
}
|
||||
*attachments = append(*attachments, attachment)
|
||||
*index++
|
||||
}
|
||||
|
||||
// Process sub-parts
|
||||
for _, subPart := range part.Parts {
|
||||
h.extractGmailAttachments(subPart, attachments, index)
|
||||
}
|
||||
}
|
||||
|
||||
// Search emails
|
||||
func (h *EmailHandler) Search(w http.ResponseWriter, r *http.Request) {
|
||||
query := r.URL.Query().Get("q")
|
||||
if query == "" {
|
||||
http.Error(w, "Search query is required", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
// Parse optional parameters
|
||||
limitStr := r.URL.Query().Get("limit")
|
||||
limit := 20
|
||||
|
||||
if limitStr != "" {
|
||||
if l, err := strconv.Atoi(limitStr); err == nil && l > 0 && l <= 100 {
|
||||
limit = l
|
||||
}
|
||||
}
|
||||
|
||||
// Search in subjects and headers
|
||||
sqlQuery := `
|
||||
SELECT e.id, e.subject, e.user_id, e.created, e.gmail_message_id, e.email_attachment_count, e.is_downloaded
|
||||
FROM emails e
|
||||
WHERE e.subject LIKE ? OR e.raw_headers LIKE ?
|
||||
ORDER BY e.id DESC
|
||||
LIMIT ?`
|
||||
|
||||
searchTerm := "%" + query + "%"
|
||||
rows, err := h.db.Query(sqlQuery, searchTerm, searchTerm, limit)
|
||||
if err != nil {
|
||||
http.Error(w, fmt.Sprintf("Database error: %v", err), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var emails []EmailResponse
|
||||
for rows.Next() {
|
||||
var email EmailResponse
|
||||
var gmailMessageID sql.NullString
|
||||
var isDownloaded sql.NullBool
|
||||
|
||||
err := rows.Scan(
|
||||
&email.ID,
|
||||
&email.Subject,
|
||||
&email.UserID,
|
||||
&email.Created,
|
||||
&gmailMessageID,
|
||||
&email.AttachmentCount,
|
||||
&isDownloaded,
|
||||
)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
if gmailMessageID.Valid {
|
||||
email.GmailMessageID = &gmailMessageID.String
|
||||
}
|
||||
if isDownloaded.Valid {
|
||||
email.IsDownloaded = &isDownloaded.Bool
|
||||
}
|
||||
|
||||
emails = append(emails, email)
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(emails)
|
||||
}
|
||||
|
||||
// Stream email content from Gmail
|
||||
func (h *EmailHandler) StreamContent(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
emailID, err := strconv.Atoi(vars["id"])
|
||||
if err != nil {
|
||||
http.Error(w, "Invalid email ID", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
// Get email details to check if it's a Gmail email
|
||||
query := `
|
||||
SELECT e.gmail_message_id, e.subject, e.created, e.user_id
|
||||
FROM emails e
|
||||
WHERE e.id = ?`
|
||||
|
||||
var gmailMessageID sql.NullString
|
||||
var subject string
|
||||
var created time.Time
|
||||
var userID int32
|
||||
|
||||
err = h.db.QueryRow(query, emailID).Scan(&gmailMessageID, &subject, &created, &userID)
|
||||
if err != nil {
|
||||
if err == sql.ErrNoRows {
|
||||
http.Error(w, "Email not found", http.StatusNotFound)
|
||||
} else {
|
||||
http.Error(w, fmt.Sprintf("Database error: %v", err), http.StatusInternalServerError)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
if !gmailMessageID.Valid {
|
||||
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||
html := `
|
||||
<div class="notification is-warning">
|
||||
<strong>Local Email</strong><br>
|
||||
This email is not from Gmail and does not have stored content available for display.
|
||||
</div>`
|
||||
w.Write([]byte(html))
|
||||
return
|
||||
}
|
||||
|
||||
// Check for stored message body content in attachments
|
||||
attachmentQuery := `
|
||||
SELECT id, name, type, size
|
||||
FROM email_attachments
|
||||
WHERE email_id = ? AND is_message_body = 1
|
||||
ORDER BY created ASC`
|
||||
|
||||
attachmentRows, err := h.db.Query(attachmentQuery, emailID)
|
||||
if err == nil {
|
||||
defer attachmentRows.Close()
|
||||
if attachmentRows.Next() {
|
||||
var attachmentID int32
|
||||
var name, attachmentType string
|
||||
var size int32
|
||||
|
||||
if attachmentRows.Scan(&attachmentID, &name, &attachmentType, &size) == nil {
|
||||
// Found stored message body - would normally read the content from file storage
|
||||
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||
html := fmt.Sprintf(`
|
||||
<div class="content">
|
||||
<div class="notification is-success is-light">
|
||||
<strong>Stored Email Content</strong><br>
|
||||
Message body is stored locally as attachment: %s (%s, %d bytes)
|
||||
</div>
|
||||
<div class="box">
|
||||
<p><em>Content would be loaded from local storage here.</em></p>
|
||||
<p>Attachment ID: %d</p>
|
||||
</div>
|
||||
</div>`, name, attachmentType, size, attachmentID)
|
||||
w.Write([]byte(html))
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Try to fetch from Gmail if service is available
|
||||
if h.gmailService != nil {
|
||||
// Fetch from Gmail
|
||||
message, err := h.gmailService.Users.Messages.Get("me", gmailMessageID.String).
|
||||
Format("RAW").Do()
|
||||
if err != nil {
|
||||
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||
html := fmt.Sprintf(`
|
||||
<div class="notification is-danger">
|
||||
<strong>Gmail API Error</strong><br>
|
||||
Failed to fetch email from Gmail: %v
|
||||
</div>`, err)
|
||||
w.Write([]byte(html))
|
||||
return
|
||||
}
|
||||
|
||||
// Decode message
|
||||
rawEmail, err := base64.URLEncoding.DecodeString(message.Raw)
|
||||
if err != nil {
|
||||
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||
html := fmt.Sprintf(`
|
||||
<div class="notification is-danger">
|
||||
<strong>Decode Error</strong><br>
|
||||
Failed to decode email: %v
|
||||
</div>`, err)
|
||||
w.Write([]byte(html))
|
||||
return
|
||||
}
|
||||
|
||||
// Parse with enmime
|
||||
env, err := enmime.ReadEnvelope(bytes.NewReader(rawEmail))
|
||||
if err != nil {
|
||||
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||
html := fmt.Sprintf(`
|
||||
<div class="notification is-danger">
|
||||
<strong>Parse Error</strong><br>
|
||||
Failed to parse email: %v
|
||||
</div>`, err)
|
||||
w.Write([]byte(html))
|
||||
return
|
||||
}
|
||||
|
||||
// Stream HTML or Text directly to client
|
||||
if env.HTML != "" {
|
||||
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||
w.Write([]byte(env.HTML))
|
||||
} else if env.Text != "" {
|
||||
// Convert plain text to HTML for better display
|
||||
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||
html := fmt.Sprintf(`
|
||||
<div class="content">
|
||||
<div class="notification is-info is-light">
|
||||
<strong>Plain Text Email</strong><br>
|
||||
This email contains only plain text content.
|
||||
</div>
|
||||
<div class="box">
|
||||
<pre style="white-space: pre-wrap; font-family: inherit;">%s</pre>
|
||||
</div>
|
||||
</div>`, env.Text)
|
||||
w.Write([]byte(html))
|
||||
} else {
|
||||
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||
html := `
|
||||
<div class="notification is-warning">
|
||||
<strong>No Content</strong><br>
|
||||
No HTML or text content found in this email.
|
||||
</div>`
|
||||
w.Write([]byte(html))
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// No Gmail service available - show error
|
||||
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||
html := fmt.Sprintf(`
|
||||
<div class="content">
|
||||
<div class="notification is-warning is-light">
|
||||
<strong>Gmail Service Unavailable</strong><br>
|
||||
<small>Subject: %s</small><br>
|
||||
<small>Date: %s</small><br>
|
||||
<small>Gmail Message ID: <code>%s</code></small>
|
||||
</div>
|
||||
|
||||
<div class="box">
|
||||
<div class="content">
|
||||
<h4>Integration Status</h4>
|
||||
<p>Gmail service is not available. To enable email content display:</p>
|
||||
<ol>
|
||||
<li>Ensure <code>credentials.json</code> and <code>token.json</code> files are present</li>
|
||||
<li>Configure Gmail API OAuth2 authentication</li>
|
||||
<li>Restart the application</li>
|
||||
</ol>
|
||||
<p class="has-text-grey-light is-size-7">
|
||||
<strong>Gmail Message ID:</strong> <code>%s</code>
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>`,
|
||||
subject,
|
||||
created.Format("2006-01-02 15:04:05"),
|
||||
gmailMessageID.String,
|
||||
gmailMessageID.String)
|
||||
|
||||
w.Write([]byte(html))
|
||||
}
|
||||
|
||||
// Stream attachment from Gmail
|
||||
func (h *EmailHandler) StreamAttachment(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
emailID, err := strconv.Atoi(vars["id"])
|
||||
if err != nil {
|
||||
http.Error(w, "Invalid email ID", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
attachmentID := vars["attachmentId"]
|
||||
|
||||
// Get email's Gmail message ID
|
||||
var gmailMessageID sql.NullString
|
||||
err = h.db.QueryRow("SELECT gmail_message_id FROM emails WHERE id = ?", emailID).Scan(&gmailMessageID)
|
||||
if err != nil || !gmailMessageID.Valid {
|
||||
http.Error(w, "Email not found or not a Gmail email", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
|
||||
if h.gmailService == nil {
|
||||
http.Error(w, "Gmail service not available", http.StatusServiceUnavailable)
|
||||
return
|
||||
}
|
||||
|
||||
// For dynamic attachments, we need to fetch the message and find the attachment
|
||||
message, err := h.gmailService.Users.Messages.Get("me", gmailMessageID.String).
|
||||
Format("FULL").Do()
|
||||
if err != nil {
|
||||
http.Error(w, "Failed to fetch email from Gmail", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
// Find the attachment by index
|
||||
var targetAttachment *gmail.MessagePart
|
||||
attachmentIndex := 1
|
||||
findAttachment(message.Payload, attachmentID, &attachmentIndex, &targetAttachment)
|
||||
|
||||
if targetAttachment == nil || targetAttachment.Body == nil || targetAttachment.Body.AttachmentId == "" {
|
||||
http.Error(w, "Attachment not found", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
|
||||
// Fetch attachment data from Gmail
|
||||
attachment, err := h.gmailService.Users.Messages.Attachments.
|
||||
Get("me", gmailMessageID.String, targetAttachment.Body.AttachmentId).Do()
|
||||
if err != nil {
|
||||
http.Error(w, "Failed to fetch attachment from Gmail", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
// Decode base64
|
||||
data, err := base64.URLEncoding.DecodeString(attachment.Data)
|
||||
if err != nil {
|
||||
http.Error(w, "Failed to decode attachment", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
// Set headers and stream
|
||||
filename := targetAttachment.Filename
|
||||
if filename == "" {
|
||||
// Generate filename from content type (same logic as extractGmailAttachments)
|
||||
switch targetAttachment.MimeType {
|
||||
case "application/pdf":
|
||||
filename = "attachment.pdf"
|
||||
case "image/png":
|
||||
filename = "image.png"
|
||||
case "image/jpeg":
|
||||
filename = "image.jpg"
|
||||
case "text/plain":
|
||||
filename = "text.txt"
|
||||
default:
|
||||
filename = "attachment"
|
||||
}
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", targetAttachment.MimeType)
|
||||
w.Header().Set("Content-Disposition", fmt.Sprintf("attachment; filename=\"%s\"", filename))
|
||||
w.Header().Set("Content-Length", fmt.Sprintf("%d", len(data)))
|
||||
w.Write(data)
|
||||
}
|
||||
|
||||
// Helper function to find attachment by index
|
||||
func findAttachment(part *gmail.MessagePart, targetID string, currentIndex *int, result **gmail.MessagePart) {
|
||||
// Check if this part is an attachment (same logic as extractGmailAttachments)
|
||||
if part.Body != nil && part.Body.AttachmentId != "" {
|
||||
fmt.Printf("Checking attachment %d (looking for %s): %s\n", *currentIndex, targetID, part.Filename)
|
||||
if strconv.Itoa(*currentIndex) == targetID {
|
||||
fmt.Printf("Found matching attachment!\n")
|
||||
*result = part
|
||||
return
|
||||
}
|
||||
*currentIndex++
|
||||
}
|
||||
|
||||
for _, subPart := range part.Parts {
|
||||
findAttachment(subPart, targetID, currentIndex, result)
|
||||
if *result != nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Helper function to join conditions
|
||||
func joinConditions(conditions []string, separator string) string {
|
||||
if len(conditions) == 0 {
|
||||
return ""
|
||||
}
|
||||
if len(conditions) == 1 {
|
||||
return conditions[0]
|
||||
}
|
||||
|
||||
result := conditions[0]
|
||||
for i := 1; i < len(conditions); i++ {
|
||||
result += separator + conditions[i]
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// Gmail OAuth2 functions
|
||||
func getGmailService(credentialsFile, tokenFile string) (*gmail.Service, error) {
|
||||
ctx := context.Background()
|
||||
|
||||
b, err := ioutil.ReadFile(credentialsFile)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("unable to read client secret file: %v", err)
|
||||
}
|
||||
|
||||
config, err := google.ConfigFromJSON(b, gmail.GmailReadonlyScope)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("unable to parse client secret file to config: %v", err)
|
||||
}
|
||||
|
||||
client := getClient(config, tokenFile)
|
||||
srv, err := gmail.NewService(ctx, option.WithHTTPClient(client))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("unable to retrieve Gmail client: %v", err)
|
||||
}
|
||||
|
||||
return srv, nil
|
||||
}
|
||||
|
||||
func getClient(config *oauth2.Config, tokFile string) *http.Client {
|
||||
tok, err := tokenFromFile(tokFile)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
return config.Client(context.Background(), tok)
|
||||
}
|
||||
|
||||
func tokenFromFile(file string) (*oauth2.Token, error) {
|
||||
f, err := os.Open(file)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer f.Close()
|
||||
tok := &oauth2.Token{}
|
||||
err = json.NewDecoder(f).Decode(tok)
|
||||
return tok, err
|
||||
}
|
||||
|
|
@ -1,9 +1,11 @@
|
|||
package handlers
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"log"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"code.springupsoftware.com/cmc/cmc-sales/internal/cmc/db"
|
||||
"code.springupsoftware.com/cmc/cmc-sales/internal/cmc/templates"
|
||||
|
|
@ -15,12 +17,14 @@ import (
|
|||
type PageHandler struct {
|
||||
queries *db.Queries
|
||||
tmpl *templates.TemplateManager
|
||||
db *sql.DB
|
||||
}
|
||||
|
||||
func NewPageHandler(queries *db.Queries, tmpl *templates.TemplateManager) *PageHandler {
|
||||
func NewPageHandler(queries *db.Queries, tmpl *templates.TemplateManager, database *sql.DB) *PageHandler {
|
||||
return &PageHandler{
|
||||
queries: queries,
|
||||
tmpl: tmpl,
|
||||
db: database,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -813,3 +817,404 @@ func (h *PageHandler) DocumentsView(w http.ResponseWriter, r *http.Request) {
|
|||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
}
|
||||
}
|
||||
|
||||
// Email page handlers
|
||||
func (h *PageHandler) EmailsIndex(w http.ResponseWriter, r *http.Request) {
|
||||
page := 1
|
||||
if p := r.URL.Query().Get("page"); p != "" {
|
||||
if val, err := strconv.Atoi(p); err == nil && val > 0 {
|
||||
page = val
|
||||
}
|
||||
}
|
||||
|
||||
limit := 30
|
||||
offset := (page - 1) * limit
|
||||
search := r.URL.Query().Get("search")
|
||||
filter := r.URL.Query().Get("filter")
|
||||
|
||||
// Build SQL query based on filters
|
||||
query := `
|
||||
SELECT e.id, e.subject, e.user_id, e.created, e.gmail_message_id,
|
||||
e.email_attachment_count, e.is_downloaded,
|
||||
u.email as user_email, u.first_name, u.last_name
|
||||
FROM emails e
|
||||
LEFT JOIN users u ON e.user_id = u.id`
|
||||
|
||||
var args []interface{}
|
||||
var conditions []string
|
||||
|
||||
// Apply search filter
|
||||
if search != "" {
|
||||
conditions = append(conditions, "(e.subject LIKE ? OR e.raw_headers LIKE ?)")
|
||||
searchTerm := "%" + search + "%"
|
||||
args = append(args, searchTerm, searchTerm)
|
||||
}
|
||||
|
||||
// Apply type filter
|
||||
switch filter {
|
||||
case "downloaded":
|
||||
conditions = append(conditions, "e.is_downloaded = 1")
|
||||
case "gmail":
|
||||
conditions = append(conditions, "e.gmail_message_id IS NOT NULL")
|
||||
case "unassociated":
|
||||
conditions = append(conditions, `NOT EXISTS (
|
||||
SELECT 1 FROM emails_enquiries WHERE email_id = e.id
|
||||
UNION SELECT 1 FROM emails_invoices WHERE email_id = e.id
|
||||
UNION SELECT 1 FROM emails_purchase_orders WHERE email_id = e.id
|
||||
UNION SELECT 1 FROM emails_jobs WHERE email_id = e.id
|
||||
)`)
|
||||
}
|
||||
|
||||
if len(conditions) > 0 {
|
||||
query += " WHERE " + joinConditions(conditions, " AND ")
|
||||
}
|
||||
|
||||
query += " ORDER BY e.id DESC LIMIT ? OFFSET ?"
|
||||
args = append(args, limit+1, offset) // Get one extra to check if there are more
|
||||
|
||||
// Execute the query to get emails
|
||||
rows, err := h.db.Query(query, args...)
|
||||
if err != nil {
|
||||
log.Printf("Error querying emails: %v", err)
|
||||
http.Error(w, "Database error", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
type EmailWithUser struct {
|
||||
ID int32 `json:"id"`
|
||||
Subject string `json:"subject"`
|
||||
UserID int32 `json:"user_id"`
|
||||
Created time.Time `json:"created"`
|
||||
GmailMessageID *string `json:"gmail_message_id"`
|
||||
AttachmentCount int32 `json:"attachment_count"`
|
||||
IsDownloaded *bool `json:"is_downloaded"`
|
||||
UserEmail *string `json:"user_email"`
|
||||
FirstName *string `json:"first_name"`
|
||||
LastName *string `json:"last_name"`
|
||||
}
|
||||
|
||||
var emails []EmailWithUser
|
||||
for rows.Next() {
|
||||
var email EmailWithUser
|
||||
var gmailMessageID, userEmail, firstName, lastName sql.NullString
|
||||
var isDownloaded sql.NullBool
|
||||
|
||||
err := rows.Scan(
|
||||
&email.ID,
|
||||
&email.Subject,
|
||||
&email.UserID,
|
||||
&email.Created,
|
||||
&gmailMessageID,
|
||||
&email.AttachmentCount,
|
||||
&isDownloaded,
|
||||
&userEmail,
|
||||
&firstName,
|
||||
&lastName,
|
||||
)
|
||||
if err != nil {
|
||||
log.Printf("Error scanning email row: %v", err)
|
||||
continue
|
||||
}
|
||||
|
||||
if gmailMessageID.Valid {
|
||||
email.GmailMessageID = &gmailMessageID.String
|
||||
}
|
||||
if isDownloaded.Valid {
|
||||
email.IsDownloaded = &isDownloaded.Bool
|
||||
}
|
||||
if userEmail.Valid {
|
||||
email.UserEmail = &userEmail.String
|
||||
}
|
||||
if firstName.Valid {
|
||||
email.FirstName = &firstName.String
|
||||
}
|
||||
if lastName.Valid {
|
||||
email.LastName = &lastName.String
|
||||
}
|
||||
|
||||
emails = append(emails, email)
|
||||
}
|
||||
|
||||
hasMore := len(emails) > limit
|
||||
if hasMore {
|
||||
emails = emails[:limit]
|
||||
}
|
||||
|
||||
data := map[string]interface{}{
|
||||
"Emails": emails,
|
||||
"Page": page,
|
||||
"PrevPage": page - 1,
|
||||
"NextPage": page + 1,
|
||||
"HasMore": hasMore,
|
||||
"TotalPages": ((len(emails) + limit - 1) / limit),
|
||||
}
|
||||
|
||||
// Check if this is an HTMX request
|
||||
if r.Header.Get("HX-Request") == "true" {
|
||||
if err := h.tmpl.RenderPartial(w, "emails/table.html", "email-table", data); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
if err := h.tmpl.Render(w, "emails/index.html", data); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
}
|
||||
}
|
||||
|
||||
func (h *PageHandler) EmailsShow(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
id, err := strconv.Atoi(vars["id"])
|
||||
if err != nil {
|
||||
http.Error(w, "Invalid email ID", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
// Get email details from database
|
||||
emailQuery := `
|
||||
SELECT e.id, e.subject, e.user_id, e.created, e.gmail_message_id,
|
||||
e.gmail_thread_id, e.raw_headers, e.is_downloaded,
|
||||
u.email as user_email, u.first_name, u.last_name
|
||||
FROM emails e
|
||||
LEFT JOIN users u ON e.user_id = u.id
|
||||
WHERE e.id = ?`
|
||||
|
||||
var email struct {
|
||||
ID int32 `json:"id"`
|
||||
Subject string `json:"subject"`
|
||||
UserID int32 `json:"user_id"`
|
||||
Created time.Time `json:"created"`
|
||||
GmailMessageID *string `json:"gmail_message_id"`
|
||||
GmailThreadID *string `json:"gmail_thread_id"`
|
||||
RawHeaders *string `json:"raw_headers"`
|
||||
IsDownloaded *bool `json:"is_downloaded"`
|
||||
User *struct {
|
||||
Email string `json:"email"`
|
||||
FirstName string `json:"first_name"`
|
||||
LastName string `json:"last_name"`
|
||||
} `json:"user"`
|
||||
Enquiries []int32 `json:"enquiries"`
|
||||
Invoices []int32 `json:"invoices"`
|
||||
PurchaseOrders []int32 `json:"purchase_orders"`
|
||||
Jobs []int32 `json:"jobs"`
|
||||
}
|
||||
|
||||
var gmailMessageID, gmailThreadID, rawHeaders sql.NullString
|
||||
var isDownloaded sql.NullBool
|
||||
var userEmail, firstName, lastName sql.NullString
|
||||
|
||||
err = h.db.QueryRow(emailQuery, id).Scan(
|
||||
&email.ID,
|
||||
&email.Subject,
|
||||
&email.UserID,
|
||||
&email.Created,
|
||||
&gmailMessageID,
|
||||
&gmailThreadID,
|
||||
&rawHeaders,
|
||||
&isDownloaded,
|
||||
&userEmail,
|
||||
&firstName,
|
||||
&lastName,
|
||||
)
|
||||
|
||||
if err != nil {
|
||||
if err == sql.ErrNoRows {
|
||||
http.Error(w, "Email not found", http.StatusNotFound)
|
||||
} else {
|
||||
log.Printf("Error fetching email %d: %v", id, err)
|
||||
http.Error(w, "Database error", http.StatusInternalServerError)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// Set nullable fields
|
||||
if gmailMessageID.Valid {
|
||||
email.GmailMessageID = &gmailMessageID.String
|
||||
}
|
||||
if gmailThreadID.Valid {
|
||||
email.GmailThreadID = &gmailThreadID.String
|
||||
}
|
||||
if rawHeaders.Valid {
|
||||
email.RawHeaders = &rawHeaders.String
|
||||
}
|
||||
if isDownloaded.Valid {
|
||||
email.IsDownloaded = &isDownloaded.Bool
|
||||
}
|
||||
|
||||
// Set user info if available
|
||||
if userEmail.Valid {
|
||||
email.User = &struct {
|
||||
Email string `json:"email"`
|
||||
FirstName string `json:"first_name"`
|
||||
LastName string `json:"last_name"`
|
||||
}{
|
||||
Email: userEmail.String,
|
||||
FirstName: firstName.String,
|
||||
LastName: lastName.String,
|
||||
}
|
||||
}
|
||||
|
||||
// Get email attachments
|
||||
attachmentQuery := `
|
||||
SELECT id, name, type, size, filename, is_message_body, gmail_attachment_id, created
|
||||
FROM email_attachments
|
||||
WHERE email_id = ?
|
||||
ORDER BY is_message_body DESC, created ASC`
|
||||
|
||||
attachmentRows, err := h.db.Query(attachmentQuery, id)
|
||||
if err != nil {
|
||||
log.Printf("Error fetching attachments for email %d: %v", id, err)
|
||||
}
|
||||
|
||||
type EmailAttachment struct {
|
||||
ID int32 `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Size int32 `json:"size"`
|
||||
Filename string `json:"filename"`
|
||||
IsMessageBody bool `json:"is_message_body"`
|
||||
GmailAttachmentID *string `json:"gmail_attachment_id"`
|
||||
Created time.Time `json:"created"`
|
||||
}
|
||||
|
||||
var attachments []EmailAttachment
|
||||
hasStoredAttachments := false
|
||||
|
||||
if attachmentRows != nil {
|
||||
defer attachmentRows.Close()
|
||||
for attachmentRows.Next() {
|
||||
hasStoredAttachments = true
|
||||
var attachment EmailAttachment
|
||||
var gmailAttachmentID sql.NullString
|
||||
|
||||
err := attachmentRows.Scan(
|
||||
&attachment.ID,
|
||||
&attachment.Name,
|
||||
&attachment.Type,
|
||||
&attachment.Size,
|
||||
&attachment.Filename,
|
||||
&attachment.IsMessageBody,
|
||||
&gmailAttachmentID,
|
||||
&attachment.Created,
|
||||
)
|
||||
if err != nil {
|
||||
log.Printf("Error scanning attachment: %v", err)
|
||||
continue
|
||||
}
|
||||
|
||||
if gmailAttachmentID.Valid {
|
||||
attachment.GmailAttachmentID = &gmailAttachmentID.String
|
||||
}
|
||||
|
||||
attachments = append(attachments, attachment)
|
||||
}
|
||||
}
|
||||
|
||||
// If no stored attachments and this is a Gmail email, show a notice
|
||||
if !hasStoredAttachments && email.GmailMessageID != nil {
|
||||
// For the page view, we'll just show a notice that attachments can be fetched
|
||||
// The actual fetching will happen via the API endpoint when needed
|
||||
log.Printf("Email %d is a Gmail email without indexed attachments", id)
|
||||
}
|
||||
|
||||
// Get associated records (simplified queries for now)
|
||||
// Enquiries
|
||||
enquiryRows, err := h.db.Query("SELECT enquiry_id FROM emails_enquiries WHERE email_id = ?", id)
|
||||
if err == nil {
|
||||
defer enquiryRows.Close()
|
||||
for enquiryRows.Next() {
|
||||
var enquiryID int32
|
||||
if enquiryRows.Scan(&enquiryID) == nil {
|
||||
email.Enquiries = append(email.Enquiries, enquiryID)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Invoices
|
||||
invoiceRows, err := h.db.Query("SELECT invoice_id FROM emails_invoices WHERE email_id = ?", id)
|
||||
if err == nil {
|
||||
defer invoiceRows.Close()
|
||||
for invoiceRows.Next() {
|
||||
var invoiceID int32
|
||||
if invoiceRows.Scan(&invoiceID) == nil {
|
||||
email.Invoices = append(email.Invoices, invoiceID)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Purchase Orders
|
||||
poRows, err := h.db.Query("SELECT purchase_order_id FROM emails_purchase_orders WHERE email_id = ?", id)
|
||||
if err == nil {
|
||||
defer poRows.Close()
|
||||
for poRows.Next() {
|
||||
var poID int32
|
||||
if poRows.Scan(&poID) == nil {
|
||||
email.PurchaseOrders = append(email.PurchaseOrders, poID)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Jobs
|
||||
jobRows, err := h.db.Query("SELECT job_id FROM emails_jobs WHERE email_id = ?", id)
|
||||
if err == nil {
|
||||
defer jobRows.Close()
|
||||
for jobRows.Next() {
|
||||
var jobID int32
|
||||
if jobRows.Scan(&jobID) == nil {
|
||||
email.Jobs = append(email.Jobs, jobID)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
data := map[string]interface{}{
|
||||
"Email": email,
|
||||
"Attachments": attachments,
|
||||
}
|
||||
|
||||
if err := h.tmpl.Render(w, "emails/show.html", data); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
}
|
||||
}
|
||||
|
||||
func (h *PageHandler) EmailsSearch(w http.ResponseWriter, r *http.Request) {
|
||||
_ = r.URL.Query().Get("search") // TODO: Implement search functionality
|
||||
|
||||
// Empty result for now - would need proper implementation
|
||||
emails := []interface{}{}
|
||||
|
||||
data := map[string]interface{}{
|
||||
"Emails": emails,
|
||||
"Page": 1,
|
||||
"PrevPage": 0,
|
||||
"NextPage": 2,
|
||||
"HasMore": false,
|
||||
"TotalPages": 1,
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", "text/html")
|
||||
if err := h.tmpl.RenderPartial(w, "emails/table.html", "email-table", data); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
}
|
||||
}
|
||||
|
||||
func (h *PageHandler) EmailsAttachments(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
_, err := strconv.Atoi(vars["id"])
|
||||
if err != nil {
|
||||
http.Error(w, "Invalid email ID", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
// Empty attachments for now - would need proper implementation
|
||||
attachments := []interface{}{}
|
||||
|
||||
data := map[string]interface{}{
|
||||
"Attachments": attachments,
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", "text/html")
|
||||
if err := h.tmpl.RenderPartial(w, "emails/attachments.html", "email-attachments", data); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -56,6 +56,10 @@ func NewTemplateManager(templatesDir string) (*TemplateManager, error) {
|
|||
"enquiries/show.html",
|
||||
"enquiries/form.html",
|
||||
"enquiries/table.html",
|
||||
"emails/index.html",
|
||||
"emails/show.html",
|
||||
"emails/table.html",
|
||||
"emails/attachments.html",
|
||||
"documents/index.html",
|
||||
"documents/show.html",
|
||||
"documents/table.html",
|
||||
|
|
|
|||
52
go-app/sql/migrations/001_add_gmail_fields.sql
Normal file
52
go-app/sql/migrations/001_add_gmail_fields.sql
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
-- +goose Up
|
||||
-- Add Gmail-specific fields to emails table
|
||||
ALTER TABLE emails
|
||||
ADD COLUMN gmail_message_id VARCHAR(255) UNIQUE AFTER id,
|
||||
ADD COLUMN gmail_thread_id VARCHAR(255) AFTER gmail_message_id,
|
||||
ADD COLUMN is_downloaded BOOLEAN DEFAULT FALSE AFTER email_attachment_count,
|
||||
ADD COLUMN raw_headers TEXT AFTER subject;
|
||||
|
||||
-- +goose StatementBegin
|
||||
CREATE INDEX idx_gmail_message_id ON emails(gmail_message_id);
|
||||
-- +goose StatementEnd
|
||||
|
||||
-- +goose StatementBegin
|
||||
CREATE INDEX idx_gmail_thread_id ON emails(gmail_thread_id);
|
||||
-- +goose StatementEnd
|
||||
|
||||
-- Add Gmail-specific fields to email_attachments
|
||||
ALTER TABLE email_attachments
|
||||
ADD COLUMN gmail_attachment_id VARCHAR(255) AFTER email_id,
|
||||
ADD COLUMN gmail_message_id VARCHAR(255) AFTER gmail_attachment_id,
|
||||
ADD COLUMN content_id VARCHAR(255) AFTER filename;
|
||||
|
||||
-- +goose StatementBegin
|
||||
CREATE INDEX idx_gmail_attachment_id ON email_attachments(gmail_attachment_id);
|
||||
-- +goose StatementEnd
|
||||
|
||||
-- +goose Down
|
||||
-- Remove indexes
|
||||
-- +goose StatementBegin
|
||||
DROP INDEX idx_gmail_attachment_id ON email_attachments;
|
||||
-- +goose StatementEnd
|
||||
|
||||
-- +goose StatementBegin
|
||||
DROP INDEX idx_gmail_thread_id ON emails;
|
||||
-- +goose StatementEnd
|
||||
|
||||
-- +goose StatementBegin
|
||||
DROP INDEX idx_gmail_message_id ON emails;
|
||||
-- +goose StatementEnd
|
||||
|
||||
-- Remove columns from email_attachments
|
||||
ALTER TABLE email_attachments
|
||||
DROP COLUMN content_id,
|
||||
DROP COLUMN gmail_message_id,
|
||||
DROP COLUMN gmail_attachment_id;
|
||||
|
||||
-- Remove columns from emails
|
||||
ALTER TABLE emails
|
||||
DROP COLUMN raw_headers,
|
||||
DROP COLUMN is_downloaded,
|
||||
DROP COLUMN gmail_thread_id,
|
||||
DROP COLUMN gmail_message_id;
|
||||
|
|
@ -107,3 +107,19 @@ body {
|
|||
border-color: #ff3860;
|
||||
box-shadow: 0 0 0 0.125em rgba(255,56,96,.25);
|
||||
}
|
||||
|
||||
/* Simple CSS loader */
|
||||
.loader {
|
||||
border: 2px solid #f3f3f3;
|
||||
border-top: 2px solid #3273dc;
|
||||
border-radius: 50%;
|
||||
width: 16px;
|
||||
height: 16px;
|
||||
animation: spin 1s linear infinite;
|
||||
display: inline-block;
|
||||
}
|
||||
|
||||
@keyframes spin {
|
||||
0% { transform: rotate(0deg); }
|
||||
100% { transform: rotate(360deg); }
|
||||
}
|
||||
67
go-app/templates/emails/attachments.html
Normal file
67
go-app/templates/emails/attachments.html
Normal file
|
|
@ -0,0 +1,67 @@
|
|||
{{define "email-attachments"}}
|
||||
<div class="table-container">
|
||||
<table class="table is-fullwidth">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Name</th>
|
||||
<th>Type</th>
|
||||
<th>Size</th>
|
||||
<th>Date</th>
|
||||
<th>Actions</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{{range .}}
|
||||
<tr>
|
||||
<td>
|
||||
{{if .IsMessageBody}}
|
||||
<span class="icon has-text-info">
|
||||
<i class="fas fa-envelope"></i>
|
||||
</span>
|
||||
<span class="has-text-weight-semibold">{{.Name}}</span>
|
||||
<span class="tag is-small is-info ml-2">Body</span>
|
||||
{{else}}
|
||||
<span class="icon has-text-grey">
|
||||
<i class="fas fa-paperclip"></i>
|
||||
</span>
|
||||
{{.Name}}
|
||||
{{end}}
|
||||
</td>
|
||||
<td>
|
||||
<span class="tag is-light">{{.Type}}</span>
|
||||
</td>
|
||||
<td>{{.Size}} bytes</td>
|
||||
<td class="is-size-7">
|
||||
{{.Created}}
|
||||
</td>
|
||||
<td>
|
||||
{{if .GmailAttachmentID}}
|
||||
<a href="/api/v1/emails/attachments/{{.ID}}/stream"
|
||||
target="_blank" class="button is-small is-info">
|
||||
<span class="icon">
|
||||
<i class="fas fa-cloud-download-alt"></i>
|
||||
</span>
|
||||
<span>Stream</span>
|
||||
</a>
|
||||
{{else}}
|
||||
<a href="/api/v1/emails/attachments/{{.ID}}"
|
||||
target="_blank" class="button is-small is-success">
|
||||
<span class="icon">
|
||||
<i class="fas fa-download"></i>
|
||||
</span>
|
||||
<span>Download</span>
|
||||
</a>
|
||||
{{end}}
|
||||
</td>
|
||||
</tr>
|
||||
{{else}}
|
||||
<tr>
|
||||
<td colspan="5" class="has-text-centered">
|
||||
<p class="has-text-grey">No attachments found</p>
|
||||
</td>
|
||||
</tr>
|
||||
{{end}}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
{{end}}
|
||||
82
go-app/templates/emails/index.html
Normal file
82
go-app/templates/emails/index.html
Normal file
|
|
@ -0,0 +1,82 @@
|
|||
{{define "title"}}Emails - CMC Sales{{end}}
|
||||
|
||||
{{define "content"}}
|
||||
<div class="level">
|
||||
<div class="level-left">
|
||||
<div class="level-item">
|
||||
<h1 class="title">Emails</h1>
|
||||
</div>
|
||||
</div>
|
||||
<div class="level-right">
|
||||
<div class="level-item">
|
||||
<div class="field has-addons">
|
||||
<div class="control">
|
||||
<div class="select">
|
||||
<select id="email-filter">
|
||||
<option value="">All Emails</option>
|
||||
<option value="downloaded">Downloaded</option>
|
||||
<option value="gmail">Gmail Only</option>
|
||||
<option value="unassociated">Unassociated</option>
|
||||
</select>
|
||||
</div>
|
||||
</div>
|
||||
<div class="control">
|
||||
<button class="button is-info" id="apply-filter">
|
||||
<span class="icon">
|
||||
<i class="fas fa-filter"></i>
|
||||
</span>
|
||||
<span>Filter</span>
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Search Box -->
|
||||
<div class="box">
|
||||
<div class="field has-addons">
|
||||
<div class="control has-icons-left is-expanded">
|
||||
<input class="input" type="text" placeholder="Search emails by subject or content..."
|
||||
name="search" id="email-search"
|
||||
hx-get="/emails/search"
|
||||
hx-trigger="keyup changed delay:500ms"
|
||||
hx-target="#email-table-container">
|
||||
<span class="icon is-left">
|
||||
<i class="fas fa-search"></i>
|
||||
</span>
|
||||
</div>
|
||||
<div class="control">
|
||||
<button class="button is-info" onclick="document.getElementById('email-search').value=''; document.getElementById('email-search').dispatchEvent(new Event('keyup'));">
|
||||
Clear
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Email Table Container -->
|
||||
<div id="email-table-container">
|
||||
{{template "email-table" .}}
|
||||
</div>
|
||||
{{end}}
|
||||
|
||||
{{define "scripts"}}
|
||||
<script>
|
||||
document.getElementById('apply-filter').addEventListener('click', function() {
|
||||
const filter = document.getElementById('email-filter').value;
|
||||
const search = document.getElementById('email-search').value;
|
||||
|
||||
let url = '/emails';
|
||||
const params = new URLSearchParams();
|
||||
|
||||
if (filter) params.append('filter', filter);
|
||||
if (search) params.append('search', search);
|
||||
|
||||
if (params.toString()) {
|
||||
url += '?' + params.toString();
|
||||
}
|
||||
|
||||
htmx.ajax('GET', url, {target: '#email-table-container'});
|
||||
});
|
||||
</script>
|
||||
{{end}}
|
||||
293
go-app/templates/emails/show.html
Normal file
293
go-app/templates/emails/show.html
Normal file
|
|
@ -0,0 +1,293 @@
|
|||
{{define "title"}}Email {{.Email.ID}} - CMC Sales{{end}}
|
||||
|
||||
{{define "content"}}
|
||||
<div class="level">
|
||||
<div class="level-left">
|
||||
<div class="level-item">
|
||||
<nav class="breadcrumb" aria-label="breadcrumbs">
|
||||
<ul>
|
||||
<li><a href="/emails">Emails</a></li>
|
||||
<li class="is-active"><a href="#" aria-current="page">Email {{.Email.ID}}</a></li>
|
||||
</ul>
|
||||
</nav>
|
||||
</div>
|
||||
</div>
|
||||
<div class="level-right">
|
||||
<div class="level-item">
|
||||
<div class="buttons">
|
||||
{{if .Email.GmailMessageID}}
|
||||
<a href="/api/v1/emails/{{.Email.ID}}/stream" target="_blank" class="button is-success">
|
||||
<span class="icon">
|
||||
<i class="fas fa-external-link-alt"></i>
|
||||
</span>
|
||||
<span>View in Gmail</span>
|
||||
</a>
|
||||
{{end}}
|
||||
<a href="/emails" class="button">
|
||||
<span class="icon">
|
||||
<i class="fas fa-arrow-left"></i>
|
||||
</span>
|
||||
<span>Back to Emails</span>
|
||||
</a>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns">
|
||||
<div class="column is-8">
|
||||
<!-- Email Details -->
|
||||
<div class="box">
|
||||
<h2 class="title is-4">
|
||||
{{if .Email.Subject}}{{.Email.Subject}}{{else}}<em>(No Subject)</em>{{end}}
|
||||
</h2>
|
||||
|
||||
<div class="content">
|
||||
<div class="columns is-multiline">
|
||||
<div class="column is-6">
|
||||
<strong>From:</strong>
|
||||
{{if .Email.User}}
|
||||
{{.Email.User.Email}} ({{.Email.User.FirstName}} {{.Email.User.LastName}})
|
||||
{{else}}
|
||||
<span class="has-text-grey">Unknown sender</span>
|
||||
{{end}}
|
||||
</div>
|
||||
<div class="column is-6">
|
||||
<strong>Date:</strong> {{.Email.Created.Format "2006-01-02 15:04:05"}}
|
||||
</div>
|
||||
<div class="column is-6">
|
||||
<strong>Type:</strong>
|
||||
{{if .Email.GmailMessageID}}
|
||||
<span class="tag is-link">
|
||||
<span class="icon is-small">
|
||||
<i class="fab fa-google"></i>
|
||||
</span>
|
||||
<span>Gmail</span>
|
||||
</span>
|
||||
{{if and .Email.IsDownloaded (not .Email.IsDownloaded)}}
|
||||
<span class="tag is-warning">Remote</span>
|
||||
{{else}}
|
||||
<span class="tag is-success">Downloaded</span>
|
||||
{{end}}
|
||||
{{else}}
|
||||
<span class="tag is-success">Local Email</span>
|
||||
{{end}}
|
||||
</div>
|
||||
<div class="column is-6">
|
||||
<strong>Attachments:</strong>
|
||||
{{if gt (len .Attachments) 0}}
|
||||
<span class="tag is-info">{{len .Attachments}} files</span>
|
||||
{{else}}
|
||||
<span class="has-text-grey">None</span>
|
||||
{{end}}
|
||||
</div>
|
||||
{{if .Email.GmailMessageID}}
|
||||
<div class="column is-12">
|
||||
<strong>Gmail Message ID:</strong>
|
||||
<code class="is-size-7">{{.Email.GmailMessageID}}</code>
|
||||
</div>
|
||||
{{end}}
|
||||
{{if .Email.GmailThreadID}}
|
||||
<div class="column is-12">
|
||||
<strong>Gmail Thread ID:</strong>
|
||||
<code class="is-size-7">{{.Email.GmailThreadID}}</code>
|
||||
</div>
|
||||
{{end}}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Email Content -->
|
||||
{{if .Email.GmailMessageID}}
|
||||
<div class="box">
|
||||
<h3 class="title is-5">Email Content</h3>
|
||||
<div id="email-content-container"
|
||||
hx-get="/api/v1/emails/{{.Email.ID}}/content"
|
||||
hx-trigger="load">
|
||||
<div class="notification is-info">
|
||||
<div class="is-flex is-align-items-center">
|
||||
<span class="loader mr-3"></span>
|
||||
<span>Loading email content...</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
{{end}}
|
||||
|
||||
<!-- Attachments -->
|
||||
{{if gt (len .Attachments) 0}}
|
||||
<div class="box">
|
||||
<h3 class="title is-5">Attachments</h3>
|
||||
<div class="table-container">
|
||||
<table class="table is-fullwidth">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Name</th>
|
||||
<th>Type</th>
|
||||
<th>Size</th>
|
||||
<th>Date</th>
|
||||
<th>Actions</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{{range .Attachments}}
|
||||
<tr>
|
||||
<td>
|
||||
{{if .IsMessageBody}}
|
||||
<span class="icon has-text-info">
|
||||
<i class="fas fa-envelope"></i>
|
||||
</span>
|
||||
{{else}}
|
||||
<span class="icon has-text-grey">
|
||||
<i class="fas fa-paperclip"></i>
|
||||
</span>
|
||||
{{end}}
|
||||
{{.Name}}
|
||||
</td>
|
||||
<td>
|
||||
<span class="tag is-light">{{.Type}}</span>
|
||||
</td>
|
||||
<td>{{.Size}} bytes</td>
|
||||
<td class="is-size-7">
|
||||
{{.Created.Format "2006-01-02 15:04"}}
|
||||
</td>
|
||||
<td>
|
||||
{{if .GmailAttachmentID}}
|
||||
<a href="/api/v1/emails/{{$.Email.ID}}/attachments/{{.ID}}/stream"
|
||||
target="_blank" class="button is-small is-info">
|
||||
<span class="icon">
|
||||
<i class="fas fa-download"></i>
|
||||
</span>
|
||||
<span>Download</span>
|
||||
</a>
|
||||
{{else}}
|
||||
<a href="/api/v1/emails/{{$.Email.ID}}/attachments/{{.ID}}"
|
||||
target="_blank" class="button is-small is-success">
|
||||
<span class="icon">
|
||||
<i class="fas fa-download"></i>
|
||||
</span>
|
||||
<span>Download</span>
|
||||
</a>
|
||||
{{end}}
|
||||
</td>
|
||||
</tr>
|
||||
{{end}}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
{{else if .Email.GmailMessageID}}
|
||||
<!-- Gmail Attachments Check -->
|
||||
<div class="box">
|
||||
<h3 class="title is-5">Attachments</h3>
|
||||
<div id="gmail-attachments-container"
|
||||
hx-get="/api/v1/emails/{{.Email.ID}}/attachments"
|
||||
hx-trigger="load"
|
||||
hx-swap="outerHTML">
|
||||
<div class="notification is-info is-light">
|
||||
<div class="is-flex is-align-items-center">
|
||||
<span class="loader mr-3"></span>
|
||||
<span>Checking for Gmail attachments...</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
{{end}}
|
||||
</div>
|
||||
|
||||
<div class="column is-4">
|
||||
<!-- Associated Records -->
|
||||
<div class="box">
|
||||
<h3 class="title is-5">Associated Records</h3>
|
||||
|
||||
{{if .Email.Enquiries}}
|
||||
<div class="field">
|
||||
<label class="label">Enquiries</label>
|
||||
<div class="tags">
|
||||
{{range .Email.Enquiries}}
|
||||
<a href="/enquiries/{{.}}" class="tag is-primary">
|
||||
ENQ-{{.}}
|
||||
</a>
|
||||
{{end}}
|
||||
</div>
|
||||
</div>
|
||||
{{end}}
|
||||
|
||||
{{if .Email.Invoices}}
|
||||
<div class="field">
|
||||
<label class="label">Invoices</label>
|
||||
<div class="tags">
|
||||
{{range .Email.Invoices}}
|
||||
<a href="/invoices/{{.}}" class="tag is-warning">
|
||||
INV-{{.}}
|
||||
</a>
|
||||
{{end}}
|
||||
</div>
|
||||
</div>
|
||||
{{end}}
|
||||
|
||||
{{if .Email.PurchaseOrders}}
|
||||
<div class="field">
|
||||
<label class="label">Purchase Orders</label>
|
||||
<div class="tags">
|
||||
{{range .Email.PurchaseOrders}}
|
||||
<a href="/purchase-orders/{{.}}" class="tag is-info">
|
||||
PO-{{.}}
|
||||
</a>
|
||||
{{end}}
|
||||
</div>
|
||||
</div>
|
||||
{{end}}
|
||||
|
||||
{{if .Email.Jobs}}
|
||||
<div class="field">
|
||||
<label class="label">Jobs</label>
|
||||
<div class="tags">
|
||||
{{range .Email.Jobs}}
|
||||
<a href="/jobs/{{.}}" class="tag is-success">
|
||||
JOB-{{.}}
|
||||
</a>
|
||||
{{end}}
|
||||
</div>
|
||||
</div>
|
||||
{{end}}
|
||||
|
||||
{{if and (not .Email.Enquiries) (not .Email.Invoices) (not .Email.PurchaseOrders) (not .Email.Jobs)}}
|
||||
<div class="notification is-warning is-light">
|
||||
<p><strong>No associations found</strong></p>
|
||||
<p class="is-size-7">This email is not associated with any enquiries, invoices, purchase orders, or jobs.</p>
|
||||
</div>
|
||||
{{end}}
|
||||
</div>
|
||||
|
||||
<!-- Quick Actions -->
|
||||
<div class="box">
|
||||
<h3 class="title is-5">Quick Actions</h3>
|
||||
<div class="buttons is-vertical is-fullwidth">
|
||||
<button class="button is-info is-outlined">
|
||||
<span class="icon">
|
||||
<i class="fas fa-link"></i>
|
||||
</span>
|
||||
<span>Associate with Record</span>
|
||||
</button>
|
||||
<button class="button is-warning is-outlined">
|
||||
<span class="icon">
|
||||
<i class="fas fa-flag"></i>
|
||||
</span>
|
||||
<span>Mark for Review</span>
|
||||
</button>
|
||||
{{if .Email.GmailMessageID}}
|
||||
<button class="button is-success is-outlined"
|
||||
hx-post="/api/v1/emails/{{.Email.ID}}/download"
|
||||
hx-confirm="Download this email and attachments locally?">
|
||||
<span class="icon">
|
||||
<i class="fas fa-download"></i>
|
||||
</span>
|
||||
<span>Download Locally</span>
|
||||
</button>
|
||||
{{end}}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
{{end}}
|
||||
143
go-app/templates/emails/table.html
Normal file
143
go-app/templates/emails/table.html
Normal file
|
|
@ -0,0 +1,143 @@
|
|||
{{define "email-table"}}
|
||||
<div class="table-container">
|
||||
<table class="table is-fullwidth is-striped is-hoverable">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>ID</th>
|
||||
<th>Subject</th>
|
||||
<th>From</th>
|
||||
<th>Date</th>
|
||||
<th>Attachments</th>
|
||||
<th>Type</th>
|
||||
<th>Associated</th>
|
||||
<th>Actions</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{{range .Emails}}
|
||||
<tr>
|
||||
<td>{{.ID}}</td>
|
||||
<td>
|
||||
<a href="/emails/{{.ID}}" class="has-text-weight-semibold">
|
||||
{{if .Subject}}{{.Subject}}{{else}}<em>(No Subject)</em>{{end}}
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
{{if .UserEmail}}
|
||||
{{.UserEmail}}
|
||||
{{else}}
|
||||
<span class="has-text-grey">Unknown</span>
|
||||
{{end}}
|
||||
</td>
|
||||
<td>
|
||||
<span class="is-size-7">
|
||||
{{.Created.Format "2006-01-02 15:04"}}
|
||||
</span>
|
||||
</td>
|
||||
<td>
|
||||
{{if gt .AttachmentCount 0}}
|
||||
<span class="tag is-info is-light">
|
||||
<span class="icon is-small">
|
||||
<i class="fas fa-paperclip"></i>
|
||||
</span>
|
||||
<span>{{.AttachmentCount}}</span>
|
||||
</span>
|
||||
{{else}}
|
||||
<span class="has-text-grey">None</span>
|
||||
{{end}}
|
||||
</td>
|
||||
<td>
|
||||
{{if .GmailMessageID}}
|
||||
<span class="tag is-link is-light">
|
||||
<span class="icon is-small">
|
||||
<i class="fab fa-google"></i>
|
||||
</span>
|
||||
<span>Gmail</span>
|
||||
</span>
|
||||
{{if and .IsDownloaded (not .IsDownloaded)}}
|
||||
<span class="tag is-warning is-light">
|
||||
<span class="icon is-small">
|
||||
<i class="fas fa-cloud"></i>
|
||||
</span>
|
||||
<span>Remote</span>
|
||||
</span>
|
||||
{{end}}
|
||||
{{else}}
|
||||
<span class="tag is-success is-light">
|
||||
<span class="icon is-small">
|
||||
<i class="fas fa-envelope"></i>
|
||||
</span>
|
||||
<span>Local</span>
|
||||
</span>
|
||||
{{end}}
|
||||
</td>
|
||||
<td>
|
||||
<span class="tag is-light">Associations TBD</span>
|
||||
</td>
|
||||
<td>
|
||||
<div class="buttons are-small">
|
||||
<a href="/emails/{{.ID}}" class="button is-info is-outlined">
|
||||
<span class="icon">
|
||||
<i class="fas fa-eye"></i>
|
||||
</span>
|
||||
</a>
|
||||
{{if gt .AttachmentCount 0}}
|
||||
<button class="button is-link is-outlined"
|
||||
hx-get="/emails/{{.ID}}/attachments"
|
||||
hx-target="#attachment-modal-content"
|
||||
onclick="document.getElementById('attachment-modal').classList.add('is-active')">
|
||||
<span class="icon">
|
||||
<i class="fas fa-paperclip"></i>
|
||||
</span>
|
||||
</button>
|
||||
{{end}}
|
||||
{{if .GmailMessageID}}
|
||||
<a href="/api/v1/emails/{{.ID}}/stream" target="_blank" class="button is-success is-outlined">
|
||||
<span class="icon">
|
||||
<i class="fas fa-external-link-alt"></i>
|
||||
</span>
|
||||
</a>
|
||||
{{end}}
|
||||
</div>
|
||||
</td>
|
||||
</tr>
|
||||
{{else}}
|
||||
<tr>
|
||||
<td colspan="8" class="has-text-centered">
|
||||
<p class="has-text-grey">No emails found</p>
|
||||
</td>
|
||||
</tr>
|
||||
{{end}}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<!-- Pagination -->
|
||||
{{if .Emails}}
|
||||
<nav class="pagination is-centered" role="navigation" aria-label="pagination">
|
||||
<a class="pagination-previous" {{if eq .Page 1}}disabled{{end}}
|
||||
hx-get="/emails?page={{.PrevPage}}"
|
||||
hx-target="#email-table-container">Previous</a>
|
||||
<a class="pagination-next" {{if not .HasMore}}disabled{{end}}
|
||||
hx-get="/emails?page={{.NextPage}}"
|
||||
hx-target="#email-table-container">Next</a>
|
||||
<ul class="pagination-list">
|
||||
<li><span class="pagination-ellipsis">Page {{.Page}} of {{.TotalPages}}</span></li>
|
||||
</ul>
|
||||
</nav>
|
||||
{{end}}
|
||||
|
||||
<!-- Attachment Modal -->
|
||||
<div class="modal" id="attachment-modal">
|
||||
<div class="modal-background" onclick="document.getElementById('attachment-modal').classList.remove('is-active')"></div>
|
||||
<div class="modal-card">
|
||||
<header class="modal-card-head">
|
||||
<p class="modal-card-title">Email Attachments</p>
|
||||
<button class="delete" aria-label="close" onclick="document.getElementById('attachment-modal').classList.remove('is-active')"></button>
|
||||
</header>
|
||||
<section class="modal-card-body" id="attachment-modal-content">
|
||||
<!-- Attachment list will be loaded here -->
|
||||
</section>
|
||||
</div>
|
||||
</div>
|
||||
{{end}}
|
||||
|
|
@ -7,3 +7,4 @@ haris:$apr1$7xqS6Oxx$3HeURNx9ceTV4WsaZEx2h1
|
|||
despina:$apr1$wyWhXD4y$UHG9//5wMwI3bkccyAMgz1
|
||||
richard:$apr1$3RMqU9gc$6iw/ZrIkSwU96YMqVr0/k.
|
||||
finley:$apr1$M4PiX6K/$k4/S5C.AMXPgpaRAirxKm0
|
||||
colleen:$apr1$ovbofsZ8$599TtnM7WVv/5eGDZpWYo0
|
||||
|
|
|
|||
|
|
@ -2,5 +2,5 @@
|
|||
|
||||
## run by cmc user cron to run the vault inside docker
|
||||
|
||||
ID=$(docker ps -q)
|
||||
ID=$(docker ps -f ancestor=cmc:latest --format "{{.ID}}" | head -n 1)
|
||||
docker exec -t $ID /var/www/cmc-sales/run_vault.sh
|
||||
|
|
|
|||
Loading…
Reference in a new issue