springup/cmc-sales
2
0
Fork 0
Code Issues 49 Pull requests Actions Packages Projects Releases Wiki Activity

Cleaning up repo, updating README.md

Browse source
This commit is contained in:
Finley Ghosh 2025-11-20 23:44:07 +11:00
parent da4eefdbf6
commit 4e769e5591
30 changed files with 105 additions and 1417 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

344
DEPLOYMENT-CADDY.md
View file

@ -1,344 +0,0 @@
# CMC Sales Deployment Guide with Caddy
## Overview
This guide covers deploying the CMC Sales application to a Debian 12 VM using Caddy as the reverse proxy with automatic HTTPS.
## Architecture
- **Production**: `https://cmc.springupsoftware.com`
- **Staging**: `https://staging.cmc.springupsoftware.com`
- **Reverse Proxy**: Caddy (running on host)
- **Applications**: Docker containers
- CakePHP legacy app
- Go modern app
- MariaDB database
- **SSL**: Automatic via Caddy (Let's Encrypt)
- **Authentication**: Basic auth configured in Caddy
## Prerequisites
### 1. Server Setup (Debian 12)
```bash
# Update system
sudo apt update && sudo apt upgrade -y
# Install Docker
sudo apt install -y docker.io docker-compose-plugin
sudo systemctl enable docker
sudo systemctl start docker
# Add user to docker group
sudo usermod -aG docker $USER
# Log out and back in
# Create directories
sudo mkdir -p /var/backups/cmc-sales
sudo chown $USER:$USER /var/backups/cmc-sales
```
### 2. Install Caddy
```bash
# Run the installation script
sudo ./scripts/install-caddy.sh
# Or manually install
sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy
```
### 3. DNS Configuration
Ensure DNS records point to your server:
- `cmc.springupsoftware.com` → Server IP
- `staging.cmc.springupsoftware.com` → Server IP
## Initial Deployment
### 1. Clone Repository
```bash
cd /home/cmc
git clone git@code.springupsoftware.com:cmc/cmc-sales.git cmc-sales
sudo chown -R $USER:$USER cmc-sales
cd cmc-sales
```
### 2. Environment Configuration
```bash
# Copy environment files
cp .env.staging go-app/.env.staging
cp .env.production go-app/.env.production
# Edit with actual passwords
nano .env.staging
nano .env.production
# Create credential directories
mkdir -p credentials/staging credentials/production
```
### 3. Setup Basic Authentication
```bash
# Generate password hashes for Caddy
./scripts/setup-caddy-auth.sh
# Or manually
caddy hash-password
# Copy the hash and update Caddyfile
```
### 4. Configure Caddy
```bash
# Copy Caddyfile
sudo cp Caddyfile /etc/caddy/Caddyfile
# Edit to update passwords and email
sudo nano /etc/caddy/Caddyfile
# Validate configuration
caddy validate --config /etc/caddy/Caddyfile
# Start Caddy
sudo systemctl start caddy
sudo systemctl status caddy
```
### 5. Gmail OAuth Setup
Same as before - set up OAuth credentials for each environment:
- Staging: `credentials/staging/credentials.json`
- Production: `credentials/production/credentials.json`
### 6. Database Initialization
```bash
# Start database containers
docker compose -f docker-compose.caddy-staging.yml up -d db-staging
docker compose -f docker-compose.caddy-production.yml up -d db-production
# Wait for databases
sleep 30
# Restore production database (if you have a backup)
./scripts/restore-db.sh production /path/to/backup.sql.gz
```
## Deployment Commands
### Starting Services
```bash
# Start staging environment
docker compose -f docker-compose.caddy-staging.yml up -d
# Start production environment
docker compose -f docker-compose.caddy-production.yml up -d
# Reload Caddy configuration
sudo systemctl reload caddy
```
### Updating Applications
```bash
# Pull latest code
git pull origin main
# Update staging
docker compose -f docker-compose.caddy-staging.yml down
docker compose -f docker-compose.caddy-staging.yml build --no-cache
docker compose -f docker-compose.caddy-staging.yml up -d
# Test staging, then update production
docker compose -f docker-compose.caddy-production.yml down
docker compose -f docker-compose.caddy-production.yml build --no-cache
docker compose -f docker-compose.caddy-production.yml up -d
```
## Caddy Management
### Configuration
```bash
# Edit Caddyfile
sudo nano /etc/caddy/Caddyfile
# Validate configuration
caddy validate --config /etc/caddy/Caddyfile
# Reload configuration (zero downtime)
sudo systemctl reload caddy
```
### Monitoring
```bash
# Check Caddy status
sudo systemctl status caddy
# View Caddy logs
sudo journalctl -u caddy -f
# View access logs
sudo tail -f /var/log/caddy/cmc-production.log
sudo tail -f /var/log/caddy/cmc-staging.log
```
### SSL Certificates
Caddy handles SSL automatically! To check certificates:
```bash
# List certificates
sudo ls -la /var/lib/caddy/.local/share/caddy/certificates/
# Force certificate renewal (rarely needed)
sudo systemctl stop caddy
sudo rm -rf /var/lib/caddy/.local/share/caddy/certificates/*
sudo systemctl start caddy
```
## Container Port Mapping
| Service | Container Port | Host Port | Access |
|---------|---------------|-----------|---------|
| cmc-php-staging | 80 | 8091 | localhost only |
| cmc-go-staging | 8080 | 8092 | localhost only |
| cmc-db-staging | 3306 | 3307 | localhost only |
| cmc-php-production | 80 | 8093 | localhost only |
| cmc-go-production | 8080 | 8094 | localhost only |
| cmc-db-production | 3306 | - | internal only |
## Monitoring and Maintenance
### Health Checks
```bash
# Check all containers
docker ps
# Check application health
curl -I https://cmc.springupsoftware.com
curl -I https://staging.cmc.springupsoftware.com
# Internal health checks (from server)
curl http://localhost:8094/api/v1/health # Production Go
curl http://localhost:8092/api/v1/health # Staging Go
```
### Database Backups
Same backup scripts work:
```bash
# Manual backup
./scripts/backup-db.sh production
./scripts/backup-db.sh staging
# Automated backups
sudo crontab -e
# Add:
# 0 2 * * * /home/cmc/cmc-sales/scripts/backup-db.sh production
# 0 3 * * * /home/cmc/cmc-sales/scripts/backup-db.sh staging
```
## Security Benefits with Caddy
1. **Automatic HTTPS**: No manual certificate management
2. **Modern TLS**: Always up-to-date TLS configuration
3. **OCSP Stapling**: Enabled by default
4. **Security Headers**: Easy to configure
5. **Rate Limiting**: Built-in support
## Troubleshooting
### Caddy Issues
```bash
# Check Caddy configuration
caddy validate --config /etc/caddy/Caddyfile
# Check Caddy service
sudo systemctl status caddy
sudo journalctl -u caddy -n 100
# Test reverse proxy
curl -v http://localhost:8094/api/v1/health
```
### Container Issues
```bash
# Check container logs
docker compose -f docker-compose.caddy-production.yml logs -f
# Restart specific service
docker compose -f docker-compose.caddy-production.yml restart cmc-go-production
```
### SSL Issues
```bash
# Caddy automatically handles SSL, but if issues arise:
# 1. Check DNS is resolving correctly
dig cmc.springupsoftware.com
# 2. Check Caddy can reach Let's Encrypt
sudo journalctl -u caddy | grep -i acme
# 3. Ensure ports 80 and 443 are open
sudo ufw status
```
## Advantages of Caddy Setup
1. **Simpler Configuration**: Caddyfile is more readable than nginx
2. **Automatic HTTPS**: No certbot or lego needed
3. **Zero-Downtime Reloads**: Config changes without dropping connections
4. **Better Performance**: More efficient than nginx for this use case
5. **Native Rate Limiting**: Built-in without additional modules
6. **Automatic Certificate Renewal**: No cron jobs needed
## Migration from Nginx
If migrating from the nginx setup:
1. Stop nginx containers: `docker compose -f docker-compose.proxy.yml down`
2. Install and configure Caddy
3. Start new containers with caddy compose files
4. Update DNS if needed
5. Monitor logs during transition
## File Structure
```
/home/cmc/cmc-sales/
├── docker-compose.caddy-staging.yml
├── docker-compose.caddy-production.yml
├── Caddyfile
├── credentials/
│ ├── staging/
│ └── production/
├── scripts/
│ ├── backup-db.sh
│ ├── restore-db.sh
│ ├── install-caddy.sh
│ └── setup-caddy-auth.sh
└── .env files
/etc/caddy/
└── Caddyfile (deployed config)
/var/log/caddy/
├── cmc-production.log
└── cmc-staging.log
```

362
DEPLOYMENT.md
View file

@ -1,362 +0,0 @@
# CMC Sales Deployment Guide
## Overview
This guide covers deploying the CMC Sales application to a Debian 12 VM at `cmc.springupsoftware.com` with both staging and production environments.
## Architecture
- **Production**: `https://cmc.springupsoftware.com`
- **Staging**: `https://staging.cmc.springupsoftware.com`
- **Components**: CakePHP legacy app, Go modern app, MariaDB, Nginx reverse proxy
- **SSL**: Let's Encrypt certificates
- **Authentication**: Basic auth for both environments
## Prerequisites
### Server Setup (Debian 12)
```bash
# Update system
sudo apt update && sudo apt upgrade -y
# Install Docker and Docker Compose
sudo apt install -y docker.io docker-compose-plugin
sudo systemctl enable docker
sudo systemctl start docker
# Add user to docker group
sudo usermod -aG docker $USER
# Log out and back in
# Create backup directory
sudo mkdir -p /var/backups/cmc-sales
sudo chown $USER:$USER /var/backups/cmc-sales
```
### DNS Configuration
Ensure these DNS records point to your server:
- `cmc.springupsoftware.com` → Server IP
- `staging.cmc.springupsoftware.com` → Server IP
## Initial Deployment
### 1. Clone Repository
```bash
cd /home/cmc
git clone git@code.springupsoftware.com:cmc/cmc-sales.git cmc-sales
sudo chown -R $USER:$USER cmc-sales
cd cmc-sales
```
### 2. Environment Configuration
```bash
# Copy environment files
cp .env.staging go-app/.env.staging
cp .env.production go-app/.env.production
# Edit with actual passwords -- up to this.
nano .env.staging
nano .env.production
# Create credential directories
mkdir -p credentials/staging credentials/production
```
### 3. Gmail OAuth Setup
For each environment (staging/production):
1. Go to [Google Cloud Console](https://console.cloud.google.com)
2. Create/select project
3. Enable Gmail API
4. Create OAuth 2.0 credentials
5. Download `credentials.json`
6. Place in appropriate directory:
- Staging: `credentials/staging/credentials.json`
- Production: `credentials/production/credentials.json`
Generate tokens (run on local machine first):
```bash
cd go-app
go run cmd/auth/main.go
# Follow OAuth flow
# Copy token.json to appropriate credential directory
```
### 4. SSL Certificates
```bash
# Start proxy services (includes Lego container)
docker compose -f docker-compose.proxy.yml up -d
# Setup SSL certificates using Lego
./scripts/setup-lego-certs.sh accounts@springupsoftware.com
# Verify certificates
./scripts/lego-list-certs.sh
```
### 5. Database Initialization
```bash
# Start database containers first
docker compose -f docker-compose.staging.yml up -d db-staging
docker compose -f docker-compose.production.yml up -d db-production
# Wait for databases to be ready
sleep 30
# Restore production database (if you have a backup)
./scripts/restore-db.sh production /path/to/backup.sql.gz
# Or initialize empty database and run migrations
# (implementation specific)
```
## Deployment Commands
### Starting Services
```bash
# Start staging environment
docker compose -f docker-compose.staging.yml up -d
# Start production environment
docker compose -f docker-compose.production.yml up -d
# Wait for services to be ready
sleep 10
# Start reverse proxy (after both environments are running)
docker compose -f docker-compose.proxy.yml up -d
# Or use the make command for full stack deployment
make full-stack
```
### Updating Applications
```bash
# Pull latest code
git pull origin main
# Rebuild and restart staging
docker compose -f docker-compose.staging.yml down
docker compose -f docker-compose.staging.yml build --no-cache
docker compose -f docker-compose.staging.yml up -d
# Test staging thoroughly, then update production
docker compose -f docker-compose.production.yml down
docker compose -f docker-compose.production.yml build --no-cache
docker compose -f docker-compose.production.yml up -d
```
## Monitoring and Maintenance
### Health Checks
```bash
# Check all containers
docker ps
# Check logs
docker compose -f docker-compose.production.yml logs -f
# Check application health
curl https://cmc.springupsoftware.com/health
curl https://staging.cmc.springupsoftware.com/health
```
### Database Backups
```bash
# Manual backup
./scripts/backup-db.sh production
./scripts/backup-db.sh staging
# Set up automated backups (cron)
sudo crontab -e
# Add: 0 2 * * * /opt/cmc-sales/scripts/backup-db.sh production
# Add: 0 3 * * * /opt/cmc-sales/scripts/backup-db.sh staging
```
### Log Management
```bash
# View nginx logs
sudo tail -f /var/log/nginx/access.log
sudo tail -f /var/log/nginx/error.log
# View application logs
docker compose -f docker-compose.production.yml logs -f cmc-go-production
docker compose -f docker-compose.staging.yml logs -f cmc-go-staging
```
### SSL Certificate Renewal
```bash
# Manual renewal
./scripts/lego-renew-cert.sh all
# Renew specific domain
./scripts/lego-renew-cert.sh cmc.springupsoftware.com
# Set up auto-renewal (cron)
sudo crontab -e
# Add: 0 2 * * * /opt/cmc-sales/scripts/lego-renew-cert.sh all
```
## Security Considerations
### Basic Authentication
Update passwords in `userpasswd` file:
```bash
# Generate new password hash
sudo apt install apache2-utils
htpasswd -c userpasswd username
# Restart nginx containers
docker compose -f docker-compose.proxy.yml restart nginx-proxy
```
### Database Security
- Use strong passwords in environment files
- Database containers are not exposed externally in production
- Regular backups with encryption at rest
### Network Security
- All traffic encrypted with SSL/TLS
- Rate limiting configured in nginx
- Security headers enabled
- Docker networks isolate environments
## Troubleshooting
### Common Issues
1. **Containers won't start**
```bash
# Check logs
docker compose logs container-name
# Check system resources
df -h
free -h
```
2. **SSL issues**
```bash
# Check certificate status
./scripts/lego-list-certs.sh
# Test SSL configuration
curl -I https://cmc.springupsoftware.com
# Manually renew certificates
./scripts/lego-renew-cert.sh all
```
3. **Database connection issues**
```bash
# Test database connectivity
docker exec -it cmc-db-production mysql -u cmc -p
```
4. **Gmail API issues**
```bash
# Check credentials are mounted
docker exec -it cmc-go-production ls -la /root/credentials/
# Check logs for OAuth errors
docker compose logs cmc-go-production | grep -i gmail
```
### Emergency Procedures
1. **Quick rollback**
```bash
# Stop current containers
docker compose -f docker-compose.production.yml down
# Restore from backup
./scripts/restore-db.sh production /var/backups/cmc-sales/latest_backup.sql.gz
# Start previous version
git checkout previous-commit
docker compose -f docker-compose.production.yml up -d
```
2. **Database corruption**
```bash
# Stop application
docker compose -f docker-compose.production.yml stop cmc-go-production cmc-php-production
# Restore from backup
./scripts/restore-db.sh production /var/backups/cmc-sales/backup_production_YYYYMMDD-HHMMSS.sql.gz
# Restart application
docker compose -f docker-compose.production.yml start cmc-go-production cmc-php-production
```
## File Structure
```
/opt/cmc-sales/
├── docker-compose.staging.yml
├── docker-compose.production.yml
├── docker-compose.proxy.yml
├── conf/
│ ├── nginx-staging.conf
│ ├── nginx-production.conf
│ └── nginx-proxy.conf
├── credentials/
│ ├── staging/
│ │ ├── credentials.json
│ │ └── token.json
│ └── production/
│ ├── credentials.json
│ └── token.json
├── scripts/
│ ├── backup-db.sh
│ ├── restore-db.sh
│ ├── lego-obtain-cert.sh
│ ├── lego-renew-cert.sh
│ ├── lego-list-certs.sh
│ └── setup-lego-certs.sh
└── .env files
```
## Performance Tuning
### Resource Limits
Resource limits are configured in the Docker Compose files:
- Production: 2 CPU cores, 2-4GB RAM per service
- Staging: More relaxed limits for testing
### Database Optimization
```sql
-- Monitor slow queries
SHOW VARIABLES LIKE 'slow_query_log';
SET GLOBAL slow_query_log = 'ON';
SET GLOBAL long_query_time = 2;
-- Check database performance
SHOW PROCESSLIST;
SHOW ENGINE INNODB STATUS;
```
### Nginx Optimization
- Gzip compression enabled
- Static file caching
- Connection keep-alive
- Rate limiting configured

18
MIGRATION.md
View file

@ -1,18 +0,0 @@
# migration instructions
mysql -u cmc -p cmc < ~/migration/latest.sql
MariaDB [(none)]> CREATE USER 'cmc'@'172.17.0.2' IDENTIFIED BY 'somepass';
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> GRANT ALL PRIVILEGES ON cmc.* TO 'cmc'@'172.17.0.2';
www-data@helios:~$ du -hcs vaultmsgs
64G vaultmsgs
64G total
www-data@helios:~$ du -hcs emails
192G emails
192G total
www-data@helios:~$

138
README.md
View file

@ -1,8 +1,22 @@
# cmc-sales # cmc-sales
## Development Setup CMC Sales is a business management system with two applications:
CMC Sales now runs both legacy CakePHP and modern Go applications side by side. - **PHP Application**: CakePHP 1.2.5 (currently the primary application)
- **Go Application**: Go + HTMX (used for select features, growing)
**Future development should be done in the Go application wherever possible.**
## Architecture
Both applications:
- Share the same MariaDB database
- Run behind a shared Caddy reverse proxy with basic authentication
- Support staging and production environments on the same server
The PHP application currently handles most functionality, while the Go application is used for select screens and new features as they're developed.
## Development Setup
### Quick Start ### Quick Start
@ -45,105 +59,53 @@ Both applications share the same database, allowing for gradual migration.
- **Go Application**: Requires Go 1.23+ (for latest sqlc) - **Go Application**: Requires Go 1.23+ (for latest sqlc)
- Alternative: Use `Dockerfile.go.legacy` with Go 1.21 and sqlc v1.26.0 - Alternative: Use `Dockerfile.go.legacy` with Go 1.21 and sqlc v1.26.0
## Deployment
## Install a new server ### Prerequisites
(TODO this is all likely out of date) The deployment scripts use SSH to connect to the server. Configure your SSH config (`~/.ssh/config`) with a host entry named `cmc` pointing to the correct server:
### Requirements
Debian or Ubuntu OS. These instructions written for Debian 9.9
Assumed pre-work:
Create a new VM with hostname newserver.cmctechnologies.com.au
Configure DNS appropriately. cmctechnologies.com.au zones is currently managed in Google Cloud DNS on Karl's account:
https://console.cloud.google.com/net-services/dns/zones/cmctechnologies?project=cmc-technologies&authuser=1&folder&organizationId
Will need to migrate that to CMC's GSuite account at some point.
1. Install ansible on your workstation
```
apt-get install ansible
```
2. Clone the playbooks
```
git clone git@gitlab.com:minimalist.software/cmc-playbooks.git
```
3. Execute the playbooks
The nginx config expects the site to be available at sales.cmctechnologies.com.au.
You'll need to add the hostname to config/nginx-site, if this isn't sales.cmctechnologies.com.au
``` ```
cd cmc-playbooks Host cmc
# Add the hostname of your new server to the inventory.txt HostName node0.prd.springupsoftware.com
ansible-playbook -i inventory.txt setup.yml User cmc
``` IdentityFile ~/.ssh/cmc
4. SSH to the new server and configure gitlab-runner
```
ssh newserver.cmctechnologies.com.au
sudo gitlab-runner register
```
5. SSH to the new server as cmc user
```
ssh cmc@newserver.cmctechnologies.com.au
``` ```
6. Add the SSH key to the cmc-sales repo on gitlab as a deploy key ### Deployment Procedures
https://gitlab.com/minimalist.software/cmc-sales/-/settings/repository
``` Deploy to staging or production using the scripts in the `deploy/` directory:
cmc@cmc:~$ cat .ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFIdoWVp2pGDb46ubW6jkfIpREMa/veD6xZVAtnj3WG1sX7NEUlQYq3RKbZ5CThlw6GKMSYoIsIqk7p6zSoJHGlJSLxoJ0edKflciMUFMTQrdm4T1USXsK+gd0C4DUCyVkYFOs37sy+JtziymnBTm7iOeVI3aMxwfoCOs6mNiD0ettjJT6WtVyy0ZTb6yU4uz7CHj1IGsvwsoKJWPGwJrZ/MfByNl6aJ8R/8zDwbtP06owKD4b3ZPgakM3nYRRoKzHZ/SClz50SXMKC4/nmFY9wLuuMhCWK+9x4/4VPSnxXESOlENMfUoa1IY4osAnZCtaFrWDyenJ+spZrNfgcscD ansible-generated on cmc **Deploy to Staging:**
```bash
./deploy/deploy-stg.sh
``` ```
6. Clone the cmc-sales repo **Deploy to Production:**
``` ```bash
git clone git@gitlab.com:minimalist.software/cmc-sales.git ./deploy/deploy-prod.sh
``` ```
7. As root on new server configure mySQL user cmc **Rebuild without cache (useful after dependency changes):**
Note: get password from app/config/database.php ```bash
(or set a new one and change it) ./deploy/deploy-prod.sh --no-cache
``` ./deploy/deploy-stg.sh --no-cache
# mysql -u root
CREATE USER 'cmc'@'localhost' IDENTIFIED BY 'password';
CREATE USER 'cmc'@'172.17.0.2' IDENTIFIED BY 'password';
CREATE database cmc;
GRANT ALL PRIVILEGES ON cmc.* TO 'cmc'@'localhost';
GRANT ALL PRIVILEGES ON cmc.* TO 'cmc'@'172.17.0.2';
``` ```
8. Get the latest backup from Google Drive ### How Deployment Works
In the shared google drive: 1. The deploy script connects to the server via the `cmc` SSH host
eg. backups/database/backup_20191217_21001.sql.gz 2. Clones or updates the appropriate git branch (`stg` or `prod`)
3. Creates environment configuration for the Go application
4. Builds and starts Docker containers using the appropriate compose file
5. Applications are accessible through Caddy reverse proxy with basic auth
Copy up to the new server: ### Deployment Environments
```
rsync backup_*.gz root@newserver:~/
``` - **Staging**: Branch `stg` → https://stg.cmctechnologies.com.au
- **Production**: Branch `prod` → https://sales.cmctechnologies.com.au or https://prod.cmctechnologies.com.au
9. Restore backup to cmc database Both environments run on the same server and share:
``` - A single Caddy reverse proxy (handles HTTPS and basic authentication for both environments)
zcat backup_* | mysql -u cmc -p - Separate Docker containers for each environment's PHP and Go applications
``` - Separate MariaDB database instances
10. Redeploy from Gitlab
https://gitlab.com/minimalist.software/cmc-sales/pipelines/new
11. You should have a new installation of cmc-sales.
12. Seems new Linux kernels break the docker
https://github.com/moby/moby/issues/28705
13. Mysql needs special args not to break
```
# /etc/mysql/my.cnf
sql_mode=ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
```

394
TESTING_DOCKER.md
View file

@ -1,394 +0,0 @@
# Running CMC Django Tests in Docker
This guide explains how to run the comprehensive CMC Django test suite using Docker for consistent, isolated testing.
## Quick Start
```bash
# 1. Setup test environment (one-time)
./run-tests-docker.sh setup
# 2. Run all tests
./run-tests-docker.sh run
# 3. Run tests with coverage
./run-tests-docker.sh coverage
```
## Test Environment Overview
The Docker test environment includes:
- **Isolated test database** (MariaDB on port 3307)
- **Django test container** with all dependencies
- **Coverage reporting** with HTML and XML output
- **PDF generation testing** with WeasyPrint/ReportLab
- **Parallel test execution** support
## Available Commands
### Setup and Management
```bash
# Build containers and setup test database
./run-tests-docker.sh setup
# Clean up all test containers and data
./run-tests-docker.sh clean
# View test container logs
./run-tests-docker.sh logs
# Open shell in test container
./run-tests-docker.sh shell
```
### Running Tests
```bash
# Run all tests
./run-tests-docker.sh run
# Run specific test suites
./run-tests-docker.sh run models # Model tests only
./run-tests-docker.sh run services # Service layer tests
./run-tests-docker.sh run auth # Authentication tests
./run-tests-docker.sh run views # View and URL tests
./run-tests-docker.sh run pdf # PDF generation tests
./run-tests-docker.sh run integration # Integration tests
# Run quick tests (models + services)
./run-tests-docker.sh quick
# Run tests with coverage reporting
./run-tests-docker.sh coverage
```
## Advanced Test Options
### Using Docker Compose Directly
```bash
# Run specific test with custom options
docker-compose -f docker-compose.test.yml run --rm cmc-django-test \
python cmcsales/manage.py test cmc.tests.test_models --verbosity=2 --keepdb
# Run tests with coverage
docker-compose -f docker-compose.test.yml run --rm cmc-django-test \
coverage run --source='.' cmcsales/manage.py test cmc.tests
# Generate coverage report
docker-compose -f docker-compose.test.yml run --rm cmc-django-test \
coverage report --show-missing
```
### Using the Test Script Directly
```bash
# Inside the container, you can use the test script with advanced options
docker-compose -f docker-compose.test.yml run --rm cmc-django-test \
/app/scripts/run-tests.sh --coverage --keepdb --failfast models
# Script options:
# -c, --coverage Enable coverage reporting
# -k, --keepdb Keep test database between runs
# -p, --parallel NUM Run tests in parallel
# -f, --failfast Stop on first failure
# -v, --verbosity NUM Verbosity level 0-3
```
## Test Suite Structure
### 1. Model Tests (`test_models.py`)
Tests all Django models including:
- Customer, Enquiry, Job, Document models
- Model validation and constraints
- Relationships and cascade behavior
- Financial calculations
```bash
./run-tests-docker.sh run models
```
### 2. Service Tests (`test_services.py`)
Tests business logic layer:
- Number generation service
- Financial calculation service
- Document service workflows
- Validation service
```bash
./run-tests-docker.sh run services
```
### 3. Authentication Tests (`test_authentication.py`)
Tests authentication system:
- Multiple authentication backends
- Permission decorators and middleware
- User management workflows
- Security features
```bash
./run-tests-docker.sh run auth
```
### 4. View Tests (`test_views.py`)
Tests web interface:
- CRUD operations for all entities
- AJAX endpoints
- Permission enforcement
- URL routing
```bash
./run-tests-docker.sh run views
```
### 5. PDF Tests (`test_pdf.py`)
Tests PDF generation:
- WeasyPrint and ReportLab engines
- Template rendering
- Document formatting
- Security and performance
```bash
./run-tests-docker.sh run pdf
```
### 6. Integration Tests (`test_integration.py`)
Tests complete workflows:
- End-to-end business processes
- Multi-user collaboration
- System integration scenarios
- Performance and security
```bash
./run-tests-docker.sh run integration
```
## Test Reports and Coverage
### Coverage Reports
After running tests with coverage, reports are available in:
- **HTML Report**: `./coverage-reports/html/index.html`
- **XML Report**: `./coverage-reports/coverage.xml`
- **Console**: Displayed after test run
```bash
# Run tests with coverage
./run-tests-docker.sh coverage
# View HTML report
open coverage-reports/html/index.html
```
### Test Artifacts
Test outputs are saved to:
- **Test Reports**: `./test-reports/`
- **Coverage Reports**: `./coverage-reports/`
- **Logs**: `./logs/`
- **PDF Test Files**: `./test-reports/pdf/`
## Configuration
### Environment Variables
The test environment uses these key variables:
```yaml
# Database configuration
DATABASE_HOST: test-db
DATABASE_NAME: test_cmc
DATABASE_USER: test_cmc
DATABASE_PASSWORD: testPassword123
# Django settings
DJANGO_SETTINGS_MODULE: cmcsales.settings
TESTING: 1
DEBUG: 0
# PDF generation
PDF_GENERATION_ENGINE: weasyprint
PDF_SAVE_DIRECTORY: /app/test-reports/pdf
```
### Test Database
- **Isolated database** separate from development/production
- **Runs on port 3307** to avoid conflicts
- **Optimized for testing** with reduced buffer sizes
- **Automatically reset** between test runs (unless `--keepdb` used)
## Performance Optimization
### Parallel Test Execution
```bash
# Run tests in parallel (faster execution)
docker-compose -f docker-compose.test.yml run --rm cmc-django-test \
/app/scripts/run-tests.sh --parallel=4 all
```
### Keeping Test Database
```bash
# Keep database between runs for faster subsequent tests
./run-tests-docker.sh run models --keepdb
```
### Quick Test Suite
```bash
# Run only essential tests for rapid feedback
./run-tests-docker.sh quick
```
## Continuous Integration
### GitHub Actions Example
```yaml
name: Test CMC Django
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Setup test environment
run: ./run-tests-docker.sh setup
- name: Run tests with coverage
run: ./run-tests-docker.sh coverage
- name: Upload coverage reports
uses: codecov/codecov-action@v3
with:
file: ./coverage-reports/coverage.xml
```
## Troubleshooting
### Database Connection Issues
```bash
# Check database status
docker-compose -f docker-compose.test.yml ps
# View database logs
docker-compose -f docker-compose.test.yml logs test-db
# Restart database
docker-compose -f docker-compose.test.yml restart test-db
```
### Test Failures
```bash
# Run with maximum verbosity for debugging
docker-compose -f docker-compose.test.yml run --rm cmc-django-test \
python cmcsales/manage.py test cmc.tests.test_models --verbosity=3
# Use failfast to stop on first error
./run-tests-docker.sh run models --failfast
# Open shell to investigate
./run-tests-docker.sh shell
```
### Permission Issues
```bash
# Fix file permissions
sudo chown -R $USER:$USER test-reports coverage-reports logs
# Check Docker permissions
docker-compose -f docker-compose.test.yml run --rm cmc-django-test whoami
```
### Memory Issues
```bash
# Run tests with reduced parallel workers
docker-compose -f docker-compose.test.yml run --rm cmc-django-test \
/app/scripts/run-tests.sh --parallel=1 all
# Monitor resource usage
docker stats
```
## Development Workflow
### Recommended Testing Workflow
1. **Initial Setup** (one-time):
```bash
./run-tests-docker.sh setup
```
2. **During Development** (fast feedback):
```bash
./run-tests-docker.sh quick --keepdb
```
3. **Before Commit** (comprehensive):
```bash
./run-tests-docker.sh coverage
```
4. **Debugging Issues**:
```bash
./run-tests-docker.sh shell
# Inside container:
python cmcsales/manage.py test cmc.tests.test_models.CustomerModelTest.test_customer_creation --verbosity=3
```
### Adding New Tests
1. Create test file in appropriate module
2. Follow existing test patterns and base classes
3. Test locally:
```bash
./run-tests-docker.sh run models --keepdb
```
4. Run full suite before committing:
```bash
./run-tests-docker.sh coverage
```
## Integration with IDE
### PyCharm/IntelliJ
Configure remote interpreter using Docker:
1. Go to Settings → Project → Python Interpreter
2. Add Docker Compose interpreter
3. Use `docker-compose.test.yml` configuration
4. Set service to `cmc-django-test`
### VS Code
Use Dev Containers extension:
1. Create `.devcontainer/devcontainer.json`
2. Configure to use test Docker environment
3. Run tests directly in integrated terminal
## Best Practices
1. **Always run tests in Docker** for consistency
2. **Use `--keepdb` during development** for speed
3. **Run coverage reports before commits**
4. **Clean up regularly** to free disk space
5. **Monitor test performance** and optimize slow tests
6. **Use parallel execution** for large test suites
7. **Keep test data realistic** but minimal
8. **Test error conditions** as well as happy paths
## Resources
- **Django Testing Documentation**: https://docs.djangoproject.com/en/5.1/topics/testing/
- **Coverage.py Documentation**: https://coverage.readthedocs.io/
- **Docker Compose Reference**: https://docs.docker.com/compose/
- **CMC Test Suite Documentation**: See individual test modules for detailed information

0
.gitlab-ci.yml → archive/ci/.gitlab-ci.yml
View file

0
Dockerfile → archive/ci/Dockerfile
View file

0
Dockerfile.go.production → archive/ci/Dockerfile.go.production
View file

0
Dockerfile.go.staging → archive/ci/Dockerfile.go.staging
View file

0
Dockerfile.local.go → archive/ci/Dockerfile.local.go
View file

0
Dockerfile.ubuntu-php → archive/ci/Dockerfile.ubuntu-php
View file

0
docker-compose.production.yml → archive/ci/docker-compose.production.yml
View file

0
docker-compose.proxy.yml → archive/ci/docker-compose.proxy.yml
View file

0
docker-compose.staging.yml → archive/ci/docker-compose.staging.yml
View file

0
Makefile → archive/scripts/Makefile
View file

0
deploy-production.sh → archive/scripts/deploy-production.sh
View file

0
deploy.sh → archive/scripts/deploy.sh
View file

0
enter_docker.sh → archive/scripts/enter_docker.sh
View file

0
fetch_latest_sql.sh → archive/scripts/fetch_latest_sql.sh
View file

0
refresh_data.sh → archive/scripts/refresh_data.sh
View file

0
run_docker_local.sh → archive/scripts/run_docker_local.sh
View file

0
run_docker_prd.sh → archive/scripts/run_docker_prd.sh
View file

0
run_docker_stg.sh → archive/scripts/run_docker_stg.sh
View file

0
run_update_invoices.sh → archive/scripts/run_update_invoices.sh
View file

0
run_vault.sh → archive/scripts/run_vault.sh
View file

0
vault_cron.sh → archive/scripts/vault_cron.sh
View file

55
deploy/Caddyfile Normal file
View file

@ -0,0 +1,55 @@
stg.cmctechnologies.com.au {
reverse_proxy localhost:8081
encode gzip
header {
Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
X-Content-Type-Options "nosniff"
X-Frame-Options "SAMEORIGIN"
X-XSS-Protection "1; mode=block"
}
log {
output file /var/log/caddy/stg-access.log
format console
}
}
mail.stg.cmctechnologies.com.au {
basic_auth {
mailpit $2a$14$yTNicvMBIwF5cBNGnM3Ya.EIagOkP1Y0..qvMfdwUzUoN6Okw.nUG
}
reverse_proxy localhost:8025
}
localhost:2019 {
log_skip
}
prod.cmctechnologies.com.au {
reverse_proxy localhost:8080
encode gzip
header {
Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
X-Content-Type-Options "nosniff"
X-Frame-Options "SAMEORIGIN"
X-XSS-Protection "1; mode=block"
}
log {
output file /var/log/caddy/prod-access.log
format console
}
}
sales.cmctechnologies.com.au {
reverse_proxy localhost:8080
encode gzip
header {
Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
X-Content-Type-Options "nosniff"
X-Frame-Options "SAMEORIGIN"
X-XSS-Protection "1; mode=block"
}
log {
output file /var/log/caddy/prod-access.log
format console
}
}

85
docker-compose.caddy-production.yml
View file

@ -1,85 +0,0 @@
version: '3.8'
services:
cmc-php-production:
build:
context: .
dockerfile: Dockerfile
platform: linux/amd64
container_name: cmc-php-production
depends_on:
- db-production
ports:
- "127.0.0.1:8093:80" # Only accessible from localhost
volumes:
- production_pdf_data:/var/www/cmc-sales/app/webroot/pdf
- production_attachments_data:/var/www/cmc-sales/app/webroot/attachments_files
restart: unless-stopped
environment:
- APP_ENV=production
deploy:
resources:
limits:
cpus: '2.0'
memory: 2G
reservations:
cpus: '0.5'
memory: 512M
db-production:
image: mariadb:latest
container_name: cmc-db-production
environment:
MYSQL_ROOT_PASSWORD: ${DB_ROOT_PASSWORD_PRODUCTION}
MYSQL_DATABASE: cmc
MYSQL_USER: cmc
MYSQL_PASSWORD: ${DB_PASSWORD_PRODUCTION}
volumes:
- production_db_data:/var/lib/mysql
- ./backups:/backups:ro
restart: unless-stopped
# No external port exposure for security
deploy:
resources:
limits:
cpus: '2.0'
memory: 4G
reservations:
cpus: '0.5'
memory: 1G
cmc-go-production:
build:
context: .
dockerfile: Dockerfile.go.production
container_name: cmc-go-production
environment:
DB_HOST: db-production
DB_PORT: 3306
DB_USER: cmc
DB_PASSWORD: ${DB_PASSWORD_PRODUCTION}
DB_NAME: cmc
PORT: 8080
APP_ENV: production
depends_on:
db-production:
condition: service_started
ports:
- "127.0.0.1:8094:8080" # Only accessible from localhost
volumes:
- production_pdf_data:/root/webroot/pdf
- ./credentials/production:/root/credentials:ro
restart: unless-stopped
deploy:
resources:
limits:
cpus: '2.0'
memory: 2G
reservations:
cpus: '0.5'
memory: 512M
volumes:
production_db_data:
production_pdf_data:
production_attachments_data:

65
docker-compose.caddy-staging-ubuntu.yml
View file

@ -1,65 +0,0 @@
version: '3.8'
services:
cmc-php-staging:
build:
context: .
dockerfile: Dockerfile.ubuntu-php
platform: linux/amd64
container_name: cmc-php-staging
depends_on:
- db-staging
ports:
- "127.0.0.1:8091:80"
volumes:
- ./app:/var/www/cmc-sales/app
- staging_pdf_data:/var/www/cmc-sales/app/webroot/pdf
- staging_attachments_data:/var/www/cmc-sales/app/webroot/attachments_files
restart: unless-stopped
environment:
- APP_ENV=staging
- DB_HOST=db-staging
- DB_NAME=cmc_staging
- DB_USER=cmc_staging
- DB_PASSWORD=${DB_PASSWORD_STAGING:-staging_password}
db-staging:
image: mariadb:10.11
container_name: cmc-db-staging
environment:
MYSQL_ROOT_PASSWORD: ${DB_ROOT_PASSWORD_STAGING:-root_password}
MYSQL_DATABASE: cmc_staging
MYSQL_USER: cmc_staging
MYSQL_PASSWORD: ${DB_PASSWORD_STAGING:-staging_password}
volumes:
- staging_db_data:/var/lib/mysql
restart: unless-stopped
ports:
- "127.0.0.1:3307:3306"
cmc-go-staging:
build:
context: .
dockerfile: Dockerfile.go.staging
container_name: cmc-go-staging
environment:
DB_HOST: db-staging
DB_PORT: 3306
DB_USER: cmc_staging
DB_PASSWORD: ${DB_PASSWORD_STAGING:-staging_password}
DB_NAME: cmc_staging
PORT: 8080
APP_ENV: staging
depends_on:
- db-staging
ports:
- "127.0.0.1:8092:8080"
volumes:
- staging_pdf_data:/root/webroot/pdf
- ./credentials/staging:/root/credentials:ro
restart: unless-stopped
volumes:
staging_db_data:
staging_pdf_data:
staging_attachments_data:

61
docker-compose.caddy-staging.yml
View file

@ -1,61 +0,0 @@
version: '3.8'
services:
cmc-php-staging:
build:
context: .
dockerfile: Dockerfile
platform: linux/amd64
container_name: cmc-php-staging
depends_on:
- db-staging
ports:
- "127.0.0.1:8091:80" # Only accessible from localhost
volumes:
- staging_pdf_data:/var/www/cmc-sales/app/webroot/pdf
- staging_attachments_data:/var/www/cmc-sales/app/webroot/attachments_files
restart: unless-stopped
environment:
- APP_ENV=staging
db-staging:
image: mariadb:latest
container_name: cmc-db-staging
environment:
MYSQL_ROOT_PASSWORD: ${DB_ROOT_PASSWORD_STAGING}
MYSQL_DATABASE: cmc_staging
MYSQL_USER: cmc_staging
MYSQL_PASSWORD: ${DB_PASSWORD_STAGING}
volumes:
- staging_db_data:/var/lib/mysql
restart: unless-stopped
ports:
- "127.0.0.1:3307:3306" # Only accessible from localhost
cmc-go-staging:
build:
context: .
dockerfile: Dockerfile.go.staging
container_name: cmc-go-staging
environment:
DB_HOST: db-staging
DB_PORT: 3306
DB_USER: cmc_staging
DB_PASSWORD: ${DB_PASSWORD_STAGING}
DB_NAME: cmc_staging
PORT: 8080
APP_ENV: staging
depends_on:
db-staging:
condition: service_started
ports:
- "127.0.0.1:8092:8080" # Only accessible from localhost
volumes:
- staging_pdf_data:/root/webroot/pdf
- ./credentials/staging:/root/credentials:ro
restart: unless-stopped
volumes:
staging_db_data:
staging_pdf_data:
staging_attachments_data: