docker-compose.prod.yml

@@ -82,8 +82,8 @@ services:
       - "8083:8082"
     volumes:
       - /home/cmc/files/pdf:/root/webroot/pdf:ro
-      - /home/cmc/files/attachments_files:/root/webroot/attachments_files
       - /home/cmc/files/emails:/var/www/emails
+      - /home/cmc/files/attachments_files:/var/www/attachments_files
       - /home/cmc/files/vault:/var/www/vault
       - /home/cmc/files/vaultmsgs:/var/www/vaultmsgs
     networks:

docker-compose.stg.yml

@@ -80,7 +80,6 @@ services:
       - "8082:8082"
     volumes:
       - /home/cmc/files/pdf:/root/webroot/pdf:ro
-      - /home/cmc/files/attachments_files:/root/webroot/attachments_files
     networks:
       - cmc-stg-network
     restart: unless-stopped

go/internal/cmc/handlers/attachments/attachments.go

@@ -172,13 +172,11 @@ func (h *AttachmentHandler) Create(w http.ResponseWriter, r *http.Request) {
 	}
 
 	// Create database record
-	// Store path in PHP format: /var/www/cmc-sales/app/webroot/attachments_files/filename
-	phpPath := "/var/www/cmc-sales/app/webroot/attachments_files/" + filename
 	params := db.CreateAttachmentParams{
 		PrincipleID: int32(principleID),
 		Name:        name,
 		Filename:    handler.Filename,
-		File:        phpPath, // Store PHP container path for compatibility
+		File:        filePath, // Store full path for PHP compatibility
 		Type:        handler.Header.Get("Content-Type"),
 		Size:        int32(handler.Size),
 		Description: description,

scripts/VAULT_PERMISSIONS_FIX.md

@@ -1,250 +0,0 @@
-# Vault Permissions Fix - Attachment Directory Issue
-
-## Problem
-
-The PHP container (`cmc-prod-php`) could not read attachments written by the Go vault because they were being written to **different directories**.
-
-### Root Cause Analysis
-
-**Go Vault Container (`cmc-prod-go`):**
-- Was writing attachments to: `/var/www/emails/` (default `--emaildir` flag)
-- Host directory: `/home/cmc/files/emails`
-- Running as: `root` user (UID 0)
-
-**PHP Application Container (`cmc-prod-php`):**
-- Reads attachments from: `/var/www/cmc-sales/app/webroot/attachments_files/`
-- Host directory: `/home/cmc/files/attachments_files`
-- Running as: `www-data` user (UID 33 or 82)
-
-### Two Issues Identified
-
-1. **Different Directory Paths** (Primary Issue) ❌
-   - Go wrote to `/var/www/emails` → Host: `/home/cmc/files/emails`
-   - PHP read from `/var/www/cmc-sales/app/webroot/attachments_files` → Host: `/home/cmc/files/attachments_files`
-   - These are **completely separate directories on the host**!
-
-2. **File Permissions** (Secondary) ⚠️
-   - Files: `0644` (`-rw-r--r--`) - World readable ✅
-   - Directories: `0755` (`drwxr-xr-x`) - World readable/executable ✅
-   - Owner: `root:root` (from Go container)
-   - Reader: `www-data` (from PHP container)
-   - **Verdict**: Permissions were actually OK (world-readable), but wrong directory!
-
-## Solution Implemented
-
-### Changes Made
-
-**1. Updated Docker Compose ([docker-compose.prod.yml](../docker-compose.prod.yml))**
-
-Added the `attachments_files` mount to the Go container:
-
-```yaml
-cmc-prod-go:
-  volumes:
-    - /home/cmc/files/attachments_files:/var/www/attachments_files  # ← ADDED
-    - /home/cmc/files/emails:/var/www/emails
-    - /home/cmc/files/vault:/var/www/vault
-    - /home/cmc/files/vaultmsgs:/var/www/vaultmsgs
-```
-
-**2. Updated Vault Cron Script ([scripts/vault-cron-prod.sh](vault-cron-prod.sh))**
-
-Changed the `--emaildir` flag to point to the correct directory:
-
-```bash
-docker exec -t "$CONTAINER_NAME" ./vault --mode=local \
-  --vaultdir=/var/www/vaultmsgs/new \
-  --processeddir=/var/www/vaultmsgs/cur \
-  --emaildir=/var/www/attachments_files \  # ← CHANGED from /var/www/emails
-  --dbhost=cmc-prod-db \
-  --dbuser=cmc \
-  --dbpass="xVRQI&cA?7AU=hqJ!%au" \
-  --dbname=cmc
-```
-
-**3. Updated Documentation**
-
-- [VAULT_QUICKSTART.md](VAULT_QUICKSTART.md) - Updated all examples
-- [VAULT_DEPLOYMENT.md](VAULT_DEPLOYMENT.md) - Updated deployment instructions
-
-## Verification Steps
-
-After deploying the fix, verify it works:
-
-### Step 1: Rebuild and Restart Containers
-
-```bash
-cd ~/src/cmc-sales
-docker compose -f docker-compose.prod.yml up -d cmc-prod-go
-```
-
-### Step 2: Verify Mount Points
-
-```bash
-# Check Go container can see the directory
-docker exec -it cmc-prod-go ls -la /var/www/attachments_files
-
-# Check PHP container can see the directory
-docker exec -it cmc-prod-php ls -la /var/www/cmc-sales/app/webroot/attachments_files
-```
-
-Both should show the same files (same inode on host).
-
-### Step 3: Test Vault Processing
-
-```bash
-# Run vault manually
-~/scripts/vault-cron-prod.sh
-
-# Check if files were written
-ls -la /home/cmc/files/attachments_files/$(date +%m-%Y)/
-
-# Check file permissions
-ls -l /home/cmc/files/attachments_files/$(date +%m-%Y)/ | head -5
-```
-
-Expected output:
-```
--rw-r--r-- 1 root root  1234 Nov 23 10:00 abc123-texthtml
--rw-r--r-- 1 root root  5678 Nov 23 10:00 abc123-document.pdf
-```
-
-### Step 4: Verify PHP Can Read Files
-
-```bash
-# Enter PHP container as www-data user
-docker exec -it -u www-data cmc-prod-php sh
-
-# Try to read an attachment file
-cd /var/www/cmc-sales/app/webroot/attachments_files/$(date +%m-%Y)
-ls -la
-cat abc123-texthtml  # Should display content
-exit
-```
-
-### Step 5: Check Database
-
-```bash
-# Verify attachments are in database with correct paths
-docker exec -it cmc-prod-db mariadb -u cmc -p cmc -e "
-SELECT id, email_id, filename, name, size
-FROM email_attachments
-ORDER BY created DESC
-LIMIT 5;
-"
-```
-
-The `name` column should contain paths like: `11-2025/abc123-filename.pdf`
-
-## File Permissions Breakdown
-
-The current permissions (`0644` for files, `0755` for dirs) are correct:
-
-### File Permissions: `0644`
-
-```
--rw-r--r--
-│││││││││
-│││└─┴─┴─── Other (everyone): read (r--)
-││└─┬─┴───── Group: read (r--)
-│└─┬┴─────── Owner: read + write (rw-)
-└─────────── Regular file (-)
-```
-
-**Result**: ✅ All users (including www-data) can **read** the files
-
-### Directory Permissions: `0755`
-
-```
-drwxr-xr-x
-││││││││││
-│││└─┴─┴─── Other: read + execute (r-x) - can list & access
-││└─┬─┴───── Group: read + execute (r-x)
-│└─┬┴─────── Owner: read + write + execute (rwx)
-└─────────── Directory (d)
-```
-
-**Result**: ✅ All users can **navigate into** and **list** the directories
-
-### Why This Works
-
-1. **Files are world-readable** (`r--` in other bits)
-2. **Directories are world-executable** (`x` in other bits)
-3. Even though owner is `root:root`, the "other" permissions allow `www-data` to read
-
-### When You'd Need to Change Permissions
-
-You would **only** need different permissions if:
-
-| Scenario | Change To | Reason |
-|----------|-----------|--------|
-| Only owner should read | `0600` / `0700` | Security/privacy |
-| Group needs write access | `0664` / `0775` | Collaborative editing |
-| Everyone needs write | `0666` / `0777` | ⚠️ **NOT recommended** - security risk |
-
-For your use case: **Keep `0644` / `0755`** ✅
-
-## Database Path Storage
-
-The vault stores relative paths in the database:
-
-```sql
--- email_attachments table
-name: "11-2025/abc123-filename.pdf"
-```
-
-The PHP application prepends the base path:
-
-```php
-$fullPath = "/var/www/cmc-sales/app/webroot/attachments_files/" . $attachment['name'];
-// Results in: /var/www/cmc-sales/app/webroot/attachments_files/11-2025/abc123-filename.pdf
-```
-
-This is why the directory mount must be correct!
-
-## Alternative Solutions (Not Implemented)
-
-### Alternative 1: Symlink (Not Needed)
-
-Could create a symlink from `/var/www/emails` → `/var/www/attachments_files`, but this adds complexity.
-
-### Alternative 2: Change User ID (Overkill)
-
-Could run Go container as `www-data` user, but:
-- Requires rebuilding the Dockerfile
-- Adds unnecessary complexity
-- Current permissions already work
-
-### Alternative 3: Set Group Permissions (Unnecessary)
-
-Could use `0664` / `0775` and shared group, but:
-- World-readable permissions already work
-- No need for write access
-- Simpler is better
-
-## Summary
-
-**What was fixed:**
-- ✅ Go vault now writes to `/var/www/attachments_files` (same as PHP reads from)
-- ✅ Added volume mount for `attachments_files` directory
-- ✅ Updated all scripts and documentation
-
-**What didn't need fixing:**
-- ✅ File permissions (`0644`) are already world-readable
-- ✅ Directory permissions (`0755`) are already world-accessible
-- ✅ No user/group changes needed
-
-**Root cause:**
-- Wrong directory path, NOT permissions issue
-
-## Deployment Checklist
-
-- [ ] Pull latest code with fixes
-- [ ] Restart Go container (`docker compose up -d cmc-prod-go`)
-- [ ] Verify mount points in both containers
-- [ ] Run vault manually to test
-- [ ] Check files appear in `/home/cmc/files/attachments_files/`
-- [ ] Verify PHP can read the files
-- [ ] Check database paths are correct
-- [ ] Enable cron job
-- [ ] Monitor logs for 24 hours

scripts/convert_charset_to_utf8mb4.sql

@@ -1,59 +0,0 @@
-ALTER TABLE `acos` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `addresses` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `aros` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `aros_acos` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `boxes` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `cake_sessions` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `contacts` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `contact_categories` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `costings` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `countries` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `currencies` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `customers` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `customer_categories` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `documents` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `doc_pages` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `emails` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `emails_enquiries` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `emails_invoices` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `emails_jobs` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `emails_purchase_orders` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `email_attachments` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `email_recipients` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `enquiries` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `enquiry_email_queues` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `freight_forwarders` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `freight_services` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `groups` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `industries` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `industries_customers` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `invoices` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `jobs` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `jobstatus` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `jobs_purchase_orders` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `job_products` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `line_items` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `order_acknowledgements` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `packing_lists` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `pages` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `principles` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `principle_addresses` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `principle_contacts` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `products` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `product_attachments` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `product_categories` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `product_options` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `product_options_categories` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `purchase_orders` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `quotes` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `quote_pages` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `quote_products` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `shipments` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `shipments_jobs` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `shipments_principles` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `shipments_purchase_orders` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `shipment_categories` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `shipment_invoices` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `states` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `statuses` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
-ALTER TABLE `users` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;

scripts/deploy/deploy-prod.sh

@@ -79,7 +79,7 @@ ENVEOF
   if [[ "$USE_CACHE" == "false" ]]; then
     echo "Building and starting docker compose for production (no cache)..."
     docker compose --env-file .env.prod -f docker-compose.prod.yml build --no-cache
-    docker compose --env-file .env.prod -f docker-compose.prod.yml up -d --force-recreate --remove-orphans
+    docker compose --env-file .env.prod -f docker-compose.prod.yml up -d --remove-orphans
   else
     echo "Building and starting docker compose for production (using cache)..."
     docker compose --env-file .env.prod -f docker-compose.prod.yml build

scripts/deploy/deploy-stg.sh

@@ -70,7 +70,7 @@ ENVEOF
   if [[ "$USE_CACHE" == "false" ]]; then
     echo "Building and starting docker compose for staging (no cache)..."
     docker compose -f docker-compose.stg.yml build --no-cache
-    docker compose -f docker-compose.stg.yml up -d --force-recreate --remove-orphans
+    docker compose -f docker-compose.stg.yml up -d --remove-orphans
   else
     echo "Building and starting docker compose for staging (using cache)..."
     docker compose -f docker-compose.stg.yml build

scripts/fix_all_corruption.sql

@@ -1,347 +0,0 @@
--- ============================================================================
--- Fix Character Corruption in ALL Tables
--- ============================================================================
--- This script fixes mojibake across the entire database
--- Run this AFTER running scan_all_tables_for_corruption.sql to see what's affected
---
--- IMPORTANT: Create a backup first!
--- docker exec cmc-db mariadb-dump -u root -psecureRootPassword --default-character-set=utf8mb4 cmc | gzip > backup_before_complete_fix_$(date +%Y%m%d).sql.gz
---
--- To run: docker exec -i cmc-db mariadb -u root -psecureRootPassword --default-character-set=utf8mb4 cmc < scripts/fix_all_corruption.sql
--- ============================================================================
-
-SET NAMES utf8mb4;
-
-SELECT '============================================================' as '';
-SELECT 'COMPREHENSIVE CORRUPTION FIX - ALL TABLES' as '';
-SELECT '============================================================' as '';
-
--- ============================================================================
--- Define reusable corruption patterns
--- These are applied to every text/varchar column
--- ============================================================================
-
--- Smart quotes and punctuation
--- ’ → ' (smart apostrophe)
--- “ → " (left double quote)
--- †→ " (right double quote)
--- â€" → – (en dash)
--- â€" → — (em dash)
--- ​ → (zero-width space, remove)
---   → (space)
--- … → …
-
--- Accented characters
--- é → é
--- É → É
--- ó → ó
--- Ã → í
--- ç → ç
--- ü → ü
--- á → á
--- ñ → ñ
--- ö → ö
--- ô → ô
--- ß → ß
--- ä → ä
-
--- Symbols
--- ° → °
--- ® → ®
--- â„¢ → ™
-
--- ============================================================================
--- ADDRESSES Table
--- ============================================================================
-SELECT 'Fixing addresses table...' as '';
-
-UPDATE addresses SET name = REPLACE(name, '’', ''') WHERE name LIKE '%’%';
-UPDATE addresses SET name = REPLACE(name, '“', '"') WHERE name LIKE '%“%';
-UPDATE addresses SET name = REPLACE(name, 'â€', '"') WHERE name LIKE '%â€%';
-UPDATE addresses SET name = REPLACE(name, 'é', 'é') WHERE name LIKE '%é%';
-UPDATE addresses SET name = REPLACE(name, 'ü', 'ü') WHERE name LIKE '%ü%';
-
-UPDATE addresses SET address = REPLACE(address, '’', ''') WHERE address LIKE '%’%';
-UPDATE addresses SET address = REPLACE(address, '“', '"') WHERE address LIKE '%“%';
-UPDATE addresses SET address = REPLACE(address, 'â€', '"') WHERE address LIKE '%â€%';
-UPDATE addresses SET address = REPLACE(address, 'â€"', '–') WHERE address LIKE '%â€"%';
-UPDATE addresses SET address = REPLACE(address, 'â€"', '—') WHERE address LIKE '%â€"%';
-UPDATE addresses SET address = REPLACE(address, 'é', 'é') WHERE address LIKE '%é%';
-UPDATE addresses SET address = REPLACE(address, 'ü', 'ü') WHERE address LIKE '%ü%';
-UPDATE addresses SET address = REPLACE(address, 'ó', 'ó') WHERE address LIKE '%ó%';
-UPDATE addresses SET address = REPLACE(address, 'ñ', 'ñ') WHERE address LIKE '%ñ%';
-
-UPDATE addresses SET city = REPLACE(city, 'é', 'é') WHERE city LIKE '%é%';
-UPDATE addresses SET city = REPLACE(city, 'ü', 'ü') WHERE city LIKE '%ü%';
-UPDATE addresses SET city = REPLACE(city, 'ó', 'ó') WHERE city LIKE '%ó%';
-
--- ============================================================================
--- ATTACHMENTS Table
--- ============================================================================
-SELECT 'Fixing attachments table...' as '';
-
-UPDATE attachments SET name = REPLACE(name, '’', ''') WHERE name LIKE '%’%';
-UPDATE attachments SET name = REPLACE(name, 'é', 'é') WHERE name LIKE '%é%';
-UPDATE attachments SET name = REPLACE(name, 'ü', 'ü') WHERE name LIKE '%ü%';
-
-UPDATE attachments SET description = REPLACE(description, '’', ''') WHERE description LIKE '%’%';
-UPDATE attachments SET description = REPLACE(description, '“', '"') WHERE description LIKE '%“%';
-UPDATE attachments SET description = REPLACE(description, 'â€', '"') WHERE description LIKE '%â€%';
-UPDATE attachments SET description = REPLACE(description, 'é', 'é') WHERE description LIKE '%é%';
-UPDATE attachments SET description = REPLACE(description, 'ü', 'ü') WHERE description LIKE '%ü%';
-
--- ============================================================================
--- CONTACTS Table
--- ============================================================================
-SELECT 'Fixing contacts table...' as '';
-
-UPDATE contacts SET name = REPLACE(name, '’', ''') WHERE name LIKE '%’%';
-UPDATE contacts SET name = REPLACE(name, 'é', 'é') WHERE name LIKE '%é%';
-UPDATE contacts SET name = REPLACE(name, 'ü', 'ü') WHERE name LIKE '%ü%';
-UPDATE contacts SET name = REPLACE(name, 'ó', 'ó') WHERE name LIKE '%ó%';
-
-UPDATE contacts SET first_name = REPLACE(first_name, '’', ''') WHERE first_name LIKE '%’%';
-UPDATE contacts SET first_name = REPLACE(first_name, 'é', 'é') WHERE first_name LIKE '%é%';
-UPDATE contacts SET first_name = REPLACE(first_name, 'ü', 'ü') WHERE first_name LIKE '%ü%';
-
-UPDATE contacts SET last_name = REPLACE(last_name, '’', ''') WHERE last_name LIKE '%’%';
-UPDATE contacts SET last_name = REPLACE(last_name, 'é', 'é') WHERE last_name LIKE '%é%';
-UPDATE contacts SET last_name = REPLACE(last_name, 'ü', 'ü') WHERE last_name LIKE '%ü%';
-
-UPDATE contacts SET notes = REPLACE(notes, '’', ''') WHERE notes LIKE '%’%';
-UPDATE contacts SET notes = REPLACE(notes, '“', '"') WHERE notes LIKE '%“%';
-UPDATE contacts SET notes = REPLACE(notes, 'â€', '"') WHERE notes LIKE '%â€%';
-UPDATE contacts SET notes = REPLACE(notes, 'é', 'é') WHERE notes LIKE '%é%';
-UPDATE contacts SET notes = REPLACE(notes, 'ü', 'ü') WHERE notes LIKE '%ü%';
-
-UPDATE contacts SET job_title = REPLACE(job_title, '’', ''') WHERE job_title LIKE '%’%';
-UPDATE contacts SET job_title = REPLACE(job_title, 'é', 'é') WHERE job_title LIKE '%é%';
-
--- ============================================================================
--- CUSTOMERS Table (re-apply in case some were missed)
--- ============================================================================
-SELECT 'Fixing customers table...' as '';
-
-UPDATE customers SET name = REPLACE(name, '’', ''') WHERE name LIKE '%’%';
-UPDATE customers SET name = REPLACE(name, '“', '"') WHERE name LIKE '%“%';
-UPDATE customers SET name = REPLACE(name, 'â€', '"') WHERE name LIKE '%â€%';
-UPDATE customers SET name = REPLACE(name, 'â€"', '–') WHERE name LIKE '%â€"%';
-UPDATE customers SET name = REPLACE(name, 'â€"', '—') WHERE name LIKE '%â€"%';
-UPDATE customers SET name = REPLACE(name, '​', '') WHERE name LIKE '%​%';
-UPDATE customers SET name = REPLACE(name, 'é', 'é') WHERE name LIKE '%é%';
-UPDATE customers SET name = REPLACE(name, 'ü', 'ü') WHERE name LIKE '%ü%';
-UPDATE customers SET name = REPLACE(name, 'ó', 'ó') WHERE name LIKE '%ó%';
-UPDATE customers SET name = REPLACE(name, 'ç', 'ç') WHERE name LIKE '%ç%';
-
-UPDATE customers SET trading_name = REPLACE(trading_name, '’', ''') WHERE trading_name LIKE '%’%';
-UPDATE customers SET trading_name = REPLACE(trading_name, 'é', 'é') WHERE trading_name LIKE '%é%';
-
-UPDATE customers SET notes = REPLACE(notes, '’', ''') WHERE notes LIKE '%’%';
-UPDATE customers SET notes = REPLACE(notes, '“', '"') WHERE notes LIKE '%“%';
-UPDATE customers SET notes = REPLACE(notes, 'â€', '"') WHERE notes LIKE '%â€%';
-UPDATE customers SET notes = REPLACE(notes, 'é', 'é') WHERE notes LIKE '%é%';
-
--- ============================================================================
--- DOCUMENTS Table
--- ============================================================================
-SELECT 'Fixing documents table...' as '';
-
-UPDATE documents SET shipping_details = REPLACE(shipping_details, '’', ''') WHERE shipping_details LIKE '%’%';
-UPDATE documents SET shipping_details = REPLACE(shipping_details, 'é', 'é') WHERE shipping_details LIKE '%é%';
-
-UPDATE documents SET bill_to = REPLACE(bill_to, '’', ''') WHERE bill_to LIKE '%’%';
-UPDATE documents SET bill_to = REPLACE(bill_to, 'é', 'é') WHERE bill_to LIKE '%é%';
-
-UPDATE documents SET ship_to = REPLACE(ship_to, '’', ''') WHERE ship_to LIKE '%’%';
-UPDATE documents SET ship_to = REPLACE(ship_to, 'é', 'é') WHERE ship_to LIKE '%é%';
-
-UPDATE documents SET subject = REPLACE(subject, '’', ''') WHERE subject LIKE '%’%';
-UPDATE documents SET subject = REPLACE(subject, '“', '"') WHERE subject LIKE '%“%';
-UPDATE documents SET subject = REPLACE(subject, 'â€', '"') WHERE subject LIKE '%â€%';
-UPDATE documents SET subject = REPLACE(subject, 'é', 'é') WHERE subject LIKE '%é%';
-
--- ============================================================================
--- EMAILS Table
--- ============================================================================
-SELECT 'Fixing emails table...' as '';
-
-UPDATE emails SET `from` = REPLACE(`from`, '’', ''') WHERE `from` LIKE '%’%';
-UPDATE emails SET `from` = REPLACE(`from`, 'é', 'é') WHERE `from` LIKE '%é%';
-
-UPDATE emails SET `to` = REPLACE(`to`, '’', ''') WHERE `to` LIKE '%’%';
-UPDATE emails SET `to` = REPLACE(`to`, 'é', 'é') WHERE `to` LIKE '%é%';
-
-UPDATE emails SET subject = REPLACE(subject, '’', ''') WHERE subject LIKE '%’%';
-UPDATE emails SET subject = REPLACE(subject, '“', '"') WHERE subject LIKE '%“%';
-UPDATE emails SET subject = REPLACE(subject, 'â€', '"') WHERE subject LIKE '%â€%';
-UPDATE emails SET subject = REPLACE(subject, 'é', 'é') WHERE subject LIKE '%é%';
-UPDATE emails SET subject = REPLACE(subject, 'ü', 'ü') WHERE subject LIKE '%ü%';
-
-UPDATE emails SET body = REPLACE(body, '’', ''') WHERE body LIKE '%’%';
-UPDATE emails SET body = REPLACE(body, '“', '"') WHERE body LIKE '%“%';
-UPDATE emails SET body = REPLACE(body, 'â€', '"') WHERE body LIKE '%â€%';
-UPDATE emails SET body = REPLACE(body, 'â€"', '–') WHERE body LIKE '%â€"%';
-UPDATE emails SET body = REPLACE(body, 'â€"', '—') WHERE body LIKE '%â€"%';
-UPDATE emails SET body = REPLACE(body, 'é', 'é') WHERE body LIKE '%é%';
-UPDATE emails SET body = REPLACE(body, 'ü', 'ü') WHERE body LIKE '%ü%';
-UPDATE emails SET body = REPLACE(body, '°', '°') WHERE body LIKE '%°%';
-UPDATE emails SET body = REPLACE(body, '®', '®') WHERE body LIKE '%®%';
-
--- ============================================================================
--- ENQUIRIES Table
--- ============================================================================
-SELECT 'Fixing enquiries table...' as '';
-
-UPDATE enquiries SET comments = REPLACE(comments, '’', ''') WHERE comments LIKE '%’%';
-UPDATE enquiries SET comments = REPLACE(comments, '“', '"') WHERE comments LIKE '%“%';
-UPDATE enquiries SET comments = REPLACE(comments, 'â€', '"') WHERE comments LIKE '%â€%';
-UPDATE enquiries SET comments = REPLACE(comments, 'é', 'é') WHERE comments LIKE '%é%';
-UPDATE enquiries SET comments = REPLACE(comments, 'ü', 'ü') WHERE comments LIKE '%ü%';
-
--- ============================================================================
--- INVOICES Table
--- ============================================================================
-SELECT 'Fixing invoices table...' as '';
-
-UPDATE invoices SET ship_via = REPLACE(ship_via, '’', ''') WHERE ship_via LIKE '%’%';
-UPDATE invoices SET ship_via = REPLACE(ship_via, 'é', 'é') WHERE ship_via LIKE '%é%';
-
-UPDATE invoices SET comments = REPLACE(comments, '’', ''') WHERE comments LIKE '%’%';
-UPDATE invoices SET comments = REPLACE(comments, '“', '"') WHERE comments LIKE '%“%';
-UPDATE invoices SET comments = REPLACE(comments, 'â€', '"') WHERE comments LIKE '%â€%';
-UPDATE invoices SET comments = REPLACE(comments, 'é', 'é') WHERE comments LIKE '%é%';
-
--- ============================================================================
--- JOBS Table
--- ============================================================================
-SELECT 'Fixing jobs table...' as '';
-
-UPDATE jobs SET comments = REPLACE(comments, '’', ''') WHERE comments LIKE '%’%';
-UPDATE jobs SET comments = REPLACE(comments, '“', '"') WHERE comments LIKE '%“%';
-UPDATE jobs SET comments = REPLACE(comments, 'â€', '"') WHERE comments LIKE '%â€%';
-UPDATE jobs SET comments = REPLACE(comments, 'é', 'é') WHERE comments LIKE '%é%';
-
--- ============================================================================
--- LINE_ITEMS Table
--- ============================================================================
-SELECT 'Fixing line_items table...' as '';
-
-UPDATE line_items SET name = REPLACE(name, '’', ''') WHERE name LIKE '%’%';
-UPDATE line_items SET name = REPLACE(name, 'é', 'é') WHERE name LIKE '%é%';
-UPDATE line_items SET name = REPLACE(name, '°', '°') WHERE name LIKE '%°%';
-UPDATE line_items SET name = REPLACE(name, '®', '®') WHERE name LIKE '%®%';
-
-UPDATE line_items SET description = REPLACE(description, '’', ''') WHERE description LIKE '%’%';
-UPDATE line_items SET description = REPLACE(description, '“', '"') WHERE description LIKE '%“%';
-UPDATE line_items SET description = REPLACE(description, 'â€', '"') WHERE description LIKE '%â€%';
-UPDATE line_items SET description = REPLACE(description, 'é', 'é') WHERE description LIKE '%é%';
-UPDATE line_items SET description = REPLACE(description, '°', '°') WHERE description LIKE '%°%';
-UPDATE line_items SET description = REPLACE(description, '®', '®') WHERE description LIKE '%®%';
-
--- ============================================================================
--- ORDER_ACKNOWLEDGEMENTS Table
--- ============================================================================
-SELECT 'Fixing order_acknowledgements table...' as '';
-
-UPDATE order_acknowledgements SET comments = REPLACE(comments, '’', ''') WHERE comments LIKE '%’%';
-UPDATE order_acknowledgements SET comments = REPLACE(comments, '“', '"') WHERE comments LIKE '%“%';
-UPDATE order_acknowledgements SET comments = REPLACE(comments, 'â€', '"') WHERE comments LIKE '%â€%';
-UPDATE order_acknowledgements SET comments = REPLACE(comments, 'é', 'é') WHERE comments LIKE '%é%';
-
--- ============================================================================
--- PRINCIPLES Table (re-apply)
--- ============================================================================
-SELECT 'Fixing principles table...' as '';
-
-UPDATE principles SET name = REPLACE(name, 'ü', 'ü') WHERE name LIKE '%ü%';
-UPDATE principles SET name = REPLACE(name, 'é', 'é') WHERE name LIKE '%é%';
-UPDATE principles SET name = REPLACE(name, 'ó', 'ó') WHERE name LIKE '%ó%';
-
-UPDATE principles SET address = REPLACE(address, 'ß', 'ß') WHERE address LIKE '%ß%';
-UPDATE principles SET address = REPLACE(address, 'ü', 'ü') WHERE address LIKE '%ü%';
-UPDATE principles SET address = REPLACE(address, 'é', 'é') WHERE address LIKE '%é%';
-UPDATE principles SET address = REPLACE(address, 'ñ', 'ñ') WHERE address LIKE '%ñ%';
-
-UPDATE principles SET city = REPLACE(city, 'ü', 'ü') WHERE city LIKE '%ü%';
-UPDATE principles SET city = REPLACE(city, 'é', 'é') WHERE city LIKE '%é%';
-
--- ============================================================================
--- PRODUCTS Table (re-apply with all patterns)
--- ============================================================================
-SELECT 'Fixing products table...' as '';
-
--- Mojibake
-UPDATE products SET title = REPLACE(title, '°', '°') WHERE title LIKE '%°%';
-UPDATE products SET title = REPLACE(title, '®', '®') WHERE title LIKE '%®%';
-UPDATE products SET title = REPLACE(title, 'â„¢', '™') WHERE title LIKE '%â„¢%';
-UPDATE products SET title = REPLACE(title, '’', ''') WHERE title LIKE '%’%';
-UPDATE products SET title = REPLACE(title, '“', '"') WHERE title LIKE '%“%';
-UPDATE products SET title = REPLACE(title, 'â€', '"') WHERE title LIKE '%â€%';
-
-UPDATE products SET description = REPLACE(description, '°', '°') WHERE description LIKE '%°%';
-UPDATE products SET description = REPLACE(description, '®', '®') WHERE description LIKE '%®%';
-UPDATE products SET description = REPLACE(description, 'â„¢', '™') WHERE description LIKE '%â„¢%';
-UPDATE products SET description = REPLACE(description, '’', ''') WHERE description LIKE '%’%';
-
-UPDATE products SET item_description = REPLACE(item_description, '°', '°') WHERE item_description LIKE '%°%';
-UPDATE products SET item_description = REPLACE(item_description, '®', '®') WHERE item_description LIKE '%®%';
-
--- HTML entities
-UPDATE products SET title = REPLACE(title, '&deg;', '°') WHERE title LIKE '%&deg;%';
-UPDATE products SET title = REPLACE(title, '&nbsp;', ' ') WHERE title LIKE '%&nbsp;%';
-UPDATE products SET title = REPLACE(title, '&amp;', '&') WHERE title LIKE '%&amp;%';
-
-UPDATE products SET description = REPLACE(description, '&deg;', '°') WHERE description LIKE '%&deg;%';
-UPDATE products SET description = REPLACE(description, '&nbsp;', ' ') WHERE description LIKE '%&nbsp;%';
-UPDATE products SET description = REPLACE(description, '&amp;', '&') WHERE description LIKE '%&amp;%';
-
--- ============================================================================
--- PURCHASE_ORDERS Table
--- ============================================================================
-SELECT 'Fixing purchase_orders table...' as '';
-
-UPDATE purchase_orders SET ship_via = REPLACE(ship_via, '’', ''') WHERE ship_via LIKE '%’%';
-UPDATE purchase_orders SET ship_via = REPLACE(ship_via, 'é', 'é') WHERE ship_via LIKE '%é%';
-
-UPDATE purchase_orders SET comments = REPLACE(comments, '’', ''') WHERE comments LIKE '%’%';
-UPDATE purchase_orders SET comments = REPLACE(comments, '“', '"') WHERE comments LIKE '%“%';
-UPDATE purchase_orders SET comments = REPLACE(comments, 'â€', '"') WHERE comments LIKE '%â€%';
-UPDATE purchase_orders SET comments = REPLACE(comments, 'é', 'é') WHERE comments LIKE '%é%';
-
--- ============================================================================
--- QUOTES Table
--- ============================================================================
-SELECT 'Fixing quotes table...' as '';
-
-UPDATE quotes SET comments = REPLACE(comments, '’', ''') WHERE comments LIKE '%’%';
-UPDATE quotes SET comments = REPLACE(comments, '“', '"') WHERE comments LIKE '%“%';
-UPDATE quotes SET comments = REPLACE(comments, 'â€', '"') WHERE comments LIKE '%â€%';
-UPDATE quotes SET comments = REPLACE(comments, 'é', 'é') WHERE comments LIKE '%é%';
-
-UPDATE quotes SET notes = REPLACE(notes, '’', ''') WHERE notes LIKE '%’%';
-UPDATE quotes SET notes = REPLACE(notes, '“', '"') WHERE notes LIKE '%“%';
-UPDATE quotes SET notes = REPLACE(notes, 'â€', '"') WHERE notes LIKE '%â€%';
-UPDATE quotes SET notes = REPLACE(notes, 'é', 'é') WHERE notes LIKE '%é%';
-
--- ============================================================================
--- SHIPMENTS Table
--- ============================================================================
-SELECT 'Fixing shipments table...' as '';
-
-UPDATE shipments SET ship_via = REPLACE(ship_via, '’', ''') WHERE ship_via LIKE '%’%';
-UPDATE shipments SET ship_via = REPLACE(ship_via, 'é', 'é') WHERE ship_via LIKE '%é%';
-
-UPDATE shipments SET comments = REPLACE(comments, '’', ''') WHERE comments LIKE '%’%';
-UPDATE shipments SET comments = REPLACE(comments, '“', '"') WHERE comments LIKE '%“%';
-UPDATE shipments SET comments = REPLACE(comments, 'â€', '"') WHERE comments LIKE '%â€%';
-UPDATE shipments SET comments = REPLACE(comments, 'é', 'é') WHERE comments LIKE '%é%';
-
--- ============================================================================
--- Verification
--- ============================================================================
-SELECT '' as '';
-SELECT '============================================================' as '';
-SELECT 'FIX COMPLETE - Verification' as '';
-SELECT '============================================================' as '';
-
-SELECT 'Run scripts/scan_all_tables_for_corruption.sql to see remaining corruption' as '';

scripts/fix_all_corruption_hex.sql

@@ -1,336 +0,0 @@
--- ============================================================================
--- Fix Character Corruption in ALL Tables (Using HEX encoding)
--- ============================================================================
--- This version uses HEX-encoded patterns to avoid encoding issues in the SQL file itself
---
--- IMPORTANT: Create a backup first!
--- To run: docker exec -i cmc-db mariadb -u root -psecureRootPassword --default-character-set=utf8mb4 cmc < scripts/fix_all_corruption_hex.sql
--- ============================================================================
-
-SET NAMES utf8mb4;
-
-SELECT '============================================================' as '';
-SELECT 'COMPREHENSIVE CORRUPTION FIX - ALL TABLES (HEX VERSION)' as '';
-SELECT '============================================================' as '';
-
--- ============================================================================
--- HEX Pattern Reference:
--- E28099 = ’ → U+2019 = ' (smart apostrophe)
--- E2809C = “ → U+201C = " (left double quote)
--- E2809D = † → U+201D = " (right double quote)
--- E28093 = â€" → U+2013 = – (en dash)
--- E28094 = â€" → U+2014 = — (em dash)
--- E2809B = ‛ (removed - zero-width space)
--- C3A9 = é → U+00E9 = é
--- C3BC = ü → U+00FC = ü
--- C3B3 = ó → U+00F3 = ó
--- C3AD = í → U+00ED = í
--- C3A7 = ç → U+00E7 = ç
--- C3B1 = ñ → U+00F1 = ñ
--- C3B6 = ö → U+00F6 = ö
--- C39F = ß → U+00DF = ß
--- C2B0 = ° → U+00B0 = °
--- C2AE = ® → U+00AE = ®
--- E284A2 = â„¢ → U+2122 = ™
--- ============================================================================
-
--- ============================================================================
--- ADDRESSES Table
--- ============================================================================
-SELECT 'Fixing addresses table...' as '';
-
-UPDATE addresses SET name = REPLACE(name, UNHEX('E28099'), ''') WHERE name LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE addresses SET name = REPLACE(name, UNHEX('E2809C'), '"') WHERE name LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE addresses SET name = REPLACE(name, UNHEX('E2809D'), '"') WHERE name LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE addresses SET name = REPLACE(name, UNHEX('C3A9'), 'é') WHERE name LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE addresses SET name = REPLACE(name, UNHEX('C3BC'), 'ü') WHERE name LIKE CONCAT('%', UNHEX('C3BC'), '%');
-
-UPDATE addresses SET address = REPLACE(address, UNHEX('E28099'), ''') WHERE address LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE addresses SET address = REPLACE(address, UNHEX('E2809C'), '"') WHERE address LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE addresses SET address = REPLACE(address, UNHEX('E2809D'), '"') WHERE address LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE addresses SET address = REPLACE(address, UNHEX('E28093'), '–') WHERE address LIKE CONCAT('%', UNHEX('E28093'), '%');
-UPDATE addresses SET address = REPLACE(address, UNHEX('E28094'), '—') WHERE address LIKE CONCAT('%', UNHEX('E28094'), '%');
-UPDATE addresses SET address = REPLACE(address, UNHEX('C3A9'), 'é') WHERE address LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE addresses SET address = REPLACE(address, UNHEX('C3BC'), 'ü') WHERE address LIKE CONCAT('%', UNHEX('C3BC'), '%');
-UPDATE addresses SET address = REPLACE(address, UNHEX('C3B3'), 'ó') WHERE address LIKE CONCAT('%', UNHEX('C3B3'), '%');
-UPDATE addresses SET address = REPLACE(address, UNHEX('C3B1'), 'ñ') WHERE address LIKE CONCAT('%', UNHEX('C3B1'), '%');
-
-UPDATE addresses SET city = REPLACE(city, UNHEX('C3A9'), 'é') WHERE city LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE addresses SET city = REPLACE(city, UNHEX('C3BC'), 'ü') WHERE city LIKE CONCAT('%', UNHEX('C3BC'), '%');
-UPDATE addresses SET city = REPLACE(city, UNHEX('C3B3'), 'ó') WHERE city LIKE CONCAT('%', UNHEX('C3B3'), '%');
-
--- ============================================================================
--- ATTACHMENTS Table
--- ============================================================================
-SELECT 'Fixing attachments table...' as '';
-
-UPDATE attachments SET name = REPLACE(name, UNHEX('E28099'), ''') WHERE name LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE attachments SET name = REPLACE(name, UNHEX('C3A9'), 'é') WHERE name LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE attachments SET name = REPLACE(name, UNHEX('C3BC'), 'ü') WHERE name LIKE CONCAT('%', UNHEX('C3BC'), '%');
-
-UPDATE attachments SET description = REPLACE(description, UNHEX('E28099'), ''') WHERE description LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE attachments SET description = REPLACE(description, UNHEX('E2809C'), '"') WHERE description LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE attachments SET description = REPLACE(description, UNHEX('E2809D'), '"') WHERE description LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE attachments SET description = REPLACE(description, UNHEX('C3A9'), 'é') WHERE description LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE attachments SET description = REPLACE(description, UNHEX('C3BC'), 'ü') WHERE description LIKE CONCAT('%', UNHEX('C3BC'), '%');
-
--- ============================================================================
--- CONTACTS Table
--- ============================================================================
-SELECT 'Fixing contacts table...' as '';
-
-UPDATE contacts SET name = REPLACE(name, UNHEX('E28099'), ''') WHERE name LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE contacts SET name = REPLACE(name, UNHEX('C3A9'), 'é') WHERE name LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE contacts SET name = REPLACE(name, UNHEX('C3BC'), 'ü') WHERE name LIKE CONCAT('%', UNHEX('C3BC'), '%');
-UPDATE contacts SET name = REPLACE(name, UNHEX('C3B3'), 'ó') WHERE name LIKE CONCAT('%', UNHEX('C3B3'), '%');
-
-UPDATE contacts SET first_name = REPLACE(first_name, UNHEX('E28099'), ''') WHERE first_name LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE contacts SET first_name = REPLACE(first_name, UNHEX('C3A9'), 'é') WHERE first_name LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE contacts SET first_name = REPLACE(first_name, UNHEX('C3BC'), 'ü') WHERE first_name LIKE CONCAT('%', UNHEX('C3BC'), '%');
-
-UPDATE contacts SET last_name = REPLACE(last_name, UNHEX('E28099'), ''') WHERE last_name LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE contacts SET last_name = REPLACE(last_name, UNHEX('C3A9'), 'é') WHERE last_name LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE contacts SET last_name = REPLACE(last_name, UNHEX('C3BC'), 'ü') WHERE last_name LIKE CONCAT('%', UNHEX('C3BC'), '%');
-
-UPDATE contacts SET notes = REPLACE(notes, UNHEX('E28099'), ''') WHERE notes LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE contacts SET notes = REPLACE(notes, UNHEX('E2809C'), '"') WHERE notes LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE contacts SET notes = REPLACE(notes, UNHEX('E2809D'), '"') WHERE notes LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE contacts SET notes = REPLACE(notes, UNHEX('C3A9'), 'é') WHERE notes LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE contacts SET notes = REPLACE(notes, UNHEX('C3BC'), 'ü') WHERE notes LIKE CONCAT('%', UNHEX('C3BC'), '%');
-
-UPDATE contacts SET job_title = REPLACE(job_title, UNHEX('E28099'), ''') WHERE job_title LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE contacts SET job_title = REPLACE(job_title, UNHEX('C3A9'), 'é') WHERE job_title LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
--- ============================================================================
--- CUSTOMERS Table
--- ============================================================================
-SELECT 'Fixing customers table...' as '';
-
-UPDATE customers SET name = REPLACE(name, UNHEX('E28099'), ''') WHERE name LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE customers SET name = REPLACE(name, UNHEX('E2809C'), '"') WHERE name LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE customers SET name = REPLACE(name, UNHEX('E2809D'), '"') WHERE name LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE customers SET name = REPLACE(name, UNHEX('E28093'), '–') WHERE name LIKE CONCAT('%', UNHEX('E28093'), '%');
-UPDATE customers SET name = REPLACE(name, UNHEX('E28094'), '—') WHERE name LIKE CONCAT('%', UNHEX('E28094'), '%');
-UPDATE customers SET name = REPLACE(name, UNHEX('E2808B'), '') WHERE name LIKE CONCAT('%', UNHEX('E2808B'), '%'); -- zero-width space
-UPDATE customers SET name = REPLACE(name, UNHEX('C3A9'), 'é') WHERE name LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE customers SET name = REPLACE(name, UNHEX('C3BC'), 'ü') WHERE name LIKE CONCAT('%', UNHEX('C3BC'), '%');
-UPDATE customers SET name = REPLACE(name, UNHEX('C3B3'), 'ó') WHERE name LIKE CONCAT('%', UNHEX('C3B3'), '%');
-UPDATE customers SET name = REPLACE(name, UNHEX('C3A7'), 'ç') WHERE name LIKE CONCAT('%', UNHEX('C3A7'), '%');
-UPDATE customers SET name = REPLACE(name, UNHEX('C3B8'), 'ø') WHERE name LIKE CONCAT('%', UNHEX('C3B8'), '%');
-
-UPDATE customers SET trading_name = REPLACE(trading_name, UNHEX('E28099'), ''') WHERE trading_name LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE customers SET trading_name = REPLACE(trading_name, UNHEX('C3A9'), 'é') WHERE trading_name LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
-UPDATE customers SET notes = REPLACE(notes, UNHEX('E28099'), ''') WHERE notes LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE customers SET notes = REPLACE(notes, UNHEX('E2809C'), '"') WHERE notes LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE customers SET notes = REPLACE(notes, UNHEX('E2809D'), '"') WHERE notes LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE customers SET notes = REPLACE(notes, UNHEX('C3A9'), 'é') WHERE notes LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
--- ============================================================================
--- DOCUMENTS Table
--- ============================================================================
-SELECT 'Fixing documents table...' as '';
-
-UPDATE documents SET shipping_details = REPLACE(shipping_details, UNHEX('E28099'), ''') WHERE shipping_details LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE documents SET shipping_details = REPLACE(shipping_details, UNHEX('C3A9'), 'é') WHERE shipping_details LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
-UPDATE documents SET bill_to = REPLACE(bill_to, UNHEX('E28099'), ''') WHERE bill_to LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE documents SET bill_to = REPLACE(bill_to, UNHEX('C3A9'), 'é') WHERE bill_to LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
-UPDATE documents SET ship_to = REPLACE(ship_to, UNHEX('E28099'), ''') WHERE ship_to LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE documents SET ship_to = REPLACE(ship_to, UNHEX('C3A9'), 'é') WHERE ship_to LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
-UPDATE documents SET subject = REPLACE(subject, UNHEX('E28099'), ''') WHERE subject LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE documents SET subject = REPLACE(subject, UNHEX('E2809C'), '"') WHERE subject LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE documents SET subject = REPLACE(subject, UNHEX('E2809D'), '"') WHERE subject LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE documents SET subject = REPLACE(subject, UNHEX('C3A9'), 'é') WHERE subject LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
--- ============================================================================
--- EMAILS Table
--- ============================================================================
-SELECT 'Fixing emails table...' as '';
-
-UPDATE emails SET `from` = REPLACE(`from`, UNHEX('E28099'), ''') WHERE `from` LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE emails SET `from` = REPLACE(`from`, UNHEX('C3A9'), 'é') WHERE `from` LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
-UPDATE emails SET `to` = REPLACE(`to`, UNHEX('E28099'), ''') WHERE `to` LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE emails SET `to` = REPLACE(`to`, UNHEX('C3A9'), 'é') WHERE `to` LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
-UPDATE emails SET subject = REPLACE(subject, UNHEX('E28099'), ''') WHERE subject LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE emails SET subject = REPLACE(subject, UNHEX('E2809C'), '"') WHERE subject LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE emails SET subject = REPLACE(subject, UNHEX('E2809D'), '"') WHERE subject LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE emails SET subject = REPLACE(subject, UNHEX('C3A9'), 'é') WHERE subject LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE emails SET subject = REPLACE(subject, UNHEX('C3BC'), 'ü') WHERE subject LIKE CONCAT('%', UNHEX('C3BC'), '%');
-
-UPDATE emails SET body = REPLACE(body, UNHEX('E28099'), ''') WHERE body LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE emails SET body = REPLACE(body, UNHEX('E2809C'), '"') WHERE body LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE emails SET body = REPLACE(body, UNHEX('E2809D'), '"') WHERE body LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE emails SET body = REPLACE(body, UNHEX('E28093'), '–') WHERE body LIKE CONCAT('%', UNHEX('E28093'), '%');
-UPDATE emails SET body = REPLACE(body, UNHEX('E28094'), '—') WHERE body LIKE CONCAT('%', UNHEX('E28094'), '%');
-UPDATE emails SET body = REPLACE(body, UNHEX('C3A9'), 'é') WHERE body LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE emails SET body = REPLACE(body, UNHEX('C3BC'), 'ü') WHERE body LIKE CONCAT('%', UNHEX('C3BC'), '%');
-UPDATE emails SET body = REPLACE(body, UNHEX('C2B0'), '°') WHERE body LIKE CONCAT('%', UNHEX('C2B0'), '%');
-UPDATE emails SET body = REPLACE(body, UNHEX('C2AE'), '®') WHERE body LIKE CONCAT('%', UNHEX('C2AE'), '%');
-
--- ============================================================================
--- ENQUIRIES Table
--- ============================================================================
-SELECT 'Fixing enquiries table...' as '';
-
-UPDATE enquiries SET comments = REPLACE(comments, UNHEX('E28099'), ''') WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE enquiries SET comments = REPLACE(comments, UNHEX('E2809C'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE enquiries SET comments = REPLACE(comments, UNHEX('E2809D'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE enquiries SET comments = REPLACE(comments, UNHEX('C3A9'), 'é') WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE enquiries SET comments = REPLACE(comments, UNHEX('C3BC'), 'ü') WHERE comments LIKE CONCAT('%', UNHEX('C3BC'), '%');
-
--- ============================================================================
--- INVOICES Table
--- ============================================================================
-SELECT 'Fixing invoices table...' as '';
-
-UPDATE invoices SET ship_via = REPLACE(ship_via, UNHEX('E28099'), ''') WHERE ship_via LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE invoices SET ship_via = REPLACE(ship_via, UNHEX('C3A9'), 'é') WHERE ship_via LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
-UPDATE invoices SET comments = REPLACE(comments, UNHEX('E28099'), ''') WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE invoices SET comments = REPLACE(comments, UNHEX('E2809C'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE invoices SET comments = REPLACE(comments, UNHEX('E2809D'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE invoices SET comments = REPLACE(comments, UNHEX('C3A9'), 'é') WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
--- ============================================================================
--- JOBS Table
--- ============================================================================
-SELECT 'Fixing jobs table...' as '';
-
-UPDATE jobs SET comments = REPLACE(comments, UNHEX('E28099'), ''') WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE jobs SET comments = REPLACE(comments, UNHEX('E2809C'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE jobs SET comments = REPLACE(comments, UNHEX('E2809D'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE jobs SET comments = REPLACE(comments, UNHEX('C3A9'), 'é') WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
--- ============================================================================
--- LINE_ITEMS Table
--- ============================================================================
-SELECT 'Fixing line_items table...' as '';
-
-UPDATE line_items SET name = REPLACE(name, UNHEX('E28099'), ''') WHERE name LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE line_items SET name = REPLACE(name, UNHEX('C3A9'), 'é') WHERE name LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE line_items SET name = REPLACE(name, UNHEX('C2B0'), '°') WHERE name LIKE CONCAT('%', UNHEX('C2B0'), '%');
-UPDATE line_items SET name = REPLACE(name, UNHEX('C2AE'), '®') WHERE name LIKE CONCAT('%', UNHEX('C2AE'), '%');
-
-UPDATE line_items SET description = REPLACE(description, UNHEX('E28099'), ''') WHERE description LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE line_items SET description = REPLACE(description, UNHEX('E2809C'), '"') WHERE description LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE line_items SET description = REPLACE(description, UNHEX('E2809D'), '"') WHERE description LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE line_items SET description = REPLACE(description, UNHEX('C3A9'), 'é') WHERE description LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE line_items SET description = REPLACE(description, UNHEX('C2B0'), '°') WHERE description LIKE CONCAT('%', UNHEX('C2B0'), '%');
-UPDATE line_items SET description = REPLACE(description, UNHEX('C2AE'), '®') WHERE description LIKE CONCAT('%', UNHEX('C2AE'), '%');
-
--- ============================================================================
--- ORDER_ACKNOWLEDGEMENTS Table
--- ============================================================================
-SELECT 'Fixing order_acknowledgements table...' as '';
-
-UPDATE order_acknowledgements SET comments = REPLACE(comments, UNHEX('E28099'), ''') WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE order_acknowledgements SET comments = REPLACE(comments, UNHEX('E2809C'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE order_acknowledgements SET comments = REPLACE(comments, UNHEX('E2809D'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE order_acknowledgements SET comments = REPLACE(comments, UNHEX('C3A9'), 'é') WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
--- ============================================================================
--- PRINCIPLES Table
--- ============================================================================
-SELECT 'Fixing principles table...' as '';
-
-UPDATE principles SET name = REPLACE(name, UNHEX('C3BC'), 'ü') WHERE name LIKE CONCAT('%', UNHEX('C3BC'), '%');
-UPDATE principles SET name = REPLACE(name, UNHEX('C3A9'), 'é') WHERE name LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE principles SET name = REPLACE(name, UNHEX('C3B3'), 'ó') WHERE name LIKE CONCAT('%', UNHEX('C3B3'), '%');
-
-UPDATE principles SET address = REPLACE(address, UNHEX('C39F'), 'ß') WHERE address LIKE CONCAT('%', UNHEX('C39F'), '%');
-UPDATE principles SET address = REPLACE(address, UNHEX('C3BC'), 'ü') WHERE address LIKE CONCAT('%', UNHEX('C3BC'), '%');
-UPDATE principles SET address = REPLACE(address, UNHEX('C3A9'), 'é') WHERE address LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE principles SET address = REPLACE(address, UNHEX('C3B1'), 'ñ') WHERE address LIKE CONCAT('%', UNHEX('C3B1'), '%');
-
-UPDATE principles SET city = REPLACE(city, UNHEX('C3BC'), 'ü') WHERE city LIKE CONCAT('%', UNHEX('C3BC'), '%');
-UPDATE principles SET city = REPLACE(city, UNHEX('C3A9'), 'é') WHERE city LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE principles SET city = REPLACE(city, UNHEX('C3B3'), 'ó') WHERE city LIKE CONCAT('%', UNHEX('C3B3'), '%');
-
--- ============================================================================
--- PRODUCTS Table
--- ============================================================================
-SELECT 'Fixing products table...' as '';
-
--- Mojibake in products
-UPDATE products SET title = REPLACE(title, UNHEX('C2B0'), '°') WHERE title LIKE CONCAT('%', UNHEX('C2B0'), '%');
-UPDATE products SET title = REPLACE(title, UNHEX('C2AE'), '®') WHERE title LIKE CONCAT('%', UNHEX('C2AE'), '%');
-UPDATE products SET title = REPLACE(title, UNHEX('E284A2'), '™') WHERE title LIKE CONCAT('%', UNHEX('E284A2'), '%');
-UPDATE products SET title = REPLACE(title, UNHEX('E28099'), ''') WHERE title LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE products SET title = REPLACE(title, UNHEX('E2809C'), '"') WHERE title LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE products SET title = REPLACE(title, UNHEX('E2809D'), '"') WHERE title LIKE CONCAT('%', UNHEX('E2809D'), '%');
-
-UPDATE products SET description = REPLACE(description, UNHEX('C2B0'), '°') WHERE description LIKE CONCAT('%', UNHEX('C2B0'), '%');
-UPDATE products SET description = REPLACE(description, UNHEX('C2AE'), '®') WHERE description LIKE CONCAT('%', UNHEX('C2AE'), '%');
-UPDATE products SET description = REPLACE(description, UNHEX('E284A2'), '™') WHERE description LIKE CONCAT('%', UNHEX('E284A2'), '%');
-UPDATE products SET description = REPLACE(description, UNHEX('E28099'), ''') WHERE description LIKE CONCAT('%', UNHEX('E28099'), '%');
-
-UPDATE products SET item_description = REPLACE(item_description, UNHEX('C2B0'), '°') WHERE item_description LIKE CONCAT('%', UNHEX('C2B0'), '%');
-UPDATE products SET item_description = REPLACE(item_description, UNHEX('C2AE'), '®') WHERE item_description LIKE CONCAT('%', UNHEX('C2AE'), '%');
-
--- HTML entities in products
-UPDATE products SET title = REPLACE(title, '&deg;', '°') WHERE title LIKE '%&deg;%';
-UPDATE products SET title = REPLACE(title, '&nbsp;', ' ') WHERE title LIKE '%&nbsp;%';
-UPDATE products SET title = REPLACE(title, '&amp;', '&') WHERE title LIKE '%&amp;%';
-UPDATE products SET title = REPLACE(title, '&ndash;', '–') WHERE title LIKE '%&ndash;%';
-UPDATE products SET title = REPLACE(title, '&rdquo;', '"') WHERE title LIKE '%&rdquo;%';
-UPDATE products SET title = REPLACE(title, '&ldquo;', '"') WHERE title LIKE '%&ldquo;%';
-
-UPDATE products SET description = REPLACE(description, '&deg;', '°') WHERE description LIKE '%&deg;%';
-UPDATE products SET description = REPLACE(description, '&nbsp;', ' ') WHERE description LIKE '%&nbsp;%';
-UPDATE products SET description = REPLACE(description, '&amp;', '&') WHERE description LIKE '%&amp;%';
-UPDATE products SET description = REPLACE(description, '&frac12;', '½') WHERE description LIKE '%&frac12;%';
-
--- ============================================================================
--- PURCHASE_ORDERS Table
--- ============================================================================
-SELECT 'Fixing purchase_orders table...' as '';
-
-UPDATE purchase_orders SET ship_via = REPLACE(ship_via, UNHEX('E28099'), ''') WHERE ship_via LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE purchase_orders SET ship_via = REPLACE(ship_via, UNHEX('C3A9'), 'é') WHERE ship_via LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
-UPDATE purchase_orders SET comments = REPLACE(comments, UNHEX('E28099'), ''') WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE purchase_orders SET comments = REPLACE(comments, UNHEX('E2809C'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE purchase_orders SET comments = REPLACE(comments, UNHEX('E2809D'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE purchase_orders SET comments = REPLACE(comments, UNHEX('C3A9'), 'é') WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
--- ============================================================================
--- QUOTES Table
--- ============================================================================
-SELECT 'Fixing quotes table...' as '';
-
-UPDATE quotes SET comments = REPLACE(comments, UNHEX('E28099'), ''') WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE quotes SET comments = REPLACE(comments, UNHEX('E2809C'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE quotes SET comments = REPLACE(comments, UNHEX('E2809D'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE quotes SET comments = REPLACE(comments, UNHEX('C3A9'), 'é') WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
-UPDATE quotes SET notes = REPLACE(notes, UNHEX('E28099'), ''') WHERE notes LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE quotes SET notes = REPLACE(notes, UNHEX('E2809C'), '"') WHERE notes LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE quotes SET notes = REPLACE(notes, UNHEX('E2809D'), '"') WHERE notes LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE quotes SET notes = REPLACE(notes, UNHEX('C3A9'), 'é') WHERE notes LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
--- ============================================================================
--- SHIPMENTS Table
--- ============================================================================
-SELECT 'Fixing shipments table...' as '';
-
-UPDATE shipments SET ship_via = REPLACE(ship_via, UNHEX('E28099'), ''') WHERE ship_via LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE shipments SET ship_via = REPLACE(ship_via, UNHEX('C3A9'), 'é') WHERE ship_via LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
-UPDATE shipments SET comments = REPLACE(comments, UNHEX('E28099'), ''') WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE shipments SET comments = REPLACE(comments, UNHEX('E2809C'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE shipments SET comments = REPLACE(comments, UNHEX('E2809D'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE shipments SET comments = REPLACE(comments, UNHEX('C3A9'), 'é') WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
--- ============================================================================
--- Verification
--- ============================================================================
-SELECT '' as '';
-SELECT '============================================================' as '';
-SELECT 'FIX COMPLETE' as '';
-SELECT '============================================================' as '';
-SELECT 'Run scan_all_tables_for_corruption.sql to verify' as '';

scripts/fix_all_corruption_safe.sql

@@ -1,336 +0,0 @@
--- ============================================================================
--- Fix Character Corruption in ALL Tables (Safe Version with CHAR())
--- ============================================================================
--- This version uses CHAR() with Unicode code points to avoid any encoding issues
---
--- IMPORTANT: Create a backup first!
--- To run: docker exec -i cmc-db mariadb -u root -psecureRootPassword --default-character-set=utf8mb4 cmc < scripts/fix_all_corruption_safe.sql
--- ============================================================================
-
-SET NAMES utf8mb4;
-
-SELECT '============================================================' as '';
-SELECT 'COMPREHENSIVE CORRUPTION FIX - ALL TABLES (SAFE VERSION)' as '';
-SELECT '============================================================' as '';
-
--- ============================================================================
--- Pattern Reference:
--- UNHEX('E28099') = ’ → CHAR(8217 USING utf8mb4) = '
--- UNHEX('E2809C') = “ → CHAR(8220 USING utf8mb4) = "
--- UNHEX('E2809D') = † → CHAR(8221 USING utf8mb4) = "
--- UNHEX('E28093') = â€" → CHAR(8211 USING utf8mb4) = –
--- UNHEX('E28094') = â€" → CHAR(8212 USING utf8mb4) = —
--- UNHEX('E2808B') = zero-width space → (remove)
--- UNHEX('C3A9') = é → CHAR(0x00E9 USING utf8mb4) = é
--- UNHEX('C3BC') = ü → CHAR(0x00FC USING utf8mb4) = ü
--- UNHEX('C3B3') = ó → CHAR(0x00F3 USING utf8mb4) = ó
--- UNHEX('C3B1') = ñ → CHAR(0x00F1 USING utf8mb4) = ñ
--- UNHEX('C3A7') = ç → CHAR(0x00E7 USING utf8mb4) = ç
--- UNHEX('C3B6') = ö → CHAR(0x00F6 USING utf8mb4) = ö
--- UNHEX('C3B8') = ø → CHAR(0x00F8 USING utf8mb4) = ø
--- UNHEX('C39F') = ß → CHAR(0x00DF USING utf8mb4) = ß
--- UNHEX('C2B0') = ° → CHAR(0x00B0 USING utf8mb4) = °
--- UNHEX('C2AE') = ® → CHAR(0x00AE USING utf8mb4) = ®
--- UNHEX('E284A2') = â„¢ → CHAR(8482 USING utf8mb4) = ™
--- ============================================================================
-
--- ============================================================================
--- ADDRESSES Table
--- ============================================================================
-SELECT 'Fixing addresses table...' as '';
-
-UPDATE addresses SET name = REPLACE(name, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE addresses SET name = REPLACE(name, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE addresses SET name = REPLACE(name, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE addresses SET name = REPLACE(name, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE addresses SET name = REPLACE(name, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3BC'), '%');
-
-UPDATE addresses SET address = REPLACE(address, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE address LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE addresses SET address = REPLACE(address, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE address LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE addresses SET address = REPLACE(address, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE address LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE addresses SET address = REPLACE(address, UNHEX('E28093'), CHAR(8211 USING utf8mb4)) WHERE address LIKE CONCAT('%', UNHEX('E28093'), '%');
-UPDATE addresses SET address = REPLACE(address, UNHEX('E28094'), CHAR(8212 USING utf8mb4)) WHERE address LIKE CONCAT('%', UNHEX('E28094'), '%');
-UPDATE addresses SET address = REPLACE(address, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE address LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE addresses SET address = REPLACE(address, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE address LIKE CONCAT('%', UNHEX('C3BC'), '%');
-UPDATE addresses SET address = REPLACE(address, UNHEX('C3B3'), CHAR(0x00F3 USING utf8mb4)) WHERE address LIKE CONCAT('%', UNHEX('C3B3'), '%');
-UPDATE addresses SET address = REPLACE(address, UNHEX('C3B1'), CHAR(0x00F1 USING utf8mb4)) WHERE address LIKE CONCAT('%', UNHEX('C3B1'), '%');
-
-UPDATE addresses SET city = REPLACE(city, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE city LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE addresses SET city = REPLACE(city, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE city LIKE CONCAT('%', UNHEX('C3BC'), '%');
-UPDATE addresses SET city = REPLACE(city, UNHEX('C3B3'), CHAR(0x00F3 USING utf8mb4)) WHERE city LIKE CONCAT('%', UNHEX('C3B3'), '%');
-
--- ============================================================================
--- ATTACHMENTS Table
--- ============================================================================
-SELECT 'Fixing attachments table...' as '';
-
-UPDATE attachments SET name = REPLACE(name, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE attachments SET name = REPLACE(name, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE attachments SET name = REPLACE(name, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3BC'), '%');
-
-UPDATE attachments SET description = REPLACE(description, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE attachments SET description = REPLACE(description, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE attachments SET description = REPLACE(description, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE attachments SET description = REPLACE(description, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE attachments SET description = REPLACE(description, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('C3BC'), '%');
-
--- ============================================================================
--- CONTACTS Table
--- ============================================================================
-SELECT 'Fixing contacts table...' as '';
-
-UPDATE contacts SET name = REPLACE(name, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE contacts SET name = REPLACE(name, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE contacts SET name = REPLACE(name, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3BC'), '%');
-UPDATE contacts SET name = REPLACE(name, UNHEX('C3B3'), CHAR(0x00F3 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3B3'), '%');
-
-UPDATE contacts SET first_name = REPLACE(first_name, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE first_name LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE contacts SET first_name = REPLACE(first_name, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE first_name LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE contacts SET first_name = REPLACE(first_name, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE first_name LIKE CONCAT('%', UNHEX('C3BC'), '%');
-
-UPDATE contacts SET last_name = REPLACE(last_name, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE last_name LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE contacts SET last_name = REPLACE(last_name, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE last_name LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE contacts SET last_name = REPLACE(last_name, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE last_name LIKE CONCAT('%', UNHEX('C3BC'), '%');
-
-UPDATE contacts SET notes = REPLACE(notes, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE notes LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE contacts SET notes = REPLACE(notes, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE notes LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE contacts SET notes = REPLACE(notes, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE notes LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE contacts SET notes = REPLACE(notes, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE notes LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE contacts SET notes = REPLACE(notes, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE notes LIKE CONCAT('%', UNHEX('C3BC'), '%');
-
-UPDATE contacts SET job_title = REPLACE(job_title, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE job_title LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE contacts SET job_title = REPLACE(job_title, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE job_title LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
--- ============================================================================
--- CUSTOMERS Table
--- ============================================================================
-SELECT 'Fixing customers table...' as '';
-
-UPDATE customers SET name = REPLACE(name, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE customers SET name = REPLACE(name, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE customers SET name = REPLACE(name, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE customers SET name = REPLACE(name, UNHEX('E28093'), CHAR(8211 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('E28093'), '%');
-UPDATE customers SET name = REPLACE(name, UNHEX('E28094'), CHAR(8212 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('E28094'), '%');
-UPDATE customers SET name = REPLACE(name, UNHEX('E2808B'), '') WHERE name LIKE CONCAT('%', UNHEX('E2808B'), '%');
-UPDATE customers SET name = REPLACE(name, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE customers SET name = REPLACE(name, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3BC'), '%');
-UPDATE customers SET name = REPLACE(name, UNHEX('C3B3'), CHAR(0x00F3 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3B3'), '%');
-UPDATE customers SET name = REPLACE(name, UNHEX('C3A7'), CHAR(0x00E7 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3A7'), '%');
-UPDATE customers SET name = REPLACE(name, UNHEX('C3B8'), CHAR(0x00F8 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3B8'), '%');
-
-UPDATE customers SET trading_name = REPLACE(trading_name, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE trading_name LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE customers SET trading_name = REPLACE(trading_name, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE trading_name LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
-UPDATE customers SET notes = REPLACE(notes, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE notes LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE customers SET notes = REPLACE(notes, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE notes LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE customers SET notes = REPLACE(notes, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE notes LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE customers SET notes = REPLACE(notes, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE notes LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
--- ============================================================================
--- DOCUMENTS Table
--- ============================================================================
-SELECT 'Fixing documents table...' as '';
-
-UPDATE documents SET shipping_details = REPLACE(shipping_details, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE shipping_details LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE documents SET shipping_details = REPLACE(shipping_details, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE shipping_details LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
-UPDATE documents SET bill_to = REPLACE(bill_to, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE bill_to LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE documents SET bill_to = REPLACE(bill_to, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE bill_to LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
-UPDATE documents SET ship_to = REPLACE(ship_to, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE ship_to LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE documents SET ship_to = REPLACE(ship_to, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE ship_to LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
-UPDATE documents SET subject = REPLACE(subject, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE subject LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE documents SET subject = REPLACE(subject, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE subject LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE documents SET subject = REPLACE(subject, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE subject LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE documents SET subject = REPLACE(subject, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE subject LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
--- ============================================================================
--- EMAILS Table
--- ============================================================================
-SELECT 'Fixing emails table...' as '';
-
-UPDATE emails SET `from` = REPLACE(`from`, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE `from` LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE emails SET `from` = REPLACE(`from`, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE `from` LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
-UPDATE emails SET `to` = REPLACE(`to`, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE `to` LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE emails SET `to` = REPLACE(`to`, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE `to` LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
-UPDATE emails SET subject = REPLACE(subject, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE subject LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE emails SET subject = REPLACE(subject, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE subject LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE emails SET subject = REPLACE(subject, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE subject LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE emails SET subject = REPLACE(subject, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE subject LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE emails SET subject = REPLACE(subject, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE subject LIKE CONCAT('%', UNHEX('C3BC'), '%');
-
-UPDATE emails SET body = REPLACE(body, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE body LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE emails SET body = REPLACE(body, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE body LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE emails SET body = REPLACE(body, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE body LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE emails SET body = REPLACE(body, UNHEX('E28093'), CHAR(8211 USING utf8mb4)) WHERE body LIKE CONCAT('%', UNHEX('E28093'), '%');
-UPDATE emails SET body = REPLACE(body, UNHEX('E28094'), CHAR(8212 USING utf8mb4)) WHERE body LIKE CONCAT('%', UNHEX('E28094'), '%');
-UPDATE emails SET body = REPLACE(body, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE body LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE emails SET body = REPLACE(body, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE body LIKE CONCAT('%', UNHEX('C3BC'), '%');
-UPDATE emails SET body = REPLACE(body, UNHEX('C2B0'), CHAR(0x00B0 USING utf8mb4)) WHERE body LIKE CONCAT('%', UNHEX('C2B0'), '%');
-UPDATE emails SET body = REPLACE(body, UNHEX('C2AE'), CHAR(0x00AE USING utf8mb4)) WHERE body LIKE CONCAT('%', UNHEX('C2AE'), '%');
-
--- ============================================================================
--- ENQUIRIES Table
--- ============================================================================
-SELECT 'Fixing enquiries table...' as '';
-
-UPDATE enquiries SET comments = REPLACE(comments, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE enquiries SET comments = REPLACE(comments, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE enquiries SET comments = REPLACE(comments, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE enquiries SET comments = REPLACE(comments, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE enquiries SET comments = REPLACE(comments, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('C3BC'), '%');
-
--- ============================================================================
--- INVOICES Table
--- ============================================================================
-SELECT 'Fixing invoices table...' as '';
-
-UPDATE invoices SET ship_via = REPLACE(ship_via, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE ship_via LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE invoices SET ship_via = REPLACE(ship_via, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE ship_via LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
-UPDATE invoices SET comments = REPLACE(comments, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE invoices SET comments = REPLACE(comments, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE invoices SET comments = REPLACE(comments, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE invoices SET comments = REPLACE(comments, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
--- ============================================================================
--- JOBS Table
--- ============================================================================
-SELECT 'Fixing jobs table...' as '';
-
-UPDATE jobs SET comments = REPLACE(comments, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE jobs SET comments = REPLACE(comments, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE jobs SET comments = REPLACE(comments, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE jobs SET comments = REPLACE(comments, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
--- ============================================================================
--- LINE_ITEMS Table
--- ============================================================================
-SELECT 'Fixing line_items table...' as '';
-
-UPDATE line_items SET name = REPLACE(name, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE line_items SET name = REPLACE(name, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE line_items SET name = REPLACE(name, UNHEX('C2B0'), CHAR(0x00B0 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C2B0'), '%');
-UPDATE line_items SET name = REPLACE(name, UNHEX('C2AE'), CHAR(0x00AE USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C2AE'), '%');
-
-UPDATE line_items SET description = REPLACE(description, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE line_items SET description = REPLACE(description, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE line_items SET description = REPLACE(description, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE line_items SET description = REPLACE(description, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE line_items SET description = REPLACE(description, UNHEX('C2B0'), CHAR(0x00B0 USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('C2B0'), '%');
-UPDATE line_items SET description = REPLACE(description, UNHEX('C2AE'), CHAR(0x00AE USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('C2AE'), '%');
-
--- ============================================================================
--- ORDER_ACKNOWLEDGEMENTS Table
--- ============================================================================
-SELECT 'Fixing order_acknowledgements table...' as '';
-
-UPDATE order_acknowledgements SET comments = REPLACE(comments, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE order_acknowledgements SET comments = REPLACE(comments, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE order_acknowledgements SET comments = REPLACE(comments, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE order_acknowledgements SET comments = REPLACE(comments, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
--- ============================================================================
--- PRINCIPLES Table
--- ============================================================================
-SELECT 'Fixing principles table...' as '';
-
-UPDATE principles SET name = REPLACE(name, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3BC'), '%');
-UPDATE principles SET name = REPLACE(name, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE principles SET name = REPLACE(name, UNHEX('C3B3'), CHAR(0x00F3 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3B3'), '%');
-
-UPDATE principles SET address = REPLACE(address, UNHEX('C39F'), CHAR(0x00DF USING utf8mb4)) WHERE address LIKE CONCAT('%', UNHEX('C39F'), '%');
-UPDATE principles SET address = REPLACE(address, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE address LIKE CONCAT('%', UNHEX('C3BC'), '%');
-UPDATE principles SET address = REPLACE(address, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE address LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE principles SET address = REPLACE(address, UNHEX('C3B1'), CHAR(0x00F1 USING utf8mb4)) WHERE address LIKE CONCAT('%', UNHEX('C3B1'), '%');
-
-UPDATE principles SET city = REPLACE(city, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE city LIKE CONCAT('%', UNHEX('C3BC'), '%');
-UPDATE principles SET city = REPLACE(city, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE city LIKE CONCAT('%', UNHEX('C3A9'), '%');
-UPDATE principles SET city = REPLACE(city, UNHEX('C3B3'), CHAR(0x00F3 USING utf8mb4)) WHERE city LIKE CONCAT('%', UNHEX('C3B3'), '%');
-
--- ============================================================================
--- PRODUCTS Table
--- ============================================================================
-SELECT 'Fixing products table...' as '';
-
--- Mojibake in products
-UPDATE products SET title = REPLACE(title, UNHEX('C2B0'), CHAR(0x00B0 USING utf8mb4)) WHERE title LIKE CONCAT('%', UNHEX('C2B0'), '%');
-UPDATE products SET title = REPLACE(title, UNHEX('C2AE'), CHAR(0x00AE USING utf8mb4)) WHERE title LIKE CONCAT('%', UNHEX('C2AE'), '%');
-UPDATE products SET title = REPLACE(title, UNHEX('E284A2'), CHAR(8482 USING utf8mb4)) WHERE title LIKE CONCAT('%', UNHEX('E284A2'), '%');
-UPDATE products SET title = REPLACE(title, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE title LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE products SET title = REPLACE(title, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE title LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE products SET title = REPLACE(title, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE title LIKE CONCAT('%', UNHEX('E2809D'), '%');
-
-UPDATE products SET description = REPLACE(description, UNHEX('C2B0'), CHAR(0x00B0 USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('C2B0'), '%');
-UPDATE products SET description = REPLACE(description, UNHEX('C2AE'), CHAR(0x00AE USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('C2AE'), '%');
-UPDATE products SET description = REPLACE(description, UNHEX('E284A2'), CHAR(8482 USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('E284A2'), '%');
-UPDATE products SET description = REPLACE(description, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('E28099'), '%');
-
-UPDATE products SET item_description = REPLACE(item_description, UNHEX('C2B0'), CHAR(0x00B0 USING utf8mb4)) WHERE item_description LIKE CONCAT('%', UNHEX('C2B0'), '%');
-UPDATE products SET item_description = REPLACE(item_description, UNHEX('C2AE'), CHAR(0x00AE USING utf8mb4)) WHERE item_description LIKE CONCAT('%', UNHEX('C2AE'), '%');
-
--- HTML entities in products (these are plain ASCII so no encoding issue)
-UPDATE products SET title = REPLACE(title, '&deg;', CHAR(0x00B0 USING utf8mb4)) WHERE title LIKE '%&deg;%';
-UPDATE products SET title = REPLACE(title, '&nbsp;', ' ') WHERE title LIKE '%&nbsp;%';
-UPDATE products SET title = REPLACE(title, '&amp;', '&') WHERE title LIKE '%&amp;%';
-UPDATE products SET title = REPLACE(title, '&ndash;', CHAR(8211 USING utf8mb4)) WHERE title LIKE '%&ndash;%';
-UPDATE products SET title = REPLACE(title, '&rdquo;', CHAR(8221 USING utf8mb4)) WHERE title LIKE '%&rdquo;%';
-UPDATE products SET title = REPLACE(title, '&ldquo;', CHAR(8220 USING utf8mb4)) WHERE title LIKE '%&ldquo;%';
-
-UPDATE products SET description = REPLACE(description, '&deg;', CHAR(0x00B0 USING utf8mb4)) WHERE description LIKE '%&deg;%';
-UPDATE products SET description = REPLACE(description, '&nbsp;', ' ') WHERE description LIKE '%&nbsp;%';
-UPDATE products SET description = REPLACE(description, '&amp;', '&') WHERE description LIKE '%&amp;%';
-UPDATE products SET description = REPLACE(description, '&frac12;', CHAR(0x00BD USING utf8mb4)) WHERE description LIKE '%&frac12;%';
-
--- ============================================================================
--- PURCHASE_ORDERS Table
--- ============================================================================
-SELECT 'Fixing purchase_orders table...' as '';
-
-UPDATE purchase_orders SET ship_via = REPLACE(ship_via, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE ship_via LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE purchase_orders SET ship_via = REPLACE(ship_via, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE ship_via LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
-UPDATE purchase_orders SET comments = REPLACE(comments, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE purchase_orders SET comments = REPLACE(comments, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE purchase_orders SET comments = REPLACE(comments, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE purchase_orders SET comments = REPLACE(comments, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
--- ============================================================================
--- QUOTES Table
--- ============================================================================
-SELECT 'Fixing quotes table...' as '';
-
-UPDATE quotes SET comments = REPLACE(comments, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE quotes SET comments = REPLACE(comments, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE quotes SET comments = REPLACE(comments, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE quotes SET comments = REPLACE(comments, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
-UPDATE quotes SET notes = REPLACE(notes, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE notes LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE quotes SET notes = REPLACE(notes, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE notes LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE quotes SET notes = REPLACE(notes, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE notes LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE quotes SET notes = REPLACE(notes, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE notes LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
--- ============================================================================
--- SHIPMENTS Table
--- ============================================================================
-SELECT 'Fixing shipments table...' as '';
-
-UPDATE shipments SET ship_via = REPLACE(ship_via, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE ship_via LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE shipments SET ship_via = REPLACE(ship_via, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE ship_via LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
-UPDATE shipments SET comments = REPLACE(comments, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
-UPDATE shipments SET comments = REPLACE(comments, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
-UPDATE shipments SET comments = REPLACE(comments, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
-UPDATE shipments SET comments = REPLACE(comments, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
-
--- ============================================================================
--- Verification
--- ============================================================================
-SELECT '' as '';
-SELECT '============================================================' as '';
-SELECT 'FIX COMPLETE' as '';
-SELECT '============================================================' as '';
-SELECT 'Run scan_all_tables_for_corruption.sql to verify' as '';

scripts/fix_corrupted_data.py

@@ -1,222 +0,0 @@
-#!/usr/bin/env python3
-"""
-Fix Corrupted Character Data in CMC Database
-Uses Python to properly handle UTF-8 encoding
-"""
-
-import subprocess
-import sys
-
-DB_USER = "root"
-DB_PASS = "secureRootPassword"
-DB_NAME = "cmc"
-CONTAINER = "cmc-db"
-
-# Mapping of corrupted patterns to correct values
-CORRUPTION_PATTERNS = {
-    # Smart quotes and punctuation
-    '\u00e2\u0080\u0099': '\u2019',  # ’ → '
-    '\u00e2\u0080\u009c': '\u201c',  # “ → "
-    '\u00e2\u0080\u009d': '\u201d',  # †→ "
-    '\u00e2\u0080\u0093': '\u2013',  # â€" → –
-    '\u00e2\u0080\u0094': '\u2014',  # â€" → —
-    '\u00e2\u0080\u008b': '',         # ​ → (zero-width space, remove)
-    '\u00e2\u0080\u0083': ' ',        #   → (em space, replace with regular space)
-
-    # Accented characters
-    '\u00c3\u00a9': '\u00e9',  # é → é
-    '\u00c3\u0089': '\u00c9',  # É → É
-    '\u00c3\u00b3': '\u00f3',  # ó → ó
-    '\u00c3\u00ad': '\u00ed',  # Ã → í
-    '\u00c3\u00a7': '\u00e7',  # ç → ç
-    '\u00c3\u00bc': '\u00fc',  # ü → ü
-    '\u00c3\u00a1': '\u00e1',  # á → á
-    '\u00c3\u00b1': '\u00f1',  # ñ → ñ
-    '\u00c3\u00b6': '\u00f6',  # ö → ö
-    '\u00c3\u00b4': '\u00f4',  # ô → ô
-    '\u00c3\u009f': '\u00df',  # ß → ß (German sharp s)
-
-    # Turkish characters
-    '\u00c4\u00b0': '\u0130',  # Ä° → İ
-    '\u00c5\u009e': '\u015e',  # Å → Ş
-    '\u00c4\u009e': '\u011e',  # Ä → Ğ
-
-    # Czech characters
-    '\u00c4\u008c': '\u010c',  # Ä → Č
-    '\u00c4\u009b': '\u011b',  # Ä› → ě
-    '\u00c4\u008d': '\u010d',  # Ä → č
-    '\u00c5\u00be': '\u017e',  # ž → ž
-    '\u00c5\u00a1': '\u0161',  # Å¡ → š
-    '\u00c5\u0099': '\u0159',  # Å™ → ř
-    '\u00c5\u0088': '\u0148',  # Å → ň
-
-    # Symbols
-    '\u00c2\u00b0': '\u00b0',  # ° → ° (degree)
-    '\u00c2\u00ae': '\u00ae',  # ® → ® (registered)
-    '\u00e2\u0084\u00a2': '\u2122',  # â„¢ → ™ (trademark)
-}
-
-def run_sql(sql):
-    """Execute SQL command via docker"""
-    cmd = [
-        'docker', 'exec', CONTAINER,
-        'mariadb',
-        '-u', DB_USER,
-        f'-p{DB_PASS}',
-        '--default-character-set=utf8mb4',
-        DB_NAME,
-        '-e', sql
-    ]
-
-    try:
-        result = subprocess.run(cmd, capture_output=True, text=True, encoding='utf-8')
-        if result.returncode != 0:
-            print(f"Error executing SQL: {result.stderr}", file=sys.stderr)
-            return False
-        return True
-    except Exception as e:
-        print(f"Exception: {e}", file=sys.stderr)
-        return False
-
-def fix_table_column(table, column, patterns):
-    """Fix corruption patterns in a specific table column"""
-    print(f"Fixing {table}.{column}...")
-
-    for corrupted, correct in patterns.items():
-        # Escape single quotes for SQL
-        corrupted_esc = corrupted.replace("'", "''")
-        correct_esc = correct.replace("'", "''")
-
-        sql = f"""
-        UPDATE {table}
-        SET {column} = REPLACE({column}, '{corrupted_esc}', '{correct_esc}')
-        WHERE {column} LIKE CONCAT('%', '{corrupted_esc}', '%');
-        """
-
-        if not run_sql(sql):
-            print(f"  Warning: Failed to fix pattern in {table}.{column}")
-
-    print(f"  ✓ {table}.{column} patterns fixed")
-
-def fix_html_entities():
-    """Fix HTML entities that shouldn't be in the database"""
-    print("\nFixing HTML entities in products table...")
-
-    entities = {
-        '&': '&',
-        ' ': ' ',
-        '–': '–',
-        '—': '—',
-        '”': '"',
-        '“': '"',
-        '’': ''',
-        '‘': ''',
-        '°': '°',
-        '½': '½',
-        '¼': '¼',
-        '¾': '¾',
-        '×': '×',
-    }
-
-    for table_col in [('products', 'title'), ('products', 'description'), ('products', 'item_description')]:
-        table, column = table_col
-        for entity, char in entities.items():
-            sql = f"""
-            UPDATE {table}
-            SET {column} = REPLACE({column}, '{entity}', '{char}')
-            WHERE {column} LIKE '%{entity}%';
-            """
-            run_sql(sql)
-
-    print("  ✓ HTML entities fixed")
-
-def fix_specific_records():
-    """Fix specific known corrupted records"""
-    print("\nFixing specific known records...")
-
-    fixes = [
-        ("customers", 253, "name", "SOLUZÉ CIVIL ENGINEERS Trading Under SOLUZE PTY LTD"),
-        ("customers", 1006, "name", "Dr. Prem's Molecules Private Limited (DPMolecules)"),
-        ("customers", 1387, "name", 'DEE ENTERPRISES (QLD) PTY LTD trading as "MEKLEK"'),
-        ("customers", 1608, "name", "Guidera O'Connor Pty Ltd"),
-        ("customers", 2174, "name", "Ingredion ANZ Pty Ltd"),
-        ("customers", 2215, "name", "Vale Nouvelle-Calédonie S.A.S"),
-        ("customers", 2375, "name", "Evaluación y Control Ambiental S.A.S."),
-        ("customers", 3143, "name", "Zontahevy Comércio e Serviços Offshore Ltda"),
-        ("customers", 3529, "name", "Société des Mines de Syama"),
-        ("customers", 4325, "name", "Rambøll Danmark"),
-        ("customers", 4350, "name", "SONNIVA ENERGY MAKİNA İNŞAAT İTH. İHR. LTD. ŞTİ."),
-        ("customers", 4669, "name", "F.C.C. Fluides Conseils Calédonie SARL"),
-        ("customers", 4743, "name", "DGPack Prostějov"),
-        ("customers", 4893, "name", "Oxiquímica, S.A.P.I. de C.V."),
-
-        ("principles", 2, "address", "Heinz-Fangman-Straße 18"),
-        ("principles", 9, "address", "Bezručova 2901\\n756 61 Rožnov pod Radhoštěm"),
-        ("principles", 9, "city", "Rožnov pod Radhoštěm"),
-        ("principles", 13, "name", "IEP Technologies GmbH - BRILEX Gesellschaft für Explosionsschutz mbH"),
-        ("principles", 14, "address", "Liebigstraße 2"),
-        ("principles", 58, "name", "Nosia S.r.l.- Señalización Industrial"),
-        ("principles", 65, "address", "Alte Emser Straße 32"),
-
-        ("addresses", 19, "address", "Lvl 3, Building B, 7–11 Talavera Road"),
-
-        ("products", 30, "title", "C95SN189C DSTGL40/C-Digital temperature probe for P8xx1 Web Sensor. Range -30 to +80°C. With CINCH connector, 1m Cable"),
-        ("products", 38, "title", "Mid-West Instrument Model 240 – SC – 02 – O(TT)"),
-        ("products", 67, "title", "Newson Gale Earth-Rite® II FIBC Static Earthing System, GRP Enclosure with 5m 2 Core Spiral Cable and FIBC Clamp"),
-        ("products", 76, "title", "Newson Gale Earth-Rite® RTR Tester - ER2/CRT"),
-        ("products", 85, "title", "Newson Gale Earth-Rite® RTR™ Tri-Mode Static Grounding System, Metal Enclosure, X90-IP Heavy Duty Clamp with 10m 2 Core Spiral Cable"),
-    ]
-
-    for table, record_id, column, value in fixes:
-        value_esc = value.replace("'", "''")
-        sql = f"UPDATE {table} SET {column} = '{value_esc}' WHERE id = {record_id};"
-        run_sql(sql)
-
-    print(f"  ✓ {len(fixes)} specific records fixed")
-
-def verify():
-    """Verify the fixes"""
-    print("\n" + "="*60)
-    print("Verification Results")
-    print("="*60)
-
-    run_sql("SELECT 'Remaining corrupted customers:', COUNT(*) FROM customers WHERE name REGEXP '[^ -~]';")
-    run_sql("SELECT 'Remaining corrupted principles:', COUNT(*) FROM principles WHERE name REGEXP '[^ -~]' OR address REGEXP '[^ -~]';")
-    run_sql("SELECT 'Remaining corrupted addresses:', COUNT(*) FROM addresses WHERE address REGEXP '[^ -~]';")
-    run_sql("SELECT 'Remaining corrupted products:', COUNT(*) FROM products WHERE title REGEXP '[^ -~]' OR description REGEXP '[^ -~]';")
-
-def main():
-    print("="*60)
-    print("CMC Database Character Corruption Fix")
-    print("="*60)
-    print("")
-
-    # Set charset
-    run_sql("SET NAMES utf8mb4;")
-
-    # Fix pattern-based corruption
-    fix_table_column("customers", "name", CORRUPTION_PATTERNS)
-    fix_table_column("principles", "name", CORRUPTION_PATTERNS)
-    fix_table_column("principles", "address", CORRUPTION_PATTERNS)
-    fix_table_column("principles", "city", CORRUPTION_PATTERNS)
-    fix_table_column("addresses", "address", CORRUPTION_PATTERNS)
-    fix_table_column("addresses", "city", CORRUPTION_PATTERNS)
-    fix_table_column("products", "title", CORRUPTION_PATTERNS)
-    fix_table_column("products", "description", CORRUPTION_PATTERNS)
-    fix_table_column("products", "item_description", CORRUPTION_PATTERNS)
-
-    # Fix HTML entities
-    fix_html_entities()
-
-    # Fix specific records
-    fix_specific_records()
-
-    # Verify
-    verify()
-
-    print("\n" + "="*60)
-    print("Fix complete!")
-    print("="*60)
-
-if __name__ == "__main__":
-    main()

scripts/fix_corrupted_data.sql

@@ -1,317 +0,0 @@
--- ============================================================================
--- Fix Corrupted Character Data in CMC Database
--- ============================================================================
--- This script fixes mojibake (corrupted UTF-8 characters) in the database
--- Run after converting tables to utf8mb4
---
--- IMPORTANT: Review this script and test on a backup first!
--- Create backup: docker exec cmc-db mariadb-dump -u root -psecureRootPassword cmc | gzip > backup_before_fix.sql.gz
---
--- To run: docker exec -i cmc-db mariadb -u root -psecureRootPassword --default-character-set=utf8mb4 cmc < scripts/fix_corrupted_data.sql
--- ============================================================================
-
--- Create a backup table for principles (optional safety measure)
--- DROP TABLE IF EXISTS principles_backup;
--- CREATE TABLE principles_backup AS SELECT * FROM principles;
-
-SET NAMES utf8mb4;
-
--- ============================================================================
--- PART 1: Global Pattern Fixes Using REPLACE
--- ============================================================================
--- These fix common mojibake patterns across all affected tables
-
--- ----------------------------------------------------------------------------
--- Fix CUSTOMERS table
--- ----------------------------------------------------------------------------
-UPDATE customers SET name = REPLACE(name, '’', ''') WHERE name LIKE '%’%';  -- Smart apostrophe
-UPDATE customers SET name = REPLACE(name, '“', '"') WHERE name LIKE '%“%';  -- Left double quote
-UPDATE customers SET name = REPLACE(name, 'â€', '"') WHERE name LIKE '%â€%';  -- Right double quote
-UPDATE customers SET name = REPLACE(name, 'â€"', '–') WHERE name LIKE '%â€"%';  -- En dash
-UPDATE customers SET name = REPLACE(name, 'â€"', '—') WHERE name LIKE '%â€"%';  -- Em dash
-UPDATE customers SET name = REPLACE(name, '​', '') WHERE name LIKE '%​%';   -- Zero-width space (remove)
-UPDATE customers SET name = REPLACE(name, ' ', ' ') WHERE name LIKE '% %';  -- Special space
-UPDATE customers SET name = REPLACE(name, 'é', 'é') WHERE name LIKE '%é%';   -- é
-UPDATE customers SET name = REPLACE(name, 'É', 'É') WHERE name LIKE '%É%';   -- É
-UPDATE customers SET name = REPLACE(name, 'ó', 'ó') WHERE name LIKE '%ó%';   -- ó
-UPDATE customers SET name = REPLACE(name, 'Ã', 'í') WHERE name LIKE '%Ã%';   -- í
-UPDATE customers SET name = REPLACE(name, 'ç', 'ç') WHERE name LIKE '%ç%';   -- ç
-UPDATE customers SET name = REPLACE(name, 'ü', 'ü') WHERE name LIKE '%ü%';   -- ü
-UPDATE customers SET name = REPLACE(name, 'á', 'á') WHERE name LIKE '%á%';   -- á
-UPDATE customers SET name = REPLACE(name, 'ñ', 'ñ') WHERE name LIKE '%ñ%';   -- ñ
-UPDATE customers SET name = REPLACE(name, 'ö', 'ö') WHERE name LIKE '%ö%';   -- ö
-UPDATE customers SET name = REPLACE(name, 'ô', 'ô') WHERE name LIKE '%ô%';   -- ô
-UPDATE customers SET name = REPLACE(name, 'Ä°', 'İ') WHERE name LIKE '%Ä°%';   -- Turkish İ
-UPDATE customers SET name = REPLACE(name, 'Å', 'Ş') WHERE name LIKE '%Å%';   -- Turkish Ş
-UPDATE customers SET name = REPLACE(name, 'Ä', 'Ğ') WHERE name LIKE '%Ä%';   -- Turkish Ğ
-UPDATE customers SET name = REPLACE(name, 'Ä', 'Č') WHERE name LIKE '%Ä%';   -- Czech Č
-UPDATE customers SET name = REPLACE(name, 'Ä›', 'ě') WHERE name LIKE '%Ä›%';   -- Czech ě
-
--- Remove trailing tabs
-UPDATE customers SET name = TRIM(TRAILING '\t' FROM name) WHERE name LIKE '%\t';
-UPDATE customers SET trading_name = TRIM(TRAILING '\t' FROM trading_name) WHERE trading_name LIKE '%\t';
-
--- ----------------------------------------------------------------------------
--- Fix PRINCIPLES table
--- ----------------------------------------------------------------------------
-UPDATE principles SET name = REPLACE(name, 'ü', 'ü') WHERE name LIKE '%ü%';
-UPDATE principles SET name = REPLACE(name, 'é', 'é') WHERE name LIKE '%é%';
-UPDATE principles SET name = REPLACE(name, 'ó', 'ó') WHERE name LIKE '%ó%';
-
-UPDATE principles SET address = REPLACE(address, 'ß', 'ß') WHERE address LIKE '%ß%';  -- German ß
-UPDATE principles SET address = REPLACE(address, 'ü', 'ü') WHERE address LIKE '%ü%';
-UPDATE principles SET address = REPLACE(address, 'é', 'é') WHERE address LIKE '%é%';
-UPDATE principles SET address = REPLACE(address, 'ñ', 'ñ') WHERE address LIKE '%ñ%';
-UPDATE principles SET address = REPLACE(address, 'Ä', 'č') WHERE address LIKE '%Ä%';   -- Czech č
-UPDATE principles SET address = REPLACE(address, 'ž', 'ž') WHERE address LIKE '%ž%';  -- Czech ž
-UPDATE principles SET address = REPLACE(address, 'Å¡', 'š') WHERE address LIKE '%Å¡%';  -- Czech š
-UPDATE principles SET address = REPLACE(address, 'Ä›', 'ě') WHERE address LIKE '%Ä›%';  -- Czech ě
-UPDATE principles SET address = REPLACE(address, 'Å', 'ň') WHERE address LIKE '%Å%';  -- Czech ň
-UPDATE principles SET address = REPLACE(address, 'Å™', 'ř') WHERE address LIKE '%Å™%';  -- Czech ř
-
-UPDATE principles SET city = REPLACE(city, 'ü', 'ü') WHERE city LIKE '%ü%';
-UPDATE principles SET city = REPLACE(city, 'é', 'é') WHERE city LIKE '%é%';
-UPDATE principles SET city = REPLACE(city, 'ó', 'ó') WHERE city LIKE '%ó%';
-
--- ----------------------------------------------------------------------------
--- Fix ADDRESSES table
--- ----------------------------------------------------------------------------
-UPDATE addresses SET address = REPLACE(address, 'â€"', '–') WHERE address LIKE '%â€"%';  -- En dash
-UPDATE addresses SET address = REPLACE(address, 'â€"', '—') WHERE address LIKE '%â€"%';  -- Em dash
-UPDATE addresses SET address = REPLACE(address, 'é', 'é') WHERE address LIKE '%é%';
-UPDATE addresses SET address = REPLACE(address, 'ü', 'ü') WHERE address LIKE '%ü%';
-
-UPDATE addresses SET city = REPLACE(city, 'é', 'é') WHERE city LIKE '%é%';
-UPDATE addresses SET city = REPLACE(city, 'ü', 'ü') WHERE city LIKE '%ü%';
-
--- ============================================================================
--- PART 2: Specific Record Fixes
--- ============================================================================
--- These fix specific known corrupted records where pattern matching isn't enough
-
--- ----------------------------------------------------------------------------
--- PRINCIPLES - Specific fixes
--- ----------------------------------------------------------------------------
-
--- ID 2: German ß
-UPDATE principles
-SET address = 'Heinz-Fangman-Straße 18'
-WHERE id = 2;
-
--- ID 9: Czech characters (this is complex, needs full correction)
-UPDATE principles
-SET address = 'Bezručova 2901\n756 61 Rožnov pod Radhoštěm',
-    city = 'Rožnov pod Radhoštěm'
-WHERE id = 9;
-
--- ID 13: German ü
-UPDATE principles
-SET name = 'IEP Technologies GmbH - BRILEX Gesellschaft für Explosionsschutz mbH'
-WHERE id = 13;
-
--- ID 14: German ß
-UPDATE principles
-SET address = 'Liebigstraße 2'
-WHERE id = 14;
-
--- ID 58: Spanish ó
-UPDATE principles
-SET name = 'Nosia S.r.l.- Señalización Industrial'
-WHERE id = 58;
-
--- ID 65: German ß
-UPDATE principles
-SET address = 'Alte Emser Straße 32'
-WHERE id = 65;
-
--- ----------------------------------------------------------------------------
--- CUSTOMERS - Specific fixes
--- ----------------------------------------------------------------------------
-
--- ID 253: É
-UPDATE customers
-SET name = 'SOLUZÉ CIVIL ENGINEERS Trading Under SOLUZE PTY LTD'
-WHERE id = 253;
-
--- ID 1006: Smart apostrophe
-UPDATE customers
-SET name = 'Dr. Prem''s Molecules Private Limited (DPMolecules)'
-WHERE id = 1006;
-
--- ID 1387: Smart quotes
-UPDATE customers
-SET name = 'DEE ENTERPRISES (QLD) PTY LTD trading as "MEKLEK"'
-WHERE id = 1387;
-
--- ID 1608: Smart apostrophe
-UPDATE customers
-SET name = 'Guidera O''Connor Pty Ltd'
-WHERE id = 1608;
-
--- ID 2174: Zero-width space
-UPDATE customers
-SET name = 'Ingredion ANZ Pty Ltd'
-WHERE id = 2174;
-
--- ID 2215: French é
-UPDATE customers
-SET name = 'Vale Nouvelle-Calédonie S.A.S'
-WHERE id = 2215;
-
--- ID 2375: Spanish ó
-UPDATE customers
-SET name = 'Evaluación y Control Ambiental S.A.S.'
-WHERE id = 2375;
-
--- ID 3143: Portuguese ç and ó
-UPDATE customers
-SET name = 'Zontahevy Comércio e Serviços Offshore Ltda'
-WHERE id = 3143;
-
--- ID 3529: French é
-UPDATE customers
-SET name = 'Société des Mines de Syama'
-WHERE id = 3529;
-
--- ID 3633: Special space (em space)
-UPDATE customers
-SET name = 'P.J. Berriman & Co PTY LTD'
-WHERE id = 3633;
-
--- ID 4325: Danish ø
-UPDATE customers
-SET name = 'Rambøll Danmark'
-WHERE id = 4325;
-
--- ID 4350: Turkish characters İ, Ş, Ğ
-UPDATE customers
-SET name = 'SONNIVA ENERGY MAKİNA İNŞAAT İTH. İHR. LTD. ŞTİ.'
-WHERE id = 4350;
-
--- ID 4669: French é
-UPDATE customers
-SET name = 'F.C.C. Fluides Conseils Calédonie SARL'
-WHERE id = 4669;
-
--- ID 4743: Czech ě
-UPDATE customers
-SET name = 'DGPack Prostějov'
-WHERE id = 4743;
-
--- ID 4764: Zero-width space
-UPDATE customers
-SET name = 'Mccready Welding Pty Ltd Trading under the entity Mccready''s Welding Services'
-WHERE id = 4764;
-
--- ID 4893: Spanish í
-UPDATE customers
-SET name = 'Oxiquímica, S.A.P.I. de C.V.'
-WHERE id = 4893;
-
--- ----------------------------------------------------------------------------
--- ADDRESSES - Specific fixes
--- ----------------------------------------------------------------------------
-
--- ID 19: En dash (7–11)
-UPDATE addresses
-SET address = 'Lvl 3, Building B, 7–11 Talavera Road'
-WHERE id = 19;
-
--- ============================================================================
--- PART 3: Fix PRODUCTS Table (12,536+ corrupted records)
--- ============================================================================
--- Products table has extensive corruption from mojibake and HTML entities
-
--- Degree symbol: ° → °
-UPDATE products SET title = REPLACE(title, '°', '°') WHERE title LIKE '%°%';
-UPDATE products SET description = REPLACE(description, '°', '°') WHERE description LIKE '%°%';
-UPDATE products SET item_description = REPLACE(item_description, '°', '°') WHERE item_description LIKE '%°%';
-
--- Registered trademark: ® → ®
-UPDATE products SET title = REPLACE(title, '®', '®') WHERE title LIKE '%®%';
-UPDATE products SET description = REPLACE(description, '®', '®') WHERE description LIKE '%®%';
-UPDATE products SET item_description = REPLACE(item_description, '®', '®') WHERE item_description LIKE '%®%';
-
--- Trademark: â„¢ → ™
-UPDATE products SET title = REPLACE(title, 'â„¢', '™') WHERE title LIKE '%â„¢%';
-UPDATE products SET description = REPLACE(description, 'â„¢', '™') WHERE description LIKE '%â„¢%';
-UPDATE products SET item_description = REPLACE(item_description, 'â„¢', '™') WHERE item_description LIKE '%â„¢%';
-
--- Smart quotes and dashes in products
-UPDATE products SET title = REPLACE(title, '’', ''') WHERE title LIKE '%’%';
-UPDATE products SET description = REPLACE(description, '’', ''') WHERE description LIKE '%’%';
-UPDATE products SET title = REPLACE(title, '“', '"') WHERE title LIKE '%“%';
-UPDATE products SET description = REPLACE(description, '“', '"') WHERE description LIKE '%“%';
-UPDATE products SET title = REPLACE(title, 'â€', '"') WHERE title LIKE '%â€%';
-UPDATE products SET description = REPLACE(description, 'â€', '"') WHERE description LIKE '%â€%';
-UPDATE products SET title = REPLACE(title, 'â€"', '–') WHERE title LIKE '%â€"%';
-UPDATE products SET description = REPLACE(description, 'â€"', '–') WHERE description LIKE '%â€"%';
-UPDATE products SET title = REPLACE(title, 'â€"', '—') WHERE title LIKE '%â€"%';
-UPDATE products SET description = REPLACE(description, 'â€"', '—') WHERE description LIKE '%â€"%';
-
--- Accented characters in products
-UPDATE products SET title = REPLACE(title, 'é', 'é') WHERE title LIKE '%é%';
-UPDATE products SET description = REPLACE(description, 'é', 'é') WHERE description LIKE '%é%';
-UPDATE products SET title = REPLACE(title, 'ü', 'ü') WHERE title LIKE '%ü%';
-UPDATE products SET description = REPLACE(description, 'ü', 'ü') WHERE description LIKE '%ü%';
-
--- HTML entities (shouldn't be in database)
-UPDATE products SET title = REPLACE(title, '&ndash;', '–') WHERE title LIKE '%&ndash;%';
-UPDATE products SET description = REPLACE(description, '&ndash;', '–') WHERE description LIKE '%&ndash;%';
-UPDATE products SET title = REPLACE(title, '&rdquo;', '"') WHERE title LIKE '%&rdquo;%';
-UPDATE products SET description = REPLACE(description, '&rdquo;', '"') WHERE description LIKE '%&rdquo;%';
-UPDATE products SET title = REPLACE(title, '&ldquo;', '"') WHERE title LIKE '%&ldquo;%';
-UPDATE products SET description = REPLACE(description, '&ldquo;', '"') WHERE description LIKE '%&ldquo;%';
-UPDATE products SET title = REPLACE(title, '&amp;', '&') WHERE title LIKE '%&amp;%';
-UPDATE products SET description = REPLACE(description, '&amp;', '&') WHERE description LIKE '%&amp;%';
-UPDATE products SET title = REPLACE(title, '&frac12;', '½') WHERE title LIKE '%&frac12;%';
-UPDATE products SET description = REPLACE(description, '&frac12;', '½') WHERE description LIKE '%&frac12;%';
-UPDATE products SET title = REPLACE(title, '&deg;', '°') WHERE title LIKE '%&deg;%';
-UPDATE products SET description = REPLACE(description, '&deg;', '°') WHERE description LIKE '%&deg;%';
-UPDATE products SET title = REPLACE(title, '&nbsp;', ' ') WHERE title LIKE '%&nbsp;%';
-UPDATE products SET description = REPLACE(description, '&nbsp;', ' ') WHERE description LIKE '%&nbsp;%';
-
--- Specific product fixes
-UPDATE products SET title = 'C95SN189C DSTGL40/C-Digital temperature probe for P8xx1 Web Sensor. Range -30 to +80°C. With CINCH connector, 1m Cable' WHERE id = 30;
-UPDATE products SET title = 'Mid-West Instrument Model 240 – SC – 02 – O(TT)' WHERE id = 38;
-UPDATE products SET title = 'Newson Gale Earth-Rite® II FIBC Static Earthing System, GRP Enclosure with 5m 2 Core Spiral Cable and FIBC Clamp' WHERE id = 67;
-UPDATE products SET title = 'Newson Gale Earth-Rite® RTR™ Tri-Mode Static Grounding System, Metal Enclosure, X90-IP Heavy Duty Clamp with 10m 2 Core Spiral Cable' WHERE id = 85;
-
--- ============================================================================
--- PART 4: Verification Queries
--- ============================================================================
--- Run these after the fixes to verify they worked
-
--- Check fixed principles
-SELECT id, name, city FROM principles WHERE id IN (2, 9, 13, 14, 58, 65);
-
--- Check fixed customers
-SELECT id, name FROM customers WHERE id IN (253, 1006, 1387, 1608, 2174, 2215, 2375, 3143, 3529, 4325, 4350, 4669, 4743, 4764, 4893);
-
--- Check fixed addresses
-SELECT id, address FROM addresses WHERE id = 19;
-
--- Count remaining corrupted records (should be much lower)
-SELECT COUNT(*) as remaining_corrupted_principles
-FROM principles
-WHERE name REGEXP '[^ -~]' OR address REGEXP '[^ -~]' OR city REGEXP '[^ -~]';
-
-SELECT COUNT(*) as remaining_corrupted_customers
-FROM customers
-WHERE name REGEXP '[^ -~]';
-
-SELECT COUNT(*) as remaining_corrupted_addresses
-FROM addresses
-WHERE address REGEXP '[^ -~]' OR city REGEXP '[^ -~]';
-
-SELECT COUNT(*) as remaining_corrupted_products
-FROM products
-WHERE title REGEXP '[^ -~]' OR description REGEXP '[^ -~]' OR item_description REGEXP '[^ -~]';
-
--- Check sample fixed products
-SELECT id, title FROM products WHERE id IN (30, 38, 67, 85);
-
--- ============================================================================
--- COMPLETION
--- ============================================================================
-SELECT 'Fix script completed! Review the verification queries above.' as status;

scripts/fix_corrupted_data_hex.sql

@@ -1,221 +0,0 @@
--- ============================================================================
--- Fix Corrupted Character Data in CMC Database (Using HEX encoding)
--- ============================================================================
--- This script fixes mojibake using HEX encoding to avoid UTF-8 interpretation issues
--- Run after converting tables to utf8mb4
---
--- IMPORTANT: Review this script and test on a backup first!
--- To run: docker exec -i cmc-db mariadb -u root -psecureRootPassword --default-character-set=utf8mb4 cmc < scripts/fix_corrupted_data_hex.sql
--- ============================================================================
-
-SET NAMES utf8mb4;
-
--- ============================================================================
--- PART 1: Fix CUSTOMERS table using HEX encoding
--- ============================================================================
-
--- Smart apostrophe: ’ → '
-UPDATE customers SET name = REPLACE(name, UNHEX('E28099'), UNHEX('E28099')) WHERE name LIKE CONCAT('%', UNHEX('E28099'), '%');
--- Actually, let's use the simpler binary approach
-UPDATE customers SET name = REPLACE(name, _binary 0xE28099, '''') WHERE name LIKE _binary CONCAT('%', 0xE28099, '%');
-
--- Left double quote: “ → "
-UPDATE customers SET name = REPLACE(name, _binary 0xE2809C, '"') WHERE name LIKE _binary CONCAT('%', 0xE2809C, '%');
-
--- Right double quote: †→ "
-UPDATE customers SET name = REPLACE(name, _binary 0xE2809D, '"') WHERE name LIKE _binary CONCAT('%', 0xE2809D, '%');
-
--- En dash: â€" → –
-UPDATE customers SET name = REPLACE(name, _binary 0xE28093, '–') WHERE name LIKE _binary CONCAT('%', 0xE28093, '%');
-
--- Em dash: â€" → —
-UPDATE customers SET name = REPLACE(name, _binary 0xE28094, '—') WHERE name LIKE _binary CONCAT('%', 0xE28094, '%');
-
--- Zero-width space: ​ → (remove)
-UPDATE customers SET name = REPLACE(name, _binary 0xE2808B, '') WHERE name LIKE _binary CONCAT('%', 0xE2808B, '%');
-
--- Em space:   → (regular space)
-UPDATE customers SET name = REPLACE(name, _binary 0xE28083, ' ') WHERE name LIKE _binary CONCAT('%', 0xE28083, '%');
-
--- Accented é: é → é
-UPDATE customers SET name = REPLACE(name, _binary 0xC3A9, 'é') WHERE name LIKE _binary CONCAT('%', 0xC3A9, '%');
-
--- Accented É: É → É
-UPDATE customers SET name = REPLACE(name, _binary 0xC389, 'É') WHERE name LIKE _binary CONCAT('%', 0xC389, '%');
-
--- Accented ó: ó → ó
-UPDATE customers SET name = REPLACE(name, _binary 0xC3B3, 'ó') WHERE name LIKE _binary CONCAT('%', 0xC3B3, '%');
-
--- Accented í: Ã → í
-UPDATE customers SET name = REPLACE(name, _binary 0xC3AD, 'í') WHERE name LIKE _binary CONCAT('%', 0xC3AD, '%');
-
--- Accented ç: ç → ç
-UPDATE customers SET name = REPLACE(name, _binary 0xC3A7, 'ç') WHERE name LIKE _binary CONCAT('%', 0xC3A7, '%');
-
--- Accented ü: ü → ü
-UPDATE customers SET name = REPLACE(name, _binary 0xC3BC, 'ü') WHERE name LIKE _binary CONCAT('%', 0xC3BC, '%');
-
--- Accented á: á → á
-UPDATE customers SET name = REPLACE(name, _binary 0xC3A1, 'á') WHERE name LIKE _binary CONCAT('%', 0xC3A1, '%');
-
--- Accented ñ: ñ → ñ
-UPDATE customers SET name = REPLACE(name, _binary 0xC3B1, 'ñ') WHERE name LIKE _binary CONCAT('%', 0xC3B1, '%');
-
--- Accented ö: ö → ö
-UPDATE customers SET name = REPLACE(name, _binary 0xC3B6, 'ö') WHERE name LIKE _binary CONCAT('%', 0xC3B6, '%');
-
--- Accented ô: ô → ô
-UPDATE customers SET name = REPLACE(name, _binary 0xC3B4, 'ô') WHERE name LIKE _binary CONCAT('%', 0xC3B4, '%');
-
--- Turkish İ: Ä° → İ
-UPDATE customers SET name = REPLACE(name, _binary 0xC4B0, 'İ') WHERE name LIKE _binary CONCAT('%', 0xC4B0, '%');
-
--- Turkish Ş: Å → Ş
-UPDATE customers SET name = REPLACE(name, _binary 0xC59E, 'Ş') WHERE name LIKE _binary CONCAT('%', 0xC59E, '%');
-
--- Turkish Ğ: Ä → Ğ
-UPDATE customers SET name = REPLACE(name, _binary 0xC49E, 'Ğ') WHERE name LIKE _binary CONCAT('%', 0xC49E, '%');
-
--- Czech Č: Ä → Č
-UPDATE customers SET name = REPLACE(name, _binary 0xC48C, 'Č') WHERE name LIKE _binary CONCAT('%', 0xC48C, '%');
-
--- Czech ě: Ä› → ě
-UPDATE customers SET name = REPLACE(name, _binary 0xC49B, 'ě') WHERE name LIKE _binary CONCAT('%', 0xC49B, '%');
-
--- Remove trailing tabs
-UPDATE customers SET name = TRIM(TRAILING '\t' FROM name) WHERE name LIKE '%\t';
-UPDATE customers SET trading_name = TRIM(TRAILING '\t' FROM trading_name) WHERE trading_name LIKE '%\t';
-
--- ============================================================================
--- PART 2: Fix PRINCIPLES table
--- ============================================================================
-
--- Accented ü: ü → ü
-UPDATE principles SET name = REPLACE(name, _binary 0xC3BC, 'ü') WHERE name LIKE _binary CONCAT('%', 0xC3BC, '%');
-UPDATE principles SET address = REPLACE(address, _binary 0xC3BC, 'ü') WHERE address LIKE _binary CONCAT('%', 0xC3BC, '%');
-UPDATE principles SET city = REPLACE(city, _binary 0xC3BC, 'ü') WHERE city LIKE _binary CONCAT('%', 0xC3BC, '%');
-
--- Accented é: é → é
-UPDATE principles SET name = REPLACE(name, _binary 0xC3A9, 'é') WHERE name LIKE _binary CONCAT('%', 0xC3A9, '%');
-UPDATE principles SET address = REPLACE(address, _binary 0xC3A9, 'é') WHERE address LIKE _binary CONCAT('%', 0xC3A9, '%');
-UPDATE principles SET city = REPLACE(city, _binary 0xC3A9, 'é') WHERE city LIKE _binary CONCAT('%', 0xC3A9, '%');
-
--- Accented ó: ó → ó
-UPDATE principles SET name = REPLACE(name, _binary 0xC3B3, 'ó') WHERE name LIKE _binary CONCAT('%', 0xC3B3, '%');
-UPDATE principles SET city = REPLACE(city, _binary 0xC3B3, 'ó') WHERE city LIKE _binary CONCAT('%', 0xC3B3, '%');
-
--- German ß: ß → ß
-UPDATE principles SET address = REPLACE(address, _binary 0xC39F, 'ß') WHERE address LIKE _binary CONCAT('%', 0xC39F, '%');
-
--- Accented ñ: ñ → ñ
-UPDATE principles SET address = REPLACE(address, _binary 0xC3B1, 'ñ') WHERE address LIKE _binary CONCAT('%', 0xC3B1, '%');
-
--- Czech č: Ä → č
-UPDATE principles SET address = REPLACE(address, _binary 0xC48D, 'č') WHERE address LIKE _binary CONCAT('%', 0xC48D, '%');
-
--- Czech ž: ž → ž
-UPDATE principles SET address = REPLACE(address, _binary 0xC5BE, 'ž') WHERE address LIKE _binary CONCAT('%', 0xC5BE, '%');
-
--- Czech š: Å¡ → š
-UPDATE principles SET address = REPLACE(address, _binary 0xC5A1, 'š') WHERE address LIKE _binary CONCAT('%', 0xC5A1, '%');
-
--- Czech ě: Ä› → ě
-UPDATE principles SET address = REPLACE(address, _binary 0xC49B, 'ě') WHERE address LIKE _binary CONCAT('%', 0xC49B, '%');
-
--- Czech ň: Å → ň
-UPDATE principles SET address = REPLACE(address, _binary 0xC588, 'ň') WHERE address LIKE _binary CONCAT('%', 0xC588, '%');
-
--- Czech ř: Å™ → ř
-UPDATE principles SET address = REPLACE(address, _binary 0xC599, 'ř') WHERE address LIKE _binary CONCAT('%', 0xC599, '%');
-
--- Specific principle fixes
-UPDATE principles SET address = 'Heinz-Fangman-Straße 18' WHERE id = 2;
-UPDATE principles SET address = 'Bezručova 2901\n756 61 Rožnov pod Radhoštěm', city = 'Rožnov pod Radhoštěm' WHERE id = 9;
-UPDATE principles SET name = 'IEP Technologies GmbH - BRILEX Gesellschaft für Explosionsschutz mbH' WHERE id = 13;
-UPDATE principles SET address = 'Liebigstraße 2' WHERE id = 14;
-UPDATE principles SET name = 'Nosia S.r.l.- Señalización Industrial' WHERE id = 58;
-UPDATE principles SET address = 'Alte Emser Straße 32' WHERE id = 65;
-
--- ============================================================================
--- PART 3: Fix ADDRESSES table
--- ============================================================================
-
--- En dash: â€" → –
-UPDATE addresses SET address = REPLACE(address, _binary 0xE28093, '–') WHERE address LIKE _binary CONCAT('%', 0xE28093, '%');
-
--- Accented é: é → é
-UPDATE addresses SET address = REPLACE(address, _binary 0xC3A9, 'é') WHERE address LIKE _binary CONCAT('%', 0xC3A9, '%');
-UPDATE addresses SET city = REPLACE(city, _binary 0xC3A9, 'é') WHERE city LIKE _binary CONCAT('%', 0xC3A9, '%');
-
--- Specific address fix
-UPDATE addresses SET address = 'Lvl 3, Building B, 7–11 Talavera Road' WHERE id = 19;
-
--- ============================================================================
--- PART 4: Fix PRODUCTS table (using simple text for known values)
--- ============================================================================
-
--- Degree symbol: ° → °
-UPDATE products SET title = REPLACE(title, CHAR(194,176), CHAR(176)) WHERE title LIKE CONCAT('%', CHAR(194,176), '%');
-UPDATE products SET description = REPLACE(description, CHAR(194,176), CHAR(176)) WHERE description LIKE CONCAT('%', CHAR(194,176), '%');
-UPDATE products SET item_description = REPLACE(item_description, CHAR(194,176), CHAR(176)) WHERE item_description LIKE CONCAT('%', CHAR(194,176), '%');
-
--- Registered trademark: ® → ®
-UPDATE products SET title = REPLACE(title, CHAR(194,174), CHAR(174)) WHERE title LIKE CONCAT('%', CHAR(194,174), '%');
-UPDATE products SET description = REPLACE(description, CHAR(194,174), CHAR(174)) WHERE description LIKE CONCAT('%', CHAR(194,174), '%');
-UPDATE products SET item_description = REPLACE(item_description, CHAR(194,174), CHAR(174)) WHERE item_description LIKE CONCAT('%', CHAR(194,174), '%');
-
--- Trademark: â„¢ → ™ (0xE284A2 → 0x2122)
-UPDATE products SET title = REPLACE(title, CHAR(226,132,162), CHAR(226,132,162)) WHERE title LIKE CONCAT('%', CHAR(226,132,162), '%');
-UPDATE products SET description = REPLACE(description, CHAR(226,132,162), CHAR(226,132,162)) WHERE description LIKE CONCAT('%', CHAR(226,132,162), '%');
-
--- Smart apostrophe ’ → '
-UPDATE products SET title = REPLACE(title, CHAR(226,128,153), '''') WHERE title LIKE CONCAT('%', CHAR(226,128,153), '%');
-UPDATE products SET description = REPLACE(description, CHAR(226,128,153), '''') WHERE description LIKE CONCAT('%', CHAR(226,128,153), '%');
-
--- En dash â€" → –
-UPDATE products SET title = REPLACE(title, CHAR(226,128,147), '–') WHERE title LIKE CONCAT('%', CHAR(226,128,147), '%');
-UPDATE products SET description = REPLACE(description, CHAR(226,128,147), '–') WHERE description LIKE CONCAT('%', CHAR(226,128,147), '%');
-
--- HTML entities (safe to use text)
-UPDATE products SET title = REPLACE(title, '&ndash;', '–') WHERE title LIKE '%&ndash;%';
-UPDATE products SET description = REPLACE(description, '&ndash;', '–') WHERE description LIKE '%&ndash;%';
-UPDATE products SET title = REPLACE(title, '&rdquo;', '"') WHERE title LIKE '%&rdquo;%';
-UPDATE products SET description = REPLACE(description, '&rdquo;', '"') WHERE description LIKE '%&rdquo;%';
-UPDATE products SET title = REPLACE(title, '&ldquo;', '"') WHERE title LIKE '%&ldquo;%';
-UPDATE products SET description = REPLACE(description, '&ldquo;', '"') WHERE description LIKE '%&ldquo;%';
-UPDATE products SET title = REPLACE(title, '&amp;', '&') WHERE title LIKE '%&amp;%';
-UPDATE products SET description = REPLACE(description, '&amp;', '&') WHERE description LIKE '%&amp;%';
-UPDATE products SET title = REPLACE(title, '&frac12;', '½') WHERE title LIKE '%&frac12;%';
-UPDATE products SET description = REPLACE(description, '&frac12;', '½') WHERE description LIKE '%&frac12;%';
-UPDATE products SET title = REPLACE(title, '&deg;', '°') WHERE title LIKE '%&deg;%';
-UPDATE products SET description = REPLACE(description, '&deg;', '°') WHERE description LIKE '%&deg;%';
-UPDATE products SET title = REPLACE(title, '&nbsp;', ' ') WHERE title LIKE '%&nbsp;%';
-UPDATE products SET description = REPLACE(description, '&nbsp;', ' ') WHERE description LIKE '%&nbsp;%';
-
--- Specific product fixes
-UPDATE products SET title = 'C95SN189C DSTGL40/C-Digital temperature probe for P8xx1 Web Sensor. Range -30 to +80°C. With CINCH connector, 1m Cable' WHERE id = 30;
-UPDATE products SET title = 'Mid-West Instrument Model 240 – SC – 02 – O(TT)' WHERE id = 38;
-UPDATE products SET title = 'Newson Gale Earth-Rite® II FIBC Static Earthing System, GRP Enclosure with 5m 2 Core Spiral Cable and FIBC Clamp' WHERE id = 67;
-UPDATE products SET title = 'Newson Gale Earth-Rite® RTR™ Tri-Mode Static Grounding System, Metal Enclosure, X90-IP Heavy Duty Clamp with 10m 2 Core Spiral Cable' WHERE id = 85;
-
--- ============================================================================
--- Verification
--- ============================================================================
-
-SELECT '===========================================================' as '';
-SELECT 'FIX COMPLETE - Verification Results' as '';
-SELECT '===========================================================' as '';
-
-SELECT 'Remaining corrupted principles:' as '', COUNT(*) as count
-FROM principles WHERE name REGEXP '[^ -~]' OR address REGEXP '[^ -~]' OR city REGEXP '[^ -~]';
-
-SELECT 'Remaining corrupted customers:' as '', COUNT(*) as count
-FROM customers WHERE name REGEXP '[^ -~]';
-
-SELECT 'Remaining corrupted addresses:' as '', COUNT(*) as count
-FROM addresses WHERE address REGEXP '[^ -~]' OR city REGEXP '[^ -~]';
-
-SELECT 'Remaining corrupted products:' as '', COUNT(*) as count
-FROM products WHERE title REGEXP '[^ -~]' OR description REGEXP '[^ -~]' OR item_description REGEXP '[^ -~]';
-
-SELECT 'Fix script completed!' as status;

scripts/fix_corrupted_data_safe.sh

@@ -1,97 +0,0 @@
-#!/bin/bash
-# ============================================================================
-# Fix Corrupted Character Data - Safe Version
-# ============================================================================
-# This script fixes mojibake by generating SQL dynamically to avoid encoding issues
-# ============================================================================
-
-DB_USER="root"
-DB_PASS="secureRootPassword"
-DB_NAME="cmc"
-CONTAINER="cmc-db"
-
-# Function to run SQL
-run_sql() {
-    docker exec $CONTAINER mariadb -u $DB_USER -p$DB_PASS --default-character-set=utf8mb4 $DB_NAME -e "$1"
-}
-
-echo "============================================================"
-echo "Fixing Character Corruption - Safe Version"
-echo "============================================================"
-echo ""
-
-# Set charset
-run_sql "SET NAMES utf8mb4;"
-
-echo "Fixing CUSTOMERS table..."
-# Fix customers - specific known records
-run_sql "UPDATE customers SET name = 'SOLUZÉ CIVIL ENGINEERS Trading Under SOLUZE PTY LTD' WHERE id = 253;"
-run_sql "UPDATE customers SET name = 'Dr. Prem''s Molecules Private Limited (DPMolecules)' WHERE id = 1006;"
-run_sql "UPDATE customers SET name = 'DEE ENTERPRISES (QLD) PTY LTD trading as \"MEKLEK\"' WHERE id = 1387;"
-run_sql "UPDATE customers SET name = 'Guidera O''Connor Pty Ltd' WHERE id = 1608;"
-run_sql "UPDATE customers SET name = 'Ingredion ANZ Pty Ltd' WHERE id = 2174;"
-run_sql "UPDATE customers SET name = 'Vale Nouvelle-Calédonie S.A.S' WHERE id = 2215;"
-run_sql "UPDATE customers SET name = 'Evaluación y Control Ambiental S.A.S.' WHERE id = 2375;"
-run_sql "UPDATE customers SET name = 'Zontahevy Comércio e Serviços Offshore Ltda' WHERE id = 3143;"
-run_sql "UPDATE customers SET name = 'Société des Mines de Syama' WHERE id = 3529;"
-run_sql "UPDATE customers SET name = 'P.J. Berriman & Co PTY LTD' WHERE id = 3633;"
-run_sql "UPDATE customers SET name = 'Rambøll Danmark' WHERE id = 4325;"
-run_sql "UPDATE customers SET name = 'SONNIVA ENERGY MAKİNA İNŞAAT İTH. İHR. LTD. ŞTİ.' WHERE id = 4350;"
-run_sql "UPDATE customers SET name = 'F.C.C. Fluides Conseils Calédonie SARL' WHERE id = 4669;"
-run_sql "UPDATE customers SET name = 'DGPack Prostějov' WHERE id = 4743;"
-run_sql "UPDATE customers SET name = 'Mccready Welding Pty Ltd Trading under the entity Mccready''s Welding Services' WHERE id = 4764;"
-run_sql "UPDATE customers SET name = 'Oxiquímica, S.A.P.I. de C.V.' WHERE id = 4893;"
-
-echo "Fixing PRINCIPLES table..."
-run_sql "UPDATE principles SET address = 'Heinz-Fangman-Straße 18' WHERE id = 2;"
-run_sql "UPDATE principles SET address = 'Bezručova 2901\n756 61 Rožnov pod Radhoštěm', city = 'Rožnov pod Radhoštěm' WHERE id = 9;"
-run_sql "UPDATE principles SET name = 'IEP Technologies GmbH - BRILEX Gesellschaft für Explosionsschutz mbH' WHERE id = 13;"
-run_sql "UPDATE principles SET address = 'Liebigstraße 2' WHERE id = 14;"
-run_sql "UPDATE principles SET name = 'Nosia S.r.l.- Señalización Industrial' WHERE id = 58;"
-run_sql "UPDATE principles SET address = 'Alte Emser Straße 32' WHERE id = 65;"
-
-echo "Fixing ADDRESSES table..."
-run_sql "UPDATE addresses SET address = 'Lvl 3, Building B, 7–11 Talavera Road' WHERE id = 19;"
-
-echo "Fixing PRODUCTS table (specific records)..."
-run_sql "UPDATE products SET title = 'C95SN189C DSTGL40/C-Digital temperature probe for P8xx1 Web Sensor. Range -30 to +80°C. With CINCH connector, 1m Cable' WHERE id = 30;"
-run_sql "UPDATE products SET title = 'Mid-West Instrument Model 240 – SC – 02 – O(TT)' WHERE id = 38;"
-run_sql "UPDATE products SET title = 'Newson Gale Earth-Rite® II FIBC Static Earthing System, GRP Enclosure with 5m 2 Core Spiral Cable and FIBC Clamp' WHERE id = 67;"
-run_sql "UPDATE products SET title = 'Newson Gale Earth-Rite® RTR Tester - ER2/CRT' WHERE id = 76;"
-run_sql "UPDATE products SET title = 'Newson Gale Earth-Rite® Installers Kit A for IIB areas only (Metal Enclosures Only)' WHERE id = 77;"
-run_sql "UPDATE products SET title = 'Newson Gale Earth-Rite® RTR™ Tri-Mode Static Grounding System, Metal Enclosure, X90-IP Heavy Duty Clamp with 10m 2 Core Spiral Cable' WHERE id = 85;"
-run_sql "UPDATE products SET title = 'Newson Gale Earth-Rite® RTR System Spares - X90-IP Heavy Duty 2 Core Clamp with 10m 2 Core Spiral Cable and Male Quick Connect' WHERE id = 86;"
-run_sql "UPDATE products SET title = 'Newson Gale Bond-Rite® EZ, VESX90-IP Heavy Duty Clamp & 3m 2 Core Spiral Cable' WHERE id = 149;"
-
-echo ""
-echo "Fixing HTML entities in products..."
-# These are safe ASCII characters, so no encoding issues
-run_sql "UPDATE products SET title = REPLACE(title, '–', '–') WHERE title LIKE '%–%';"
-run_sql "UPDATE products SET description = REPLACE(description, '–', '–') WHERE description LIKE '%–%';"
-run_sql "UPDATE products SET title = REPLACE(title, '&', '&') WHERE title LIKE '%&%';"
-run_sql "UPDATE products SET description = REPLACE(description, '&', '&') WHERE description LIKE '%&%';"
-run_sql "UPDATE products SET title = REPLACE(title, ' ', ' ') WHERE title LIKE '% %';"
-run_sql "UPDATE products SET description = REPLACE(description, ' ', ' ') WHERE description LIKE '% %';"
-run_sql "UPDATE products SET title = REPLACE(title, '°', '°') WHERE title LIKE '%°%';"
-run_sql "UPDATE products SET description = REPLACE(description, '°', '°') WHERE description LIKE '%°%';"
-run_sql "UPDATE products SET title = REPLACE(title, '½', '½') WHERE title LIKE '%½%';"
-run_sql "UPDATE products SET description = REPLACE(description, '½', '½') WHERE description LIKE '%½%';"
-run_sql "UPDATE products SET title = REPLACE(title, '”', '\"') WHERE title LIKE '%”%';"
-run_sql "UPDATE products SET description = REPLACE(description, '”', '\"') WHERE description LIKE '%”%';"
-run_sql "UPDATE products SET title = REPLACE(title, '“', '\"') WHERE title LIKE '%“%';"
-run_sql "UPDATE products SET description = REPLACE(description, '“', '\"') WHERE description LIKE '%“%';"
-
-echo ""
-echo "============================================================"
-echo "Fix complete! Verification:"
-echo "============================================================"
-
-run_sql "SELECT 'Remaining corrupted customers:' as status, COUNT(*) as count FROM customers WHERE name REGEXP '[^ -~]';"
-run_sql "SELECT 'Remaining corrupted principles:' as status, COUNT(*) as count FROM principles WHERE name REGEXP '[^ -~]' OR address REGEXP '[^ -~]';"
-run_sql "SELECT 'Remaining corrupted products:' as status, COUNT(*) as count FROM products WHERE title REGEXP '[^ -~]' OR description REGEXP '[^ -~]';"
-
-echo ""
-echo "Done! Check specific records:"
-echo "SELECT id, name FROM customers WHERE id IN (253, 1006, 1387);"
-echo "SELECT id, name, city FROM principles WHERE id IN (2, 9, 13);"
-echo "SELECT id, title FROM products WHERE id IN (30, 38, 67, 85);"

scripts/fix_products_corruption.sql

@@ -1,291 +0,0 @@
--- ============================================================================
--- Fix Corrupted Character Data in PRODUCTS Table
--- ============================================================================
--- This script fixes mojibake in the products table (12,536+ affected records)
--- Also fixes HTML entities that shouldn't be in the raw data
---
--- IMPORTANT: Review this script and test on a backup first!
--- To run: docker exec -i cmc-db mariadb -u root -psecureRootPassword --default-character-set=utf8mb4 cmc < scripts/fix_products_corruption.sql
--- ============================================================================
-
-SET NAMES utf8mb4;
-
-SELECT '============================================================' as '';
-SELECT 'PRODUCTS TABLE - CHARACTER CORRUPTION FIX' as '';
-SELECT '============================================================' as '';
-
--- Count before fix
-SELECT 'BEFORE FIX - Corrupted records count:' as '';
-SELECT COUNT(*) as corrupted_products
-FROM products
-WHERE title REGEXP '[^ -~]'
-   OR description REGEXP '[^ -~]'
-   OR item_description REGEXP '[^ -~]'
-   OR notes REGEXP '[^ -~]';
-
--- ============================================================================
--- PART 1: Fix Mojibake Patterns (UTF-8 corruption)
--- ============================================================================
-
--- Degree symbol: ° → °
-UPDATE products SET title = REPLACE(title, '°', '°') WHERE title LIKE '%°%';
-UPDATE products SET description = REPLACE(description, '°', '°') WHERE description LIKE '%°%';
-UPDATE products SET item_description = REPLACE(item_description, '°', '°') WHERE item_description LIKE '%°%';
-UPDATE products SET notes = REPLACE(notes, '°', '°') WHERE notes LIKE '%°%';
-
--- Registered trademark: ® → ®
-UPDATE products SET title = REPLACE(title, '®', '®') WHERE title LIKE '%®%';
-UPDATE products SET description = REPLACE(description, '®', '®') WHERE description LIKE '%®%';
-UPDATE products SET item_description = REPLACE(item_description, '®', '®') WHERE item_description LIKE '%®%';
-UPDATE products SET notes = REPLACE(notes, '®', '®') WHERE notes LIKE '%®%';
-
--- Trademark: â„¢ → ™
-UPDATE products SET title = REPLACE(title, 'â„¢', '™') WHERE title LIKE '%â„¢%';
-UPDATE products SET description = REPLACE(description, 'â„¢', '™') WHERE description LIKE '%â„¢%';
-UPDATE products SET item_description = REPLACE(item_description, 'â„¢', '™') WHERE item_description LIKE '%â„¢%';
-UPDATE products SET notes = REPLACE(notes, 'â„¢', '™') WHERE notes LIKE '%â„¢%';
-
--- Smart apostrophe: ’ → '
-UPDATE products SET title = REPLACE(title, '’', ''') WHERE title LIKE '%’%';
-UPDATE products SET description = REPLACE(description, '’', ''') WHERE description LIKE '%’%';
-UPDATE products SET item_description = REPLACE(item_description, '’', ''') WHERE item_description LIKE '%’%';
-UPDATE products SET notes = REPLACE(notes, '’', ''') WHERE notes LIKE '%’%';
-
--- Smart left double quote: “ → "
-UPDATE products SET title = REPLACE(title, '“', '"') WHERE title LIKE '%“%';
-UPDATE products SET description = REPLACE(description, '“', '"') WHERE description LIKE '%“%';
-UPDATE products SET item_description = REPLACE(item_description, '“', '"') WHERE item_description LIKE '%“%';
-UPDATE products SET notes = REPLACE(notes, '“', '"') WHERE notes LIKE '%“%';
-
--- Smart right double quote: †→ "
-UPDATE products SET title = REPLACE(title, 'â€', '"') WHERE title LIKE '%â€%';
-UPDATE products SET description = REPLACE(description, 'â€', '"') WHERE description LIKE '%â€%';
-UPDATE products SET item_description = REPLACE(item_description, 'â€', '"') WHERE item_description LIKE '%â€%';
-UPDATE products SET notes = REPLACE(notes, 'â€', '"') WHERE notes LIKE '%â€%';
-
--- En dash: â€" → –
-UPDATE products SET title = REPLACE(title, 'â€"', '–') WHERE title LIKE '%â€"%';
-UPDATE products SET description = REPLACE(description, 'â€"', '–') WHERE description LIKE '%â€"%';
-UPDATE products SET item_description = REPLACE(item_description, 'â€"', '–') WHERE item_description LIKE '%â€"%';
-UPDATE products SET notes = REPLACE(notes, 'â€"', '–') WHERE notes LIKE '%â€"%';
-
--- Em dash: â€" → —
-UPDATE products SET title = REPLACE(title, 'â€"', '—') WHERE title LIKE '%â€"%';
-UPDATE products SET description = REPLACE(description, 'â€"', '—') WHERE description LIKE '%â€"%';
-UPDATE products SET item_description = REPLACE(item_description, 'â€"', '—') WHERE item_description LIKE '%â€"%';
-UPDATE products SET notes = REPLACE(notes, 'â€"', '—') WHERE notes LIKE '%â€"%';
-
--- Ellipsis: … → …
-UPDATE products SET title = REPLACE(title, '…', '…') WHERE title LIKE '%…%';
-UPDATE products SET description = REPLACE(description, '…', '…') WHERE description LIKE '%…%';
-UPDATE products SET item_description = REPLACE(item_description, '…', '…') WHERE item_description LIKE '%…%';
-UPDATE products SET notes = REPLACE(notes, '…', '…') WHERE notes LIKE '%…%';
-
--- Accented characters (common in product descriptions)
--- é
-UPDATE products SET title = REPLACE(title, 'é', 'é') WHERE title LIKE '%é%';
-UPDATE products SET description = REPLACE(description, 'é', 'é') WHERE description LIKE '%é%';
-UPDATE products SET item_description = REPLACE(item_description, 'é', 'é') WHERE item_description LIKE '%é%';
-UPDATE products SET notes = REPLACE(notes, 'é', 'é') WHERE notes LIKE '%é%';
-
--- ü
-UPDATE products SET title = REPLACE(title, 'ü', 'ü') WHERE title LIKE '%ü%';
-UPDATE products SET description = REPLACE(description, 'ü', 'ü') WHERE description LIKE '%ü%';
-UPDATE products SET item_description = REPLACE(item_description, 'ü', 'ü') WHERE item_description LIKE '%ü%';
-UPDATE products SET notes = REPLACE(notes, 'ü', 'ü') WHERE notes LIKE '%ü%';
-
--- ö
-UPDATE products SET title = REPLACE(title, 'ö', 'ö') WHERE title LIKE '%ö%';
-UPDATE products SET description = REPLACE(description, 'ö', 'ö') WHERE description LIKE '%ö%';
-UPDATE products SET item_description = REPLACE(item_description, 'ö', 'ö') WHERE item_description LIKE '%ö%';
-UPDATE products SET notes = REPLACE(notes, 'ö', 'ö') WHERE notes LIKE '%ö%';
-
--- ä
-UPDATE products SET title = REPLACE(title, 'ä', 'ä') WHERE title LIKE '%ä%';
-UPDATE products SET description = REPLACE(description, 'ä', 'ä') WHERE description LIKE '%ä%';
-UPDATE products SET item_description = REPLACE(item_description, 'ä', 'ä') WHERE item_description LIKE '%ä%';
-UPDATE products SET notes = REPLACE(notes, 'ä', 'ä') WHERE notes LIKE '%ä%';
-
--- ß (German sharp s)
-UPDATE products SET title = REPLACE(title, 'ß', 'ß') WHERE title LIKE '%ß%';
-UPDATE products SET description = REPLACE(description, 'ß', 'ß') WHERE description LIKE '%ß%';
-UPDATE products SET item_description = REPLACE(item_description, 'ß', 'ß') WHERE item_description LIKE '%ß%';
-UPDATE products SET notes = REPLACE(notes, 'ß', 'ß') WHERE notes LIKE '%ß%';
-
--- ============================================================================
--- PART 2: Fix HTML Entities (shouldn't be in raw database data)
--- ============================================================================
--- Note: These are likely from copy-paste from HTML or rich text editors
-
--- &ndash; → –
-UPDATE products SET title = REPLACE(title, '&ndash;', '–') WHERE title LIKE '%&ndash;%';
-UPDATE products SET description = REPLACE(description, '&ndash;', '–') WHERE description LIKE '%&ndash;%';
-UPDATE products SET item_description = REPLACE(item_description, '&ndash;', '–') WHERE item_description LIKE '%&ndash;%';
-UPDATE products SET notes = REPLACE(notes, '&ndash;', '–') WHERE notes LIKE '%&ndash;%';
-
--- &mdash; → —
-UPDATE products SET title = REPLACE(title, '&mdash;', '—') WHERE title LIKE '%&mdash;%';
-UPDATE products SET description = REPLACE(description, '&mdash;', '—') WHERE description LIKE '%&mdash;%';
-UPDATE products SET item_description = REPLACE(item_description, '&mdash;', '—') WHERE item_description LIKE '%&mdash;%';
-UPDATE products SET notes = REPLACE(notes, '&mdash;', '—') WHERE notes LIKE '%&mdash;%';
-
--- &rdquo; → "
-UPDATE products SET title = REPLACE(title, '&rdquo;', '"') WHERE title LIKE '%&rdquo;%';
-UPDATE products SET description = REPLACE(description, '&rdquo;', '"') WHERE description LIKE '%&rdquo;%';
-UPDATE products SET item_description = REPLACE(item_description, '&rdquo;', '"') WHERE item_description LIKE '%&rdquo;%';
-UPDATE products SET notes = REPLACE(notes, '&rdquo;', '"') WHERE notes LIKE '%&rdquo;%';
-
--- &ldquo; → "
-UPDATE products SET title = REPLACE(title, '&ldquo;', '"') WHERE title LIKE '%&ldquo;%';
-UPDATE products SET description = REPLACE(description, '&ldquo;', '"') WHERE description LIKE '%&ldquo;%';
-UPDATE products SET item_description = REPLACE(item_description, '&ldquo;', '"') WHERE item_description LIKE '%&ldquo;%';
-UPDATE products SET notes = REPLACE(notes, '&ldquo;', '"') WHERE notes LIKE '%&ldquo;%';
-
--- &rsquo; → '
-UPDATE products SET title = REPLACE(title, '&rsquo;', ''') WHERE title LIKE '%&rsquo;%';
-UPDATE products SET description = REPLACE(description, '&rsquo;', ''') WHERE description LIKE '%&rsquo;%';
-UPDATE products SET item_description = REPLACE(item_description, '&rsquo;', ''') WHERE item_description LIKE '%&rsquo;%';
-UPDATE products SET notes = REPLACE(notes, '&rsquo;', ''') WHERE notes LIKE '%&rsquo;%';
-
--- &lsquo; → '
-UPDATE products SET title = REPLACE(title, '&lsquo;', ''') WHERE title LIKE '%&lsquo;%';
-UPDATE products SET description = REPLACE(description, '&lsquo;', ''') WHERE description LIKE '%&lsquo;%';
-UPDATE products SET item_description = REPLACE(item_description, '&lsquo;', ''') WHERE item_description LIKE '%&lsquo;%';
-UPDATE products SET notes = REPLACE(notes, '&lsquo;', ''') WHERE notes LIKE '%&lsquo;%';
-
--- &quot; → " (keep as is, but decode entity)
-UPDATE products SET title = REPLACE(title, '&quot;', '"') WHERE title LIKE '%&quot;%';
-UPDATE products SET description = REPLACE(description, '&quot;', '"') WHERE description LIKE '%&quot;%';
-UPDATE products SET item_description = REPLACE(item_description, '&quot;', '"') WHERE item_description LIKE '%&quot;%';
-UPDATE products SET notes = REPLACE(notes, '&quot;', '"') WHERE notes LIKE '%&quot;%';
-
--- &amp; → & (ampersand)
-UPDATE products SET title = REPLACE(title, '&amp;', '&') WHERE title LIKE '%&amp;%';
-UPDATE products SET description = REPLACE(description, '&amp;', '&') WHERE description LIKE '%&amp;%';
-UPDATE products SET item_description = REPLACE(item_description, '&amp;', '&') WHERE item_description LIKE '%&amp;%';
-UPDATE products SET notes = REPLACE(notes, '&amp;', '&') WHERE notes LIKE '%&amp;%';
-
--- Common fractions
--- &frac12; → ½
-UPDATE products SET title = REPLACE(title, '&frac12;', '½') WHERE title LIKE '%&frac12;%';
-UPDATE products SET description = REPLACE(description, '&frac12;', '½') WHERE description LIKE '%&frac12;%';
-UPDATE products SET item_description = REPLACE(item_description, '&frac12;', '½') WHERE item_description LIKE '%&frac12;%';
-UPDATE products SET notes = REPLACE(notes, '&frac12;', '½') WHERE notes LIKE '%&frac12;%';
-
--- &frac14; → ¼
-UPDATE products SET title = REPLACE(title, '&frac14;', '¼') WHERE title LIKE '%&frac14;%';
-UPDATE products SET description = REPLACE(description, '&frac14;', '¼') WHERE description LIKE '%&frac14;%';
-UPDATE products SET item_description = REPLACE(item_description, '&frac14;', '¼') WHERE item_description LIKE '%&frac14;%';
-UPDATE products SET notes = REPLACE(notes, '&frac14;', '¼') WHERE notes LIKE '%&frac14;%';
-
--- &frac34; → ¾
-UPDATE products SET title = REPLACE(title, '&frac34;', '¾') WHERE title LIKE '%&frac34;%';
-UPDATE products SET description = REPLACE(description, '&frac34;', '¾') WHERE description LIKE '%&frac34;%';
-UPDATE products SET item_description = REPLACE(item_description, '&frac34;', '¾') WHERE item_description LIKE '%&frac34;%';
-UPDATE products SET notes = REPLACE(notes, '&frac34;', '¾') WHERE notes LIKE '%&frac34;%';
-
--- &deg; → °
-UPDATE products SET title = REPLACE(title, '&deg;', '°') WHERE title LIKE '%&deg;%';
-UPDATE products SET description = REPLACE(description, '&deg;', '°') WHERE description LIKE '%&deg;%';
-UPDATE products SET item_description = REPLACE(item_description, '&deg;', '°') WHERE item_description LIKE '%&deg;%';
-UPDATE products SET notes = REPLACE(notes, '&deg;', '°') WHERE notes LIKE '%&deg;%';
-
--- &times; → ×
-UPDATE products SET title = REPLACE(title, '&times;', '×') WHERE title LIKE '%&times;%';
-UPDATE products SET description = REPLACE(description, '&times;', '×') WHERE description LIKE '%&times;%';
-UPDATE products SET item_description = REPLACE(item_description, '&times;', '×') WHERE item_description LIKE '%&times;%';
-UPDATE products SET notes = REPLACE(notes, '&times;', '×') WHERE notes LIKE '%&times;%';
-
--- &nbsp; → (space) - non-breaking space
-UPDATE products SET title = REPLACE(title, '&nbsp;', ' ') WHERE title LIKE '%&nbsp;%';
-UPDATE products SET description = REPLACE(description, '&nbsp;', ' ') WHERE description LIKE '%&nbsp;%';
-UPDATE products SET item_description = REPLACE(item_description, '&nbsp;', ' ') WHERE item_description LIKE '%&nbsp;%';
-UPDATE products SET notes = REPLACE(notes, '&nbsp;', ' ') WHERE notes LIKE '%&nbsp;%';
-
--- ============================================================================
--- PART 3: Specific Known Corrupted Records
--- ============================================================================
-
--- ID 30: Degree symbol in title
-UPDATE products
-SET title = 'C95SN189C DSTGL40/C-Digital temperature probe for P8xx1 Web Sensor. Range -30 to +80°C. With CINCH connector, 1m Cable'
-WHERE id = 30;
-
--- ID 38: Em dashes in title
-UPDATE products
-SET title = 'Mid-West Instrument Model 240 – SC – 02 – O(TT)'
-WHERE id = 38;
-
--- ID 67: Registered trademark
-UPDATE products
-SET title = 'Newson Gale Earth-Rite® II FIBC Static Earthing System, GRP Enclosure with 5m 2 Core Spiral Cable and FIBC Clamp'
-WHERE id = 67;
-
--- ID 76: Registered trademark
-UPDATE products
-SET title = 'Newson Gale Earth-Rite® RTR Tester - ER2/CRT'
-WHERE id = 76;
-
--- ID 77: Registered trademark
-UPDATE products
-SET title = 'Newson Gale Earth-Rite® Installers Kit A for IIB areas only (Metal Enclosures Only)'
-WHERE id = 77;
-
--- ID 85: Registered trademark and trademark
-UPDATE products
-SET title = 'Newson Gale Earth-Rite® RTR™ Tri-Mode Static Grounding System, Metal Enclosure, X90-IP Heavy Duty Clamp with 10m 2 Core Spiral Cable'
-WHERE id = 85;
-
--- ID 86: Registered trademark
-UPDATE products
-SET title = 'Newson Gale Earth-Rite® RTR System Spares - X90-IP Heavy Duty 2 Core Clamp with 10m 2 Core Spiral Cable and Male Quick Connect'
-WHERE id = 86;
-
--- ID 149: Registered trademark
-UPDATE products
-SET title = 'Newson Gale Bond-Rite® EZ, VESX90-IP Heavy Duty Clamp & 3m 2 Core Spiral Cable'
-WHERE id = 149;
-
--- ID 151: Registered trademark
-UPDATE products
-SET title = 'UNSURE Newson Gale Bond-Rite® Clamp with 20ft (6.1m) Cable Reel'
-WHERE id = 151;
-
--- ID 155: Registered trademark
-UPDATE products
-SET title = 'Newson Gale Earth-Rite® MGV Tri-Mode Static Grounding System, GRP Enclosure, X90-IP Heavy Duty Clamp with 15m 2 Core Spiral Cable'
-WHERE id = 155;
-
--- ID 220: Registered trademark
-UPDATE products
-SET title = 'Newson Gale Earth-Rite® TELLUS II Static Earthing System, X90-IP Heavy Duty Clamp with 10m 2 Core Spiral Cable'
-WHERE id = 220;
-
--- ID 249: Double registered trademarks
-UPDATE products
-SET title = 'Newson Gale Bond-Rite® Remote GRP Multi-Way Junction Box for 2 to 4 Bond-Rite® Remote EP Units'
-WHERE id = 249;
-
--- ============================================================================
--- PART 4: Verification
--- ============================================================================
-
-SELECT '' as '';
-SELECT '============================================================' as '';
-SELECT 'AFTER FIX - Corrupted records count:' as '';
-SELECT COUNT(*) as remaining_corrupted_products
-FROM products
-WHERE title REGEXP '[^ -~]'
-   OR description REGEXP '[^ -~]'
-   OR item_description REGEXP '[^ -~]'
-   OR notes REGEXP '[^ -~]';
-
--- Show sample fixed records
-SELECT '' as '';
-SELECT 'Sample fixed records:' as '';
-SELECT id, title FROM products WHERE id IN (30, 38, 67, 76, 85, 149);
-
-SELECT '' as '';
-SELECT '============================================================' as '';
-SELECT 'PRODUCTS TABLE FIX COMPLETE' as '';
-SELECT '============================================================' as '';

scripts/report_remaining_corruption.sh

@@ -1,15 +0,0 @@
-#!/bin/bash
-# ============================================================================
-# Generate Report of Remaining Corrupted Data
-# ============================================================================
-
-DB_USER="root"
-DB_PASS="secureRootPassword"
-DB_NAME="cmc"
-CONTAINER="cmc-db"
-
-docker exec $CONTAINER mariadb \
-    -u $DB_USER \
-    -p$DB_PASS \
-    --default-character-set=utf8mb4 \
-    $DB_NAME < scripts/report_remaining_corruption.sql

scripts/report_remaining_corruption.sql

@@ -1,191 +0,0 @@
--- ============================================================================
--- Report Remaining Corrupted Data After Fix
--- ============================================================================
--- Run this after fix_corrupted_data.sql to find records that still need manual fixes
---
--- To run: docker exec cmc-db mariadb -u root -psecureRootPassword --default-character-set=utf8mb4 cmc < scripts/report_remaining_corruption.sql
--- ============================================================================
-
-SET NAMES utf8mb4;
-
-SELECT '============================================================' as '';
-SELECT 'CORRUPTED DATA REPORT' as '';
-SELECT '============================================================' as '';
-
--- Summary counts
-SELECT '' as '';
-SELECT 'SUMMARY OF REMAINING CORRUPTION' as '';
-SELECT '-------------------------------' as '';
-
-SELECT
-    'principles' as table_name,
-    COUNT(*) as corrupted_records,
-    'name, address, city' as affected_columns
-FROM principles
-WHERE name REGEXP '[^ -~]'
-   OR address REGEXP '[^ -~]'
-   OR city REGEXP '[^ -~]'
-
-UNION ALL
-
-SELECT
-    'customers',
-    COUNT(*),
-    'name'
-FROM customers
-WHERE name REGEXP '[^ -~]'
-
-UNION ALL
-
-SELECT
-    'addresses',
-    COUNT(*),
-    'address, city'
-FROM addresses
-WHERE address REGEXP '[^ -~]'
-   OR city REGEXP '[^ -~]'
-
-UNION ALL
-
-SELECT
-    'contacts',
-    COUNT(*),
-    'name'
-FROM contacts
-WHERE name REGEXP '[^ -~]'
-
-UNION ALL
-
-SELECT
-    'products',
-    COUNT(*),
-    'name, description'
-FROM products
-WHERE name REGEXP '[^ -~]'
-   OR description REGEXP '[^ -~]';
-
--- Detailed listings
-SELECT '' as '';
-SELECT '============================================================' as '';
-SELECT 'DETAILED CORRUPTED RECORDS' as '';
-SELECT '============================================================' as '';
-
--- Principles with corruption
-SELECT '' as '';
-SELECT 'PRINCIPLES TABLE - Corrupted Records:' as '';
-SELECT '-------------------------------------' as '';
-SELECT
-    id,
-    name,
-    LEFT(address, 50) as address_preview,
-    city
-FROM principles
-WHERE name REGEXP '[^ -~]'
-   OR address REGEXP '[^ -~]'
-   OR city REGEXP '[^ -~]'
-ORDER BY id;
-
--- Customers with corruption
-SELECT '' as '';
-SELECT 'CUSTOMERS TABLE - Corrupted Records:' as '';
-SELECT '-------------------------------------' as '';
-SELECT
-    id,
-    name
-FROM customers
-WHERE name REGEXP '[^ -~]'
-ORDER BY id;
-
--- Addresses with corruption
-SELECT '' as '';
-SELECT 'ADDRESSES TABLE - Corrupted Records:' as '';
-SELECT '-------------------------------------' as '';
-SELECT
-    id,
-    LEFT(address, 60) as address,
-    city
-FROM addresses
-WHERE address REGEXP '[^ -~]'
-   OR city REGEXP '[^ -~]'
-ORDER BY id
-LIMIT 50;
-
--- Contacts with corruption
-SELECT '' as '';
-SELECT 'CONTACTS TABLE - Corrupted Records:' as '';
-SELECT '-------------------------------------' as '';
-SELECT
-    id,
-    name
-FROM contacts
-WHERE name REGEXP '[^ -~]'
-ORDER BY id
-LIMIT 50;
-
--- Products with corruption
-SELECT '' as '';
-SELECT 'PRODUCTS TABLE - Corrupted Records:' as '';
-SELECT '-------------------------------------' as '';
-SELECT
-    id,
-    name,
-    LEFT(description, 50) as description_preview
-FROM products
-WHERE name REGEXP '[^ -~]'
-   OR description REGEXP '[^ -~]'
-ORDER BY id
-LIMIT 50;
-
--- Common corruption patterns found
-SELECT '' as '';
-SELECT '============================================================' as '';
-SELECT 'COMMON CORRUPTION PATTERNS DETECTED' as '';
-SELECT '============================================================' as '';
-
-SELECT
-    '’' as mojibake_pattern,
-    'Smart apostrophe (')' as should_be,
-    COUNT(*) as occurrences_in_customers
-FROM customers
-WHERE name LIKE '%’%'
-
-UNION ALL
-
-SELECT
-    '“ or â€',
-    'Smart quotes (" ")',
-    COUNT(*)
-FROM customers
-WHERE name LIKE '%“%' OR name LIKE '%â€%'
-
-UNION ALL
-
-SELECT
-    'é, ü, ó, etc',
-    'Accented chars (é, ü, ó)',
-    COUNT(*)
-FROM customers
-WHERE name REGEXP 'Ã[©¼³±§]'
-
-UNION ALL
-
-SELECT
-    'â€" or â€"',
-    'Dashes (– —)',
-    COUNT(*)
-FROM addresses
-WHERE address LIKE '%â€"%' OR address LIKE '%â€"%'
-
-UNION ALL
-
-SELECT
-    'Ä, ž, Å¡, etc',
-    'Czech/Polish chars',
-    COUNT(*)
-FROM principles
-WHERE address REGEXP '[ÄÅ][¾¡›™]';
-
-SELECT '' as '';
-SELECT '============================================================' as '';
-SELECT 'END OF REPORT' as '';
-SELECT '============================================================' as '';

scripts/run_comprehensive_fix.sh

@@ -1,115 +0,0 @@
-#!/bin/bash
-# ============================================================================
-# Comprehensive Corruption Fix - All Tables
-# ============================================================================
-# This script:
-# 1. Creates a backup
-# 2. Scans all tables for corruption
-# 3. Prompts for confirmation
-# 4. Applies fixes to ALL tables
-# 5. Shows before/after comparison
-# ============================================================================
-
-set -e  # Exit on error
-
-DB_USER="root"
-DB_PASS="secureRootPassword"
-DB_NAME="cmc"
-CONTAINER="cmc-db"
-BACKUP_DIR="./backups"
-TIMESTAMP=$(date +%Y%m%d_%H%M%S)
-
-# Colors for output
-RED='\033[0;31m'
-GREEN='\033[0;32m'
-YELLOW='\033[1;33m'
-BLUE='\033[0;34m'
-NC='\033[0m' # No Color
-
-echo -e "${BLUE}============================================================${NC}"
-echo -e "${BLUE}CMC Database - Comprehensive Corruption Fix${NC}"
-echo -e "${BLUE}============================================================${NC}"
-echo ""
-
-# Create backups directory
-mkdir -p "$BACKUP_DIR"
-
-# Step 1: Create backup
-echo -e "${YELLOW}Step 1: Creating full database backup...${NC}"
-docker exec $CONTAINER mariadb-dump \
-    -u $DB_USER \
-    -p$DB_PASS \
-    --default-character-set=utf8mb4 \
-    --single-transaction \
-    $DB_NAME | gzip > "$BACKUP_DIR/backup_comprehensive_fix_$TIMESTAMP.sql.gz"
-
-BACKUP_SIZE=$(du -h "$BACKUP_DIR/backup_comprehensive_fix_$TIMESTAMP.sql.gz" | cut -f1)
-echo -e "${GREEN}✓ Backup created: backup_comprehensive_fix_$TIMESTAMP.sql.gz (${BACKUP_SIZE})${NC}"
-echo ""
-
-# Step 2: Scan all tables
-echo -e "${YELLOW}Step 2: Scanning ALL tables for corruption...${NC}"
-echo "-------------------------------------------"
-docker exec $CONTAINER mariadb \
-    -u $DB_USER \
-    -p$DB_PASS \
-    --default-character-set=utf8mb4 \
-    $DB_NAME < scripts/scan_all_tables_for_corruption.sql | head -100
-echo ""
-
-# Step 3: Prompt for confirmation
-echo -e "${RED}IMPORTANT:${NC}"
-echo "- Backup created at: $BACKUP_DIR/backup_comprehensive_fix_$TIMESTAMP.sql.gz"
-echo "- This will modify ALL text columns in ALL tables"
-echo "- Fixes mojibake patterns like: ’ → ', é → é, ° → °, etc."
-echo "- Tables affected: customers, principles, addresses, products, contacts,"
-echo "  emails, invoices, jobs, quotes, purchase_orders, shipments, etc."
-echo ""
-read -p "Do you want to proceed with the comprehensive fix? (yes/no): " CONFIRM
-
-if [ "$CONFIRM" != "yes" ]; then
-    echo -e "${RED}Fix cancelled by user${NC}"
-    exit 0
-fi
-
-# Step 4: Apply comprehensive fixes
-echo ""
-echo -e "${YELLOW}Step 4: Applying comprehensive corruption fixes...${NC}"
-docker exec -i $CONTAINER mariadb \
-    -u $DB_USER \
-    -p$DB_PASS \
-    --default-character-set=utf8mb4 \
-    $DB_NAME < scripts/fix_all_corruption_safe.sql
-
-echo -e "${GREEN}✓ Comprehensive fix executed${NC}"
-echo ""
-
-# Step 5: Re-scan to show results
-echo -e "${YELLOW}Step 5: Re-scanning to verify fixes...${NC}"
-echo "---------------------------------------"
-docker exec $CONTAINER mariadb \
-    -u $DB_USER \
-    -p$DB_PASS \
-    --default-character-set=utf8mb4 \
-    $DB_NAME < scripts/scan_all_tables_for_corruption.sql | head -100
-echo ""
-
-# Final summary
-echo -e "${BLUE}============================================================${NC}"
-echo -e "${GREEN}COMPREHENSIVE FIX COMPLETED!${NC}"
-echo -e "${BLUE}============================================================${NC}"
-echo ""
-echo "Summary:"
-echo "- Backup location: $BACKUP_DIR/backup_comprehensive_fix_$TIMESTAMP.sql.gz"
-echo "- Fixed corruption patterns across ALL tables"
-echo "- Common patterns: smart quotes, accented characters, symbols, HTML entities"
-echo ""
-echo "Next steps:"
-echo "1. Review the fixed data in the application"
-echo "2. Test critical functionality (quotes, invoices, emails, etc.)"
-echo "3. Run: bash scripts/report_remaining_corruption.sh if any issues remain"
-echo ""
-echo "To rollback if needed:"
-echo "  gunzip < $BACKUP_DIR/backup_comprehensive_fix_$TIMESTAMP.sql.gz | \\"
-echo "  docker exec -i $CONTAINER mariadb -u $DB_USER -p$DB_PASS $DB_NAME"
-echo ""

scripts/run_corruption_fix.sh

@@ -1,187 +0,0 @@
-#!/bin/bash
-# ============================================================================
-# Safe Corruption Fix Execution Script
-# ============================================================================
-# This script:
-# 1. Creates a backup
-# 2. Shows current corruption status
-# 3. Prompts for confirmation
-# 4. Applies fixes
-# 5. Shows before/after comparison
-# ============================================================================
-
-set -e  # Exit on error
-
-DB_USER="root"
-DB_PASS="secureRootPassword"
-DB_NAME="cmc"
-CONTAINER="cmc-db"
-BACKUP_DIR="./backups"
-TIMESTAMP=$(date +%Y%m%d_%H%M%S)
-
-# Colors for output
-RED='\033[0;31m'
-GREEN='\033[0;32m'
-YELLOW='\033[1;33m'
-BLUE='\033[0;34m'
-NC='\033[0m' # No Color
-
-echo -e "${BLUE}============================================================${NC}"
-echo -e "${BLUE}CMC Database Corruption Fix Script${NC}"
-echo -e "${BLUE}============================================================${NC}"
-echo ""
-
-# Create backups directory
-mkdir -p "$BACKUP_DIR"
-
-# Step 1: Create backup
-echo -e "${YELLOW}Step 1: Creating database backup...${NC}"
-docker exec $CONTAINER mariadb-dump \
-    -u $DB_USER \
-    -p$DB_PASS \
-    --default-character-set=utf8mb4 \
-    --single-transaction \
-    $DB_NAME | gzip > "$BACKUP_DIR/backup_before_corruption_fix_$TIMESTAMP.sql.gz"
-
-BACKUP_SIZE=$(du -h "$BACKUP_DIR/backup_before_corruption_fix_$TIMESTAMP.sql.gz" | cut -f1)
-echo -e "${GREEN}✓ Backup created: backup_before_corruption_fix_$TIMESTAMP.sql.gz (${BACKUP_SIZE})${NC}"
-echo ""
-
-# Step 2: Show current corruption status
-echo -e "${YELLOW}Step 2: Current corruption status${NC}"
-echo "-----------------------------------"
-docker exec $CONTAINER mariadb -u $DB_USER -p$DB_PASS --default-character-set=utf8mb4 $DB_NAME -e "
-SELECT
-    'principles' as table_name,
-    COUNT(*) as corrupted_records
-FROM principles
-WHERE name REGEXP '[^ -~]'
-   OR address REGEXP '[^ -~]'
-   OR city REGEXP '[^ -~]'
-UNION ALL
-SELECT 'customers', COUNT(*)
-FROM customers
-WHERE name REGEXP '[^ -~]'
-UNION ALL
-SELECT 'addresses', COUNT(*)
-FROM addresses
-WHERE address REGEXP '[^ -~]'
-   OR city REGEXP '[^ -~]'
-UNION ALL
-SELECT 'products', COUNT(*)
-FROM products
-WHERE title REGEXP '[^ -~]'
-   OR description REGEXP '[^ -~]'
-   OR item_description REGEXP '[^ -~]';
-"
-echo ""
-
-# Step 3: Show sample corrupted data
-echo -e "${YELLOW}Step 3: Sample corrupted data (BEFORE fix)${NC}"
-echo "-------------------------------------------"
-docker exec $CONTAINER mariadb -u $DB_USER -p$DB_PASS --default-character-set=utf8mb4 $DB_NAME -e "
-SELECT 'CUSTOMERS' as table_name, id, name FROM customers WHERE id IN (253, 1006, 1387, 1608)
-UNION ALL
-SELECT 'PRINCIPLES', id, name FROM principles WHERE id IN (2, 9, 13, 14)
-UNION ALL
-SELECT 'PRODUCTS', id, title FROM products WHERE id IN (30, 38, 67);
-"
-echo ""
-
-# Step 4: Prompt for confirmation
-echo -e "${RED}IMPORTANT:${NC}"
-echo "- Backup created at: $BACKUP_DIR/backup_before_corruption_fix_$TIMESTAMP.sql.gz"
-echo "- This will modify data in tables: principles, customers, addresses, products"
-echo "- Changes fix mojibake like: ’ → ', é → é, etc."
-echo ""
-read -p "Do you want to proceed with the fix? (yes/no): " CONFIRM
-
-if [ "$CONFIRM" != "yes" ]; then
-    echo -e "${RED}Fix cancelled by user${NC}"
-    exit 0
-fi
-
-# Step 5: Apply fixes
-echo ""
-echo -e "${YELLOW}Step 5: Applying corruption fixes...${NC}"
-docker exec -i $CONTAINER mariadb \
-    -u $DB_USER \
-    -p$DB_PASS \
-    --default-character-set=utf8mb4 \
-    $DB_NAME < scripts/fix_corrupted_data.sql
-
-echo -e "${GREEN}✓ Fix script executed${NC}"
-echo ""
-
-# Step 6: Show results
-echo -e "${YELLOW}Step 6: Results (AFTER fix)${NC}"
-echo "----------------------------"
-echo ""
-echo "Sample fixed data:"
-docker exec $CONTAINER mariadb -u $DB_USER -p$DB_PASS --default-character-set=utf8mb4 $DB_NAME -e "
-SELECT 'CUSTOMERS' as table_name, id, name FROM customers WHERE id IN (253, 1006, 1387, 1608)
-UNION ALL
-SELECT 'PRINCIPLES', id, name FROM principles WHERE id IN (2, 9, 13, 14)
-UNION ALL
-SELECT 'PRODUCTS', id, title FROM products WHERE id IN (30, 38, 67);
-"
-echo ""
-
-echo "Remaining corruption counts:"
-docker exec $CONTAINER mariadb -u $DB_USER -p$DB_PASS --default-character-set=utf8mb4 $DB_NAME -e "
-SELECT
-    'principles' as table_name,
-    COUNT(*) as corrupted_records
-FROM principles
-WHERE name REGEXP '[^ -~]'
-   OR address REGEXP '[^ -~]'
-   OR city REGEXP '[^ -~]'
-UNION ALL
-SELECT 'customers', COUNT(*)
-FROM customers
-WHERE name REGEXP '[^ -~]'
-UNION ALL
-SELECT 'addresses', COUNT(*)
-FROM addresses
-WHERE address REGEXP '[^ -~]'
-   OR city REGEXP '[^ -~]'
-UNION ALL
-SELECT 'products', COUNT(*)
-FROM products
-WHERE title REGEXP '[^ -~]'
-   OR description REGEXP '[^ -~]'
-   OR item_description REGEXP '[^ -~]';
-"
-echo ""
-
-# Step 7: Test new insert
-echo -e "${YELLOW}Step 7: Testing new inserts with special characters...${NC}"
-docker exec $CONTAINER mariadb -u $DB_USER -p$DB_PASS --default-character-set=utf8mb4 $DB_NAME -e "
-START TRANSACTION;
-INSERT INTO principles (name, code, address, city, state, postcode, country_id, currency_id, short_name, url)
-VALUES ('Test Müller & Söhne GmbH', 999999, 'Straße 123', 'München', 'Bayern', '80331', 1, 1, 'TEST', 'http://test.com');
-SELECT name, address, city FROM principles WHERE code = 999999;
-ROLLBACK;
-"
-echo -e "${GREEN}✓ New inserts working correctly${NC}"
-echo ""
-
-# Final summary
-echo -e "${BLUE}============================================================${NC}"
-echo -e "${GREEN}FIX COMPLETED SUCCESSFULLY!${NC}"
-echo -e "${BLUE}============================================================${NC}"
-echo ""
-echo "Summary:"
-echo "- Backup location: $BACKUP_DIR/backup_before_corruption_fix_$TIMESTAMP.sql.gz"
-echo "- Fixed common patterns: smart quotes, accented characters, dashes"
-echo "- Fixed specific records: ~30+ known corrupted records"
-echo ""
-echo "Next steps:"
-echo "1. Review the fixed data in the application"
-echo "2. Run: bash scripts/report_remaining_corruption.sh to see what's left"
-echo "3. Manually fix any remaining corrupted records"
-echo ""
-echo "To rollback if needed:"
-echo "  gunzip < $BACKUP_DIR/backup_before_corruption_fix_$TIMESTAMP.sql.gz | \\"
-echo "  docker exec -i $CONTAINER mariadb -u $DB_USER -p$DB_PASS $DB_NAME"
-echo ""

scripts/scan_all_tables_for_corruption.sql

@@ -1,126 +0,0 @@
--- ============================================================================
--- Scan ALL Tables for Character Corruption
--- ============================================================================
--- This script scans every table in the database for mojibake/corruption
--- Finds all VARCHAR and TEXT columns and checks them for non-ASCII characters
---
--- To run: docker exec cmc-db mariadb -u root -psecureRootPassword --default-character-set=utf8mb4 cmc < scripts/scan_all_tables_for_corruption.sql
--- ============================================================================
-
-SET NAMES utf8mb4;
-
-SELECT '============================================================' as '';
-SELECT 'COMPREHENSIVE CORRUPTION SCAN - ALL TABLES' as '';
-SELECT '============================================================' as '';
-SELECT '' as '';
-
--- Scan all major tables with text content
-SELECT 'Table: addresses' as '';
-SELECT COUNT(*) as corrupted_records, 'name' as column_name FROM addresses WHERE name REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'address' FROM addresses WHERE address REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'city' FROM addresses WHERE city REGEXP '[^ -~]';
-
-SELECT '' as '';
-SELECT 'Table: attachments' as '';
-SELECT COUNT(*) as corrupted_records, 'name' as column_name FROM attachments WHERE name REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'filename' FROM attachments WHERE filename REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'description' FROM attachments WHERE description REGEXP '[^ -~]';
-
-SELECT '' as '';
-SELECT 'Table: contacts' as '';
-SELECT COUNT(*) as corrupted_records, 'name' as column_name FROM contacts WHERE name REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'first_name' FROM contacts WHERE first_name REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'last_name' FROM contacts WHERE last_name REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'notes' FROM contacts WHERE notes REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'job_title' FROM contacts WHERE job_title REGEXP '[^ -~]';
-
-SELECT '' as '';
-SELECT 'Table: customers' as '';
-SELECT COUNT(*) as corrupted_records, 'name' as column_name FROM customers WHERE name REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'trading_name' FROM customers WHERE trading_name REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'notes' FROM customers WHERE notes REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'payment_terms' FROM customers WHERE payment_terms REGEXP '[^ -~]';
-
-SELECT '' as '';
-SELECT 'Table: documents' as '';
-SELECT COUNT(*) as corrupted_records, 'cmc_reference' as column_name FROM documents WHERE cmc_reference REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'shipping_details' FROM documents WHERE shipping_details REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'bill_to' FROM documents WHERE bill_to REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'ship_to' FROM documents WHERE ship_to REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'name' FROM documents WHERE name REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'subject' FROM documents WHERE subject REGEXP '[^ -~]';
-
-SELECT '' as '';
-SELECT 'Table: emails' as '';
-SELECT COUNT(*) as corrupted_records, 'from' as column_name FROM emails WHERE `from` REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'to' FROM emails WHERE `to` REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'subject' FROM emails WHERE subject REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'body' FROM emails WHERE body REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'bcc' FROM emails WHERE bcc REGEXP '[^ -~]';
-
-SELECT '' as '';
-SELECT 'Table: enquiries' as '';
-SELECT COUNT(*) as corrupted_records, 'title' as column_name FROM enquiries WHERE title REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'comments' FROM enquiries WHERE comments REGEXP '[^ -~]';
-
-SELECT '' as '';
-SELECT 'Table: invoices' as '';
-SELECT COUNT(*) as corrupted_records, 'title' as column_name FROM invoices WHERE title REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'ship_via' FROM invoices WHERE ship_via REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'fob' FROM invoices WHERE fob REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'comments' FROM invoices WHERE comments REGEXP '[^ -~]';
-
-SELECT '' as '';
-SELECT 'Table: jobs' as '';
-SELECT COUNT(*) as corrupted_records, 'title' as column_name FROM jobs WHERE title REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'comments' FROM jobs WHERE comments REGEXP '[^ -~]';
-
-SELECT '' as '';
-SELECT 'Table: line_items' as '';
-SELECT COUNT(*) as corrupted_records, 'name' as column_name FROM line_items WHERE name REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'description' FROM line_items WHERE description REGEXP '[^ -~]';
-
-SELECT '' as '';
-SELECT 'Table: order_acknowledgements' as '';
-SELECT COUNT(*) as corrupted_records, 'title' as column_name FROM order_acknowledgements WHERE title REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'comments' FROM order_acknowledgements WHERE comments REGEXP '[^ -~]';
-
-SELECT '' as '';
-SELECT 'Table: principles' as '';
-SELECT COUNT(*) as corrupted_records, 'name' as column_name FROM principles WHERE name REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'address' FROM principles WHERE address REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'city' FROM principles WHERE city REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'short_name' FROM principles WHERE short_name REGEXP '[^ -~]';
-
-SELECT '' as '';
-SELECT 'Table: products' as '';
-SELECT COUNT(*) as corrupted_records, 'name' as column_name FROM products WHERE name REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'title' FROM products WHERE title REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'description' FROM products WHERE description REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'item_description' FROM products WHERE item_description REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'notes' FROM products WHERE notes REGEXP '[^ -~]';
-
-SELECT '' as '';
-SELECT 'Table: purchase_orders' as '';
-SELECT COUNT(*) as corrupted_records, 'title' as column_name FROM purchase_orders WHERE title REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'ship_via' FROM purchase_orders WHERE ship_via REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'comments' FROM purchase_orders WHERE comments REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'fob' FROM purchase_orders WHERE fob REGEXP '[^ -~]';
-
-SELECT '' as '';
-SELECT 'Table: quotes' as '';
-SELECT COUNT(*) as corrupted_records, 'title' as column_name FROM quotes WHERE title REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'comments' FROM quotes WHERE comments REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'notes' FROM quotes WHERE notes REGEXP '[^ -~]';
-
-SELECT '' as '';
-SELECT 'Table: shipments' as '';
-SELECT COUNT(*) as corrupted_records, 'title' as column_name FROM shipments WHERE title REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'ship_via' FROM shipments WHERE ship_via REGEXP '[^ -~]'
-UNION ALL SELECT COUNT(*), 'comments' FROM shipments WHERE comments REGEXP '[^ -~]';
-
-SELECT '' as '';
-SELECT '============================================================' as '';
-SELECT 'SUMMARY - Tables with corruption (>0 records)' as '';
-SELECT '============================================================' as '';
--- This gives you a quick overview of which tables need fixing

scripts/verify_charset_fix.sh

@@ -1,163 +0,0 @@
-#!/bin/bash
-# Script to verify character encoding fix in MariaDB
-
-DB_USER="root"
-DB_PASS="secureRootPassword"
-DB_NAME="cmc"
-CONTAINER="cmc-db"
-
-echo "=========================================="
-echo "Character Encoding Verification Script"
-echo "=========================================="
-echo ""
-
-# Function to run MariaDB command
-run_sql() {
-    docker exec $CONTAINER mariadb -u $DB_USER -p$DB_PASS $DB_NAME -e "$1"
-}
-
-# Function to run MariaDB command with UTF-8 client
-run_sql_utf8() {
-    docker exec $CONTAINER mariadb -u $DB_USER -p$DB_PASS --default-character-set=utf8mb4 $DB_NAME -e "$1"
-}
-
-echo "1. Check database default charset"
-echo "-----------------------------------"
-docker exec $CONTAINER mariadb -u $DB_USER -p$DB_PASS -e "
-SELECT @@character_set_database, @@collation_database;
-"
-echo ""
-
-echo "2. Count tables by charset (BEFORE should show ~59 latin1)"
-echo "------------------------------------------------------------"
-run_sql "
-SELECT
-    CASE
-        WHEN TABLE_COLLATION LIKE 'latin1%' THEN 'latin1'
-        WHEN TABLE_COLLATION LIKE 'utf8mb4%' THEN 'utf8mb4'
-        WHEN TABLE_COLLATION LIKE 'utf8mb3%' THEN 'utf8mb3'
-        ELSE TABLE_COLLATION
-    END as charset,
-    COUNT(*) as table_count
-FROM information_schema.TABLES
-WHERE TABLE_SCHEMA='$DB_NAME'
-GROUP BY charset
-ORDER BY table_count DESC;
-"
-echo ""
-
-echo "3. Check specific table charsets (sample)"
-echo "------------------------------------------"
-run_sql "
-SELECT TABLE_NAME, TABLE_COLLATION
-FROM information_schema.TABLES
-WHERE TABLE_SCHEMA='$DB_NAME'
-AND TABLE_NAME IN ('principles', 'customers', 'addresses', 'contacts', 'products')
-ORDER BY TABLE_NAME;
-"
-echo ""
-
-echo "4. Show column charsets for principles table"
-echo "---------------------------------------------"
-run_sql "
-SELECT COLUMN_NAME, CHARACTER_SET_NAME, COLLATION_NAME
-FROM information_schema.COLUMNS
-WHERE TABLE_SCHEMA='$DB_NAME'
-AND TABLE_NAME='principles'
-AND DATA_TYPE IN ('varchar', 'text', 'char')
-ORDER BY ORDINAL_POSITION;
-"
-echo ""
-
-echo "5. Test INSERT of special characters (will be rolled back)"
-echo "-----------------------------------------------------------"
-echo "Attempting to insert test record with special characters..."
-run_sql_utf8 "
-START TRANSACTION;
-
-INSERT INTO principles (name, code, address, city, state, postcode, country_id, currency_id, short_name, url)
-VALUES (
-    'Test Müller & Söhne GmbH',  -- German umlauts
-    999999,
-    'Straße 123, 2. Stock',      -- German ß and ² symbol
-    'München',                    -- German ü
-    'Bayern',
-    '80331',
-    1,
-    1,
-    'TEST',
-    'http://test.com'
-);
-
--- Read it back immediately
-SELECT id, name, address, city
-FROM principles
-WHERE code = 999999;
-
-ROLLBACK;
-" 2>&1 | grep -A 10 "id\|Test Müller\|ERROR"
-echo "✓ Test insert rolled back"
-echo ""
-
-echo "6. Verify current corrupted data in principles"
-echo "-----------------------------------------------"
-echo "These should show corruption if NOT yet fixed:"
-run_sql_utf8 "
-SELECT id, name, city
-FROM principles
-WHERE id IN (9, 18);
-"
-echo ""
-
-echo "7. Verify corrupted data in customers"
-echo "--------------------------------------"
-run_sql_utf8 "
-SELECT id, name
-FROM customers
-WHERE id IN (253, 1006, 1387, 1608);
-"
-echo ""
-
-echo "8. Count potentially corrupted records across tables"
-echo "-----------------------------------------------------"
-run_sql "
-SELECT
-    'principles' as table_name,
-    COUNT(*) as rows_with_non_ascii
-FROM principles
-WHERE name REGEXP '[^ -~]'
-   OR address REGEXP '[^ -~]'
-   OR city REGEXP '[^ -~]'
-UNION ALL
-SELECT
-    'customers',
-    COUNT(*)
-FROM customers
-WHERE name REGEXP '[^ -~]'
-UNION ALL
-SELECT
-    'addresses',
-    COUNT(*)
-FROM addresses
-WHERE address REGEXP '[^ -~]'
-   OR city REGEXP '[^ -~]'
-UNION ALL
-SELECT
-    'contacts',
-    COUNT(*)
-FROM contacts
-WHERE firstname REGEXP '[^ -~]'
-   OR lastname REGEXP '[^ -~]';
-"
-echo ""
-
-echo "=========================================="
-echo "Verification complete!"
-echo "=========================================="
-echo ""
-echo "INTERPRETATION:"
-echo "- If table_count for 'latin1' is ~59: NOT YET FIXED"
-echo "- If table_count for 'utf8mb4' is ~62: FIXED!"
-echo "- Test insert should show proper characters if fixed"
-echo "- Corrupted data will still show mojibake until manually repaired"
-echo ""

scripts/verify_charset_manual.sql

@@ -1,207 +0,0 @@
--- ============================================================================
--- Manual MariaDB Character Encoding Verification Commands
--- ============================================================================
--- Run these commands directly in MariaDB to verify the charset fix
---
--- Connect to database:
--- docker exec -it cmc-db mariadb -u root -psecureRootPassword --default-character-set=utf8mb4 cmc
--- ============================================================================
-
--- 1. CHECK DATABASE-LEVEL SETTINGS
--- Should show: utf8mb4
-SELECT @@character_set_database as db_charset,
-       @@collation_database as db_collation;
-
--- 2. COUNT TABLES BY CHARACTER SET
--- BEFORE FIX: ~59 latin1 tables
--- AFTER FIX: ~62 utf8mb4 tables
-SELECT
-    CASE
-        WHEN TABLE_COLLATION LIKE 'latin1%' THEN 'latin1'
-        WHEN TABLE_COLLATION LIKE 'utf8mb4%' THEN 'utf8mb4'
-        WHEN TABLE_COLLATION LIKE 'utf8mb3%' THEN 'utf8mb3'
-        ELSE TABLE_COLLATION
-    END as charset,
-    COUNT(*) as table_count
-FROM information_schema.TABLES
-WHERE TABLE_SCHEMA='cmc'
-GROUP BY charset
-ORDER BY table_count DESC;
-
--- 3. CHECK SPECIFIC TABLES (Key business tables)
-SELECT TABLE_NAME, TABLE_COLLATION, ENGINE
-FROM information_schema.TABLES
-WHERE TABLE_SCHEMA='cmc'
-AND TABLE_NAME IN (
-    'principles', 'customers', 'addresses', 'contacts',
-    'products', 'quotes', 'purchase_orders', 'invoices'
-)
-ORDER BY TABLE_NAME;
-
--- 4. CHECK ALL COLUMN CHARSETS FOR 'principles' TABLE
--- All text columns should show utf8mb4 after fix
-SELECT
-    COLUMN_NAME,
-    DATA_TYPE,
-    CHARACTER_SET_NAME,
-    COLLATION_NAME
-FROM information_schema.COLUMNS
-WHERE TABLE_SCHEMA='cmc'
-AND TABLE_NAME='principles'
-AND DATA_TYPE IN ('varchar', 'text', 'char')
-ORDER BY ORDINAL_POSITION;
-
--- 5. TEST INSERT WITH SPECIAL CHARACTERS
--- This will be rolled back - just testing if charset accepts them
-START TRANSACTION;
-
--- Insert test record with various special characters
-INSERT INTO principles (
-    name, code, address, city, state, postcode,
-    country_id, currency_id, short_name, url
-)
-VALUES (
-    'Test Müller & Söhne GmbH',     -- German umlauts (ü, ö)
-    999999,
-    'Straße 123, 2. Stock',         -- German ß
-    'München',                       -- German ü
-    'Bayern',
-    '80331',
-    1, 1, 'TEST', 'http://test.com'
-);
-
--- Read it back - should show properly if charset is correct
-SELECT id, name, address, city
-FROM principles
-WHERE code = 999999;
-
--- Expected output if FIXED:
--- name: Test Müller & Söhne GmbH (proper umlauts)
--- address: Straße 123, 2. Stock
--- city: München
-
--- Expected output if NOT FIXED:
--- Characters will be corrupted or insert will fail
-
-ROLLBACK;  -- Don't actually save the test record
-
--- 6. VIEW KNOWN CORRUPTED RECORDS
--- These show what the corruption looks like before manual repair
-
--- Principle ID 9 - Czech characters corrupted
-SELECT id, name, address, city
-FROM principles
-WHERE id = 9;
--- BEFORE FIX: BezruÄova 2901, Rožnov pod RadhoÅ¡tÄ›m
--- AFTER FIX (still needs manual repair): Same corruption
--- AFTER MANUAL REPAIR: Bezručova 2901, Rožnov pod Radhoštěm
-
--- Principle ID 18 - German umlaut in city
-SELECT id, short_name, city, country_id
-FROM principles
-WHERE id = 18;
-
--- Customer corrupted names
-SELECT id, name
-FROM customers
-WHERE id IN (253, 1006, 1387, 1608, 2174);
--- Look for: ’ (should be '), É (should be É), “ (should be ")
-
--- Address corrupted data
-SELECT id, address, city
-FROM addresses
-WHERE id IN (19, 25);
--- Look for: â€" (should be –)
-
--- 7. COUNT RECORDS WITH NON-ASCII CHARACTERS
--- Shows how many records might need manual repair after charset fix
-SELECT
-    'principles' as table_name,
-    COUNT(*) as records_with_non_ascii
-FROM principles
-WHERE name REGEXP '[^ -~]'
-   OR address REGEXP '[^ -~]'
-   OR city REGEXP '[^ -~]'
-
-UNION ALL
-
-SELECT 'customers', COUNT(*)
-FROM customers
-WHERE name REGEXP '[^ -~]'
-
-UNION ALL
-
-SELECT 'addresses', COUNT(*)
-FROM addresses
-WHERE address REGEXP '[^ -~]'
-   OR city REGEXP '[^ -~]'
-
-UNION ALL
-
-SELECT 'contacts', COUNT(*)
-FROM contacts
-WHERE firstname REGEXP '[^ -~]'
-   OR lastname REGEXP '[^ -~]';
-
--- 8. VERIFY CONNECTION CHARACTER SET
--- Your client should also be using utf8mb4
-SHOW VARIABLES LIKE 'character_set_%';
-SHOW VARIABLES LIKE 'collation_%';
-
--- Should see:
--- character_set_client: utf8mb4
--- character_set_connection: utf8mb4
--- character_set_results: utf8mb4
--- character_set_database: utf8mb4
-
--- ============================================================================
--- AFTER CONVERSION - TEST NEW INSERTS
--- ============================================================================
--- After running convert_charset_to_utf8mb4.sql, try these tests:
-
--- Test 1: Insert with German characters
-START TRANSACTION;
-INSERT INTO principles (name, code, address, city, state, postcode, country_id, currency_id, short_name, url)
-VALUES ('Müller GmbH', 999991, 'Hauptstraße 1', 'München', 'BY', '80331', 1, 1, 'MUL', 'https://test.de');
-SELECT * FROM principles WHERE code = 999991;
--- Should show: Müller, Hauptstraße, München (correctly)
-ROLLBACK;
-
--- Test 2: Insert with French characters
-START TRANSACTION;
-INSERT INTO principles (name, code, address, city, state, postcode, country_id, currency_id, short_name, url)
-VALUES ('Société François', 999992, 'Rue de la Paix', 'Orléans', 'FR', '45000', 1, 1, 'SOC', 'https://test.fr');
-SELECT * FROM principles WHERE code = 999992;
--- Should show: Société, François, Orléans (correctly)
-ROLLBACK;
-
--- Test 3: Insert with special punctuation
-START TRANSACTION;
-INSERT INTO customers (name, trading_name, abn, created, notes, discount_pricing_policies, payment_terms, customer_category_id, url, country_id)
-VALUES ('O'Connor & Sons — "Quality" Products', 'O'Connor Trading', '', NOW(), '', '', '', 1, '', 1);
-SELECT name FROM customers WHERE name LIKE '%O''Connor%' ORDER BY id DESC LIMIT 1;
--- Should show: O'Connor & Sons — "Quality" Products (with proper apostrophes, em-dash, and quotes)
-ROLLBACK;
-
--- ============================================================================
--- SUMMARY OF WHAT TO LOOK FOR
--- ============================================================================
-
-/*
-BEFORE FIX:
-- Table charset: latin1_swedish_ci
-- Column charset: latin1
-- Test insert: May fail or corrupt characters
-- Existing data: Shows mojibake (É, ’, etc.)
-
-AFTER FIX (charset conversion only):
-- Table charset: utf8mb4_general_ci
-- Column charset: utf8mb4
-- Test insert: Succeeds with proper characters
-- Existing data: STILL shows mojibake (needs manual repair)
-
-AFTER MANUAL REPAIR:
-- Everything displays correctly
-- New inserts work properly
-- Old data has been corrected
-*/