Compare commits
No commits in common. "master" and "finley/139" have entirely different histories.
master
...
finley/139
|
|
@ -82,8 +82,8 @@ services:
|
|||
- "8083:8082"
|
||||
volumes:
|
||||
- /home/cmc/files/pdf:/root/webroot/pdf:ro
|
||||
- /home/cmc/files/attachments_files:/root/webroot/attachments_files
|
||||
- /home/cmc/files/emails:/var/www/emails
|
||||
- /home/cmc/files/attachments_files:/var/www/attachments_files
|
||||
- /home/cmc/files/vault:/var/www/vault
|
||||
- /home/cmc/files/vaultmsgs:/var/www/vaultmsgs
|
||||
networks:
|
||||
|
|
|
|||
|
|
@ -80,7 +80,6 @@ services:
|
|||
- "8082:8082"
|
||||
volumes:
|
||||
- /home/cmc/files/pdf:/root/webroot/pdf:ro
|
||||
- /home/cmc/files/attachments_files:/root/webroot/attachments_files
|
||||
networks:
|
||||
- cmc-stg-network
|
||||
restart: unless-stopped
|
||||
|
|
|
|||
|
|
@ -172,13 +172,11 @@ func (h *AttachmentHandler) Create(w http.ResponseWriter, r *http.Request) {
|
|||
}
|
||||
|
||||
// Create database record
|
||||
// Store path in PHP format: /var/www/cmc-sales/app/webroot/attachments_files/filename
|
||||
phpPath := "/var/www/cmc-sales/app/webroot/attachments_files/" + filename
|
||||
params := db.CreateAttachmentParams{
|
||||
PrincipleID: int32(principleID),
|
||||
Name: name,
|
||||
Filename: handler.Filename,
|
||||
File: phpPath, // Store PHP container path for compatibility
|
||||
File: filePath, // Store full path for PHP compatibility
|
||||
Type: handler.Header.Get("Content-Type"),
|
||||
Size: int32(handler.Size),
|
||||
Description: description,
|
||||
|
|
|
|||
|
|
@ -1,250 +0,0 @@
|
|||
# Vault Permissions Fix - Attachment Directory Issue
|
||||
|
||||
## Problem
|
||||
|
||||
The PHP container (`cmc-prod-php`) could not read attachments written by the Go vault because they were being written to **different directories**.
|
||||
|
||||
### Root Cause Analysis
|
||||
|
||||
**Go Vault Container (`cmc-prod-go`):**
|
||||
- Was writing attachments to: `/var/www/emails/` (default `--emaildir` flag)
|
||||
- Host directory: `/home/cmc/files/emails`
|
||||
- Running as: `root` user (UID 0)
|
||||
|
||||
**PHP Application Container (`cmc-prod-php`):**
|
||||
- Reads attachments from: `/var/www/cmc-sales/app/webroot/attachments_files/`
|
||||
- Host directory: `/home/cmc/files/attachments_files`
|
||||
- Running as: `www-data` user (UID 33 or 82)
|
||||
|
||||
### Two Issues Identified
|
||||
|
||||
1. **Different Directory Paths** (Primary Issue) ❌
|
||||
- Go wrote to `/var/www/emails` → Host: `/home/cmc/files/emails`
|
||||
- PHP read from `/var/www/cmc-sales/app/webroot/attachments_files` → Host: `/home/cmc/files/attachments_files`
|
||||
- These are **completely separate directories on the host**!
|
||||
|
||||
2. **File Permissions** (Secondary) ⚠️
|
||||
- Files: `0644` (`-rw-r--r--`) - World readable ✅
|
||||
- Directories: `0755` (`drwxr-xr-x`) - World readable/executable ✅
|
||||
- Owner: `root:root` (from Go container)
|
||||
- Reader: `www-data` (from PHP container)
|
||||
- **Verdict**: Permissions were actually OK (world-readable), but wrong directory!
|
||||
|
||||
## Solution Implemented
|
||||
|
||||
### Changes Made
|
||||
|
||||
**1. Updated Docker Compose ([docker-compose.prod.yml](../docker-compose.prod.yml))**
|
||||
|
||||
Added the `attachments_files` mount to the Go container:
|
||||
|
||||
```yaml
|
||||
cmc-prod-go:
|
||||
volumes:
|
||||
- /home/cmc/files/attachments_files:/var/www/attachments_files # ← ADDED
|
||||
- /home/cmc/files/emails:/var/www/emails
|
||||
- /home/cmc/files/vault:/var/www/vault
|
||||
- /home/cmc/files/vaultmsgs:/var/www/vaultmsgs
|
||||
```
|
||||
|
||||
**2. Updated Vault Cron Script ([scripts/vault-cron-prod.sh](vault-cron-prod.sh))**
|
||||
|
||||
Changed the `--emaildir` flag to point to the correct directory:
|
||||
|
||||
```bash
|
||||
docker exec -t "$CONTAINER_NAME" ./vault --mode=local \
|
||||
--vaultdir=/var/www/vaultmsgs/new \
|
||||
--processeddir=/var/www/vaultmsgs/cur \
|
||||
--emaildir=/var/www/attachments_files \ # ← CHANGED from /var/www/emails
|
||||
--dbhost=cmc-prod-db \
|
||||
--dbuser=cmc \
|
||||
--dbpass="xVRQI&cA?7AU=hqJ!%au" \
|
||||
--dbname=cmc
|
||||
```
|
||||
|
||||
**3. Updated Documentation**
|
||||
|
||||
- [VAULT_QUICKSTART.md](VAULT_QUICKSTART.md) - Updated all examples
|
||||
- [VAULT_DEPLOYMENT.md](VAULT_DEPLOYMENT.md) - Updated deployment instructions
|
||||
|
||||
## Verification Steps
|
||||
|
||||
After deploying the fix, verify it works:
|
||||
|
||||
### Step 1: Rebuild and Restart Containers
|
||||
|
||||
```bash
|
||||
cd ~/src/cmc-sales
|
||||
docker compose -f docker-compose.prod.yml up -d cmc-prod-go
|
||||
```
|
||||
|
||||
### Step 2: Verify Mount Points
|
||||
|
||||
```bash
|
||||
# Check Go container can see the directory
|
||||
docker exec -it cmc-prod-go ls -la /var/www/attachments_files
|
||||
|
||||
# Check PHP container can see the directory
|
||||
docker exec -it cmc-prod-php ls -la /var/www/cmc-sales/app/webroot/attachments_files
|
||||
```
|
||||
|
||||
Both should show the same files (same inode on host).
|
||||
|
||||
### Step 3: Test Vault Processing
|
||||
|
||||
```bash
|
||||
# Run vault manually
|
||||
~/scripts/vault-cron-prod.sh
|
||||
|
||||
# Check if files were written
|
||||
ls -la /home/cmc/files/attachments_files/$(date +%m-%Y)/
|
||||
|
||||
# Check file permissions
|
||||
ls -l /home/cmc/files/attachments_files/$(date +%m-%Y)/ | head -5
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
-rw-r--r-- 1 root root 1234 Nov 23 10:00 abc123-texthtml
|
||||
-rw-r--r-- 1 root root 5678 Nov 23 10:00 abc123-document.pdf
|
||||
```
|
||||
|
||||
### Step 4: Verify PHP Can Read Files
|
||||
|
||||
```bash
|
||||
# Enter PHP container as www-data user
|
||||
docker exec -it -u www-data cmc-prod-php sh
|
||||
|
||||
# Try to read an attachment file
|
||||
cd /var/www/cmc-sales/app/webroot/attachments_files/$(date +%m-%Y)
|
||||
ls -la
|
||||
cat abc123-texthtml # Should display content
|
||||
exit
|
||||
```
|
||||
|
||||
### Step 5: Check Database
|
||||
|
||||
```bash
|
||||
# Verify attachments are in database with correct paths
|
||||
docker exec -it cmc-prod-db mariadb -u cmc -p cmc -e "
|
||||
SELECT id, email_id, filename, name, size
|
||||
FROM email_attachments
|
||||
ORDER BY created DESC
|
||||
LIMIT 5;
|
||||
"
|
||||
```
|
||||
|
||||
The `name` column should contain paths like: `11-2025/abc123-filename.pdf`
|
||||
|
||||
## File Permissions Breakdown
|
||||
|
||||
The current permissions (`0644` for files, `0755` for dirs) are correct:
|
||||
|
||||
### File Permissions: `0644`
|
||||
|
||||
```
|
||||
-rw-r--r--
|
||||
│││││││││
|
||||
│││└─┴─┴─── Other (everyone): read (r--)
|
||||
││└─┬─┴───── Group: read (r--)
|
||||
│└─┬┴─────── Owner: read + write (rw-)
|
||||
└─────────── Regular file (-)
|
||||
```
|
||||
|
||||
**Result**: ✅ All users (including www-data) can **read** the files
|
||||
|
||||
### Directory Permissions: `0755`
|
||||
|
||||
```
|
||||
drwxr-xr-x
|
||||
││││││││││
|
||||
│││└─┴─┴─── Other: read + execute (r-x) - can list & access
|
||||
││└─┬─┴───── Group: read + execute (r-x)
|
||||
│└─┬┴─────── Owner: read + write + execute (rwx)
|
||||
└─────────── Directory (d)
|
||||
```
|
||||
|
||||
**Result**: ✅ All users can **navigate into** and **list** the directories
|
||||
|
||||
### Why This Works
|
||||
|
||||
1. **Files are world-readable** (`r--` in other bits)
|
||||
2. **Directories are world-executable** (`x` in other bits)
|
||||
3. Even though owner is `root:root`, the "other" permissions allow `www-data` to read
|
||||
|
||||
### When You'd Need to Change Permissions
|
||||
|
||||
You would **only** need different permissions if:
|
||||
|
||||
| Scenario | Change To | Reason |
|
||||
|----------|-----------|--------|
|
||||
| Only owner should read | `0600` / `0700` | Security/privacy |
|
||||
| Group needs write access | `0664` / `0775` | Collaborative editing |
|
||||
| Everyone needs write | `0666` / `0777` | ⚠️ **NOT recommended** - security risk |
|
||||
|
||||
For your use case: **Keep `0644` / `0755`** ✅
|
||||
|
||||
## Database Path Storage
|
||||
|
||||
The vault stores relative paths in the database:
|
||||
|
||||
```sql
|
||||
-- email_attachments table
|
||||
name: "11-2025/abc123-filename.pdf"
|
||||
```
|
||||
|
||||
The PHP application prepends the base path:
|
||||
|
||||
```php
|
||||
$fullPath = "/var/www/cmc-sales/app/webroot/attachments_files/" . $attachment['name'];
|
||||
// Results in: /var/www/cmc-sales/app/webroot/attachments_files/11-2025/abc123-filename.pdf
|
||||
```
|
||||
|
||||
This is why the directory mount must be correct!
|
||||
|
||||
## Alternative Solutions (Not Implemented)
|
||||
|
||||
### Alternative 1: Symlink (Not Needed)
|
||||
|
||||
Could create a symlink from `/var/www/emails` → `/var/www/attachments_files`, but this adds complexity.
|
||||
|
||||
### Alternative 2: Change User ID (Overkill)
|
||||
|
||||
Could run Go container as `www-data` user, but:
|
||||
- Requires rebuilding the Dockerfile
|
||||
- Adds unnecessary complexity
|
||||
- Current permissions already work
|
||||
|
||||
### Alternative 3: Set Group Permissions (Unnecessary)
|
||||
|
||||
Could use `0664` / `0775` and shared group, but:
|
||||
- World-readable permissions already work
|
||||
- No need for write access
|
||||
- Simpler is better
|
||||
|
||||
## Summary
|
||||
|
||||
**What was fixed:**
|
||||
- ✅ Go vault now writes to `/var/www/attachments_files` (same as PHP reads from)
|
||||
- ✅ Added volume mount for `attachments_files` directory
|
||||
- ✅ Updated all scripts and documentation
|
||||
|
||||
**What didn't need fixing:**
|
||||
- ✅ File permissions (`0644`) are already world-readable
|
||||
- ✅ Directory permissions (`0755`) are already world-accessible
|
||||
- ✅ No user/group changes needed
|
||||
|
||||
**Root cause:**
|
||||
- Wrong directory path, NOT permissions issue
|
||||
|
||||
## Deployment Checklist
|
||||
|
||||
- [ ] Pull latest code with fixes
|
||||
- [ ] Restart Go container (`docker compose up -d cmc-prod-go`)
|
||||
- [ ] Verify mount points in both containers
|
||||
- [ ] Run vault manually to test
|
||||
- [ ] Check files appear in `/home/cmc/files/attachments_files/`
|
||||
- [ ] Verify PHP can read the files
|
||||
- [ ] Check database paths are correct
|
||||
- [ ] Enable cron job
|
||||
- [ ] Monitor logs for 24 hours
|
||||
|
|
@ -1,59 +0,0 @@
|
|||
ALTER TABLE `acos` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `addresses` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `aros` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `aros_acos` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `boxes` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `cake_sessions` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `contacts` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `contact_categories` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `costings` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `countries` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `currencies` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `customers` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `customer_categories` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `documents` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `doc_pages` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `emails` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `emails_enquiries` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `emails_invoices` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `emails_jobs` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `emails_purchase_orders` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `email_attachments` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `email_recipients` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `enquiries` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `enquiry_email_queues` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `freight_forwarders` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `freight_services` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `groups` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `industries` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `industries_customers` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `invoices` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `jobs` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `jobstatus` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `jobs_purchase_orders` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `job_products` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `line_items` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `order_acknowledgements` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `packing_lists` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `pages` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `principles` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `principle_addresses` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `principle_contacts` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `products` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `product_attachments` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `product_categories` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `product_options` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `product_options_categories` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `purchase_orders` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `quotes` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `quote_pages` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `quote_products` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `shipments` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `shipments_jobs` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `shipments_principles` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `shipments_purchase_orders` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `shipment_categories` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `shipment_invoices` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `states` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `statuses` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
ALTER TABLE `users` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
|
||||
|
|
@ -79,7 +79,7 @@ ENVEOF
|
|||
if [[ "$USE_CACHE" == "false" ]]; then
|
||||
echo "Building and starting docker compose for production (no cache)..."
|
||||
docker compose --env-file .env.prod -f docker-compose.prod.yml build --no-cache
|
||||
docker compose --env-file .env.prod -f docker-compose.prod.yml up -d --force-recreate --remove-orphans
|
||||
docker compose --env-file .env.prod -f docker-compose.prod.yml up -d --remove-orphans
|
||||
else
|
||||
echo "Building and starting docker compose for production (using cache)..."
|
||||
docker compose --env-file .env.prod -f docker-compose.prod.yml build
|
||||
|
|
|
|||
|
|
@ -70,7 +70,7 @@ ENVEOF
|
|||
if [[ "$USE_CACHE" == "false" ]]; then
|
||||
echo "Building and starting docker compose for staging (no cache)..."
|
||||
docker compose -f docker-compose.stg.yml build --no-cache
|
||||
docker compose -f docker-compose.stg.yml up -d --force-recreate --remove-orphans
|
||||
docker compose -f docker-compose.stg.yml up -d --remove-orphans
|
||||
else
|
||||
echo "Building and starting docker compose for staging (using cache)..."
|
||||
docker compose -f docker-compose.stg.yml build
|
||||
|
|
|
|||
|
|
@ -1,347 +0,0 @@
|
|||
-- ============================================================================
|
||||
-- Fix Character Corruption in ALL Tables
|
||||
-- ============================================================================
|
||||
-- This script fixes mojibake across the entire database
|
||||
-- Run this AFTER running scan_all_tables_for_corruption.sql to see what's affected
|
||||
--
|
||||
-- IMPORTANT: Create a backup first!
|
||||
-- docker exec cmc-db mariadb-dump -u root -psecureRootPassword --default-character-set=utf8mb4 cmc | gzip > backup_before_complete_fix_$(date +%Y%m%d).sql.gz
|
||||
--
|
||||
-- To run: docker exec -i cmc-db mariadb -u root -psecureRootPassword --default-character-set=utf8mb4 cmc < scripts/fix_all_corruption.sql
|
||||
-- ============================================================================
|
||||
|
||||
SET NAMES utf8mb4;
|
||||
|
||||
SELECT '============================================================' as '';
|
||||
SELECT 'COMPREHENSIVE CORRUPTION FIX - ALL TABLES' as '';
|
||||
SELECT '============================================================' as '';
|
||||
|
||||
-- ============================================================================
|
||||
-- Define reusable corruption patterns
|
||||
-- These are applied to every text/varchar column
|
||||
-- ============================================================================
|
||||
|
||||
-- Smart quotes and punctuation
|
||||
-- ’ → ' (smart apostrophe)
|
||||
-- “ → " (left double quote)
|
||||
-- †→ " (right double quote)
|
||||
-- â€" → – (en dash)
|
||||
-- â€" → — (em dash)
|
||||
-- ​ → (zero-width space, remove)
|
||||
--   → (space)
|
||||
-- … → …
|
||||
|
||||
-- Accented characters
|
||||
-- é → é
|
||||
-- É → É
|
||||
-- ó → ó
|
||||
-- Ã → í
|
||||
-- ç → ç
|
||||
-- ü → ü
|
||||
-- á → á
|
||||
-- ñ → ñ
|
||||
-- ö → ö
|
||||
-- ô → ô
|
||||
-- ß → ß
|
||||
-- ä → ä
|
||||
|
||||
-- Symbols
|
||||
-- ° → °
|
||||
-- ® → ®
|
||||
-- â„¢ → ™
|
||||
|
||||
-- ============================================================================
|
||||
-- ADDRESSES Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing addresses table...' as '';
|
||||
|
||||
UPDATE addresses SET name = REPLACE(name, '’', ''') WHERE name LIKE '%’%';
|
||||
UPDATE addresses SET name = REPLACE(name, '“', '"') WHERE name LIKE '%“%';
|
||||
UPDATE addresses SET name = REPLACE(name, 'â€', '"') WHERE name LIKE '%â€%';
|
||||
UPDATE addresses SET name = REPLACE(name, 'é', 'é') WHERE name LIKE '%é%';
|
||||
UPDATE addresses SET name = REPLACE(name, 'ü', 'ü') WHERE name LIKE '%ü%';
|
||||
|
||||
UPDATE addresses SET address = REPLACE(address, '’', ''') WHERE address LIKE '%’%';
|
||||
UPDATE addresses SET address = REPLACE(address, '“', '"') WHERE address LIKE '%“%';
|
||||
UPDATE addresses SET address = REPLACE(address, 'â€', '"') WHERE address LIKE '%â€%';
|
||||
UPDATE addresses SET address = REPLACE(address, 'â€"', '–') WHERE address LIKE '%â€"%';
|
||||
UPDATE addresses SET address = REPLACE(address, 'â€"', '—') WHERE address LIKE '%â€"%';
|
||||
UPDATE addresses SET address = REPLACE(address, 'é', 'é') WHERE address LIKE '%é%';
|
||||
UPDATE addresses SET address = REPLACE(address, 'ü', 'ü') WHERE address LIKE '%ü%';
|
||||
UPDATE addresses SET address = REPLACE(address, 'ó', 'ó') WHERE address LIKE '%ó%';
|
||||
UPDATE addresses SET address = REPLACE(address, 'ñ', 'ñ') WHERE address LIKE '%ñ%';
|
||||
|
||||
UPDATE addresses SET city = REPLACE(city, 'é', 'é') WHERE city LIKE '%é%';
|
||||
UPDATE addresses SET city = REPLACE(city, 'ü', 'ü') WHERE city LIKE '%ü%';
|
||||
UPDATE addresses SET city = REPLACE(city, 'ó', 'ó') WHERE city LIKE '%ó%';
|
||||
|
||||
-- ============================================================================
|
||||
-- ATTACHMENTS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing attachments table...' as '';
|
||||
|
||||
UPDATE attachments SET name = REPLACE(name, '’', ''') WHERE name LIKE '%’%';
|
||||
UPDATE attachments SET name = REPLACE(name, 'é', 'é') WHERE name LIKE '%é%';
|
||||
UPDATE attachments SET name = REPLACE(name, 'ü', 'ü') WHERE name LIKE '%ü%';
|
||||
|
||||
UPDATE attachments SET description = REPLACE(description, '’', ''') WHERE description LIKE '%’%';
|
||||
UPDATE attachments SET description = REPLACE(description, '“', '"') WHERE description LIKE '%“%';
|
||||
UPDATE attachments SET description = REPLACE(description, 'â€', '"') WHERE description LIKE '%â€%';
|
||||
UPDATE attachments SET description = REPLACE(description, 'é', 'é') WHERE description LIKE '%é%';
|
||||
UPDATE attachments SET description = REPLACE(description, 'ü', 'ü') WHERE description LIKE '%ü%';
|
||||
|
||||
-- ============================================================================
|
||||
-- CONTACTS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing contacts table...' as '';
|
||||
|
||||
UPDATE contacts SET name = REPLACE(name, '’', ''') WHERE name LIKE '%’%';
|
||||
UPDATE contacts SET name = REPLACE(name, 'é', 'é') WHERE name LIKE '%é%';
|
||||
UPDATE contacts SET name = REPLACE(name, 'ü', 'ü') WHERE name LIKE '%ü%';
|
||||
UPDATE contacts SET name = REPLACE(name, 'ó', 'ó') WHERE name LIKE '%ó%';
|
||||
|
||||
UPDATE contacts SET first_name = REPLACE(first_name, '’', ''') WHERE first_name LIKE '%’%';
|
||||
UPDATE contacts SET first_name = REPLACE(first_name, 'é', 'é') WHERE first_name LIKE '%é%';
|
||||
UPDATE contacts SET first_name = REPLACE(first_name, 'ü', 'ü') WHERE first_name LIKE '%ü%';
|
||||
|
||||
UPDATE contacts SET last_name = REPLACE(last_name, '’', ''') WHERE last_name LIKE '%’%';
|
||||
UPDATE contacts SET last_name = REPLACE(last_name, 'é', 'é') WHERE last_name LIKE '%é%';
|
||||
UPDATE contacts SET last_name = REPLACE(last_name, 'ü', 'ü') WHERE last_name LIKE '%ü%';
|
||||
|
||||
UPDATE contacts SET notes = REPLACE(notes, '’', ''') WHERE notes LIKE '%’%';
|
||||
UPDATE contacts SET notes = REPLACE(notes, '“', '"') WHERE notes LIKE '%“%';
|
||||
UPDATE contacts SET notes = REPLACE(notes, 'â€', '"') WHERE notes LIKE '%â€%';
|
||||
UPDATE contacts SET notes = REPLACE(notes, 'é', 'é') WHERE notes LIKE '%é%';
|
||||
UPDATE contacts SET notes = REPLACE(notes, 'ü', 'ü') WHERE notes LIKE '%ü%';
|
||||
|
||||
UPDATE contacts SET job_title = REPLACE(job_title, '’', ''') WHERE job_title LIKE '%’%';
|
||||
UPDATE contacts SET job_title = REPLACE(job_title, 'é', 'é') WHERE job_title LIKE '%é%';
|
||||
|
||||
-- ============================================================================
|
||||
-- CUSTOMERS Table (re-apply in case some were missed)
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing customers table...' as '';
|
||||
|
||||
UPDATE customers SET name = REPLACE(name, '’', ''') WHERE name LIKE '%’%';
|
||||
UPDATE customers SET name = REPLACE(name, '“', '"') WHERE name LIKE '%“%';
|
||||
UPDATE customers SET name = REPLACE(name, 'â€', '"') WHERE name LIKE '%â€%';
|
||||
UPDATE customers SET name = REPLACE(name, 'â€"', '–') WHERE name LIKE '%â€"%';
|
||||
UPDATE customers SET name = REPLACE(name, 'â€"', '—') WHERE name LIKE '%â€"%';
|
||||
UPDATE customers SET name = REPLACE(name, '​', '') WHERE name LIKE '%​%';
|
||||
UPDATE customers SET name = REPLACE(name, 'é', 'é') WHERE name LIKE '%é%';
|
||||
UPDATE customers SET name = REPLACE(name, 'ü', 'ü') WHERE name LIKE '%ü%';
|
||||
UPDATE customers SET name = REPLACE(name, 'ó', 'ó') WHERE name LIKE '%ó%';
|
||||
UPDATE customers SET name = REPLACE(name, 'ç', 'ç') WHERE name LIKE '%ç%';
|
||||
|
||||
UPDATE customers SET trading_name = REPLACE(trading_name, '’', ''') WHERE trading_name LIKE '%’%';
|
||||
UPDATE customers SET trading_name = REPLACE(trading_name, 'é', 'é') WHERE trading_name LIKE '%é%';
|
||||
|
||||
UPDATE customers SET notes = REPLACE(notes, '’', ''') WHERE notes LIKE '%’%';
|
||||
UPDATE customers SET notes = REPLACE(notes, '“', '"') WHERE notes LIKE '%“%';
|
||||
UPDATE customers SET notes = REPLACE(notes, 'â€', '"') WHERE notes LIKE '%â€%';
|
||||
UPDATE customers SET notes = REPLACE(notes, 'é', 'é') WHERE notes LIKE '%é%';
|
||||
|
||||
-- ============================================================================
|
||||
-- DOCUMENTS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing documents table...' as '';
|
||||
|
||||
UPDATE documents SET shipping_details = REPLACE(shipping_details, '’', ''') WHERE shipping_details LIKE '%’%';
|
||||
UPDATE documents SET shipping_details = REPLACE(shipping_details, 'é', 'é') WHERE shipping_details LIKE '%é%';
|
||||
|
||||
UPDATE documents SET bill_to = REPLACE(bill_to, '’', ''') WHERE bill_to LIKE '%’%';
|
||||
UPDATE documents SET bill_to = REPLACE(bill_to, 'é', 'é') WHERE bill_to LIKE '%é%';
|
||||
|
||||
UPDATE documents SET ship_to = REPLACE(ship_to, '’', ''') WHERE ship_to LIKE '%’%';
|
||||
UPDATE documents SET ship_to = REPLACE(ship_to, 'é', 'é') WHERE ship_to LIKE '%é%';
|
||||
|
||||
UPDATE documents SET subject = REPLACE(subject, '’', ''') WHERE subject LIKE '%’%';
|
||||
UPDATE documents SET subject = REPLACE(subject, '“', '"') WHERE subject LIKE '%“%';
|
||||
UPDATE documents SET subject = REPLACE(subject, 'â€', '"') WHERE subject LIKE '%â€%';
|
||||
UPDATE documents SET subject = REPLACE(subject, 'é', 'é') WHERE subject LIKE '%é%';
|
||||
|
||||
-- ============================================================================
|
||||
-- EMAILS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing emails table...' as '';
|
||||
|
||||
UPDATE emails SET `from` = REPLACE(`from`, '’', ''') WHERE `from` LIKE '%’%';
|
||||
UPDATE emails SET `from` = REPLACE(`from`, 'é', 'é') WHERE `from` LIKE '%é%';
|
||||
|
||||
UPDATE emails SET `to` = REPLACE(`to`, '’', ''') WHERE `to` LIKE '%’%';
|
||||
UPDATE emails SET `to` = REPLACE(`to`, 'é', 'é') WHERE `to` LIKE '%é%';
|
||||
|
||||
UPDATE emails SET subject = REPLACE(subject, '’', ''') WHERE subject LIKE '%’%';
|
||||
UPDATE emails SET subject = REPLACE(subject, '“', '"') WHERE subject LIKE '%“%';
|
||||
UPDATE emails SET subject = REPLACE(subject, 'â€', '"') WHERE subject LIKE '%â€%';
|
||||
UPDATE emails SET subject = REPLACE(subject, 'é', 'é') WHERE subject LIKE '%é%';
|
||||
UPDATE emails SET subject = REPLACE(subject, 'ü', 'ü') WHERE subject LIKE '%ü%';
|
||||
|
||||
UPDATE emails SET body = REPLACE(body, '’', ''') WHERE body LIKE '%’%';
|
||||
UPDATE emails SET body = REPLACE(body, '“', '"') WHERE body LIKE '%“%';
|
||||
UPDATE emails SET body = REPLACE(body, 'â€', '"') WHERE body LIKE '%â€%';
|
||||
UPDATE emails SET body = REPLACE(body, 'â€"', '–') WHERE body LIKE '%â€"%';
|
||||
UPDATE emails SET body = REPLACE(body, 'â€"', '—') WHERE body LIKE '%â€"%';
|
||||
UPDATE emails SET body = REPLACE(body, 'é', 'é') WHERE body LIKE '%é%';
|
||||
UPDATE emails SET body = REPLACE(body, 'ü', 'ü') WHERE body LIKE '%ü%';
|
||||
UPDATE emails SET body = REPLACE(body, '°', '°') WHERE body LIKE '%°%';
|
||||
UPDATE emails SET body = REPLACE(body, '®', '®') WHERE body LIKE '%®%';
|
||||
|
||||
-- ============================================================================
|
||||
-- ENQUIRIES Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing enquiries table...' as '';
|
||||
|
||||
UPDATE enquiries SET comments = REPLACE(comments, '’', ''') WHERE comments LIKE '%’%';
|
||||
UPDATE enquiries SET comments = REPLACE(comments, '“', '"') WHERE comments LIKE '%“%';
|
||||
UPDATE enquiries SET comments = REPLACE(comments, 'â€', '"') WHERE comments LIKE '%â€%';
|
||||
UPDATE enquiries SET comments = REPLACE(comments, 'é', 'é') WHERE comments LIKE '%é%';
|
||||
UPDATE enquiries SET comments = REPLACE(comments, 'ü', 'ü') WHERE comments LIKE '%ü%';
|
||||
|
||||
-- ============================================================================
|
||||
-- INVOICES Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing invoices table...' as '';
|
||||
|
||||
UPDATE invoices SET ship_via = REPLACE(ship_via, '’', ''') WHERE ship_via LIKE '%’%';
|
||||
UPDATE invoices SET ship_via = REPLACE(ship_via, 'é', 'é') WHERE ship_via LIKE '%é%';
|
||||
|
||||
UPDATE invoices SET comments = REPLACE(comments, '’', ''') WHERE comments LIKE '%’%';
|
||||
UPDATE invoices SET comments = REPLACE(comments, '“', '"') WHERE comments LIKE '%“%';
|
||||
UPDATE invoices SET comments = REPLACE(comments, 'â€', '"') WHERE comments LIKE '%â€%';
|
||||
UPDATE invoices SET comments = REPLACE(comments, 'é', 'é') WHERE comments LIKE '%é%';
|
||||
|
||||
-- ============================================================================
|
||||
-- JOBS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing jobs table...' as '';
|
||||
|
||||
UPDATE jobs SET comments = REPLACE(comments, '’', ''') WHERE comments LIKE '%’%';
|
||||
UPDATE jobs SET comments = REPLACE(comments, '“', '"') WHERE comments LIKE '%“%';
|
||||
UPDATE jobs SET comments = REPLACE(comments, 'â€', '"') WHERE comments LIKE '%â€%';
|
||||
UPDATE jobs SET comments = REPLACE(comments, 'é', 'é') WHERE comments LIKE '%é%';
|
||||
|
||||
-- ============================================================================
|
||||
-- LINE_ITEMS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing line_items table...' as '';
|
||||
|
||||
UPDATE line_items SET name = REPLACE(name, '’', ''') WHERE name LIKE '%’%';
|
||||
UPDATE line_items SET name = REPLACE(name, 'é', 'é') WHERE name LIKE '%é%';
|
||||
UPDATE line_items SET name = REPLACE(name, '°', '°') WHERE name LIKE '%°%';
|
||||
UPDATE line_items SET name = REPLACE(name, '®', '®') WHERE name LIKE '%®%';
|
||||
|
||||
UPDATE line_items SET description = REPLACE(description, '’', ''') WHERE description LIKE '%’%';
|
||||
UPDATE line_items SET description = REPLACE(description, '“', '"') WHERE description LIKE '%“%';
|
||||
UPDATE line_items SET description = REPLACE(description, 'â€', '"') WHERE description LIKE '%â€%';
|
||||
UPDATE line_items SET description = REPLACE(description, 'é', 'é') WHERE description LIKE '%é%';
|
||||
UPDATE line_items SET description = REPLACE(description, '°', '°') WHERE description LIKE '%°%';
|
||||
UPDATE line_items SET description = REPLACE(description, '®', '®') WHERE description LIKE '%®%';
|
||||
|
||||
-- ============================================================================
|
||||
-- ORDER_ACKNOWLEDGEMENTS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing order_acknowledgements table...' as '';
|
||||
|
||||
UPDATE order_acknowledgements SET comments = REPLACE(comments, '’', ''') WHERE comments LIKE '%’%';
|
||||
UPDATE order_acknowledgements SET comments = REPLACE(comments, '“', '"') WHERE comments LIKE '%“%';
|
||||
UPDATE order_acknowledgements SET comments = REPLACE(comments, 'â€', '"') WHERE comments LIKE '%â€%';
|
||||
UPDATE order_acknowledgements SET comments = REPLACE(comments, 'é', 'é') WHERE comments LIKE '%é%';
|
||||
|
||||
-- ============================================================================
|
||||
-- PRINCIPLES Table (re-apply)
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing principles table...' as '';
|
||||
|
||||
UPDATE principles SET name = REPLACE(name, 'ü', 'ü') WHERE name LIKE '%ü%';
|
||||
UPDATE principles SET name = REPLACE(name, 'é', 'é') WHERE name LIKE '%é%';
|
||||
UPDATE principles SET name = REPLACE(name, 'ó', 'ó') WHERE name LIKE '%ó%';
|
||||
|
||||
UPDATE principles SET address = REPLACE(address, 'ß', 'ß') WHERE address LIKE '%ß%';
|
||||
UPDATE principles SET address = REPLACE(address, 'ü', 'ü') WHERE address LIKE '%ü%';
|
||||
UPDATE principles SET address = REPLACE(address, 'é', 'é') WHERE address LIKE '%é%';
|
||||
UPDATE principles SET address = REPLACE(address, 'ñ', 'ñ') WHERE address LIKE '%ñ%';
|
||||
|
||||
UPDATE principles SET city = REPLACE(city, 'ü', 'ü') WHERE city LIKE '%ü%';
|
||||
UPDATE principles SET city = REPLACE(city, 'é', 'é') WHERE city LIKE '%é%';
|
||||
|
||||
-- ============================================================================
|
||||
-- PRODUCTS Table (re-apply with all patterns)
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing products table...' as '';
|
||||
|
||||
-- Mojibake
|
||||
UPDATE products SET title = REPLACE(title, '°', '°') WHERE title LIKE '%°%';
|
||||
UPDATE products SET title = REPLACE(title, '®', '®') WHERE title LIKE '%®%';
|
||||
UPDATE products SET title = REPLACE(title, 'â„¢', '™') WHERE title LIKE '%â„¢%';
|
||||
UPDATE products SET title = REPLACE(title, '’', ''') WHERE title LIKE '%’%';
|
||||
UPDATE products SET title = REPLACE(title, '“', '"') WHERE title LIKE '%“%';
|
||||
UPDATE products SET title = REPLACE(title, 'â€', '"') WHERE title LIKE '%â€%';
|
||||
|
||||
UPDATE products SET description = REPLACE(description, '°', '°') WHERE description LIKE '%°%';
|
||||
UPDATE products SET description = REPLACE(description, '®', '®') WHERE description LIKE '%®%';
|
||||
UPDATE products SET description = REPLACE(description, 'â„¢', '™') WHERE description LIKE '%â„¢%';
|
||||
UPDATE products SET description = REPLACE(description, '’', ''') WHERE description LIKE '%’%';
|
||||
|
||||
UPDATE products SET item_description = REPLACE(item_description, '°', '°') WHERE item_description LIKE '%°%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, '®', '®') WHERE item_description LIKE '%®%';
|
||||
|
||||
-- HTML entities
|
||||
UPDATE products SET title = REPLACE(title, '°', '°') WHERE title LIKE '%°%';
|
||||
UPDATE products SET title = REPLACE(title, ' ', ' ') WHERE title LIKE '% %';
|
||||
UPDATE products SET title = REPLACE(title, '&', '&') WHERE title LIKE '%&%';
|
||||
|
||||
UPDATE products SET description = REPLACE(description, '°', '°') WHERE description LIKE '%°%';
|
||||
UPDATE products SET description = REPLACE(description, ' ', ' ') WHERE description LIKE '% %';
|
||||
UPDATE products SET description = REPLACE(description, '&', '&') WHERE description LIKE '%&%';
|
||||
|
||||
-- ============================================================================
|
||||
-- PURCHASE_ORDERS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing purchase_orders table...' as '';
|
||||
|
||||
UPDATE purchase_orders SET ship_via = REPLACE(ship_via, '’', ''') WHERE ship_via LIKE '%’%';
|
||||
UPDATE purchase_orders SET ship_via = REPLACE(ship_via, 'é', 'é') WHERE ship_via LIKE '%é%';
|
||||
|
||||
UPDATE purchase_orders SET comments = REPLACE(comments, '’', ''') WHERE comments LIKE '%’%';
|
||||
UPDATE purchase_orders SET comments = REPLACE(comments, '“', '"') WHERE comments LIKE '%“%';
|
||||
UPDATE purchase_orders SET comments = REPLACE(comments, 'â€', '"') WHERE comments LIKE '%â€%';
|
||||
UPDATE purchase_orders SET comments = REPLACE(comments, 'é', 'é') WHERE comments LIKE '%é%';
|
||||
|
||||
-- ============================================================================
|
||||
-- QUOTES Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing quotes table...' as '';
|
||||
|
||||
UPDATE quotes SET comments = REPLACE(comments, '’', ''') WHERE comments LIKE '%’%';
|
||||
UPDATE quotes SET comments = REPLACE(comments, '“', '"') WHERE comments LIKE '%“%';
|
||||
UPDATE quotes SET comments = REPLACE(comments, 'â€', '"') WHERE comments LIKE '%â€%';
|
||||
UPDATE quotes SET comments = REPLACE(comments, 'é', 'é') WHERE comments LIKE '%é%';
|
||||
|
||||
UPDATE quotes SET notes = REPLACE(notes, '’', ''') WHERE notes LIKE '%’%';
|
||||
UPDATE quotes SET notes = REPLACE(notes, '“', '"') WHERE notes LIKE '%“%';
|
||||
UPDATE quotes SET notes = REPLACE(notes, 'â€', '"') WHERE notes LIKE '%â€%';
|
||||
UPDATE quotes SET notes = REPLACE(notes, 'é', 'é') WHERE notes LIKE '%é%';
|
||||
|
||||
-- ============================================================================
|
||||
-- SHIPMENTS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing shipments table...' as '';
|
||||
|
||||
UPDATE shipments SET ship_via = REPLACE(ship_via, '’', ''') WHERE ship_via LIKE '%’%';
|
||||
UPDATE shipments SET ship_via = REPLACE(ship_via, 'é', 'é') WHERE ship_via LIKE '%é%';
|
||||
|
||||
UPDATE shipments SET comments = REPLACE(comments, '’', ''') WHERE comments LIKE '%’%';
|
||||
UPDATE shipments SET comments = REPLACE(comments, '“', '"') WHERE comments LIKE '%“%';
|
||||
UPDATE shipments SET comments = REPLACE(comments, 'â€', '"') WHERE comments LIKE '%â€%';
|
||||
UPDATE shipments SET comments = REPLACE(comments, 'é', 'é') WHERE comments LIKE '%é%';
|
||||
|
||||
-- ============================================================================
|
||||
-- Verification
|
||||
-- ============================================================================
|
||||
SELECT '' as '';
|
||||
SELECT '============================================================' as '';
|
||||
SELECT 'FIX COMPLETE - Verification' as '';
|
||||
SELECT '============================================================' as '';
|
||||
|
||||
SELECT 'Run scripts/scan_all_tables_for_corruption.sql to see remaining corruption' as '';
|
||||
|
|
@ -1,336 +0,0 @@
|
|||
-- ============================================================================
|
||||
-- Fix Character Corruption in ALL Tables (Using HEX encoding)
|
||||
-- ============================================================================
|
||||
-- This version uses HEX-encoded patterns to avoid encoding issues in the SQL file itself
|
||||
--
|
||||
-- IMPORTANT: Create a backup first!
|
||||
-- To run: docker exec -i cmc-db mariadb -u root -psecureRootPassword --default-character-set=utf8mb4 cmc < scripts/fix_all_corruption_hex.sql
|
||||
-- ============================================================================
|
||||
|
||||
SET NAMES utf8mb4;
|
||||
|
||||
SELECT '============================================================' as '';
|
||||
SELECT 'COMPREHENSIVE CORRUPTION FIX - ALL TABLES (HEX VERSION)' as '';
|
||||
SELECT '============================================================' as '';
|
||||
|
||||
-- ============================================================================
|
||||
-- HEX Pattern Reference:
|
||||
-- E28099 = ’ → U+2019 = ' (smart apostrophe)
|
||||
-- E2809C = “ → U+201C = " (left double quote)
|
||||
-- E2809D = †→ U+201D = " (right double quote)
|
||||
-- E28093 = â€" → U+2013 = – (en dash)
|
||||
-- E28094 = â€" → U+2014 = — (em dash)
|
||||
-- E2809B = ‛ (removed - zero-width space)
|
||||
-- C3A9 = é → U+00E9 = é
|
||||
-- C3BC = ü → U+00FC = ü
|
||||
-- C3B3 = ó → U+00F3 = ó
|
||||
-- C3AD = Ã → U+00ED = í
|
||||
-- C3A7 = ç → U+00E7 = ç
|
||||
-- C3B1 = ñ → U+00F1 = ñ
|
||||
-- C3B6 = ö → U+00F6 = ö
|
||||
-- C39F = ß → U+00DF = ß
|
||||
-- C2B0 = ° → U+00B0 = °
|
||||
-- C2AE = ® → U+00AE = ®
|
||||
-- E284A2 = â„¢ → U+2122 = ™
|
||||
-- ============================================================================
|
||||
|
||||
-- ============================================================================
|
||||
-- ADDRESSES Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing addresses table...' as '';
|
||||
|
||||
UPDATE addresses SET name = REPLACE(name, UNHEX('E28099'), ''') WHERE name LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE addresses SET name = REPLACE(name, UNHEX('E2809C'), '"') WHERE name LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE addresses SET name = REPLACE(name, UNHEX('E2809D'), '"') WHERE name LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE addresses SET name = REPLACE(name, UNHEX('C3A9'), 'é') WHERE name LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE addresses SET name = REPLACE(name, UNHEX('C3BC'), 'ü') WHERE name LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
|
||||
UPDATE addresses SET address = REPLACE(address, UNHEX('E28099'), ''') WHERE address LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE addresses SET address = REPLACE(address, UNHEX('E2809C'), '"') WHERE address LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE addresses SET address = REPLACE(address, UNHEX('E2809D'), '"') WHERE address LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE addresses SET address = REPLACE(address, UNHEX('E28093'), '–') WHERE address LIKE CONCAT('%', UNHEX('E28093'), '%');
|
||||
UPDATE addresses SET address = REPLACE(address, UNHEX('E28094'), '—') WHERE address LIKE CONCAT('%', UNHEX('E28094'), '%');
|
||||
UPDATE addresses SET address = REPLACE(address, UNHEX('C3A9'), 'é') WHERE address LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE addresses SET address = REPLACE(address, UNHEX('C3BC'), 'ü') WHERE address LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
UPDATE addresses SET address = REPLACE(address, UNHEX('C3B3'), 'ó') WHERE address LIKE CONCAT('%', UNHEX('C3B3'), '%');
|
||||
UPDATE addresses SET address = REPLACE(address, UNHEX('C3B1'), 'ñ') WHERE address LIKE CONCAT('%', UNHEX('C3B1'), '%');
|
||||
|
||||
UPDATE addresses SET city = REPLACE(city, UNHEX('C3A9'), 'é') WHERE city LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE addresses SET city = REPLACE(city, UNHEX('C3BC'), 'ü') WHERE city LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
UPDATE addresses SET city = REPLACE(city, UNHEX('C3B3'), 'ó') WHERE city LIKE CONCAT('%', UNHEX('C3B3'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- ATTACHMENTS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing attachments table...' as '';
|
||||
|
||||
UPDATE attachments SET name = REPLACE(name, UNHEX('E28099'), ''') WHERE name LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE attachments SET name = REPLACE(name, UNHEX('C3A9'), 'é') WHERE name LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE attachments SET name = REPLACE(name, UNHEX('C3BC'), 'ü') WHERE name LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
|
||||
UPDATE attachments SET description = REPLACE(description, UNHEX('E28099'), ''') WHERE description LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE attachments SET description = REPLACE(description, UNHEX('E2809C'), '"') WHERE description LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE attachments SET description = REPLACE(description, UNHEX('E2809D'), '"') WHERE description LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE attachments SET description = REPLACE(description, UNHEX('C3A9'), 'é') WHERE description LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE attachments SET description = REPLACE(description, UNHEX('C3BC'), 'ü') WHERE description LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- CONTACTS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing contacts table...' as '';
|
||||
|
||||
UPDATE contacts SET name = REPLACE(name, UNHEX('E28099'), ''') WHERE name LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE contacts SET name = REPLACE(name, UNHEX('C3A9'), 'é') WHERE name LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE contacts SET name = REPLACE(name, UNHEX('C3BC'), 'ü') WHERE name LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
UPDATE contacts SET name = REPLACE(name, UNHEX('C3B3'), 'ó') WHERE name LIKE CONCAT('%', UNHEX('C3B3'), '%');
|
||||
|
||||
UPDATE contacts SET first_name = REPLACE(first_name, UNHEX('E28099'), ''') WHERE first_name LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE contacts SET first_name = REPLACE(first_name, UNHEX('C3A9'), 'é') WHERE first_name LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE contacts SET first_name = REPLACE(first_name, UNHEX('C3BC'), 'ü') WHERE first_name LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
|
||||
UPDATE contacts SET last_name = REPLACE(last_name, UNHEX('E28099'), ''') WHERE last_name LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE contacts SET last_name = REPLACE(last_name, UNHEX('C3A9'), 'é') WHERE last_name LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE contacts SET last_name = REPLACE(last_name, UNHEX('C3BC'), 'ü') WHERE last_name LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
|
||||
UPDATE contacts SET notes = REPLACE(notes, UNHEX('E28099'), ''') WHERE notes LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE contacts SET notes = REPLACE(notes, UNHEX('E2809C'), '"') WHERE notes LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE contacts SET notes = REPLACE(notes, UNHEX('E2809D'), '"') WHERE notes LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE contacts SET notes = REPLACE(notes, UNHEX('C3A9'), 'é') WHERE notes LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE contacts SET notes = REPLACE(notes, UNHEX('C3BC'), 'ü') WHERE notes LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
|
||||
UPDATE contacts SET job_title = REPLACE(job_title, UNHEX('E28099'), ''') WHERE job_title LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE contacts SET job_title = REPLACE(job_title, UNHEX('C3A9'), 'é') WHERE job_title LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- CUSTOMERS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing customers table...' as '';
|
||||
|
||||
UPDATE customers SET name = REPLACE(name, UNHEX('E28099'), ''') WHERE name LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE customers SET name = REPLACE(name, UNHEX('E2809C'), '"') WHERE name LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE customers SET name = REPLACE(name, UNHEX('E2809D'), '"') WHERE name LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE customers SET name = REPLACE(name, UNHEX('E28093'), '–') WHERE name LIKE CONCAT('%', UNHEX('E28093'), '%');
|
||||
UPDATE customers SET name = REPLACE(name, UNHEX('E28094'), '—') WHERE name LIKE CONCAT('%', UNHEX('E28094'), '%');
|
||||
UPDATE customers SET name = REPLACE(name, UNHEX('E2808B'), '') WHERE name LIKE CONCAT('%', UNHEX('E2808B'), '%'); -- zero-width space
|
||||
UPDATE customers SET name = REPLACE(name, UNHEX('C3A9'), 'é') WHERE name LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE customers SET name = REPLACE(name, UNHEX('C3BC'), 'ü') WHERE name LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
UPDATE customers SET name = REPLACE(name, UNHEX('C3B3'), 'ó') WHERE name LIKE CONCAT('%', UNHEX('C3B3'), '%');
|
||||
UPDATE customers SET name = REPLACE(name, UNHEX('C3A7'), 'ç') WHERE name LIKE CONCAT('%', UNHEX('C3A7'), '%');
|
||||
UPDATE customers SET name = REPLACE(name, UNHEX('C3B8'), 'ø') WHERE name LIKE CONCAT('%', UNHEX('C3B8'), '%');
|
||||
|
||||
UPDATE customers SET trading_name = REPLACE(trading_name, UNHEX('E28099'), ''') WHERE trading_name LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE customers SET trading_name = REPLACE(trading_name, UNHEX('C3A9'), 'é') WHERE trading_name LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
UPDATE customers SET notes = REPLACE(notes, UNHEX('E28099'), ''') WHERE notes LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE customers SET notes = REPLACE(notes, UNHEX('E2809C'), '"') WHERE notes LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE customers SET notes = REPLACE(notes, UNHEX('E2809D'), '"') WHERE notes LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE customers SET notes = REPLACE(notes, UNHEX('C3A9'), 'é') WHERE notes LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- DOCUMENTS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing documents table...' as '';
|
||||
|
||||
UPDATE documents SET shipping_details = REPLACE(shipping_details, UNHEX('E28099'), ''') WHERE shipping_details LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE documents SET shipping_details = REPLACE(shipping_details, UNHEX('C3A9'), 'é') WHERE shipping_details LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
UPDATE documents SET bill_to = REPLACE(bill_to, UNHEX('E28099'), ''') WHERE bill_to LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE documents SET bill_to = REPLACE(bill_to, UNHEX('C3A9'), 'é') WHERE bill_to LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
UPDATE documents SET ship_to = REPLACE(ship_to, UNHEX('E28099'), ''') WHERE ship_to LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE documents SET ship_to = REPLACE(ship_to, UNHEX('C3A9'), 'é') WHERE ship_to LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
UPDATE documents SET subject = REPLACE(subject, UNHEX('E28099'), ''') WHERE subject LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE documents SET subject = REPLACE(subject, UNHEX('E2809C'), '"') WHERE subject LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE documents SET subject = REPLACE(subject, UNHEX('E2809D'), '"') WHERE subject LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE documents SET subject = REPLACE(subject, UNHEX('C3A9'), 'é') WHERE subject LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- EMAILS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing emails table...' as '';
|
||||
|
||||
UPDATE emails SET `from` = REPLACE(`from`, UNHEX('E28099'), ''') WHERE `from` LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE emails SET `from` = REPLACE(`from`, UNHEX('C3A9'), 'é') WHERE `from` LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
UPDATE emails SET `to` = REPLACE(`to`, UNHEX('E28099'), ''') WHERE `to` LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE emails SET `to` = REPLACE(`to`, UNHEX('C3A9'), 'é') WHERE `to` LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
UPDATE emails SET subject = REPLACE(subject, UNHEX('E28099'), ''') WHERE subject LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE emails SET subject = REPLACE(subject, UNHEX('E2809C'), '"') WHERE subject LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE emails SET subject = REPLACE(subject, UNHEX('E2809D'), '"') WHERE subject LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE emails SET subject = REPLACE(subject, UNHEX('C3A9'), 'é') WHERE subject LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE emails SET subject = REPLACE(subject, UNHEX('C3BC'), 'ü') WHERE subject LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
|
||||
UPDATE emails SET body = REPLACE(body, UNHEX('E28099'), ''') WHERE body LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE emails SET body = REPLACE(body, UNHEX('E2809C'), '"') WHERE body LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE emails SET body = REPLACE(body, UNHEX('E2809D'), '"') WHERE body LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE emails SET body = REPLACE(body, UNHEX('E28093'), '–') WHERE body LIKE CONCAT('%', UNHEX('E28093'), '%');
|
||||
UPDATE emails SET body = REPLACE(body, UNHEX('E28094'), '—') WHERE body LIKE CONCAT('%', UNHEX('E28094'), '%');
|
||||
UPDATE emails SET body = REPLACE(body, UNHEX('C3A9'), 'é') WHERE body LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE emails SET body = REPLACE(body, UNHEX('C3BC'), 'ü') WHERE body LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
UPDATE emails SET body = REPLACE(body, UNHEX('C2B0'), '°') WHERE body LIKE CONCAT('%', UNHEX('C2B0'), '%');
|
||||
UPDATE emails SET body = REPLACE(body, UNHEX('C2AE'), '®') WHERE body LIKE CONCAT('%', UNHEX('C2AE'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- ENQUIRIES Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing enquiries table...' as '';
|
||||
|
||||
UPDATE enquiries SET comments = REPLACE(comments, UNHEX('E28099'), ''') WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE enquiries SET comments = REPLACE(comments, UNHEX('E2809C'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE enquiries SET comments = REPLACE(comments, UNHEX('E2809D'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE enquiries SET comments = REPLACE(comments, UNHEX('C3A9'), 'é') WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE enquiries SET comments = REPLACE(comments, UNHEX('C3BC'), 'ü') WHERE comments LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- INVOICES Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing invoices table...' as '';
|
||||
|
||||
UPDATE invoices SET ship_via = REPLACE(ship_via, UNHEX('E28099'), ''') WHERE ship_via LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE invoices SET ship_via = REPLACE(ship_via, UNHEX('C3A9'), 'é') WHERE ship_via LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
UPDATE invoices SET comments = REPLACE(comments, UNHEX('E28099'), ''') WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE invoices SET comments = REPLACE(comments, UNHEX('E2809C'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE invoices SET comments = REPLACE(comments, UNHEX('E2809D'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE invoices SET comments = REPLACE(comments, UNHEX('C3A9'), 'é') WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- JOBS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing jobs table...' as '';
|
||||
|
||||
UPDATE jobs SET comments = REPLACE(comments, UNHEX('E28099'), ''') WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE jobs SET comments = REPLACE(comments, UNHEX('E2809C'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE jobs SET comments = REPLACE(comments, UNHEX('E2809D'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE jobs SET comments = REPLACE(comments, UNHEX('C3A9'), 'é') WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- LINE_ITEMS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing line_items table...' as '';
|
||||
|
||||
UPDATE line_items SET name = REPLACE(name, UNHEX('E28099'), ''') WHERE name LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE line_items SET name = REPLACE(name, UNHEX('C3A9'), 'é') WHERE name LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE line_items SET name = REPLACE(name, UNHEX('C2B0'), '°') WHERE name LIKE CONCAT('%', UNHEX('C2B0'), '%');
|
||||
UPDATE line_items SET name = REPLACE(name, UNHEX('C2AE'), '®') WHERE name LIKE CONCAT('%', UNHEX('C2AE'), '%');
|
||||
|
||||
UPDATE line_items SET description = REPLACE(description, UNHEX('E28099'), ''') WHERE description LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE line_items SET description = REPLACE(description, UNHEX('E2809C'), '"') WHERE description LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE line_items SET description = REPLACE(description, UNHEX('E2809D'), '"') WHERE description LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE line_items SET description = REPLACE(description, UNHEX('C3A9'), 'é') WHERE description LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE line_items SET description = REPLACE(description, UNHEX('C2B0'), '°') WHERE description LIKE CONCAT('%', UNHEX('C2B0'), '%');
|
||||
UPDATE line_items SET description = REPLACE(description, UNHEX('C2AE'), '®') WHERE description LIKE CONCAT('%', UNHEX('C2AE'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- ORDER_ACKNOWLEDGEMENTS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing order_acknowledgements table...' as '';
|
||||
|
||||
UPDATE order_acknowledgements SET comments = REPLACE(comments, UNHEX('E28099'), ''') WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE order_acknowledgements SET comments = REPLACE(comments, UNHEX('E2809C'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE order_acknowledgements SET comments = REPLACE(comments, UNHEX('E2809D'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE order_acknowledgements SET comments = REPLACE(comments, UNHEX('C3A9'), 'é') WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- PRINCIPLES Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing principles table...' as '';
|
||||
|
||||
UPDATE principles SET name = REPLACE(name, UNHEX('C3BC'), 'ü') WHERE name LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
UPDATE principles SET name = REPLACE(name, UNHEX('C3A9'), 'é') WHERE name LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE principles SET name = REPLACE(name, UNHEX('C3B3'), 'ó') WHERE name LIKE CONCAT('%', UNHEX('C3B3'), '%');
|
||||
|
||||
UPDATE principles SET address = REPLACE(address, UNHEX('C39F'), 'ß') WHERE address LIKE CONCAT('%', UNHEX('C39F'), '%');
|
||||
UPDATE principles SET address = REPLACE(address, UNHEX('C3BC'), 'ü') WHERE address LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
UPDATE principles SET address = REPLACE(address, UNHEX('C3A9'), 'é') WHERE address LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE principles SET address = REPLACE(address, UNHEX('C3B1'), 'ñ') WHERE address LIKE CONCAT('%', UNHEX('C3B1'), '%');
|
||||
|
||||
UPDATE principles SET city = REPLACE(city, UNHEX('C3BC'), 'ü') WHERE city LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
UPDATE principles SET city = REPLACE(city, UNHEX('C3A9'), 'é') WHERE city LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE principles SET city = REPLACE(city, UNHEX('C3B3'), 'ó') WHERE city LIKE CONCAT('%', UNHEX('C3B3'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- PRODUCTS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing products table...' as '';
|
||||
|
||||
-- Mojibake in products
|
||||
UPDATE products SET title = REPLACE(title, UNHEX('C2B0'), '°') WHERE title LIKE CONCAT('%', UNHEX('C2B0'), '%');
|
||||
UPDATE products SET title = REPLACE(title, UNHEX('C2AE'), '®') WHERE title LIKE CONCAT('%', UNHEX('C2AE'), '%');
|
||||
UPDATE products SET title = REPLACE(title, UNHEX('E284A2'), '™') WHERE title LIKE CONCAT('%', UNHEX('E284A2'), '%');
|
||||
UPDATE products SET title = REPLACE(title, UNHEX('E28099'), ''') WHERE title LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE products SET title = REPLACE(title, UNHEX('E2809C'), '"') WHERE title LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE products SET title = REPLACE(title, UNHEX('E2809D'), '"') WHERE title LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
|
||||
UPDATE products SET description = REPLACE(description, UNHEX('C2B0'), '°') WHERE description LIKE CONCAT('%', UNHEX('C2B0'), '%');
|
||||
UPDATE products SET description = REPLACE(description, UNHEX('C2AE'), '®') WHERE description LIKE CONCAT('%', UNHEX('C2AE'), '%');
|
||||
UPDATE products SET description = REPLACE(description, UNHEX('E284A2'), '™') WHERE description LIKE CONCAT('%', UNHEX('E284A2'), '%');
|
||||
UPDATE products SET description = REPLACE(description, UNHEX('E28099'), ''') WHERE description LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
|
||||
UPDATE products SET item_description = REPLACE(item_description, UNHEX('C2B0'), '°') WHERE item_description LIKE CONCAT('%', UNHEX('C2B0'), '%');
|
||||
UPDATE products SET item_description = REPLACE(item_description, UNHEX('C2AE'), '®') WHERE item_description LIKE CONCAT('%', UNHEX('C2AE'), '%');
|
||||
|
||||
-- HTML entities in products
|
||||
UPDATE products SET title = REPLACE(title, '°', '°') WHERE title LIKE '%°%';
|
||||
UPDATE products SET title = REPLACE(title, ' ', ' ') WHERE title LIKE '% %';
|
||||
UPDATE products SET title = REPLACE(title, '&', '&') WHERE title LIKE '%&%';
|
||||
UPDATE products SET title = REPLACE(title, '–', '–') WHERE title LIKE '%–%';
|
||||
UPDATE products SET title = REPLACE(title, '”', '"') WHERE title LIKE '%”%';
|
||||
UPDATE products SET title = REPLACE(title, '“', '"') WHERE title LIKE '%“%';
|
||||
|
||||
UPDATE products SET description = REPLACE(description, '°', '°') WHERE description LIKE '%°%';
|
||||
UPDATE products SET description = REPLACE(description, ' ', ' ') WHERE description LIKE '% %';
|
||||
UPDATE products SET description = REPLACE(description, '&', '&') WHERE description LIKE '%&%';
|
||||
UPDATE products SET description = REPLACE(description, '½', '½') WHERE description LIKE '%½%';
|
||||
|
||||
-- ============================================================================
|
||||
-- PURCHASE_ORDERS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing purchase_orders table...' as '';
|
||||
|
||||
UPDATE purchase_orders SET ship_via = REPLACE(ship_via, UNHEX('E28099'), ''') WHERE ship_via LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE purchase_orders SET ship_via = REPLACE(ship_via, UNHEX('C3A9'), 'é') WHERE ship_via LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
UPDATE purchase_orders SET comments = REPLACE(comments, UNHEX('E28099'), ''') WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE purchase_orders SET comments = REPLACE(comments, UNHEX('E2809C'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE purchase_orders SET comments = REPLACE(comments, UNHEX('E2809D'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE purchase_orders SET comments = REPLACE(comments, UNHEX('C3A9'), 'é') WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- QUOTES Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing quotes table...' as '';
|
||||
|
||||
UPDATE quotes SET comments = REPLACE(comments, UNHEX('E28099'), ''') WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE quotes SET comments = REPLACE(comments, UNHEX('E2809C'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE quotes SET comments = REPLACE(comments, UNHEX('E2809D'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE quotes SET comments = REPLACE(comments, UNHEX('C3A9'), 'é') WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
UPDATE quotes SET notes = REPLACE(notes, UNHEX('E28099'), ''') WHERE notes LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE quotes SET notes = REPLACE(notes, UNHEX('E2809C'), '"') WHERE notes LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE quotes SET notes = REPLACE(notes, UNHEX('E2809D'), '"') WHERE notes LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE quotes SET notes = REPLACE(notes, UNHEX('C3A9'), 'é') WHERE notes LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- SHIPMENTS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing shipments table...' as '';
|
||||
|
||||
UPDATE shipments SET ship_via = REPLACE(ship_via, UNHEX('E28099'), ''') WHERE ship_via LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE shipments SET ship_via = REPLACE(ship_via, UNHEX('C3A9'), 'é') WHERE ship_via LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
UPDATE shipments SET comments = REPLACE(comments, UNHEX('E28099'), ''') WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE shipments SET comments = REPLACE(comments, UNHEX('E2809C'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE shipments SET comments = REPLACE(comments, UNHEX('E2809D'), '"') WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE shipments SET comments = REPLACE(comments, UNHEX('C3A9'), 'é') WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- Verification
|
||||
-- ============================================================================
|
||||
SELECT '' as '';
|
||||
SELECT '============================================================' as '';
|
||||
SELECT 'FIX COMPLETE' as '';
|
||||
SELECT '============================================================' as '';
|
||||
SELECT 'Run scan_all_tables_for_corruption.sql to verify' as '';
|
||||
|
|
@ -1,336 +0,0 @@
|
|||
-- ============================================================================
|
||||
-- Fix Character Corruption in ALL Tables (Safe Version with CHAR())
|
||||
-- ============================================================================
|
||||
-- This version uses CHAR() with Unicode code points to avoid any encoding issues
|
||||
--
|
||||
-- IMPORTANT: Create a backup first!
|
||||
-- To run: docker exec -i cmc-db mariadb -u root -psecureRootPassword --default-character-set=utf8mb4 cmc < scripts/fix_all_corruption_safe.sql
|
||||
-- ============================================================================
|
||||
|
||||
SET NAMES utf8mb4;
|
||||
|
||||
SELECT '============================================================' as '';
|
||||
SELECT 'COMPREHENSIVE CORRUPTION FIX - ALL TABLES (SAFE VERSION)' as '';
|
||||
SELECT '============================================================' as '';
|
||||
|
||||
-- ============================================================================
|
||||
-- Pattern Reference:
|
||||
-- UNHEX('E28099') = ’ → CHAR(8217 USING utf8mb4) = '
|
||||
-- UNHEX('E2809C') = “ → CHAR(8220 USING utf8mb4) = "
|
||||
-- UNHEX('E2809D') = †→ CHAR(8221 USING utf8mb4) = "
|
||||
-- UNHEX('E28093') = â€" → CHAR(8211 USING utf8mb4) = –
|
||||
-- UNHEX('E28094') = â€" → CHAR(8212 USING utf8mb4) = —
|
||||
-- UNHEX('E2808B') = zero-width space → (remove)
|
||||
-- UNHEX('C3A9') = é → CHAR(0x00E9 USING utf8mb4) = é
|
||||
-- UNHEX('C3BC') = ü → CHAR(0x00FC USING utf8mb4) = ü
|
||||
-- UNHEX('C3B3') = ó → CHAR(0x00F3 USING utf8mb4) = ó
|
||||
-- UNHEX('C3B1') = ñ → CHAR(0x00F1 USING utf8mb4) = ñ
|
||||
-- UNHEX('C3A7') = ç → CHAR(0x00E7 USING utf8mb4) = ç
|
||||
-- UNHEX('C3B6') = ö → CHAR(0x00F6 USING utf8mb4) = ö
|
||||
-- UNHEX('C3B8') = ø → CHAR(0x00F8 USING utf8mb4) = ø
|
||||
-- UNHEX('C39F') = ß → CHAR(0x00DF USING utf8mb4) = ß
|
||||
-- UNHEX('C2B0') = ° → CHAR(0x00B0 USING utf8mb4) = °
|
||||
-- UNHEX('C2AE') = ® → CHAR(0x00AE USING utf8mb4) = ®
|
||||
-- UNHEX('E284A2') = â„¢ → CHAR(8482 USING utf8mb4) = ™
|
||||
-- ============================================================================
|
||||
|
||||
-- ============================================================================
|
||||
-- ADDRESSES Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing addresses table...' as '';
|
||||
|
||||
UPDATE addresses SET name = REPLACE(name, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE addresses SET name = REPLACE(name, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE addresses SET name = REPLACE(name, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE addresses SET name = REPLACE(name, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE addresses SET name = REPLACE(name, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
|
||||
UPDATE addresses SET address = REPLACE(address, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE address LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE addresses SET address = REPLACE(address, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE address LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE addresses SET address = REPLACE(address, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE address LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE addresses SET address = REPLACE(address, UNHEX('E28093'), CHAR(8211 USING utf8mb4)) WHERE address LIKE CONCAT('%', UNHEX('E28093'), '%');
|
||||
UPDATE addresses SET address = REPLACE(address, UNHEX('E28094'), CHAR(8212 USING utf8mb4)) WHERE address LIKE CONCAT('%', UNHEX('E28094'), '%');
|
||||
UPDATE addresses SET address = REPLACE(address, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE address LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE addresses SET address = REPLACE(address, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE address LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
UPDATE addresses SET address = REPLACE(address, UNHEX('C3B3'), CHAR(0x00F3 USING utf8mb4)) WHERE address LIKE CONCAT('%', UNHEX('C3B3'), '%');
|
||||
UPDATE addresses SET address = REPLACE(address, UNHEX('C3B1'), CHAR(0x00F1 USING utf8mb4)) WHERE address LIKE CONCAT('%', UNHEX('C3B1'), '%');
|
||||
|
||||
UPDATE addresses SET city = REPLACE(city, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE city LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE addresses SET city = REPLACE(city, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE city LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
UPDATE addresses SET city = REPLACE(city, UNHEX('C3B3'), CHAR(0x00F3 USING utf8mb4)) WHERE city LIKE CONCAT('%', UNHEX('C3B3'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- ATTACHMENTS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing attachments table...' as '';
|
||||
|
||||
UPDATE attachments SET name = REPLACE(name, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE attachments SET name = REPLACE(name, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE attachments SET name = REPLACE(name, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
|
||||
UPDATE attachments SET description = REPLACE(description, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE attachments SET description = REPLACE(description, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE attachments SET description = REPLACE(description, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE attachments SET description = REPLACE(description, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE attachments SET description = REPLACE(description, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- CONTACTS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing contacts table...' as '';
|
||||
|
||||
UPDATE contacts SET name = REPLACE(name, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE contacts SET name = REPLACE(name, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE contacts SET name = REPLACE(name, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
UPDATE contacts SET name = REPLACE(name, UNHEX('C3B3'), CHAR(0x00F3 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3B3'), '%');
|
||||
|
||||
UPDATE contacts SET first_name = REPLACE(first_name, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE first_name LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE contacts SET first_name = REPLACE(first_name, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE first_name LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE contacts SET first_name = REPLACE(first_name, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE first_name LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
|
||||
UPDATE contacts SET last_name = REPLACE(last_name, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE last_name LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE contacts SET last_name = REPLACE(last_name, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE last_name LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE contacts SET last_name = REPLACE(last_name, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE last_name LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
|
||||
UPDATE contacts SET notes = REPLACE(notes, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE notes LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE contacts SET notes = REPLACE(notes, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE notes LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE contacts SET notes = REPLACE(notes, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE notes LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE contacts SET notes = REPLACE(notes, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE notes LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE contacts SET notes = REPLACE(notes, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE notes LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
|
||||
UPDATE contacts SET job_title = REPLACE(job_title, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE job_title LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE contacts SET job_title = REPLACE(job_title, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE job_title LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- CUSTOMERS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing customers table...' as '';
|
||||
|
||||
UPDATE customers SET name = REPLACE(name, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE customers SET name = REPLACE(name, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE customers SET name = REPLACE(name, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE customers SET name = REPLACE(name, UNHEX('E28093'), CHAR(8211 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('E28093'), '%');
|
||||
UPDATE customers SET name = REPLACE(name, UNHEX('E28094'), CHAR(8212 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('E28094'), '%');
|
||||
UPDATE customers SET name = REPLACE(name, UNHEX('E2808B'), '') WHERE name LIKE CONCAT('%', UNHEX('E2808B'), '%');
|
||||
UPDATE customers SET name = REPLACE(name, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE customers SET name = REPLACE(name, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
UPDATE customers SET name = REPLACE(name, UNHEX('C3B3'), CHAR(0x00F3 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3B3'), '%');
|
||||
UPDATE customers SET name = REPLACE(name, UNHEX('C3A7'), CHAR(0x00E7 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3A7'), '%');
|
||||
UPDATE customers SET name = REPLACE(name, UNHEX('C3B8'), CHAR(0x00F8 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3B8'), '%');
|
||||
|
||||
UPDATE customers SET trading_name = REPLACE(trading_name, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE trading_name LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE customers SET trading_name = REPLACE(trading_name, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE trading_name LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
UPDATE customers SET notes = REPLACE(notes, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE notes LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE customers SET notes = REPLACE(notes, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE notes LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE customers SET notes = REPLACE(notes, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE notes LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE customers SET notes = REPLACE(notes, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE notes LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- DOCUMENTS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing documents table...' as '';
|
||||
|
||||
UPDATE documents SET shipping_details = REPLACE(shipping_details, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE shipping_details LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE documents SET shipping_details = REPLACE(shipping_details, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE shipping_details LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
UPDATE documents SET bill_to = REPLACE(bill_to, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE bill_to LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE documents SET bill_to = REPLACE(bill_to, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE bill_to LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
UPDATE documents SET ship_to = REPLACE(ship_to, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE ship_to LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE documents SET ship_to = REPLACE(ship_to, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE ship_to LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
UPDATE documents SET subject = REPLACE(subject, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE subject LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE documents SET subject = REPLACE(subject, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE subject LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE documents SET subject = REPLACE(subject, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE subject LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE documents SET subject = REPLACE(subject, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE subject LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- EMAILS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing emails table...' as '';
|
||||
|
||||
UPDATE emails SET `from` = REPLACE(`from`, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE `from` LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE emails SET `from` = REPLACE(`from`, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE `from` LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
UPDATE emails SET `to` = REPLACE(`to`, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE `to` LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE emails SET `to` = REPLACE(`to`, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE `to` LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
UPDATE emails SET subject = REPLACE(subject, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE subject LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE emails SET subject = REPLACE(subject, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE subject LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE emails SET subject = REPLACE(subject, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE subject LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE emails SET subject = REPLACE(subject, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE subject LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE emails SET subject = REPLACE(subject, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE subject LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
|
||||
UPDATE emails SET body = REPLACE(body, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE body LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE emails SET body = REPLACE(body, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE body LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE emails SET body = REPLACE(body, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE body LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE emails SET body = REPLACE(body, UNHEX('E28093'), CHAR(8211 USING utf8mb4)) WHERE body LIKE CONCAT('%', UNHEX('E28093'), '%');
|
||||
UPDATE emails SET body = REPLACE(body, UNHEX('E28094'), CHAR(8212 USING utf8mb4)) WHERE body LIKE CONCAT('%', UNHEX('E28094'), '%');
|
||||
UPDATE emails SET body = REPLACE(body, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE body LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE emails SET body = REPLACE(body, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE body LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
UPDATE emails SET body = REPLACE(body, UNHEX('C2B0'), CHAR(0x00B0 USING utf8mb4)) WHERE body LIKE CONCAT('%', UNHEX('C2B0'), '%');
|
||||
UPDATE emails SET body = REPLACE(body, UNHEX('C2AE'), CHAR(0x00AE USING utf8mb4)) WHERE body LIKE CONCAT('%', UNHEX('C2AE'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- ENQUIRIES Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing enquiries table...' as '';
|
||||
|
||||
UPDATE enquiries SET comments = REPLACE(comments, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE enquiries SET comments = REPLACE(comments, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE enquiries SET comments = REPLACE(comments, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE enquiries SET comments = REPLACE(comments, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE enquiries SET comments = REPLACE(comments, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- INVOICES Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing invoices table...' as '';
|
||||
|
||||
UPDATE invoices SET ship_via = REPLACE(ship_via, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE ship_via LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE invoices SET ship_via = REPLACE(ship_via, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE ship_via LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
UPDATE invoices SET comments = REPLACE(comments, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE invoices SET comments = REPLACE(comments, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE invoices SET comments = REPLACE(comments, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE invoices SET comments = REPLACE(comments, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- JOBS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing jobs table...' as '';
|
||||
|
||||
UPDATE jobs SET comments = REPLACE(comments, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE jobs SET comments = REPLACE(comments, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE jobs SET comments = REPLACE(comments, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE jobs SET comments = REPLACE(comments, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- LINE_ITEMS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing line_items table...' as '';
|
||||
|
||||
UPDATE line_items SET name = REPLACE(name, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE line_items SET name = REPLACE(name, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE line_items SET name = REPLACE(name, UNHEX('C2B0'), CHAR(0x00B0 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C2B0'), '%');
|
||||
UPDATE line_items SET name = REPLACE(name, UNHEX('C2AE'), CHAR(0x00AE USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C2AE'), '%');
|
||||
|
||||
UPDATE line_items SET description = REPLACE(description, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE line_items SET description = REPLACE(description, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE line_items SET description = REPLACE(description, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE line_items SET description = REPLACE(description, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE line_items SET description = REPLACE(description, UNHEX('C2B0'), CHAR(0x00B0 USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('C2B0'), '%');
|
||||
UPDATE line_items SET description = REPLACE(description, UNHEX('C2AE'), CHAR(0x00AE USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('C2AE'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- ORDER_ACKNOWLEDGEMENTS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing order_acknowledgements table...' as '';
|
||||
|
||||
UPDATE order_acknowledgements SET comments = REPLACE(comments, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE order_acknowledgements SET comments = REPLACE(comments, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE order_acknowledgements SET comments = REPLACE(comments, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE order_acknowledgements SET comments = REPLACE(comments, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- PRINCIPLES Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing principles table...' as '';
|
||||
|
||||
UPDATE principles SET name = REPLACE(name, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
UPDATE principles SET name = REPLACE(name, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE principles SET name = REPLACE(name, UNHEX('C3B3'), CHAR(0x00F3 USING utf8mb4)) WHERE name LIKE CONCAT('%', UNHEX('C3B3'), '%');
|
||||
|
||||
UPDATE principles SET address = REPLACE(address, UNHEX('C39F'), CHAR(0x00DF USING utf8mb4)) WHERE address LIKE CONCAT('%', UNHEX('C39F'), '%');
|
||||
UPDATE principles SET address = REPLACE(address, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE address LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
UPDATE principles SET address = REPLACE(address, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE address LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE principles SET address = REPLACE(address, UNHEX('C3B1'), CHAR(0x00F1 USING utf8mb4)) WHERE address LIKE CONCAT('%', UNHEX('C3B1'), '%');
|
||||
|
||||
UPDATE principles SET city = REPLACE(city, UNHEX('C3BC'), CHAR(0x00FC USING utf8mb4)) WHERE city LIKE CONCAT('%', UNHEX('C3BC'), '%');
|
||||
UPDATE principles SET city = REPLACE(city, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE city LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
UPDATE principles SET city = REPLACE(city, UNHEX('C3B3'), CHAR(0x00F3 USING utf8mb4)) WHERE city LIKE CONCAT('%', UNHEX('C3B3'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- PRODUCTS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing products table...' as '';
|
||||
|
||||
-- Mojibake in products
|
||||
UPDATE products SET title = REPLACE(title, UNHEX('C2B0'), CHAR(0x00B0 USING utf8mb4)) WHERE title LIKE CONCAT('%', UNHEX('C2B0'), '%');
|
||||
UPDATE products SET title = REPLACE(title, UNHEX('C2AE'), CHAR(0x00AE USING utf8mb4)) WHERE title LIKE CONCAT('%', UNHEX('C2AE'), '%');
|
||||
UPDATE products SET title = REPLACE(title, UNHEX('E284A2'), CHAR(8482 USING utf8mb4)) WHERE title LIKE CONCAT('%', UNHEX('E284A2'), '%');
|
||||
UPDATE products SET title = REPLACE(title, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE title LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE products SET title = REPLACE(title, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE title LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE products SET title = REPLACE(title, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE title LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
|
||||
UPDATE products SET description = REPLACE(description, UNHEX('C2B0'), CHAR(0x00B0 USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('C2B0'), '%');
|
||||
UPDATE products SET description = REPLACE(description, UNHEX('C2AE'), CHAR(0x00AE USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('C2AE'), '%');
|
||||
UPDATE products SET description = REPLACE(description, UNHEX('E284A2'), CHAR(8482 USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('E284A2'), '%');
|
||||
UPDATE products SET description = REPLACE(description, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE description LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
|
||||
UPDATE products SET item_description = REPLACE(item_description, UNHEX('C2B0'), CHAR(0x00B0 USING utf8mb4)) WHERE item_description LIKE CONCAT('%', UNHEX('C2B0'), '%');
|
||||
UPDATE products SET item_description = REPLACE(item_description, UNHEX('C2AE'), CHAR(0x00AE USING utf8mb4)) WHERE item_description LIKE CONCAT('%', UNHEX('C2AE'), '%');
|
||||
|
||||
-- HTML entities in products (these are plain ASCII so no encoding issue)
|
||||
UPDATE products SET title = REPLACE(title, '°', CHAR(0x00B0 USING utf8mb4)) WHERE title LIKE '%°%';
|
||||
UPDATE products SET title = REPLACE(title, ' ', ' ') WHERE title LIKE '% %';
|
||||
UPDATE products SET title = REPLACE(title, '&', '&') WHERE title LIKE '%&%';
|
||||
UPDATE products SET title = REPLACE(title, '–', CHAR(8211 USING utf8mb4)) WHERE title LIKE '%–%';
|
||||
UPDATE products SET title = REPLACE(title, '”', CHAR(8221 USING utf8mb4)) WHERE title LIKE '%”%';
|
||||
UPDATE products SET title = REPLACE(title, '“', CHAR(8220 USING utf8mb4)) WHERE title LIKE '%“%';
|
||||
|
||||
UPDATE products SET description = REPLACE(description, '°', CHAR(0x00B0 USING utf8mb4)) WHERE description LIKE '%°%';
|
||||
UPDATE products SET description = REPLACE(description, ' ', ' ') WHERE description LIKE '% %';
|
||||
UPDATE products SET description = REPLACE(description, '&', '&') WHERE description LIKE '%&%';
|
||||
UPDATE products SET description = REPLACE(description, '½', CHAR(0x00BD USING utf8mb4)) WHERE description LIKE '%½%';
|
||||
|
||||
-- ============================================================================
|
||||
-- PURCHASE_ORDERS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing purchase_orders table...' as '';
|
||||
|
||||
UPDATE purchase_orders SET ship_via = REPLACE(ship_via, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE ship_via LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE purchase_orders SET ship_via = REPLACE(ship_via, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE ship_via LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
UPDATE purchase_orders SET comments = REPLACE(comments, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE purchase_orders SET comments = REPLACE(comments, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE purchase_orders SET comments = REPLACE(comments, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE purchase_orders SET comments = REPLACE(comments, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- QUOTES Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing quotes table...' as '';
|
||||
|
||||
UPDATE quotes SET comments = REPLACE(comments, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE quotes SET comments = REPLACE(comments, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE quotes SET comments = REPLACE(comments, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE quotes SET comments = REPLACE(comments, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
UPDATE quotes SET notes = REPLACE(notes, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE notes LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE quotes SET notes = REPLACE(notes, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE notes LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE quotes SET notes = REPLACE(notes, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE notes LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE quotes SET notes = REPLACE(notes, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE notes LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- SHIPMENTS Table
|
||||
-- ============================================================================
|
||||
SELECT 'Fixing shipments table...' as '';
|
||||
|
||||
UPDATE shipments SET ship_via = REPLACE(ship_via, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE ship_via LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE shipments SET ship_via = REPLACE(ship_via, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE ship_via LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
UPDATE shipments SET comments = REPLACE(comments, UNHEX('E28099'), CHAR(8217 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
UPDATE shipments SET comments = REPLACE(comments, UNHEX('E2809C'), CHAR(8220 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809C'), '%');
|
||||
UPDATE shipments SET comments = REPLACE(comments, UNHEX('E2809D'), CHAR(8221 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('E2809D'), '%');
|
||||
UPDATE shipments SET comments = REPLACE(comments, UNHEX('C3A9'), CHAR(0x00E9 USING utf8mb4)) WHERE comments LIKE CONCAT('%', UNHEX('C3A9'), '%');
|
||||
|
||||
-- ============================================================================
|
||||
-- Verification
|
||||
-- ============================================================================
|
||||
SELECT '' as '';
|
||||
SELECT '============================================================' as '';
|
||||
SELECT 'FIX COMPLETE' as '';
|
||||
SELECT '============================================================' as '';
|
||||
SELECT 'Run scan_all_tables_for_corruption.sql to verify' as '';
|
||||
|
|
@ -1,222 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
Fix Corrupted Character Data in CMC Database
|
||||
Uses Python to properly handle UTF-8 encoding
|
||||
"""
|
||||
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
DB_USER = "root"
|
||||
DB_PASS = "secureRootPassword"
|
||||
DB_NAME = "cmc"
|
||||
CONTAINER = "cmc-db"
|
||||
|
||||
# Mapping of corrupted patterns to correct values
|
||||
CORRUPTION_PATTERNS = {
|
||||
# Smart quotes and punctuation
|
||||
'\u00e2\u0080\u0099': '\u2019', # ’ → '
|
||||
'\u00e2\u0080\u009c': '\u201c', # “ → "
|
||||
'\u00e2\u0080\u009d': '\u201d', # †→ "
|
||||
'\u00e2\u0080\u0093': '\u2013', # â€" → –
|
||||
'\u00e2\u0080\u0094': '\u2014', # â€" → —
|
||||
'\u00e2\u0080\u008b': '', # ​ → (zero-width space, remove)
|
||||
'\u00e2\u0080\u0083': ' ', #   → (em space, replace with regular space)
|
||||
|
||||
# Accented characters
|
||||
'\u00c3\u00a9': '\u00e9', # é → é
|
||||
'\u00c3\u0089': '\u00c9', # É → É
|
||||
'\u00c3\u00b3': '\u00f3', # ó → ó
|
||||
'\u00c3\u00ad': '\u00ed', # Ã → í
|
||||
'\u00c3\u00a7': '\u00e7', # ç → ç
|
||||
'\u00c3\u00bc': '\u00fc', # ü → ü
|
||||
'\u00c3\u00a1': '\u00e1', # á → á
|
||||
'\u00c3\u00b1': '\u00f1', # ñ → ñ
|
||||
'\u00c3\u00b6': '\u00f6', # ö → ö
|
||||
'\u00c3\u00b4': '\u00f4', # ô → ô
|
||||
'\u00c3\u009f': '\u00df', # ß → ß (German sharp s)
|
||||
|
||||
# Turkish characters
|
||||
'\u00c4\u00b0': '\u0130', # Ä° → İ
|
||||
'\u00c5\u009e': '\u015e', # Å → Ş
|
||||
'\u00c4\u009e': '\u011e', # Ä → Ğ
|
||||
|
||||
# Czech characters
|
||||
'\u00c4\u008c': '\u010c', # Ä → Č
|
||||
'\u00c4\u009b': '\u011b', # Ä› → ě
|
||||
'\u00c4\u008d': '\u010d', # Ä → č
|
||||
'\u00c5\u00be': '\u017e', # ž → ž
|
||||
'\u00c5\u00a1': '\u0161', # Å¡ → š
|
||||
'\u00c5\u0099': '\u0159', # Å™ → ř
|
||||
'\u00c5\u0088': '\u0148', # Å → ň
|
||||
|
||||
# Symbols
|
||||
'\u00c2\u00b0': '\u00b0', # ° → ° (degree)
|
||||
'\u00c2\u00ae': '\u00ae', # ® → ® (registered)
|
||||
'\u00e2\u0084\u00a2': '\u2122', # â„¢ → ™ (trademark)
|
||||
}
|
||||
|
||||
def run_sql(sql):
|
||||
"""Execute SQL command via docker"""
|
||||
cmd = [
|
||||
'docker', 'exec', CONTAINER,
|
||||
'mariadb',
|
||||
'-u', DB_USER,
|
||||
f'-p{DB_PASS}',
|
||||
'--default-character-set=utf8mb4',
|
||||
DB_NAME,
|
||||
'-e', sql
|
||||
]
|
||||
|
||||
try:
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, encoding='utf-8')
|
||||
if result.returncode != 0:
|
||||
print(f"Error executing SQL: {result.stderr}", file=sys.stderr)
|
||||
return False
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"Exception: {e}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
def fix_table_column(table, column, patterns):
|
||||
"""Fix corruption patterns in a specific table column"""
|
||||
print(f"Fixing {table}.{column}...")
|
||||
|
||||
for corrupted, correct in patterns.items():
|
||||
# Escape single quotes for SQL
|
||||
corrupted_esc = corrupted.replace("'", "''")
|
||||
correct_esc = correct.replace("'", "''")
|
||||
|
||||
sql = f"""
|
||||
UPDATE {table}
|
||||
SET {column} = REPLACE({column}, '{corrupted_esc}', '{correct_esc}')
|
||||
WHERE {column} LIKE CONCAT('%', '{corrupted_esc}', '%');
|
||||
"""
|
||||
|
||||
if not run_sql(sql):
|
||||
print(f" Warning: Failed to fix pattern in {table}.{column}")
|
||||
|
||||
print(f" ✓ {table}.{column} patterns fixed")
|
||||
|
||||
def fix_html_entities():
|
||||
"""Fix HTML entities that shouldn't be in the database"""
|
||||
print("\nFixing HTML entities in products table...")
|
||||
|
||||
entities = {
|
||||
'&': '&',
|
||||
' ': ' ',
|
||||
'–': '–',
|
||||
'—': '—',
|
||||
'”': '"',
|
||||
'“': '"',
|
||||
'’': ''',
|
||||
'‘': ''',
|
||||
'°': '°',
|
||||
'½': '½',
|
||||
'¼': '¼',
|
||||
'¾': '¾',
|
||||
'×': '×',
|
||||
}
|
||||
|
||||
for table_col in [('products', 'title'), ('products', 'description'), ('products', 'item_description')]:
|
||||
table, column = table_col
|
||||
for entity, char in entities.items():
|
||||
sql = f"""
|
||||
UPDATE {table}
|
||||
SET {column} = REPLACE({column}, '{entity}', '{char}')
|
||||
WHERE {column} LIKE '%{entity}%';
|
||||
"""
|
||||
run_sql(sql)
|
||||
|
||||
print(" ✓ HTML entities fixed")
|
||||
|
||||
def fix_specific_records():
|
||||
"""Fix specific known corrupted records"""
|
||||
print("\nFixing specific known records...")
|
||||
|
||||
fixes = [
|
||||
("customers", 253, "name", "SOLUZÉ CIVIL ENGINEERS Trading Under SOLUZE PTY LTD"),
|
||||
("customers", 1006, "name", "Dr. Prem's Molecules Private Limited (DPMolecules)"),
|
||||
("customers", 1387, "name", 'DEE ENTERPRISES (QLD) PTY LTD trading as "MEKLEK"'),
|
||||
("customers", 1608, "name", "Guidera O'Connor Pty Ltd"),
|
||||
("customers", 2174, "name", "Ingredion ANZ Pty Ltd"),
|
||||
("customers", 2215, "name", "Vale Nouvelle-Calédonie S.A.S"),
|
||||
("customers", 2375, "name", "Evaluación y Control Ambiental S.A.S."),
|
||||
("customers", 3143, "name", "Zontahevy Comércio e Serviços Offshore Ltda"),
|
||||
("customers", 3529, "name", "Société des Mines de Syama"),
|
||||
("customers", 4325, "name", "Rambøll Danmark"),
|
||||
("customers", 4350, "name", "SONNIVA ENERGY MAKİNA İNŞAAT İTH. İHR. LTD. ŞTİ."),
|
||||
("customers", 4669, "name", "F.C.C. Fluides Conseils Calédonie SARL"),
|
||||
("customers", 4743, "name", "DGPack Prostějov"),
|
||||
("customers", 4893, "name", "Oxiquímica, S.A.P.I. de C.V."),
|
||||
|
||||
("principles", 2, "address", "Heinz-Fangman-Straße 18"),
|
||||
("principles", 9, "address", "Bezručova 2901\\n756 61 Rožnov pod Radhoštěm"),
|
||||
("principles", 9, "city", "Rožnov pod Radhoštěm"),
|
||||
("principles", 13, "name", "IEP Technologies GmbH - BRILEX Gesellschaft für Explosionsschutz mbH"),
|
||||
("principles", 14, "address", "Liebigstraße 2"),
|
||||
("principles", 58, "name", "Nosia S.r.l.- Señalización Industrial"),
|
||||
("principles", 65, "address", "Alte Emser Straße 32"),
|
||||
|
||||
("addresses", 19, "address", "Lvl 3, Building B, 7–11 Talavera Road"),
|
||||
|
||||
("products", 30, "title", "C95SN189C DSTGL40/C-Digital temperature probe for P8xx1 Web Sensor. Range -30 to +80°C. With CINCH connector, 1m Cable"),
|
||||
("products", 38, "title", "Mid-West Instrument Model 240 – SC – 02 – O(TT)"),
|
||||
("products", 67, "title", "Newson Gale Earth-Rite® II FIBC Static Earthing System, GRP Enclosure with 5m 2 Core Spiral Cable and FIBC Clamp"),
|
||||
("products", 76, "title", "Newson Gale Earth-Rite® RTR Tester - ER2/CRT"),
|
||||
("products", 85, "title", "Newson Gale Earth-Rite® RTR™ Tri-Mode Static Grounding System, Metal Enclosure, X90-IP Heavy Duty Clamp with 10m 2 Core Spiral Cable"),
|
||||
]
|
||||
|
||||
for table, record_id, column, value in fixes:
|
||||
value_esc = value.replace("'", "''")
|
||||
sql = f"UPDATE {table} SET {column} = '{value_esc}' WHERE id = {record_id};"
|
||||
run_sql(sql)
|
||||
|
||||
print(f" ✓ {len(fixes)} specific records fixed")
|
||||
|
||||
def verify():
|
||||
"""Verify the fixes"""
|
||||
print("\n" + "="*60)
|
||||
print("Verification Results")
|
||||
print("="*60)
|
||||
|
||||
run_sql("SELECT 'Remaining corrupted customers:', COUNT(*) FROM customers WHERE name REGEXP '[^ -~]';")
|
||||
run_sql("SELECT 'Remaining corrupted principles:', COUNT(*) FROM principles WHERE name REGEXP '[^ -~]' OR address REGEXP '[^ -~]';")
|
||||
run_sql("SELECT 'Remaining corrupted addresses:', COUNT(*) FROM addresses WHERE address REGEXP '[^ -~]';")
|
||||
run_sql("SELECT 'Remaining corrupted products:', COUNT(*) FROM products WHERE title REGEXP '[^ -~]' OR description REGEXP '[^ -~]';")
|
||||
|
||||
def main():
|
||||
print("="*60)
|
||||
print("CMC Database Character Corruption Fix")
|
||||
print("="*60)
|
||||
print("")
|
||||
|
||||
# Set charset
|
||||
run_sql("SET NAMES utf8mb4;")
|
||||
|
||||
# Fix pattern-based corruption
|
||||
fix_table_column("customers", "name", CORRUPTION_PATTERNS)
|
||||
fix_table_column("principles", "name", CORRUPTION_PATTERNS)
|
||||
fix_table_column("principles", "address", CORRUPTION_PATTERNS)
|
||||
fix_table_column("principles", "city", CORRUPTION_PATTERNS)
|
||||
fix_table_column("addresses", "address", CORRUPTION_PATTERNS)
|
||||
fix_table_column("addresses", "city", CORRUPTION_PATTERNS)
|
||||
fix_table_column("products", "title", CORRUPTION_PATTERNS)
|
||||
fix_table_column("products", "description", CORRUPTION_PATTERNS)
|
||||
fix_table_column("products", "item_description", CORRUPTION_PATTERNS)
|
||||
|
||||
# Fix HTML entities
|
||||
fix_html_entities()
|
||||
|
||||
# Fix specific records
|
||||
fix_specific_records()
|
||||
|
||||
# Verify
|
||||
verify()
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("Fix complete!")
|
||||
print("="*60)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -1,317 +0,0 @@
|
|||
-- ============================================================================
|
||||
-- Fix Corrupted Character Data in CMC Database
|
||||
-- ============================================================================
|
||||
-- This script fixes mojibake (corrupted UTF-8 characters) in the database
|
||||
-- Run after converting tables to utf8mb4
|
||||
--
|
||||
-- IMPORTANT: Review this script and test on a backup first!
|
||||
-- Create backup: docker exec cmc-db mariadb-dump -u root -psecureRootPassword cmc | gzip > backup_before_fix.sql.gz
|
||||
--
|
||||
-- To run: docker exec -i cmc-db mariadb -u root -psecureRootPassword --default-character-set=utf8mb4 cmc < scripts/fix_corrupted_data.sql
|
||||
-- ============================================================================
|
||||
|
||||
-- Create a backup table for principles (optional safety measure)
|
||||
-- DROP TABLE IF EXISTS principles_backup;
|
||||
-- CREATE TABLE principles_backup AS SELECT * FROM principles;
|
||||
|
||||
SET NAMES utf8mb4;
|
||||
|
||||
-- ============================================================================
|
||||
-- PART 1: Global Pattern Fixes Using REPLACE
|
||||
-- ============================================================================
|
||||
-- These fix common mojibake patterns across all affected tables
|
||||
|
||||
-- ----------------------------------------------------------------------------
|
||||
-- Fix CUSTOMERS table
|
||||
-- ----------------------------------------------------------------------------
|
||||
UPDATE customers SET name = REPLACE(name, '’', ''') WHERE name LIKE '%’%'; -- Smart apostrophe
|
||||
UPDATE customers SET name = REPLACE(name, '“', '"') WHERE name LIKE '%“%'; -- Left double quote
|
||||
UPDATE customers SET name = REPLACE(name, 'â€', '"') WHERE name LIKE '%â€%'; -- Right double quote
|
||||
UPDATE customers SET name = REPLACE(name, 'â€"', '–') WHERE name LIKE '%â€"%'; -- En dash
|
||||
UPDATE customers SET name = REPLACE(name, 'â€"', '—') WHERE name LIKE '%â€"%'; -- Em dash
|
||||
UPDATE customers SET name = REPLACE(name, '​', '') WHERE name LIKE '%​%'; -- Zero-width space (remove)
|
||||
UPDATE customers SET name = REPLACE(name, ' ', ' ') WHERE name LIKE '% %'; -- Special space
|
||||
UPDATE customers SET name = REPLACE(name, 'é', 'é') WHERE name LIKE '%é%'; -- é
|
||||
UPDATE customers SET name = REPLACE(name, 'É', 'É') WHERE name LIKE '%É%'; -- É
|
||||
UPDATE customers SET name = REPLACE(name, 'ó', 'ó') WHERE name LIKE '%ó%'; -- ó
|
||||
UPDATE customers SET name = REPLACE(name, 'Ã', 'í') WHERE name LIKE '%Ã%'; -- í
|
||||
UPDATE customers SET name = REPLACE(name, 'ç', 'ç') WHERE name LIKE '%ç%'; -- ç
|
||||
UPDATE customers SET name = REPLACE(name, 'ü', 'ü') WHERE name LIKE '%ü%'; -- ü
|
||||
UPDATE customers SET name = REPLACE(name, 'á', 'á') WHERE name LIKE '%á%'; -- á
|
||||
UPDATE customers SET name = REPLACE(name, 'ñ', 'ñ') WHERE name LIKE '%ñ%'; -- ñ
|
||||
UPDATE customers SET name = REPLACE(name, 'ö', 'ö') WHERE name LIKE '%ö%'; -- ö
|
||||
UPDATE customers SET name = REPLACE(name, 'ô', 'ô') WHERE name LIKE '%ô%'; -- ô
|
||||
UPDATE customers SET name = REPLACE(name, 'Ä°', 'İ') WHERE name LIKE '%Ä°%'; -- Turkish İ
|
||||
UPDATE customers SET name = REPLACE(name, 'Å', 'Ş') WHERE name LIKE '%Å%'; -- Turkish Ş
|
||||
UPDATE customers SET name = REPLACE(name, 'Ä', 'Ğ') WHERE name LIKE '%Ä%'; -- Turkish Ğ
|
||||
UPDATE customers SET name = REPLACE(name, 'Ä', 'Č') WHERE name LIKE '%Ä%'; -- Czech Č
|
||||
UPDATE customers SET name = REPLACE(name, 'Ä›', 'ě') WHERE name LIKE '%Ä›%'; -- Czech ě
|
||||
|
||||
-- Remove trailing tabs
|
||||
UPDATE customers SET name = TRIM(TRAILING '\t' FROM name) WHERE name LIKE '%\t';
|
||||
UPDATE customers SET trading_name = TRIM(TRAILING '\t' FROM trading_name) WHERE trading_name LIKE '%\t';
|
||||
|
||||
-- ----------------------------------------------------------------------------
|
||||
-- Fix PRINCIPLES table
|
||||
-- ----------------------------------------------------------------------------
|
||||
UPDATE principles SET name = REPLACE(name, 'ü', 'ü') WHERE name LIKE '%ü%';
|
||||
UPDATE principles SET name = REPLACE(name, 'é', 'é') WHERE name LIKE '%é%';
|
||||
UPDATE principles SET name = REPLACE(name, 'ó', 'ó') WHERE name LIKE '%ó%';
|
||||
|
||||
UPDATE principles SET address = REPLACE(address, 'ß', 'ß') WHERE address LIKE '%ß%'; -- German ß
|
||||
UPDATE principles SET address = REPLACE(address, 'ü', 'ü') WHERE address LIKE '%ü%';
|
||||
UPDATE principles SET address = REPLACE(address, 'é', 'é') WHERE address LIKE '%é%';
|
||||
UPDATE principles SET address = REPLACE(address, 'ñ', 'ñ') WHERE address LIKE '%ñ%';
|
||||
UPDATE principles SET address = REPLACE(address, 'Ä', 'č') WHERE address LIKE '%Ä%'; -- Czech č
|
||||
UPDATE principles SET address = REPLACE(address, 'ž', 'ž') WHERE address LIKE '%ž%'; -- Czech ž
|
||||
UPDATE principles SET address = REPLACE(address, 'Å¡', 'š') WHERE address LIKE '%Å¡%'; -- Czech š
|
||||
UPDATE principles SET address = REPLACE(address, 'Ä›', 'ě') WHERE address LIKE '%Ä›%'; -- Czech ě
|
||||
UPDATE principles SET address = REPLACE(address, 'Å', 'ň') WHERE address LIKE '%Å%'; -- Czech ň
|
||||
UPDATE principles SET address = REPLACE(address, 'Å™', 'ř') WHERE address LIKE '%Å™%'; -- Czech ř
|
||||
|
||||
UPDATE principles SET city = REPLACE(city, 'ü', 'ü') WHERE city LIKE '%ü%';
|
||||
UPDATE principles SET city = REPLACE(city, 'é', 'é') WHERE city LIKE '%é%';
|
||||
UPDATE principles SET city = REPLACE(city, 'ó', 'ó') WHERE city LIKE '%ó%';
|
||||
|
||||
-- ----------------------------------------------------------------------------
|
||||
-- Fix ADDRESSES table
|
||||
-- ----------------------------------------------------------------------------
|
||||
UPDATE addresses SET address = REPLACE(address, 'â€"', '–') WHERE address LIKE '%â€"%'; -- En dash
|
||||
UPDATE addresses SET address = REPLACE(address, 'â€"', '—') WHERE address LIKE '%â€"%'; -- Em dash
|
||||
UPDATE addresses SET address = REPLACE(address, 'é', 'é') WHERE address LIKE '%é%';
|
||||
UPDATE addresses SET address = REPLACE(address, 'ü', 'ü') WHERE address LIKE '%ü%';
|
||||
|
||||
UPDATE addresses SET city = REPLACE(city, 'é', 'é') WHERE city LIKE '%é%';
|
||||
UPDATE addresses SET city = REPLACE(city, 'ü', 'ü') WHERE city LIKE '%ü%';
|
||||
|
||||
-- ============================================================================
|
||||
-- PART 2: Specific Record Fixes
|
||||
-- ============================================================================
|
||||
-- These fix specific known corrupted records where pattern matching isn't enough
|
||||
|
||||
-- ----------------------------------------------------------------------------
|
||||
-- PRINCIPLES - Specific fixes
|
||||
-- ----------------------------------------------------------------------------
|
||||
|
||||
-- ID 2: German ß
|
||||
UPDATE principles
|
||||
SET address = 'Heinz-Fangman-Straße 18'
|
||||
WHERE id = 2;
|
||||
|
||||
-- ID 9: Czech characters (this is complex, needs full correction)
|
||||
UPDATE principles
|
||||
SET address = 'Bezručova 2901\n756 61 Rožnov pod Radhoštěm',
|
||||
city = 'Rožnov pod Radhoštěm'
|
||||
WHERE id = 9;
|
||||
|
||||
-- ID 13: German ü
|
||||
UPDATE principles
|
||||
SET name = 'IEP Technologies GmbH - BRILEX Gesellschaft für Explosionsschutz mbH'
|
||||
WHERE id = 13;
|
||||
|
||||
-- ID 14: German ß
|
||||
UPDATE principles
|
||||
SET address = 'Liebigstraße 2'
|
||||
WHERE id = 14;
|
||||
|
||||
-- ID 58: Spanish ó
|
||||
UPDATE principles
|
||||
SET name = 'Nosia S.r.l.- Señalización Industrial'
|
||||
WHERE id = 58;
|
||||
|
||||
-- ID 65: German ß
|
||||
UPDATE principles
|
||||
SET address = 'Alte Emser Straße 32'
|
||||
WHERE id = 65;
|
||||
|
||||
-- ----------------------------------------------------------------------------
|
||||
-- CUSTOMERS - Specific fixes
|
||||
-- ----------------------------------------------------------------------------
|
||||
|
||||
-- ID 253: É
|
||||
UPDATE customers
|
||||
SET name = 'SOLUZÉ CIVIL ENGINEERS Trading Under SOLUZE PTY LTD'
|
||||
WHERE id = 253;
|
||||
|
||||
-- ID 1006: Smart apostrophe
|
||||
UPDATE customers
|
||||
SET name = 'Dr. Prem''s Molecules Private Limited (DPMolecules)'
|
||||
WHERE id = 1006;
|
||||
|
||||
-- ID 1387: Smart quotes
|
||||
UPDATE customers
|
||||
SET name = 'DEE ENTERPRISES (QLD) PTY LTD trading as "MEKLEK"'
|
||||
WHERE id = 1387;
|
||||
|
||||
-- ID 1608: Smart apostrophe
|
||||
UPDATE customers
|
||||
SET name = 'Guidera O''Connor Pty Ltd'
|
||||
WHERE id = 1608;
|
||||
|
||||
-- ID 2174: Zero-width space
|
||||
UPDATE customers
|
||||
SET name = 'Ingredion ANZ Pty Ltd'
|
||||
WHERE id = 2174;
|
||||
|
||||
-- ID 2215: French é
|
||||
UPDATE customers
|
||||
SET name = 'Vale Nouvelle-Calédonie S.A.S'
|
||||
WHERE id = 2215;
|
||||
|
||||
-- ID 2375: Spanish ó
|
||||
UPDATE customers
|
||||
SET name = 'Evaluación y Control Ambiental S.A.S.'
|
||||
WHERE id = 2375;
|
||||
|
||||
-- ID 3143: Portuguese ç and ó
|
||||
UPDATE customers
|
||||
SET name = 'Zontahevy Comércio e Serviços Offshore Ltda'
|
||||
WHERE id = 3143;
|
||||
|
||||
-- ID 3529: French é
|
||||
UPDATE customers
|
||||
SET name = 'Société des Mines de Syama'
|
||||
WHERE id = 3529;
|
||||
|
||||
-- ID 3633: Special space (em space)
|
||||
UPDATE customers
|
||||
SET name = 'P.J. Berriman & Co PTY LTD'
|
||||
WHERE id = 3633;
|
||||
|
||||
-- ID 4325: Danish ø
|
||||
UPDATE customers
|
||||
SET name = 'Rambøll Danmark'
|
||||
WHERE id = 4325;
|
||||
|
||||
-- ID 4350: Turkish characters İ, Ş, Ğ
|
||||
UPDATE customers
|
||||
SET name = 'SONNIVA ENERGY MAKİNA İNŞAAT İTH. İHR. LTD. ŞTİ.'
|
||||
WHERE id = 4350;
|
||||
|
||||
-- ID 4669: French é
|
||||
UPDATE customers
|
||||
SET name = 'F.C.C. Fluides Conseils Calédonie SARL'
|
||||
WHERE id = 4669;
|
||||
|
||||
-- ID 4743: Czech ě
|
||||
UPDATE customers
|
||||
SET name = 'DGPack Prostějov'
|
||||
WHERE id = 4743;
|
||||
|
||||
-- ID 4764: Zero-width space
|
||||
UPDATE customers
|
||||
SET name = 'Mccready Welding Pty Ltd Trading under the entity Mccready''s Welding Services'
|
||||
WHERE id = 4764;
|
||||
|
||||
-- ID 4893: Spanish í
|
||||
UPDATE customers
|
||||
SET name = 'Oxiquímica, S.A.P.I. de C.V.'
|
||||
WHERE id = 4893;
|
||||
|
||||
-- ----------------------------------------------------------------------------
|
||||
-- ADDRESSES - Specific fixes
|
||||
-- ----------------------------------------------------------------------------
|
||||
|
||||
-- ID 19: En dash (7–11)
|
||||
UPDATE addresses
|
||||
SET address = 'Lvl 3, Building B, 7–11 Talavera Road'
|
||||
WHERE id = 19;
|
||||
|
||||
-- ============================================================================
|
||||
-- PART 3: Fix PRODUCTS Table (12,536+ corrupted records)
|
||||
-- ============================================================================
|
||||
-- Products table has extensive corruption from mojibake and HTML entities
|
||||
|
||||
-- Degree symbol: ° → °
|
||||
UPDATE products SET title = REPLACE(title, '°', '°') WHERE title LIKE '%°%';
|
||||
UPDATE products SET description = REPLACE(description, '°', '°') WHERE description LIKE '%°%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, '°', '°') WHERE item_description LIKE '%°%';
|
||||
|
||||
-- Registered trademark: ® → ®
|
||||
UPDATE products SET title = REPLACE(title, '®', '®') WHERE title LIKE '%®%';
|
||||
UPDATE products SET description = REPLACE(description, '®', '®') WHERE description LIKE '%®%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, '®', '®') WHERE item_description LIKE '%®%';
|
||||
|
||||
-- Trademark: â„¢ → ™
|
||||
UPDATE products SET title = REPLACE(title, 'â„¢', '™') WHERE title LIKE '%â„¢%';
|
||||
UPDATE products SET description = REPLACE(description, 'â„¢', '™') WHERE description LIKE '%â„¢%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, 'â„¢', '™') WHERE item_description LIKE '%â„¢%';
|
||||
|
||||
-- Smart quotes and dashes in products
|
||||
UPDATE products SET title = REPLACE(title, '’', ''') WHERE title LIKE '%’%';
|
||||
UPDATE products SET description = REPLACE(description, '’', ''') WHERE description LIKE '%’%';
|
||||
UPDATE products SET title = REPLACE(title, '“', '"') WHERE title LIKE '%“%';
|
||||
UPDATE products SET description = REPLACE(description, '“', '"') WHERE description LIKE '%“%';
|
||||
UPDATE products SET title = REPLACE(title, 'â€', '"') WHERE title LIKE '%â€%';
|
||||
UPDATE products SET description = REPLACE(description, 'â€', '"') WHERE description LIKE '%â€%';
|
||||
UPDATE products SET title = REPLACE(title, 'â€"', '–') WHERE title LIKE '%â€"%';
|
||||
UPDATE products SET description = REPLACE(description, 'â€"', '–') WHERE description LIKE '%â€"%';
|
||||
UPDATE products SET title = REPLACE(title, 'â€"', '—') WHERE title LIKE '%â€"%';
|
||||
UPDATE products SET description = REPLACE(description, 'â€"', '—') WHERE description LIKE '%â€"%';
|
||||
|
||||
-- Accented characters in products
|
||||
UPDATE products SET title = REPLACE(title, 'é', 'é') WHERE title LIKE '%é%';
|
||||
UPDATE products SET description = REPLACE(description, 'é', 'é') WHERE description LIKE '%é%';
|
||||
UPDATE products SET title = REPLACE(title, 'ü', 'ü') WHERE title LIKE '%ü%';
|
||||
UPDATE products SET description = REPLACE(description, 'ü', 'ü') WHERE description LIKE '%ü%';
|
||||
|
||||
-- HTML entities (shouldn't be in database)
|
||||
UPDATE products SET title = REPLACE(title, '–', '–') WHERE title LIKE '%–%';
|
||||
UPDATE products SET description = REPLACE(description, '–', '–') WHERE description LIKE '%–%';
|
||||
UPDATE products SET title = REPLACE(title, '”', '"') WHERE title LIKE '%”%';
|
||||
UPDATE products SET description = REPLACE(description, '”', '"') WHERE description LIKE '%”%';
|
||||
UPDATE products SET title = REPLACE(title, '“', '"') WHERE title LIKE '%“%';
|
||||
UPDATE products SET description = REPLACE(description, '“', '"') WHERE description LIKE '%“%';
|
||||
UPDATE products SET title = REPLACE(title, '&', '&') WHERE title LIKE '%&%';
|
||||
UPDATE products SET description = REPLACE(description, '&', '&') WHERE description LIKE '%&%';
|
||||
UPDATE products SET title = REPLACE(title, '½', '½') WHERE title LIKE '%½%';
|
||||
UPDATE products SET description = REPLACE(description, '½', '½') WHERE description LIKE '%½%';
|
||||
UPDATE products SET title = REPLACE(title, '°', '°') WHERE title LIKE '%°%';
|
||||
UPDATE products SET description = REPLACE(description, '°', '°') WHERE description LIKE '%°%';
|
||||
UPDATE products SET title = REPLACE(title, ' ', ' ') WHERE title LIKE '% %';
|
||||
UPDATE products SET description = REPLACE(description, ' ', ' ') WHERE description LIKE '% %';
|
||||
|
||||
-- Specific product fixes
|
||||
UPDATE products SET title = 'C95SN189C DSTGL40/C-Digital temperature probe for P8xx1 Web Sensor. Range -30 to +80°C. With CINCH connector, 1m Cable' WHERE id = 30;
|
||||
UPDATE products SET title = 'Mid-West Instrument Model 240 – SC – 02 – O(TT)' WHERE id = 38;
|
||||
UPDATE products SET title = 'Newson Gale Earth-Rite® II FIBC Static Earthing System, GRP Enclosure with 5m 2 Core Spiral Cable and FIBC Clamp' WHERE id = 67;
|
||||
UPDATE products SET title = 'Newson Gale Earth-Rite® RTR™ Tri-Mode Static Grounding System, Metal Enclosure, X90-IP Heavy Duty Clamp with 10m 2 Core Spiral Cable' WHERE id = 85;
|
||||
|
||||
-- ============================================================================
|
||||
-- PART 4: Verification Queries
|
||||
-- ============================================================================
|
||||
-- Run these after the fixes to verify they worked
|
||||
|
||||
-- Check fixed principles
|
||||
SELECT id, name, city FROM principles WHERE id IN (2, 9, 13, 14, 58, 65);
|
||||
|
||||
-- Check fixed customers
|
||||
SELECT id, name FROM customers WHERE id IN (253, 1006, 1387, 1608, 2174, 2215, 2375, 3143, 3529, 4325, 4350, 4669, 4743, 4764, 4893);
|
||||
|
||||
-- Check fixed addresses
|
||||
SELECT id, address FROM addresses WHERE id = 19;
|
||||
|
||||
-- Count remaining corrupted records (should be much lower)
|
||||
SELECT COUNT(*) as remaining_corrupted_principles
|
||||
FROM principles
|
||||
WHERE name REGEXP '[^ -~]' OR address REGEXP '[^ -~]' OR city REGEXP '[^ -~]';
|
||||
|
||||
SELECT COUNT(*) as remaining_corrupted_customers
|
||||
FROM customers
|
||||
WHERE name REGEXP '[^ -~]';
|
||||
|
||||
SELECT COUNT(*) as remaining_corrupted_addresses
|
||||
FROM addresses
|
||||
WHERE address REGEXP '[^ -~]' OR city REGEXP '[^ -~]';
|
||||
|
||||
SELECT COUNT(*) as remaining_corrupted_products
|
||||
FROM products
|
||||
WHERE title REGEXP '[^ -~]' OR description REGEXP '[^ -~]' OR item_description REGEXP '[^ -~]';
|
||||
|
||||
-- Check sample fixed products
|
||||
SELECT id, title FROM products WHERE id IN (30, 38, 67, 85);
|
||||
|
||||
-- ============================================================================
|
||||
-- COMPLETION
|
||||
-- ============================================================================
|
||||
SELECT 'Fix script completed! Review the verification queries above.' as status;
|
||||
|
|
@ -1,221 +0,0 @@
|
|||
-- ============================================================================
|
||||
-- Fix Corrupted Character Data in CMC Database (Using HEX encoding)
|
||||
-- ============================================================================
|
||||
-- This script fixes mojibake using HEX encoding to avoid UTF-8 interpretation issues
|
||||
-- Run after converting tables to utf8mb4
|
||||
--
|
||||
-- IMPORTANT: Review this script and test on a backup first!
|
||||
-- To run: docker exec -i cmc-db mariadb -u root -psecureRootPassword --default-character-set=utf8mb4 cmc < scripts/fix_corrupted_data_hex.sql
|
||||
-- ============================================================================
|
||||
|
||||
SET NAMES utf8mb4;
|
||||
|
||||
-- ============================================================================
|
||||
-- PART 1: Fix CUSTOMERS table using HEX encoding
|
||||
-- ============================================================================
|
||||
|
||||
-- Smart apostrophe: ’ → '
|
||||
UPDATE customers SET name = REPLACE(name, UNHEX('E28099'), UNHEX('E28099')) WHERE name LIKE CONCAT('%', UNHEX('E28099'), '%');
|
||||
-- Actually, let's use the simpler binary approach
|
||||
UPDATE customers SET name = REPLACE(name, _binary 0xE28099, '''') WHERE name LIKE _binary CONCAT('%', 0xE28099, '%');
|
||||
|
||||
-- Left double quote: “ → "
|
||||
UPDATE customers SET name = REPLACE(name, _binary 0xE2809C, '"') WHERE name LIKE _binary CONCAT('%', 0xE2809C, '%');
|
||||
|
||||
-- Right double quote: †→ "
|
||||
UPDATE customers SET name = REPLACE(name, _binary 0xE2809D, '"') WHERE name LIKE _binary CONCAT('%', 0xE2809D, '%');
|
||||
|
||||
-- En dash: â€" → –
|
||||
UPDATE customers SET name = REPLACE(name, _binary 0xE28093, '–') WHERE name LIKE _binary CONCAT('%', 0xE28093, '%');
|
||||
|
||||
-- Em dash: â€" → —
|
||||
UPDATE customers SET name = REPLACE(name, _binary 0xE28094, '—') WHERE name LIKE _binary CONCAT('%', 0xE28094, '%');
|
||||
|
||||
-- Zero-width space: ​ → (remove)
|
||||
UPDATE customers SET name = REPLACE(name, _binary 0xE2808B, '') WHERE name LIKE _binary CONCAT('%', 0xE2808B, '%');
|
||||
|
||||
-- Em space:   → (regular space)
|
||||
UPDATE customers SET name = REPLACE(name, _binary 0xE28083, ' ') WHERE name LIKE _binary CONCAT('%', 0xE28083, '%');
|
||||
|
||||
-- Accented é: é → é
|
||||
UPDATE customers SET name = REPLACE(name, _binary 0xC3A9, 'é') WHERE name LIKE _binary CONCAT('%', 0xC3A9, '%');
|
||||
|
||||
-- Accented É: É → É
|
||||
UPDATE customers SET name = REPLACE(name, _binary 0xC389, 'É') WHERE name LIKE _binary CONCAT('%', 0xC389, '%');
|
||||
|
||||
-- Accented ó: ó → ó
|
||||
UPDATE customers SET name = REPLACE(name, _binary 0xC3B3, 'ó') WHERE name LIKE _binary CONCAT('%', 0xC3B3, '%');
|
||||
|
||||
-- Accented í: Ã → í
|
||||
UPDATE customers SET name = REPLACE(name, _binary 0xC3AD, 'í') WHERE name LIKE _binary CONCAT('%', 0xC3AD, '%');
|
||||
|
||||
-- Accented ç: ç → ç
|
||||
UPDATE customers SET name = REPLACE(name, _binary 0xC3A7, 'ç') WHERE name LIKE _binary CONCAT('%', 0xC3A7, '%');
|
||||
|
||||
-- Accented ü: ü → ü
|
||||
UPDATE customers SET name = REPLACE(name, _binary 0xC3BC, 'ü') WHERE name LIKE _binary CONCAT('%', 0xC3BC, '%');
|
||||
|
||||
-- Accented á: á → á
|
||||
UPDATE customers SET name = REPLACE(name, _binary 0xC3A1, 'á') WHERE name LIKE _binary CONCAT('%', 0xC3A1, '%');
|
||||
|
||||
-- Accented ñ: ñ → ñ
|
||||
UPDATE customers SET name = REPLACE(name, _binary 0xC3B1, 'ñ') WHERE name LIKE _binary CONCAT('%', 0xC3B1, '%');
|
||||
|
||||
-- Accented ö: ö → ö
|
||||
UPDATE customers SET name = REPLACE(name, _binary 0xC3B6, 'ö') WHERE name LIKE _binary CONCAT('%', 0xC3B6, '%');
|
||||
|
||||
-- Accented ô: ô → ô
|
||||
UPDATE customers SET name = REPLACE(name, _binary 0xC3B4, 'ô') WHERE name LIKE _binary CONCAT('%', 0xC3B4, '%');
|
||||
|
||||
-- Turkish İ: Ä° → İ
|
||||
UPDATE customers SET name = REPLACE(name, _binary 0xC4B0, 'İ') WHERE name LIKE _binary CONCAT('%', 0xC4B0, '%');
|
||||
|
||||
-- Turkish Ş: Å → Ş
|
||||
UPDATE customers SET name = REPLACE(name, _binary 0xC59E, 'Ş') WHERE name LIKE _binary CONCAT('%', 0xC59E, '%');
|
||||
|
||||
-- Turkish Ğ: Ä → Ğ
|
||||
UPDATE customers SET name = REPLACE(name, _binary 0xC49E, 'Ğ') WHERE name LIKE _binary CONCAT('%', 0xC49E, '%');
|
||||
|
||||
-- Czech Č: Ä → Č
|
||||
UPDATE customers SET name = REPLACE(name, _binary 0xC48C, 'Č') WHERE name LIKE _binary CONCAT('%', 0xC48C, '%');
|
||||
|
||||
-- Czech ě: Ä› → ě
|
||||
UPDATE customers SET name = REPLACE(name, _binary 0xC49B, 'ě') WHERE name LIKE _binary CONCAT('%', 0xC49B, '%');
|
||||
|
||||
-- Remove trailing tabs
|
||||
UPDATE customers SET name = TRIM(TRAILING '\t' FROM name) WHERE name LIKE '%\t';
|
||||
UPDATE customers SET trading_name = TRIM(TRAILING '\t' FROM trading_name) WHERE trading_name LIKE '%\t';
|
||||
|
||||
-- ============================================================================
|
||||
-- PART 2: Fix PRINCIPLES table
|
||||
-- ============================================================================
|
||||
|
||||
-- Accented ü: ü → ü
|
||||
UPDATE principles SET name = REPLACE(name, _binary 0xC3BC, 'ü') WHERE name LIKE _binary CONCAT('%', 0xC3BC, '%');
|
||||
UPDATE principles SET address = REPLACE(address, _binary 0xC3BC, 'ü') WHERE address LIKE _binary CONCAT('%', 0xC3BC, '%');
|
||||
UPDATE principles SET city = REPLACE(city, _binary 0xC3BC, 'ü') WHERE city LIKE _binary CONCAT('%', 0xC3BC, '%');
|
||||
|
||||
-- Accented é: é → é
|
||||
UPDATE principles SET name = REPLACE(name, _binary 0xC3A9, 'é') WHERE name LIKE _binary CONCAT('%', 0xC3A9, '%');
|
||||
UPDATE principles SET address = REPLACE(address, _binary 0xC3A9, 'é') WHERE address LIKE _binary CONCAT('%', 0xC3A9, '%');
|
||||
UPDATE principles SET city = REPLACE(city, _binary 0xC3A9, 'é') WHERE city LIKE _binary CONCAT('%', 0xC3A9, '%');
|
||||
|
||||
-- Accented ó: ó → ó
|
||||
UPDATE principles SET name = REPLACE(name, _binary 0xC3B3, 'ó') WHERE name LIKE _binary CONCAT('%', 0xC3B3, '%');
|
||||
UPDATE principles SET city = REPLACE(city, _binary 0xC3B3, 'ó') WHERE city LIKE _binary CONCAT('%', 0xC3B3, '%');
|
||||
|
||||
-- German ß: ß → ß
|
||||
UPDATE principles SET address = REPLACE(address, _binary 0xC39F, 'ß') WHERE address LIKE _binary CONCAT('%', 0xC39F, '%');
|
||||
|
||||
-- Accented ñ: ñ → ñ
|
||||
UPDATE principles SET address = REPLACE(address, _binary 0xC3B1, 'ñ') WHERE address LIKE _binary CONCAT('%', 0xC3B1, '%');
|
||||
|
||||
-- Czech č: Ä → č
|
||||
UPDATE principles SET address = REPLACE(address, _binary 0xC48D, 'č') WHERE address LIKE _binary CONCAT('%', 0xC48D, '%');
|
||||
|
||||
-- Czech ž: ž → ž
|
||||
UPDATE principles SET address = REPLACE(address, _binary 0xC5BE, 'ž') WHERE address LIKE _binary CONCAT('%', 0xC5BE, '%');
|
||||
|
||||
-- Czech š: Å¡ → š
|
||||
UPDATE principles SET address = REPLACE(address, _binary 0xC5A1, 'š') WHERE address LIKE _binary CONCAT('%', 0xC5A1, '%');
|
||||
|
||||
-- Czech ě: Ä› → ě
|
||||
UPDATE principles SET address = REPLACE(address, _binary 0xC49B, 'ě') WHERE address LIKE _binary CONCAT('%', 0xC49B, '%');
|
||||
|
||||
-- Czech ň: Å → ň
|
||||
UPDATE principles SET address = REPLACE(address, _binary 0xC588, 'ň') WHERE address LIKE _binary CONCAT('%', 0xC588, '%');
|
||||
|
||||
-- Czech ř: Å™ → ř
|
||||
UPDATE principles SET address = REPLACE(address, _binary 0xC599, 'ř') WHERE address LIKE _binary CONCAT('%', 0xC599, '%');
|
||||
|
||||
-- Specific principle fixes
|
||||
UPDATE principles SET address = 'Heinz-Fangman-Straße 18' WHERE id = 2;
|
||||
UPDATE principles SET address = 'Bezručova 2901\n756 61 Rožnov pod Radhoštěm', city = 'Rožnov pod Radhoštěm' WHERE id = 9;
|
||||
UPDATE principles SET name = 'IEP Technologies GmbH - BRILEX Gesellschaft für Explosionsschutz mbH' WHERE id = 13;
|
||||
UPDATE principles SET address = 'Liebigstraße 2' WHERE id = 14;
|
||||
UPDATE principles SET name = 'Nosia S.r.l.- Señalización Industrial' WHERE id = 58;
|
||||
UPDATE principles SET address = 'Alte Emser Straße 32' WHERE id = 65;
|
||||
|
||||
-- ============================================================================
|
||||
-- PART 3: Fix ADDRESSES table
|
||||
-- ============================================================================
|
||||
|
||||
-- En dash: â€" → –
|
||||
UPDATE addresses SET address = REPLACE(address, _binary 0xE28093, '–') WHERE address LIKE _binary CONCAT('%', 0xE28093, '%');
|
||||
|
||||
-- Accented é: é → é
|
||||
UPDATE addresses SET address = REPLACE(address, _binary 0xC3A9, 'é') WHERE address LIKE _binary CONCAT('%', 0xC3A9, '%');
|
||||
UPDATE addresses SET city = REPLACE(city, _binary 0xC3A9, 'é') WHERE city LIKE _binary CONCAT('%', 0xC3A9, '%');
|
||||
|
||||
-- Specific address fix
|
||||
UPDATE addresses SET address = 'Lvl 3, Building B, 7–11 Talavera Road' WHERE id = 19;
|
||||
|
||||
-- ============================================================================
|
||||
-- PART 4: Fix PRODUCTS table (using simple text for known values)
|
||||
-- ============================================================================
|
||||
|
||||
-- Degree symbol: ° → °
|
||||
UPDATE products SET title = REPLACE(title, CHAR(194,176), CHAR(176)) WHERE title LIKE CONCAT('%', CHAR(194,176), '%');
|
||||
UPDATE products SET description = REPLACE(description, CHAR(194,176), CHAR(176)) WHERE description LIKE CONCAT('%', CHAR(194,176), '%');
|
||||
UPDATE products SET item_description = REPLACE(item_description, CHAR(194,176), CHAR(176)) WHERE item_description LIKE CONCAT('%', CHAR(194,176), '%');
|
||||
|
||||
-- Registered trademark: ® → ®
|
||||
UPDATE products SET title = REPLACE(title, CHAR(194,174), CHAR(174)) WHERE title LIKE CONCAT('%', CHAR(194,174), '%');
|
||||
UPDATE products SET description = REPLACE(description, CHAR(194,174), CHAR(174)) WHERE description LIKE CONCAT('%', CHAR(194,174), '%');
|
||||
UPDATE products SET item_description = REPLACE(item_description, CHAR(194,174), CHAR(174)) WHERE item_description LIKE CONCAT('%', CHAR(194,174), '%');
|
||||
|
||||
-- Trademark: â„¢ → ™ (0xE284A2 → 0x2122)
|
||||
UPDATE products SET title = REPLACE(title, CHAR(226,132,162), CHAR(226,132,162)) WHERE title LIKE CONCAT('%', CHAR(226,132,162), '%');
|
||||
UPDATE products SET description = REPLACE(description, CHAR(226,132,162), CHAR(226,132,162)) WHERE description LIKE CONCAT('%', CHAR(226,132,162), '%');
|
||||
|
||||
-- Smart apostrophe ’ → '
|
||||
UPDATE products SET title = REPLACE(title, CHAR(226,128,153), '''') WHERE title LIKE CONCAT('%', CHAR(226,128,153), '%');
|
||||
UPDATE products SET description = REPLACE(description, CHAR(226,128,153), '''') WHERE description LIKE CONCAT('%', CHAR(226,128,153), '%');
|
||||
|
||||
-- En dash â€" → –
|
||||
UPDATE products SET title = REPLACE(title, CHAR(226,128,147), '–') WHERE title LIKE CONCAT('%', CHAR(226,128,147), '%');
|
||||
UPDATE products SET description = REPLACE(description, CHAR(226,128,147), '–') WHERE description LIKE CONCAT('%', CHAR(226,128,147), '%');
|
||||
|
||||
-- HTML entities (safe to use text)
|
||||
UPDATE products SET title = REPLACE(title, '–', '–') WHERE title LIKE '%–%';
|
||||
UPDATE products SET description = REPLACE(description, '–', '–') WHERE description LIKE '%–%';
|
||||
UPDATE products SET title = REPLACE(title, '”', '"') WHERE title LIKE '%”%';
|
||||
UPDATE products SET description = REPLACE(description, '”', '"') WHERE description LIKE '%”%';
|
||||
UPDATE products SET title = REPLACE(title, '“', '"') WHERE title LIKE '%“%';
|
||||
UPDATE products SET description = REPLACE(description, '“', '"') WHERE description LIKE '%“%';
|
||||
UPDATE products SET title = REPLACE(title, '&', '&') WHERE title LIKE '%&%';
|
||||
UPDATE products SET description = REPLACE(description, '&', '&') WHERE description LIKE '%&%';
|
||||
UPDATE products SET title = REPLACE(title, '½', '½') WHERE title LIKE '%½%';
|
||||
UPDATE products SET description = REPLACE(description, '½', '½') WHERE description LIKE '%½%';
|
||||
UPDATE products SET title = REPLACE(title, '°', '°') WHERE title LIKE '%°%';
|
||||
UPDATE products SET description = REPLACE(description, '°', '°') WHERE description LIKE '%°%';
|
||||
UPDATE products SET title = REPLACE(title, ' ', ' ') WHERE title LIKE '% %';
|
||||
UPDATE products SET description = REPLACE(description, ' ', ' ') WHERE description LIKE '% %';
|
||||
|
||||
-- Specific product fixes
|
||||
UPDATE products SET title = 'C95SN189C DSTGL40/C-Digital temperature probe for P8xx1 Web Sensor. Range -30 to +80°C. With CINCH connector, 1m Cable' WHERE id = 30;
|
||||
UPDATE products SET title = 'Mid-West Instrument Model 240 – SC – 02 – O(TT)' WHERE id = 38;
|
||||
UPDATE products SET title = 'Newson Gale Earth-Rite® II FIBC Static Earthing System, GRP Enclosure with 5m 2 Core Spiral Cable and FIBC Clamp' WHERE id = 67;
|
||||
UPDATE products SET title = 'Newson Gale Earth-Rite® RTR™ Tri-Mode Static Grounding System, Metal Enclosure, X90-IP Heavy Duty Clamp with 10m 2 Core Spiral Cable' WHERE id = 85;
|
||||
|
||||
-- ============================================================================
|
||||
-- Verification
|
||||
-- ============================================================================
|
||||
|
||||
SELECT '===========================================================' as '';
|
||||
SELECT 'FIX COMPLETE - Verification Results' as '';
|
||||
SELECT '===========================================================' as '';
|
||||
|
||||
SELECT 'Remaining corrupted principles:' as '', COUNT(*) as count
|
||||
FROM principles WHERE name REGEXP '[^ -~]' OR address REGEXP '[^ -~]' OR city REGEXP '[^ -~]';
|
||||
|
||||
SELECT 'Remaining corrupted customers:' as '', COUNT(*) as count
|
||||
FROM customers WHERE name REGEXP '[^ -~]';
|
||||
|
||||
SELECT 'Remaining corrupted addresses:' as '', COUNT(*) as count
|
||||
FROM addresses WHERE address REGEXP '[^ -~]' OR city REGEXP '[^ -~]';
|
||||
|
||||
SELECT 'Remaining corrupted products:' as '', COUNT(*) as count
|
||||
FROM products WHERE title REGEXP '[^ -~]' OR description REGEXP '[^ -~]' OR item_description REGEXP '[^ -~]';
|
||||
|
||||
SELECT 'Fix script completed!' as status;
|
||||
|
|
@ -1,97 +0,0 @@
|
|||
#!/bin/bash
|
||||
# ============================================================================
|
||||
# Fix Corrupted Character Data - Safe Version
|
||||
# ============================================================================
|
||||
# This script fixes mojibake by generating SQL dynamically to avoid encoding issues
|
||||
# ============================================================================
|
||||
|
||||
DB_USER="root"
|
||||
DB_PASS="secureRootPassword"
|
||||
DB_NAME="cmc"
|
||||
CONTAINER="cmc-db"
|
||||
|
||||
# Function to run SQL
|
||||
run_sql() {
|
||||
docker exec $CONTAINER mariadb -u $DB_USER -p$DB_PASS --default-character-set=utf8mb4 $DB_NAME -e "$1"
|
||||
}
|
||||
|
||||
echo "============================================================"
|
||||
echo "Fixing Character Corruption - Safe Version"
|
||||
echo "============================================================"
|
||||
echo ""
|
||||
|
||||
# Set charset
|
||||
run_sql "SET NAMES utf8mb4;"
|
||||
|
||||
echo "Fixing CUSTOMERS table..."
|
||||
# Fix customers - specific known records
|
||||
run_sql "UPDATE customers SET name = 'SOLUZÉ CIVIL ENGINEERS Trading Under SOLUZE PTY LTD' WHERE id = 253;"
|
||||
run_sql "UPDATE customers SET name = 'Dr. Prem''s Molecules Private Limited (DPMolecules)' WHERE id = 1006;"
|
||||
run_sql "UPDATE customers SET name = 'DEE ENTERPRISES (QLD) PTY LTD trading as \"MEKLEK\"' WHERE id = 1387;"
|
||||
run_sql "UPDATE customers SET name = 'Guidera O''Connor Pty Ltd' WHERE id = 1608;"
|
||||
run_sql "UPDATE customers SET name = 'Ingredion ANZ Pty Ltd' WHERE id = 2174;"
|
||||
run_sql "UPDATE customers SET name = 'Vale Nouvelle-Calédonie S.A.S' WHERE id = 2215;"
|
||||
run_sql "UPDATE customers SET name = 'Evaluación y Control Ambiental S.A.S.' WHERE id = 2375;"
|
||||
run_sql "UPDATE customers SET name = 'Zontahevy Comércio e Serviços Offshore Ltda' WHERE id = 3143;"
|
||||
run_sql "UPDATE customers SET name = 'Société des Mines de Syama' WHERE id = 3529;"
|
||||
run_sql "UPDATE customers SET name = 'P.J. Berriman & Co PTY LTD' WHERE id = 3633;"
|
||||
run_sql "UPDATE customers SET name = 'Rambøll Danmark' WHERE id = 4325;"
|
||||
run_sql "UPDATE customers SET name = 'SONNIVA ENERGY MAKİNA İNŞAAT İTH. İHR. LTD. ŞTİ.' WHERE id = 4350;"
|
||||
run_sql "UPDATE customers SET name = 'F.C.C. Fluides Conseils Calédonie SARL' WHERE id = 4669;"
|
||||
run_sql "UPDATE customers SET name = 'DGPack Prostějov' WHERE id = 4743;"
|
||||
run_sql "UPDATE customers SET name = 'Mccready Welding Pty Ltd Trading under the entity Mccready''s Welding Services' WHERE id = 4764;"
|
||||
run_sql "UPDATE customers SET name = 'Oxiquímica, S.A.P.I. de C.V.' WHERE id = 4893;"
|
||||
|
||||
echo "Fixing PRINCIPLES table..."
|
||||
run_sql "UPDATE principles SET address = 'Heinz-Fangman-Straße 18' WHERE id = 2;"
|
||||
run_sql "UPDATE principles SET address = 'Bezručova 2901\n756 61 Rožnov pod Radhoštěm', city = 'Rožnov pod Radhoštěm' WHERE id = 9;"
|
||||
run_sql "UPDATE principles SET name = 'IEP Technologies GmbH - BRILEX Gesellschaft für Explosionsschutz mbH' WHERE id = 13;"
|
||||
run_sql "UPDATE principles SET address = 'Liebigstraße 2' WHERE id = 14;"
|
||||
run_sql "UPDATE principles SET name = 'Nosia S.r.l.- Señalización Industrial' WHERE id = 58;"
|
||||
run_sql "UPDATE principles SET address = 'Alte Emser Straße 32' WHERE id = 65;"
|
||||
|
||||
echo "Fixing ADDRESSES table..."
|
||||
run_sql "UPDATE addresses SET address = 'Lvl 3, Building B, 7–11 Talavera Road' WHERE id = 19;"
|
||||
|
||||
echo "Fixing PRODUCTS table (specific records)..."
|
||||
run_sql "UPDATE products SET title = 'C95SN189C DSTGL40/C-Digital temperature probe for P8xx1 Web Sensor. Range -30 to +80°C. With CINCH connector, 1m Cable' WHERE id = 30;"
|
||||
run_sql "UPDATE products SET title = 'Mid-West Instrument Model 240 – SC – 02 – O(TT)' WHERE id = 38;"
|
||||
run_sql "UPDATE products SET title = 'Newson Gale Earth-Rite® II FIBC Static Earthing System, GRP Enclosure with 5m 2 Core Spiral Cable and FIBC Clamp' WHERE id = 67;"
|
||||
run_sql "UPDATE products SET title = 'Newson Gale Earth-Rite® RTR Tester - ER2/CRT' WHERE id = 76;"
|
||||
run_sql "UPDATE products SET title = 'Newson Gale Earth-Rite® Installers Kit A for IIB areas only (Metal Enclosures Only)' WHERE id = 77;"
|
||||
run_sql "UPDATE products SET title = 'Newson Gale Earth-Rite® RTR™ Tri-Mode Static Grounding System, Metal Enclosure, X90-IP Heavy Duty Clamp with 10m 2 Core Spiral Cable' WHERE id = 85;"
|
||||
run_sql "UPDATE products SET title = 'Newson Gale Earth-Rite® RTR System Spares - X90-IP Heavy Duty 2 Core Clamp with 10m 2 Core Spiral Cable and Male Quick Connect' WHERE id = 86;"
|
||||
run_sql "UPDATE products SET title = 'Newson Gale Bond-Rite® EZ, VESX90-IP Heavy Duty Clamp & 3m 2 Core Spiral Cable' WHERE id = 149;"
|
||||
|
||||
echo ""
|
||||
echo "Fixing HTML entities in products..."
|
||||
# These are safe ASCII characters, so no encoding issues
|
||||
run_sql "UPDATE products SET title = REPLACE(title, '–', '–') WHERE title LIKE '%–%';"
|
||||
run_sql "UPDATE products SET description = REPLACE(description, '–', '–') WHERE description LIKE '%–%';"
|
||||
run_sql "UPDATE products SET title = REPLACE(title, '&', '&') WHERE title LIKE '%&%';"
|
||||
run_sql "UPDATE products SET description = REPLACE(description, '&', '&') WHERE description LIKE '%&%';"
|
||||
run_sql "UPDATE products SET title = REPLACE(title, ' ', ' ') WHERE title LIKE '% %';"
|
||||
run_sql "UPDATE products SET description = REPLACE(description, ' ', ' ') WHERE description LIKE '% %';"
|
||||
run_sql "UPDATE products SET title = REPLACE(title, '°', '°') WHERE title LIKE '%°%';"
|
||||
run_sql "UPDATE products SET description = REPLACE(description, '°', '°') WHERE description LIKE '%°%';"
|
||||
run_sql "UPDATE products SET title = REPLACE(title, '½', '½') WHERE title LIKE '%½%';"
|
||||
run_sql "UPDATE products SET description = REPLACE(description, '½', '½') WHERE description LIKE '%½%';"
|
||||
run_sql "UPDATE products SET title = REPLACE(title, '”', '\"') WHERE title LIKE '%”%';"
|
||||
run_sql "UPDATE products SET description = REPLACE(description, '”', '\"') WHERE description LIKE '%”%';"
|
||||
run_sql "UPDATE products SET title = REPLACE(title, '“', '\"') WHERE title LIKE '%“%';"
|
||||
run_sql "UPDATE products SET description = REPLACE(description, '“', '\"') WHERE description LIKE '%“%';"
|
||||
|
||||
echo ""
|
||||
echo "============================================================"
|
||||
echo "Fix complete! Verification:"
|
||||
echo "============================================================"
|
||||
|
||||
run_sql "SELECT 'Remaining corrupted customers:' as status, COUNT(*) as count FROM customers WHERE name REGEXP '[^ -~]';"
|
||||
run_sql "SELECT 'Remaining corrupted principles:' as status, COUNT(*) as count FROM principles WHERE name REGEXP '[^ -~]' OR address REGEXP '[^ -~]';"
|
||||
run_sql "SELECT 'Remaining corrupted products:' as status, COUNT(*) as count FROM products WHERE title REGEXP '[^ -~]' OR description REGEXP '[^ -~]';"
|
||||
|
||||
echo ""
|
||||
echo "Done! Check specific records:"
|
||||
echo "SELECT id, name FROM customers WHERE id IN (253, 1006, 1387);"
|
||||
echo "SELECT id, name, city FROM principles WHERE id IN (2, 9, 13);"
|
||||
echo "SELECT id, title FROM products WHERE id IN (30, 38, 67, 85);"
|
||||
|
|
@ -1,291 +0,0 @@
|
|||
-- ============================================================================
|
||||
-- Fix Corrupted Character Data in PRODUCTS Table
|
||||
-- ============================================================================
|
||||
-- This script fixes mojibake in the products table (12,536+ affected records)
|
||||
-- Also fixes HTML entities that shouldn't be in the raw data
|
||||
--
|
||||
-- IMPORTANT: Review this script and test on a backup first!
|
||||
-- To run: docker exec -i cmc-db mariadb -u root -psecureRootPassword --default-character-set=utf8mb4 cmc < scripts/fix_products_corruption.sql
|
||||
-- ============================================================================
|
||||
|
||||
SET NAMES utf8mb4;
|
||||
|
||||
SELECT '============================================================' as '';
|
||||
SELECT 'PRODUCTS TABLE - CHARACTER CORRUPTION FIX' as '';
|
||||
SELECT '============================================================' as '';
|
||||
|
||||
-- Count before fix
|
||||
SELECT 'BEFORE FIX - Corrupted records count:' as '';
|
||||
SELECT COUNT(*) as corrupted_products
|
||||
FROM products
|
||||
WHERE title REGEXP '[^ -~]'
|
||||
OR description REGEXP '[^ -~]'
|
||||
OR item_description REGEXP '[^ -~]'
|
||||
OR notes REGEXP '[^ -~]';
|
||||
|
||||
-- ============================================================================
|
||||
-- PART 1: Fix Mojibake Patterns (UTF-8 corruption)
|
||||
-- ============================================================================
|
||||
|
||||
-- Degree symbol: ° → °
|
||||
UPDATE products SET title = REPLACE(title, '°', '°') WHERE title LIKE '%°%';
|
||||
UPDATE products SET description = REPLACE(description, '°', '°') WHERE description LIKE '%°%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, '°', '°') WHERE item_description LIKE '%°%';
|
||||
UPDATE products SET notes = REPLACE(notes, '°', '°') WHERE notes LIKE '%°%';
|
||||
|
||||
-- Registered trademark: ® → ®
|
||||
UPDATE products SET title = REPLACE(title, '®', '®') WHERE title LIKE '%®%';
|
||||
UPDATE products SET description = REPLACE(description, '®', '®') WHERE description LIKE '%®%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, '®', '®') WHERE item_description LIKE '%®%';
|
||||
UPDATE products SET notes = REPLACE(notes, '®', '®') WHERE notes LIKE '%®%';
|
||||
|
||||
-- Trademark: â„¢ → ™
|
||||
UPDATE products SET title = REPLACE(title, 'â„¢', '™') WHERE title LIKE '%â„¢%';
|
||||
UPDATE products SET description = REPLACE(description, 'â„¢', '™') WHERE description LIKE '%â„¢%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, 'â„¢', '™') WHERE item_description LIKE '%â„¢%';
|
||||
UPDATE products SET notes = REPLACE(notes, 'â„¢', '™') WHERE notes LIKE '%â„¢%';
|
||||
|
||||
-- Smart apostrophe: ’ → '
|
||||
UPDATE products SET title = REPLACE(title, '’', ''') WHERE title LIKE '%’%';
|
||||
UPDATE products SET description = REPLACE(description, '’', ''') WHERE description LIKE '%’%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, '’', ''') WHERE item_description LIKE '%’%';
|
||||
UPDATE products SET notes = REPLACE(notes, '’', ''') WHERE notes LIKE '%’%';
|
||||
|
||||
-- Smart left double quote: “ → "
|
||||
UPDATE products SET title = REPLACE(title, '“', '"') WHERE title LIKE '%“%';
|
||||
UPDATE products SET description = REPLACE(description, '“', '"') WHERE description LIKE '%“%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, '“', '"') WHERE item_description LIKE '%“%';
|
||||
UPDATE products SET notes = REPLACE(notes, '“', '"') WHERE notes LIKE '%“%';
|
||||
|
||||
-- Smart right double quote: †→ "
|
||||
UPDATE products SET title = REPLACE(title, 'â€', '"') WHERE title LIKE '%â€%';
|
||||
UPDATE products SET description = REPLACE(description, 'â€', '"') WHERE description LIKE '%â€%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, 'â€', '"') WHERE item_description LIKE '%â€%';
|
||||
UPDATE products SET notes = REPLACE(notes, 'â€', '"') WHERE notes LIKE '%â€%';
|
||||
|
||||
-- En dash: â€" → –
|
||||
UPDATE products SET title = REPLACE(title, 'â€"', '–') WHERE title LIKE '%â€"%';
|
||||
UPDATE products SET description = REPLACE(description, 'â€"', '–') WHERE description LIKE '%â€"%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, 'â€"', '–') WHERE item_description LIKE '%â€"%';
|
||||
UPDATE products SET notes = REPLACE(notes, 'â€"', '–') WHERE notes LIKE '%â€"%';
|
||||
|
||||
-- Em dash: â€" → —
|
||||
UPDATE products SET title = REPLACE(title, 'â€"', '—') WHERE title LIKE '%â€"%';
|
||||
UPDATE products SET description = REPLACE(description, 'â€"', '—') WHERE description LIKE '%â€"%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, 'â€"', '—') WHERE item_description LIKE '%â€"%';
|
||||
UPDATE products SET notes = REPLACE(notes, 'â€"', '—') WHERE notes LIKE '%â€"%';
|
||||
|
||||
-- Ellipsis: … → …
|
||||
UPDATE products SET title = REPLACE(title, '…', '…') WHERE title LIKE '%…%';
|
||||
UPDATE products SET description = REPLACE(description, '…', '…') WHERE description LIKE '%…%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, '…', '…') WHERE item_description LIKE '%…%';
|
||||
UPDATE products SET notes = REPLACE(notes, '…', '…') WHERE notes LIKE '%…%';
|
||||
|
||||
-- Accented characters (common in product descriptions)
|
||||
-- é
|
||||
UPDATE products SET title = REPLACE(title, 'é', 'é') WHERE title LIKE '%é%';
|
||||
UPDATE products SET description = REPLACE(description, 'é', 'é') WHERE description LIKE '%é%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, 'é', 'é') WHERE item_description LIKE '%é%';
|
||||
UPDATE products SET notes = REPLACE(notes, 'é', 'é') WHERE notes LIKE '%é%';
|
||||
|
||||
-- ü
|
||||
UPDATE products SET title = REPLACE(title, 'ü', 'ü') WHERE title LIKE '%ü%';
|
||||
UPDATE products SET description = REPLACE(description, 'ü', 'ü') WHERE description LIKE '%ü%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, 'ü', 'ü') WHERE item_description LIKE '%ü%';
|
||||
UPDATE products SET notes = REPLACE(notes, 'ü', 'ü') WHERE notes LIKE '%ü%';
|
||||
|
||||
-- ö
|
||||
UPDATE products SET title = REPLACE(title, 'ö', 'ö') WHERE title LIKE '%ö%';
|
||||
UPDATE products SET description = REPLACE(description, 'ö', 'ö') WHERE description LIKE '%ö%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, 'ö', 'ö') WHERE item_description LIKE '%ö%';
|
||||
UPDATE products SET notes = REPLACE(notes, 'ö', 'ö') WHERE notes LIKE '%ö%';
|
||||
|
||||
-- ä
|
||||
UPDATE products SET title = REPLACE(title, 'ä', 'ä') WHERE title LIKE '%ä%';
|
||||
UPDATE products SET description = REPLACE(description, 'ä', 'ä') WHERE description LIKE '%ä%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, 'ä', 'ä') WHERE item_description LIKE '%ä%';
|
||||
UPDATE products SET notes = REPLACE(notes, 'ä', 'ä') WHERE notes LIKE '%ä%';
|
||||
|
||||
-- ß (German sharp s)
|
||||
UPDATE products SET title = REPLACE(title, 'ß', 'ß') WHERE title LIKE '%ß%';
|
||||
UPDATE products SET description = REPLACE(description, 'ß', 'ß') WHERE description LIKE '%ß%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, 'ß', 'ß') WHERE item_description LIKE '%ß%';
|
||||
UPDATE products SET notes = REPLACE(notes, 'ß', 'ß') WHERE notes LIKE '%ß%';
|
||||
|
||||
-- ============================================================================
|
||||
-- PART 2: Fix HTML Entities (shouldn't be in raw database data)
|
||||
-- ============================================================================
|
||||
-- Note: These are likely from copy-paste from HTML or rich text editors
|
||||
|
||||
-- – → –
|
||||
UPDATE products SET title = REPLACE(title, '–', '–') WHERE title LIKE '%–%';
|
||||
UPDATE products SET description = REPLACE(description, '–', '–') WHERE description LIKE '%–%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, '–', '–') WHERE item_description LIKE '%–%';
|
||||
UPDATE products SET notes = REPLACE(notes, '–', '–') WHERE notes LIKE '%–%';
|
||||
|
||||
-- — → —
|
||||
UPDATE products SET title = REPLACE(title, '—', '—') WHERE title LIKE '%—%';
|
||||
UPDATE products SET description = REPLACE(description, '—', '—') WHERE description LIKE '%—%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, '—', '—') WHERE item_description LIKE '%—%';
|
||||
UPDATE products SET notes = REPLACE(notes, '—', '—') WHERE notes LIKE '%—%';
|
||||
|
||||
-- ” → "
|
||||
UPDATE products SET title = REPLACE(title, '”', '"') WHERE title LIKE '%”%';
|
||||
UPDATE products SET description = REPLACE(description, '”', '"') WHERE description LIKE '%”%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, '”', '"') WHERE item_description LIKE '%”%';
|
||||
UPDATE products SET notes = REPLACE(notes, '”', '"') WHERE notes LIKE '%”%';
|
||||
|
||||
-- “ → "
|
||||
UPDATE products SET title = REPLACE(title, '“', '"') WHERE title LIKE '%“%';
|
||||
UPDATE products SET description = REPLACE(description, '“', '"') WHERE description LIKE '%“%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, '“', '"') WHERE item_description LIKE '%“%';
|
||||
UPDATE products SET notes = REPLACE(notes, '“', '"') WHERE notes LIKE '%“%';
|
||||
|
||||
-- ’ → '
|
||||
UPDATE products SET title = REPLACE(title, '’', ''') WHERE title LIKE '%’%';
|
||||
UPDATE products SET description = REPLACE(description, '’', ''') WHERE description LIKE '%’%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, '’', ''') WHERE item_description LIKE '%’%';
|
||||
UPDATE products SET notes = REPLACE(notes, '’', ''') WHERE notes LIKE '%’%';
|
||||
|
||||
-- ‘ → '
|
||||
UPDATE products SET title = REPLACE(title, '‘', ''') WHERE title LIKE '%‘%';
|
||||
UPDATE products SET description = REPLACE(description, '‘', ''') WHERE description LIKE '%‘%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, '‘', ''') WHERE item_description LIKE '%‘%';
|
||||
UPDATE products SET notes = REPLACE(notes, '‘', ''') WHERE notes LIKE '%‘%';
|
||||
|
||||
-- " → " (keep as is, but decode entity)
|
||||
UPDATE products SET title = REPLACE(title, '"', '"') WHERE title LIKE '%"%';
|
||||
UPDATE products SET description = REPLACE(description, '"', '"') WHERE description LIKE '%"%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, '"', '"') WHERE item_description LIKE '%"%';
|
||||
UPDATE products SET notes = REPLACE(notes, '"', '"') WHERE notes LIKE '%"%';
|
||||
|
||||
-- & → & (ampersand)
|
||||
UPDATE products SET title = REPLACE(title, '&', '&') WHERE title LIKE '%&%';
|
||||
UPDATE products SET description = REPLACE(description, '&', '&') WHERE description LIKE '%&%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, '&', '&') WHERE item_description LIKE '%&%';
|
||||
UPDATE products SET notes = REPLACE(notes, '&', '&') WHERE notes LIKE '%&%';
|
||||
|
||||
-- Common fractions
|
||||
-- ½ → ½
|
||||
UPDATE products SET title = REPLACE(title, '½', '½') WHERE title LIKE '%½%';
|
||||
UPDATE products SET description = REPLACE(description, '½', '½') WHERE description LIKE '%½%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, '½', '½') WHERE item_description LIKE '%½%';
|
||||
UPDATE products SET notes = REPLACE(notes, '½', '½') WHERE notes LIKE '%½%';
|
||||
|
||||
-- ¼ → ¼
|
||||
UPDATE products SET title = REPLACE(title, '¼', '¼') WHERE title LIKE '%¼%';
|
||||
UPDATE products SET description = REPLACE(description, '¼', '¼') WHERE description LIKE '%¼%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, '¼', '¼') WHERE item_description LIKE '%¼%';
|
||||
UPDATE products SET notes = REPLACE(notes, '¼', '¼') WHERE notes LIKE '%¼%';
|
||||
|
||||
-- ¾ → ¾
|
||||
UPDATE products SET title = REPLACE(title, '¾', '¾') WHERE title LIKE '%¾%';
|
||||
UPDATE products SET description = REPLACE(description, '¾', '¾') WHERE description LIKE '%¾%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, '¾', '¾') WHERE item_description LIKE '%¾%';
|
||||
UPDATE products SET notes = REPLACE(notes, '¾', '¾') WHERE notes LIKE '%¾%';
|
||||
|
||||
-- ° → °
|
||||
UPDATE products SET title = REPLACE(title, '°', '°') WHERE title LIKE '%°%';
|
||||
UPDATE products SET description = REPLACE(description, '°', '°') WHERE description LIKE '%°%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, '°', '°') WHERE item_description LIKE '%°%';
|
||||
UPDATE products SET notes = REPLACE(notes, '°', '°') WHERE notes LIKE '%°%';
|
||||
|
||||
-- × → ×
|
||||
UPDATE products SET title = REPLACE(title, '×', '×') WHERE title LIKE '%×%';
|
||||
UPDATE products SET description = REPLACE(description, '×', '×') WHERE description LIKE '%×%';
|
||||
UPDATE products SET item_description = REPLACE(item_description, '×', '×') WHERE item_description LIKE '%×%';
|
||||
UPDATE products SET notes = REPLACE(notes, '×', '×') WHERE notes LIKE '%×%';
|
||||
|
||||
-- → (space) - non-breaking space
|
||||
UPDATE products SET title = REPLACE(title, ' ', ' ') WHERE title LIKE '% %';
|
||||
UPDATE products SET description = REPLACE(description, ' ', ' ') WHERE description LIKE '% %';
|
||||
UPDATE products SET item_description = REPLACE(item_description, ' ', ' ') WHERE item_description LIKE '% %';
|
||||
UPDATE products SET notes = REPLACE(notes, ' ', ' ') WHERE notes LIKE '% %';
|
||||
|
||||
-- ============================================================================
|
||||
-- PART 3: Specific Known Corrupted Records
|
||||
-- ============================================================================
|
||||
|
||||
-- ID 30: Degree symbol in title
|
||||
UPDATE products
|
||||
SET title = 'C95SN189C DSTGL40/C-Digital temperature probe for P8xx1 Web Sensor. Range -30 to +80°C. With CINCH connector, 1m Cable'
|
||||
WHERE id = 30;
|
||||
|
||||
-- ID 38: Em dashes in title
|
||||
UPDATE products
|
||||
SET title = 'Mid-West Instrument Model 240 – SC – 02 – O(TT)'
|
||||
WHERE id = 38;
|
||||
|
||||
-- ID 67: Registered trademark
|
||||
UPDATE products
|
||||
SET title = 'Newson Gale Earth-Rite® II FIBC Static Earthing System, GRP Enclosure with 5m 2 Core Spiral Cable and FIBC Clamp'
|
||||
WHERE id = 67;
|
||||
|
||||
-- ID 76: Registered trademark
|
||||
UPDATE products
|
||||
SET title = 'Newson Gale Earth-Rite® RTR Tester - ER2/CRT'
|
||||
WHERE id = 76;
|
||||
|
||||
-- ID 77: Registered trademark
|
||||
UPDATE products
|
||||
SET title = 'Newson Gale Earth-Rite® Installers Kit A for IIB areas only (Metal Enclosures Only)'
|
||||
WHERE id = 77;
|
||||
|
||||
-- ID 85: Registered trademark and trademark
|
||||
UPDATE products
|
||||
SET title = 'Newson Gale Earth-Rite® RTR™ Tri-Mode Static Grounding System, Metal Enclosure, X90-IP Heavy Duty Clamp with 10m 2 Core Spiral Cable'
|
||||
WHERE id = 85;
|
||||
|
||||
-- ID 86: Registered trademark
|
||||
UPDATE products
|
||||
SET title = 'Newson Gale Earth-Rite® RTR System Spares - X90-IP Heavy Duty 2 Core Clamp with 10m 2 Core Spiral Cable and Male Quick Connect'
|
||||
WHERE id = 86;
|
||||
|
||||
-- ID 149: Registered trademark
|
||||
UPDATE products
|
||||
SET title = 'Newson Gale Bond-Rite® EZ, VESX90-IP Heavy Duty Clamp & 3m 2 Core Spiral Cable'
|
||||
WHERE id = 149;
|
||||
|
||||
-- ID 151: Registered trademark
|
||||
UPDATE products
|
||||
SET title = 'UNSURE Newson Gale Bond-Rite® Clamp with 20ft (6.1m) Cable Reel'
|
||||
WHERE id = 151;
|
||||
|
||||
-- ID 155: Registered trademark
|
||||
UPDATE products
|
||||
SET title = 'Newson Gale Earth-Rite® MGV Tri-Mode Static Grounding System, GRP Enclosure, X90-IP Heavy Duty Clamp with 15m 2 Core Spiral Cable'
|
||||
WHERE id = 155;
|
||||
|
||||
-- ID 220: Registered trademark
|
||||
UPDATE products
|
||||
SET title = 'Newson Gale Earth-Rite® TELLUS II Static Earthing System, X90-IP Heavy Duty Clamp with 10m 2 Core Spiral Cable'
|
||||
WHERE id = 220;
|
||||
|
||||
-- ID 249: Double registered trademarks
|
||||
UPDATE products
|
||||
SET title = 'Newson Gale Bond-Rite® Remote GRP Multi-Way Junction Box for 2 to 4 Bond-Rite® Remote EP Units'
|
||||
WHERE id = 249;
|
||||
|
||||
-- ============================================================================
|
||||
-- PART 4: Verification
|
||||
-- ============================================================================
|
||||
|
||||
SELECT '' as '';
|
||||
SELECT '============================================================' as '';
|
||||
SELECT 'AFTER FIX - Corrupted records count:' as '';
|
||||
SELECT COUNT(*) as remaining_corrupted_products
|
||||
FROM products
|
||||
WHERE title REGEXP '[^ -~]'
|
||||
OR description REGEXP '[^ -~]'
|
||||
OR item_description REGEXP '[^ -~]'
|
||||
OR notes REGEXP '[^ -~]';
|
||||
|
||||
-- Show sample fixed records
|
||||
SELECT '' as '';
|
||||
SELECT 'Sample fixed records:' as '';
|
||||
SELECT id, title FROM products WHERE id IN (30, 38, 67, 76, 85, 149);
|
||||
|
||||
SELECT '' as '';
|
||||
SELECT '============================================================' as '';
|
||||
SELECT 'PRODUCTS TABLE FIX COMPLETE' as '';
|
||||
SELECT '============================================================' as '';
|
||||
|
|
@ -1,15 +0,0 @@
|
|||
#!/bin/bash
|
||||
# ============================================================================
|
||||
# Generate Report of Remaining Corrupted Data
|
||||
# ============================================================================
|
||||
|
||||
DB_USER="root"
|
||||
DB_PASS="secureRootPassword"
|
||||
DB_NAME="cmc"
|
||||
CONTAINER="cmc-db"
|
||||
|
||||
docker exec $CONTAINER mariadb \
|
||||
-u $DB_USER \
|
||||
-p$DB_PASS \
|
||||
--default-character-set=utf8mb4 \
|
||||
$DB_NAME < scripts/report_remaining_corruption.sql
|
||||
|
|
@ -1,191 +0,0 @@
|
|||
-- ============================================================================
|
||||
-- Report Remaining Corrupted Data After Fix
|
||||
-- ============================================================================
|
||||
-- Run this after fix_corrupted_data.sql to find records that still need manual fixes
|
||||
--
|
||||
-- To run: docker exec cmc-db mariadb -u root -psecureRootPassword --default-character-set=utf8mb4 cmc < scripts/report_remaining_corruption.sql
|
||||
-- ============================================================================
|
||||
|
||||
SET NAMES utf8mb4;
|
||||
|
||||
SELECT '============================================================' as '';
|
||||
SELECT 'CORRUPTED DATA REPORT' as '';
|
||||
SELECT '============================================================' as '';
|
||||
|
||||
-- Summary counts
|
||||
SELECT '' as '';
|
||||
SELECT 'SUMMARY OF REMAINING CORRUPTION' as '';
|
||||
SELECT '-------------------------------' as '';
|
||||
|
||||
SELECT
|
||||
'principles' as table_name,
|
||||
COUNT(*) as corrupted_records,
|
||||
'name, address, city' as affected_columns
|
||||
FROM principles
|
||||
WHERE name REGEXP '[^ -~]'
|
||||
OR address REGEXP '[^ -~]'
|
||||
OR city REGEXP '[^ -~]'
|
||||
|
||||
UNION ALL
|
||||
|
||||
SELECT
|
||||
'customers',
|
||||
COUNT(*),
|
||||
'name'
|
||||
FROM customers
|
||||
WHERE name REGEXP '[^ -~]'
|
||||
|
||||
UNION ALL
|
||||
|
||||
SELECT
|
||||
'addresses',
|
||||
COUNT(*),
|
||||
'address, city'
|
||||
FROM addresses
|
||||
WHERE address REGEXP '[^ -~]'
|
||||
OR city REGEXP '[^ -~]'
|
||||
|
||||
UNION ALL
|
||||
|
||||
SELECT
|
||||
'contacts',
|
||||
COUNT(*),
|
||||
'name'
|
||||
FROM contacts
|
||||
WHERE name REGEXP '[^ -~]'
|
||||
|
||||
UNION ALL
|
||||
|
||||
SELECT
|
||||
'products',
|
||||
COUNT(*),
|
||||
'name, description'
|
||||
FROM products
|
||||
WHERE name REGEXP '[^ -~]'
|
||||
OR description REGEXP '[^ -~]';
|
||||
|
||||
-- Detailed listings
|
||||
SELECT '' as '';
|
||||
SELECT '============================================================' as '';
|
||||
SELECT 'DETAILED CORRUPTED RECORDS' as '';
|
||||
SELECT '============================================================' as '';
|
||||
|
||||
-- Principles with corruption
|
||||
SELECT '' as '';
|
||||
SELECT 'PRINCIPLES TABLE - Corrupted Records:' as '';
|
||||
SELECT '-------------------------------------' as '';
|
||||
SELECT
|
||||
id,
|
||||
name,
|
||||
LEFT(address, 50) as address_preview,
|
||||
city
|
||||
FROM principles
|
||||
WHERE name REGEXP '[^ -~]'
|
||||
OR address REGEXP '[^ -~]'
|
||||
OR city REGEXP '[^ -~]'
|
||||
ORDER BY id;
|
||||
|
||||
-- Customers with corruption
|
||||
SELECT '' as '';
|
||||
SELECT 'CUSTOMERS TABLE - Corrupted Records:' as '';
|
||||
SELECT '-------------------------------------' as '';
|
||||
SELECT
|
||||
id,
|
||||
name
|
||||
FROM customers
|
||||
WHERE name REGEXP '[^ -~]'
|
||||
ORDER BY id;
|
||||
|
||||
-- Addresses with corruption
|
||||
SELECT '' as '';
|
||||
SELECT 'ADDRESSES TABLE - Corrupted Records:' as '';
|
||||
SELECT '-------------------------------------' as '';
|
||||
SELECT
|
||||
id,
|
||||
LEFT(address, 60) as address,
|
||||
city
|
||||
FROM addresses
|
||||
WHERE address REGEXP '[^ -~]'
|
||||
OR city REGEXP '[^ -~]'
|
||||
ORDER BY id
|
||||
LIMIT 50;
|
||||
|
||||
-- Contacts with corruption
|
||||
SELECT '' as '';
|
||||
SELECT 'CONTACTS TABLE - Corrupted Records:' as '';
|
||||
SELECT '-------------------------------------' as '';
|
||||
SELECT
|
||||
id,
|
||||
name
|
||||
FROM contacts
|
||||
WHERE name REGEXP '[^ -~]'
|
||||
ORDER BY id
|
||||
LIMIT 50;
|
||||
|
||||
-- Products with corruption
|
||||
SELECT '' as '';
|
||||
SELECT 'PRODUCTS TABLE - Corrupted Records:' as '';
|
||||
SELECT '-------------------------------------' as '';
|
||||
SELECT
|
||||
id,
|
||||
name,
|
||||
LEFT(description, 50) as description_preview
|
||||
FROM products
|
||||
WHERE name REGEXP '[^ -~]'
|
||||
OR description REGEXP '[^ -~]'
|
||||
ORDER BY id
|
||||
LIMIT 50;
|
||||
|
||||
-- Common corruption patterns found
|
||||
SELECT '' as '';
|
||||
SELECT '============================================================' as '';
|
||||
SELECT 'COMMON CORRUPTION PATTERNS DETECTED' as '';
|
||||
SELECT '============================================================' as '';
|
||||
|
||||
SELECT
|
||||
'’' as mojibake_pattern,
|
||||
'Smart apostrophe (')' as should_be,
|
||||
COUNT(*) as occurrences_in_customers
|
||||
FROM customers
|
||||
WHERE name LIKE '%’%'
|
||||
|
||||
UNION ALL
|
||||
|
||||
SELECT
|
||||
'“ or â€',
|
||||
'Smart quotes (" ")',
|
||||
COUNT(*)
|
||||
FROM customers
|
||||
WHERE name LIKE '%“%' OR name LIKE '%â€%'
|
||||
|
||||
UNION ALL
|
||||
|
||||
SELECT
|
||||
'é, ü, ó, etc',
|
||||
'Accented chars (é, ü, ó)',
|
||||
COUNT(*)
|
||||
FROM customers
|
||||
WHERE name REGEXP 'Ã[©¼³±§]'
|
||||
|
||||
UNION ALL
|
||||
|
||||
SELECT
|
||||
'â€" or â€"',
|
||||
'Dashes (– —)',
|
||||
COUNT(*)
|
||||
FROM addresses
|
||||
WHERE address LIKE '%â€"%' OR address LIKE '%â€"%'
|
||||
|
||||
UNION ALL
|
||||
|
||||
SELECT
|
||||
'Ä, ž, Å¡, etc',
|
||||
'Czech/Polish chars',
|
||||
COUNT(*)
|
||||
FROM principles
|
||||
WHERE address REGEXP '[ÄÅ][¾¡›™]';
|
||||
|
||||
SELECT '' as '';
|
||||
SELECT '============================================================' as '';
|
||||
SELECT 'END OF REPORT' as '';
|
||||
SELECT '============================================================' as '';
|
||||
|
|
@ -1,115 +0,0 @@
|
|||
#!/bin/bash
|
||||
# ============================================================================
|
||||
# Comprehensive Corruption Fix - All Tables
|
||||
# ============================================================================
|
||||
# This script:
|
||||
# 1. Creates a backup
|
||||
# 2. Scans all tables for corruption
|
||||
# 3. Prompts for confirmation
|
||||
# 4. Applies fixes to ALL tables
|
||||
# 5. Shows before/after comparison
|
||||
# ============================================================================
|
||||
|
||||
set -e # Exit on error
|
||||
|
||||
DB_USER="root"
|
||||
DB_PASS="secureRootPassword"
|
||||
DB_NAME="cmc"
|
||||
CONTAINER="cmc-db"
|
||||
BACKUP_DIR="./backups"
|
||||
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
|
||||
|
||||
# Colors for output
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
BLUE='\033[0;34m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
echo -e "${BLUE}============================================================${NC}"
|
||||
echo -e "${BLUE}CMC Database - Comprehensive Corruption Fix${NC}"
|
||||
echo -e "${BLUE}============================================================${NC}"
|
||||
echo ""
|
||||
|
||||
# Create backups directory
|
||||
mkdir -p "$BACKUP_DIR"
|
||||
|
||||
# Step 1: Create backup
|
||||
echo -e "${YELLOW}Step 1: Creating full database backup...${NC}"
|
||||
docker exec $CONTAINER mariadb-dump \
|
||||
-u $DB_USER \
|
||||
-p$DB_PASS \
|
||||
--default-character-set=utf8mb4 \
|
||||
--single-transaction \
|
||||
$DB_NAME | gzip > "$BACKUP_DIR/backup_comprehensive_fix_$TIMESTAMP.sql.gz"
|
||||
|
||||
BACKUP_SIZE=$(du -h "$BACKUP_DIR/backup_comprehensive_fix_$TIMESTAMP.sql.gz" | cut -f1)
|
||||
echo -e "${GREEN}✓ Backup created: backup_comprehensive_fix_$TIMESTAMP.sql.gz (${BACKUP_SIZE})${NC}"
|
||||
echo ""
|
||||
|
||||
# Step 2: Scan all tables
|
||||
echo -e "${YELLOW}Step 2: Scanning ALL tables for corruption...${NC}"
|
||||
echo "-------------------------------------------"
|
||||
docker exec $CONTAINER mariadb \
|
||||
-u $DB_USER \
|
||||
-p$DB_PASS \
|
||||
--default-character-set=utf8mb4 \
|
||||
$DB_NAME < scripts/scan_all_tables_for_corruption.sql | head -100
|
||||
echo ""
|
||||
|
||||
# Step 3: Prompt for confirmation
|
||||
echo -e "${RED}IMPORTANT:${NC}"
|
||||
echo "- Backup created at: $BACKUP_DIR/backup_comprehensive_fix_$TIMESTAMP.sql.gz"
|
||||
echo "- This will modify ALL text columns in ALL tables"
|
||||
echo "- Fixes mojibake patterns like: ’ → ', é → é, ° → °, etc."
|
||||
echo "- Tables affected: customers, principles, addresses, products, contacts,"
|
||||
echo " emails, invoices, jobs, quotes, purchase_orders, shipments, etc."
|
||||
echo ""
|
||||
read -p "Do you want to proceed with the comprehensive fix? (yes/no): " CONFIRM
|
||||
|
||||
if [ "$CONFIRM" != "yes" ]; then
|
||||
echo -e "${RED}Fix cancelled by user${NC}"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Step 4: Apply comprehensive fixes
|
||||
echo ""
|
||||
echo -e "${YELLOW}Step 4: Applying comprehensive corruption fixes...${NC}"
|
||||
docker exec -i $CONTAINER mariadb \
|
||||
-u $DB_USER \
|
||||
-p$DB_PASS \
|
||||
--default-character-set=utf8mb4 \
|
||||
$DB_NAME < scripts/fix_all_corruption_safe.sql
|
||||
|
||||
echo -e "${GREEN}✓ Comprehensive fix executed${NC}"
|
||||
echo ""
|
||||
|
||||
# Step 5: Re-scan to show results
|
||||
echo -e "${YELLOW}Step 5: Re-scanning to verify fixes...${NC}"
|
||||
echo "---------------------------------------"
|
||||
docker exec $CONTAINER mariadb \
|
||||
-u $DB_USER \
|
||||
-p$DB_PASS \
|
||||
--default-character-set=utf8mb4 \
|
||||
$DB_NAME < scripts/scan_all_tables_for_corruption.sql | head -100
|
||||
echo ""
|
||||
|
||||
# Final summary
|
||||
echo -e "${BLUE}============================================================${NC}"
|
||||
echo -e "${GREEN}COMPREHENSIVE FIX COMPLETED!${NC}"
|
||||
echo -e "${BLUE}============================================================${NC}"
|
||||
echo ""
|
||||
echo "Summary:"
|
||||
echo "- Backup location: $BACKUP_DIR/backup_comprehensive_fix_$TIMESTAMP.sql.gz"
|
||||
echo "- Fixed corruption patterns across ALL tables"
|
||||
echo "- Common patterns: smart quotes, accented characters, symbols, HTML entities"
|
||||
echo ""
|
||||
echo "Next steps:"
|
||||
echo "1. Review the fixed data in the application"
|
||||
echo "2. Test critical functionality (quotes, invoices, emails, etc.)"
|
||||
echo "3. Run: bash scripts/report_remaining_corruption.sh if any issues remain"
|
||||
echo ""
|
||||
echo "To rollback if needed:"
|
||||
echo " gunzip < $BACKUP_DIR/backup_comprehensive_fix_$TIMESTAMP.sql.gz | \\"
|
||||
echo " docker exec -i $CONTAINER mariadb -u $DB_USER -p$DB_PASS $DB_NAME"
|
||||
echo ""
|
||||
|
|
@ -1,187 +0,0 @@
|
|||
#!/bin/bash
|
||||
# ============================================================================
|
||||
# Safe Corruption Fix Execution Script
|
||||
# ============================================================================
|
||||
# This script:
|
||||
# 1. Creates a backup
|
||||
# 2. Shows current corruption status
|
||||
# 3. Prompts for confirmation
|
||||
# 4. Applies fixes
|
||||
# 5. Shows before/after comparison
|
||||
# ============================================================================
|
||||
|
||||
set -e # Exit on error
|
||||
|
||||
DB_USER="root"
|
||||
DB_PASS="secureRootPassword"
|
||||
DB_NAME="cmc"
|
||||
CONTAINER="cmc-db"
|
||||
BACKUP_DIR="./backups"
|
||||
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
|
||||
|
||||
# Colors for output
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
BLUE='\033[0;34m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
echo -e "${BLUE}============================================================${NC}"
|
||||
echo -e "${BLUE}CMC Database Corruption Fix Script${NC}"
|
||||
echo -e "${BLUE}============================================================${NC}"
|
||||
echo ""
|
||||
|
||||
# Create backups directory
|
||||
mkdir -p "$BACKUP_DIR"
|
||||
|
||||
# Step 1: Create backup
|
||||
echo -e "${YELLOW}Step 1: Creating database backup...${NC}"
|
||||
docker exec $CONTAINER mariadb-dump \
|
||||
-u $DB_USER \
|
||||
-p$DB_PASS \
|
||||
--default-character-set=utf8mb4 \
|
||||
--single-transaction \
|
||||
$DB_NAME | gzip > "$BACKUP_DIR/backup_before_corruption_fix_$TIMESTAMP.sql.gz"
|
||||
|
||||
BACKUP_SIZE=$(du -h "$BACKUP_DIR/backup_before_corruption_fix_$TIMESTAMP.sql.gz" | cut -f1)
|
||||
echo -e "${GREEN}✓ Backup created: backup_before_corruption_fix_$TIMESTAMP.sql.gz (${BACKUP_SIZE})${NC}"
|
||||
echo ""
|
||||
|
||||
# Step 2: Show current corruption status
|
||||
echo -e "${YELLOW}Step 2: Current corruption status${NC}"
|
||||
echo "-----------------------------------"
|
||||
docker exec $CONTAINER mariadb -u $DB_USER -p$DB_PASS --default-character-set=utf8mb4 $DB_NAME -e "
|
||||
SELECT
|
||||
'principles' as table_name,
|
||||
COUNT(*) as corrupted_records
|
||||
FROM principles
|
||||
WHERE name REGEXP '[^ -~]'
|
||||
OR address REGEXP '[^ -~]'
|
||||
OR city REGEXP '[^ -~]'
|
||||
UNION ALL
|
||||
SELECT 'customers', COUNT(*)
|
||||
FROM customers
|
||||
WHERE name REGEXP '[^ -~]'
|
||||
UNION ALL
|
||||
SELECT 'addresses', COUNT(*)
|
||||
FROM addresses
|
||||
WHERE address REGEXP '[^ -~]'
|
||||
OR city REGEXP '[^ -~]'
|
||||
UNION ALL
|
||||
SELECT 'products', COUNT(*)
|
||||
FROM products
|
||||
WHERE title REGEXP '[^ -~]'
|
||||
OR description REGEXP '[^ -~]'
|
||||
OR item_description REGEXP '[^ -~]';
|
||||
"
|
||||
echo ""
|
||||
|
||||
# Step 3: Show sample corrupted data
|
||||
echo -e "${YELLOW}Step 3: Sample corrupted data (BEFORE fix)${NC}"
|
||||
echo "-------------------------------------------"
|
||||
docker exec $CONTAINER mariadb -u $DB_USER -p$DB_PASS --default-character-set=utf8mb4 $DB_NAME -e "
|
||||
SELECT 'CUSTOMERS' as table_name, id, name FROM customers WHERE id IN (253, 1006, 1387, 1608)
|
||||
UNION ALL
|
||||
SELECT 'PRINCIPLES', id, name FROM principles WHERE id IN (2, 9, 13, 14)
|
||||
UNION ALL
|
||||
SELECT 'PRODUCTS', id, title FROM products WHERE id IN (30, 38, 67);
|
||||
"
|
||||
echo ""
|
||||
|
||||
# Step 4: Prompt for confirmation
|
||||
echo -e "${RED}IMPORTANT:${NC}"
|
||||
echo "- Backup created at: $BACKUP_DIR/backup_before_corruption_fix_$TIMESTAMP.sql.gz"
|
||||
echo "- This will modify data in tables: principles, customers, addresses, products"
|
||||
echo "- Changes fix mojibake like: ’ → ', é → é, etc."
|
||||
echo ""
|
||||
read -p "Do you want to proceed with the fix? (yes/no): " CONFIRM
|
||||
|
||||
if [ "$CONFIRM" != "yes" ]; then
|
||||
echo -e "${RED}Fix cancelled by user${NC}"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Step 5: Apply fixes
|
||||
echo ""
|
||||
echo -e "${YELLOW}Step 5: Applying corruption fixes...${NC}"
|
||||
docker exec -i $CONTAINER mariadb \
|
||||
-u $DB_USER \
|
||||
-p$DB_PASS \
|
||||
--default-character-set=utf8mb4 \
|
||||
$DB_NAME < scripts/fix_corrupted_data.sql
|
||||
|
||||
echo -e "${GREEN}✓ Fix script executed${NC}"
|
||||
echo ""
|
||||
|
||||
# Step 6: Show results
|
||||
echo -e "${YELLOW}Step 6: Results (AFTER fix)${NC}"
|
||||
echo "----------------------------"
|
||||
echo ""
|
||||
echo "Sample fixed data:"
|
||||
docker exec $CONTAINER mariadb -u $DB_USER -p$DB_PASS --default-character-set=utf8mb4 $DB_NAME -e "
|
||||
SELECT 'CUSTOMERS' as table_name, id, name FROM customers WHERE id IN (253, 1006, 1387, 1608)
|
||||
UNION ALL
|
||||
SELECT 'PRINCIPLES', id, name FROM principles WHERE id IN (2, 9, 13, 14)
|
||||
UNION ALL
|
||||
SELECT 'PRODUCTS', id, title FROM products WHERE id IN (30, 38, 67);
|
||||
"
|
||||
echo ""
|
||||
|
||||
echo "Remaining corruption counts:"
|
||||
docker exec $CONTAINER mariadb -u $DB_USER -p$DB_PASS --default-character-set=utf8mb4 $DB_NAME -e "
|
||||
SELECT
|
||||
'principles' as table_name,
|
||||
COUNT(*) as corrupted_records
|
||||
FROM principles
|
||||
WHERE name REGEXP '[^ -~]'
|
||||
OR address REGEXP '[^ -~]'
|
||||
OR city REGEXP '[^ -~]'
|
||||
UNION ALL
|
||||
SELECT 'customers', COUNT(*)
|
||||
FROM customers
|
||||
WHERE name REGEXP '[^ -~]'
|
||||
UNION ALL
|
||||
SELECT 'addresses', COUNT(*)
|
||||
FROM addresses
|
||||
WHERE address REGEXP '[^ -~]'
|
||||
OR city REGEXP '[^ -~]'
|
||||
UNION ALL
|
||||
SELECT 'products', COUNT(*)
|
||||
FROM products
|
||||
WHERE title REGEXP '[^ -~]'
|
||||
OR description REGEXP '[^ -~]'
|
||||
OR item_description REGEXP '[^ -~]';
|
||||
"
|
||||
echo ""
|
||||
|
||||
# Step 7: Test new insert
|
||||
echo -e "${YELLOW}Step 7: Testing new inserts with special characters...${NC}"
|
||||
docker exec $CONTAINER mariadb -u $DB_USER -p$DB_PASS --default-character-set=utf8mb4 $DB_NAME -e "
|
||||
START TRANSACTION;
|
||||
INSERT INTO principles (name, code, address, city, state, postcode, country_id, currency_id, short_name, url)
|
||||
VALUES ('Test Müller & Söhne GmbH', 999999, 'Straße 123', 'München', 'Bayern', '80331', 1, 1, 'TEST', 'http://test.com');
|
||||
SELECT name, address, city FROM principles WHERE code = 999999;
|
||||
ROLLBACK;
|
||||
"
|
||||
echo -e "${GREEN}✓ New inserts working correctly${NC}"
|
||||
echo ""
|
||||
|
||||
# Final summary
|
||||
echo -e "${BLUE}============================================================${NC}"
|
||||
echo -e "${GREEN}FIX COMPLETED SUCCESSFULLY!${NC}"
|
||||
echo -e "${BLUE}============================================================${NC}"
|
||||
echo ""
|
||||
echo "Summary:"
|
||||
echo "- Backup location: $BACKUP_DIR/backup_before_corruption_fix_$TIMESTAMP.sql.gz"
|
||||
echo "- Fixed common patterns: smart quotes, accented characters, dashes"
|
||||
echo "- Fixed specific records: ~30+ known corrupted records"
|
||||
echo ""
|
||||
echo "Next steps:"
|
||||
echo "1. Review the fixed data in the application"
|
||||
echo "2. Run: bash scripts/report_remaining_corruption.sh to see what's left"
|
||||
echo "3. Manually fix any remaining corrupted records"
|
||||
echo ""
|
||||
echo "To rollback if needed:"
|
||||
echo " gunzip < $BACKUP_DIR/backup_before_corruption_fix_$TIMESTAMP.sql.gz | \\"
|
||||
echo " docker exec -i $CONTAINER mariadb -u $DB_USER -p$DB_PASS $DB_NAME"
|
||||
echo ""
|
||||
|
|
@ -1,126 +0,0 @@
|
|||
-- ============================================================================
|
||||
-- Scan ALL Tables for Character Corruption
|
||||
-- ============================================================================
|
||||
-- This script scans every table in the database for mojibake/corruption
|
||||
-- Finds all VARCHAR and TEXT columns and checks them for non-ASCII characters
|
||||
--
|
||||
-- To run: docker exec cmc-db mariadb -u root -psecureRootPassword --default-character-set=utf8mb4 cmc < scripts/scan_all_tables_for_corruption.sql
|
||||
-- ============================================================================
|
||||
|
||||
SET NAMES utf8mb4;
|
||||
|
||||
SELECT '============================================================' as '';
|
||||
SELECT 'COMPREHENSIVE CORRUPTION SCAN - ALL TABLES' as '';
|
||||
SELECT '============================================================' as '';
|
||||
SELECT '' as '';
|
||||
|
||||
-- Scan all major tables with text content
|
||||
SELECT 'Table: addresses' as '';
|
||||
SELECT COUNT(*) as corrupted_records, 'name' as column_name FROM addresses WHERE name REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'address' FROM addresses WHERE address REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'city' FROM addresses WHERE city REGEXP '[^ -~]';
|
||||
|
||||
SELECT '' as '';
|
||||
SELECT 'Table: attachments' as '';
|
||||
SELECT COUNT(*) as corrupted_records, 'name' as column_name FROM attachments WHERE name REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'filename' FROM attachments WHERE filename REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'description' FROM attachments WHERE description REGEXP '[^ -~]';
|
||||
|
||||
SELECT '' as '';
|
||||
SELECT 'Table: contacts' as '';
|
||||
SELECT COUNT(*) as corrupted_records, 'name' as column_name FROM contacts WHERE name REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'first_name' FROM contacts WHERE first_name REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'last_name' FROM contacts WHERE last_name REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'notes' FROM contacts WHERE notes REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'job_title' FROM contacts WHERE job_title REGEXP '[^ -~]';
|
||||
|
||||
SELECT '' as '';
|
||||
SELECT 'Table: customers' as '';
|
||||
SELECT COUNT(*) as corrupted_records, 'name' as column_name FROM customers WHERE name REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'trading_name' FROM customers WHERE trading_name REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'notes' FROM customers WHERE notes REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'payment_terms' FROM customers WHERE payment_terms REGEXP '[^ -~]';
|
||||
|
||||
SELECT '' as '';
|
||||
SELECT 'Table: documents' as '';
|
||||
SELECT COUNT(*) as corrupted_records, 'cmc_reference' as column_name FROM documents WHERE cmc_reference REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'shipping_details' FROM documents WHERE shipping_details REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'bill_to' FROM documents WHERE bill_to REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'ship_to' FROM documents WHERE ship_to REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'name' FROM documents WHERE name REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'subject' FROM documents WHERE subject REGEXP '[^ -~]';
|
||||
|
||||
SELECT '' as '';
|
||||
SELECT 'Table: emails' as '';
|
||||
SELECT COUNT(*) as corrupted_records, 'from' as column_name FROM emails WHERE `from` REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'to' FROM emails WHERE `to` REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'subject' FROM emails WHERE subject REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'body' FROM emails WHERE body REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'bcc' FROM emails WHERE bcc REGEXP '[^ -~]';
|
||||
|
||||
SELECT '' as '';
|
||||
SELECT 'Table: enquiries' as '';
|
||||
SELECT COUNT(*) as corrupted_records, 'title' as column_name FROM enquiries WHERE title REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'comments' FROM enquiries WHERE comments REGEXP '[^ -~]';
|
||||
|
||||
SELECT '' as '';
|
||||
SELECT 'Table: invoices' as '';
|
||||
SELECT COUNT(*) as corrupted_records, 'title' as column_name FROM invoices WHERE title REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'ship_via' FROM invoices WHERE ship_via REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'fob' FROM invoices WHERE fob REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'comments' FROM invoices WHERE comments REGEXP '[^ -~]';
|
||||
|
||||
SELECT '' as '';
|
||||
SELECT 'Table: jobs' as '';
|
||||
SELECT COUNT(*) as corrupted_records, 'title' as column_name FROM jobs WHERE title REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'comments' FROM jobs WHERE comments REGEXP '[^ -~]';
|
||||
|
||||
SELECT '' as '';
|
||||
SELECT 'Table: line_items' as '';
|
||||
SELECT COUNT(*) as corrupted_records, 'name' as column_name FROM line_items WHERE name REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'description' FROM line_items WHERE description REGEXP '[^ -~]';
|
||||
|
||||
SELECT '' as '';
|
||||
SELECT 'Table: order_acknowledgements' as '';
|
||||
SELECT COUNT(*) as corrupted_records, 'title' as column_name FROM order_acknowledgements WHERE title REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'comments' FROM order_acknowledgements WHERE comments REGEXP '[^ -~]';
|
||||
|
||||
SELECT '' as '';
|
||||
SELECT 'Table: principles' as '';
|
||||
SELECT COUNT(*) as corrupted_records, 'name' as column_name FROM principles WHERE name REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'address' FROM principles WHERE address REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'city' FROM principles WHERE city REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'short_name' FROM principles WHERE short_name REGEXP '[^ -~]';
|
||||
|
||||
SELECT '' as '';
|
||||
SELECT 'Table: products' as '';
|
||||
SELECT COUNT(*) as corrupted_records, 'name' as column_name FROM products WHERE name REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'title' FROM products WHERE title REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'description' FROM products WHERE description REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'item_description' FROM products WHERE item_description REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'notes' FROM products WHERE notes REGEXP '[^ -~]';
|
||||
|
||||
SELECT '' as '';
|
||||
SELECT 'Table: purchase_orders' as '';
|
||||
SELECT COUNT(*) as corrupted_records, 'title' as column_name FROM purchase_orders WHERE title REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'ship_via' FROM purchase_orders WHERE ship_via REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'comments' FROM purchase_orders WHERE comments REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'fob' FROM purchase_orders WHERE fob REGEXP '[^ -~]';
|
||||
|
||||
SELECT '' as '';
|
||||
SELECT 'Table: quotes' as '';
|
||||
SELECT COUNT(*) as corrupted_records, 'title' as column_name FROM quotes WHERE title REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'comments' FROM quotes WHERE comments REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'notes' FROM quotes WHERE notes REGEXP '[^ -~]';
|
||||
|
||||
SELECT '' as '';
|
||||
SELECT 'Table: shipments' as '';
|
||||
SELECT COUNT(*) as corrupted_records, 'title' as column_name FROM shipments WHERE title REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'ship_via' FROM shipments WHERE ship_via REGEXP '[^ -~]'
|
||||
UNION ALL SELECT COUNT(*), 'comments' FROM shipments WHERE comments REGEXP '[^ -~]';
|
||||
|
||||
SELECT '' as '';
|
||||
SELECT '============================================================' as '';
|
||||
SELECT 'SUMMARY - Tables with corruption (>0 records)' as '';
|
||||
SELECT '============================================================' as '';
|
||||
-- This gives you a quick overview of which tables need fixing
|
||||
|
|
@ -1,163 +0,0 @@
|
|||
#!/bin/bash
|
||||
# Script to verify character encoding fix in MariaDB
|
||||
|
||||
DB_USER="root"
|
||||
DB_PASS="secureRootPassword"
|
||||
DB_NAME="cmc"
|
||||
CONTAINER="cmc-db"
|
||||
|
||||
echo "=========================================="
|
||||
echo "Character Encoding Verification Script"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
# Function to run MariaDB command
|
||||
run_sql() {
|
||||
docker exec $CONTAINER mariadb -u $DB_USER -p$DB_PASS $DB_NAME -e "$1"
|
||||
}
|
||||
|
||||
# Function to run MariaDB command with UTF-8 client
|
||||
run_sql_utf8() {
|
||||
docker exec $CONTAINER mariadb -u $DB_USER -p$DB_PASS --default-character-set=utf8mb4 $DB_NAME -e "$1"
|
||||
}
|
||||
|
||||
echo "1. Check database default charset"
|
||||
echo "-----------------------------------"
|
||||
docker exec $CONTAINER mariadb -u $DB_USER -p$DB_PASS -e "
|
||||
SELECT @@character_set_database, @@collation_database;
|
||||
"
|
||||
echo ""
|
||||
|
||||
echo "2. Count tables by charset (BEFORE should show ~59 latin1)"
|
||||
echo "------------------------------------------------------------"
|
||||
run_sql "
|
||||
SELECT
|
||||
CASE
|
||||
WHEN TABLE_COLLATION LIKE 'latin1%' THEN 'latin1'
|
||||
WHEN TABLE_COLLATION LIKE 'utf8mb4%' THEN 'utf8mb4'
|
||||
WHEN TABLE_COLLATION LIKE 'utf8mb3%' THEN 'utf8mb3'
|
||||
ELSE TABLE_COLLATION
|
||||
END as charset,
|
||||
COUNT(*) as table_count
|
||||
FROM information_schema.TABLES
|
||||
WHERE TABLE_SCHEMA='$DB_NAME'
|
||||
GROUP BY charset
|
||||
ORDER BY table_count DESC;
|
||||
"
|
||||
echo ""
|
||||
|
||||
echo "3. Check specific table charsets (sample)"
|
||||
echo "------------------------------------------"
|
||||
run_sql "
|
||||
SELECT TABLE_NAME, TABLE_COLLATION
|
||||
FROM information_schema.TABLES
|
||||
WHERE TABLE_SCHEMA='$DB_NAME'
|
||||
AND TABLE_NAME IN ('principles', 'customers', 'addresses', 'contacts', 'products')
|
||||
ORDER BY TABLE_NAME;
|
||||
"
|
||||
echo ""
|
||||
|
||||
echo "4. Show column charsets for principles table"
|
||||
echo "---------------------------------------------"
|
||||
run_sql "
|
||||
SELECT COLUMN_NAME, CHARACTER_SET_NAME, COLLATION_NAME
|
||||
FROM information_schema.COLUMNS
|
||||
WHERE TABLE_SCHEMA='$DB_NAME'
|
||||
AND TABLE_NAME='principles'
|
||||
AND DATA_TYPE IN ('varchar', 'text', 'char')
|
||||
ORDER BY ORDINAL_POSITION;
|
||||
"
|
||||
echo ""
|
||||
|
||||
echo "5. Test INSERT of special characters (will be rolled back)"
|
||||
echo "-----------------------------------------------------------"
|
||||
echo "Attempting to insert test record with special characters..."
|
||||
run_sql_utf8 "
|
||||
START TRANSACTION;
|
||||
|
||||
INSERT INTO principles (name, code, address, city, state, postcode, country_id, currency_id, short_name, url)
|
||||
VALUES (
|
||||
'Test Müller & Söhne GmbH', -- German umlauts
|
||||
999999,
|
||||
'Straße 123, 2. Stock', -- German ß and ² symbol
|
||||
'München', -- German ü
|
||||
'Bayern',
|
||||
'80331',
|
||||
1,
|
||||
1,
|
||||
'TEST',
|
||||
'http://test.com'
|
||||
);
|
||||
|
||||
-- Read it back immediately
|
||||
SELECT id, name, address, city
|
||||
FROM principles
|
||||
WHERE code = 999999;
|
||||
|
||||
ROLLBACK;
|
||||
" 2>&1 | grep -A 10 "id\|Test Müller\|ERROR"
|
||||
echo "✓ Test insert rolled back"
|
||||
echo ""
|
||||
|
||||
echo "6. Verify current corrupted data in principles"
|
||||
echo "-----------------------------------------------"
|
||||
echo "These should show corruption if NOT yet fixed:"
|
||||
run_sql_utf8 "
|
||||
SELECT id, name, city
|
||||
FROM principles
|
||||
WHERE id IN (9, 18);
|
||||
"
|
||||
echo ""
|
||||
|
||||
echo "7. Verify corrupted data in customers"
|
||||
echo "--------------------------------------"
|
||||
run_sql_utf8 "
|
||||
SELECT id, name
|
||||
FROM customers
|
||||
WHERE id IN (253, 1006, 1387, 1608);
|
||||
"
|
||||
echo ""
|
||||
|
||||
echo "8. Count potentially corrupted records across tables"
|
||||
echo "-----------------------------------------------------"
|
||||
run_sql "
|
||||
SELECT
|
||||
'principles' as table_name,
|
||||
COUNT(*) as rows_with_non_ascii
|
||||
FROM principles
|
||||
WHERE name REGEXP '[^ -~]'
|
||||
OR address REGEXP '[^ -~]'
|
||||
OR city REGEXP '[^ -~]'
|
||||
UNION ALL
|
||||
SELECT
|
||||
'customers',
|
||||
COUNT(*)
|
||||
FROM customers
|
||||
WHERE name REGEXP '[^ -~]'
|
||||
UNION ALL
|
||||
SELECT
|
||||
'addresses',
|
||||
COUNT(*)
|
||||
FROM addresses
|
||||
WHERE address REGEXP '[^ -~]'
|
||||
OR city REGEXP '[^ -~]'
|
||||
UNION ALL
|
||||
SELECT
|
||||
'contacts',
|
||||
COUNT(*)
|
||||
FROM contacts
|
||||
WHERE firstname REGEXP '[^ -~]'
|
||||
OR lastname REGEXP '[^ -~]';
|
||||
"
|
||||
echo ""
|
||||
|
||||
echo "=========================================="
|
||||
echo "Verification complete!"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
echo "INTERPRETATION:"
|
||||
echo "- If table_count for 'latin1' is ~59: NOT YET FIXED"
|
||||
echo "- If table_count for 'utf8mb4' is ~62: FIXED!"
|
||||
echo "- Test insert should show proper characters if fixed"
|
||||
echo "- Corrupted data will still show mojibake until manually repaired"
|
||||
echo ""
|
||||
|
|
@ -1,207 +0,0 @@
|
|||
-- ============================================================================
|
||||
-- Manual MariaDB Character Encoding Verification Commands
|
||||
-- ============================================================================
|
||||
-- Run these commands directly in MariaDB to verify the charset fix
|
||||
--
|
||||
-- Connect to database:
|
||||
-- docker exec -it cmc-db mariadb -u root -psecureRootPassword --default-character-set=utf8mb4 cmc
|
||||
-- ============================================================================
|
||||
|
||||
-- 1. CHECK DATABASE-LEVEL SETTINGS
|
||||
-- Should show: utf8mb4
|
||||
SELECT @@character_set_database as db_charset,
|
||||
@@collation_database as db_collation;
|
||||
|
||||
-- 2. COUNT TABLES BY CHARACTER SET
|
||||
-- BEFORE FIX: ~59 latin1 tables
|
||||
-- AFTER FIX: ~62 utf8mb4 tables
|
||||
SELECT
|
||||
CASE
|
||||
WHEN TABLE_COLLATION LIKE 'latin1%' THEN 'latin1'
|
||||
WHEN TABLE_COLLATION LIKE 'utf8mb4%' THEN 'utf8mb4'
|
||||
WHEN TABLE_COLLATION LIKE 'utf8mb3%' THEN 'utf8mb3'
|
||||
ELSE TABLE_COLLATION
|
||||
END as charset,
|
||||
COUNT(*) as table_count
|
||||
FROM information_schema.TABLES
|
||||
WHERE TABLE_SCHEMA='cmc'
|
||||
GROUP BY charset
|
||||
ORDER BY table_count DESC;
|
||||
|
||||
-- 3. CHECK SPECIFIC TABLES (Key business tables)
|
||||
SELECT TABLE_NAME, TABLE_COLLATION, ENGINE
|
||||
FROM information_schema.TABLES
|
||||
WHERE TABLE_SCHEMA='cmc'
|
||||
AND TABLE_NAME IN (
|
||||
'principles', 'customers', 'addresses', 'contacts',
|
||||
'products', 'quotes', 'purchase_orders', 'invoices'
|
||||
)
|
||||
ORDER BY TABLE_NAME;
|
||||
|
||||
-- 4. CHECK ALL COLUMN CHARSETS FOR 'principles' TABLE
|
||||
-- All text columns should show utf8mb4 after fix
|
||||
SELECT
|
||||
COLUMN_NAME,
|
||||
DATA_TYPE,
|
||||
CHARACTER_SET_NAME,
|
||||
COLLATION_NAME
|
||||
FROM information_schema.COLUMNS
|
||||
WHERE TABLE_SCHEMA='cmc'
|
||||
AND TABLE_NAME='principles'
|
||||
AND DATA_TYPE IN ('varchar', 'text', 'char')
|
||||
ORDER BY ORDINAL_POSITION;
|
||||
|
||||
-- 5. TEST INSERT WITH SPECIAL CHARACTERS
|
||||
-- This will be rolled back - just testing if charset accepts them
|
||||
START TRANSACTION;
|
||||
|
||||
-- Insert test record with various special characters
|
||||
INSERT INTO principles (
|
||||
name, code, address, city, state, postcode,
|
||||
country_id, currency_id, short_name, url
|
||||
)
|
||||
VALUES (
|
||||
'Test Müller & Söhne GmbH', -- German umlauts (ü, ö)
|
||||
999999,
|
||||
'Straße 123, 2. Stock', -- German ß
|
||||
'München', -- German ü
|
||||
'Bayern',
|
||||
'80331',
|
||||
1, 1, 'TEST', 'http://test.com'
|
||||
);
|
||||
|
||||
-- Read it back - should show properly if charset is correct
|
||||
SELECT id, name, address, city
|
||||
FROM principles
|
||||
WHERE code = 999999;
|
||||
|
||||
-- Expected output if FIXED:
|
||||
-- name: Test Müller & Söhne GmbH (proper umlauts)
|
||||
-- address: Straße 123, 2. Stock
|
||||
-- city: München
|
||||
|
||||
-- Expected output if NOT FIXED:
|
||||
-- Characters will be corrupted or insert will fail
|
||||
|
||||
ROLLBACK; -- Don't actually save the test record
|
||||
|
||||
-- 6. VIEW KNOWN CORRUPTED RECORDS
|
||||
-- These show what the corruption looks like before manual repair
|
||||
|
||||
-- Principle ID 9 - Czech characters corrupted
|
||||
SELECT id, name, address, city
|
||||
FROM principles
|
||||
WHERE id = 9;
|
||||
-- BEFORE FIX: BezruÄova 2901, Rožnov pod RadhoÅ¡tÄ›m
|
||||
-- AFTER FIX (still needs manual repair): Same corruption
|
||||
-- AFTER MANUAL REPAIR: Bezručova 2901, Rožnov pod Radhoštěm
|
||||
|
||||
-- Principle ID 18 - German umlaut in city
|
||||
SELECT id, short_name, city, country_id
|
||||
FROM principles
|
||||
WHERE id = 18;
|
||||
|
||||
-- Customer corrupted names
|
||||
SELECT id, name
|
||||
FROM customers
|
||||
WHERE id IN (253, 1006, 1387, 1608, 2174);
|
||||
-- Look for: ’ (should be '), É (should be É), “ (should be ")
|
||||
|
||||
-- Address corrupted data
|
||||
SELECT id, address, city
|
||||
FROM addresses
|
||||
WHERE id IN (19, 25);
|
||||
-- Look for: â€" (should be –)
|
||||
|
||||
-- 7. COUNT RECORDS WITH NON-ASCII CHARACTERS
|
||||
-- Shows how many records might need manual repair after charset fix
|
||||
SELECT
|
||||
'principles' as table_name,
|
||||
COUNT(*) as records_with_non_ascii
|
||||
FROM principles
|
||||
WHERE name REGEXP '[^ -~]'
|
||||
OR address REGEXP '[^ -~]'
|
||||
OR city REGEXP '[^ -~]'
|
||||
|
||||
UNION ALL
|
||||
|
||||
SELECT 'customers', COUNT(*)
|
||||
FROM customers
|
||||
WHERE name REGEXP '[^ -~]'
|
||||
|
||||
UNION ALL
|
||||
|
||||
SELECT 'addresses', COUNT(*)
|
||||
FROM addresses
|
||||
WHERE address REGEXP '[^ -~]'
|
||||
OR city REGEXP '[^ -~]'
|
||||
|
||||
UNION ALL
|
||||
|
||||
SELECT 'contacts', COUNT(*)
|
||||
FROM contacts
|
||||
WHERE firstname REGEXP '[^ -~]'
|
||||
OR lastname REGEXP '[^ -~]';
|
||||
|
||||
-- 8. VERIFY CONNECTION CHARACTER SET
|
||||
-- Your client should also be using utf8mb4
|
||||
SHOW VARIABLES LIKE 'character_set_%';
|
||||
SHOW VARIABLES LIKE 'collation_%';
|
||||
|
||||
-- Should see:
|
||||
-- character_set_client: utf8mb4
|
||||
-- character_set_connection: utf8mb4
|
||||
-- character_set_results: utf8mb4
|
||||
-- character_set_database: utf8mb4
|
||||
|
||||
-- ============================================================================
|
||||
-- AFTER CONVERSION - TEST NEW INSERTS
|
||||
-- ============================================================================
|
||||
-- After running convert_charset_to_utf8mb4.sql, try these tests:
|
||||
|
||||
-- Test 1: Insert with German characters
|
||||
START TRANSACTION;
|
||||
INSERT INTO principles (name, code, address, city, state, postcode, country_id, currency_id, short_name, url)
|
||||
VALUES ('Müller GmbH', 999991, 'Hauptstraße 1', 'München', 'BY', '80331', 1, 1, 'MUL', 'https://test.de');
|
||||
SELECT * FROM principles WHERE code = 999991;
|
||||
-- Should show: Müller, Hauptstraße, München (correctly)
|
||||
ROLLBACK;
|
||||
|
||||
-- Test 2: Insert with French characters
|
||||
START TRANSACTION;
|
||||
INSERT INTO principles (name, code, address, city, state, postcode, country_id, currency_id, short_name, url)
|
||||
VALUES ('Société François', 999992, 'Rue de la Paix', 'Orléans', 'FR', '45000', 1, 1, 'SOC', 'https://test.fr');
|
||||
SELECT * FROM principles WHERE code = 999992;
|
||||
-- Should show: Société, François, Orléans (correctly)
|
||||
ROLLBACK;
|
||||
|
||||
-- Test 3: Insert with special punctuation
|
||||
START TRANSACTION;
|
||||
INSERT INTO customers (name, trading_name, abn, created, notes, discount_pricing_policies, payment_terms, customer_category_id, url, country_id)
|
||||
VALUES ('O'Connor & Sons — "Quality" Products', 'O'Connor Trading', '', NOW(), '', '', '', 1, '', 1);
|
||||
SELECT name FROM customers WHERE name LIKE '%O''Connor%' ORDER BY id DESC LIMIT 1;
|
||||
-- Should show: O'Connor & Sons — "Quality" Products (with proper apostrophes, em-dash, and quotes)
|
||||
ROLLBACK;
|
||||
|
||||
-- ============================================================================
|
||||
-- SUMMARY OF WHAT TO LOOK FOR
|
||||
-- ============================================================================
|
||||
|
||||
/*
|
||||
BEFORE FIX:
|
||||
- Table charset: latin1_swedish_ci
|
||||
- Column charset: latin1
|
||||
- Test insert: May fail or corrupt characters
|
||||
- Existing data: Shows mojibake (É, ’, etc.)
|
||||
|
||||
AFTER FIX (charset conversion only):
|
||||
- Table charset: utf8mb4_general_ci
|
||||
- Column charset: utf8mb4
|
||||
- Test insert: Succeeds with proper characters
|
||||
- Existing data: STILL shows mojibake (needs manual repair)
|
||||
|
||||
AFTER MANUAL REPAIR:
|
||||
- Everything displays correctly
|
||||
- New inserts work properly
|
||||
- Old data has been corrected
|
||||
*/
|
||||
Loading…
Reference in a new issue