# Database Synchronization Scripts

This directory contains scripts for synchronizing SQLite-vec databases across multiple machines using JSON export/import and Litestream replication.

## Overview

The synchronization system enables you to:
- Export memories from one machine to JSON format
- Import memories from multiple JSON files into a central database
- Set up real-time replication using Litestream
- Maintain consistent memory databases across multiple devices

## Scripts

### export_memories.py

Export memories from a local SQLite-vec database to JSON format.

**Usage:**
```bash
# Basic export
python export_memories.py

# Export from specific database
python export_memories.py --db-path /path/to/sqlite_vec.db

# Export to specific file
python export_memories.py --output my_export.json

# Export with embedding vectors (large file size)
python export_memories.py --include-embeddings

# Export only specific tags
python export_memories.py --filter-tags claude-code,architecture
```

**Features:**
- Preserves original timestamps and metadata
- Adds source machine tracking
- Supports tag filtering
- Optional embedding vector export
- Cross-platform compatibility

### import_memories.py

Import memories from JSON export files into a central database.

**Usage:**
```bash
# Import single file
python import_memories.py windows_export.json

# Import multiple files
python import_memories.py windows_export.json macbook_export.json

# Dry run analysis
python import_memories.py --dry-run exports/*.json

# Import to specific database
python import_memories.py --db-path /path/to/central.db exports/*.json
```

**Features:**
- Intelligent deduplication based on content hash
- Preserves original timestamps
- Adds source tracking tags
- Conflict detection and resolution
- Comprehensive import statistics

## Typical Workflow

### Phase 1: Initial Consolidation

1. **Export from each machine:**
   ```bash
   # On Windows PC
   python export_memories.py --output windows_memories.json
   
   # On MacBook
   python export_memories.py --output macbook_memories.json
   ```

2. **Transfer files to central server:**
   ```bash
   scp windows_memories.json central-server:/tmp/
   scp macbook_memories.json central-server:/tmp/
   ```

3. **Import on central server:**
   ```bash
   # Analyze first
   python import_memories.py --dry-run /tmp/*.json
   
   # Import for real
   python import_memories.py /tmp/windows_memories.json /tmp/macbook_memories.json
   ```

### Phase 2: Set up Litestream

After consolidating all memories into the central database, set up Litestream for ongoing synchronization.

## JSON Export Format

The export format preserves all memory data:

```json
{
  "export_metadata": {
    "source_machine": "machine-name",
    "export_timestamp": "2025-08-12T10:30:00Z",
    "total_memories": 450,
    "database_path": "/path/to/sqlite_vec.db",
    "platform": "Windows",
    "exporter_version": "5.0.0"
  },
  "memories": [
    {
      "content": "Memory content here",
      "content_hash": "sha256hash",
      "tags": ["tag1", "tag2"],
      "created_at": 1673545200.0,
      "updated_at": 1673545200.0,
      "memory_type": "note",
      "metadata": {},
      "export_source": "machine-name"
    }
  ]
}
```

## Deduplication Strategy

Memories are deduplicated based on content hash:
- Same content hash = duplicate (skipped)
- Different content hash = unique (imported)
- Original timestamps are preserved
- Source machine tags are added for tracking

## Error Handling

Both scripts include comprehensive error handling:
- JSON format validation
- Database connectivity checks
- File existence verification
- Transaction rollback on failures
- Detailed error logging

## Performance Considerations

- **Export**: ~1000 memories/second
- **Import**: ~500 memories/second with deduplication
- **File Size**: ~1KB per memory (without embeddings)
- **Memory Usage**: Processes files in streaming fashion

## Troubleshooting

### Common Issues

1. **Database not found:**
   ```bash
   # Check default location
   python -c "from mcp_memory_service.config import get_storage_path; print(get_storage_path())"
   ```

2. **Permission errors:**
   ```bash
   # Ensure database directory is writable
   chmod 755 ~/.local/share/mcp-memory/
   ```

3. **JSON format errors:**
   ```bash
   # Validate JSON file
   python -m json.tool export.json > /dev/null
   ```

### Logging

Enable verbose logging for debugging:
```bash
python export_memories.py --verbose
python import_memories.py --verbose
```

## Next Steps

After using these scripts for initial consolidation:

1. Set up Litestream for real-time sync
2. Configure replica nodes
3. Implement monitoring and alerting
4. Schedule regular backups

See the main documentation for complete Litestream setup instructions.