Development Guide
This guide covers building, testing, and contributing to SchemaX.
Prerequisites
Required
- Python 3.11 or higher - For Python SDK and CLI development
- Node.js 18 or higher - For VS Code extension development
- npm 9 or higher - For JavaScript dependencies
- Visual Studio Code 1.90.0 or higher - For extension development and testing
Recommended (Optional)
-
uv - Fast Python package installer and virtual environment manager (10-100x faster than pip)
Install uv:
# macOS/Linux
curl -LsSf https://astral.sh/uv/install.sh | sh
# Windows
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
# Or via pip
pip install uv
Getting Started
1. Clone Repository
git clone https://github.com/vb-dbrks/schemax-vscode.git
cd schemax-vscode
2. Choose Your Setup Path
Depending on what you're working on, follow one of these paths:
Option A: Full Setup (Extension + Python SDK)
Install VS Code Extension Dependencies:
npm install
npm run build
Install Python SDK:
Using uv (recommended - much faster):
cd packages/python-sdk
uv venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
uv pip install -e ".[dev]"
cd ../..
Using traditional pip/venv:
cd packages/python-sdk
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
pip install -e ".[dev]"
cd ../..
Option B: Extension Only
If you're only working on the VS Code extension:
npm install
npm run build
Option C: Python SDK Only
If you're only working on the Python SDK/CLI:
cd packages/python-sdk
# With uv (recommended)
uv venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
uv pip install -e ".[dev]"
# Or with traditional tools
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
pip install -e ".[dev]"
3. Verify Installation
VS Code Extension:
npm run build # Should complete without errors
# Then press F5 in VS Code to launch Extension Development Host
Python SDK:
# Check CLI is installed
schemax --version
# Run tests
cd packages/python-sdk
pytest
# Run linter
ruff check src/
Development Workflows
VS Code Extension Development
Build:
# Build everything (extension + webview)
npm run build
# Build individually
npm run build:ext # Extension only
npm run build:webview # Webview only
Watch Mode (automatic rebuilds):
npm run watch
This runs both extension and webview watchers concurrently.
Testing:
- Open the project in VS Code
- Press
F5to launch the Extension Development Host - In the new window, open a test workspace
- Press
Cmd+Shift+Pand run SchemaX: Open Designer - Check logs: View → Output → Select "SchemaX"
Python SDK Development
Activate Virtual Environment:
cd packages/python-sdk
source .venv/bin/activate # On Windows: .venv\Scripts\activate
Run Tests:
# Run all tests
pytest
# Run with coverage
pytest --cov=schemax --cov-report=term-missing
# Run specific test file
pytest tests/unit/test_sql_generator.py
# Run specific test class
pytest tests/unit/test_sql_generator.py::TestTableSQL -v
# Run with verbose output
pytest -xvs
Test Status:
- ✅ 124 passing tests (91.2% of total)
- ⏸️ 12 skipped tests (8.8%)
Linting and Type Checking:
# Run linter
ruff check src/
# Auto-fix linting issues
ruff check --fix src/
# Type checking
mypy src/
SQL Validation (Optional):
SchemaX includes optional SQLGlot integration for validating generated SQL:
# Install SQLGlot
pip install "schemaxpy[validation]"
# or
pip install sqlglot>=20.0.0
# Use in tests or scripts
python -c "
import sqlglot
sql = 'ALTER TABLE \`catalog\`.\`schema\`.\`table\` ADD COLUMN \`id\` BIGINT'
parsed = sqlglot.parse_one(sql, dialect='databricks')
print('Valid SQL!' if parsed else 'Invalid SQL')
"
Manual Testing:
# Navigate to test project
cd ../../examples/basic-schema
# Test CLI commands
schemax validate
schemax sql
schemax sql --output test-migration.sql
Project Structure
schemax-vscode/
├── packages/
│ ├── vscode-extension/ # VS Code Extension
│ │ ├── src/
│ │ │ ├── extension.ts # Extension entry point
│ │ │ ├── storage-v3.ts # Provider-aware storage
│ │ │ ├── providers/ # Provider system
│ │ │ │ ├── base/ # Base interfaces
│ │ │ │ │ ├── provider.ts
│ │ │ │ │ ├── models.ts
│ │ │ │ │ ├── operations.ts
│ │ │ │ │ ├── sql-generator.ts
│ │ │ │ │ └── hierarchy.ts
│ │ │ │ ├── registry.ts # Provider registry
│ │ │ │ ├── index.ts # Auto-registration
│ │ │ │ └── unity/ # Unity Catalog provider
│ │ │ │ ├── index.ts
│ │ │ │ ├── models.ts
│ │ │ │ ├── operations.ts
│ │ │ │ ├── sql-generator.ts
│ │ │ │ ├── state-reducer.ts
│ │ │ │ ├── hierarchy.ts
│ │ │ │ └── provider.ts
│ │ │ ├── shared/ # Legacy (V2 compat)
│ │ │ │ ├── model.ts
│ │ │ │ └── ops.ts
│ │ │ └── webview/
│ │ │ ├── main.tsx # Webview entry
│ │ │ ├── App.tsx # Main component
│ │ │ ├── state/
│ │ │ │ └── useDesignerStore.ts # Provider-aware store
│ │ │ └── components/
│ │ │ ├── Sidebar.tsx
│ │ │ ├── TableDesigner.tsx
│ │ │ ├── ColumnGrid.tsx
│ │ │ └── SnapshotPanel.tsx
│ │ ├── dist/ # Compiled extension
│ │ ├── media/ # Compiled webview
│ │ └── package.json
│ │
│ └── python-sdk/ # Python SDK & CLI
│ ├── src/schemax/
│ │ ├── providers/ # Provider system
│ │ │ ├── base/
│ │ │ │ ├── provider.py
│ │ │ │ ├── models.py
│ │ │ │ ├── operations.py
│ │ │ │ ├── sql_generator.py
│ │ │ │ └── hierarchy.py
│ │ │ ├── registry.py
│ │ │ └── unity/
│ │ │ ├── __init__.py
│ │ │ ├── models.py
│ │ │ ├── operations.py
│ │ │ ├── sql_generator.py
│ │ │ ├── state_reducer.py
│ │ │ ├── hierarchy.py
│ │ │ └── provider.py
│ │ ├── storage_v3.py # Provider-aware storage
│ │ ├── cli.py # CLI commands
│ │ └── models.py # Legacy (V2 compat)
│ └── pyproject.toml
│
├── docs/ # Documentation
├── examples/ # Working examples
└── scripts/ # Utility scripts
Architecture
Provider-Based System
SchemaX uses a provider-based architecture to support multiple catalog types (Unity Catalog, Hive, PostgreSQL, etc.). Each provider implements a standard interface.
Key Components:
- Provider Registry - Manages available providers
- Provider Interface - Standard contract all providers implement
- Storage V3 - Provider-aware file operations
- State Reducer - Provider-specific state modifications
- SQL Generator - Provider-specific DDL generation
See Architecture for detailed architecture documentation.
Extension Host
The extension host (src/extension.ts) runs in Node.js and:
- Registers VS Code commands
- Manages file I/O via
storage-v3.ts - Loads provider from project file
- Creates and controls the webview
- Handles message passing with the webview
- Validates operations using provider
Webview
The webview (src/webview/) is a React application that:
- Provides the visual designer UI
- Uses Zustand for state management
- Stores current provider information
- Generates provider-prefixed operations
- Sends operations to the extension via
postMessage
Communication Flow (V3)
User Action → Webview (React)
→ Get provider from store
→ Generate Op with provider prefix (e.g., unity.add_catalog)
→ postMessage('append-ops')
→ Extension
→ Get provider from registry
→ Validate operation
→ Append to changelog.json
→ Apply op using provider's state reducer
→ postMessage('project-updated' with provider info)
→ Webview updates UI
Storage Architecture (V3)
SchemaX V3 adds provider awareness to the snapshot-based architecture:
Files:
.schemax/project.json- Metadata + provider selection.schemax/changelog.json- Uncommitted provider-prefixed operations.schemax/snapshots/vX.Y.Z.json- Full state snapshots
Loading:
- Read project.json → get provider type
- Load provider from registry
- Read latest snapshot file (or use provider's initial state)
- Read changelog
- Apply operations using provider's state reducer
- Result = current state
Migration:
- V2 projects automatically migrate to V3 on first load
- Operations get prefixed with provider ID (e.g.,
unity.) - Provider field added to project.json
Saving:
- Operations append to
changelog.json - State changes are computed by applying operations
- Snapshots freeze the state and clear changelog
See Architecture for detailed design information.
Key Concepts
Operations
Operations are immutable records of user actions:
{
id: "op_uuid",
ts: "2025-10-06T...",
op: "add_column",
target: "col_uuid",
payload: { tableId, colId, name, type, nullable }
}
All schema changes generate operations. Operations are:
- Append-only (never modified)
- Tracked with UUIDs
- Stored in
changelog.json - Used to generate migrations (future)
Snapshots
Snapshots capture complete schema state:
Metadata (in project.json):
{
id: "snap_uuid",
version: "v0.1.0",
name: "Initial schema",
file: ".schemax/snapshots/v0.1.0.json",
opsCount: 15,
...
}
Full Snapshot (in snapshots/v0.1.0.json):
{
id: "snap_uuid",
version: "v0.1.0",
state: { catalogs: [...] }, // Complete schema
opsIncluded: ["op_1", "op_2", ...],
hash: "sha256...",
...
}
Build System
Extension Build (esbuild)
- Entry:
src/extension.ts - Output:
dist/extension.js - Target: Node.js (CommonJS)
- Externals:
vscodemodule
Webview Build (Vite)
- Entry:
src/webview/main.tsx - Output:
media/directory - Target: Browser (ES modules)
- Bundles: React, Zustand, CSS
Coding Guidelines
Logging
Always use the output channel for logging:
outputChannel.appendLine('[SchemaX] Your message');
Never use console.log() in extension code (it goes to Extension Host console, not visible to users).
File I/O
TypeScript (Extension):
All file operations should go through storage-v3.ts:
import * as storageV3 from './storage-v3';
// Good
const project = await storageV3.readProject(workspaceUri);
const { state, changelog, provider } = await storageV3.loadCurrentState(workspaceUri);
await storageV3.appendOps(workspaceUri, ops);
// Bad - don't access files directly
const content = await fs.readFile('.schemax/project.json');
Python (SDK):
All file operations should go through storage_v3.py:
from schemax.storage_v3 import read_project, load_current_state, append_ops
# Good
project = read_project(workspace_path)
state, changelog, provider = load_current_state(workspace_path)
append_ops(workspace_path, operations)
# Bad - don't access files directly
with open('.schemax/project.json') as f:
project = json.load(f)
State Management
The webview uses Zustand. All mutations must generate operations:
// In useDesignerStore.ts
addColumn: (tableId, name, type, nullable) => {
const op: Op = {
id: `op_${uuidv4()}`,
ts: new Date().toISOString(),
op: 'add_column',
target: colId,
payload: { tableId, colId, name, type, nullable }
};
emitOps([op]); // Sends to extension
}
Data Validation
TypeScript:
All external data must be validated. The storage layer handles validation internally:
import * as storageV3 from './storage-v3';
// Validates and throws if invalid
const project = await storageV3.readProject(workspacePath);
Python:
All external data must be validated. The storage layer handles validation internally:
from schemax.storage_v3 import read_project
# Validates and raises if invalid
project = read_project(workspace_path)
Code Formatting
TypeScript:
- Use TypeScript strict mode
- 2 spaces indentation
- Single quotes for strings
- Semicolons required
- Max line length: 100 characters
Python:
- Use Ruff for linting and formatting (configured in
pyproject.toml) - Line length: 100 characters
- Type hints required for all function signatures
- Follow PEP 8 naming conventions
- Docstrings for public APIs
Format code before committing:
# Python
cd packages/python-sdk
ruff check --fix src/
ruff format src/
# TypeScript (manual formatting for now)
# Follow existing code style
Testing Checklist
Before submitting a pull request, verify:
VS Code Extension
- Extension builds without errors (
npm run build) - No TypeScript errors
- Designer opens successfully
- Can create catalogs, schemas, tables, columns
- Can edit column properties inline
- Can create snapshots
- Snapshots persist after reloading
- Changelog clears after snapshot
- SchemaX output logs show no errors
Python SDK
- All tests pass (
pytest) - No new test failures (124 passing baseline)
- Test coverage maintained at 30%+ (
pytest --cov=schemax --cov-fail-under=30) - No linting errors (
ruff check src/) - Type checking passes (
mypy src/) - Code is formatted (
ruff format src/) - CLI commands work (
schemax validate,schemax sql) - Documentation updated if API changed
- SQL validated with SQLGlot (if modifying SQL generator)
Note: Coverage threshold is currently set to 30% (baseline). As test coverage improves, this threshold should be gradually increased.
Test Structure:
tests/
├── conftest.py # Shared fixtures
├── utils/
│ └── operation_builders.py # OperationBuilder helper
├── unit/
│ ├── test_storage_v3.py # Storage layer tests
│ ├── test_state_reducer.py # State reducer tests
│ ├── test_sql_generator.py # SQL generation tests
│ └── test_provider.py # Provider system tests
└── integration/
└── test_workflows.py # End-to-end workflow tests
Using Test Helpers:
Tests use OperationBuilder for creating test operations:
from tests.utils import OperationBuilder
builder = OperationBuilder()
op = builder.add_catalog("cat_123", "bronze", op_id="op_001")
# Automatically includes: id, ts, provider, op, target, payload
Both (if applicable)
- Provider changes implemented in both TypeScript and Python
- State reducer logic matches between implementations
- SQL generator produces identical output
- Operation definitions synchronized
Debugging
Extension Logs
View → Output → Select "SchemaX" from dropdown
Shows:
- Extension activation
- File operations
- Operation appends
- Snapshot creation
- Errors with stack traces
Webview Console
- With designer open: Help → Toggle Developer Tools
- Go to Console tab
- Look for
[SchemaX Webview]logs
Common Issues
Blank webview:
- Check webview console for JavaScript errors
- Verify
media/directory hasindex.htmlandassets/ - Rebuild webview:
npm run build:webview
Operations not saving:
- Check SchemaX output logs
- Verify
.schemax/directory exists - Check file permissions
Snapshots disappearing:
- Verify
project.jsonhassnapshotsarray - Check
changelog.jsonexists - Look for errors in output logs
Provider Development
Overview
Adding a new provider to SchemaX involves implementing the provider interface in both TypeScript and Python. This section guides you through the process.
See also: Provider Contract for detailed API documentation.
Quick Start: Adding a New Provider
1. Create Provider Directories
# TypeScript
mkdir -p packages/vscode-extension/src/providers/myprovider
touch packages/vscode-extension/src/providers/myprovider/{index,models,operations,sql-generator,state-reducer,hierarchy,provider}.ts
# Python
mkdir -p packages/python-sdk/src/schemax/providers/myprovider
touch packages/python-sdk/src/schemax/providers/myprovider/{__init__,models,operations,sql_generator,state_reducer,hierarchy,provider}.py
2. Define Provider Metadata
// providers/myprovider/provider.ts
export const myProvider: Provider = {
info: {
id: 'myprovider',
name: 'My Provider',
version: '1.0.0',
description: 'Support for MyDB',
author: 'Your Name',
docsUrl: 'https://docs.mydb.com',
},
capabilities: {
supportedOperations: ['myprovider.add_database', 'myprovider.add_table'],
supportedObjectTypes: ['database', 'table'],
hierarchy: myProviderHierarchy,
features: {
constraints: true,
rowFilters: false,
columnMasks: false,
columnTags: false,
},
},
// ... implement interface methods
};
3. Define Hierarchy
// providers/myprovider/hierarchy.ts
export const myProviderHierarchy = new Hierarchy([
{
name: 'database',
displayName: 'Database',
pluralName: 'databases',
icon: 'database',
isContainer: true,
},
{
name: 'table',
displayName: 'Table',
pluralName: 'tables',
icon: 'table',
isContainer: false,
},
]);
4. Define Models
// providers/myprovider/models.ts
import { z } from 'zod';
export const MyDatabase = z.object({
id: z.string(),
name: z.string(),
tables: z.array(MyTable),
});
export const MyTable = z.object({
id: z.string(),
name: z.string(),
columns: z.array(MyColumn),
});
export const MyProviderState = z.object({
databases: z.array(MyDatabase),
});
export type MyDatabase = z.infer<typeof MyDatabase>;
export type MyTable = z.infer<typeof MyTable>;
export type MyProviderState = z.infer<typeof MyProviderState>;
5. Define Operations
// providers/myprovider/operations.ts
export const MY_PROVIDER_OPERATIONS = {
ADD_DATABASE: 'myprovider.add_database',
RENAME_DATABASE: 'myprovider.rename_database',
DROP_DATABASE: 'myprovider.drop_database',
ADD_TABLE: 'myprovider.add_table',
// ... more operations
};
export const myProviderOperationMetadata: OperationMetadata[] = [
{
type: MY_PROVIDER_OPERATIONS.ADD_DATABASE,
displayName: 'Add Database',
description: 'Create a new database',
category: OperationCategory.DATABASE,
requiredFields: ['databaseId', 'name'],
optionalFields: [],
isDestructive: false,
},
// ... more metadata
];
6. Implement State Reducer
// providers/myprovider/state-reducer.ts
export function applyOperation(
state: MyProviderState,
op: Operation
): MyProviderState {
// Deep clone for immutability
const newState = JSON.parse(JSON.stringify(state));
// Strip provider prefix
const opType = op.op.replace('myprovider.', '');
switch (opType) {
case 'add_database': {
const database: MyDatabase = {
id: op.payload.databaseId,
name: op.payload.name,
tables: [],
};
newState.databases.push(database);
break;
}
case 'add_table': {
const db = newState.databases.find(d => d.id === op.payload.databaseId);
if (db) {
const table: MyTable = {
id: op.payload.tableId,
name: op.payload.name,
columns: [],
};
db.tables.push(table);
}
break;
}
// ... more cases
}
return newState;
}
7. Implement SQL Generator
// providers/myprovider/sql-generator.ts
export class MyProviderSQLGenerator extends BaseSQLGenerator {
generateSQLForOperation(op: Operation): SQLGenerationResult {
const opType = op.op.replace('myprovider.', '');
switch (opType) {
case 'add_database':
return {
sql: `CREATE DATABASE IF NOT EXISTS ${this.escapeIdentifier(op.payload.name)}`,
warnings: [],
isIdempotent: true,
};
case 'add_table':
const dbName = this.idNameMap[op.payload.databaseId];
const tableName = op.payload.name;
return {
sql: `CREATE TABLE IF NOT EXISTS ${dbName}.${this.escapeIdentifier(tableName)} ()`,
warnings: [],
isIdempotent: true,
};
// ... more cases
}
}
}
8. Implement Provider Class
// providers/myprovider/provider.ts
export class MyProvider extends BaseProvider {
get info(): ProviderInfo {
return {
id: 'myprovider',
name: 'My Provider',
version: '1.0.0',
description: 'Support for MyDB',
author: 'Your Name',
docsUrl: 'https://docs.mydb.com',
};
}
get capabilities(): ProviderCapabilities {
return {
supportedOperations: Object.values(MY_PROVIDER_OPERATIONS),
supportedObjectTypes: ['database', 'table'],
hierarchy: myProviderHierarchy,
features: {
constraints: true,
rowFilters: false,
columnMasks: false,
columnTags: false,
},
};
}
createInitialState(): ProviderState {
return { databases: [] };
}
applyOperation(state: ProviderState, op: Operation): ProviderState {
return applyOperation(state as MyProviderState, op);
}
getSQLGenerator(state: ProviderState): SQLGenerator {
return new MyProviderSQLGenerator(state);
}
validateOperation(op: Operation): ValidationResult {
// Implementation
}
validateState(state: ProviderState): ValidationResult {
// Implementation
}
}
9. Register Provider
// providers/myprovider/index.ts
export * from './models';
export * from './operations';
export * from './provider';
import { MyProvider } from './provider';
export const myProvider = new MyProvider();
// providers/index.ts - Add to auto-registration
import { myProvider } from './myprovider';
export function initializeProviders() {
ProviderRegistry.register(unityProvider);
ProviderRegistry.register(myProvider); // NEW!
}
initializeProviders();
10. Python Implementation
Follow the same structure in Python:
# packages/python-sdk/src/schemax/providers/myprovider/provider.py
class MyProvider(BaseProvider):
@property
def info(self) -> ProviderInfo:
return ProviderInfo(
id="myprovider",
name="My Provider",
version="1.0.0",
description="Support for MyDB",
author="Your Name",
docs_url="https://docs.mydb.com",
)
# ... implement all methods
Testing Your Provider
1. Create Test File
// providers/myprovider/__tests__/myprovider.test.ts
import { testProviderCompliance } from '../../base/__tests__/provider-compliance.test';
import { myProvider } from '../provider';
// Run compliance tests
testProviderCompliance(myProvider);
// Provider-specific tests
describe('MyProvider', () => {
test('creates initial state', () => {
const state = myProvider.createInitialState();
expect(state.databases).toEqual([]);
});
test('applies add_database operation', () => {
// ... test implementation
});
});
2. Run Tests
# TypeScript
cd packages/vscode-extension
npm test
# Python
cd packages/python-sdk
pytest tests/providers/myprovider/
Common Patterns
Immutable State Updates:
// ❌ BAD - Mutates state
state.databases.push(newDb);
// ✅ GOOD - Creates new state
const newState = {
...state,
databases: [...state.databases, newDb],
};
Operation Validation:
validateOperation(op: Operation): ValidationResult {
const errors: ValidationError[] = [];
const metadata = this.getOperationMetadata(op.op);
// Check required fields
for (const field of metadata.requiredFields) {
if (!op.payload[field]) {
errors.push({
field: `payload.${field}`,
message: `Required field missing: ${field}`,
code: 'MISSING_REQUIRED_FIELD',
});
}
}
return { valid: errors.length === 0, errors };
}
SQL Escaping:
// Always escape identifiers
this.escapeIdentifier(name); // `name`
// Always escape strings
this.escapeString(value); // 'value'
Provider Development Checklist
Before submitting a provider:
- All required interface methods implemented
- Provider registered in registry
- Hierarchy defined
- All operations have metadata
- State reducer handles all operations
- State reducer is pure (no side effects)
- SQL generator handles all operations
- SQL is idempotent (safe to run multiple times)
- Operation validation implemented
- State validation implemented
- Provider compliance tests pass
- Provider-specific tests written
- Python implementation matches TypeScript
- Documentation written
- Examples provided
Resources
- Provider Contract: Provider Contract
- Architecture: Architecture
- Unity Provider: Reference implementation in
providers/unity/ - Base Provider: Abstract base in
providers/base/
Contributing
Reporting Issues
When reporting bugs, include:
- Steps to reproduce
- Expected vs actual behavior
- SchemaX output logs
- Webview console errors (if any)
- VS Code version
- Operating system
Pull Requests
- Fork the repository
- Create a feature branch:
git checkout -b feature/your-feature - Make changes and test thoroughly
- Commit with clear messages
- Push and create a pull request
Code Style
- Use TypeScript strict mode
- Follow existing code formatting
- Add comments for complex logic
- Keep functions focused and small
- Use descriptive variable names
Release Process
- Update version in
package.json - Update CHANGELOG.md
- Build:
npm run build - Package:
npm run package - Test the
.vsixfile - Create GitHub release
- Publish to marketplace