Skip to main content

Development Guide

This guide covers building, testing, and contributing to SchemaX.

Prerequisites

Required

  • Python 3.11 or higher - For Python SDK and CLI development
  • Node.js 18 or higher - For VS Code extension development
  • npm 9 or higher - For JavaScript dependencies
  • Visual Studio Code 1.90.0 or higher - For extension development and testing
  • uv - Fast Python package installer and virtual environment manager (10-100x faster than pip)

    Install uv:

    # macOS/Linux
    curl -LsSf https://astral.sh/uv/install.sh | sh

    # Windows
    powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

    # Or via pip
    pip install uv

Getting Started

1. Clone Repository

git clone https://github.com/vb-dbrks/schemax-vscode.git
cd schemax-vscode

2. Choose Your Setup Path

Depending on what you're working on, follow one of these paths:

Option A: Full Setup (Extension + Python SDK)

Install VS Code Extension Dependencies:

npm install
npm run build

Install Python SDK:

Using uv (recommended - much faster):

cd packages/python-sdk
uv venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
uv pip install -e ".[dev]"
cd ../..

Using traditional pip/venv:

cd packages/python-sdk
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
pip install -e ".[dev]"
cd ../..

Option B: Extension Only

If you're only working on the VS Code extension:

npm install
npm run build

Option C: Python SDK Only

If you're only working on the Python SDK/CLI:

cd packages/python-sdk

# With uv (recommended)
uv venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
uv pip install -e ".[dev]"

# Or with traditional tools
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
pip install -e ".[dev]"

3. Verify Installation

VS Code Extension:

npm run build  # Should complete without errors
# Then press F5 in VS Code to launch Extension Development Host

Python SDK:

# Check CLI is installed
schemax --version

# Run tests
cd packages/python-sdk
pytest

# Run linter
ruff check src/

Development Workflows

VS Code Extension Development

Build:

# Build everything (extension + webview)
npm run build

# Build individually
npm run build:ext # Extension only
npm run build:webview # Webview only

Watch Mode (automatic rebuilds):

npm run watch

This runs both extension and webview watchers concurrently.

Testing:

  1. Open the project in VS Code
  2. Press F5 to launch the Extension Development Host
  3. In the new window, open a test workspace
  4. Press Cmd+Shift+P and run SchemaX: Open Designer
  5. Check logs: View → Output → Select "SchemaX"

Python SDK Development

Activate Virtual Environment:

cd packages/python-sdk
source .venv/bin/activate # On Windows: .venv\Scripts\activate

Run Tests:

# Run all tests
pytest

# Run with coverage
pytest --cov=schemax --cov-report=term-missing

# Run specific test file
pytest tests/unit/test_sql_generator.py

# Run specific test class
pytest tests/unit/test_sql_generator.py::TestTableSQL -v

# Run with verbose output
pytest -xvs

Test Status:

  • ✅ 124 passing tests (91.2% of total)
  • ⏸️ 12 skipped tests (8.8%)
    • 6 tests: Governance SQL generation (issue #19)
    • 2 tests: Constraint SQL fixes (issue #20)
    • 4 tests: Integration tests (blocked by above)

Linting and Type Checking:

# Run linter
ruff check src/

# Auto-fix linting issues
ruff check --fix src/

# Type checking
mypy src/

SQL Validation (Optional):

SchemaX includes optional SQLGlot integration for validating generated SQL:

# Install SQLGlot
pip install "schemaxpy[validation]"
# or
pip install sqlglot>=20.0.0

# Use in tests or scripts
python -c "
import sqlglot
sql = 'ALTER TABLE \`catalog\`.\`schema\`.\`table\` ADD COLUMN \`id\` BIGINT'
parsed = sqlglot.parse_one(sql, dialect='databricks')
print('Valid SQL!' if parsed else 'Invalid SQL')
"

Manual Testing:

# Navigate to test project
cd ../../examples/basic-schema

# Test CLI commands
schemax validate
schemax sql
schemax sql --output test-migration.sql

Project Structure

schemax-vscode/
├── packages/
│ ├── vscode-extension/ # VS Code Extension
│ │ ├── src/
│ │ │ ├── extension.ts # Extension entry point
│ │ │ ├── storage-v3.ts # Provider-aware storage
│ │ │ ├── providers/ # Provider system
│ │ │ │ ├── base/ # Base interfaces
│ │ │ │ │ ├── provider.ts
│ │ │ │ │ ├── models.ts
│ │ │ │ │ ├── operations.ts
│ │ │ │ │ ├── sql-generator.ts
│ │ │ │ │ └── hierarchy.ts
│ │ │ │ ├── registry.ts # Provider registry
│ │ │ │ ├── index.ts # Auto-registration
│ │ │ │ └── unity/ # Unity Catalog provider
│ │ │ │ ├── index.ts
│ │ │ │ ├── models.ts
│ │ │ │ ├── operations.ts
│ │ │ │ ├── sql-generator.ts
│ │ │ │ ├── state-reducer.ts
│ │ │ │ ├── hierarchy.ts
│ │ │ │ └── provider.ts
│ │ │ ├── shared/ # Legacy (V2 compat)
│ │ │ │ ├── model.ts
│ │ │ │ └── ops.ts
│ │ │ └── webview/
│ │ │ ├── main.tsx # Webview entry
│ │ │ ├── App.tsx # Main component
│ │ │ ├── state/
│ │ │ │ └── useDesignerStore.ts # Provider-aware store
│ │ │ └── components/
│ │ │ ├── Sidebar.tsx
│ │ │ ├── TableDesigner.tsx
│ │ │ ├── ColumnGrid.tsx
│ │ │ └── SnapshotPanel.tsx
│ │ ├── dist/ # Compiled extension
│ │ ├── media/ # Compiled webview
│ │ └── package.json
│ │
│ └── python-sdk/ # Python SDK & CLI
│ ├── src/schemax/
│ │ ├── providers/ # Provider system
│ │ │ ├── base/
│ │ │ │ ├── provider.py
│ │ │ │ ├── models.py
│ │ │ │ ├── operations.py
│ │ │ │ ├── sql_generator.py
│ │ │ │ └── hierarchy.py
│ │ │ ├── registry.py
│ │ │ └── unity/
│ │ │ ├── __init__.py
│ │ │ ├── models.py
│ │ │ ├── operations.py
│ │ │ ├── sql_generator.py
│ │ │ ├── state_reducer.py
│ │ │ ├── hierarchy.py
│ │ │ └── provider.py
│ │ ├── storage_v3.py # Provider-aware storage
│ │ ├── cli.py # CLI commands
│ │ └── models.py # Legacy (V2 compat)
│ └── pyproject.toml

├── docs/ # Documentation
├── examples/ # Working examples
└── scripts/ # Utility scripts

Architecture

Provider-Based System

SchemaX uses a provider-based architecture to support multiple catalog types (Unity Catalog, Hive, PostgreSQL, etc.). Each provider implements a standard interface.

Key Components:

  • Provider Registry - Manages available providers
  • Provider Interface - Standard contract all providers implement
  • Storage V3 - Provider-aware file operations
  • State Reducer - Provider-specific state modifications
  • SQL Generator - Provider-specific DDL generation

See Architecture for detailed architecture documentation.

Extension Host

The extension host (src/extension.ts) runs in Node.js and:

  • Registers VS Code commands
  • Manages file I/O via storage-v3.ts
  • Loads provider from project file
  • Creates and controls the webview
  • Handles message passing with the webview
  • Validates operations using provider

Webview

The webview (src/webview/) is a React application that:

  • Provides the visual designer UI
  • Uses Zustand for state management
  • Stores current provider information
  • Generates provider-prefixed operations
  • Sends operations to the extension via postMessage

Communication Flow (V3)

User Action → Webview (React)
→ Get provider from store
→ Generate Op with provider prefix (e.g., unity.add_catalog)
→ postMessage('append-ops')
→ Extension
→ Get provider from registry
→ Validate operation
→ Append to changelog.json
→ Apply op using provider's state reducer
→ postMessage('project-updated' with provider info)
→ Webview updates UI

Storage Architecture (V3)

SchemaX V3 adds provider awareness to the snapshot-based architecture:

Files:

  • .schemax/project.json - Metadata + provider selection
  • .schemax/changelog.json - Uncommitted provider-prefixed operations
  • .schemax/snapshots/vX.Y.Z.json - Full state snapshots

Loading:

  1. Read project.json → get provider type
  2. Load provider from registry
  3. Read latest snapshot file (or use provider's initial state)
  4. Read changelog
  5. Apply operations using provider's state reducer
  6. Result = current state

Migration:

  • V2 projects automatically migrate to V3 on first load
  • Operations get prefixed with provider ID (e.g., unity.)
  • Provider field added to project.json

Saving:

  • Operations append to changelog.json
  • State changes are computed by applying operations
  • Snapshots freeze the state and clear changelog

See Architecture for detailed design information.

Key Concepts

Operations

Operations are immutable records of user actions:

{
id: "op_uuid",
ts: "2025-10-06T...",
op: "add_column",
target: "col_uuid",
payload: { tableId, colId, name, type, nullable }
}

All schema changes generate operations. Operations are:

  • Append-only (never modified)
  • Tracked with UUIDs
  • Stored in changelog.json
  • Used to generate migrations (future)

Snapshots

Snapshots capture complete schema state:

Metadata (in project.json):

{
id: "snap_uuid",
version: "v0.1.0",
name: "Initial schema",
file: ".schemax/snapshots/v0.1.0.json",
opsCount: 15,
...
}

Full Snapshot (in snapshots/v0.1.0.json):

{
id: "snap_uuid",
version: "v0.1.0",
state: { catalogs: [...] }, // Complete schema
opsIncluded: ["op_1", "op_2", ...],
hash: "sha256...",
...
}

Build System

Extension Build (esbuild)

  • Entry: src/extension.ts
  • Output: dist/extension.js
  • Target: Node.js (CommonJS)
  • Externals: vscode module

Webview Build (Vite)

  • Entry: src/webview/main.tsx
  • Output: media/ directory
  • Target: Browser (ES modules)
  • Bundles: React, Zustand, CSS

Coding Guidelines

Logging

Always use the output channel for logging:

outputChannel.appendLine('[SchemaX] Your message');

Never use console.log() in extension code (it goes to Extension Host console, not visible to users).

File I/O

TypeScript (Extension):

All file operations should go through storage-v3.ts:

import * as storageV3 from './storage-v3';

// Good
const project = await storageV3.readProject(workspaceUri);
const { state, changelog, provider } = await storageV3.loadCurrentState(workspaceUri);
await storageV3.appendOps(workspaceUri, ops);

// Bad - don't access files directly
const content = await fs.readFile('.schemax/project.json');

Python (SDK):

All file operations should go through storage_v3.py:

from schemax.storage_v3 import read_project, load_current_state, append_ops

# Good
project = read_project(workspace_path)
state, changelog, provider = load_current_state(workspace_path)
append_ops(workspace_path, operations)

# Bad - don't access files directly
with open('.schemax/project.json') as f:
project = json.load(f)

State Management

The webview uses Zustand. All mutations must generate operations:

// In useDesignerStore.ts
addColumn: (tableId, name, type, nullable) => {
const op: Op = {
id: `op_${uuidv4()}`,
ts: new Date().toISOString(),
op: 'add_column',
target: colId,
payload: { tableId, colId, name, type, nullable }
};
emitOps([op]); // Sends to extension
}

Data Validation

TypeScript:

All external data must be validated. The storage layer handles validation internally:

import * as storageV3 from './storage-v3';

// Validates and throws if invalid
const project = await storageV3.readProject(workspacePath);

Python:

All external data must be validated. The storage layer handles validation internally:

from schemax.storage_v3 import read_project

# Validates and raises if invalid
project = read_project(workspace_path)

Code Formatting

TypeScript:

  • Use TypeScript strict mode
  • 2 spaces indentation
  • Single quotes for strings
  • Semicolons required
  • Max line length: 100 characters

Python:

  • Use Ruff for linting and formatting (configured in pyproject.toml)
  • Line length: 100 characters
  • Type hints required for all function signatures
  • Follow PEP 8 naming conventions
  • Docstrings for public APIs

Format code before committing:

# Python
cd packages/python-sdk
ruff check --fix src/
ruff format src/

# TypeScript (manual formatting for now)
# Follow existing code style

Testing Checklist

Before submitting a pull request, verify:

VS Code Extension

  • Extension builds without errors (npm run build)
  • No TypeScript errors
  • Designer opens successfully
  • Can create catalogs, schemas, tables, columns
  • Can edit column properties inline
  • Can create snapshots
  • Snapshots persist after reloading
  • Changelog clears after snapshot
  • SchemaX output logs show no errors

Python SDK

  • All tests pass (pytest)
  • No new test failures (124 passing baseline)
  • Test coverage maintained at 30%+ (pytest --cov=schemax --cov-fail-under=30)
  • No linting errors (ruff check src/)
  • Type checking passes (mypy src/)
  • Code is formatted (ruff format src/)
  • CLI commands work (schemax validate, schemax sql)
  • Documentation updated if API changed
  • SQL validated with SQLGlot (if modifying SQL generator)

Note: Coverage threshold is currently set to 30% (baseline). As test coverage improves, this threshold should be gradually increased.

Test Structure:

tests/
├── conftest.py # Shared fixtures
├── utils/
│ └── operation_builders.py # OperationBuilder helper
├── unit/
│ ├── test_storage_v3.py # Storage layer tests
│ ├── test_state_reducer.py # State reducer tests
│ ├── test_sql_generator.py # SQL generation tests
│ └── test_provider.py # Provider system tests
└── integration/
└── test_workflows.py # End-to-end workflow tests

Using Test Helpers:

Tests use OperationBuilder for creating test operations:

from tests.utils import OperationBuilder

builder = OperationBuilder()
op = builder.add_catalog("cat_123", "bronze", op_id="op_001")
# Automatically includes: id, ts, provider, op, target, payload

Both (if applicable)

  • Provider changes implemented in both TypeScript and Python
  • State reducer logic matches between implementations
  • SQL generator produces identical output
  • Operation definitions synchronized

Debugging

Extension Logs

View → Output → Select "SchemaX" from dropdown

Shows:

  • Extension activation
  • File operations
  • Operation appends
  • Snapshot creation
  • Errors with stack traces

Webview Console

  1. With designer open: Help → Toggle Developer Tools
  2. Go to Console tab
  3. Look for [SchemaX Webview] logs

Common Issues

Blank webview:

  • Check webview console for JavaScript errors
  • Verify media/ directory has index.html and assets/
  • Rebuild webview: npm run build:webview

Operations not saving:

  • Check SchemaX output logs
  • Verify .schemax/ directory exists
  • Check file permissions

Snapshots disappearing:

  • Verify project.json has snapshots array
  • Check changelog.json exists
  • Look for errors in output logs

Provider Development

Overview

Adding a new provider to SchemaX involves implementing the provider interface in both TypeScript and Python. This section guides you through the process.

See also: Provider Contract for detailed API documentation.

Quick Start: Adding a New Provider

1. Create Provider Directories

# TypeScript
mkdir -p packages/vscode-extension/src/providers/myprovider
touch packages/vscode-extension/src/providers/myprovider/{index,models,operations,sql-generator,state-reducer,hierarchy,provider}.ts

# Python
mkdir -p packages/python-sdk/src/schemax/providers/myprovider
touch packages/python-sdk/src/schemax/providers/myprovider/{__init__,models,operations,sql_generator,state_reducer,hierarchy,provider}.py

2. Define Provider Metadata

// providers/myprovider/provider.ts
export const myProvider: Provider = {
info: {
id: 'myprovider',
name: 'My Provider',
version: '1.0.0',
description: 'Support for MyDB',
author: 'Your Name',
docsUrl: 'https://docs.mydb.com',
},
capabilities: {
supportedOperations: ['myprovider.add_database', 'myprovider.add_table'],
supportedObjectTypes: ['database', 'table'],
hierarchy: myProviderHierarchy,
features: {
constraints: true,
rowFilters: false,
columnMasks: false,
columnTags: false,
},
},
// ... implement interface methods
};

3. Define Hierarchy

// providers/myprovider/hierarchy.ts
export const myProviderHierarchy = new Hierarchy([
{
name: 'database',
displayName: 'Database',
pluralName: 'databases',
icon: 'database',
isContainer: true,
},
{
name: 'table',
displayName: 'Table',
pluralName: 'tables',
icon: 'table',
isContainer: false,
},
]);

4. Define Models

// providers/myprovider/models.ts
import { z } from 'zod';

export const MyDatabase = z.object({
id: z.string(),
name: z.string(),
tables: z.array(MyTable),
});

export const MyTable = z.object({
id: z.string(),
name: z.string(),
columns: z.array(MyColumn),
});

export const MyProviderState = z.object({
databases: z.array(MyDatabase),
});

export type MyDatabase = z.infer<typeof MyDatabase>;
export type MyTable = z.infer<typeof MyTable>;
export type MyProviderState = z.infer<typeof MyProviderState>;

5. Define Operations

// providers/myprovider/operations.ts
export const MY_PROVIDER_OPERATIONS = {
ADD_DATABASE: 'myprovider.add_database',
RENAME_DATABASE: 'myprovider.rename_database',
DROP_DATABASE: 'myprovider.drop_database',
ADD_TABLE: 'myprovider.add_table',
// ... more operations
};

export const myProviderOperationMetadata: OperationMetadata[] = [
{
type: MY_PROVIDER_OPERATIONS.ADD_DATABASE,
displayName: 'Add Database',
description: 'Create a new database',
category: OperationCategory.DATABASE,
requiredFields: ['databaseId', 'name'],
optionalFields: [],
isDestructive: false,
},
// ... more metadata
];

6. Implement State Reducer

// providers/myprovider/state-reducer.ts
export function applyOperation(
state: MyProviderState,
op: Operation
): MyProviderState {
// Deep clone for immutability
const newState = JSON.parse(JSON.stringify(state));

// Strip provider prefix
const opType = op.op.replace('myprovider.', '');

switch (opType) {
case 'add_database': {
const database: MyDatabase = {
id: op.payload.databaseId,
name: op.payload.name,
tables: [],
};
newState.databases.push(database);
break;
}

case 'add_table': {
const db = newState.databases.find(d => d.id === op.payload.databaseId);
if (db) {
const table: MyTable = {
id: op.payload.tableId,
name: op.payload.name,
columns: [],
};
db.tables.push(table);
}
break;
}

// ... more cases
}

return newState;
}

7. Implement SQL Generator

// providers/myprovider/sql-generator.ts
export class MyProviderSQLGenerator extends BaseSQLGenerator {
generateSQLForOperation(op: Operation): SQLGenerationResult {
const opType = op.op.replace('myprovider.', '');

switch (opType) {
case 'add_database':
return {
sql: `CREATE DATABASE IF NOT EXISTS ${this.escapeIdentifier(op.payload.name)}`,
warnings: [],
isIdempotent: true,
};

case 'add_table':
const dbName = this.idNameMap[op.payload.databaseId];
const tableName = op.payload.name;
return {
sql: `CREATE TABLE IF NOT EXISTS ${dbName}.${this.escapeIdentifier(tableName)} ()`,
warnings: [],
isIdempotent: true,
};

// ... more cases
}
}
}

8. Implement Provider Class

// providers/myprovider/provider.ts
export class MyProvider extends BaseProvider {
get info(): ProviderInfo {
return {
id: 'myprovider',
name: 'My Provider',
version: '1.0.0',
description: 'Support for MyDB',
author: 'Your Name',
docsUrl: 'https://docs.mydb.com',
};
}

get capabilities(): ProviderCapabilities {
return {
supportedOperations: Object.values(MY_PROVIDER_OPERATIONS),
supportedObjectTypes: ['database', 'table'],
hierarchy: myProviderHierarchy,
features: {
constraints: true,
rowFilters: false,
columnMasks: false,
columnTags: false,
},
};
}

createInitialState(): ProviderState {
return { databases: [] };
}

applyOperation(state: ProviderState, op: Operation): ProviderState {
return applyOperation(state as MyProviderState, op);
}

getSQLGenerator(state: ProviderState): SQLGenerator {
return new MyProviderSQLGenerator(state);
}

validateOperation(op: Operation): ValidationResult {
// Implementation
}

validateState(state: ProviderState): ValidationResult {
// Implementation
}
}

9. Register Provider

// providers/myprovider/index.ts
export * from './models';
export * from './operations';
export * from './provider';

import { MyProvider } from './provider';
export const myProvider = new MyProvider();

// providers/index.ts - Add to auto-registration
import { myProvider } from './myprovider';

export function initializeProviders() {
ProviderRegistry.register(unityProvider);
ProviderRegistry.register(myProvider); // NEW!
}

initializeProviders();

10. Python Implementation

Follow the same structure in Python:

# packages/python-sdk/src/schemax/providers/myprovider/provider.py
class MyProvider(BaseProvider):
@property
def info(self) -> ProviderInfo:
return ProviderInfo(
id="myprovider",
name="My Provider",
version="1.0.0",
description="Support for MyDB",
author="Your Name",
docs_url="https://docs.mydb.com",
)

# ... implement all methods

Testing Your Provider

1. Create Test File

// providers/myprovider/__tests__/myprovider.test.ts
import { testProviderCompliance } from '../../base/__tests__/provider-compliance.test';
import { myProvider } from '../provider';

// Run compliance tests
testProviderCompliance(myProvider);

// Provider-specific tests
describe('MyProvider', () => {
test('creates initial state', () => {
const state = myProvider.createInitialState();
expect(state.databases).toEqual([]);
});

test('applies add_database operation', () => {
// ... test implementation
});
});

2. Run Tests

# TypeScript
cd packages/vscode-extension
npm test

# Python
cd packages/python-sdk
pytest tests/providers/myprovider/

Common Patterns

Immutable State Updates:

// ❌ BAD - Mutates state
state.databases.push(newDb);

// ✅ GOOD - Creates new state
const newState = {
...state,
databases: [...state.databases, newDb],
};

Operation Validation:

validateOperation(op: Operation): ValidationResult {
const errors: ValidationError[] = [];
const metadata = this.getOperationMetadata(op.op);

// Check required fields
for (const field of metadata.requiredFields) {
if (!op.payload[field]) {
errors.push({
field: `payload.${field}`,
message: `Required field missing: ${field}`,
code: 'MISSING_REQUIRED_FIELD',
});
}
}

return { valid: errors.length === 0, errors };
}

SQL Escaping:

// Always escape identifiers
this.escapeIdentifier(name); // `name`

// Always escape strings
this.escapeString(value); // 'value'

Provider Development Checklist

Before submitting a provider:

  • All required interface methods implemented
  • Provider registered in registry
  • Hierarchy defined
  • All operations have metadata
  • State reducer handles all operations
  • State reducer is pure (no side effects)
  • SQL generator handles all operations
  • SQL is idempotent (safe to run multiple times)
  • Operation validation implemented
  • State validation implemented
  • Provider compliance tests pass
  • Provider-specific tests written
  • Python implementation matches TypeScript
  • Documentation written
  • Examples provided

Resources

  • Provider Contract: Provider Contract
  • Architecture: Architecture
  • Unity Provider: Reference implementation in providers/unity/
  • Base Provider: Abstract base in providers/base/

Contributing

Reporting Issues

When reporting bugs, include:

  • Steps to reproduce
  • Expected vs actual behavior
  • SchemaX output logs
  • Webview console errors (if any)
  • VS Code version
  • Operating system

Pull Requests

  1. Fork the repository
  2. Create a feature branch: git checkout -b feature/your-feature
  3. Make changes and test thoroughly
  4. Commit with clear messages
  5. Push and create a pull request

Code Style

  • Use TypeScript strict mode
  • Follow existing code formatting
  • Add comments for complex logic
  • Keep functions focused and small
  • Use descriptive variable names

Release Process

  1. Update version in package.json
  2. Update CHANGELOG.md
  3. Build: npm run build
  4. Package: npm run package
  5. Test the .vsix file
  6. Create GitHub release
  7. Publish to marketplace

Resources

Getting Help

  • Check docs/ for other documentation
  • Review .cursorrules in the repo root for project context (or see GitHub)
  • Open an issue on GitHub
  • Review existing issues and discussions