Standards and Conventions

CEX

To ensure data consistency and reliability, the following standards and conventions are followed:

1. Timestamps:

   • Fee details are stored in JSON format for compatibility across exchanges.

   • Each trade fee is linked to its respective trade to ensure accurate calculations.

2. Data Integrity:

   • Each trade has a unique id that differentiates it from others.

   • symbol values follow the standard market notation (e.g., BTC/USD, ETH/USD).

3. Trade Side Representation:

   • The side field is always either "buy" or "sell" to indicate trade direction.

4. Order Execution Types:

   • The type field supports various order types, including:

     • Market Order: Executes immediately at the best available price.

     • Limit Order: Executes at a specific price or better.

     • Stop-Limit Order: Becomes a limit order once the stop price is reached.

5. Fees and Costs:

   • Fee details are stored in JSON format for compatibility across exchanges.

   • Each trade fee is linked to its respective trade to ensure accurate calculations.

DEX

Data Normalization

Normalization simplifies DEX data by standardizing formats across multiple chains and protocols.

This section explores these processes, trade-offs, and the reasoning behind our schema design.

Cross-Chain Standardization

– Ethereum: Timestamps are captured directly from block data.

– Solana: Timestamps are converted from UNIX format to ISO-8601.

– Token amounts are normalized according to their decimal places.

– Prices are standardized to USD value using oracle feeds.

Pool Liquidity Normalization

– Total Value Locked (TVL) is calculated in USD.

– Volume metrics are normalized to 24-hour periods.

– Fee revenues are converted to APR values.

– Position concentrations are expressed as percentile distributions.

Transaction Feature Engineering

– Price impact calculations using constant product formulas.

– Slippage measurements comparing quoted vs. executed prices.

– Gas costs in USD for Ethereum transactions.

– Fee costs in USD for Solana transactions.

Time Series Resampling

– 1-minute candles for high-frequency analysis.

– 5-minute, 15-minute, hourly, and daily aggregations.

– Volume-weighted average prices (VWAP) calculated per interval.

– Liquidity depth measurements at standard intervals.

Simulated Order Book Construction

– AMM curve discretization into bid/ask levels.

– Concentrated liquidity position mapping to limit orders.

– Fee tier normalization across platforms.

– Multi-pool aggregation for complete market depth.

Pricing Methodology

– Market prices derived from the latest swap execution.

– Theoretical prices calculated from pool reserves.

– Slippage curves modeled for various trade sizes.

– Cross-DEX price normalization for arbitrage detection.

Data Quality Assurance

– Automated validation of schema compliance.

– Outlier detection for anomalous price or volume data.

– Chain reorganization handling with transaction finality checks.

– Missing data imputation for analytics continuity.