| OS / Rust | stable | beta | nightly |
|---|---|---|---|
| Ubuntu 22.04 |  |
 |
 |
| Ubuntu 24.04 |  |
 |
 |
The Investors Exchange (IEX) provides free access to historical datasets such as Top of Book (TOPS) and Depth of Book (DEEP) through its web interface. Unfortunately, downloading these files manually requires clicking through each link — impractical for large-scale research or backtesting. This project provides iex-download, a Rust-based automation tool for fetching these datasets programmatically.
Because with this tool you can fetch over 17 TB of IEX tick data (≈17,548 GB across 4,984 files as of 2026-03-30) in a single run as long as you agree to the IEX Historical Data Terms of Use. In a recent blog post I mentioned only ~6 TB — but that was just the TOPS feed. The full picture includes the depth feeds (DEEP and the recently released DEEP+), which are far larger. Think of trading data like an iceberg: TOPS is the shiny tip (best bid/ask + last trade), while DEEP/DEEP+ contain the mass below the surface — depth, weight, and real research value. That’s where this tool helps you dive in.
Here’s the lay of the land as of 2026 March 30:
|
|
| Feature | TOPS (Top-of-book) | DEEP (Aggregated) | DEEP+ (Order-by-order) |
|---|---|---|---|
| Order granularity | Only best bid/ask + last trade | Aggregated by price level (size summed) | Individual orders (each displayed order) |
| OrderID / update per order | Not present | Not present | Present |
| Hidden / non-display / reserve portions | Not shown | Not shown | Not shown |
| Memory / bandwidth load | Lowest (very compact, minimal updates) | Lower (fewer messages, coarser updates) | Higher (tracking many individual orders, cancels, modifications) |
| Use-cases | Quote feeds, NBBO tracking, top-level liquidity, lightweight apps | General depth, price level elasticity, coarse modelling, liquidity at price tiers | Detailed book shape, order flow-level strategy, detailed execution modelling, microstructure research |
- Progress bar with attitude → because watching terabytes flow should feel satisfying.
- PEG-based date parser → type
2025-01-??or2025-01-02,2025-01-03,2025-01-06,2025-01-2?and it just works, no regex headaches. - One tiny ELF → a single 3.5 MB executable (
-rwxrwxr-x 2 steven steven 3.5M Sep 23 11:00 target/release/iex-download).
No Python venvs, no dependency jungles. Drop it anywhere,chmod +x, and let it rip. - Need details? Just ask my imaginary friend, Manual. He’s got you covered.
man iex-downloadofiex-download --help
Next Steps: From PCAP to HDF5
Downloading is just the first half of the journey. To make the IEX datasets usable for analysis and backtesting, pair this tool with iex2h5.
iex-download→ grabs raw gzipped PCAP files from IEXiex2h5→ converts PCAP streams into efficient HDF5 datasets (RTS/IRTS, statistics, matrices)
Example workflow:
# Download a batch of gzip-compressed PCAP files
iex-download --tops --directory ./data 2016-12-01..2016-12-31
# Convert the IRTS stream into an HDF5 container
iex2h5 -c irts -o iex-archive.h5 ./data/*.pcap.gz
# Convert IRTS streams into daily price matrices sampled every 10 seconds for the month of September 2025
iex2h5 -c rts --time-interval 00:00:10 --date-range 2025-09-01:2025-09-30 -o experiment-001.h5 iex-archive.h5Tip: Don’t try to pull the entire dataset in one go (we’re talking 13+ TB and counting). Instead, download in manageable batches and incrementally add the IrRegular Time Series (IRTS) stream into your HDF5 container. Once caught up to the current trading day, you only need to maintain it with daily updates. That way the commands are tight, the warning about data size is still there, but placed below as advice.
- A recent Rust toolchain (e.g. via rustup)
Clone the repository:
git clone git@github.com:vargalabs/iex-download.git
cd iex-downloadBuild and run:
make && make install“Data provided for free by IEX. By accessing or using IEX Historical Data, you agree to the IEX Historical Data Terms of Use.”