ai-data-extractor
Free open-source extractor for AI coding assistant chat histories. Supports Claude Code, Cursor, Windsurf, Aider, Cline/Roo Code, and more.
What is it?
What it is
A Python command‑line toolkit that auto‑discovers and extracts chat histories from many AI coding assistants (Claude, Cursor, Windsurf, Trae, Continue, Gemini, etc.) and normalizes them into a single JSONL format.
Why it exists
To let users back up, analyze, or fine‑tune on their own local conversation data before an app’s storage is cleared, and to provide consistent access to data from tools with differing storage formats.
Who should use it
AI researchers, ML engineers, and data scientists who need to extract and normalize conversation data from various AI coding assistants for fine-tuning language models, personal analytics, or data backup purposes. Also suitable for developers who want to audit or migrate their AI assistant conversation history.
Who should avoid it
Users seeking a graphical interface or web-based solution, those without Python programming experience, or individuals looking for real-time data extraction from running AI applications. Also not suitable for users who need to extract data from AI tools not listed in the supported sources.
How it works
A quick walkthrough in plain English
How ai-data-extractor works in 4 steps
Step 1 of 4
Something triggers the flow
A schedule, webhook, or manual click tells ai-data-extractor to start.
Features
Advantages
- Unified backup of years of AI coding conversations for fine‑tuning, analytics, or archiving
- Preserves rich context (code snippets, diffs, tool usage) that raw chat logs lack
- Zero external dependencies – runs with Python 3.9+ standard library only
- Cross‑platform convenience; no need to manually locate tool data directories
- Robust error handling yields partial results even if some files are locked or malformed
- Easy to script for CI/CD or regular backups (`--all`, `--merge`, custom search paths)
- MIT‑licensed, open‑source toolkit that can be forked or extended
- Provides a ready‑to‑use dataset format for training ML models (e.g., Hugging Face datasets)
- Clear separation of concerns – each tool’s extractor is isolated and testable
- Privacy‑first approach with built‑in warnings and secret‑scanning instructions
Disadvantages
- Heuristic extraction for Windsurf and Trae may miss conversations if the storage schema changes
- Aider extraction relies on scanning project directories; non‑standard locations require `--search-path`
- Not every tool stores all metadata (e.g., `code_context`, `tool_use`, `project_path` are optional)
- Future updates to AI tools could break extraction logic, requiring manual extractor updates
- Potential to inadvertently capture sensitive information or proprietary code if not scanned
- Requires Python 3.9+ and a writable output directory; no pre‑installed binary package
- Large histories can be slow to process; no built‑in throttling or progress bars
- Heuristic key‑matching may produce false positives or miss chats in undocumented formats
- No guarantee of extracting every conversation if the tool’s storage layout is non‑standard or hidden
Installation
cloud
No cloud installation instructions found in README
docker
No Docker installation instructions found in README
native
python --version # 3.9+ required, 3.10+ recommended # Interactive: pick which sources to extract from a numbered menu python extract.py # Or drive it directly python extract.py --all python extract.py --sources cursor,claude_code,aider python extract.py --list # just show what's installed, don't extract python extract.py --all --merge # also write all_conversations.jsonl # Shorthand for "extract everything" ./extract_all.sh
compose
No Docker Compose installation instructions found in README
FAQ
How can I extract conversations from all supported tools at once?
Run `python extract.py --all` (or use the convenience script `./extract_all.sh`). This will automatically detect every installed source, extract its history, and write timestamped JSONL files under `extracted_data/`.
What does the extracted JSONL look like and where is it saved?
Each run creates a timestamped file per source under `extracted_data/` (e.g., `claude_code_conversations_20260816_143022.jsonl`). Every line is a JSON object with a `messages` array, plus fields like `source`, `session_id`, `project_path`, `name`, and `created_at`. The schema is documented in the README under **Output format**.
How do I add a new AI tool to the extractor?
Copy an existing extractor module (e.g., `continue_ext.py`), implement `find_installations` and `extract` functions, fill in `DISPLAY_NAME` and `SOURCE_ID`, then add your module to the `REGISTRY` list in `extract.py`. The `extractors/common.py` helpers handle JSONL, JSON, and SQLite reading.
What steps should I take to protect sensitive data in the extracted conversations?
1. Scan the output files for secrets: `detect-secrets scan extracted_data/*.jsonl`. 2. Review `code_context` and `tool_use` fields for proprietary code, API keys, or personal paths. 3. Never commit `extracted_data/` to a public repo (it’s already in `.gitignore`). Keep the directory on encrypted storage if it contains client or proprietary work.
How does the tool handle locked databases (e.g., Cursor) and what if no conversations are found?
All SQLite reads are opened with `mode=ro` so a running editor won’t block extraction. If you still see errors, close the app and re‑run. If an installation is found but no conversations are extracted, the heuristic key‑matching may not match the current schema. Use `--list` to confirm the app directory was located, inspect the DB keys directly (`sqlite3 state.vscdb "SELECT key FROM ItemTable"`), and update `KEY_HINTS` in the relevant extractor file.
Featured in Videos
YouTube tutorials and walkthroughs for ai-data-extractor
Alternatives
Similar projects ranked by category, topics, and text overlap.
