Hey, I’m still getting used to working with big sets of financial data. Just wondering how others here keep things organized?
Do you store everything in CSVs, use a database, or maybe work mostly in pandas? I’m trying different ways but it gets messy fast 
Would be cool to hear how others do it, especially if you work with EOD data regularly.
For now I’m just using folders + CSVs named by ticker and date. Not perfect, but at least I know where stuff is Will probably switch to a proper DB later.
I mostly use SQLite for local stuff. Easy to query and keeps things clean. CSVs get messy fast when files pile up.
I actually built a small tool around the EODHD API that grabs daily data and dumps it into a PostgreSQL database. Then I use Metabase to visualize some stuff. Bit overkill maybe, but works for me.
I dump everything into pandas DataFrames and just save as pickle files. It’s quick for analysis but yeah, not the best long-term storage if you have tons of tickers.
I’m using a Postgres + Airflow setup: a daily DAG downloads raw data from EODHD, writes it to a staging area, then cleans and loads it into fact tables by ticker. This makes it easier to maintain auditing and quickly spot any “gaps” in the data. As an alternative — has anyone tried MongoDB? I’m curious to hear the pros and cons compared to a relational database.
We used Mongo for raw EODHD dumps — works great for quick ingest, especially when the structure varies.
But querying? Not fun. Aggregations are clunky, and performance drops fast with time-series data.
Ended up keeping Mongo for raw storage only, and moved cleaned data to ClickHouse. Way faster for analytics.
TL;DR — good for ingest, not so great for analysis.