The vendor CSV spec is a suggestion. Production parsers assume extra columns, weird encodings, and a header that moved last Thursday.
Tags: Python, Data Engineering, Vendors, ETL, unlisted
I stopped treating vendor files as contracts. I treat them as weather. The parser sniffs delimiter and encoding, maps columns by fuzzy header names, and refuses to run if required fields are missing.
A dry-run mode shows row counts and five sample mapped records before write. That preview has saved more relationships than any angry email about their spec.
Version the mapping, not your dignity
When they add a column, the mapping table changes. The script does not grow another if-brand-x block if I can help it.
I hash the file and store it. When they swear they sent the same file, we can know.
Partial success is a decision
Some imports can commit the good rows and quarantine the rest. Some cannot. Make that explicit per vendor. Silent partials are how finance and the floor disagree until month end.
