We haven’t had this question in a while, but it’s a valid and interesting one.
For safe measure, I usually tell folks to have 10x the storage of their input data, so 8x makes sense as an actual value you’re seeing.
You’ve got a couple options, but the simplest is to add the optimize-disk-space flag:
Delete heavy intermediate files to optimize disk space usage. This affects the ability to restart the pipeline from an intermediate stage, but allows datasets to be processed on machines that don’t have sufficient disk space available. Default:
False