I am not sure whether you are aware about the major changes in Spark 3.x versions but in the 3.x versions there is something called as Adaptive Query Execution which can be used for the multiple use cases when enabled. Like for example, if I consider your point where you are talking about the number of partitions, if you are not sure how to calculate the actual number of partitions needed for your dataset then you can just leave on AQE to take care of this.