Yeah, this definitely helps. Thanks a lot for your answer @Thiago. So, after observing the maximum shuffle read (using spark web ui) I can calculate and set the numbers accordingly.