Required executor memory (1024), overhead (384 MB), and PySpark memory (0 MB) is above the max threshold (1024 MB) of this cluster!
Getting this? java.lang.IllegalArgumentException: Required executor memory (1024), overhead (384 MB), and PySpark memory (0 MB) is above the max threshold (1024 MB) of this cluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'. Either increase the Container Memory: yarn.nodemanager.resource.memory-mb to 2GB from 1GB Or reduce the maximum container memory: yarn.scheduler.maximum-allocation-mb from 1GB to 512MB in the […]