You can create your custom ESP-IDF board template based on an existing one or exported code, and editing ‘sdkconfig’ in it, or set it with ‘idf.py menuconfig’ according to your memory/device requirements. It might need some trials with different settings until it works. Especially if the display-diver is not supported by default in ESP-IDF and you need to integrate a driver into the source.
A similar topic including our tutorial on creating custom board templates: How do I create a project for a custom STM32 board?
And a successful custom ESP32 board template as a possible example (albeit with different display/etc) for you:
Sunton ESP32 5.0" RGB display board