What is Data Execution Prevention (DEP) and how does it prevent shellcode execution?
Data Execution Prevention (DEP) is a security feature introduced in Windows XP SP2 that marks memory pages containing data as non-executable. When a stack overflow successfully redirects execution to shellcode in the data segment, DEP triggers a CPU exception instead of executing the malicious instructions. This makes direct shellcode execution impossible, as explained in Windows Shellcode Study Notes - Bypassing DEP via VirtualProtect.
DEPData Execution Preventionshellcodestack overflowmemory protection