In WHM, there are some limits on the resources you can appply at Home » Service Configuration » Apache Configuration » Memory Usage Restrictions.
There is also some information on how to further customize the settingsin the cpanel forums: https://forums.cpanel.net/threads/cpanel-resource-limit.169942/