Both HANA Studio and HANA Cockpit provide administrative tools to visualize memory usage by service. If these tools are not available, you can query the M_SERVICE_MEMORY System View using the hdbsql command line program.
SELECT * FROM SYS.M_SERVICE_MEMORY
If your HANA system contains multiple tenant databases, you will need to login to each tenant database and execute the query above. There are two fields from the query result that are of interest (EFFECTIVE_ALLOCATION_LIMIT, TOTAL_MEMORY_USED_SIZE).
The effective allocation limit of a service indicates how much physical memory a service can use given the current memory usage of other services. The total memory used size is the total memory being used by the service. You should ensure that TOTAL_MEMORY_USED_SIZE is below the EFFECTIVE_ALLOCATION_LIMIT. If the total memory used begins to reach the allocation limit, it may lead to out-of-memory failure.