Skip to main content

 

Summary Orbix service raises an error in the PSS layer "errno is 12 (not enough space)"
Article Number 13412
Environment

Orbix 6.x

All Supported Operating Systems

Question/Problem Description An Orbix service raises error in PSS layer:
"errno is 12 (not enough space)"

At runtime, the Orbix service raises the following error in its log file: "errno is 12 (not enough space)"

A possible full error message for the IFR could read:
 
ERROR: DB del failed; env is ifr_store, db is IRObjectPSHomeImpl:1.0, errno is 12 (Not enough space)

This error can also occur on any of the Orbix services, e.g in the CFR:

E - DB del failed; env is cfr_store, db is IT_CFRStoreImpl/BindingHomeImpl:1.0, errno is 12 (Not enough space)


The problem only occurs under higher load.

Clarifying Information  
Error Message  
Defect/Enhancement Number  
Cause The underlying Berkeley DB is running out of locks and lock objects. The default value is 1000. By increasing the maximum number of locks available to the Berkeley database, the problem can be overcome.
Resolution

If the error is happening for the interface repository (IFR), change this variable in the configuration scope of your Orbix service:

 plugins:pss_db:envs:ifr_store:lk_max_locks = "10000";

 plugins:pss_db:envs:ifr_store:lk_max_lockers = "10000";

 plugins:pss_db:envs:ifr_store:lk_max_objects = "10000";

The same variable might be set for other services. For the naming service, it would be:

  plugins:pss_db:envs:it_naming_store:lk_max_locks = "10000";

  plugins:pss_db:envs:it_naming_store:lk_max_lockers = "10000";

 plugins:pss_db:envs:it_naming_store:lk_max_objects = "10000";

Changing the maximum allowed number of locks and lock objects to 10000 typically resolves the problem. However if you want to really find out what the required max number of locks for your system is, then monitor the amount of locks used for the database using the Berkeley DB tool db_stat that can be downloaded from the Berkeley DB website.

Workaround  
Notes The Berkeley DB documentation page can be found at http://www.oracle.com/technetwork/database/berkeleydb/documentation/index.html and the documentation page for the Berkeley DB command line utilities (including db_stat) can be found at http://download.oracle.com/docs/cd/E17076_02/html/api_reference/C/utilities.html

To monitor the locks db_stat must be used, this can easily be built using the Berkeley source, just ensure you build the same version of db_stat as is the version of the database.

After that the configuration will need to be changed in order to use db_stat to monitor the locks. The reason for this is that by default DB_PRIVATE is set for all Orbix databases. This flag basically only allows one process to access the database at a time, which means if the database and application are up and running then it is not possible to simultaneously get the information using db_stat. This is easy to overcome by setting the following in the Orbix configuration file - the example here is for naming database .....

plugins:pss_db:envs:it_naming_store:private = "false";

Now after re-starting the services the lock usage can be determined by using the following command (it is a requirement that db_stat is run from where it was built rather than copied away from there):

/users/dublin/rd 29> /vob/tools_src/db-4.5.20/build_unix/db_stat -c -h /local4/users/rd/6.3.4/product/var/simples/dbs/naming_pivo
28 Last allocated locker ID
0x7fffffff Current maximum unused locker ID
9 Number of lock modes
50000 Maximum number of locks possible
50000 Maximum number of lockers possible
50000 Maximum number of lock objects possible
1360 Number of current locks
1370 Maximum number of locks at any one time
17 Number of current lockers
20 Maximum number of lockers at any one time
5 Number of current lock objects
15 Maximum number of lock objects at any one time
444356 Total number of locks requested
442996 Total number of locks released
0 Total number of locks upgraded
5 Total number of locks downgraded
24484 Lock requests not available due to conflicts, for which we waited
1355 Lock requests not available due to conflicts, for which we did not wait
0 Number of deadlocks
0 Lock timeout value
0 Number of locks that have timed out
0 Transaction timeout value
0 Number of transactions that have timed out
16MB 768KB The size of the lock region
5950 The number of region locks that required waiting (0%)

Attachment  

#Orbix
#KnowledgeDocs
#Orbix6
#database