A buffer pool manger is an important feature in database systems, introduced to improve performance.
If a database block is required, it is not directly read from disk, but it is requested from the buffer pool manager. The buffer pool manager keeps database blocks in memory - the buffer pool. If a database block is requested, the buffer pool manager checks whether the requested block is already in the buffer pool. If yes, it is not necessary to read it again from disk, and it can directly be returned to the caller. Only if the block is not yet in the buffer pool, does it have to be read from disk. Because typically most requested database blocks are used relatively often, this way, the number of physical disk I/Os can be reduced dramatically. A buffer pool efficiency defined as (1 - physical I/Os / logical I/Os) >= 99% is often achieved.
Adabas also uses the buffer pool for storing temporary blocks (NUCTMP, NUCSRT). These blocks are only relevant for the current nucleus session and for allocating internal work areas.
Not only the number of read I/Os is reduced with the buffer pool, also the number of write I/Os: if database blocks are logically written to the buffer pool, they are not written directly to disk, but only if a certain number of blocks is marked as modified - then a “buffer flush” is performed. Because the same database block is often updated several times in quick succession, this way the number of write I/Os is reduced. Nevertheless, it must be guaranteed that no transactions are lost in case of a nucleus crash, for example because of a power failure. This is achieved via a log, that is required anyway in order to be able to roll back transactions. In Adabas, the WORK container is used for this purpose.
Buffer flushes are normally performed in parallel to the normal processing of the Adabas nucleus by a separate buffer flush thread. Therefore, the impact of a buffer flush usually is not very big, but the following performance reductions are possible:
Because of the high I/O load generated by a buffer flush, other I/Os performed by the nucleus may become slower. As a result, commands requiring I/Os may become slower.
I/Os may also result in additional CPU load, for example for copying the data to I/O buffers or for encrypting the data, if data is written encrypted to disk.
The amount of performance degradation caused by buffer flushes is very much system dependent; in some environments it may be nearly zero, while in other environments it may be noticeable or even disturbing.
Usually you want to keep the amount of modified database blocks not yet written to disk relatively small in order to keep the time small that is required for the regeneration of these blocks from the log on the WORK. In contrast, temporary blocks are relevant only for the current Adabas nucleus session and therefore need not be regenerated during a nucleus crash and should only be written to disk in order to avoid a buffer overflow.
For this reason, we have defined two separate flush lists (lists of blocks to be written to disk): one for database blocks and one for temporary blocks. These flush lists also result in separate buffer flushes for database blocks and for temporary blocks. If you have enough memory available, you should use a buffer pool size that temporary blocks need not be written to disk at all.
There are the following reasons for performing buffer flushes:
Explicit buffer flushes: in some situations, Adabas explicitly starts a buffer flush, for example the nucleus performs an explicit buffer flush after it has created the checkpoint entry during the nucleus start, or during shutdown processing. Some utilities initiate an explicit buffer flush because they read Adabas files directly from disk, and therefore all modified database blocks of these files in the buffer pool must be written to disk before the utility can start processing.
The write limit is exceeded. There are different write limits for database blocks and for temporary blocks: The write limit for database blocks can be defined via the nucleus parameter WRITE_LIMIT; for temporary blocks, the write limit is 50%.
WORK limit buffer flush: blocks not yet written to disk must be regenerated via the log records stored on the WORK. If too many WORK blocks are required for the regeneration of these blocks, a buffer flush is started to avoid a WORK overflow.
Wait-for-space buffer flush: there is no more free space in the buffer pool to allocate another block. If this happens, the command processing waits until the buffer flush is finished. Because this can result in large command execution times, this should be avoided: reducing the write limit or increasing the buffer pool, if enough memory is available.
Emergency buffer flush: if the WORK really overflows, the command processing stops until the buffer flush is finished. Because this can result in large command execution times, this should be avoided: reducing the write limit or increasing the WORK may help.
Ignored blocks buffer flush: When a modified block is being updated again just when the buffer flush tries to write the block to disk, the buffer flush ignores this block and continues with the next block to be written. However, in order to avoid emergency buffer flushes, directly after the termination of the buffer flush, another buffer flush is initiated to get these ignored blocks written to disk as well.
An important issue related to the buffer pool management is the autorestart processing: An autorestart is performed during nucleus start after an abnormal termination of the nucleus, for example because of a power failure. During the autorestart, the blocks that were in the buffer pool at the abnormal nucleus termination, but not yet written to disk, are regenerated and all open transactions are rolled back. For production database usually the maximum autorestart times should be kept small, because the autorestart times mean production downtime. The following is relevant for the autorestart time:
The write limit for database blocks (nucleus parameter WRITE_LIMIT). The higher the write limit, the more modified blocks may be in the buffer pool that must be regenerated during a buffer flush. This leads to longer autorestart times.
For some applications, the size of the WORK container is important. The WORK is used to log the database modification commands for the autorestart processing. If the same database blocks are updated very frequently, it can happen that a large number of WORK blocks are written to log the update operations. The more WORK blocks are written before a buffer flush occurs, the more update operations must be regenerated. This leads to longer autorestart times. In such a case, you can reduce the autorestart times by using a smaller WORK, because then a Work autorestart is performed, before the write limit triggers a buffer flush.
You can imagine that the configuration of the buffer pool has a large impact on the overall database performance. There are the following Adabas nucleus parameters for this purpose:
BFIO_PARALLEL_LIMIT - Buffer flush I/O parallel limit. This parameter limits the number of I/Os that are started in parallel. If this value is too high, a read I/O done in parallel to the buffer flush, may become very slow, because before the read operation, at first the write I/Os are performed. The optimal value depends very much from the I/O optimizations done by the storage system and the operation system, but in most cases the default value of 50 will result in a good performance. This parameter can be changed dynamically with the utility ADAOPR while the nucleus is running.
LBP - Length of buffer pool: You should try to define the buffer pool large enough, that you achieve a buffer pool efficiency >= 99%. It is also desirable that the buffer pool has a size where no buffer flushes for temporary blocks are performed. On the other hand, the optimal buffer pool size is limited by the available memory; LBP must not be so large that the operating system starts paging.
READ_PARALLEL_LIMITS allows you to read ahead database blocks for sequential processing. This parameter can be changed dynamically with the utility ADAOPR while the nucleus is running. However, this increases the performance only if most of the Adabas commands process the database sequentially, for example in batch programs. Otherwise, you read blocks that are never used. Therefore it is recommended to switch this parameter on only explicitly when you run programs that really take advantage of the reading ahead.
WRITE_LIMIT. This parameter specifies how many database blocks are modified until a buffer flush is started:
A larger value will usually reduce the number of write I/Os: Assume a block is updated twice. With the smaller WRITE_LIMIT, it may happen that the block is already written after the first update and before the second update, while with the larger WRITE_LIMIT, it takes longer
On the other hand, a larger value of WRITE_LIMIT also means that in case of a nucleus crash, for example after a power failure, there may be more modified blocks in the buffer pool that have not yet been written to disk. These blocks must be recreated in the “autorestart” performed when the database is restarted again. This means tt takes longer until the database is up again after the failure.
A smaller value for WRITE_LIMIT may be useful to avoid wait-for-space and emergency buffer flushes. Then a buffer flush may be triggered earlier, and modified blocks can be written to disk before they could cause a wait-for-space or emergency buffer flush.
Data-at-rest encryption requires additional space for encryption purposes. The required size depends on the parameters LBP and WRITE_LIMIT. The size amounts to ‘2 * LBP * WRITE_LIMIT’ and must not be greater than 4GB. Otherwise, ADANUC terminates with an error during startup.
For example, a buffer pool of 20GB with a write limit of 10% will result in 4GB of reserved encryption space within the buffer pool.
Note
To use the encryption functionality, the license for Adabas Encryption for
Linux (AEL) is required.
Because the WORK container is used to log the database modifications and is required for an autorestart, also the WORK size is important for the buffer pool management. Please consider the following when you define the WORK size of a database:
The WORK must be large enough to log all open transactions. If it is too small, transactions are rolled back, and an Adabas response code 9, subcode 3, “LP” is returned. If this happens too often, and not only for transactions that are open for a long time, you should increase the WORK.
A lot of log space on WORK is required in particular if you use the following features:
Large object (LOB) values. For the insert or delete of a LOB value, the additional WORK space required is the size of the LOB value plus some administration data; for an update of a LOB value you need the double amount of space.
Adabas-to-Adabas replication. All database modifications are logged in Adabas files, the replication system files, and the modifications in the replication system files are also logged on WORK. The WORK space requirements with replication may be up to a factor 4 compared to the WORK requirements without replication.
If the same database blocks are updated very often, it may happen that a buffer flush must be triggered to avoid a WORK overflow, before the write limit is exceeded. If this happens with a large WORK container, the WORK size may be a significant factor for the autorestart time, and it may be useful to reduce the WORK size. However, you should not reduce the WORK size so much that transactions must be rolled back.
If a WORK buffer flush is triggered, and the update load for the database is high it can happen that an emergency buffer flush becomes necessary, because the database blocks are not written to disk fast enough. This should be avoided, because this can significantly reduce the Adabas performance. Either a larger WORK or a lower write limit may help.
When you define a database container, the block size can be any multiple of 1 KB up to 32 KB; NUCTMP blocks generally have a block size of 4 KB.
However, it is recommended to use only block sizes that are multiples of each other, in particular it is recommended to use block sizes that are powers of 2. Otherwise the buffer pool may become fragmented. It may happen that you cannot use the complete space in the buffer pool.
For monitoring the buffer pool behavior (and generally the database behavior), you can use:
The ADAOPR DISPLAY command, in particular DISPLAY=BP_STATISTICS and DISPLAY=BF_STATISTICS.
The ADAMON utility.
For more information see the Utilities documentation.
