(PR3)RMS Global buffer expected behavior? CHAMP Service Request Extract Sequence Number: C010205-002461Owner: 152303 Customer: DSN-UNIX SRQ, HK SYSTEMS (PR3)RMS Global buffer expected behavior? WorkUnit 0001 From: (60867) bill.norton@wcfa.com@norton.60867.dsn ======== Operating System Info : V7.2-1 Hardware Info : AlphaServer ES40 Software Info : We started with a 1.6 M-block indexed file with 11 keys. Added up all the allocations for the index areas, and set that many global buffers on the file. Then $TYPE-ed it on one process and ran an app that accessed it using one of the alternate keys in another process. Some questions came up. 1. When the first process (TYPE) opened the file, we saw the global section creation results with INSTALL LIST/GLOBAL/SUMMARY, and it seemed to have used 23.2 pagelets per global buffer. How is that factor arrived at? Is it a constant, or does it depend on system parameters or user quotas or something? 2. Opening the file seemed to produce a refcount that was 5% of the total pagelet count as seen by INSTALL LIST/GLOBAL. Thinking we'd overestimated the requirement, we reduced the number of global buffers so the number of pagelets would be about the same as the refcount. We were surprised that opening the file now produced a refcount that was 5% of the new, smaller pageletcnt. We had expected that once we had determined the number of global buffers for one user to open the file, that the total pagelet count would reflect that, and each process would end up adding about that many to the refcount. Where are we mistaken? ****************************************************************************** ** Name : Bill Norton Phone : 414-662-4029 Additional : OpenVMS ASE WorkUnit 0002 \+----------------------- Beginning of Screener's Text -----------------------+ \(PR3)RMS Global buffer expected behavior? \** This call was created by the customer using DSNlink. \** The first work unit in the SRQ has more information. \+--------------------------- End of Screener's Text --------------------------+ Working. WorkUnit 0003 Bill, Below I have included multiple articles from our DSNlink database that should help to address your questions. If you have additional questions or problems after reviewing this information, please let us know. Joe Albanese OpenVMS Support Internals/Drivers Rules of Thumb: Number of Buffers For Indexed Files Any party granted access to the following copyrighted information (protected under Federal Copyright Laws), pursuant to a duly executed Compaq Service Agreement may, under the terms of such agreement copy all or selected portions of this information for internal use and distribution only. No other copying or distribution for any other purpose is authorized. Copyright (c) Digital Equipment Corporation 1987, 1991. All rights reserved COMPONENT: Record Management Services (RMS) OP/SYS: VMS SOURCE: Digital Customer Support Center VERSION INFORMATION: Information Applies To: VMS, All Versions OVERVIEW: This information was gathered from the "RMS Structures and Utilities" training class available through Educational Services of Digital Equipment Corporation. *** CAUTION *** Too many global buffers can have a serious negative impact on performance. Depending on the number of active concurrent users, the locking contention on the global section required to scan the GBD (Global Buffer Descriptor) list and the sequential scan of the GBD linked list itself can make performance worse than if the file had no global buffers. Before implementing global buffers on a particular file, always perform benchmarks with and without them. DISCUSSION: General Guidelines: Number Of Buffers For Indexed Files In general, the maximum number of buffers (local or global) that should be used for any index tree to be accessed in an indexed file is the following: o Maximum # buffers = SUM of [(total # index buckets) + 1 (for data)] for each index PLUS the average number of users The "sum" indicated would be across all index trees to be accessed. In the case of local buffers, the average number of users would be equal to 1. This total number provides the user with an "upperbound" estimate. (Refer to Note 1 below.) For moderate to large sized files this upperbound buffer estimate is generally more than the user can afford for local buffering or the system can afford for global buffering. NOTE: As of VMS 4.5, the maximum number of local buffers that can be specified for any file is 127. Algorithms: Estimating Minimum LOCAL and GLOBAL Buffers The following algorithms provide a good starting point for estimating the "minimum" number of LOCAL buffers that should be used. The algorithms described below for "clustered" reads or updates provide a good "lowerbound" estimate for GLOBAL buffers as well, but note that these algorithms assume the average number of users is 1. In practice the actual number of buffers used will fall somewhere in the range between the "lowerbound" and "upperbound" estimates derived. In the case of shared files that have a number of concurrent accessors, every attempt should be made to cache the top-most levels of the index structure in global buffers, particularly for files that have more than two (2) index levels in the primary structure. 1. KEYED READ_ONLY OPERATIONS: A. KEY 0: o Minimum # buffers = (1 for root index bucket) + 1 (data) o lower-bound est. clustered reads = (# levels in index tree) + 1 (data) B. ALTERNATE KEY: o Minimum # buffers = (1 for root index bucket alt key) + 1 (data alt key) + 1 (data key 0) + 1 (RRV) o lower-bound est. clustered reads = (# levels in index tree alt key) + 1 (data alt key) + 1 (data key 0) + 1 (RRV) (Refer to Note 2 below for an explanation of RRV) 2. KEYED (KEY 0) INSERTS OR DELETES (fast delete not enabled): The following estimate should be calculated for each index tree in the file. The minimum number of buffers would be the SUM of the estimates across all index trees: o Minimum # buffers = (# levels in index tree) + 2 (data) (Refer to Note 3 below) 3. KEYED UPDATES (MODIFYs): A. Using Key 0: If the size of the records will not grow larger (i.e., remain the same size or decrease) and no alternate key value will typically be changed by the update: o Minimum # buffers = (1 for root index bucket) + 1 (data) o lower-bound est. clustered updates = (# levels in index tree) + 1 (data) If the size of a record may be expanded by the update, then the possibility that bucket splits may occur must be considered. o Minimum # buffers = (# levels in index tree) + 2 (data) (Refer to Note 3 below) B. Using an Alternate Key: If the update is done using an alternate key, then add 1 to the above Key 0 algorithms to allow for RRVs. (Refer to Note 2 below) SETTING GLOBAL BUFFERS: Global buffers are set on a file-by-file basis either at creation time or at runtime. For example: Creation time ------------- FAB FAB$W_GBC=n $CREATE/FDL=fdl FILE GLOBAL_BUFFER_COUNT=n $CONVERT/FDL=fdl FILE GLOBAL_BUFFER_COUNT=n Runtime ------- FAB FAB$W_GBC=n FDL$PARSE(fdl,...) FILE GLOBAL_BUFFER_COUNT=n DCL $ SET FILE/GLOBAL_BUFFER=n WHERE n = 0 to 32,767 *** CAUTION *** Global Buffers are ignored for process permanent files (DCL OPEN) and when an image is linked using the LINK option keyword IOSEGMENT=NOP0BUFS. NOTES: Note 1: In most cases all index paths down to the level 0 data buckets will not be followed, so the total number of buffers estimated from this algorithm would typically not be needed. Any surplus would in fact be used to cache more data buckets than the one bucket allowed above. Note 2: If the retrieval is done using an alternate key, one additional I/O operation is required if the path to the record to be retrieved from a Key 0, Level 0 data bucket is via an RRV. Note 3: Allow 2 for data to provide for at least a simple bucket split. A bucket split also causes the index structure to have to be updated (or in case of deletion of highest record in data bucket). To locate an index bucket to be updated at any tree level, RMS must re-work its way back down through the tree from the top root index bucket. Requests for bucket locks are only allowed by RMS from one direction -- top down -- otherwise deadlock conditions could occur. This algorithm provides enough buffers to cache the index buckets for the path followed to the current data bucket. [OpenVMS] SYSGEN Parameters Related To RMS Global Buffers Any party granted access to the following copyrighted information (protected under Federal Copyright Laws), pursuant to a duly executed Compaq Service Agreement may, under the terms of such agreement copy all or selected portions of this information for internal use and distribution only. No other copying or distribution for any other purpose is authorized. Copyright (c) 1988, 2000. Compaq Computer Corporation. All rights reserved. PRODUCT: Compaq OpenVMS VAX, All Versions COMPONENT: RMS SOURCE: Compaq Computer Corporation OVERVIEW: There are eight SYSGEN parameters that are directly related to RMS global buffers. They are the following: 1. RMS_GBLBUFQUO* (see note) 5. VIRTUALPAGECNT 2. GBLSECTIONS 6. SYSMWCNT 3. GBLPAGES 7. LOCKIDTBL 4. GBLPAGFIL 8. RESHASHTBL *Note: RMS_GBLBUFQUO is obsolete as of OpenVMS V7.1. Algorithms are provided that can be used to determine the number of units of the first four SYSGEN parameters that are used for RMS global buffer caches. These algorithms require the following two items of information about an RMS file that global buffers are to be added to: 1) Global buffer count (GBC) 2) Maximum bucket size (BKS) NOTE: The "virtual" bucket size for a shared write sequential file is set to the multiblock count (MBC) of the first accessor of the file. The MBC system default is 16. The command procedure SYS$UPDATE:AUTOGEN.COM should be used to adjust the last four parameters in accordance with any modifications made to the first four. DISCUSSION: 1. RMS_GBLBUFQUO (Note: RMS_GBLBUFQUO is obsolete as of OpenVMS V7.1) This is the total number of global buffers over all RMS files allowed to be active simultaneously on a particular system. The need to have a separate SYSGEN parameter for RMS global buffers is explained by the fact that each global buffer requires a system lock. Lock structures are allocated from nonpaged pool (SYSGEN parameter NPAGEDYN). Thus, this quota sets an upper limit on how much of nonpaged pool can be used for system locks on RMS global buffers. RMS_GBLBUFQUO per global cache (file)= GBC When a file is opened that has global buffers specified, the number of global buffers it is requesting is not checked against this quota. This quota is not checked until a global buffer must be used. At that time a check is made to determine whether this quota allows another RMS global buffer to be active. If the quota has been exhausted, then a global buffer is not used. Lack of this resource does NOT cause any error to be returned by RMS (specifically, %RMS-F-CRMP, CRMPSC system service failed to map global buffers). AUTOGEN does not calculate this parameter. 2. GBLSECTIONS This number indicates the number of global section table entries needed in the system header. Each RMS file opened with global buffers requires one GBLSECTION entry (and associated global section descriptor). GBLSECTIONS per global cache (file) = 1 Each global section table entry occupies 32 bytes in the nonpageable area of the system header. Thus, each 19 sections of GBLSECTIONS uses 1 page of nonpaged pool. NOTE: The descriptor associated with the global section is pageable. The AUTOGEN "FEEDBACK" mechanism provides input for this parameter so, if it is specified in the MODPARAMS.DAT file, it should be preceded by a MIN_ or ADD_ prefix. Lack of this resource causes the following error when the record stream is connected to the file: %RMS-F-CRMP, CRMPSC system service failed to map global buffers See note (1) for a workaround to this error. 3. GBLPAGES The maximum number of pages in memory that can be accessed by more than one user. The GBLPAGES used by RMS global buffers are always writable. GBLPAGES per global cache (file) = TRUN [ ( (GBC * BKS * 512) + k + 1023 ) /512 ] + 2 Where k = 64 + ( GBC * 48) 'k' equals the number of bytes needed for the following RMS data structures: Global buffer header (GBH) - 64 bytes: one per global cache (file) Global buffer descriptor (GBD) - 48 bytes: one per global buffer The 1023 bytes listed above are added by RMS for page alignment and internal purposes. The two additional pages (the "+ 2" in the above formula) serve as "stoppers" (limits to modified page write clustering). One is created at the beginning of the section and one at the end. The system page file is used as the backing storage for these pages (with the exception of the 2 stopper pages) when they are removed from the modified page list. GBLPAGES should be generously estimated. It is a cheap parameter. Each 128 entries in the global page table use 4 bytes of nonpaged pool, or another way of expressing it is: each 131,072 pages of GBLPAGES use only 8 pages of nonpaged pool. And the global page table itself is pageable. The AUTOGEN "FEEDBACK" mechanism provides input for this parameter so, if it is specified in the MODPARAMS.DAT file, it should be preceded by a MIN_ or ADD_ prefix. Lack of this resource caused the following error when the record stream is connected to the file: %RMS-F-CRMP, CRMPSC system service failed to map global buffers See note (1) for a workaround to this error. This error occurs if there is an insufficient number of entries remaining in the global page table, but may also occur for a less obvious reason. The entries for a global section in the global page table must be contiguous. Thus, sometimes the number of entries remaining are more than the number needed, but the table is fragmented and does not have the number needed contiguous to each other. The DCL lexical function F$GETSYI can be used to determine whether lack of sufficient contiguous entries is the problem: F$GETSYI ("CONTIG_GBLPAGES") Also note that when the last image mapped to an RMS global section exits, the entries in the global page table do not get immediately freed. The pages mapped through the table must first go to the modified page list and then eventually out to the system paging file. Then the global section can be deleted. Once the global section is deleted, the entries in the global page table are freed up. 4. GBLPAGFIL The maximum number of system-wide pages allowed for global page file sections (or scratch global sections) in the system's PAGEFILE.SYS. GBLPAGFIL per global cache (file) = value calculated for GBLPAGES minus 2 The two 'stopper' pages added to GBLPAGES do not require the system page file as a backing storage and are thus not included in the GBLPAGFIL count. GBLPAGFIL indicates how many of the blocks allocated for the primary and secondary page files can be used for pages paged out of global sections. In order to delete the global section associated with an RMS global cache, the system has to be able to page out the pages used by these buffers. The delete is pending until the pages can be paged out to the page files. The AUTOGEN "FEEDBACK" mechanism provides input for this parameter so, if it is specified in the MODPARAMS.DAT file, it should be preceded by a MIN_ or ADD_ prefix. Lack of this resource causes the following error when the record stream is connected to the file: %RMS-F-CRMP, CRMPSC system service failed to map global buffers See note (1) for a workaround to this error. The GBLPAGES used by RMS global buffers are always writable. Prior to VMS Version 5.0, the size of the primary page file had to include enough blocks to support the maximum number of writable GBLPAGES that could be paged out at any one time. As of VMS 5.0, this is no longer a requirement. They can be paged out across the primary or secondary page files. 5. VIRTUALPAGECNT Global buffers are mapped through a process's virtual address space. While the process quota PGFLQUOTA sets an upper limit on the maximum number of pages that a process can use in the system paging file, the SYSGEN parameter VIRTUALPAGECNT sets an upper limit on the maximum any process's PGFLQUOTA can be increased to. Global buffers are not allocated until runtime after an image has been executed. If a process's PGFLQUOTA is insufficient for the number of global (and local) buffers that an image attempts to connect to the record streams for one or more files, the following runtime error occurs: RMS-F-DME, Dynamic memory exhausted In other words, the failure occurs due to the exhaustion of a process quota. However, since its upperbound is restricted by a SYSGEN parameter, it is included here. In many cases when this error occurs, PGFLQUOTA is set much lower than the SYSGEN parameter VIRTUALPAGECNT, so the process quota can be increased without having to reboot the system. NOTE: When PGFLQUOTA is modified, the user must log out and back in again for the new value to take effect. The number of PGFLQUOTA entries that will be needed for each global cache (file) is as follows: PGFLQUOTA per global cache (file) = value calculated for GBLPAGES Thus: VIRTUALPAGECNT per global cache (file) = PGFLQUOTA per global cache (file) AUTOGEN calculates this parameter. 6. SYSMWCNT SYSMWCNT sets the quota for the size of the system working set (nonpageable). The global page table itself is part of the system working set. Each 128 entries (each 4 bytes in size) in the global page table use one page of the system working set. The "VMS System Generation Utility Manual", April 1988, page A-12 under the description of GBLPAGES, points out the need to adjust SYSMWCNT by one per 128 GBLPAGE entries as GBLPAGES is increased beyond its default. SYSMWCNT per global cache = [ GBLPAGES / 128 ] rounded up AUTOGEN calculates the value for SYSMWCNT based on GBLPAGES and PAGEDYN, provided it is either omitted from the MODPARAMS.DAT file or, if it is specified in the MODPARAMS.DAT file, it is preceded by a MIN_ or ADD_ prefix. No error occurs if this parameter is set too low. 7. LOCKIDTBL The number of entries in the system Lock ID table (see Note 2), which limits the number of locks in the system. Each global buffer has a system lock on it. LOCKIDTBL per global cache (file) = GBC + 1 AUTOGEN will automatically adjust this parameter, if necessary, in accordance with changes made to RMS_GBLBUFQUO, provided that this parameter is either omitted from the MODPARAMS.DAT file or any appearance is preceded by a MIN_ or ADD_ prefix. The following error occurs if this parameter is set too low: %SYSTEM-E-NOLOCKID, no lock id available. However, this error is not seen using RMS. If this problem is encountered by RMS it most likely results in a nonfatal bugcheck and the process is deleted. If a bugcheck does not occur, the error that RMS might return is very dependent on what it was doing when it encountered the problem. In the latter case, displaying the RAB$L_STV value (accessible from program control only) associated with the STS could be helpful. 8. RESHASHTBL The number of entries in the lock management resource name hash table is related to the LOCKIDTBL parameter. As a general guideline, there should be one resource hash table entry for every four locks in the system. It can be derived directly from the value of LOCKIDTBL. Specifically, it should be one-quarter the value of LOCKIDTBL, rounded up to the closest power of two. AUTOGEN will automatically adjust this parameter, if necessary, in accordance with changes made to LOCKIDTBL, provided that this parameter is either omitted from the MODPARAMS.DAT file or any appearance is preceded by a MIN_ or ADD_ prefix. No error is associated with this parameter being set too low. If it is inappropriately low, there may be a slowdown in the response rate for $ENQ's and $DEQ's (enqueuing and dequeuing locks). ------------------------------------------------------------------------ NOTES: (1) Applications which use global buffers on RMS files should include a workaround for the RMS-F-CRMP error so that the programs do not abort. The workaround is to include a test for a return from a $CONNECT of the status code RMS$_CRMP and, when this is returned, take the following actions: 1. Print a message to the user that the RMS-F-CRMP error occurred and the system manager should be informed that one or more of the SYSGEN parameters GBLSECTIONS, GBLPAGES or GBLPAGFIL was set too low on this particular system to map a global section for RMS global buffers. The filename should be displayed as part of the error message. 2. Assign a zero to the FAB$W_GBC field. This will override the number of global buffers requested (either in the file header or by a nonzero assignment to FAB$W_GBC), and RMS will not attempt to set up a global cache for this file on a connect. 3. Re-call $CONNECT. NOTE: The above suggestions assume that an application is using RMS directly or is using a high level language that provides a user-action routine option as part of the file open. (2) The following information discussing the SYSGEN parameter LOCKIDTBL_MAX is version specific. OpenVMS VAX versions 5.0 through 5.4-3 -------------------------------------- LOCKIDTBL_MAX establishes the maximum size of the system Lock ID table. As a general rule, it should be twice the value of LOCKIDTBL. AUTOGEN does not calculate this value. If it is omitted from the MODPARAMS.DAT file, it will be set to the default. On booting a system, a lock database with LOCKIDTBL entries is built. If this becomes full, it can be expanded to at least LOCKIDTBL_MAX entries (provided that LOCKIDTBL_MAX is greater than LOCKIDTBL) while the system is running. The expansion is done in increments of LOCKIDTBL. For Example: LOCKIDTBL is 65000 and LOCKIDTBL_MAX is 65001. The lock ID table becomes full. The system finds that LOCKIDTBL_MAX is greater than LOCKIDTBL. Since it is, it will expand the lock ID table by LOCKIDTBL. The lock ID table now has 130,000 entries. However, if it becomes full again, LOCKIDTBL_MAX (65001) will not be greater than LOCKIDTBL (130,000). In this example, the lock id table will only be expanded once, but note that its size will exceed LOCKIDTBL_MAX. The expanded lock id table is built in a block of memory allocated from nonpaged pool. You need to have enough contiguous pages in nonpaged pool to accommodate the entire size of the NEW expanded table because the old table isn't deleted until the new one is successfully created. OpenVMS VAX versions 5.5 through 6.2 ------------------------------------ Starting in OpenVMS VAX versions 5.5, the LOCKIDTBL_MAX parameter is used at SYSBOOT time in the calculation which determines possible "System Page Table Entry" (SPTE) requirements. The algorithm governing the extension of the Lock ID table was also modified. The algorithm now only checks the availability of SPTE's and fluid pages before a 1 or 2 page extension is added to the Lock ID table. REFERENCES: Additional information about this topic can be found in other database articles with the following titles: Process Limits and Quotas Related to RMS Buffers Command Procedure To Estimate SYSGEN Units for RMS Global Buffers Estimating SYSGEN Units Currently in Use for RMS Global Buffers [OpenVMS] OPEN with Global Buffers Fails with RMS-F-CRMP, RMS-F-DME For additional information on the AUTOGEN command procedure refer to the "Guide to Setting Up a VMS System", April 1988, (AA-LA25A-TE), Chapter 6. For additional information on VMS memory management and lock management refer to the "Introduction to VMS System Services", April 1988, (AA-LA68A-TE), Chapters 11 & 12. For additional information on file performance tuning and file sharing refer to the "Guide to VMS File Applications", (April 1988), (AA-LA78A-TE), Chapters 3 & 7. Process Limits and Quotas Related to RMS Buffers Any party granted access to the following copyrighted information (protected under Federal Copyright Laws), pursuant to a duly executed Compaq Service Agreement may, under the terms of such agreement copy all or selected portions of this information for internal use and distribution only. No other copying or distribution for any other purpose is authorized. Copyright (c) Digital Equipment Corporation 1990, 1991. All rights reserved COMPONENT: RMS OP/SYS: VMS, All Versions SOURCE: Digital Customer Support Center OVERVIEW: This article describes the following process limits and quotas directly related to RMS local and global buffers: 1. Working set size (WSQUOTA, WSEXTENT) 2. PGFLQUOTA 3. PIOPAGES 4. ENQLM DISCUSSION: 1. WORKING SET SIZE (WSQUOTA, WSEXTENT) o Using LOCAL buffers The number of pages of working set needed to support local buffers for each file is as follows: Number of pages used per local cache = number of buffers * maximum_bucket_size NOTE: The 'virtual' bucket size for a sequential file equals the multiblock count (MBC). If a sequential file is opened with shared write access, it is set to the MBC of the first accessor of the file. The MBC system default is set by the SYSGEN parameter RMS_DFMBC (current default is 16). Since these pages are private to a process, the process must have a working set size large enough to hold the number of pages required. Larger local caches without an increase in working set size can induce excessive paging. o Using GLOBAL buffers The pages used by global buffers are mapped through a process's PO space. The pages of a particular buffer referenced by the process will be part of the working set of that process. However, no one process mapped to a global cache has to have all the global buffers in their working set at one time. The responsibility is distributed across a number of processes. The more processes mapped to a particular global cache and actively doing I/O to that file, the more distributed the actual pages in the global cache are across the working sets of these processes. On the other hand, if the global cache is large and only a few processes are actively using the cache, the more likely paging will occur. Under these conditions, the state of the free page list is very important. The more free memory that exists on a system, the better the chances that buffer pages which are paged out of the working set will still be on the modified page list (or even the free page list) when requested again. In other words, a 'hard' page fault (one that requires a disk I/O) may not occur. 2. PGFLQUOTA o Both local and global buffers are mapped through a process's virtual address space. PGFLQUOTA sets an upper limit on the maximum number of pages that a process can use in the system paging file. o Buffers are allocated at image runtime. If several files are opened by an image and a large number of local and global buffers are connected to the record streams for these files, the following error may occur (at runtime): RMS-F-DME, Dynamic memory exhausted o The SYSGEN parameter VIRTUALPAGECNT sets an upper limit on the maximum any process's PGFLQUOTA can be raised to. In order to get a good estimate of what PGFLQUOTA should be increased to, first use the AUTHORIZE utility to set it to a high value for the particular user who incurred the error. The user should then log off, log back in, and run the image again. When the image completes, the user should logout using the qualifier '/FULL'. The display that appears includes a 'Peak page file size' entry that indicates how much PGFLQUOTA was needed by this job. This provides a lower-bound estimate of what PGFLQUOTA needs to be for this user. o If the DME error is incurred on the open of a process permanent file (one opened from DCL), the SYSGEN parameter PIOPAGES should be increased. 3. PIOPAGES o Files opened at DCL level are opened as process permanent files. Files opened from program control are opened as image files. o Process permanent files (notably SYS$INPUT, SYS$OUTPUT, and SYS$ERROR) must have their RMS internal structures mapped through P1 process space, in order to survive image deletion. PIOPAGES sets an upper limit on the number of P1 pages that can be used for this purpose. o When the RMS-F-DME error is incurred trying to open a process permanent file, PIOPAGES needs to be increased. o The algorithm for estimating the number of PIOPAGES needed for each process permanent file is: Number of PIOPAGES per process permanent file = Number of buffers * maximum_bucket_size NOTE: The 'virtual' bucket size for a sequential file equals the multiblock count (MBC). If a sequential file is opened with shared write access, it is set to the MBC of the first accessor of the file. The MBC system default is set by the SYSGEN parameter RMS_DFMBC (current default is 16). 4. ENQLM o Using LOCAL buffers If the file is not shared write, 2 units of ENQLM are required. One for a file lock and one for an extend lock, which is a sublock of the file lock. If the file is shared write, the number of ENQLM units can be estimated as follows: File and extend locks: 2 Local buffer locks - Number of local buffers minus the one used for extend: # of buffers - 1 Record locks for indexed files (no record locking is done for sequential or relative files): -- Automatic locking - Only one record lock block (RLB) is used at a time: 1 -- Manual locking - Requires as many RLB's as number of records requested before an explicit UNLOCK is done on the file or the file is closed: # of records - 1 NOTE: As of VMS Version 5.4, the default number of local buffers that are allocated when an indexed file is accessed may increase slightly. This will result in more ENQLM units being used. Any process that ran on a marginal ENQLM under pre-VMS 5.4 could experience an ENQLM problem after upgrading. For more information, see page 4-41 of the "VMS Version 5.4 Release Notes", August 1990, (AA-LB22C-TE). o Using GLOBAL buffers The number of ENQLM units can be estimated for global buffers as follows: File and extend lock: 2 Global section lock: 1 Local buffer locks - Number of local buffers minus the one used for extend: # of buffers - 1 Global buffer pointer block (GBPB) maintained in local cache. No matter how large the global cache is, there will only be 2 GBPB's (see note 1): 2 Record locks for indexed files (no record locking is done for sequential or relative files): -- Automatic locking - Only one record lock block (RLB) used at a time 1 -- Manual locking - Requires as many RLB's as records requested before an explicit UNLOCK is done on the file or the file is closed. # of records - 1 ----------------------------------------------------------------- Note (1): The two GBPB's are used for any process locks taken out on a global buffer accessed by a process. When the process lock is released, it is converted to a system NL lock in order to survive the closing of the file by the process, image exit or process deletion. System locks are maintained for each global buffer (once they are used) until the global section for that file is removed. No process limit is charged for system locks. REFERENCES: Additional information about this topic can be found in three other database articles with the following titles: Command Procedure To Estimate SYSGEN Units for RMS Global Buffers Estimating SYSGEN Units Currently in Use for RMS Global Buffers SYSGEN Parameters Related To RMS Global Buffers For additional information on file performance tuning and file sharing refer to "Guide to VMS File Applications", (April 1988), (AA-LA78A-TE), Ch. 3 & 7. For additional information on VMS memory management and lock management refer to the "Introduction to VMS System Services", April 1988, (AA-LA68A-TE), Chapters 11 & 12. [OpenVMS] Guidelines for Indexed File Optimization and Performance Any party granted access to the following copyrighted information (protected under Federal Copyright Laws), pursuant to a duly executed Compaq Service Agreement may, under the terms of such agreement copy all or selected portions of this information for internal use and distribution only. No other copying or distribution for any other purpose is authorized. Copyright (c) Compaq Computer Corporation 1995, 1998. All rights reserved. PRODUCT: DIGITAL OpenVMS VAX, All Versions DIGITAL OpenVMS Alpha, All Versions COMPONENT: Record Management Services (RMS) SOURCE: Compaq Computer Corporation OVERVIEW: RMS indexed files that are actively updated (i.e., with record writes, updates and deletions) need regular maintenance and can benefit greatly in terms of performance by regularly tuning the file's internal structure. This article discusses the general tuning procedure and makes tuning recommendations based on file design criteria. PROCEDURE: The tuning of an indexed file involves using the CONVERT Utility on the file to reload all the data and key indexes in key order, to eliminate bucket splits and reclaim deleted buckets. 1. The procedure to tune an indexed file's size and structure involves three steps: A. Create an FDL file that contains file attribute definitions and statistical information that describe the current state of the indexed file. For example, the following command analyzes ABC.DAT and creates file ABC.FDL;1: $ ANALYZE /RMS /FDL ABC.DAT B. Invoke the FDL Editor's Optimize script, (reads the FDL language and statistical information in the FDL file created by ANALYZE/RMS/FDL) and calculates the optimal number of data and index areas, bucket sizes, and index depth for this indexed file. The following command uses the /NOINTERACTIVE qualifier to direct the FDL Editor to execute the Optimize script without a terminal dialog: $ EDIT /FDL /ANALYSIS=ABC.FDL /NOINTERACTIVE ABC.FDL The default file type for this command is .FDL. On successful execution of this command, the FDL Editor creates ABC.FDL;2. This file contains definition language that, by default, describes a "flatter" indexed file (that is, one with fewer index levels). To create an output FDL file with a unique name, specify the /OUTPUT= qualifier on the command line: $ EDIT /FDL /ANALYSIS=ABC.FDL - /NOINTERACTIVE/OUTPUT=ABC_NEW ABC This command invokes the interactive Optimize script: $ EDIT /FDL /ANALYSIS=ABC.FDL /SCRIPT=OPTIMIZE ABC.FDL NOTE: This command may create an FDL file that defines a different bucket sizes than the original file has. If you've assigned global buffers to the indexed file, the change in bucket size may impact the number of global buffers desired for the file. C. Use the FDL file, created by the FDL Editor, to create an optimized version of the indexed file. For example: $ CONVERT/FDL=ABC_NEW/STATISTICS /NOSORT ABC.DAT ABC.NEW $ RENAME ABC.NEW ABC.DAT 2. Using the FDL Editor's Optimize script yields different results, depending on whether or not it runs interactively: A. Use of the /NOINTERACTIVE qualifier invokes the Optimize script. This produces an FDL file that may contain: o A different number of index and data areas. For example, if there is only 1 area in the original indexed file, the optimized FDL file defines two or more areas, unless /GRANULARITY=1 was specified on the EDIT/FDL command line o New bucket sizes for the index and/or data areas o A different number of blocks allocated to the file o A default extend size for each area that is recalculated to about 25% of the initial allocation for that area B. Using /SCRIPT=OPTIMIZE (without /NOINTERACTIVE) causes the FDL Editor to invoke the interactive Optimize script. By answering the questions in the terminal dialog, one can select different characteristics for each key. One can also change the default emphasis of the file (i.e., to smaller buffers or flatten the file's index structure), and select a new bucket size for the index and data buckets of that index structure. 3. The frequency of file optimization depends on the file's use: A. If the file is actively updated (with records being added, updated and deleted) but the number of records is relatively constant, it should be necessary to tune the file only once. Invoke the FDL Editor's Optimize script (/SCRIPT=OPTIMIZE) and indicate the maximum number of records the file should contain at any one time, by answering these script questions: Number of Records that will be Reloaded into the File (0-2Giga)[n] : Number of Additional Records to be Added After the Reloading the File (0-2147482672)[0] : Because activity on the file can cause internal fragmentation, use CONVERT to periodically rebuild the indexed file: $ CONVERT /STATISTICS /NOSORT ABC.DAT ABC.NEW $ RENAME ABC.NEW ABC.DAT B. If the number of records in the file increases over time, it should be tuned periodically, in order to plan for the file's expected growth during the period of time until the file is next tuned. To do this, execute the FDL Editor's Optimize script and indicate the number of records to be added during that time period: Number of Additional Records to be Added After the Reloading the File (0-2147482672)[0] : For example, if you plan to tune the file every month, indicate the number of additional records you will be adding in that month. Between file tuning, regular use of CONVERT to rebuild the file keeps internal fragmentation from bucket splitting to a minimum. 4. Suggestions for file definition attributes, to increase performance: A. Disable index compression. When index compression is enabled, RMS must sequentially search through the index when performing a lookup of a key value or writing a new record to the file. When it is disabled, RMS performs a binary search. B. Disable data key compression. This does not dramatically improve RMS performance, but reduces the amount of CPU that is needed to compress and decompress the data key values when writing, updating or reading a record. C. If available disk space is not an issue and data record compression is not efficiently reducing the space needed to store a record, disable data record compression. Review the FDL file created by ANALYZE/RMS/FDL, to find the compression efficiency percentage. This information is found in the 'ANALYSIS OF KEY 0' section, in the value for the DATA_RECORD_COMPRESSION field. The larger the number, the more effective the data record compression. D. Avoid duplicate values for key values or make key values as unique as possible. When RMS searches for a particular key value in an alternate key and duplicate records are allowed for the index of reference, RMS may have to search through a chain of records with the same key value to find the record in question. Because RMS sequentially traverses this duplicate chain, a search for a specific record in a long duplicate chain represents a major performance problem for read and delete operations. This problem is compounded for update operations that change the key value, since RMS must traverse the duplicate chain to find the records that has the old key value, delete that alternate key's record (known as a SIDR), traverse the duplicate chain that has the new key value and add a SIDR for the record to the end of that chain. If it is not possible to disallow duplicate key values, it may be possible to make the key more unique. For example, a string key can be made more unique by adding to it a key segment for a field that has some unique value, or by adding to the key field a more unique value. Then, if the key value is not unique, a call to $GET might specify only the first part of the key; in that case, sequential $GET calls can be used to find the record with the the particular key that is being sought. E. Use a fill factor of less than 100%, if records will be randomly added to the file. If records are added with an increasing key value (i.e., key is a log number that is incremented for each new record), select a load fill of 100%. When selecting a load fill factor that is less than 100%, set it to an amount that allows room in the bucket for at least one additional record. For example, if four records can fit into a data bucket, setting the fill factor to 75% causes the CONVERT utility to place three records per bucket, leaving room for one record in each data bucket. F. Allocate additional space in the indexed file, for records that will be added after CONVERT initially loads records to the file. As described earlier in this article, execute the FDL Editor's Optimize script and indicate the number of additional records that will be added after CONVERT reloads the file: Number of Additional Records to be Added After the Reloading the File (0-2147482672)[0] : 5. To reduce index fragmentation when writing records to a new file: When loading records to a new file from program control, write the records to a sequential file first. Then use CONVERT/FDL to load the indexed file from the sequential file. Since CONVERT builds one index at a time, this method loads the file's index structures without causing the internal fragmentation that can result if the program loads the records to an indexed file. RELATED ARTICLES: Other articles in the OPENVMS database provide additional information on RMS indexed files. These articles can be found using a search string of: RMS INDEXED FILES ANALYZE/RMS FDL REFERENCES: "Guide to OpenVMS File Applications", (AA-PV6PA-TK), May 1993, Chapter 3, Chapter 10, Appendix A "OpenVMS Record Management Services Reference Manual", (AA-PV6RA-TK), May 1993 "OpenVMS Record Management Utilities Reference Manual", (AA-PV6QA-TK), May 1993, Chapter 1, Chapter 2, Chapter 6 [OpenVMS] Estimating SYSGEN Units Currently in Use for RMS Global Buffers Any party granted access to the following copyrighted information (protected under Federal Copyright Laws), pursuant to a duly executed Compaq Service Agreement may, under the terms of such agreement copy all or selected portions of this information for internal use and distribution only. No other copying or distribution for any other purpose is authorized. Copyright (c) 1990, 2000. Compaq Computer Corporation. All rights reserved. PRODUCT: Compaq OpenVMS VAX, All Versions COMPONENT: RMS SOURCE: Compaq Computer Corporation OVERVIEW: This article describes how to estimate units currently used by RMS global buffers for the following SYSGEN parameters: 1. GBLSECTIONS 2. GBLPAGES 3. GBLPAGFIL 4. RMS_GBLBUFQUO* (see note below) *Note: RMS_GBLBUFQUO is obsolete as of OpenVMS V7.1. Caution: In deriving any estimates of how many of the units set up by these SYSGEN parameters are currently in use, it must be noted that they are based on a "snapshot" of the system at a particular moment. Any RMS file with global buffers defined (either by a SET FILE/GLOBAL DCL command or via setting FAB$W_GBC at runtime) that is not open at this moment will not be included in the estimation. Any program containing a call to the system service $CRMPSC that is not running at this moment will not be included in the estimation. In other words, any estimate that is derived should be considered an "UNDERESTIMATE." DISCUSSION: 1. GBLSECTIONS The DCL lexical function F$GETSYI can be used to determine how many units of GBLSECTIONS are NOT in use: $ WRITE SYS$OUTPUT F$GETSYI("FREE_GBLSECTS") 9 2. GBLPAGES The DCL lexical function F$GETSYI can be used to determine how many contiguous, free GBLPAGES are available: $ WRITE SYS$OUTPUT F$GETSYI("CONTIG_GBLPAGES") 396 The number of unused pages reported by the INSTALL utility includes the TOTAL number of free pages and can be misleading. $ INSTALL LIST/GLOBAL/SUMMARY Summary of Local Memory Global Sections 261 Global Sections Used, 14998/402 Global Pages Used/Unused 3. GBLPAGFIL There are two alternative ways to estimate GBLPAGFIL units currently in use. ALTERNATIVE #1: This is the simplest way to estimate GBLPAGFIL units currently in use. However, it requires SYSPRV and CMKRNL privileges. In addition, it is based on a symbol that is available through the current version of VMS, but there is no guarantee that this symbol will be available in the future. $ SET PROCESS/PRIVILEGE=(SYSPRV,CMKRNL) $ ANALYZE/SYSTEM VAX/VMS System analyzer SDA> EXAMINE SGN$GL_GBLPAGFIL ! SYSGEN GBLPAGFIL SGN$GL_GBLPAGFIL: 00000400 "...." ! Hex 400 = decimal 1024 SDA> EXAMINE MMG$GL_GBLPAGFIL ! GBLPAGFIL remaining MMG$GL_GBLPAGFIL: 000003B6 "¶..." ! Hex 3B6 = decimal 950 ! MMG$GL_GBLPAGFIL starts out as a positive integer equal ! to SGN$GL_GBLPAGFIL. MMG$GL_GBLPAGFIL is decremented for ! each page created for a WRT/DZRO global section. In other ! words, the current value in MMG$GL_GBLPAGFIL indicates the ! number of unused GBLPAGFIL units. When MMG$GL_GBLPAGFIL ! drops to zero, then no more GBLPAGFIL quota exists. SDA> EXIT ! Symbols used to find amount of GBLPAGFIL currently used $ GBLPAGFIL_SYSGEN = 1024 $ GBLPAGFIL_AVAILABLE = %X000003B6 $ SHOW SYMBOL GBLPAGFIL_AVAILABLE GBLPAGFIL_AVAILABLE = 950 Hex = 000003B6 Octal = 00000001666 ! GBLPAGFIL_USED = GBLPAGFIL_SYSGEN - GBLPAGFIL_AVAILABLE ! = 1024 - 950 ! GBLPAGFIL_USED = 74 ALTERNATIVE #2: Any global sections that have GBLPAGFIL units deducted must have the PSECT attributes WRT (writable) and DZRO (PAGEFILE.SYS used as backing storage). This method uses the INSTALL Utility's LIST/GLOBAL command to identify any global sections that have both of these attributes set. Step 1: Obtain listing of all global sections on system: $ DEFINE/USER SYS$OUTPUT glob_install.lis $ INSTALL LIST/GLOBAL $ DIRECTORY/SIZE glob_install.lis GLOB_INSTALL.LIS;1 38 Step 2: Use DCL SEARCH command to select only those global sections that have the PSECT ATTRIBUTES writable (WRT) and DZRO (PAGEFILE.SYS used as backing storage). $ SEARCH/EXACT/MATCH=AND/OUTPUT=glob_wrt.lis - _$ glob_install.lis WRT,DZRO $ TYPE glob_wrt.lis OSIF$MANAGEMENT(0A000001) WRT DZRO PRM SYS Pagcnt/Refcnt=5/5 RMS$804BDCA0 (00000000) WRT DZRO TMP SYS Pagcnt/Refcnt=69/69 Refcnt = pagcnt * number of users Step 3: Sum 'pagcnt' entries with both WRT and DZRO attributes. The total is equal to the number of GBLPAGFIL units currently in use. ----- Number of GBLPAGFIL units in use 74 The global sections for RMS global buffers always begin with an RMS$ prefix. In the output above, one RMS file active on the system currently has global buffers set (RMS$804BDCA0). It indicates that 69 GBLPAGES are being used by this file. This does not tell us how many global buffers (RMS_GBLBUFQUO units) are being used. The estimation procedure for RMS_GBLBUFQUO below is in "buffer" units. NOTE: There is one case in which a data file can be used as the backing storage for a global section. This occurs when a call is made to the system service SYS$CRMPSC specifying a channel to the data file and the DZRO attribute (as well as GBL and WRT). Since the system PAGEFILE.SYS is not used as the backing storage for this section, GBLPAGFIL units are not deducted. However, this global section DOES have both the WRT and DZRO attributes in the INSTALL listing so there is no way to identify that this section is not using GBLPAGFIL units. Thus, the estimate of GBLPAGFIL units currently in use derived from this alternative could provide an "overestimate". In contrast, the first alternative always provides an exact number at the particular time it is checked using SDA. 4. RMS_GBLBUFQUO (Note: RMS_GBLBUFQUO is obsolete as of OpenVMS V7.1.) SDA provides the only means of finding out how many active RMS global buffers are currently in use on a particular system. A logfile with the steps necessary follows: Step 1: Put EXEC writable page offsets into an object file. $ MACRO/OBJECT=ew_data SYS$INPUT:+SYS$LIBRARY:LIB/LIB $EWDATADEF GLOBAL .END $ DIRECTORY/SIZE ew_data.obj EW_DATA.OBJ;1 3 Step 2: Invoke SDA $ SET PROCESS/PRIVILEGE=(SYSPRV,CMKRNL) $ ANALYZE/SYSTEM VAX/VMS System analyzer Step 3: Read in EXEC writable page offsets SDA> READ ew_data.obj %SDA-I-READSYM, reading symbol table DISK:[USER]EW_DATA.OBJ;1 Step 4: Examine GBLBUFQUO current value SDA> EXAMINE @exe$AR_EWDATA+ew_RMS$GW_GBLBUFQUO SYS$NETWORK_SERVICES+00778: 0000FFFD "ý..." ! decimal -3 ! RMS$GW_GBLBUFQUO is a negative counter. It is initialized ! to zero when the system boots and is decremented for each ! GBLBUFQUO unit used. The -3 (FFFD) is accounted for by the ! one RMS global section located in the INSTALL listing: ! ! RMS$804BDCA0 WRT DZRO TMP SYS Pagcnt/Refcnt=69/69 ! ! Three global buffers are being used by this file. ! ! A break was done here to run an image that opens an RMS ! indexed file (INTER11.DAT) which had 10 global buffers set ! on it. A DIRECTORY/FULL of the file after the global ! buffers were set showed the following file attributes: ! ! $ DIRECTORY/FULL INTER11.DAT ! ... ! Allocation: 21, Extend: 1, Maximum bucket size: 1 ! Global buffer count: 10, No version limit ! ... ! ! Two records were added to the file and the image was ! paused just prior to the $CLOSE operation. ! Return to the SDA session. SDA> ex @exe$AR_EWDATA+ew_RMS$GW_GBLBUFQUO SYS$NETWORK_SERVICES+00778: 0000FFFA "ú..." ! decimal -6 ! Three additional RMS_GBLBUFQUO units are in use. But the ! file had 10 global buffers set on it! Why the discrepancy? ! Because only 3 out of 10 global buffers are currently in ! use for the INTER11.DAT file! In other words, units of ! GBLBUFQUO are deducted as they are used! SDA> SET PROCESS/INDEX=21600572 SDA> SET RMS=GBDSUM SDA> SHOW PROCESS/RMS Process index: 0072 Name: _LTA31: Extended PID: 21600572 ------------------------------------------------------------ GBD Summary ----------- GBD USE REL_ CACHE_ Address CNT SIZE NUMB VBN VBNSEQNUM ADDR VAL LOCK_ID ------- ----- ---- ---- --- ------- ---- --- ------- 00033488 0 512 512 8 00000000 00000600 1 018C0309 00033458 0 512 512 9 00000000 00000400 2 00140321 000334B8 0 512 512 14 00000000 00000800 1 00200320 00033608 0 512 0 -1 00000000 00001600 0 00000000 000335D8 0 512 0 -1 00000000 00001400 0 00000000 000335A8 0 512 0 -1 00000000 00001200 0 00000000 00033578 0 512 0 -1 00000000 00001000 0 00000000 00033548 0 512 0 -1 00000000 00000E00 0 00000000 00033518 0 512 0 -1 00000000 00000C00 0 00000000 000334E8 0 512 0 -1 00000000 00000A00 0 00000000 10 GBDs were processed ! NOTE: File has not been closed in image yet. ! Exit from SDA and obtain information from INSTALL while ! global section is still active for this file. SDA> EXIT $ DEFINE/USER SYS$OUTPUT glob_install.lis $ INSTALL LIST/GLOBAL $ SEARCH/EXACT/MATCH=AND/OUTPUT=glob_writ.lis - _$ glob_install.lis WRT,DZRO $ TYPE glob_writ.lis RMS$8053CBD0 (00000000) WRT DZRO TMP SYS Pagcnt/Refcnt=13/13 OSIF$MANAGEMENT(0A000001) WRT DZRO PRM SYS Pagcnt/Refcnt=5/5 RMS$804BDCA0 (00000000) WRT DZRO TMP SYS Pagcnt/Refcnt=69/69 REFERENCES: Additional information about this topic can be found in other database articles with the following titles: Process Limits and Quotas Related to RMS Buffers Command Procedure To Estimate SYSGEN Units for RMS Global Buffers SYSGEN Parameters Related To RMS Global Buffers For additional information on file performance tuning and file sharing refer to "Guide to VMS File Applications", April 1988, (AA-LA78A-TE), Chapters 3 & 7. For additional information on VMS memory management and lock management refer to the "Introduction to VMS System Services", April 1988, (AA-LA68A-TE), Chapters 11 & 12. For additional information on using SDA refer to the "VMS System Dump Analyzer Utility Manual", April 1988, (AA-LA87A-TE). [OpenVMS] Command Procedure To Estimate SYSGEN Units for RMS Global Buffers Any party granted access to the following copyrighted information (protected under Federal Copyright Laws), pursuant to a duly executed Compaq Service Agreement may, under the terms of such agreement copy all or selected portions of this information for internal use and distribution only. No other copying or distribution for any other purpose is authorized. Copyright (c) 1990, 2000. Compaq Computer Corporation. All rights reserved. PRODUCT: Compaq OpenVMS VAX, All Version COMPONENT: RMS SOURCE: Compaq Computer Corporation OVERVIEW: This article contains a command procedure that can be used to estimate the units needed to support the addition of new RMS global buffers to a system. The following SYSGEN parameter additions are estimated: 1. RMS_GBLBUFQUO *(see note below) 2. GBLSECTIONS 3. GBLPAGES 4. GBLPAGFIL *Note: RMS_GBLBUFQUO is obsolete as of OpenVMS V7.1. The algorithms used in the command procedure to calculate the values are described in another database article. Only two items of information are needed for each indexed file on which global buffers have been set to estimate the number of SYSGEN units that will be consumed by that file. Use the DCL command DIRECTORY/FULL or the lexical function F$FILE_ATTRIBUTES to obtain the following information: 1. Global buffer count (GBC) 2. Maximum bucket size (BKS) PROCEDURE: $ ! RMSGLOB_SYSGEN.COM $ !+ $ ! $ ! Command procedure to estimate RMS global buffer SYSGEN $ ! units needed for one indexed file. This command procedure $ ! can be adapted to accumulate these units across a number $ ! of indexed files. $ ! $ ! CAUTION: This sample command procedure has been tested $ ! using VMS 5.4-3. However, we cannot guarantee its $ ! effectiveness because of the possibility of error in $ ! transmitting or implementing it. It is meant to be used $ ! as a template for writing your own command procedure, and $ ! it may require modification for use on your system. $ ! $ !- $ ON WARNING THEN GOTO DONE $ IF P1 .EQS. "" THEN $INQUIRE P1 "FILE SPEC " $ P1 = F$SEARCH (P1, 1) ! Returns full filespec $ gbc = F$FILE_ATTRIBUTES (P1,"GBC") ! GBC available as of $ IF gbc .EQ. 0 THEN GOTO NOGBC ! VMS V5.2 $ bks = F$FILE_ATTRIBUTES (P1,"BKS") $ RMS_GBLBUFQUO = gbc $ GBLSECTIONS = 1 $ k = 64 + (gbc * 48) ! GBH and GBD descriptor bytes $ GBLPAGFIL = ((gbc * bks * 512) + k + 1023)/512 $ GBLPAGES = GBLPAGFIL + 2 ! plus 2 stopper pages $ WRITE SYS$OUTPUT "" $ WRITE SYS$OUTPUT "SYSGEN values needed by ",P1 $ WRITE SYS$OUTPUT "" $ WRITE SYS$OUTPUT " RMS_GBLBUFQUO = ",RMS_GBLBUFQUO $ WRITE SYS$OUTPUT " GBLSECTIONS = ",GBLSECTIONS $ WRITE SYS$OUTPUT " GBLPAGES = ",GBLPAGES $ WRITE SYS$OUTPUT " GBLPAGFIL = ",GBLPAGFIL $ GOTO DONE $ NOGBC: $ WRITE SYS$OUTPUT "" $ WRITE SYS$OUTPUT "Following file did not have global buffers set: " $ WRITE SYS$OUTPUT " ",P1 $ DONE: $ EXIT EXAMPLE: $ DIRECTORY/FULL tuning.dat Directory DISK:[DIR]TUNING.DAT;1 TUNING.DAT;1 File ID: (22486,1,0) Size: 1554/1554 Owner: [USER] Created: 18-JAN-1990 21:00:48.01 Revised: 17-MAR-1990 15:51:05.08 (11) Expires: Backup: File organization: Indexed, Prolog: 3, Using 4 keys In 3 areas > File attributes: Allocation: 1554, Extend: 252, Maximum bucket size: 18 > Global buffer count: 50, No version limit Contiguous best try Record format: Fixed length 80 byte records Record attributes: Carriage return carriage control RMS attributes: None Journaling enabled: None File protection: System:RWED, Owner:RWED, Group:RE, World: Access Cntrl List: None $ @rmsglob_sysgen TUNING.DAT SYSGEN values needed by DISK:[DIR]TUNING.DAT;1 RMS_GBLBUFQUO = 50 GBLSECTIONS = 1 GBLPAGES = 908 GBLPAGFIL = 906 $ SET FILE/GLOBAL=0 TUNING.DAT $ @rmsglob_sysgen TUNING.DAT Following file did not have global buffers set: DISK:[DIR]TUNING.DAT;1 REFERENCES: Additional information about this topic can be found in other database articles with the following titles: Process Limits and Quotas Related to RMS Buffers Estimating SYSGEN Units Currently in Use for RMS Global Buffers SYSGEN Parameters Related To RMS Global Buffers For additional information on file performance tuning and file sharing, refer to the "Guide to VMS File Applications", April 1988, (AA-LA78A-TE), Chapters 3 & 7. For additional information on VMS memory management and lock management, refer to "Introduction to VMS System Services", April 1988, (AA-LA68A-TE), Chapters 11 & 12. WorkUnit 0004 ** The previous work unit was queued by DSNlink to be sent to the customer. WorkUnit 0005 ---------< This work unit was entered by DSNlink - 11 lines >---------- DSNlink Mail Delivery Failure Notice ------------------------------------ DSNlink was unable to deliver your mail message (workunit) to this customer after repeated attempts. Please contact your customer via an alternative method, such as voice, until this problem can be corrected. WorkUnit 0006 \ dsnlink copy failure; re-sending w/u 3 (first re-try) \ tb 329637 \ +----------------------- Beginning of Screener's Text -----------------------+ \(PR3)RMS Global buffer expected behavior? \DSNlink File Copy Failure \** This is additional information to the call owned by 152303 \+--------------------------- End of Screener's Text --------------------------+ Bill, Below I have included multiple articles from our DSNlink database that should help to address your questions. If you have additional questions or problems after reviewing this information, please let us know. Joe Albanese OpenVMS Support Internals/Drivers Rules of Thumb: Number of Buffers For Indexed Files Any party granted access to the following copyrighted information (protected under Federal Copyright Laws), pursuant to a duly executed Compaq Service Agreement may, under the terms of such agreement copy all or selected portions of this information for internal use and distribution only. No other copying or distribution for any other purpose is authorized. Copyright (c) Digital Equipment Corporation 1987, 1991. All rights reserved COMPONENT: Record Management Services (RMS) OP/SYS: VMS SOURCE: Digital Customer Support Center VERSION INFORMATION: Information Applies To: VMS, All Versions OVERVIEW: This information was gathered from the "RMS Structures and Utilities" training class available through Educational Services of Digital Equipment Corporation. *** CAUTION *** Too many global buffers can have a serious negative impact on performance. Depending on the number of active concurrent users, the locking contention on the global section required to scan the GBD (Global Buffer Descriptor) list and the sequential scan of the GBD linked list itself can make performance worse than if the file had no global buffers. Before implementing global buffers on a particular file, always perform benchmarks with and without them. DISCUSSION: General Guidelines: Number Of Buffers For Indexed Files In general, the maximum number of buffers (local or global) that should be used for any index tree to be accessed in an indexed file is the following: o Maximum # buffers = SUM of [(total # index buckets) + 1 (for data)] for each index PLUS the average number of users The "sum" indicated would be across all index trees to be accessed. In the case of local buffers, the average number of users would be equal to 1. This total number provides the user with an "upperbound" estimate. (Refer to Note 1 below.) For moderate to large sized files this upperbound buffer estimate is generally more than the user can afford for local buffering or the system can afford for global buffering. NOTE: As of VMS 4.5, the maximum number of local buffers that can be specified for any file is 127. Algorithms: Estimating Minimum LOCAL and GLOBAL Buffers The following algorithms provide a good starting point for estimating the "minimum" number of LOCAL buffers that should be used. The algorithms described below for "clustered" reads or updates provide a good "lowerbound" estimate for GLOBAL buffers as well, but note that these algorithms assume the average number of users is 1. In practice the actual number of buffers used will fall somewhere in the range between the "lowerbound" and "upperbound" estimates derived. In the case of shared files that have a number of concurrent accessors, every attempt should be made to cache the top-most levels of the index structure in global buffers, particularly for files that have more than two (2) index levels in the primary structure. 1. KEYED READ_ONLY OPERATIONS: A. KEY 0: o Minimum # buffers = (1 for root index bucket) + 1 (data) o lower-bound est. clustered reads = (# levels in index tree) + 1 (data) B. ALTERNATE KEY: o Minimum # buffers = (1 for root index bucket alt key) + 1 (data alt key) + 1 (data key 0) + 1 (RRV) o lower-bound est. clustered reads = (# levels in index tree alt key) + 1 (data alt key) + 1 (data key 0) + 1 (RRV) (Refer to Note 2 below for an explanation of RRV) 2. KEYED (KEY 0) INSERTS OR DELETES (fast delete not enabled): The following estimate should be calculated for each index tree in the file. The minimum number of buffers would be the SUM of the estimates across all index trees: o Minimum # buffers = (# levels in index tree) + 2 (data) (Refer to Note 3 below) 3. KEYED UPDATES (MODIFYs): A. Using Key 0: If the size of the records will not grow larger (i.e., remain the same size or decrease) and no alternate key value will typically be changed by the update: o Minimum # buffers = (1 for root index bucket) + 1 (data) o lower-bound est. clustered updates = (# levels in index tree) + 1 (data) If the size of a record may be expanded by the update, then the possibility that bucket splits may occur must be considered. o Minimum # buffers = (# levels in index tree) + 2 (data) (Refer to Note 3 below) B. Using an Alternate Key: If the update is done using an alternate key, then add 1 to the above Key 0 algorithms to allow for RRVs. (Refer to Note 2 below) SETTING GLOBAL BUFFERS: Global buffers are set on a file-by-file basis either at creation time or at runtime. For example: Creation time ------------- FAB FAB$W_GBC=n $CREATE/FDL=fdl FILE GLOBAL_BUFFER_COUNT=n $CONVERT/FDL=fdl FILE GLOBAL_BUFFER_COUNT=n Runtime ------- FAB FAB$W_GBC=n FDL$PARSE(fdl,...) FILE GLOBAL_BUFFER_COUNT=n DCL $ SET FILE/GLOBAL_BUFFER=n WHERE n = 0 to 32,767 *** CAUTION *** Global Buffers are ignored for process permanent files (DCL OPEN) and when an image is linked using the LINK option keyword IOSEGMENT=NOP0BUFS. NOTES: Note 1: In most cases all index paths down to the level 0 data buckets will not be followed, so the total number of buffers estimated from this algorithm would typically not be needed. Any surplus would in fact be used to cache more data buckets than the one bucket allowed above. Note 2: If the retrieval is done using an alternate key, one additional I/O operation is required if the path to the record to be retrieved from a Key 0, Level 0 data bucket is via an RRV. Note 3: Allow 2 for data to provide for at least a simple bucket split. A bucket split also causes the index structure to have to be updated (or in case of deletion of highest record in data bucket). To locate an index bucket to be updated at any tree level, RMS must re-work its way back down through the tree from the top root index bucket. Requests for bucket locks are only allowed by RMS from one direction -- top down -- otherwise deadlock conditions could occur. This algorithm provides enough buffers to cache the index buckets for the path followed to the current data bucket. [OpenVMS] SYSGEN Parameters Related To RMS Global Buffers Any party granted access to the following copyrighted information (protected under Federal Copyright Laws), pursuant to a duly executed Compaq Service Agreement may, under the terms of such agreement copy all or selected portions of this information for internal use and distribution only. No other copying or distribution for any other purpose is authorized. Copyright (c) 1988, 2000. Compaq Computer Corporation. All rights reserved. PRODUCT: Compaq OpenVMS VAX, All Versions COMPONENT: RMS SOURCE: Compaq Computer Corporation OVERVIEW: There are eight SYSGEN parameters that are directly related to RMS global buffers. They are the following: 1. RMS_GBLBUFQUO* (see note) 5. VIRTUALPAGECNT 2. GBLSECTIONS 6. SYSMWCNT 3. GBLPAGES 7. LOCKIDTBL 4. GBLPAGFIL 8. RESHASHTBL *Note: RMS_GBLBUFQUO is obsolete as of OpenVMS V7.1. Algorithms are provided that can be used to determine the number of units of the first four SYSGEN parameters that are used for RMS global buffer caches. These algorithms require the following two items of information about an RMS file that global buffers are to be added to: 1) Global buffer count (GBC) 2) Maximum bucket size (BKS) NOTE: The "virtual" bucket size for a shared write sequential file is set to the multiblock count (MBC) of the first accessor of the file. The MBC system default is 16. The command procedure SYS$UPDATE:AUTOGEN.COM should be used to adjust the last four parameters in accordance with any modifications made to the first four. DISCUSSION: 1. RMS_GBLBUFQUO (Note: RMS_GBLBUFQUO is obsolete as of OpenVMS V7.1) This is the total number of global buffers over all RMS files allowed to be active simultaneously on a particular system. The need to have a separate SYSGEN parameter for RMS global buffers is explained by the fact that each global buffer requires a system lock. Lock structures are allocated from nonpaged pool (SYSGEN parameter NPAGEDYN). Thus, this quota sets an upper limit on how much of nonpaged pool can be used for system locks on RMS global buffers. RMS_GBLBUFQUO per global cache (file)= GBC When a file is opened that has global buffers specified, the number of global buffers it is requesting is not checked against this quota. This quota is not checked until a global buffer must be used. At that time a check is made to determine whether this quota allows another RMS global buffer to be active. If the quota has been exhausted, then a global buffer is not used. Lack of this resource does NOT cause any error to be returned by RMS (specifically, %RMS-F-CRMP, CRMPSC system service failed to map global buffers). AUTOGEN does not calculate this parameter. 2. GBLSECTIONS This number indicates the number of global section table entries needed in the system header. Each RMS file opened with global buffers requires one GBLSECTION entry (and associated global section descriptor). GBLSECTIONS per global cache (file) = 1 Each global section table entry occupies 32 bytes in the nonpageable area of the system header. Thus, each 19 sections of GBLSECTIONS uses 1 page of nonpaged pool. NOTE: The descriptor associated with the global section is pageable. The AUTOGEN "FEEDBACK" mechanism provides input for this parameter so, if it is specified in the MODPARAMS.DAT file, it should be preceded by a MIN_ or ADD_ prefix. Lack of this resource causes the following error when the record stream is connected to the file: %RMS-F-CRMP, CRMPSC system service failed to map global buffers See note (1) for a workaround to this error. 3. GBLPAGES The maximum number of pages in memory that can be accessed by more than one user. The GBLPAGES used by RMS global buffers are always writable. GBLPAGES per global cache (file) = TRUN [ ( (GBC * BKS * 512) + k + 1023 ) /512 ] + 2 Where k = 64 + ( GBC * 48) 'k' equals the number of bytes needed for the following RMS data structures: Global buffer header (GBH) - 64 bytes: one per global cache (file) Global buffer descriptor (GBD) - 48 bytes: one per global buffer The 1023 bytes listed above are added by RMS for page alignment and internal purposes. The two additional pages (the "+ 2" in the above formula) serve as "stoppers" (limits to modified page write clustering). One is created at the beginning of the section and one at the end. The system page file is used as the backing storage for these pages (with the exception of the 2 stopper pages) when they are removed from the modified page list. GBLPAGES should be generously estimated. It is a cheap parameter. Each 128 entries in the global page table use 4 bytes of nonpaged pool, or another way of expressing it is: each 131,072 pages of GBLPAGES use only 8 pages of nonpaged pool. And the global page table itself is pageable. The AUTOGEN "FEEDBACK" mechanism provides input for this parameter so, if it is specified in the MODPARAMS.DAT file, it should be preceded by a MIN_ or ADD_ prefix. Lack of this resource caused the following error when the record stream is connected to the file: %RMS-F-CRMP, CRMPSC system service failed to map global buffers See note (1) for a workaround to this error. This error occurs if there is an insufficient number of entries remaining in the global page table, but may also occur for a less obvious reason. The entries for a global section in the global page table must be contiguous. Thus, sometimes the number of entries remaining are more than the number needed, but the table is fragmented and does not have the number needed contiguous to each other. The DCL lexical function F$GETSYI can be used to determine whether lack of sufficient contiguous entries is the problem: F$GETSYI ("CONTIG_GBLPAGES") Also note that when the last image mapped to an RMS global section exits, the entries in the global page table do not get immediately freed. The pages mapped through the table must first go to the modified page list and then eventually out to the system paging file. Then the global section can be deleted. Once the global section is deleted, the entries in the global page table are freed up. 4. GBLPAGFIL The maximum number of system-wide pages allowed for global page file sections (or scratch global sections) in the system's PAGEFILE.SYS. GBLPAGFIL per global cache (file) = value calculated for GBLPAGES minus 2 The two 'stopper' pages added to GBLPAGES do not require the system page file as a backing storage and are thus not included in the GBLPAGFIL count. GBLPAGFIL indicates how many of the blocks allocated for the primary and secondary page files can be used for pages paged out of global sections. In order to delete the global section associated with an RMS global cache, the system has to be able to page out the pages used by these buffers. The delete is pending until the pages can be paged out to the page files. The AUTOGEN "FEEDBACK" mechanism provides input for this parameter so, if it is specified in the MODPARAMS.DAT file, it should be preceded by a MIN_ or ADD_ prefix. Lack of this resource causes the following error when the record stream is connected to the file: %RMS-F-CRMP, CRMPSC system service failed to map global buffers See note (1) for a workaround to this error. The GBLPAGES used by RMS global buffers are always writable. Prior to VMS Version 5.0, the size of the primary page file had to include enough blocks to support the maximum number of writable GBLPAGES that could be paged out at any one time. As of VMS 5.0, this is no longer a requirement. They can be paged out across the primary or secondary page files. 5. VIRTUALPAGECNT Global buffers are mapped through a process's virtual address space. While the process quota PGFLQUOTA sets an upper limit on the maximum number of pages that a process can use in the system paging file, the SYSGEN parameter VIRTUALPAGECNT sets an upper limit on the maximum any process's PGFLQUOTA can be increased to. Global buffers are not allocated until runtime after an image has been executed. If a process's PGFLQUOTA is insufficient for the number of global (and local) buffers that an image attempts to connect to the record streams for one or more files, the following runtime error occurs: RMS-F-DME, Dynamic memory exhausted In other words, the failure occurs due to the exhaustion of a process quota. However, since its upperbound is restricted by a SYSGEN parameter, it is included here. In many cases when this error occurs, PGFLQUOTA is set much lower than the SYSGEN parameter VIRTUALPAGECNT, so the process quota can be increased without having to reboot the system. NOTE: When PGFLQUOTA is modified, the user must log out and back in again for the new value to take effect. The number of PGFLQUOTA entries that will be needed for each global cache (file) is as follows: PGFLQUOTA per global cache (file) = value calculated for GBLPAGES Thus: VIRTUALPAGECNT per global cache (file) = PGFLQUOTA per global cache (file) AUTOGEN calculates this parameter. 6. SYSMWCNT SYSMWCNT sets the quota for the size of the system working set (nonpageable). The global page table itself is part of the system working set. Each 128 entries (each 4 bytes in size) in the global page table use one page of the system working set. The "VMS System Generation Utility Manual", April 1988, page A-12 under the description of GBLPAGES, points out the need to adjust SYSMWCNT by one per 128 GBLPAGE entries as GBLPAGES is increased beyond its default. SYSMWCNT per global cache = [ GBLPAGES / 128 ] rounded up AUTOGEN calculates the value for SYSMWCNT based on GBLPAGES and PAGEDYN, provided it is either omitted from the MODPARAMS.DAT file or, if it is specified in the MODPARAMS.DAT file, it is preceded by a MIN_ or ADD_ prefix. No error occurs if this parameter is set too low. 7. LOCKIDTBL The number of entries in the system Lock ID table (see Note 2), which limits the number of locks in the system. Each global buffer has a system lock on it. LOCKIDTBL per global cache (file) = GBC + 1 AUTOGEN will automatically adjust this parameter, if necessary, in accordance with changes made to RMS_GBLBUFQUO, provided that this parameter is either omitted from the MODPARAMS.DAT file or any appearance is preceded by a MIN_ or ADD_ prefix. The following error occurs if this parameter is set too low: %SYSTEM-E-NOLOCKID, no lock id available. However, this error is not seen using RMS. If this problem is encountered by RMS it most likely results in a nonfatal bugcheck and the process is deleted. If a bugcheck does not occur, the error that RMS might return is very dependent on what it was doing when it encountered the problem. In the latter case, displaying the RAB$L_STV value (accessible from program control only) associated with the STS could be helpful. 8. RESHASHTBL The number of entries in the lock management resource name hash table is related to the LOCKIDTBL parameter. As a general guideline, there should be one resource hash table entry for every four locks in the system. It can be derived directly from the value of LOCKIDTBL. Specifically, it should be one-quarter the value of LOCKIDTBL, rounded up to the closest power of two. AUTOGEN will automatically adjust this parameter, if necessary, in accordance with changes made to LOCKIDTBL, provided that this parameter is either omitted from the MODPARAMS.DAT file or any appearance is preceded by a MIN_ or ADD_ prefix. No error is associated with this parameter being set too low. If it is inappropriately low, there may be a slowdown in the response rate for $ENQ's and $DEQ's (enqueuing and dequeuing locks). ------------------------------------------------------------------------ NOTES: (1) Applications which use global buffers on RMS files should include a workaround for the RMS-F-CRMP error so that the programs do not abort. The workaround is to include a test for a return from a $CONNECT of the status code RMS$_CRMP and, when this is returned, take the following actions: 1. Print a message to the user that the RMS-F-CRMP error occurred and the system manager should be informed that one or more of the SYSGEN parameters GBLSECTIONS, GBLPAGES or GBLPAGFIL was set too low on this particular system to map a global section for RMS global buffers. The filename should be displayed as part of the error message. 2. Assign a zero to the FAB$W_GBC field. This will override the number of global buffers requested (either in the file header or by a nonzero assignment to FAB$W_GBC), and RMS will not attempt to set up a global cache for this file on a connect. 3. Re-call $CONNECT. NOTE: The above suggestions assume that an application is using RMS directly or is using a high level language that provides a user-action routine option as part of the file open. (2) The following information discussing the SYSGEN parameter LOCKIDTBL_MAX is version specific. OpenVMS VAX versions 5.0 through 5.4-3 -------------------------------------- LOCKIDTBL_MAX establishes the maximum size of the system Lock ID table. As a general rule, it should be twice the value of LOCKIDTBL. AUTOGEN does not calculate this value. If it is omitted from the MODPARAMS.DAT file, it will be set to the default. On booting a system, a lock database with LOCKIDTBL entries is built. If this becomes full, it can be expanded to at least LOCKIDTBL_MAX entries (provided that LOCKIDTBL_MAX is greater than LOCKIDTBL) while the system is running. The expansion is done in increments of LOCKIDTBL. For Example: LOCKIDTBL is 65000 and LOCKIDTBL_MAX is 65001. The lock ID table becomes full. The system finds that LOCKIDTBL_MAX is greater than LOCKIDTBL. Since it is, it will expand the lock ID table by LOCKIDTBL. The lock ID table now has 130,000 entries. However, if it becomes full again, LOCKIDTBL_MAX (65001) will not be greater than LOCKIDTBL (130,000). In this example, the lock id table will only be expanded once, but note that its size will exceed LOCKIDTBL_MAX. The expanded lock id table is built in a block of memory allocated from nonpaged pool. You need to have enough contiguous pages in nonpaged pool to accommodate the entire size of the NEW expanded table because the old table isn't deleted until the new one is successfully created. OpenVMS VAX versions 5.5 through 6.2 ------------------------------------ Starting in OpenVMS VAX versions 5.5, the LOCKIDTBL_MAX parameter is used at SYSBOOT time in the calculation which determines possible "System Page Table Entry" (SPTE) requirements. The algorithm governing the extension of the Lock ID table was also modified. The algorithm now only checks the availability of SPTE's and fluid pages before a 1 or 2 page extension is added to the Lock ID table. REFERENCES: Additional information about this topic can be found in other database articles with the following titles: Process Limits and Quotas Related to RMS Buffers Command Procedure To Estimate SYSGEN Units for RMS Global Buffers Estimating SYSGEN Units Currently in Use for RMS Global Buffers [OpenVMS] OPEN with Global Buffers Fails with RMS-F-CRMP, RMS-F-DME For additional information on the AUTOGEN command procedure refer to the "Guide to Setting Up a VMS System", April 1988, (AA-LA25A-TE), Chapter 6. For additional information on VMS memory management and lock management refer to the "Introduction to VMS System Services", April 1988, (AA-LA68A-TE), Chapters 11 & 12. For additional information on file performance tuning and file sharing refer to the "Guide to VMS File Applications", (April 1988), (AA-LA78A-TE), Chapters 3 & 7. Process Limits and Quotas Related to RMS Buffers Any party granted access to the following copyrighted information (protected under Federal Copyright Laws), pursuant to a duly executed Compaq Service Agreement may, under the terms of such agreement copy all or selected portions of this information for internal use and distribution only. No other copying or distribution for any other purpose is authorized. Copyright (c) Digital Equipment Corporation 1990, 1991. All rights reserved COMPONENT: RMS OP/SYS: VMS, All Versions SOURCE: Digital Customer Support Center OVERVIEW: This article describes the following process limits and quotas directly related to RMS local and global buffers: 1. Working set size (WSQUOTA, WSEXTENT) 2. PGFLQUOTA 3. PIOPAGES 4. ENQLM DISCUSSION: 1. WORKING SET SIZE (WSQUOTA, WSEXTENT) o Using LOCAL buffers The number of pages of working set needed to support local buffers for each file is as follows: Number of pages used per local cache = number of buffers * maximum_bucket_size NOTE: The 'virtual' bucket size for a sequential file equals the multiblock count (MBC). If a sequential file is opened with shared write access, it is set to the MBC of the first accessor of the file. The MBC system default is set by the SYSGEN parameter RMS_DFMBC (current default is 16). Since these pages are private to a process, the process must have a working set size large enough to hold the number of pages required. Larger local caches without an increase in working set size can induce excessive paging. o Using GLOBAL buffers The pages used by global buffers are mapped through a process's PO space. The pages of a particular buffer referenced by the process will be part of the working set of that process. However, no one process mapped to a global cache has to have all the global buffers in their working set at one time. The responsibility is distributed across a number of processes. The more processes mapped to a particular global cache and actively doing I/O to that file, the more distributed the actual pages in the global cache are across the working sets of these processes. On the other hand, if the global cache is large and only a few processes are actively using the cache, the more likely paging will occur. Under these conditions, the state of the free page list is very important. The more free memory that exists on a system, the better the chances that buffer pages which are paged out of the working set will still be on the modified page list (or even the free page list) when requested again. In other words, a 'hard' page fault (one that requires a disk I/O) may not occur. 2. PGFLQUOTA o Both local and global buffers are mapped through a process's virtual address space. PGFLQUOTA sets an upper limit on the maximum number of pages that a process can use in the system paging file. o Buffers are allocated at image runtime. If several files are opened by an image and a large number of local and global buffers are connected to the record streams for these files, the following error may occur (at runtime): RMS-F-DME, Dynamic memory exhausted o The SYSGEN parameter VIRTUALPAGECNT sets an upper limit on the maximum any process's PGFLQUOTA can be raised to. In order to get a good estimate of what PGFLQUOTA should be increased to, first use the AUTHORIZE utility to set it to a high value for the particular user who incurred the error. The user should then log off, log back in, and run the image again. When the image completes, the user should logout using the qualifier '/FULL'. The display that appears includes a 'Peak page file size' entry that indicates how much PGFLQUOTA was needed by this job. This provides a lower-bound estimate of what PGFLQUOTA needs to be for this user. o If the DME error is incurred on the open of a process permanent file (one opened from DCL), the SYSGEN parameter PIOPAGES should be increased. 3. PIOPAGES o Files opened at DCL level are opened as process permanent files. Files opened from program control are opened as image files. o Process permanent files (notably SYS$INPUT, SYS$OUTPUT, and SYS$ERROR) must have their RMS internal structures mapped through P1 process space, in order to survive image deletion. PIOPAGES sets an upper limit on the number of P1 pages that can be used for this purpose. o When the RMS-F-DME error is incurred trying to open a process permanent file, PIOPAGES needs to be increased. o The algorithm for estimating the number of PIOPAGES needed for each process permanent file is: Number of PIOPAGES per process permanent file = Number of buffers * maximum_bucket_size NOTE: The 'virtual' bucket size for a sequential file equals the multiblock count (MBC). If a sequential file is opened with shared write access, it is set to the MBC of the first accessor of the file. The MBC system default is set by the SYSGEN parameter RMS_DFMBC (current default is 16). 4. ENQLM o Using LOCAL buffers If the file is not shared write, 2 units of ENQLM are required. One for a file lock and one for an extend lock, which is a sublock of the file lock. If the file is shared write, the number of ENQLM units can be estimated as follows: File and extend locks: 2 Local buffer locks - Number of local buffers minus the one used for extend: # of buffers - 1 Record locks for indexed files (no record locking is done for sequential or relative files): -- Automatic locking - Only one record lock block (RLB) is used at a time: 1 -- Manual locking - Requires as many RLB's as number of records requested before an explicit UNLOCK is done on the file or the file is closed: # of records - 1 NOTE: As of VMS Version 5.4, the default number of local buffers that are allocated when an indexed file is accessed may increase slightly. This will result in more ENQLM units being used. Any process that ran on a marginal ENQLM under pre-VMS 5.4 could experience an ENQLM problem after upgrading. For more information, see page 4-41 of the "VMS Version 5.4 Release Notes", August 1990, (AA-LB22C-TE). o Using GLOBAL buffers The number of ENQLM units can be estimated for global buffers as follows: File and extend lock: 2 Global section lock: 1 Local buffer locks - Number of local buffers minus the one used for extend: # of buffers - 1 Global buffer pointer block (GBPB) maintained in local cache. No matter how large the global cache is, there will only be 2 GBPB's (see note 1): 2 Record locks for indexed files (no record locking is done for sequential or relative files): -- Automatic locking - Only one record lock block (RLB) used at a time 1 -- Manual locking - Requires as many RLB's as records requested before an explicit UNLOCK is done on the file or the file is closed. # of records - 1 ----------------------------------------------------------------- Note (1): The two GBPB's are used for any process locks taken out on a global buffer accessed by a process. When the process lock is released, it is converted to a system NL lock in order to survive the closing of the file by the process, image exit or process deletion. System locks are maintained for each global buffer (once they are used) until the global section for that file is removed. No process limit is charged for system locks. REFERENCES: Additional information about this topic can be found in three other database articles with the following titles: Command Procedure To Estimate SYSGEN Units for RMS Global Buffers Estimating SYSGEN Units Currently in Use for RMS Global Buffers SYSGEN Parameters Related To RMS Global Buffers For additional information on file performance tuning and file sharing refer to "Guide to VMS File Applications", (April 1988), (AA-LA78A-TE), Ch. 3 & 7. For additional information on VMS memory management and lock management refer to the "Introduction to VMS System Services", April 1988, (AA-LA68A-TE), Chapters 11 & 12. [OpenVMS] Guidelines for Indexed File Optimization and Performance Any party granted access to the following copyrighted information (protected under Federal Copyright Laws), pursuant to a duly executed Compaq Service Agreement may, under the terms of such agreement copy all or selected portions of this information for internal use and distribution only. No other copying or distribution for any other purpose is authorized. Copyright (c) Compaq Computer Corporation 1995, 1998. All rights reserved. PRODUCT: DIGITAL OpenVMS VAX, All Versions DIGITAL OpenVMS Alpha, All Versions COMPONENT: Record Management Services (RMS) SOURCE: Compaq Computer Corporation OVERVIEW: RMS indexed files that are actively updated (i.e., with record writes, updates and deletions) need regular maintenance and can benefit greatly in terms of performance by regularly tuning the file's internal structure. This article discusses the general tuning procedure and makes tuning recommendations based on file design criteria. PROCEDURE: The tuning of an indexed file involves using the CONVERT Utility on the file to reload all the data and key indexes in key order, to eliminate bucket splits and reclaim deleted buckets. 1. The procedure to tune an indexed file's size and structure involves three steps: A. Create an FDL file that contains file attribute definitions and statistical information that describe the current state of the indexed file. For example, the following command analyzes ABC.DAT and creates file ABC.FDL;1: $ ANALYZE /RMS /FDL ABC.DAT B. Invoke the FDL Editor's Optimize script, (reads the FDL language and statistical information in the FDL file created by ANALYZE/RMS/FDL) and calculates the optimal number of data and index areas, bucket sizes, and index depth for this indexed file. The following command uses the /NOINTERACTIVE qualifier to direct the FDL Editor to execute the Optimize script without a terminal dialog: $ EDIT /FDL /ANALYSIS=ABC.FDL /NOINTERACTIVE ABC.FDL The default file type for this command is .FDL. On successful execution of this command, the FDL Editor creates ABC.FDL;2. This file contains definition language that, by default, describes a "flatter" indexed file (that is, one with fewer index levels). To create an output FDL file with a unique name, specify the /OUTPUT= qualifier on the command line: $ EDIT /FDL /ANALYSIS=ABC.FDL - /NOINTERACTIVE/OUTPUT=ABC_NEW ABC This command invokes the interactive Optimize script: $ EDIT /FDL /ANALYSIS=ABC.FDL /SCRIPT=OPTIMIZE ABC.FDL NOTE: This command may create an FDL file that defines a different bucket sizes than the original file has. If you've assigned global buffers to the indexed file, the change in bucket size may impact the number of global buffers desired for the file. C. Use the FDL file, created by the FDL Editor, to create an optimized version of the indexed file. For example: $ CONVERT/FDL=ABC_NEW/STATISTICS /NOSORT ABC.DAT ABC.NEW $ RENAME ABC.NEW ABC.DAT 2. Using the FDL Editor's Optimize script yields different results, depending on whether or not it runs interactively: A. Use of the /NOINTERACTIVE qualifier invokes the Optimize script. This produces an FDL file that may contain: o A different number of index and data areas. For example, if there is only 1 area in the original indexed file, the optimized FDL file defines two or more areas, unless /GRANULARITY=1 was specified on the EDIT/FDL command line o New bucket sizes for the index and/or data areas o A different number of blocks allocated to the file o A default extend size for each area that is recalculated to about 25% of the initial allocation for that area B. Using /SCRIPT=OPTIMIZE (without /NOINTERACTIVE) causes the FDL Editor to invoke the interactive Optimize script. By answering the questions in the terminal dialog, one can select different characteristics for each key. One can also change the default emphasis of the file (i.e., to smaller buffers or flatten the file's index structure), and select a new bucket size for the index and data buckets of that index structure. 3. The frequency of file optimization depends on the file's use: A. If the file is actively updated (with records being added, updated and deleted) but the number of records is relatively constant, it should be necessary to tune the file only once. Invoke the FDL Editor's Optimize script (/SCRIPT=OPTIMIZE) and indicate the maximum number of records the file should contain at any one time, by answering these script questions: Number of Records that will be Reloaded into the File (0-2Giga)[n] : Number of Additional Records to be Added After the Reloading the File (0-2147482672)[0] : Because activity on the file can cause internal fragmentation, use CONVERT to periodically rebuild the indexed file: $ CONVERT /STATISTICS /NOSORT ABC.DAT ABC.NEW $ RENAME ABC.NEW ABC.DAT B. If the number of records in the file increases over time, it should be tuned periodically, in order to plan for the file's expected growth during the period of time until the file is next tuned. To do this, execute the FDL Editor's Optimize script and indicate the number of records to be added during that time period: Number of Additional Records to be Added After the Reloading the File (0-2147482672)[0] : For example, if you plan to tune the file every month, indicate the number of additional records you will be adding in that month. Between file tuning, regular use of CONVERT to rebuild the file keeps internal fragmentation from bucket splitting to a minimum. 4. Suggestions for file definition attributes, to increase performance: A. Disable index compression. When index compression is enabled, RMS must sequentially search through the index when performing a lookup of a key value or writing a new record to the file. When it is disabled, RMS performs a binary search. B. Disable data key compression. This does not dramatically improve RMS performance, but reduces the amount of CPU that is needed to compress and decompress the data key values when writing, updating or reading a record. C. If available disk space is not an issue and data record compression is not efficiently reducing the space needed to store a record, disable data record compression. Review the FDL file created by ANALYZE/RMS/FDL, to find the compression efficiency percentage. This information is found in the 'ANALYSIS OF KEY 0' section, in the value for the DATA_RECORD_COMPRESSION field. The larger the number, the more effective the data record compression. D. Avoid duplicate values for key values or make key values as unique as possible. When RMS searches for a particular key value in an alternate key and duplicate records are allowed for the index of reference, RMS may have to search through a chain of records with the same key value to find the record in question. Because RMS sequentially traverses this duplicate chain, a search for a specific record in a long duplicate chain represents a major performance problem for read and delete operations. This problem is compounded for update operations that change the key value, since RMS must traverse the duplicate chain to find the records that has the old key value, delete that alternate key's record (known as a SIDR), traverse the duplicate chain that has the new key value and add a SIDR for the record to the end of that chain. If it is not possible to disallow duplicate key values, it may be possible to make the key more unique. For example, a string key can be made more unique by adding to it a key segment for a field that has some unique value, or by adding to the key field a more unique value. Then, if the key value is not unique, a call to $GET might specify only the first part of the key; in that case, sequential $GET calls can be used to find the record with the the particular key that is being sought. E. Use a fill factor of less than 100%, if records will be randomly added to the file. If records are added with an increasing key value (i.e., key is a log number that is incremented for each new record), select a load fill of 100%. When selecting a load fill factor that is less than 100%, set it to an amount that allows room in the bucket for at least one additional record. For example, if four records can fit into a data bucket, setting the fill factor to 75% causes the CONVERT utility to place three records per bucket, leaving room for one record in each data bucket. F. Allocate additional space in the indexed file, for records that will be added after CONVERT initially loads records to the file. As described earlier in this article, execute the FDL Editor's Optimize script and indicate the number of additional records that will be added after CONVERT reloads the file: Number of Additional Records to be Added After the Reloading the File (0-2147482672)[0] : 5. To reduce index fragmentation when writing records to a new file: When loading records to a new file from program control, write the records to a sequential file first. Then use CONVERT/FDL to load the indexed file from the sequential file. Since CONVERT builds one index at a time, this method loads the file's index structures without causing the internal fragmentation that can result if the program loads the records to an indexed file. RELATED ARTICLES: Other articles in the OPENVMS database provide additional information on RMS indexed files. These articles can be found using a search string of: RMS INDEXED FILES ANALYZE/RMS FDL REFERENCES: "Guide to OpenVMS File Applications", (AA-PV6PA-TK), May 1993, Chapter 3, Chapter 10, Appendix A "OpenVMS Record Management Services Reference Manual", (AA-PV6RA-TK), May 1993 "OpenVMS Record Management Utilities Reference Manual", (AA-PV6QA-TK), May 1993, Chapter 1, Chapter 2, Chapter 6 [OpenVMS] Estimating SYSGEN Units Currently in Use for RMS Global Buffers Any party granted access to the following copyrighted information (protected under Federal Copyright Laws), pursuant to a duly executed Compaq Service Agreement may, under the terms of such agreement copy all or selected portions of this information for internal use and distribution only. No other copying or distribution for any other purpose is authorized. Copyright (c) 1990, 2000. Compaq Computer Corporation. All rights reserved. PRODUCT: Compaq OpenVMS VAX, All Versions COMPONENT: RMS SOURCE: Compaq Computer Corporation OVERVIEW: This article describes how to estimate units currently used by RMS global buffers for the following SYSGEN parameters: 1. GBLSECTIONS 2. GBLPAGES 3. GBLPAGFIL 4. RMS_GBLBUFQUO* (see note below) *Note: RMS_GBLBUFQUO is obsolete as of OpenVMS V7.1. Caution: In deriving any estimates of how many of the units set up by these SYSGEN parameters are currently in use, it must be noted that they are based on a "snapshot" of the system at a particular moment. Any RMS file with global buffers defined (either by a SET FILE/GLOBAL DCL command or via setting FAB$W_GBC at runtime) that is not open at this moment will not be included in the estimation. Any program containing a call to the system service $CRMPSC that is not running at this moment will not be included in the estimation. In other words, any estimate that is derived should be considered an "UNDERESTIMATE." DISCUSSION: 1. GBLSECTIONS The DCL lexical function F$GETSYI can be used to determine how many units of GBLSECTIONS are NOT in use: $ WRITE SYS$OUTPUT F$GETSYI("FREE_GBLSECTS") 9 2. GBLPAGES The DCL lexical function F$GETSYI can be used to determine how many contiguous, free GBLPAGES are available: $ WRITE SYS$OUTPUT F$GETSYI("CONTIG_GBLPAGES") 396 The number of unused pages reported by the INSTALL utility includes the TOTAL number of free pages and can be misleading. $ INSTALL LIST/GLOBAL/SUMMARY Summary of Local Memory Global Sections 261 Global Sections Used, 14998/402 Global Pages Used/Unused 3. GBLPAGFIL There are two alternative ways to estimate GBLPAGFIL units currently in use. ALTERNATIVE #1: This is the simplest way to estimate GBLPAGFIL units currently in use. However, it requires SYSPRV and CMKRNL privileges. In addition, it is based on a symbol that is available through the current version of VMS, but there is no guarantee that this symbol will be available in the future. $ SET PROCESS/PRIVILEGE=(SYSPRV,CMKRNL) $ ANALYZE/SYSTEM VAX/VMS System analyzer SDA> EXAMINE SGN$GL_GBLPAGFIL ! SYSGEN GBLPAGFIL SGN$GL_GBLPAGFIL: 00000400 "...." ! Hex 400 = decimal 1024 SDA> EXAMINE MMG$GL_GBLPAGFIL ! GBLPAGFIL remaining MMG$GL_GBLPAGFIL: 000003B6 "¶..." ! Hex 3B6 = decimal 950 ! MMG$GL_GBLPAGFIL starts out as a positive integer equal ! to SGN$GL_GBLPAGFIL. MMG$GL_GBLPAGFIL is decremented for ! each page created for a WRT/DZRO global section. In other ! words, the current value in MMG$GL_GBLPAGFIL indicates the ! number of unused GBLPAGFIL units. When MMG$GL_GBLPAGFIL ! drops to zero, then no more GBLPAGFIL quota exists. SDA> EXIT ! Symbols used to find amount of GBLPAGFIL currently used $ GBLPAGFIL_SYSGEN = 1024 $ GBLPAGFIL_AVAILABLE = %X000003B6 $ SHOW SYMBOL GBLPAGFIL_AVAILABLE GBLPAGFIL_AVAILABLE = 950 Hex = 000003B6 Octal = 00000001666 ! GBLPAGFIL_USED = GBLPAGFIL_SYSGEN - GBLPAGFIL_AVAILABLE ! = 1024 - 950 ! GBLPAGFIL_USED = 74 ALTERNATIVE #2: Any global sections that have GBLPAGFIL units deducted must have the PSECT attributes WRT (writable) and DZRO (PAGEFILE.SYS used as backing storage). This method uses the INSTALL Utility's LIST/GLOBAL command to identify any global sections that have both of these attributes set. Step 1: Obtain listing of all global sections on system: $ DEFINE/USER SYS$OUTPUT glob_install.lis $ INSTALL LIST/GLOBAL $ DIRECTORY/SIZE glob_install.lis GLOB_INSTALL.LIS;1 38 Step 2: Use DCL SEARCH command to select only those global sections that have the PSECT ATTRIBUTES writable (WRT) and DZRO (PAGEFILE.SYS used as backing storage). $ SEARCH/EXACT/MATCH=AND/OUTPUT=glob_wrt.lis - _$ glob_install.lis WRT,DZRO $ TYPE glob_wrt.lis OSIF$MANAGEMENT(0A000001) WRT DZRO PRM SYS Pagcnt/Refcnt=5/5 RMS$804BDCA0 (00000000) WRT DZRO TMP SYS Pagcnt/Refcnt=69/69 Refcnt = pagcnt * number of users Step 3: Sum 'pagcnt' entries with both WRT and DZRO attributes. The total is equal to the number of GBLPAGFIL units currently in use. ----- Number of GBLPAGFIL units in use 74 The global sections for RMS global buffers always begin with an RMS$ prefix. In the output above, one RMS file active on the system currently has global buffers set (RMS$804BDCA0). It indicates that 69 GBLPAGES are being used by this file. This does not tell us how many global buffers (RMS_GBLBUFQUO units) are being used. The estimation procedure for RMS_GBLBUFQUO below is in "buffer" units. NOTE: There is one case in which a data file can be used as the backing storage for a global section. This occurs when a call is made to the system service SYS$CRMPSC specifying a channel to the data file and the DZRO attribute (as well as GBL and WRT). Since the system PAGEFILE.SYS is not used as the backing storage for this section, GBLPAGFIL units are not deducted. However, this global section DOES have both the WRT and DZRO attributes in the INSTALL listing so there is no way to identify that this section is not using GBLPAGFIL units. Thus, the estimate of GBLPAGFIL units currently in use derived from this alternative could provide an "overestimate". In contrast, the first alternative always provides an exact number at the particular time it is checked using SDA. 4. RMS_GBLBUFQUO (Note: RMS_GBLBUFQUO is obsolete as of OpenVMS V7.1.) SDA provides the only means of finding out how many active RMS global buffers are currently in use on a particular system. A logfile with the steps necessary follows: Step 1: Put EXEC writable page offsets into an object file. $ MACRO/OBJECT=ew_data SYS$INPUT:+SYS$LIBRARY:LIB/LIB $EWDATADEF GLOBAL .END $ DIRECTORY/SIZE ew_data.obj EW_DATA.OBJ;1 3 Step 2: Invoke SDA $ SET PROCESS/PRIVILEGE=(SYSPRV,CMKRNL) $ ANALYZE/SYSTEM VAX/VMS System analyzer Step 3: Read in EXEC writable page offsets SDA> READ ew_data.obj %SDA-I-READSYM, reading symbol table DISK:[USER]EW_DATA.OBJ;1 Step 4: Examine GBLBUFQUO current value SDA> EXAMINE @exe$AR_EWDATA+ew_RMS$GW_GBLBUFQUO SYS$NETWORK_SERVICES+00778: 0000FFFD "ý..." ! decimal -3 ! RMS$GW_GBLBUFQUO is a negative counter. It is initialized ! to zero when the system boots and is decremented for each ! GBLBUFQUO unit used. The -3 (FFFD) is accounted for by the ! one RMS global section located in the INSTALL listing: ! ! RMS$804BDCA0 WRT DZRO TMP SYS Pagcnt/Refcnt=69/69 ! ! Three global buffers are being used by this file. ! ! A break was done here to run an image that opens an RMS ! indexed file (INTER11.DAT) which had 10 global buffers set ! on it. A DIRECTORY/FULL of the file after the global ! buffers were set showed the following file attributes: ! ! $ DIRECTORY/FULL INTER11.DAT ! ... ! Allocation: 21, Extend: 1, Maximum bucket size: 1 ! Global buffer count: 10, No version limit ! ... ! ! Two records were added to the file and the image was ! paused just prior to the $CLOSE operation. ! Return to the SDA session. SDA> ex @exe$AR_EWDATA+ew_RMS$GW_GBLBUFQUO SYS$NETWORK_SERVICES+00778: 0000FFFA "ú..." ! decimal -6 ! Three additional RMS_GBLBUFQUO units are in use. But the ! file had 10 global buffers set on it! Why the discrepancy? ! Because only 3 out of 10 global buffers are currently in ! use for the INTER11.DAT file! In other words, units of ! GBLBUFQUO are deducted as they are used! SDA> SET PROCESS/INDEX=21600572 SDA> SET RMS=GBDSUM SDA> SHOW PROCESS/RMS Process index: 0072 Name: _LTA31: Extended PID: 21600572 ------------------------------------------------------------ GBD Summary ----------- GBD USE REL_ CACHE_ Address CNT SIZE NUMB VBN VBNSEQNUM ADDR VAL LOCK_ID ------- ----- ---- ---- --- ------- ---- --- ------- 00033488 0 512 512 8 00000000 00000600 1 018C0309 00033458 0 512 512 9 00000000 00000400 2 00140321 000334B8 0 512 512 14 00000000 00000800 1 00200320 00033608 0 512 0 -1 00000000 00001600 0 00000000 000335D8 0 512 0 -1 00000000 00001400 0 00000000 000335A8 0 512 0 -1 00000000 00001200 0 00000000 00033578 0 512 0 -1 00000000 00001000 0 00000000 00033548 0 512 0 -1 00000000 00000E00 0 00000000 00033518 0 512 0 -1 00000000 00000C00 0 00000000 000334E8 0 512 0 -1 00000000 00000A00 0 00000000 10 GBDs were processed ! NOTE: File has not been closed in image yet. ! Exit from SDA and obtain information from INSTALL while ! global section is still active for this file. SDA> EXIT $ DEFINE/USER SYS$OUTPUT glob_install.lis $ INSTALL LIST/GLOBAL $ SEARCH/EXACT/MATCH=AND/OUTPUT=glob_writ.lis - _$ glob_install.lis WRT,DZRO $ TYPE glob_writ.lis RMS$8053CBD0 (00000000) WRT DZRO TMP SYS Pagcnt/Refcnt=13/13 OSIF$MANAGEMENT(0A000001) WRT DZRO PRM SYS Pagcnt/Refcnt=5/5 RMS$804BDCA0 (00000000) WRT DZRO TMP SYS Pagcnt/Refcnt=69/69 REFERENCES: Additional information about this topic can be found in other database articles with the following titles: Process Limits and Quotas Related to RMS Buffers Command Procedure To Estimate SYSGEN Units for RMS Global Buffers SYSGEN Parameters Related To RMS Global Buffers For additional information on file performance tuning and file sharing refer to "Guide to VMS File Applications", April 1988, (AA-LA78A-TE), Chapters 3 & 7. For additional information on VMS memory management and lock management refer to the "Introduction to VMS System Services", April 1988, (AA-LA68A-TE), Chapters 11 & 12. For additional information on using SDA refer to the "VMS System Dump Analyzer Utility Manual", April 1988, (AA-LA87A-TE). [OpenVMS] Command Procedure To Estimate SYSGEN Units for RMS Global Buffers Any party granted access to the following copyrighted information (protected under Federal Copyright Laws), pursuant to a duly executed Compaq Service Agreement may, under the terms of such agreement copy all or selected portions of this information for internal use and distribution only. No other copying or distribution for any other purpose is authorized. Copyright (c) 1990, 2000. Compaq Computer Corporation. All rights reserved. PRODUCT: Compaq OpenVMS VAX, All Version COMPONENT: RMS SOURCE: Compaq Computer Corporation OVERVIEW: This article contains a command procedure that can be used to estimate the units needed to support the addition of new RMS global buffers to a system. The following SYSGEN parameter additions are estimated: 1. RMS_GBLBUFQUO *(see note below) 2. GBLSECTIONS 3. GBLPAGES 4. GBLPAGFIL *Note: RMS_GBLBUFQUO is obsolete as of OpenVMS V7.1. The algorithms used in the command procedure to calculate the values are described in another database article. Only two items of information are needed for each indexed file on which global buffers have been set to estimate the number of SYSGEN units that will be consumed by that file. Use the DCL command DIRECTORY/FULL or the lexical function F$FILE_ATTRIBUTES to obtain the following information: 1. Global buffer count (GBC) 2. Maximum bucket size (BKS) PROCEDURE: $ ! RMSGLOB_SYSGEN.COM $ !+ $ ! $ ! Command procedure to estimate RMS global buffer SYSGEN $ ! units needed for one indexed file. This command procedure $ ! can be adapted to accumulate these units across a number $ ! of indexed files. $ ! $ ! CAUTION: This sample command procedure has been tested $ ! using VMS 5.4-3. However, we cannot guarantee its $ ! effectiveness because of the possibility of error in $ ! transmitting or implementing it. It is meant to be used $ ! as a template for writing your own command procedure, and $ ! it may require modification for use on your system. $ ! $ !- $ ON WARNING THEN GOTO DONE $ IF P1 .EQS. "" THEN $INQUIRE P1 "FILE SPEC " $ P1 = F$SEARCH (P1, 1) ! Returns full filespec $ gbc = F$FILE_ATTRIBUTES (P1,"GBC") ! GBC available as of $ IF gbc .EQ. 0 THEN GOTO NOGBC ! VMS V5.2 $ bks = F$FILE_ATTRIBUTES (P1,"BKS") $ RMS_GBLBUFQUO = gbc $ GBLSECTIONS = 1 $ k = 64 + (gbc * 48) ! GBH and GBD descriptor bytes $ GBLPAGFIL = ((gbc * bks * 512) + k + 1023)/512 $ GBLPAGES = GBLPAGFIL + 2 ! plus 2 stopper pages $ WRITE SYS$OUTPUT "" $ WRITE SYS$OUTPUT "SYSGEN values needed by ",P1 $ WRITE SYS$OUTPUT "" $ WRITE SYS$OUTPUT " RMS_GBLBUFQUO = ",RMS_GBLBUFQUO $ WRITE SYS$OUTPUT " GBLSECTIONS = ",GBLSECTIONS $ WRITE SYS$OUTPUT " GBLPAGES = ",GBLPAGES $ WRITE SYS$OUTPUT " GBLPAGFIL = ",GBLPAGFIL $ GOTO DONE $ NOGBC: $ WRITE SYS$OUTPUT "" $ WRITE SYS$OUTPUT "Following file did not have global buffers set: " $ WRITE SYS$OUTPUT " ",P1 $ DONE: $ EXIT EXAMPLE: $ DIRECTORY/FULL tuning.dat Directory DISK:[DIR]TUNING.DAT;1 TUNING.DAT;1 File ID: (22486,1,0) Size: 1554/1554 Owner: [USER] Created: 18-JAN-1990 21:00:48.01 Revised: 17-MAR-1990 15:51:05.08 (11) Expires: Backup: File organization: Indexed, Prolog: 3, Using 4 keys In 3 areas > File attributes: Allocation: 1554, Extend: 252, Maximum bucket size: 18 > Global buffer count: 50, No version limit Contiguous best try Record format: Fixed length 80 byte records Record attributes: Carriage return carriage control RMS attributes: None Journaling enabled: None File protection: System:RWED, Owner:RWED, Group:RE, World: Access Cntrl List: None $ @rmsglob_sysgen TUNING.DAT SYSGEN values needed by DISK:[DIR]TUNING.DAT;1 RMS_GBLBUFQUO = 50 GBLSECTIONS = 1 GBLPAGES = 908 GBLPAGFIL = 906 $ SET FILE/GLOBAL=0 TUNING.DAT $ @rmsglob_sysgen TUNING.DAT Following file did not have global buffers set: DISK:[DIR]TUNING.DAT;1 REFERENCES: Additional information about this topic can be found in other database articles with the following titles: Process Limits and Quotas Related to RMS Buffers Estimating SYSGEN Units Currently in Use for RMS Global Buffers SYSGEN Parameters Related To RMS Global Buffers For additional information on file performance tuning and file sharing, refer to the "Guide to VMS File Applications", April 1988, (AA-LA78A-TE), Chapters 3 & 7. For additional information on VMS memory management and lock management, refer to "Introduction to VMS System Services", April 1988, (AA-LA68A-TE), Chapters 11 & 12. WorkUnit 0007 ** The previous work unit was queued by DSNlink to be sent to the customer. WorkUnit 0008 ---------< This work unit was entered by DSNlink - 11 lines >---------- DSNlink Mail Delivery Failure Notice ------------------------------------ DSNlink was unable to deliver your mail message (workunit) to this customer after repeated attempts. Please contact your customer via an alternative method, such as voice, until this problem can be corrected. WorkUnit 0009 The following Email was sent to BILL.NORTON@WCFA.COM: Bill, I replied yesterday to your questions posted in DSNlink service request C010205-2461. However, multiple DSNlink delivery attempts have failed so I am sending this information directly to your Email address and will also try posting it one more time to DSNlink. If you do receive this information in Email, please reply back so that I will know that you have received it. Joe Albanese OpenVMS Support Internals/Drivers Response posted on February 5, 2000: Bill, Below I have included multiple articles from our DSNlink database that should help to address your questions. If you have additional questions or problems after reviewing this information, please let us know. Joe Albanese OpenVMS Support Internals/Drivers Rules of Thumb: Number of Buffers For Indexed Files Any party granted access to the following copyrighted information (protected under Federal Copyright Laws), pursuant to a duly executed Compaq Service Agreement may, under the terms of such agreement copy all or selected portions of this information for internal use and distribution only. No other copying or distribution for any other purpose is authorized. Copyright (c) Digital Equipment Corporation 1987, 1991. All rights reserved COMPONENT: Record Management Services (RMS) OP/SYS: VMS SOURCE: Digital Customer Support Center VERSION INFORMATION: Information Applies To: VMS, All Versions OVERVIEW: This information was gathered from the "RMS Structures and Utilities" training class available through Educational Services of Digital Equipment Corporation. *** CAUTION *** Too many global buffers can have a serious negative impact on performance. Depending on the number of active concurrent users, the locking contention on the global section required to scan the GBD (Global Buffer Descriptor) list and the sequential scan of the GBD linked list itself can make performance worse than if the file had no global buffers. Before implementing global buffers on a particular file, always perform benchmarks with and without them. DISCUSSION: General Guidelines: Number Of Buffers For Indexed Files In general, the maximum number of buffers (local or global) that should be used for any index tree to be accessed in an indexed file is the following: o Maximum # buffers = SUM of [(total # index buckets) + 1 (for data)] for each index PLUS the average number of users The "sum" indicated would be across all index trees to be accessed. In the case of local buffers, the average number of users would be equal to 1. This total number provides the user with an "upperbound" estimate. (Refer to Note 1 below.) For moderate to large sized files this upperbound buffer estimate is generally more than the user can afford for local buffering or the system can afford for global buffering. NOTE: As of VMS 4.5, the maximum number of local buffers that can be specified for any file is 127. Algorithms: Estimating Minimum LOCAL and GLOBAL Buffers The following algorithms provide a good starting point for estimating the "minimum" number of LOCAL buffers that should be used. The algorithms described below for "clustered" reads or updates provide a good "lowerbound" estimate for GLOBAL buffers as well, but note that these algorithms assume the average number of users is 1. In practice the actual number of buffers used will fall somewhere in the range between the "lowerbound" and "upperbound" estimates derived. In the case of shared files that have a number of concurrent accessors, every attempt should be made to cache the top-most levels of the index structure in global buffers, particularly for files that have more than two (2) index levels in the primary structure. 1. KEYED READ_ONLY OPERATIONS: A. KEY 0: o Minimum # buffers = (1 for root index bucket) + 1 (data) o lower-bound est. clustered reads = (# levels in index tree) + 1 (data) B. ALTERNATE KEY: o Minimum # buffers = (1 for root index bucket alt key) + 1 (data alt key) + 1 (data key 0) + 1 (RRV) o lower-bound est. clustered reads = (# levels in index tree alt key) + 1 (data alt key) + 1 (data key 0) + 1 (RRV) (Refer to Note 2 below for an explanation of RRV) 2. KEYED (KEY 0) INSERTS OR DELETES (fast delete not enabled): The following estimate should be calculated for each index tree in the file. The minimum number of buffers would be the SUM of the estimates across all index trees: o Minimum # buffers = (# levels in index tree) + 2 (data) (Refer to Note 3 below) 3. KEYED UPDATES (MODIFYs): A. Using Key 0: If the size of the records will not grow larger (i.e., remain the same size or decrease) and no alternate key value will typically be changed by the update: o Minimum # buffers = (1 for root index bucket) + 1 (data) o lower-bound est. clustered updates = (# levels in index tree) + 1 (data) If the size of a record may be expanded by the update, then the possibility that bucket splits may occur must be considered. o Minimum # buffers = (# levels in index tree) + 2 (data) (Refer to Note 3 below) B. Using an Alternate Key: If the update is done using an alternate key, then add 1 to the above Key 0 algorithms to allow for RRVs. (Refer to Note 2 below) SETTING GLOBAL BUFFERS: Global buffers are set on a file-by-file basis either at creation time or at runtime. For example: Creation time ------------- FAB FAB$W_GBC=n $CREATE/FDL=fdl FILE GLOBAL_BUFFER_COUNT=n $CONVERT/FDL=fdl FILE GLOBAL_BUFFER_COUNT=n Runtime ------- FAB FAB$W_GBC=n FDL$PARSE(fdl,...) FILE GLOBAL_BUFFER_COUNT=n DCL $ SET FILE/GLOBAL_BUFFER=n WHERE n = 0 to 32,767 *** CAUTION *** Global Buffers are ignored for process permanent files (DCL OPEN) and when an image is linked using the LINK option keyword IOSEGMENT=NOP0BUFS. NOTES: Note 1: In most cases all index paths down to the level 0 data buckets will not be followed, so the total number of buffers estimated from this algorithm would typically not be needed. Any surplus would in fact be used to cache more data buckets than the one bucket allowed above. Note 2: If the retrieval is done using an alternate key, one additional I/O operation is required if the path to the record to be retrieved from a Key 0, Level 0 data bucket is via an RRV. Note 3: Allow 2 for data to provide for at least a simple bucket split. A bucket split also causes the index structure to have to be updated (or in case of deletion of highest record in data bucket). To locate an index bucket to be updated at any tree level, RMS must re-work its way back down through the tree from the top root index bucket. Requests for bucket locks are only allowed by RMS from one direction -- top down -- otherwise deadlock conditions could occur. This algorithm provides enough buffers to cache the index buckets for the path followed to the current data bucket. [OpenVMS] SYSGEN Parameters Related To RMS Global Buffers Any party granted access to the following copyrighted information (protected under Federal Copyright Laws), pursuant to a duly executed Compaq Service Agreement may, under the terms of such agreement copy all or selected portions of this information for internal use and distribution only. No other copying or distribution for any other purpose is authorized. Copyright (c) 1988, 2000. Compaq Computer Corporation. All rights reserved. PRODUCT: Compaq OpenVMS VAX, All Versions COMPONENT: RMS SOURCE: Compaq Computer Corporation OVERVIEW: There are eight SYSGEN parameters that are directly related to RMS global buffers. They are the following: 1. RMS_GBLBUFQUO* (see note) 5. VIRTUALPAGECNT 2. GBLSECTIONS 6. SYSMWCNT 3. GBLPAGES 7. LOCKIDTBL 4. GBLPAGFIL 8. RESHASHTBL *Note: RMS_GBLBUFQUO is obsolete as of OpenVMS V7.1. Algorithms are provided that can be used to determine the number of units of the first four SYSGEN parameters that are used for RMS global buffer caches. These algorithms require the following two items of information about an RMS file that global buffers are to be added to: 1) Global buffer count (GBC) 2) Maximum bucket size (BKS) NOTE: The "virtual" bucket size for a shared write sequential file is set to the multiblock count (MBC) of the first accessor of the file. The MBC system default is 16. The command procedure SYS$UPDATE:AUTOGEN.COM should be used to adjust the last four parameters in accordance with any modifications made to the first four. DISCUSSION: 1. RMS_GBLBUFQUO (Note: RMS_GBLBUFQUO is obsolete as of OpenVMS V7.1) This is the total number of global buffers over all RMS files allowed to be active simultaneously on a particular system. The need to have a separate SYSGEN parameter for RMS global buffers is explained by the fact that each global buffer requires a system lock. Lock structures are allocated from nonpaged pool (SYSGEN parameter NPAGEDYN). Thus, this quota sets an upper limit on how much of nonpaged pool can be used for system locks on RMS global buffers. RMS_GBLBUFQUO per global cache (file)= GBC When a file is opened that has global buffers specified, the number of global buffers it is requesting is not checked against this quota. This quota is not checked until a global buffer must be used. At that time a check is made to determine whether this quota allows another RMS global buffer to be active. If the quota has been exhausted, then a global buffer is not used. Lack of this resource does NOT cause any error to be returned by RMS (specifically, %RMS-F-CRMP, CRMPSC system service failed to map global buffers). AUTOGEN does not calculate this parameter. 2. GBLSECTIONS This number indicates the number of global section table entries needed in the system header. Each RMS file opened with global buffers requires one GBLSECTION entry (and associated global section descriptor). GBLSECTIONS per global cache (file) = 1 Each global section table entry occupies 32 bytes in the nonpageable area of the system header. Thus, each 19 sections of GBLSECTIONS uses 1 page of nonpaged pool. NOTE: The descriptor associated with the global section is pageable. The AUTOGEN "FEEDBACK" mechanism provides input for this parameter so, if it is specified in the MODPARAMS.DAT file, it should be preceded by a MIN_ or ADD_ prefix. Lack of this resource causes the following error when the record stream is connected to the file: %RMS-F-CRMP, CRMPSC system service failed to map global buffers See note (1) for a workaround to this error. 3. GBLPAGES The maximum number of pages in memory that can be accessed by more than one user. The GBLPAGES used by RMS global buffers are always writable. GBLPAGES per global cache (file) = TRUN [ ( (GBC * BKS * 512) + k + 1023 ) /512 ] + 2 Where k = 64 + ( GBC * 48) 'k' equals the number of bytes needed for the following RMS data structures: Global buffer header (GBH) - 64 bytes: one per global cache (file) Global buffer descriptor (GBD) - 48 bytes: one per global buffer The 1023 bytes listed above are added by RMS for page alignment and internal purposes. The two additional pages (the "+ 2" in the above formula) serve as "stoppers" (limits to modified page write clustering). One is created at the beginning of the section and one at the end. The system page file is used as the backing storage for these pages (with the exception of the 2 stopper pages) when they are removed from the modified page list. GBLPAGES should be generously estimated. It is a cheap parameter. Each 128 entries in the global page table use 4 bytes of nonpaged pool, or another way of expressing it is: each 131,072 pages of GBLPAGES use only 8 pages of nonpaged pool. And the global page table itself is pageable. The AUTOGEN "FEEDBACK" mechanism provides input for this parameter so, if it is specified in the MODPARAMS.DAT file, it should be preceded by a MIN_ or ADD_ prefix. Lack of this resource caused the following error when the record stream is connected to the file: %RMS-F-CRMP, CRMPSC system service failed to map global buffers See note (1) for a workaround to this error. This error occurs if there is an insufficient number of entries remaining in the global page table, but may also occur for a less obvious reason. The entries for a global section in the global page table must be contiguous. Thus, sometimes the number of entries remaining are more than the number needed, but the table is fragmented and does not have the number needed contiguous to each other. The DCL lexical function F$GETSYI can be used to determine whether lack of sufficient contiguous entries is the problem: F$GETSYI ("CONTIG_GBLPAGES") Also note that when the last image mapped to an RMS global section exits, the entries in the global page table do not get immediately freed. The pages mapped through the table must first go to the modified page list and then eventually out to the system paging file. Then the global section can be deleted. Once the global section is deleted, the entries in the global page table are freed up. 4. GBLPAGFIL The maximum number of system-wide pages allowed for global page file sections (or scratch global sections) in the system's PAGEFILE.SYS. GBLPAGFIL per global cache (file) = value calculated for GBLPAGES minus 2 The two 'stopper' pages added to GBLPAGES do not require the system page file as a backing storage and are thus not included in the GBLPAGFIL count. GBLPAGFIL indicates how many of the blocks allocated for the primary and secondary page files can be used for pages paged out of global sections. In order to delete the global section associated with an RMS global cache, the system has to be able to page out the pages used by these buffers. The delete is pending until the pages can be paged out to the page files. The AUTOGEN "FEEDBACK" mechanism provides input for this parameter so, if it is specified in the MODPARAMS.DAT file, it should be preceded by a MIN_ or ADD_ prefix. Lack of this resource causes the following error when the record stream is connected to the file: %RMS-F-CRMP, CRMPSC system service failed to map global buffers See note (1) for a workaround to this error. The GBLPAGES used by RMS global buffers are always writable. Prior to VMS Version 5.0, the size of the primary page file had to include enough blocks to support the maximum number of writable GBLPAGES that could be paged out at any one time. As of VMS 5.0, this is no longer a requirement. They can be paged out across the primary or secondary page files. 5. VIRTUALPAGECNT Global buffers are mapped through a process's virtual address space. While the process quota PGFLQUOTA sets an upper limit on the maximum number of pages that a process can use in the system paging file, the SYSGEN parameter VIRTUALPAGECNT sets an upper limit on the maximum any process's PGFLQUOTA can be increased to. Global buffers are not allocated until runtime after an image has been executed. If a process's PGFLQUOTA is insufficient for the number of global (and local) buffers that an image attempts to connect to the record streams for one or more files, the following runtime error occurs: RMS-F-DME, Dynamic memory exhausted In other words, the failure occurs due to the exhaustion of a process quota. However, since its upperbound is restricted by a SYSGEN parameter, it is included here. In many cases when this error occurs, PGFLQUOTA is set much lower than the SYSGEN parameter VIRTUALPAGECNT, so the process quota can be increased without having to reboot the system. NOTE: When PGFLQUOTA is modified, the user must log out and back in again for the new value to take effect. The number of PGFLQUOTA entries that will be needed for each global cache (file) is as follows: PGFLQUOTA per global cache (file) = value calculated for GBLPAGES Thus: VIRTUALPAGECNT per global cache (file) = PGFLQUOTA per global cache (file) AUTOGEN calculates this parameter. 6. SYSMWCNT SYSMWCNT sets the quota for the size of the system working set (nonpageable). The global page table itself is part of the system working set. Each 128 entries (each 4 bytes in size) in the global page table use one page of the system working set. The "VMS System Generation Utility Manual", April 1988, page A-12 under the description of GBLPAGES, points out the need to adjust SYSMWCNT by one per 128 GBLPAGE entries as GBLPAGES is increased beyond its default. SYSMWCNT per global cache = [ GBLPAGES / 128 ] rounded up AUTOGEN calculates the value for SYSMWCNT based on GBLPAGES and PAGEDYN, provided it is either omitted from the MODPARAMS.DAT file or, if it is specified in the MODPARAMS.DAT file, it is preceded by a MIN_ or ADD_ prefix. No error occurs if this parameter is set too low. 7. LOCKIDTBL The number of entries in the system Lock ID table (see Note 2), which limits the number of locks in the system. Each global buffer has a system lock on it. LOCKIDTBL per global cache (file) = GBC + 1 AUTOGEN will automatically adjust this parameter, if necessary, in accordance with changes made to RMS_GBLBUFQUO, provided that this parameter is either omitted from the MODPARAMS.DAT file or any appearance is preceded by a MIN_ or ADD_ prefix. The following error occurs if this parameter is set too low: %SYSTEM-E-NOLOCKID, no lock id available. However, this error is not seen using RMS. If this problem is encountered by RMS it most likely results in a nonfatal bugcheck and the process is deleted. If a bugcheck does not occur, the error that RMS might return is very dependent on what it was doing when it encountered the problem. In the latter case, displaying the RAB$L_STV value (accessible from program control only) associated with the STS could be helpful. 8. RESHASHTBL The number of entries in the lock management resource name hash table is related to the LOCKIDTBL parameter. As a general guideline, there should be one resource hash table entry for every four locks in the system. It can be derived directly from the value of LOCKIDTBL. Specifically, it should be one-quarter the value of LOCKIDTBL, rounded up to the closest power of two. AUTOGEN will automatically adjust this parameter, if necessary, in accordance with changes made to LOCKIDTBL, provided that this parameter is either omitted from the MODPARAMS.DAT file or any appearance is preceded by a MIN_ or ADD_ prefix. No error is associated with this parameter being set too low. If it is inappropriately low, there may be a slowdown in the response rate for $ENQ's and $DEQ's (enqueuing and dequeuing locks). ------------------------------------------------------------------------ NOTES: (1) Applications which use global buffers on RMS files should include a workaround for the RMS-F-CRMP error so that the programs do not abort. The workaround is to include a test for a return from a $CONNECT of the status code RMS$_CRMP and, when this is returned, take the following actions: 1. Print a message to the user that the RMS-F-CRMP error occurred and the system manager should be informed that one or more of the SYSGEN parameters GBLSECTIONS, GBLPAGES or GBLPAGFIL was set too low on this particular system to map a global section for RMS global buffers. The filename should be displayed as part of the error message. 2. Assign a zero to the FAB$W_GBC field. This will override the number of global buffers requested (either in the file header or by a nonzero assignment to FAB$W_GBC), and RMS will not attempt to set up a global cache for this file on a connect. 3. Re-call $CONNECT. NOTE: The above suggestions assume that an application is using RMS directly or is using a high level language that provides a user-action routine option as part of the file open. (2) The following information discussing the SYSGEN parameter LOCKIDTBL_MAX is version specific. OpenVMS VAX versions 5.0 through 5.4-3 -------------------------------------- LOCKIDTBL_MAX establishes the maximum size of the system Lock ID table. As a general rule, it should be twice the value of LOCKIDTBL. AUTOGEN does not calculate this value. If it is omitted from the MODPARAMS.DAT file, it will be set to the default. On booting a system, a lock database with LOCKIDTBL entries is built. If this becomes full, it can be expanded to at least LOCKIDTBL_MAX entries (provided that LOCKIDTBL_MAX is greater than LOCKIDTBL) while the system is running. The expansion is done in increments of LOCKIDTBL. For Example: LOCKIDTBL is 65000 and LOCKIDTBL_MAX is 65001. The lock ID table becomes full. The system finds that LOCKIDTBL_MAX is greater than LOCKIDTBL. Since it is, it will expand the lock ID table by LOCKIDTBL. The lock ID table now has 130,000 entries. However, if it becomes full again, LOCKIDTBL_MAX (65001) will not be greater than LOCKIDTBL (130,000). In this example, the lock id table will only be expanded once, but note that its size will exceed LOCKIDTBL_MAX. The expanded lock id table is built in a block of memory allocated from nonpaged pool. You need to have enough contiguous pages in nonpaged pool to accommodate the entire size of the NEW expanded table because the old table isn't deleted until the new one is successfully created. OpenVMS VAX versions 5.5 through 6.2 ------------------------------------ Starting in OpenVMS VAX versions 5.5, the LOCKIDTBL_MAX parameter is used at SYSBOOT time in the calculation which determines possible "System Page Table Entry" (SPTE) requirements. The algorithm governing the extension of the Lock ID table was also modified. The algorithm now only checks the availability of SPTE's and fluid pages before a 1 or 2 page extension is added to the Lock ID table. REFERENCES: Additional information about this topic can be found in other database articles with the following titles: Process Limits and Quotas Related to RMS Buffers Command Procedure To Estimate SYSGEN Units for RMS Global Buffers Estimating SYSGEN Units Currently in Use for RMS Global Buffers [OpenVMS] OPEN with Global Buffers Fails with RMS-F-CRMP, RMS-F-DME For additional information on the AUTOGEN command procedure refer to the "Guide to Setting Up a VMS System", April 1988, (AA-LA25A-TE), Chapter 6. For additional information on VMS memory management and lock management refer to the "Introduction to VMS System Services", April 1988, (AA-LA68A-TE), Chapters 11 & 12. For additional information on file performance tuning and file sharing refer to the "Guide to VMS File Applications", (April 1988), (AA-LA78A-TE), Chapters 3 & 7. Process Limits and Quotas Related to RMS Buffers Any party granted access to the following copyrighted information (protected under Federal Copyright Laws), pursuant to a duly executed Compaq Service Agreement may, under the terms of such agreement copy all or selected portions of this information for internal use and distribution only. No other copying or distribution for any other purpose is authorized. Copyright (c) Digital Equipment Corporation 1990, 1991. All rights reserved COMPONENT: RMS OP/SYS: VMS, All Versions SOURCE: Digital Customer Support Center OVERVIEW: This article describes the following process limits and quotas directly related to RMS local and global buffers: 1. Working set size (WSQUOTA, WSEXTENT) 2. PGFLQUOTA 3. PIOPAGES 4. ENQLM DISCUSSION: 1. WORKING SET SIZE (WSQUOTA, WSEXTENT) o Using LOCAL buffers The number of pages of working set needed to support local buffers for each file is as follows: Number of pages used per local cache = number of buffers * maximum_bucket_size NOTE: The 'virtual' bucket size for a sequential file equals the multiblock count (MBC). If a sequential file is opened with shared write access, it is set to the MBC of the first accessor of the file. The MBC system default is set by the SYSGEN parameter RMS_DFMBC (current default is 16). Since these pages are private to a process, the process must have a working set size large enough to hold the number of pages required. Larger local caches without an increase in working set size can induce excessive paging. o Using GLOBAL buffers The pages used by global buffers are mapped through a process's PO space. The pages of a particular buffer referenced by the process will be part of the working set of that process. However, no one process mapped to a global cache has to have all the global buffers in their working set at one time. The responsibility is distributed across a number of processes. The more processes mapped to a particular global cache and actively doing I/O to that file, the more distributed the actual pages in the global cache are across the working sets of these processes. On the other hand, if the global cache is large and only a few processes are actively using the cache, the more likely paging will occur. Under these conditions, the state of the free page list is very important. The more free memory that exists on a system, the better the chances that buffer pages which are paged out of the working set will still be on the modified page list (or even the free page list) when requested again. In other words, a 'hard' page fault (one that requires a disk I/O) may not occur. 2. PGFLQUOTA o Both local and global buffers are mapped through a process's virtual address space. PGFLQUOTA sets an upper limit on the maximum number of pages that a process can use in the system paging file. o Buffers are allocated at image runtime. If several files are opened by an image and a large number of local and global buffers are connected to the record streams for these files, the following error may occur (at runtime): RMS-F-DME, Dynamic memory exhausted o The SYSGEN parameter VIRTUALPAGECNT sets an upper limit on the maximum any process's PGFLQUOTA can be raised to. In order to get a good estimate of what PGFLQUOTA should be increased to, first use the AUTHORIZE utility to set it to a high value for the particular user who incurred the error. The user should then log off, log back in, and run the image again. When the image completes, the user should logout using the qualifier '/FULL'. The display that appears includes a 'Peak page file size' entry that indicates how much PGFLQUOTA was needed by this job. This provides a lower-bound estimate of what PGFLQUOTA needs to be for this user. o If the DME error is incurred on the open of a process permanent file (one opened from DCL), the SYSGEN parameter PIOPAGES should be increased. 3. PIOPAGES o Files opened at DCL level are opened as process permanent files. Files opened from program control are opened as image files. o Process permanent files (notably SYS$INPUT, SYS$OUTPUT, and SYS$ERROR) must have their RMS internal structures mapped through P1 process space, in order to survive image deletion. PIOPAGES sets an upper limit on the number of P1 pages that can be used for this purpose. o When the RMS-F-DME error is incurred trying to open a process permanent file, PIOPAGES needs to be increased. o The algorithm for estimating the number of PIOPAGES needed for each process permanent file is: Number of PIOPAGES per process permanent file = Number of buffers * maximum_bucket_size NOTE: The 'virtual' bucket size for a sequential file equals the multiblock count (MBC). If a sequential file is opened with shared write access, it is set to the MBC of the first accessor of the file. The MBC system default is set by the SYSGEN parameter RMS_DFMBC (current default is 16). 4. ENQLM o Using LOCAL buffers If the file is not shared write, 2 units of ENQLM are required. One for a file lock and one for an extend lock, which is a sublock of the file lock. If the file is shared write, the number of ENQLM units can be estimated as follows: File and extend locks: 2 Local buffer locks - Number of local buffers minus the one used for extend: # of buffers - 1 Record locks for indexed files (no record locking is done for sequential or relative files): -- Automatic locking - Only one record lock block (RLB) is used at a time: 1 -- Manual locking - Requires as many RLB's as number of records requested before an explicit UNLOCK is done on the file or the file is closed: # of records - 1 NOTE: As of VMS Version 5.4, the default number of local buffers that are allocated when an indexed file is accessed may increase slightly. This will result in more ENQLM units being used. Any process that ran on a marginal ENQLM under pre-VMS 5.4 could experience an ENQLM problem after upgrading. For more information, see page 4-41 of the "VMS Version 5.4 Release Notes", August 1990, (AA-LB22C-TE). o Using GLOBAL buffers The number of ENQLM units can be estimated for global buffers as follows: File and extend lock: 2 Global section lock: 1 Local buffer locks - Number of local buffers minus the one used for extend: # of buffers - 1 Global buffer pointer block (GBPB) maintained in local cache. No matter how large the global cache is, there will only be 2 GBPB's (see note 1): 2 Record locks for indexed files (no record locking is done for sequential or relative files): -- Automatic locking - Only one record lock block (RLB) used at a time 1 -- Manual locking - Requires as many RLB's as records requested before an explicit UNLOCK is done on the file or the file is closed. # of records - 1 ----------------------------------------------------------------- Note (1): The two GBPB's are used for any process locks taken out on a global buffer accessed by a process. When the process lock is released, it is converted to a system NL lock in order to survive the closing of the file by the process, image exit or process deletion. System locks are maintained for each global buffer (once they are used) until the global section for that file is removed. No process limit is charged for system locks. REFERENCES: Additional information about this topic can be found in three other database articles with the following titles: Command Procedure To Estimate SYSGEN Units for RMS Global Buffers Estimating SYSGEN Units Currently in Use for RMS Global Buffers SYSGEN Parameters Related To RMS Global Buffers For additional information on file performance tuning and file sharing refer to "Guide to VMS File Applications", (April 1988), (AA-LA78A-TE), Ch. 3 & 7. For additional information on VMS memory management and lock management refer to the "Introduction to VMS System Services", April 1988, (AA-LA68A-TE), Chapters 11 & 12. [OpenVMS] Guidelines for Indexed File Optimization and Performance Any party granted access to the following copyrighted information (protected under Federal Copyright Laws), pursuant to a duly executed Compaq Service Agreement may, under the terms of such agreement copy all or selected portions of this information for internal use and distribution only. No other copying or distribution for any other purpose is authorized. Copyright (c) Compaq Computer Corporation 1995, 1998. All rights reserved. PRODUCT: DIGITAL OpenVMS VAX, All Versions DIGITAL OpenVMS Alpha, All Versions COMPONENT: Record Management Services (RMS) SOURCE: Compaq Computer Corporation OVERVIEW: RMS indexed files that are actively updated (i.e., with record writes, updates and deletions) need regular maintenance and can benefit greatly in terms of performance by regularly tuning the file's internal structure. This article discusses the general tuning procedure and makes tuning recommendations based on file design criteria. PROCEDURE: The tuning of an indexed file involves using the CONVERT Utility on the file to reload all the data and key indexes in key order, to eliminate bucket splits and reclaim deleted buckets. 1. The procedure to tune an indexed file's size and structure involves three steps: A. Create an FDL file that contains file attribute definitions and statistical information that describe the current state of the indexed file. For example, the following command analyzes ABC.DAT and creates file ABC.FDL;1: $ ANALYZE /RMS /FDL ABC.DAT B. Invoke the FDL Editor's Optimize script, (reads the FDL language and statistical information in the FDL file created by ANALYZE/RMS/FDL) and calculates the optimal number of data and index areas, bucket sizes, and index depth for this indexed file. The following command uses the /NOINTERACTIVE qualifier to direct the FDL Editor to execute the Optimize script without a terminal dialog: $ EDIT /FDL /ANALYSIS=ABC.FDL /NOINTERACTIVE ABC.FDL The default file type for this command is .FDL. On successful execution of this command, the FDL Editor creates ABC.FDL;2. This file contains definition language that, by default, describes a "flatter" indexed file (that is, one with fewer index levels). To create an output FDL file with a unique name, specify the /OUTPUT= qualifier on the command line: $ EDIT /FDL /ANALYSIS=ABC.FDL - /NOINTERACTIVE/OUTPUT=ABC_NEW ABC This command invokes the interactive Optimize script: $ EDIT /FDL /ANALYSIS=ABC.FDL /SCRIPT=OPTIMIZE ABC.FDL NOTE: This command may create an FDL file that defines a different bucket sizes than the original file has. If you've assigned global buffers to the indexed file, the change in bucket size may impact the number of global buffers desired for the file. C. Use the FDL file, created by the FDL Editor, to create an optimized version of the indexed file. For example: $ CONVERT/FDL=ABC_NEW/STATISTICS /NOSORT ABC.DAT ABC.NEW $ RENAME ABC.NEW ABC.DAT 2. Using the FDL Editor's Optimize script yields different results, depending on whether or not it runs interactively: A. Use of the /NOINTERACTIVE qualifier invokes the Optimize script. This produces an FDL file that may contain: o A different number of index and data areas. For example, if there is only 1 area in the original indexed file, the optimized FDL file defines two or more areas, unless /GRANULARITY=1 was specified on the EDIT/FDL command line o New bucket sizes for the index and/or data areas o A different number of blocks allocated to the file o A default extend size for each area that is recalculated to about 25% of the initial allocation for that area B. Using /SCRIPT=OPTIMIZE (without /NOINTERACTIVE) causes the FDL Editor to invoke the interactive Optimize script. By answering the questions in the terminal dialog, one can select different characteristics for each key. One can also change the default emphasis of the file (i.e., to smaller buffers or flatten the file's index structure), and select a new bucket size for the index and data buckets of that index structure. 3. The frequency of file optimization depends on the file's use: A. If the file is actively updated (with records being added, updated and deleted) but the number of records is relatively constant, it should be necessary to tune the file only once. Invoke the FDL Editor's Optimize script (/SCRIPT=OPTIMIZE) and indicate the maximum number of records the file should contain at any one time, by answering these script questions: Number of Records that will be Reloaded into the File (0-2Giga)[n] : Number of Additional Records to be Added After the Reloading the File (0-2147482672)[0] : Because activity on the file can cause internal fragmentation, use CONVERT to periodically rebuild the indexed file: $ CONVERT /STATISTICS /NOSORT ABC.DAT ABC.NEW $ RENAME ABC.NEW ABC.DAT B. If the number of records in the file increases over time, it should be tuned periodically, in order to plan for the file's expected growth during the period of time until the file is next tuned. To do this, execute the FDL Editor's Optimize script and indicate the number of records to be added during that time period: Number of Additional Records to be Added After the Reloading the File (0-2147482672)[0] : For example, if you plan to tune the file every month, indicate the number of additional records you will be adding in that month. Between file tuning, regular use of CONVERT to rebuild the file keeps internal fragmentation from bucket splitting to a minimum. 4. Suggestions for file definition attributes, to increase performance: A. Disable index compression. When index compression is enabled, RMS must sequentially search through the index when performing a lookup of a key value or writing a new record to the file. When it is disabled, RMS performs a binary search. B. Disable data key compression. This does not dramatically improve RMS performance, but reduces the amount of CPU that is needed to compress and decompress the data key values when writing, updating or reading a record. C. If available disk space is not an issue and data record compression is not efficiently reducing the space needed to store a record, disable data record compression. Review the FDL file created by ANALYZE/RMS/FDL, to find the compression efficiency percentage. This information is found in the 'ANALYSIS OF KEY 0' section, in the value for the DATA_RECORD_COMPRESSION field. The larger the number, the more effective the data record compression. D. Avoid duplicate values for key values or make key values as unique as possible. When RMS searches for a particular key value in an alternate key and duplicate records are allowed for the index of reference, RMS may have to search through a chain of records with the same key value to find the record in question. Because RMS sequentially traverses this duplicate chain, a search for a specific record in a long duplicate chain represents a major performance problem for read and delete operations. This problem is compounded for update operations that change the key value, since RMS must traverse the duplicate chain to find the records that has the old key value, delete that alternate key's record (known as a SIDR), traverse the duplicate chain that has the new key value and add a SIDR for the record to the end of that chain. If it is not possible to disallow duplicate key values, it may be possible to make the key more unique. For example, a string key can be made more unique by adding to it a key segment for a field that has some unique value, or by adding to the key field a more unique value. Then, if the key value is not unique, a call to $GET might specify only the first part of the key; in that case, sequential $GET calls can be used to find the record with the the particular key that is being sought. E. Use a fill factor of less than 100%, if records will be randomly added to the file. If records are added with an increasing key value (i.e., key is a log number that is incremented for each new record), select a load fill of 100%. When selecting a load fill factor that is less than 100%, set it to an amount that allows room in the bucket for at least one additional record. For example, if four records can fit into a data bucket, setting the fill factor to 75% causes the CONVERT utility to place three records per bucket, leaving room for one record in each data bucket. F. Allocate additional space in the indexed file, for records that will be added after CONVERT initially loads records to the file. As described earlier in this article, execute the FDL Editor's Optimize script and indicate the number of additional records that will be added after CONVERT reloads the file: Number of Additional Records to be Added After the Reloading the File (0-2147482672)[0] : 5. To reduce index fragmentation when writing records to a new file: When loading records to a new file from program control, write the records to a sequential file first. Then use CONVERT/FDL to load the indexed file from the sequential file. Since CONVERT builds one index at a time, this method loads the file's index structures without causing the internal fragmentation that can result if the program loads the records to an indexed file. RELATED ARTICLES: Other articles in the OPENVMS database provide additional information on RMS indexed files. These articles can be found using a search string of: RMS INDEXED FILES ANALYZE/RMS FDL REFERENCES: "Guide to OpenVMS File Applications", (AA-PV6PA-TK), May 1993, Chapter 3, Chapter 10, Appendix A "OpenVMS Record Management Services Reference Manual", (AA-PV6RA-TK), May 1993 "OpenVMS Record Management Utilities Reference Manual", (AA-PV6QA-TK), May 1993, Chapter 1, Chapter 2, Chapter 6 [OpenVMS] Estimating SYSGEN Units Currently in Use for RMS Global Buffers Any party granted access to the following copyrighted information (protected under Federal Copyright Laws), pursuant to a duly executed Compaq Service Agreement may, under the terms of such agreement copy all or selected portions of this information for internal use and distribution only. No other copying or distribution for any other purpose is authorized. Copyright (c) 1990, 2000. Compaq Computer Corporation. All rights reserved. PRODUCT: Compaq OpenVMS VAX, All Versions COMPONENT: RMS SOURCE: Compaq Computer Corporation OVERVIEW: This article describes how to estimate units currently used by RMS global buffers for the following SYSGEN parameters: 1. GBLSECTIONS 2. GBLPAGES 3. GBLPAGFIL 4. RMS_GBLBUFQUO* (see note below) *Note: RMS_GBLBUFQUO is obsolete as of OpenVMS V7.1. Caution: In deriving any estimates of how many of the units set up by these SYSGEN parameters are currently in use, it must be noted that they are based on a "snapshot" of the system at a particular moment. Any RMS file with global buffers defined (either by a SET FILE/GLOBAL DCL command or via setting FAB$W_GBC at runtime) that is not open at this moment will not be included in the estimation. Any program containing a call to the system service $CRMPSC that is not running at this moment will not be included in the estimation. In other words, any estimate that is derived should be considered an "UNDERESTIMATE." DISCUSSION: 1. GBLSECTIONS The DCL lexical function F$GETSYI can be used to determine how many units of GBLSECTIONS are NOT in use: $ WRITE SYS$OUTPUT F$GETSYI("FREE_GBLSECTS") 9 2. GBLPAGES The DCL lexical function F$GETSYI can be used to determine how many contiguous, free GBLPAGES are available: $ WRITE SYS$OUTPUT F$GETSYI("CONTIG_GBLPAGES") 396 The number of unused pages reported by the INSTALL utility includes the TOTAL number of free pages and can be misleading. $ INSTALL LIST/GLOBAL/SUMMARY Summary of Local Memory Global Sections 261 Global Sections Used, 14998/402 Global Pages Used/Unused 3. GBLPAGFIL There are two alternative ways to estimate GBLPAGFIL units currently in use. ALTERNATIVE #1: This is the simplest way to estimate GBLPAGFIL units currently in use. However, it requires SYSPRV and CMKRNL privileges. In addition, it is based on a symbol that is available through the current version of VMS, but there is no guarantee that this symbol will be available in the future. $ SET PROCESS/PRIVILEGE=(SYSPRV,CMKRNL) $ ANALYZE/SYSTEM VAX/VMS System analyzer SDA> EXAMINE SGN$GL_GBLPAGFIL ! SYSGEN GBLPAGFIL SGN$GL_GBLPAGFIL: 00000400 "...." ! Hex 400 = decimal 1024 SDA> EXAMINE MMG$GL_GBLPAGFIL ! GBLPAGFIL remaining MMG$GL_GBLPAGFIL: 000003B6 "¶..." ! Hex 3B6 = decimal 950 ! MMG$GL_GBLPAGFIL starts out as a positive integer equal ! to SGN$GL_GBLPAGFIL. MMG$GL_GBLPAGFIL is decremented for ! each page created for a WRT/DZRO global section. In other ! words, the current value in MMG$GL_GBLPAGFIL indicates the ! number of unused GBLPAGFIL units. When MMG$GL_GBLPAGFIL ! drops to zero, then no more GBLPAGFIL quota exists. SDA> EXIT ! Symbols used to find amount of GBLPAGFIL currently used $ GBLPAGFIL_SYSGEN = 1024 $ GBLPAGFIL_AVAILABLE = %X000003B6 $ SHOW SYMBOL GBLPAGFIL_AVAILABLE GBLPAGFIL_AVAILABLE = 950 Hex = 000003B6 Octal = 00000001666 ! GBLPAGFIL_USED = GBLPAGFIL_SYSGEN - GBLPAGFIL_AVAILABLE ! = 1024 - 950 ! GBLPAGFIL_USED = 74 ALTERNATIVE #2: Any global sections that have GBLPAGFIL units deducted must have the PSECT attributes WRT (writable) and DZRO (PAGEFILE.SYS used as backing storage). This method uses the INSTALL Utility's LIST/GLOBAL command to identify any global sections that have both of these attributes set. Step 1: Obtain listing of all global sections on system: $ DEFINE/USER SYS$OUTPUT glob_install.lis $ INSTALL LIST/GLOBAL $ DIRECTORY/SIZE glob_install.lis GLOB_INSTALL.LIS;1 38 Step 2: Use DCL SEARCH command to select only those global sections that have the PSECT ATTRIBUTES writable (WRT) and DZRO (PAGEFILE.SYS used as backing storage). $ SEARCH/EXACT/MATCH=AND/OUTPUT=glob_wrt.lis - _$ glob_install.lis WRT,DZRO $ TYPE glob_wrt.lis OSIF$MANAGEMENT(0A000001) WRT DZRO PRM SYS Pagcnt/Refcnt=5/5 RMS$804BDCA0 (00000000) WRT DZRO TMP SYS Pagcnt/Refcnt=69/69 Refcnt = pagcnt * number of users Step 3: Sum 'pagcnt' entries with both WRT and DZRO attributes. The total is equal to the number of GBLPAGFIL units currently in use. ----- Number of GBLPAGFIL units in use 74 The global sections for RMS global buffers always begin with an RMS$ prefix. In the output above, one RMS file active on the system currently has global buffers set (RMS$804BDCA0). It indicates that 69 GBLPAGES are being used by this file. This does not tell us how many global buffers (RMS_GBLBUFQUO units) are being used. The estimation procedure for RMS_GBLBUFQUO below is in "buffer" units. NOTE: There is one case in which a data file can be used as the backing storage for a global section. This occurs when a call is made to the system service SYS$CRMPSC specifying a channel to the data file and the DZRO attribute (as well as GBL and WRT). Since the system PAGEFILE.SYS is not used as the backing storage for this section, GBLPAGFIL units are not deducted. However, this global section DOES have both the WRT and DZRO attributes in the INSTALL listing so there is no way to identify that this section is not using GBLPAGFIL units. Thus, the estimate of GBLPAGFIL units currently in use derived from this alternative could provide an "overestimate". In contrast, the first alternative always provides an exact number at the particular time it is checked using SDA. 4. RMS_GBLBUFQUO (Note: RMS_GBLBUFQUO is obsolete as of OpenVMS V7.1.) SDA provides the only means of finding out how many active RMS global buffers are currently in use on a particular system. A logfile with the steps necessary follows: Step 1: Put EXEC writable page offsets into an object file. $ MACRO/OBJECT=ew_data SYS$INPUT:+SYS$LIBRARY:LIB/LIB $EWDATADEF GLOBAL .END $ DIRECTORY/SIZE ew_data.obj EW_DATA.OBJ;1 3 Step 2: Invoke SDA $ SET PROCESS/PRIVILEGE=(SYSPRV,CMKRNL) $ ANALYZE/SYSTEM VAX/VMS System analyzer Step 3: Read in EXEC writable page offsets SDA> READ ew_data.obj %SDA-I-READSYM, reading symbol table DISK:[USER]EW_DATA.OBJ;1 Step 4: Examine GBLBUFQUO current value SDA> EXAMINE @exe$AR_EWDATA+ew_RMS$GW_GBLBUFQUO SYS$NETWORK_SERVICES+00778: 0000FFFD "ý..." ! decimal -3 ! RMS$GW_GBLBUFQUO is a negative counter. It is initialized ! to zero when the system boots and is decremented for each ! GBLBUFQUO unit used. The -3 (FFFD) is accounted for by the ! one RMS global section located in the INSTALL listing: ! ! RMS$804BDCA0 WRT DZRO TMP SYS Pagcnt/Refcnt=69/69 ! ! Three global buffers are being used by this file. ! ! A break was done here to run an image that opens an RMS ! indexed file (INTER11.DAT) which had 10 global buffers set ! on it. A DIRECTORY/FULL of the file after the global ! buffers were set showed the following file attributes: ! ! $ DIRECTORY/FULL INTER11.DAT ! ... ! Allocation: 21, Extend: 1, Maximum bucket size: 1 ! Global buffer count: 10, No version limit ! ... ! ! Two records were added to the file and the image was ! paused just prior to the $CLOSE operation. ! Return to the SDA session. SDA> ex @exe$AR_EWDATA+ew_RMS$GW_GBLBUFQUO SYS$NETWORK_SERVICES+00778: 0000FFFA "ú..." ! decimal -6 ! Three additional RMS_GBLBUFQUO units are in use. But the ! file had 10 global buffers set on it! Why the discrepancy? ! Because only 3 out of 10 global buffers are currently in ! use for the INTER11.DAT file! In other words, units of ! GBLBUFQUO are deducted as they are used! SDA> SET PROCESS/INDEX=21600572 SDA> SET RMS=GBDSUM SDA> SHOW PROCESS/RMS Process index: 0072 Name: _LTA31: Extended PID: 21600572 ------------------------------------------------------------ GBD Summary ----------- GBD USE REL_ CACHE_ Address CNT SIZE NUMB VBN VBNSEQNUM ADDR VAL LOCK_ID ------- ----- ---- ---- --- ------- ---- --- ------- 00033488 0 512 512 8 00000000 00000600 1 018C0309 00033458 0 512 512 9 00000000 00000400 2 00140321 000334B8 0 512 512 14 00000000 00000800 1 00200320 00033608 0 512 0 -1 00000000 00001600 0 00000000 000335D8 0 512 0 -1 00000000 00001400 0 00000000 000335A8 0 512 0 -1 00000000 00001200 0 00000000 00033578 0 512 0 -1 00000000 00001000 0 00000000 00033548 0 512 0 -1 00000000 00000E00 0 00000000 00033518 0 512 0 -1 00000000 00000C00 0 00000000 000334E8 0 512 0 -1 00000000 00000A00 0 00000000 10 GBDs were processed ! NOTE: File has not been closed in image yet. ! Exit from SDA and obtain information from INSTALL while ! global section is still active for this file. SDA> EXIT $ DEFINE/USER SYS$OUTPUT glob_install.lis $ INSTALL LIST/GLOBAL $ SEARCH/EXACT/MATCH=AND/OUTPUT=glob_writ.lis - _$ glob_install.lis WRT,DZRO $ TYPE glob_writ.lis RMS$8053CBD0 (00000000) WRT DZRO TMP SYS Pagcnt/Refcnt=13/13 OSIF$MANAGEMENT(0A000001) WRT DZRO PRM SYS Pagcnt/Refcnt=5/5 RMS$804BDCA0 (00000000) WRT DZRO TMP SYS Pagcnt/Refcnt=69/69 REFERENCES: Additional information about this topic can be found in other database articles with the following titles: Process Limits and Quotas Related to RMS Buffers Command Procedure To Estimate SYSGEN Units for RMS Global Buffers SYSGEN Parameters Related To RMS Global Buffers For additional information on file performance tuning and file sharing refer to "Guide to VMS File Applications", April 1988, (AA-LA78A-TE), Chapters 3 & 7. For additional information on VMS memory management and lock management refer to the "Introduction to VMS System Services", April 1988, (AA-LA68A-TE), Chapters 11 & 12. For additional information on using SDA refer to the "VMS System Dump Analyzer Utility Manual", April 1988, (AA-LA87A-TE). [OpenVMS] Command Procedure To Estimate SYSGEN Units for RMS Global Buffers Any party granted access to the following copyrighted information (protected under Federal Copyright Laws), pursuant to a duly executed Compaq Service Agreement may, under the terms of such agreement copy all or selected portions of this information for internal use and distribution only. No other copying or distribution for any other purpose is authorized. Copyright (c) 1990, 2000. Compaq Computer Corporation. All rights reserved. PRODUCT: Compaq OpenVMS VAX, All Version COMPONENT: RMS SOURCE: Compaq Computer Corporation OVERVIEW: This article contains a command procedure that can be used to estimate the units needed to support the addition of new RMS global buffers to a system. The following SYSGEN parameter additions are estimated: 1. RMS_GBLBUFQUO *(see note below) 2. GBLSECTIONS 3. GBLPAGES 4. GBLPAGFIL *Note: RMS_GBLBUFQUO is obsolete as of OpenVMS V7.1. The algorithms used in the command procedure to calculate the values are described in another database article. Only two items of information are needed for each indexed file on which global buffers have been set to estimate the number of SYSGEN units that will be consumed by that file. Use the DCL command DIRECTORY/FULL or the lexical function F$FILE_ATTRIBUTES to obtain the following information: 1. Global buffer count (GBC) 2. Maximum bucket size (BKS) PROCEDURE: $ ! RMSGLOB_SYSGEN.COM $ !+ $ ! $ ! Command procedure to estimate RMS global buffer SYSGEN $ ! units needed for one indexed file. This command procedure $ ! can be adapted to accumulate these units across a number $ ! of indexed files. $ ! $ ! CAUTION: This sample command procedure has been tested $ ! using VMS 5.4-3. However, we cannot guarantee its $ ! effectiveness because of the possibility of error in $ ! transmitting or implementing it. It is meant to be used $ ! as a template for writing your own command procedure, and $ ! it may require modification for use on your system. $ ! $ !- $ ON WARNING THEN GOTO DONE $ IF P1 .EQS. "" THEN $INQUIRE P1 "FILE SPEC " $ P1 = F$SEARCH (P1, 1) ! Returns full filespec $ gbc = F$FILE_ATTRIBUTES (P1,"GBC") ! GBC available as of $ IF gbc .EQ. 0 THEN GOTO NOGBC ! VMS V5.2 $ bks = F$FILE_ATTRIBUTES (P1,"BKS") $ RMS_GBLBUFQUO = gbc $ GBLSECTIONS = 1 $ k = 64 + (gbc * 48) ! GBH and GBD descriptor bytes $ GBLPAGFIL = ((gbc * bks * 512) + k + 1023)/512 $ GBLPAGES = GBLPAGFIL + 2 ! plus 2 stopper pages $ WRITE SYS$OUTPUT "" $ WRITE SYS$OUTPUT "SYSGEN values needed by ",P1 $ WRITE SYS$OUTPUT "" $ WRITE SYS$OUTPUT " RMS_GBLBUFQUO = ",RMS_GBLBUFQUO $ WRITE SYS$OUTPUT " GBLSECTIONS = ",GBLSECTIONS $ WRITE SYS$OUTPUT " GBLPAGES = ",GBLPAGES $ WRITE SYS$OUTPUT " GBLPAGFIL = ",GBLPAGFIL $ GOTO DONE $ NOGBC: $ WRITE SYS$OUTPUT "" $ WRITE SYS$OUTPUT "Following file did not have global buffers set: " $ WRITE SYS$OUTPUT " ",P1 $ DONE: $ EXIT EXAMPLE: $ DIRECTORY/FULL tuning.dat Directory DISK:[DIR]TUNING.DAT;1 TUNING.DAT;1 File ID: (22486,1,0) Size: 1554/1554 Owner: [USER] Created: 18-JAN-1990 21:00:48.01 Revised: 17-MAR-1990 15:51:05.08 (11) Expires: Backup: File organization: Indexed, Prolog: 3, Using 4 keys In 3 areas > File attributes: Allocation: 1554, Extend: 252, Maximum bucket size: 18 > Global buffer count: 50, No version limit Contiguous best try Record format: Fixed length 80 byte records Record attributes: Carriage return carriage control RMS attributes: None Journaling enabled: None File protection: System:RWED, Owner:RWED, Group:RE, World: Access Cntrl List: None $ @rmsglob_sysgen TUNING.DAT SYSGEN values needed by DISK:[DIR]TUNING.DAT;1 RMS_GBLBUFQUO = 50 GBLSECTIONS = 1 GBLPAGES = 908 GBLPAGFIL = 906 $ SET FILE/GLOBAL=0 TUNING.DAT $ @rmsglob_sysgen TUNING.DAT Following file did not have global buffers set: DISK:[DIR]TUNING.DAT;1 REFERENCES: Additional information about this topic can be found in other database articles with the following titles: Process Limits and Quotas Related to RMS Buffers Estimating SYSGEN Units Currently in Use for RMS Global Buffers SYSGEN Parameters Related To RMS Global Buffers For additional information on file performance tuning and file sharing, refer to the "Guide to VMS File Applications", April 1988, (AA-LA78A-TE), Chapters 3 & 7. For additional information on VMS memory management and lock management, refer to "Introduction to VMS System Services", April 1988, (AA-LA68A-TE), Chapters 11 & 12. WorkUnit 0010 ** The previous work unit was queued by DSNlink to be sent to the customer. WorkUnit 0011 ---------< This work unit was entered by DSNlink - 11 lines >---------- DSNlink Mail Delivery Failure Notice ------------------------------------ DSNlink was unable to deliver your mail message (workunit) to this customer after repeated attempts. Please contact your customer via an alternative method, such as voice, until this problem can be corrected. WorkUnit 0012 \+----------------------- Beginning of Screener's Text -----------------------+ \(PR3)RMS Global buffer expected behavior? \DSNlink File Copy Failure \** This is additional information to the call owned by 152303 \+--------------------------- End of Screener's Text --------------------------+ Since I sent Email to Bill concerning this and since DSNlink keeps failing. I will close this for now and reopen as apporpriate if Bill replies back to me. WorkUnit 0013 The following Email was sent to BILL.NORTON@WCFA.COM: Bill, I replied yesterday to your questions posted in DSNlink service request C010205-2461. However, multiple DSNlink delivery attempts have failed so I am sending this information directly to your Email address and will also try posting it one more time to DSNlink. If you do receive this information in Email, please reply back so that I will know that you have received it. Joe Albanese OpenVMS Support Internals/Drivers Response posted on February 5, 2000: Bill, Below I have included multiple articles from our DSNlink database that should help to address your questions. If you have additional questions or problems after reviewing this information, please let us know. Joe Albanese OpenVMS Support Internals/Drivers Rules of Thumb: Number of Buffers For Indexed Files Any party granted access to the following copyrighted information (protected under Federal Copyright Laws), pursuant to a duly executed Compaq Service Agreement may, under the terms of such agreement copy all or selected portions of this information for internal use and distribution only. No other copying or distribution for any other purpose is authorized. Copyright (c) Digital Equipment Corporation 1987, 1991. All rights reserved COMPONENT: Record Management Services (RMS) OP/SYS: VMS SOURCE: Digital Customer Support Center VERSION INFORMATION: Information Applies To: VMS, All Versions OVERVIEW: This information was gathered from the "RMS Structures and Utilities" training class available through Educational Services of Digital Equipment Corporation. *** CAUTION *** Too many global buffers can have a serious negative impact on performance. Depending on the number of active concurrent users, the locking contention on the global section required to scan the GBD (Global Buffer Descriptor) list and the sequential scan of the GBD linked list itself can make performance worse than if the file had no global buffers. Before implementing global buffers on a particular file, always perform benchmarks with and without them. DISCUSSION: General Guidelines: Number Of Buffers For Indexed Files In general, the maximum number of buffers (local or global) that should be used for any index tree to be accessed in an indexed file is the following: o Maximum # buffers = SUM of [(total # index buckets) + 1 (for data)] for each index PLUS the average number of users The "sum" indicated would be across all index trees to be accessed. In the case of local buffers, the average number of users would be equal to 1. This total number provides the user with an "upperbound" estimate. (Refer to Note 1 below.) For moderate to large sized files this upperbound buffer estimate is generally more than the user can afford for local buffering or the system can afford for global buffering. NOTE: As of VMS 4.5, the maximum number of local buffers that can be specified for any file is 127. Algorithms: Estimating Minimum LOCAL and GLOBAL Buffers The following algorithms provide a good starting point for estimating the "minimum" number of LOCAL buffers that should be used. The algorithms described below for "clustered" reads or updates provide a good "lowerbound" estimate for GLOBAL buffers as well, but note that these algorithms assume the average number of users is 1. In practice the actual number of buffers used will fall somewhere in the range between the "lowerbound" and "upperbound" estimates derived. In the case of shared files that have a number of concurrent accessors, every attempt should be made to cache the top-most levels of the index structure in global buffers, particularly for files that have more than two (2) index levels in the primary structure. 1. KEYED READ_ONLY OPERATIONS: A. KEY 0: o Minimum # buffers = (1 for root index bucket) + 1 (data) o lower-bound est. clustered reads = (# levels in index tree) + 1 (data) B. ALTERNATE KEY: o Minimum # buffers = (1 for root index bucket alt key) + 1 (data alt key) + 1 (data key 0) + 1 (RRV) o lower-bound est. clustered reads = (# levels in index tree alt key) + 1 (data alt key) + 1 (data key 0) + 1 (RRV) (Refer to Note 2 below for an explanation of RRV) 2. KEYED (KEY 0) INSERTS OR DELETES (fast delete not enabled): The following estimate should be calculated for each index tree in the file. The minimum number of buffers would be the SUM of the estimates across all index trees: o Minimum # buffers = (# levels in index tree) + 2 (data) (Refer to Note 3 below) 3. KEYED UPDATES (MODIFYs): A. Using Key 0: If the size of the records will not grow larger (i.e., remain the same size or decrease) and no alternate key value will typically be changed by the update: o Minimum # buffers = (1 for root index bucket) + 1 (data) o lower-bound est. clustered updates = (# levels in index tree) + 1 (data) If the size of a record may be expanded by the update, then the possibility that bucket splits may occur must be considered. o Minimum # buffers = (# levels in index tree) + 2 (data) (Refer to Note 3 below) B. Using an Alternate Key: If the update is done using an alternate key, then add 1 to the above Key 0 algorithms to allow for RRVs. (Refer to Note 2 below) SETTING GLOBAL BUFFERS: Global buffers are set on a file-by-file basis either at creation time or at runtime. For example: Creation time ------------- FAB FAB$W_GBC=n $CREATE/FDL=fdl FILE GLOBAL_BUFFER_COUNT=n $CONVERT/FDL=fdl FILE GLOBAL_BUFFER_COUNT=n Runtime ------- FAB FAB$W_GBC=n FDL$PARSE(fdl,...) FILE GLOBAL_BUFFER_COUNT=n DCL $ SET FILE/GLOBAL_BUFFER=n WHERE n = 0 to 32,767 *** CAUTION *** Global Buffers are ignored for process permanent files (DCL OPEN) and when an image is linked using the LINK option keyword IOSEGMENT=NOP0BUFS. NOTES: Note 1: In most cases all index paths down to the level 0 data buckets will not be followed, so the total number of buffers estimated from this algorithm would typically not be needed. Any surplus would in fact be used to cache more data buckets than the one bucket allowed above. Note 2: If the retrieval is done using an alternate key, one additional I/O operation is required if the path to the record to be retrieved from a Key 0, Level 0 data bucket is via an RRV. Note 3: Allow 2 for data to provide for at least a simple bucket split. A bucket split also causes the index structure to have to be updated (or in case of deletion of highest record in data bucket). To locate an index bucket to be updated at any tree level, RMS must re-work its way back down through the tree from the top root index bucket. Requests for bucket locks are only allowed by RMS from one direction -- top down -- otherwise deadlock conditions could occur. This algorithm provides enough buffers to cache the index buckets for the path followed to the current data bucket. [OpenVMS] SYSGEN Parameters Related To RMS Global Buffers Any party granted access to the following copyrighted information (protected under Federal Copyright Laws), pursuant to a duly executed Compaq Service Agreement may, under the terms of such agreement copy all or selected portions of this information for internal use and distribution only. No other copying or distribution for any other purpose is authorized. Copyright (c) 1988, 2000. Compaq Computer Corporation. All rights reserved. PRODUCT: Compaq OpenVMS VAX, All Versions COMPONENT: RMS SOURCE: Compaq Computer Corporation OVERVIEW: There are eight SYSGEN parameters that are directly related to RMS global buffers. They are the following: 1. RMS_GBLBUFQUO* (see note) 5. VIRTUALPAGECNT 2. GBLSECTIONS 6. SYSMWCNT 3. GBLPAGES 7. LOCKIDTBL 4. GBLPAGFIL 8. RESHASHTBL *Note: RMS_GBLBUFQUO is obsolete as of OpenVMS V7.1. Algorithms are provided that can be used to determine the number of units of the first four SYSGEN parameters that are used for RMS global buffer caches. These algorithms require the following two items of information about an RMS file that global buffers are to be added to: 1) Global buffer count (GBC) 2) Maximum bucket size (BKS) NOTE: The "virtual" bucket size for a shared write sequential file is set to the multiblock count (MBC) of the first accessor of the file. The MBC system default is 16. The command procedure SYS$UPDATE:AUTOGEN.COM should be used to adjust the last four parameters in accordance with any modifications made to the first four. DISCUSSION: 1. RMS_GBLBUFQUO (Note: RMS_GBLBUFQUO is obsolete as of OpenVMS V7.1) This is the total number of global buffers over all RMS files allowed to be active simultaneously on a particular system. The need to have a separate SYSGEN parameter for RMS global buffers is explained by the fact that each global buffer requires a system lock. Lock structures are allocated from nonpaged pool (SYSGEN parameter NPAGEDYN). Thus, this quota sets an upper limit on how much of nonpaged pool can be used for system locks on RMS global buffers. RMS_GBLBUFQUO per global cache (file)= GBC When a file is opened that has global buffers specified, the number of global buffers it is requesting is not checked against this quota. This quota is not checked until a global buffer must be used. At that time a check is made to determine whether this quota allows another RMS global buffer to be active. If the quota has been exhausted, then a global buffer is not used. Lack of this resource does NOT cause any error to be returned by RMS (specifically, %RMS-F-CRMP, CRMPSC system service failed to map global buffers). AUTOGEN does not calculate this parameter. 2. GBLSECTIONS This number indicates the number of global section table entries needed in the system header. Each RMS file opened with global buffers requires one GBLSECTION entry (and associated global section descriptor). GBLSECTIONS per global cache (file) = 1 Each global section table entry occupies 32 bytes in the nonpageable area of the system header. Thus, each 19 sections of GBLSECTIONS uses 1 page of nonpaged pool. NOTE: The descriptor associated with the global section is pageable. The AUTOGEN "FEEDBACK" mechanism provides input for this parameter so, if it is specified in the MODPARAMS.DAT file, it should be preceded by a MIN_ or ADD_ prefix. Lack of this resource causes the following error when the record stream is connected to the file: %RMS-F-CRMP, CRMPSC system service failed to map global buffers See note (1) for a workaround to this error. 3. GBLPAGES The maximum number of pages in memory that can be accessed by more than one user. The GBLPAGES used by RMS global buffers are always writable. GBLPAGES per global cache (file) = TRUN [ ( (GBC * BKS * 512) + k + 1023 ) /512 ] + 2 Where k = 64 + ( GBC * 48) 'k' equals the number of bytes needed for the following RMS data structures: Global buffer header (GBH) - 64 bytes: one per global cache (file) Global buffer descriptor (GBD) - 48 bytes: one per global buffer The 1023 bytes listed above are added by RMS for page alignment and internal purposes. The two additional pages (the "+ 2" in the above formula) serve as "stoppers" (limits to modified page write clustering). One is created at the beginning of the section and one at the end. The system page file is used as the backing storage for these pages (with the exception of the 2 stopper pages) when they are removed from the modified page list. GBLPAGES should be generously estimated. It is a cheap parameter. Each 128 entries in the global page table use 4 bytes of nonpaged pool, or another way of expressing it is: each 131,072 pages of GBLPAGES use only 8 pages of nonpaged pool. And the global page table itself is pageable. The AUTOGEN "FEEDBACK" mechanism provides input for this parameter so, if it is specified in the MODPARAMS.DAT file, it should be preceded by a MIN_ or ADD_ prefix. Lack of this resource caused the following error when the record stream is connected to the file: %RMS-F-CRMP, CRMPSC system service failed to map global buffers See note (1) for a workaround to this error. This error occurs if there is an insufficient number of entries remaining in the global page table, but may also occur for a less obvious reason. The entries for a global section in the global page table must be contiguous. Thus, sometimes the number of entries remaining are more than the number needed, but the table is fragmented and does not have the number needed contiguous to each other. The DCL lexical function F$GETSYI can be used to determine whether lack of sufficient contiguous entries is the problem: F$GETSYI ("CONTIG_GBLPAGES") Also note that when the last image mapped to an RMS global section exits, the entries in the global page table do not get immediately freed. The pages mapped through the table must first go to the modified page list and then eventually out to the system paging file. Then the global section can be deleted. Once the global section is deleted, the entries in the global page table are freed up. 4. GBLPAGFIL The maximum number of system-wide pages allowed for global page file sections (or scratch global sections) in the system's PAGEFILE.SYS. GBLPAGFIL per global cache (file) = value calculated for GBLPAGES minus 2 The two 'stopper' pages added to GBLPAGES do not require the system page file as a backing storage and are thus not included in the GBLPAGFIL count. GBLPAGFIL indicates how many of the blocks allocated for the primary and secondary page files can be used for pages paged out of global sections. In order to delete the global section associated with an RMS global cache, the system has to be able to page out the pages used by these buffers. The delete is pending until the pages can be paged out to the page files. The AUTOGEN "FEEDBACK" mechanism provides input for this parameter so, if it is specified in the MODPARAMS.DAT file, it should be preceded by a MIN_ or ADD_ prefix. Lack of this resource causes the following error when the record stream is connected to the file: %RMS-F-CRMP, CRMPSC system service failed to map global buffers See note (1) for a workaround to this error. The GBLPAGES used by RMS global buffers are always writable. Prior to VMS Version 5.0, the size of the primary page file had to include enough blocks to support the maximum number of writable GBLPAGES that could be paged out at any one time. As of VMS 5.0, this is no longer a requirement. They can be paged out across the primary or secondary page files. 5. VIRTUALPAGECNT Global buffers are mapped through a process's virtual address space. While the process quota PGFLQUOTA sets an upper limit on the maximum number of pages that a process can use in the system paging file, the SYSGEN parameter VIRTUALPAGECNT sets an upper limit on the maximum any process's PGFLQUOTA can be increased to. Global buffers are not allocated until runtime after an image has been executed. If a process's PGFLQUOTA is insufficient for the number of global (and local) buffers that an image attempts to connect to the record streams for one or more files, the following runtime error occurs: RMS-F-DME, Dynamic memory exhausted In other words, the failure occurs due to the exhaustion of a process quota. However, since its upperbound is restricted by a SYSGEN parameter, it is included here. In many cases when this error occurs, PGFLQUOTA is set much lower than the SYSGEN parameter VIRTUALPAGECNT, so the process quota can be increased without having to reboot the system. NOTE: When PGFLQUOTA is modified, the user must log out and back in again for the new value to take effect. The number of PGFLQUOTA entries that will be needed for each global cache (file) is as follows: PGFLQUOTA per global cache (file) = value calculated for GBLPAGES Thus: VIRTUALPAGECNT per global cache (file) = PGFLQUOTA per global cache (file) AUTOGEN calculates this parameter. 6. SYSMWCNT SYSMWCNT sets the quota for the size of the system working set (nonpageable). The global page table itself is part of the system working set. Each 128 entries (each 4 bytes in size) in the global page table use one page of the system working set. The "VMS System Generation Utility Manual", April 1988, page A-12 under the description of GBLPAGES, points out the need to adjust SYSMWCNT by one per 128 GBLPAGE entries as GBLPAGES is increased beyond its default. SYSMWCNT per global cache = [ GBLPAGES / 128 ] rounded up AUTOGEN calculates the value for SYSMWCNT based on GBLPAGES and PAGEDYN, provided it is either omitted from the MODPARAMS.DAT file or, if it is specified in the MODPARAMS.DAT file, it is preceded by a MIN_ or ADD_ prefix. No error occurs if this parameter is set too low. 7. LOCKIDTBL The number of entries in the system Lock ID table (see Note 2), which limits the number of locks in the system. Each global buffer has a system lock on it. LOCKIDTBL per global cache (file) = GBC + 1 AUTOGEN will automatically adjust this parameter, if necessary, in accordance with changes made to RMS_GBLBUFQUO, provided that this parameter is either omitted from the MODPARAMS.DAT file or any appearance is preceded by a MIN_ or ADD_ prefix. The following error occurs if this parameter is set too low: %SYSTEM-E-NOLOCKID, no lock id available. However, this error is not seen using RMS. If this problem is encountered by RMS it most likely results in a nonfatal bugcheck and the process is deleted. If a bugcheck does not occur, the error that RMS might return is very dependent on what it was doing when it encountered the problem. In the latter case, displaying the RAB$L_STV value (accessible from program control only) associated with the STS could be helpful. 8. RESHASHTBL The number of entries in the lock management resource name hash table is related to the LOCKIDTBL parameter. As a general guideline, there should be one resource hash table entry for every four locks in the system. It can be derived directly from the value of LOCKIDTBL. Specifically, it should be one-quarter the value of LOCKIDTBL, rounded up to the closest power of two. AUTOGEN will automatically adjust this parameter, if necessary, in accordance with changes made to LOCKIDTBL, provided that this parameter is either omitted from the MODPARAMS.DAT file or any appearance is preceded by a MIN_ or ADD_ prefix. No error is associated with this parameter being set too low. If it is inappropriately low, there may be a slowdown in the response rate for $ENQ's and $DEQ's (enqueuing and dequeuing locks). ------------------------------------------------------------------------ NOTES: (1) Applications which use global buffers on RMS files should include a workaround for the RMS-F-CRMP error so that the programs do not abort. The workaround is to include a test for a return from a $CONNECT of the status code RMS$_CRMP and, when this is returned, take the following actions: 1. Print a message to the user that the RMS-F-CRMP error occurred and the system manager should be informed that one or more of the SYSGEN parameters GBLSECTIONS, GBLPAGES or GBLPAGFIL was set too low on this particular system to map a global section for RMS global buffers. The filename should be displayed as part of the error message. 2. Assign a zero to the FAB$W_GBC field. This will override the number of global buffers requested (either in the file header or by a nonzero assignment to FAB$W_GBC), and RMS will not attempt to set up a global cache for this file on a connect. 3. Re-call $CONNECT. NOTE: The above suggestions assume that an application is using RMS directly or is using a high level language that provides a user-action routine option as part of the file open. (2) The following information discussing the SYSGEN parameter LOCKIDTBL_MAX is version specific. OpenVMS VAX versions 5.0 through 5.4-3 -------------------------------------- LOCKIDTBL_MAX establishes the maximum size of the system Lock ID table. As a general rule, it should be twice the value of LOCKIDTBL. AUTOGEN does not calculate this value. If it is omitted from the MODPARAMS.DAT file, it will be set to the default. On booting a system, a lock database with LOCKIDTBL entries is built. If this becomes full, it can be expanded to at least LOCKIDTBL_MAX entries (provided that LOCKIDTBL_MAX is greater than LOCKIDTBL) while the system is running. The expansion is done in increments of LOCKIDTBL. For Example: LOCKIDTBL is 65000 and LOCKIDTBL_MAX is 65001. The lock ID table becomes full. The system finds that LOCKIDTBL_MAX is greater than LOCKIDTBL. Since it is, it will expand the lock ID table by LOCKIDTBL. The lock ID table now has 130,000 entries. However, if it becomes full again, LOCKIDTBL_MAX (65001) will not be greater than LOCKIDTBL (130,000). In this example, the lock id table will only be expanded once, but note that its size will exceed LOCKIDTBL_MAX. The expanded lock id table is built in a block of memory allocated from nonpaged pool. You need to have enough contiguous pages in nonpaged pool to accommodate the entire size of the NEW expanded table because the old table isn't deleted until the new one is successfully created. OpenVMS VAX versions 5.5 through 6.2 ------------------------------------ Starting in OpenVMS VAX versions 5.5, the LOCKIDTBL_MAX parameter is used at SYSBOOT time in the calculation which determines possible "System Page Table Entry" (SPTE) requirements. The algorithm governing the extension of the Lock ID table was also modified. The algorithm now only checks the availability of SPTE's and fluid pages before a 1 or 2 page extension is added to the Lock ID table. REFERENCES: Additional information about this topic can be found in other database articles with the following titles: Process Limits and Quotas Related to RMS Buffers Command Procedure To Estimate SYSGEN Units for RMS Global Buffers Estimating SYSGEN Units Currently in Use for RMS Global Buffers [OpenVMS] OPEN with Global Buffers Fails with RMS-F-CRMP, RMS-F-DME For additional information on the AUTOGEN command procedure refer to the "Guide to Setting Up a VMS System", April 1988, (AA-LA25A-TE), Chapter 6. For additional information on VMS memory management and lock management refer to the "Introduction to VMS System Services", April 1988, (AA-LA68A-TE), Chapters 11 & 12. For additional information on file performance tuning and file sharing refer to the "Guide to VMS File Applications", (April 1988), (AA-LA78A-TE), Chapters 3 & 7. Process Limits and Quotas Related to RMS Buffers Any party granted access to the following copyrighted information (protected under Federal Copyright Laws), pursuant to a duly executed Compaq Service Agreement may, under the terms of such agreement copy all or selected portions of this information for internal use and distribution only. No other copying or distribution for any other purpose is authorized. Copyright (c) Digital Equipment Corporation 1990, 1991. All rights reserved COMPONENT: RMS OP/SYS: VMS, All Versions SOURCE: Digital Customer Support Center OVERVIEW: This article describes the following process limits and quotas directly related to RMS local and global buffers: 1. Working set size (WSQUOTA, WSEXTENT) 2. PGFLQUOTA 3. PIOPAGES 4. ENQLM DISCUSSION: 1. WORKING SET SIZE (WSQUOTA, WSEXTENT) o Using LOCAL buffers The number of pages of working set needed to support local buffers for each file is as follows: Number of pages used per local cache = number of buffers * maximum_bucket_size NOTE: The 'virtual' bucket size for a sequential file equals the multiblock count (MBC). If a sequential file is opened with shared write access, it is set to the MBC of the first accessor of the file. The MBC system default is set by the SYSGEN parameter RMS_DFMBC (current default is 16). Since these pages are private to a process, the process must have a working set size large enough to hold the number of pages required. Larger local caches without an increase in working set size can induce excessive paging. o Using GLOBAL buffers The pages used by global buffers are mapped through a process's PO space. The pages of a particular buffer referenced by the process will be part of the working set of that process. However, no one process mapped to a global cache has to have all the global buffers in their working set at one time. The responsibility is distributed across a number of processes. The more processes mapped to a particular global cache and actively doing I/O to that file, the more distributed the actual pages in the global cache are across the working sets of these processes. On the other hand, if the global cache is large and only a few processes are actively using the cache, the more likely paging will occur. Under these conditions, the state of the free page list is very important. The more free memory that exists on a system, the better the chances that buffer pages which are paged out of the working set will still be on the modified page list (or even the free page list) when requested again. In other words, a 'hard' page fault (one that requires a disk I/O) may not occur. 2. PGFLQUOTA o Both local and global buffers are mapped through a process's virtual address space. PGFLQUOTA sets an upper limit on the maximum number of pages that a process can use in the system paging file. o Buffers are allocated at image runtime. If several files are opened by an image and a large number of local and global buffers are connected to the record streams for these files, the following error may occur (at runtime): RMS-F-DME, Dynamic memory exhausted o The SYSGEN parameter VIRTUALPAGECNT sets an upper limit on the maximum any process's PGFLQUOTA can be raised to. In order to get a good estimate of what PGFLQUOTA should be increased to, first use the AUTHORIZE utility to set it to a high value for the particular user who incurred the error. The user should then log off, log back in, and run the image again. When the image completes, the user should logout using the qualifier '/FULL'. The display that appears includes a 'Peak page file size' entry that indicates how much PGFLQUOTA was needed by this job. This provides a lower-bound estimate of what PGFLQUOTA needs to be for this user. o If the DME error is incurred on the open of a process permanent file (one opened from DCL), the SYSGEN parameter PIOPAGES should be increased. 3. PIOPAGES o Files opened at DCL level are opened as process permanent files. Files opened from program control are opened as image files. o Process permanent files (notably SYS$INPUT, SYS$OUTPUT, and SYS$ERROR) must have their RMS internal structures mapped through P1 process space, in order to survive image deletion. PIOPAGES sets an upper limit on the number of P1 pages that can be used for this purpose. o When the RMS-F-DME error is incurred trying to open a process permanent file, PIOPAGES needs to be increased. o The algorithm for estimating the number of PIOPAGES needed for each process permanent file is: Number of PIOPAGES per process permanent file = Number of buffers * maximum_bucket_size NOTE: The 'virtual' bucket size for a sequential file equals the multiblock count (MBC). If a sequential file is opened with shared write access, it is set to the MBC of the first accessor of the file. The MBC system default is set by the SYSGEN parameter RMS_DFMBC (current default is 16). 4. ENQLM o Using LOCAL buffers If the file is not shared write, 2 units of ENQLM are required. One for a file lock and one for an extend lock, which is a sublock of the file lock. If the file is shared write, the number of ENQLM units can be estimated as follows: File and extend locks: 2 Local buffer locks - Number of local buffers minus the one used for extend: # of buffers - 1 Record locks for indexed files (no record locking is done for sequential or relative files): -- Automatic locking - Only one record lock block (RLB) is used at a time: 1 -- Manual locking - Requires as many RLB's as number of records requested before an explicit UNLOCK is done on the file or the file is closed: # of records - 1 NOTE: As of VMS Version 5.4, the default number of local buffers that are allocated when an indexed file is accessed may increase slightly. This will result in more ENQLM units being used. Any process that ran on a marginal ENQLM under pre-VMS 5.4 could experience an ENQLM problem after upgrading. For more information, see page 4-41 of the "VMS Version 5.4 Release Notes", August 1990, (AA-LB22C-TE). o Using GLOBAL buffers The number of ENQLM units can be estimated for global buffers as follows: File and extend lock: 2 Global section lock: 1 Local buffer locks - Number of local buffers minus the one used for extend: # of buffers - 1 Global buffer pointer block (GBPB) maintained in local cache. No matter how large the global cache is, there will only be 2 GBPB's (see note 1): 2 Record locks for indexed files (no record locking is done for sequential or relative files): -- Automatic locking - Only one record lock block (RLB) used at a time 1 -- Manual locking - Requires as many RLB's as records requested before an explicit UNLOCK is done on the file or the file is closed. # of records - 1 ----------------------------------------------------------------- Note (1): The two GBPB's are used for any process locks taken out on a global buffer accessed by a process. When the process lock is released, it is converted to a system NL lock in order to survive the closing of the file by the process, image exit or process deletion. System locks are maintained for each global buffer (once they are used) until the global section for that file is removed. No process limit is charged for system locks. REFERENCES: Additional information about this topic can be found in three other database articles with the following titles: Command Procedure To Estimate SYSGEN Units for RMS Global Buffers Estimating SYSGEN Units Currently in Use for RMS Global Buffers SYSGEN Parameters Related To RMS Global Buffers For additional information on file performance tuning and file sharing refer to "Guide to VMS File Applications", (April 1988), (AA-LA78A-TE), Ch. 3 & 7. For additional information on VMS memory management and lock management refer to the "Introduction to VMS System Services", April 1988, (AA-LA68A-TE), Chapters 11 & 12. [OpenVMS] Guidelines for Indexed File Optimization and Performance Any party granted access to the following copyrighted information (protected under Federal Copyright Laws), pursuant to a duly executed Compaq Service Agreement may, under the terms of such agreement copy all or selected portions of this information for internal use and distribution only. No other copying or distribution for any other purpose is authorized. Copyright (c) Compaq Computer Corporation 1995, 1998. All rights reserved. PRODUCT: DIGITAL OpenVMS VAX, All Versions DIGITAL OpenVMS Alpha, All Versions COMPONENT: Record Management Services (RMS) SOURCE: Compaq Computer Corporation OVERVIEW: RMS indexed files that are actively updated (i.e., with record writes, updates and deletions) need regular maintenance and can benefit greatly in terms of performance by regularly tuning the file's internal structure. This article discusses the general tuning procedure and makes tuning recommendations based on file design criteria. PROCEDURE: The tuning of an indexed file involves using the CONVERT Utility on the file to reload all the data and key indexes in key order, to eliminate bucket splits and reclaim deleted buckets. 1. The procedure to tune an indexed file's size and structure involves three steps: A. Create an FDL file that contains file attribute definitions and statistical information that describe the current state of the indexed file. For example, the following command analyzes ABC.DAT and creates file ABC.FDL;1: $ ANALYZE /RMS /FDL ABC.DAT B. Invoke the FDL Editor's Optimize script, (reads the FDL language and statistical information in the FDL file created by ANALYZE/RMS/FDL) and calculates the optimal number of data and index areas, bucket sizes, and index depth for this indexed file. The following command uses the /NOINTERACTIVE qualifier to direct the FDL Editor to execute the Optimize script without a terminal dialog: $ EDIT /FDL /ANALYSIS=ABC.FDL /NOINTERACTIVE ABC.FDL The default file type for this command is .FDL. On successful execution of this command, the FDL Editor creates ABC.FDL;2. This file contains definition language that, by default, describes a "flatter" indexed file (that is, one with fewer index levels). To create an output FDL file with a unique name, specify the /OUTPUT= qualifier on the command line: $ EDIT /FDL /ANALYSIS=ABC.FDL - /NOINTERACTIVE/OUTPUT=ABC_NEW ABC This command invokes the interactive Optimize script: $ EDIT /FDL /ANALYSIS=ABC.FDL /SCRIPT=OPTIMIZE ABC.FDL NOTE: This command may create an FDL file that defines a different bucket sizes than the original file has. If you've assigned global buffers to the indexed file, the change in bucket size may impact the number of global buffers desired for the file. C. Use the FDL file, created by the FDL Editor, to create an optimized version of the indexed file. For example: $ CONVERT/FDL=ABC_NEW/STATISTICS /NOSORT ABC.DAT ABC.NEW $ RENAME ABC.NEW ABC.DAT 2. Using the FDL Editor's Optimize script yields different results, depending on whether or not it runs interactively: A. Use of the /NOINTERACTIVE qualifier invokes the Optimize script. This produces an FDL file that may contain: o A different number of index and data areas. For example, if there is only 1 area in the original indexed file, the optimized FDL file defines two or more areas, unless /GRANULARITY=1 was specified on the EDIT/FDL command line o New bucket sizes for the index and/or data areas o A different number of blocks allocated to the file o A default extend size for each area that is recalculated to about 25% of the initial allocation for that area B. Using /SCRIPT=OPTIMIZE (without /NOINTERACTIVE) causes the FDL Editor to invoke the interactive Optimize script. By answering the questions in the terminal dialog, one can select different characteristics for each key. One can also change the default emphasis of the file (i.e., to smaller buffers or flatten the file's index structure), and select a new bucket size for the index and data buckets of that index structure. 3. The frequency of file optimization depends on the file's use: A. If the file is actively updated (with records being added, updated and deleted) but the number of records is relatively constant, it should be necessary to tune the file only once. Invoke the FDL Editor's Optimize script (/SCRIPT=OPTIMIZE) and indicate the maximum number of records the file should contain at any one time, by answering these script questions: Number of Records that will be Reloaded into the File (0-2Giga)[n] : Number of Additional Records to be Added After the Reloading the File (0-2147482672)[0] : Because activity on the file can cause internal fragmentation, use CONVERT to periodically rebuild the indexed file: $ CONVERT /STATISTICS /NOSORT ABC.DAT ABC.NEW $ RENAME ABC.NEW ABC.DAT B. If the number of records in the file increases over time, it should be tuned periodically, in order to plan for the file's expected growth during the period of time until the file is next tuned. To do this, execute the FDL Editor's Optimize script and indicate the number of records to be added during that time period: Number of Additional Records to be Added After the Reloading the File (0-2147482672)[0] : For example, if you plan to tune the file every month, indicate the number of additional records you will be adding in that month. Between file tuning, regular use of CONVERT to rebuild the file keeps internal fragmentation from bucket splitting to a minimum. 4. Suggestions for file definition attributes, to increase performance: A. Disable index compression. When index compression is enabled, RMS must sequentially search through the index when performing a lookup of a key value or writing a new record to the file. When it is disabled, RMS performs a binary search. B. Disable data key compression. This does not dramatically improve RMS performance, but reduces the amount of CPU that is needed to compress and decompress the data key values when writing, updating or reading a record. C. If available disk space is not an issue and data record compression is not efficiently reducing the space needed to store a record, disable data record compression. Review the FDL file created by ANALYZE/RMS/FDL, to find the compression efficiency percentage. This information is found in the 'ANALYSIS OF KEY 0' section, in the value for the DATA_RECORD_COMPRESSION field. The larger the number, the more effective the data record compression. D. Avoid duplicate values for key values or make key values as unique as possible. When RMS searches for a particular key value in an alternate key and duplicate records are allowed for the index of reference, RMS may have to search through a chain of records with the same key value to find the record in question. Because RMS sequentially traverses this duplicate chain, a search for a specific record in a long duplicate chain represents a major performance problem for read and delete operations. This problem is compounded for update operations that change the key value, since RMS must traverse the duplicate chain to find the records that has the old key value, delete that alternate key's record (known as a SIDR), traverse the duplicate chain that has the new key value and add a SIDR for the record to the end of that chain. If it is not possible to disallow duplicate key values, it may be possible to make the key more unique. For example, a string key can be made more unique by adding to it a key segment for a field that has some unique value, or by adding to the key field a more unique value. Then, if the key value is not unique, a call to $GET might specify only the first part of the key; in that case, sequential $GET calls can be used to find the record with the the particular key that is being sought. E. Use a fill factor of less than 100%, if records will be randomly added to the file. If records are added with an increasing key value (i.e., key is a log number that is incremented for each new record), select a load fill of 100%. When selecting a load fill factor that is less than 100%, set it to an amount that allows room in the bucket for at least one additional record. For example, if four records can fit into a data bucket, setting the fill factor to 75% causes the CONVERT utility to place three records per bucket, leaving room for one record in each data bucket. F. Allocate additional space in the indexed file, for records that will be added after CONVERT initially loads records to the file. As described earlier in this article, execute the FDL Editor's Optimize script and indicate the number of additional records that will be added after CONVERT reloads the file: Number of Additional Records to be Added After the Reloading the File (0-2147482672)[0] : 5. To reduce index fragmentation when writing records to a new file: When loading records to a new file from program control, write the records to a sequential file first. Then use CONVERT/FDL to load the indexed file from the sequential file. Since CONVERT builds one index at a time, this method loads the file's index structures without causing the internal fragmentation that can result if the program loads the records to an indexed file. RELATED ARTICLES: Other articles in the OPENVMS database provide additional information on RMS indexed files. These articles can be found using a search string of: RMS INDEXED FILES ANALYZE/RMS FDL REFERENCES: "Guide to OpenVMS File Applications", (AA-PV6PA-TK), May 1993, Chapter 3, Chapter 10, Appendix A "OpenVMS Record Management Services Reference Manual", (AA-PV6RA-TK), May 1993 "OpenVMS Record Management Utilities Reference Manual", (AA-PV6QA-TK), May 1993, Chapter 1, Chapter 2, Chapter 6 [OpenVMS] Estimating SYSGEN Units Currently in Use for RMS Global Buffers Any party granted access to the following copyrighted information (protected under Federal Copyright Laws), pursuant to a duly executed Compaq Service Agreement may, under the terms of such agreement copy all or selected portions of this information for internal use and distribution only. No other copying or distribution for any other purpose is authorized. Copyright (c) 1990, 2000. Compaq Computer Corporation. All rights reserved. PRODUCT: Compaq OpenVMS VAX, All Versions COMPONENT: RMS SOURCE: Compaq Computer Corporation OVERVIEW: This article describes how to estimate units currently used by RMS global buffers for the following SYSGEN parameters: 1. GBLSECTIONS 2. GBLPAGES 3. GBLPAGFIL 4. RMS_GBLBUFQUO* (see note below) *Note: RMS_GBLBUFQUO is obsolete as of OpenVMS V7.1. Caution: In deriving any estimates of how many of the units set up by these SYSGEN parameters are currently in use, it must be noted that they are based on a "snapshot" of the system at a particular moment. Any RMS file with global buffers defined (either by a SET FILE/GLOBAL DCL command or via setting FAB$W_GBC at runtime) that is not open at this moment will not be included in the estimation. Any program containing a call to the system service $CRMPSC that is not running at this moment will not be included in the estimation. In other words, any estimate that is derived should be considered an "UNDERESTIMATE." DISCUSSION: 1. GBLSECTIONS The DCL lexical function F$GETSYI can be used to determine how many units of GBLSECTIONS are NOT in use: $ WRITE SYS$OUTPUT F$GETSYI("FREE_GBLSECTS") 9 2. GBLPAGES The DCL lexical function F$GETSYI can be used to determine how many contiguous, free GBLPAGES are available: $ WRITE SYS$OUTPUT F$GETSYI("CONTIG_GBLPAGES") 396 The number of unused pages reported by the INSTALL utility includes the TOTAL number of free pages and can be misleading. $ INSTALL LIST/GLOBAL/SUMMARY Summary of Local Memory Global Sections 261 Global Sections Used, 14998/402 Global Pages Used/Unused 3. GBLPAGFIL There are two alternative ways to estimate GBLPAGFIL units currently in use. ALTERNATIVE #1: This is the simplest way to estimate GBLPAGFIL units currently in use. However, it requires SYSPRV and CMKRNL privileges. In addition, it is based on a symbol that is available through the current version of VMS, but there is no guarantee that this symbol will be available in the future. $ SET PROCESS/PRIVILEGE=(SYSPRV,CMKRNL) $ ANALYZE/SYSTEM VAX/VMS System analyzer SDA> EXAMINE SGN$GL_GBLPAGFIL ! SYSGEN GBLPAGFIL SGN$GL_GBLPAGFIL: 00000400 "...." ! Hex 400 = decimal 1024 SDA> EXAMINE MMG$GL_GBLPAGFIL ! GBLPAGFIL remaining MMG$GL_GBLPAGFIL: 000003B6 "¶..." ! Hex 3B6 = decimal 950 ! MMG$GL_GBLPAGFIL starts out as a positive integer equal ! to SGN$GL_GBLPAGFIL. MMG$GL_GBLPAGFIL is decremented for ! each page created for a WRT/DZRO global section. In other ! words, the current value in MMG$GL_GBLPAGFIL indicates the ! number of unused GBLPAGFIL units. When MMG$GL_GBLPAGFIL ! drops to zero, then no more GBLPAGFIL quota exists. SDA> EXIT ! Symbols used to find amount of GBLPAGFIL currently used $ GBLPAGFIL_SYSGEN = 1024 $ GBLPAGFIL_AVAILABLE = %X000003B6 $ SHOW SYMBOL GBLPAGFIL_AVAILABLE GBLPAGFIL_AVAILABLE = 950 Hex = 000003B6 Octal = 00000001666 ! GBLPAGFIL_USED = GBLPAGFIL_SYSGEN - GBLPAGFIL_AVAILABLE ! = 1024 - 950 ! GBLPAGFIL_USED = 74 ALTERNATIVE #2: Any global sections that have GBLPAGFIL units deducted must have the PSECT attributes WRT (writable) and DZRO (PAGEFILE.SYS used as backing storage). This method uses the INSTALL Utility's LIST/GLOBAL command to identify any global sections that have both of these attributes set. Step 1: Obtain listing of all global sections on system: $ DEFINE/USER SYS$OUTPUT glob_install.lis $ INSTALL LIST/GLOBAL $ DIRECTORY/SIZE glob_install.lis GLOB_INSTALL.LIS;1 38 Step 2: Use DCL SEARCH command to select only those global sections that have the PSECT ATTRIBUTES writable (WRT) and DZRO (PAGEFILE.SYS used as backing storage). $ SEARCH/EXACT/MATCH=AND/OUTPUT=glob_wrt.lis - _$ glob_install.lis WRT,DZRO $ TYPE glob_wrt.lis OSIF$MANAGEMENT(0A000001) WRT DZRO PRM SYS Pagcnt/Refcnt=5/5 RMS$804BDCA0 (00000000) WRT DZRO TMP SYS Pagcnt/Refcnt=69/69 Refcnt = pagcnt * number of users Step 3: Sum 'pagcnt' entries with both WRT and DZRO attributes. The total is equal to the number of GBLPAGFIL units currently in use. ----- Number of GBLPAGFIL units in use 74 The global sections for RMS global buffers always begin with an RMS$ prefix. In the output above, one RMS file active on the system currently has global buffers set (RMS$804BDCA0). It indicates that 69 GBLPAGES are being used by this file. This does not tell us how many global buffers (RMS_GBLBUFQUO units) are being used. The estimation procedure for RMS_GBLBUFQUO below is in "buffer" units. NOTE: There is one case in which a data file can be used as the backing storage for a global section. This occurs when a call is made to the system service SYS$CRMPSC specifying a channel to the data file and the DZRO attribute (as well as GBL and WRT). Since the system PAGEFILE.SYS is not used as the backing storage for this section, GBLPAGFIL units are not deducted. However, this global section DOES have both the WRT and DZRO attributes in the INSTALL listing so there is no way to identify that this section is not using GBLPAGFIL units. Thus, the estimate of GBLPAGFIL units currently in use derived from this alternative could provide an "overestimate". In contrast, the first alternative always provides an exact number at the particular time it is checked using SDA. 4. RMS_GBLBUFQUO (Note: RMS_GBLBUFQUO is obsolete as of OpenVMS V7.1.) SDA provides the only means of finding out how many active RMS global buffers are currently in use on a particular system. A logfile with the steps necessary follows: Step 1: Put EXEC writable page offsets into an object file. $ MACRO/OBJECT=ew_data SYS$INPUT:+SYS$LIBRARY:LIB/LIB $EWDATADEF GLOBAL .END $ DIRECTORY/SIZE ew_data.obj EW_DATA.OBJ;1 3 Step 2: Invoke SDA $ SET PROCESS/PRIVILEGE=(SYSPRV,CMKRNL) $ ANALYZE/SYSTEM VAX/VMS System analyzer Step 3: Read in EXEC writable page offsets SDA> READ ew_data.obj %SDA-I-READSYM, reading symbol table DISK:[USER]EW_DATA.OBJ;1 Step 4: Examine GBLBUFQUO current value SDA> EXAMINE @exe$AR_EWDATA+ew_RMS$GW_GBLBUFQUO SYS$NETWORK_SERVICES+00778: 0000FFFD "ý..." ! decimal -3 ! RMS$GW_GBLBUFQUO is a negative counter. It is initialized ! to zero when the system boots and is decremented for each ! GBLBUFQUO unit used. The -3 (FFFD) is accounted for by the ! one RMS global section located in the INSTALL listing: ! ! RMS$804BDCA0 WRT DZRO TMP SYS Pagcnt/Refcnt=69/69 ! ! Three global buffers are being used by this file. ! ! A break was done here to run an image that opens an RMS ! indexed file (INTER11.DAT) which had 10 global buffers set ! on it. A DIRECTORY/FULL of the file after the global ! buffers were set showed the following file attributes: ! ! $ DIRECTORY/FULL INTER11.DAT ! ... ! Allocation: 21, Extend: 1, Maximum bucket size: 1 ! Global buffer count: 10, No version limit ! ... ! ! Two records were added to the file and the image was ! paused just prior to the $CLOSE operation. ! Return to the SDA session. SDA> ex @exe$AR_EWDATA+ew_RMS$GW_GBLBUFQUO SYS$NETWORK_SERVICES+00778: 0000FFFA "ú..." ! decimal -6 ! Three additional RMS_GBLBUFQUO units are in use. But the ! file had 10 global buffers set on it! Why the discrepancy? ! Because only 3 out of 10 global buffers are currently in ! use for the INTER11.DAT file! In other words, units of ! GBLBUFQUO are deducted as they are used! SDA> SET PROCESS/INDEX=21600572 SDA> SET RMS=GBDSUM SDA> SHOW PROCESS/RMS Process index: 0072 Name: _LTA31: Extended PID: 21600572 ------------------------------------------------------------ GBD Summary ----------- GBD USE REL_ CACHE_ Address CNT SIZE NUMB VBN VBNSEQNUM ADDR VAL LOCK_ID ------- ----- ---- ---- --- ------- ---- --- ------- 00033488 0 512 512 8 00000000 00000600 1 018C0309 00033458 0 512 512 9 00000000 00000400 2 00140321 000334B8 0 512 512 14 00000000 00000800 1 00200320 00033608 0 512 0 -1 00000000 00001600 0 00000000 000335D8 0 512 0 -1 00000000 00001400 0 00000000 000335A8 0 512 0 -1 00000000 00001200 0 00000000 00033578 0 512 0 -1 00000000 00001000 0 00000000 00033548 0 512 0 -1 00000000 00000E00 0 00000000 00033518 0 512 0 -1 00000000 00000C00 0 00000000 000334E8 0 512 0 -1 00000000 00000A00 0 00000000 10 GBDs were processed ! NOTE: File has not been closed in image yet. ! Exit from SDA and obtain information from INSTALL while ! global section is still active for this file. SDA> EXIT $ DEFINE/USER SYS$OUTPUT glob_install.lis $ INSTALL LIST/GLOBAL $ SEARCH/EXACT/MATCH=AND/OUTPUT=glob_writ.lis - _$ glob_install.lis WRT,DZRO $ TYPE glob_writ.lis RMS$8053CBD0 (00000000) WRT DZRO TMP SYS Pagcnt/Refcnt=13/13 OSIF$MANAGEMENT(0A000001) WRT DZRO PRM SYS Pagcnt/Refcnt=5/5 RMS$804BDCA0 (00000000) WRT DZRO TMP SYS Pagcnt/Refcnt=69/69 REFERENCES: Additional information about this topic can be found in other database articles with the following titles: Process Limits and Quotas Related to RMS Buffers Command Procedure To Estimate SYSGEN Units for RMS Global Buffers SYSGEN Parameters Related To RMS Global Buffers For additional information on file performance tuning and file sharing refer to "Guide to VMS File Applications", April 1988, (AA-LA78A-TE), Chapters 3 & 7. For additional information on VMS memory management and lock management refer to the "Introduction to VMS System Services", April 1988, (AA-LA68A-TE), Chapters 11 & 12. For additional information on using SDA refer to the "VMS System Dump Analyzer Utility Manual", April 1988, (AA-LA87A-TE). [OpenVMS] Command Procedure To Estimate SYSGEN Units for RMS Global Buffers Any party granted access to the following copyrighted information (protected under Federal Copyright Laws), pursuant to a duly executed Compaq Service Agreement may, under the terms of such agreement copy all or selected portions of this information for internal use and distribution only. No other copying or distribution for any other purpose is authorized. Copyright (c) 1990, 2000. Compaq Computer Corporation. All rights reserved. PRODUCT: Compaq OpenVMS VAX, All Version COMPONENT: RMS SOURCE: Compaq Computer Corporation OVERVIEW: This article contains a command procedure that can be used to estimate the units needed to support the addition of new RMS global buffers to a system. The following SYSGEN parameter additions are estimated: 1. RMS_GBLBUFQUO *(see note below) 2. GBLSECTIONS 3. GBLPAGES 4. GBLPAGFIL *Note: RMS_GBLBUFQUO is obsolete as of OpenVMS V7.1. The algorithms used in the command procedure to calculate the values are described in another database article. Only two items of information are needed for each indexed file on which global buffers have been set to estimate the number of SYSGEN units that will be consumed by that file. Use the DCL command DIRECTORY/FULL or the lexical function F$FILE_ATTRIBUTES to obtain the following information: 1. Global buffer count (GBC) 2. Maximum bucket size (BKS) PROCEDURE: $ ! RMSGLOB_SYSGEN.COM $ !+ $ ! $ ! Command procedure to estimate RMS global buffer SYSGEN $ ! units needed for one indexed file. This command procedure $ ! can be adapted to accumulate these units across a number $ ! of indexed files. $ ! $ ! CAUTION: This sample command procedure has been tested $ ! using VMS 5.4-3. However, we cannot guarantee its $ ! effectiveness because of the possibility of error in $ ! transmitting or implementing it. It is meant to be used $ ! as a template for writing your own command procedure, and $ ! it may require modification for use on your system. $ ! $ !- $ ON WARNING THEN GOTO DONE $ IF P1 .EQS. "" THEN $INQUIRE P1 "FILE SPEC " $ P1 = F$SEARCH (P1, 1) ! Returns full filespec $ gbc = F$FILE_ATTRIBUTES (P1,"GBC") ! GBC available as of $ IF gbc .EQ. 0 THEN GOTO NOGBC ! VMS V5.2 $ bks = F$FILE_ATTRIBUTES (P1,"BKS") $ RMS_GBLBUFQUO = gbc $ GBLSECTIONS = 1 $ k = 64 + (gbc * 48) ! GBH and GBD descriptor bytes $ GBLPAGFIL = ((gbc * bks * 512) + k + 1023)/512 $ GBLPAGES = GBLPAGFIL + 2 ! plus 2 stopper pages $ WRITE SYS$OUTPUT "" $ WRITE SYS$OUTPUT "SYSGEN values needed by ",P1 $ WRITE SYS$OUTPUT "" $ WRITE SYS$OUTPUT " RMS_GBLBUFQUO = ",RMS_GBLBUFQUO $ WRITE SYS$OUTPUT " GBLSECTIONS = ",GBLSECTIONS $ WRITE SYS$OUTPUT " GBLPAGES = ",GBLPAGES $ WRITE SYS$OUTPUT " GBLPAGFIL = ",GBLPAGFIL $ GOTO DONE $ NOGBC: $ WRITE SYS$OUTPUT "" $ WRITE SYS$OUTPUT "Following file did not have global buffers set: " $ WRITE SYS$OUTPUT " ",P1 $ DONE: $ EXIT EXAMPLE: $ DIRECTORY/FULL tuning.dat Directory DISK:[DIR]TUNING.DAT;1 TUNING.DAT;1 File ID: (22486,1,0) Size: 1554/1554 Owner: [USER] Created: 18-JAN-1990 21:00:48.01 Revised: 17-MAR-1990 15:51:05.08 (11) Expires: Backup: File organization: Indexed, Prolog: 3, Using 4 keys In 3 areas > File attributes: Allocation: 1554, Extend: 252, Maximum bucket size: 18 > Global buffer count: 50, No version limit Contiguous best try Record format: Fixed length 80 byte records Record attributes: Carriage return carriage control RMS attributes: None Journaling enabled: None File protection: System:RWED, Owner:RWED, Group:RE, World: Access Cntrl List: None $ @rmsglob_sysgen TUNING.DAT SYSGEN values needed by DISK:[DIR]TUNING.DAT;1 RMS_GBLBUFQUO = 50 GBLSECTIONS = 1 GBLPAGES = 908 GBLPAGFIL = 906 $ SET FILE/GLOBAL=0 TUNING.DAT $ @rmsglob_sysgen TUNING.DAT Following file did not have global buffers set: DISK:[DIR]TUNING.DAT;1 REFERENCES: Additional information about this topic can be found in other database articles with the following titles: Process Limits and Quotas Related to RMS Buffers Estimating SYSGEN Units Currently in Use for RMS Global Buffers SYSGEN Parameters Related To RMS Global Buffers For additional information on file performance tuning and file sharing, refer to the "Guide to VMS File Applications", April 1988, (AA-LA78A-TE), Chapters 3 & 7. For additional information on VMS memory management and lock management, refer to "Introduction to VMS System Services", April 1988, (AA-LA68A-TE), Chapters 11 & 12.