Merge branch 'hpc-gridware:master' into master

Author: ernst-bablick

doc/markdown/manual/release-notes/03_major_enhancements.md

@@ -1,14 +1,87 @@
 # Major Enhancements 
 
-## qconf support to add/modify/delete/show complex entries individually
+## v9.0.1beta
+
+### Utilization of additional data stores and activation of new thread pools
+
+Beginning with patch v9.0.1 the new internal architecture of `sge_qmaster` is activated so that the component can
+utilize additional data stores by starting new thread pools.
+
+* Listener thread pool: The listener thread pool was already activated in v9.0.0. It is used to handle incoming 
+  requests from clients and to distribute them to the corresponding processing components. Additionally, this pool
+  of threads utilizes a new data store to answer authentication requests. Beginning with v9.0.1 this data store
+  is used for even more requests to relieve other internal components within `sge_qmaster`.
+
+* Reader thread pool: The reader thread pool is activated and can now utilize a corresponding thread pool.
+  This will boost the performance of clusters in large environments where also users tend to request the status of the
+  system very often, by using client commands like `qstat`, `qhost` or other commands that send read-only requests 
+  to `sge_qmaster`. The additional data store needs to be enabled manually by setting following qmaster parameter in the 
+  *qmaster_params* of the cluster configuration:
+
+  ```
+  > qconf -mconf
+  ...
+  qmaster_params ...,DISABLE_SECONDARY_DS_READER=false
+  ...
+  ```
+  
+  Please note that requests answered by the reader thread pool might deliver slightly outdated data compared to the 
+  requests answered with data from the main data store because both data stores can be slightly out of sync. The 
+  maximum deviation can be configured by setting the `MAX_DS_DEVIATION` in milliseconds within in the `qmaster_params`.
+
+  ```
+  > qconf -mconf
+  ...
+  qmaster_params ...,MAX_DS_DEVIATION=1000 
+  ...
+  ```
+  
+  The default value is 1000 milliseconds. The value should be chosen carefully to balance the performance gain with
+  the accuracy of the data.
+ 
+  With one of the upcoming patches we will introduce an addition concept of automatic-sessions that will allow to
+  synchronize the data stores more efficiently and so that client commands can be enforced to get the most recent data.
+
+* Enhanced monitoring: The monitoring of `sge_qmaster` has been enhanced to provide more detailed information about
+  the utilization of the different thread pools. As also in the past the monitoring is enabled by setting the monitor 
+  time:
+
+  ```
+  > qconf -mconf
+  ...
+  qmaster_params ...,MONITOR_TIME=10
+  ...
+  ```
+  
+  `qping` will then show statistics about the handled requests per thread.
+
+  ``` 
+  qping -i 1 -f  <master_host> $SGE_QMASTER_PORT qmaster 1
+  ...
+  10/11/2024 12:54:53 | reader: runs: 261.04r/s (GDI (a:0.00,g:2871.45,m:0.00,d:0.00,c:0.00,t:0.00,p:0.00)/s 
+                                                 OTHER (ql:0)) 
+                                                 out: 261.04m/s APT: 0.0007s/m idle: 80.88% wait: 0.01% time: 9.99s
+  10/11/2024 12:54:53 | reader: runs: 279.50r/s (GDI (a:0.00,g:3074.50,m:0.00,d:0.00,c:0.00,t:0.00,p:0.00)/s 
+                                                 OTHER (ql:0)) 
+                                                 out: 279.50m/s APT: 0.0007s/m idle: 79.08% wait: 0.01% time: 10.00s
+  10/11/2024 12:54:53 | listener: runs: 268.65r/s (in (g:268.34 a:0.00 e:0.00 r:0.30)/s 
+                                                   GDI (g:0.00,t:0.00,p:0.00)/s) 
+                                                   out: 0.00m/s APT: 0.0001s/m idle: 98.42% wait: 0.00% time: 9.99s
+  10/11/2024 12:54:53 | listener: runs: 255.37r/s (in (g:255.37 a:0.00 e:0.00 r:0.00)/s GDI (g:0.00,t:0.00,p:0.00)/s) 
+                                                   out: 0.00m/s APT: 0.0001s/m idle: 98.54% wait: 0.00% time: 10.00s
+  ```
+
+## v9.0.0
+
+### qconf support to add/modify/delete/show complex entries individually
 
 `Qconf` also allows you to add, modify, delete and display complexes individually using the new `-ace`, `-Ace`, 
 `-mce`, `-Mce`, `-sce` and `-scel` switches. Previously this was only possible as a group command for the whole 
 complex set with `-mq`. More information can be found in the qconf(1) man page or by running `qconf -help`.
 
 (Available in Open Cluster Scheduler and Gridware Cluster Scheduler)
 
-## Added support to supplementary group IDs in user, operator and manager lists. 
+### Added support to supplementary group IDs in user, operator and manager lists. 
 
 Additionally, to user and primary group names, it is now possible to specify supplementary group IDs in user, operator,
 and manager lists. User lists can be specified in host, queue, configuration, and parallel environment objects to allow
@@ -30,7 +103,7 @@ in a production environment. Enabling caching services like `nscd` can help redu
 
 (Available in Gridware Cluster Scheduler only)
 
-## New internal architecture to support multiple Data Stores
+### New internal architecture to support multiple Data Stores
 
 The internal data architecture of `sge_qmaster` has been changed to support multiple data stores. This change 
 does not have a major impact currently and is not visible to the user. However, it is a prerequisite for future 
@@ -40,7 +113,7 @@ enhancing the performance of the cluster in large environments.
 
 (Available in Open Cluster Scheduler and Gridware Cluster Scheduler)
 
-## New RSMAP (Resource Map) complex type
+### New RSMAP (Resource Map) complex type
 
 Resource Maps are a new complex type that allows administrators to define a list of special resources which
 are available on a host, e.g. GPU devices, networking devices, lists of network ports, or other special resources.
@@ -111,7 +184,7 @@ qrsh -l port_numbers=8 env | grep SGE_HGR
 SGE_HGR_port_number=65000 65001 65002 65003 65004 65005 65006 65007
 ```
 
-## Per HOST complex variables
+### Per HOST complex variables
 
 The definition of complex variables contains the attribute `consumable` which could so far have the following values:
 
@@ -125,7 +198,7 @@ of parallel jobs multiple tasks are running on the same host, the requested amou
 once. E.g. multiple tasks of a parallel job can share the same GPU.
 
 
-## One-line JSON format for accounting and reporting files
+### One-line JSON format for accounting and reporting files
 
 The accounting and reporting files contain one line per record.
 The format of the records used to be a column-based format with a fixed number of columns,
@@ -202,7 +275,7 @@ as extensions to the accounting and reporting records (e.g. more exact timestamp
 additional usage values like maxrss) are only done in the new format.
 
 
-## Resource and queue requests per scope (global, master, slave) for parallel jobs
+### Resource and queue requests per scope (global, master, slave) for parallel jobs
 
 In former product versions resource requests for parallel jobs were only possible on the global level.
 Resource requests for parallel jobs were applied to all tasks of the job, both master task (the job script)

source/daemons/qmaster/sge_c_gdi.cc

@@ -269,23 +269,17 @@ sge_c_gdi_process_in_listener(sge_gdi_packet_class_t *packet, sge_gdi_task_class
    sge_pack_buffer *pb = &(packet->pb);
    switch (operation) {
       case SGE_GDI_TRIGGER:
-#if 0
-         MONITOR_GDI_TRIG(monitor);
-#endif
+         MONITOR_LIS_GDI_TRIG(monitor);
          sge_c_gdi_trigger_in_listener(packet, task, monitor);
          sge_gdi_packet_pack_task(packet, task, answer_list, pb);
          DRETURN(true);
       case SGE_GDI_PERMCHECK:
-#if 0
-         MONITOR_GDI_PERM(monitor);
-#endif
+         MONITOR_LIS_GDI_PERM(monitor);
          sge_c_gdi_permcheck(packet, task, monitor);
          sge_gdi_packet_pack_task(packet, task, answer_list, pb);
          DRETURN(true);
       case SGE_GDI_GET:
-#if 0
-         MONITOR_GDI_GET(monitor);
-#endif
+         MONITOR_LIS_GDI_GET(monitor);
          sge_c_gdi_get_in_listener(ao, packet, task, monitor);
          sge_gdi_packet_pack_task(packet, task, answer_list, pb);
          DRETURN(true);

source/daemons/qmaster/sge_thread_reader.cc

@@ -27,9 +27,12 @@
 #include "uti/sge_os.h"
 #include "uti/sge_profiling.h"
 #include "uti/sge_rmon_macros.h"
+#include "uti/sge_time.h"
 
 #include "sgeobj/ocs_DataStore.h"
 
+#include "sge_thread_ctrl.h"
+
 #ifdef OBSERVE
 #  include "cull/cull_observe.h"
 #endif
@@ -143,7 +146,7 @@ sge_reader_main(void *arg) {
 
    // init monitoring
    cl_thread_func_startup(thread_config);
-   sge_monitor_init(p_monitor, thread_config->thread_name, GDI_EXT, MT_WARNING, MT_ERROR);
+   sge_monitor_init(p_monitor, thread_config->thread_name, GDI_EXT, RT_WARNING, RT_ERROR);
    sge_qmaster_thread_init(QMASTER, READER_THREAD, true);
 
    /* register at profiling module */
@@ -164,7 +167,8 @@ sge_reader_main(void *arg) {
 
       MONITOR_SET_QLEN(p_monitor, sge_tq_get_task_count(ReaderRequestQueue));
 
-      if (packet != nullptr) {
+      // handle the packet only if it is not nullptr and the shutdown has not started
+      if (packet != nullptr && !sge_thread_has_shutdown_started()) {
          sge_gdi_task_class_t *task;
          bool is_only_read_request = true;
 
@@ -216,7 +220,7 @@ sge_reader_main(void *arg) {
 
          // handle the request (GDI/Report/Ack ...
          if (packet->request_type == PACKET_GDI_REQUEST) {
-            // sge_usleep(3000000);
+            //sge_usleep(1000000);
 
             task = packet->first_task;
             while (task != nullptr) {
@@ -229,6 +233,7 @@ sge_reader_main(void *arg) {
             sge_c_report(packet, task, packet->host, packet->commproc, packet->commproc_id, task->data_list, p_monitor);
          } else if (packet->request_type == PACKET_ACK_REQUEST) {
             task = packet->first_task;
+            // @TODO: This could be done by listener already?
             sge_c_ack(packet, task, p_monitor);
          } else {
             DPRINTF("unknown request type %d\n", packet->request_type);
@@ -298,14 +303,25 @@ sge_reader_main(void *arg) {
          thread_output_profiling("reader thread profiling summary:\n", &next_prof_output);
 
          sge_monitor_output(p_monitor);
-      } else {
-         int execute = 0;
+      }
 
+      // pass the cancellation point at least once or stay here if shutdown was triggered
+      bool shutdown_started = false;
+      do {
          // pthread cancellation point
+         int execute = 0;
          pthread_cleanup_push(sge_reader_cleanup_monitor, static_cast<void *>(p_monitor));
          cl_thread_func_testcancel(thread_config);
          pthread_cleanup_pop(execute); // cleanup monitor
-      }
+
+         // shutdown in process?
+         shutdown_started = sge_thread_has_shutdown_started();
+
+         // if we will wait here than do not eat up all cpu time
+         if (shutdown_started) {
+            sge_usleep(25000);
+         }
+      } while (shutdown_started);
    }
 
    // Don't add cleanup code here. It will never be executed. Instead, register a cleanup function with

source/daemons/qmaster/sge_thread_worker.cc

@@ -41,9 +41,12 @@
 #include "uti/sge_os.h"
 #include "uti/sge_profiling.h"
 #include "uti/sge_rmon_macros.h"
+#include "uti/sge_time.h"
 
 #include "sgeobj/ocs_DataStore.h"
 
+#include "sge_thread_ctrl.h"
+
 #ifdef OBSERVE
 #  include "cull/cull_observe.h"
 #endif
@@ -193,7 +196,7 @@ sge_worker_main(void *arg) {
 
    // init monitoring
    cl_thread_func_startup(thread_config);
-   sge_monitor_init(p_monitor, thread_config->thread_name, GDI_EXT, MT_WARNING, MT_ERROR);
+   sge_monitor_init(p_monitor, thread_config->thread_name, GDI_EXT, WT_WARNING, WT_ERROR);
    sge_qmaster_thread_init(QMASTER, WORKER_THREAD, true);
 
    /* register at profiling module */
@@ -214,6 +217,7 @@ sge_worker_main(void *arg) {
 
       MONITOR_SET_QLEN(p_monitor, sge_tq_get_task_count(GlobalRequestQueue));
 
+      // handle the packet only if it is not nullptr and the shutdown has not started
       if (packet != nullptr) {
          sge_gdi_task_class_t *task;
          bool is_only_read_request = true;
@@ -346,14 +350,13 @@ sge_worker_main(void *arg) {
          thread_output_profiling("worker thread profiling summary:\n", &next_prof_output);
 
          sge_monitor_output(p_monitor);
-      } else {
-         int execute = 0;
-
-         // pthread cancellation point
-         pthread_cleanup_push(sge_worker_cleanup_monitor, static_cast<void *>(p_monitor));
-         cl_thread_func_testcancel(thread_config);
-         pthread_cleanup_pop(execute); // cleanup monitor
       }
+
+      // pthread cancellation point
+      int execute = 0;
+      pthread_cleanup_push(sge_worker_cleanup_monitor, static_cast<void *>(p_monitor));
+      cl_thread_func_testcancel(thread_config);
+      pthread_cleanup_pop(execute); // cleanup monitor
    }
 
    // Don't add cleanup code here. It will never be executed. Instead, register a cleanup function with

source/dist/inst_sge

@@ -40,7 +40,7 @@
 # set -x
 
 SCRIPT_VERSION="9"
-SGE_VERSION="9.0.0"
+SGE_VERSION="9.0.1beta"
 
 #Reset PATH to a safe value
 #

source/libs/gdi/version.cc

@@ -36,7 +36,7 @@
 
 #include "gdi/version.h"
 
-const char GDI_VERSION[] = "9.0.0";
+const char GDI_VERSION[] = "9.0.1beta";
 
 // TODO: Add also an entry to the table further down below when you change this
 //       And change SGE_VERSION in dist/inst_sge

source/libs/uti/msg_utilib.h

@@ -154,7 +154,7 @@
 #define MSG_UTI_MONITOR_MEMERROREXT             _MESSAGE(59133, _("not enough memory for monitor extension"))
 #define MSG_UTI_MONITOR_TETEXT_FF               _MESSAGE(59134, _("pending: %.2f executed: %.2f/s"))
 #define MSG_UTI_MONITOR_EDTEXT_FFFFFFFF         _MESSAGE(59135, _("clients: %.2f mod: %.2f/s ack: %.2f/s blocked: %.2f busy: %.2f | events: %.2f/s added: %.2f/s skipt: %.2f/s"))
-#define MSG_UTI_MONITOR_LISEXT_FFFF             _MESSAGE(59136, _("in (g:%.2f a:%.2f e:%.2f r:%.2f)/s"))
+#define MSG_UTI_MONITOR_LISEXT_FFFFFFF          _MESSAGE(59136, _("in (g:%.2f a:%.2f e:%.2f r:%.2f)/s GDI (g:%.2f,t:%.2f,p:%.2f)/s"))
 #define MSG_UTI_MONITOR_SCHEXT_UUUUUUUUUU       _MESSAGE(59137, _("malloc:                   arena(" sge_U32CFormat ") |ordblks(" sge_U32CFormat ") | smblks(" sge_U32CFormat ") | hblksr(" sge_U32CFormat ") | hblhkd(" sge_U32CFormat ") usmblks(" sge_U32CFormat ") | fsmblks(" sge_U32CFormat ") | uordblks(" sge_U32CFormat ") | fordblks(" sge_U32CFormat ") | keepcost(" sge_U32CFormat ")"))
 #define MSG_UTI_DAEMONIZE_CANT_PIPE             _MESSAGE(59140, _("can't create pipe"))
 #define MSG_UTI_DAEMONIZE_CANT_FCNTL_PIPE       _MESSAGE(59141, _("can't set daemonize pipe to not blocking mode"))