bogus usage jobs may be created when usage is behind in processing

[DaanHoogland]

problem

During trouble shooting missing data, when usage is behind it was found that a lot of usage_jobs are created that won’t have a period to process:

2026-01-13 01:00:38,566 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) it's been 86438558 ms since last usage job and 66261443 ms until next job, scheduling an immediate job to catch up (aggregation duration is 1440 minutes)
2026-01-13 01:00:38,568 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) Scheduling Usage job...
2026-01-13 01:00:38,570 INFO  [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) starting usage job...
2026-01-13 01:00:38,606 INFO  [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) Parsing usage records between Mon Jan 12 01:00:00 CET 2026 and Tue Jan 13 00:59:59 CET 2026
2026-01-13 01:01:38,581 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) it's been 86498581 ms since last usage job and 66201420 ms until next job, scheduling an immediate job to catch up (aggregation duration is 1440 minutes)
2026-01-13 01:01:38,581 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) Scheduling Usage job...
2026-01-13 01:02:38,559 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) it's been 86558559 ms since last usage job and 66141442 ms until next job, scheduling an immediate job to catch up (aggregation duration is 1440 minutes)
2026-01-13 01:02:38,559 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) Scheduling Usage job...
2026-01-13 01:03:38,558 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) it's been 86618558 ms since last usage job and 66081443 ms until next job, scheduling an immediate job to catch up (aggregation duration is 1440 minutes)
2026-01-13 01:03:38,558 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) Scheduling Usage job...
2026-01-13 01:04:38,559 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) it's been 86678559 ms since last usage job and 66021442 ms until next job, scheduling an immediate job to catch up (aggregation duration is 1440 minutes)
2026-01-13 01:04:38,559 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) Scheduling Usage job...
2026-01-13 01:05:38,558 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) it's been 86738559 ms since last usage job and 65961442 ms until next job, scheduling an immediate job to catch up (aggregation duration is 1440 minutes)
2026-01-13 01:05:38,558 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) Scheduling Usage job...
2026-01-13 01:06:02,952 DEBUG [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) Creating networking offering: 18 for Vm: 75432 for account: 239496
2026-01-13 01:06:03,028 DEBUG [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) create volume with id : 163459 for account: 239496
2026-01-13 01:06:03,497 DEBUG [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) deleting network offering: 18 from Vm: 96981

the HeartBeat thread should be clever about existing jobs and not create the extra jobs.

versions

ACS 4.19

The steps to reproduce the bug

1. have an environment running for a while
2. prune intermediate and usage tables a couple of months
3. set the events of those months to unprocessed
4. start usage
5. observe the a job starts and multiple other jobs are created.
6. the extra jobs seem to have an interval of -1 second and fail (of course)

What to do about it?

make sure the heartbeat job does not create jobs while jobs are being processed/scheduled already.

[abh1sar]

I was able to reproduce the issue (With some code orchestration to speed up the process)

Steps:

1. Set usage.stats.job.aggregation.range to 5 minutes

2. Put a sleep in UsageManagerImpl.parse(UsageJobVO job, long startDateMillis, long endDateMillis) for 5 minutes (we need a long running usage parsing job)
3. stop the cloudstack-usage service.
4. start it after ~12 minutes
5. The heartbeat code sees that it has been long since last job was scheduled, so it schedules it.
6. Usage job scheduled sleeps for 5 minutes
7. Heartbeat code runs every minute and logs the Usage job starting every one minute (It doesn't actually start new jobs, just updates the last running job with the new heartbeat)

2026-01-29 11:31:58,846 INFO  [cloud.usage.UsageServer] (main:[]) (logid:) UsageServer ready...                                                                                                                                                                                                                                     
2026-01-29 11:32:13,827 DEBUG [usage.UsageManagerImpl.Heartbeat] (Usage-HB-1:[]) (logid:) it's been 733825 ms since last usage job and 63166176 ms until next job, scheduling an immediate job to catch up (aggregation duration is 5 minutes)                                                                                      
2026-01-29 11:32:13,828 DEBUG [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_7373c433] (Usage-HB-1:[]) (logid:) Scheduling Usage job...                                                                                                                                                                                         
2026-01-29 11:32:13,829 INFO  [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_7373c433] (Usage-Job-1:[]) (logid:) starting usage job...                                                                                                                                                                                          
2026-01-29 11:32:13,881 INFO  [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_7373c433] (Usage-Job-1:[]) (logid:) Parsing usage records between [2026-01-29T11:20:00+0000] and [2026-01-29T11:31:59+0000].                                                                                                                       
2026-01-29 11:33:13,808 DEBUG [usage.UsageManagerImpl.Heartbeat] (Usage-HB-1:[]) (logid:) it's been 793808 ms since last usage job and 63106193 ms until next job, scheduling an immediate job to catch up (aggregation duration is 5 minutes)                                                                                      
2026-01-29 11:33:13,808 DEBUG [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_7373c433] (Usage-HB-1:[]) (logid:) Scheduling Usage job...                                                                                                                                                                                         
2026-01-29 11:34:13,806 DEBUG [usage.UsageManagerImpl.Heartbeat] (Usage-HB-1:[]) (logid:) it's been 853806 ms since last usage job and 63046195 ms until next job, scheduling an immediate job to catch up (aggregation duration is 5 minutes)                                                                                      
2026-01-29 11:34:13,806 DEBUG [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_7373c433] (Usage-HB-1:[]) (logid:) Scheduling Usage job...                                                                                                                                                                                         
2026-01-29 11:35:13,816 DEBUG [usage.UsageManagerImpl.Heartbeat] (Usage-HB-1:[]) (logid:) it's been 913816 ms since last usage job and 62986185 ms until next job, scheduling an immediate job to catch up (aggregation duration is 5 minutes)                                                                                      
2026-01-29 11:35:13,816 DEBUG [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_7373c433] (Usage-HB-1:[]) (logid:) Scheduling Usage job...                                                                                                                                                                                         
2026-01-29 11:36:13,804 DEBUG [usage.UsageManagerImpl.Heartbeat] (Usage-HB-1:[]) (logid:) it's been 973804 ms since last usage job and 62926197 ms until next job, scheduling an immediate job to catch up (aggregation duration is 5 minutes)                                                                                      
2026-01-29 11:36:13,805 DEBUG [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_7373c433] (Usage-HB-1:[]) (logid:) Scheduling Usage job...                                                                                                                                                                                         
2026-01-29 11:37:13,804 DEBUG [usage.UsageManagerImpl.Heartbeat] (Usage-HB-1:[]) (logid:) it's been 1033804 ms since last usage job and 62866197 ms until next job, scheduling an immediate job to catch up (aggregation duration is 5 minutes)                                                                                     
2026-01-29 11:37:13,804 DEBUG [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_7373c433] (Usage-HB-1:[]) (logid:) Scheduling Usage job...      

mysql> mysql> select * from usage_job where end_millis=0;
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
| id   | host                                                 | pid    | job_type | scheduled | start_millis | end_millis | exec_time | start_date | end_date | success | heartbeat           |
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
| 1811 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 393299 |        0 |         0 |            0 |          0 |         0 | NULL       | NULL     |    NULL | 2026-01-29 11:34:13 |
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
1 row in set (0.00 sec)

mysql> select * from usage_job where end_millis=0;
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
| id   | host                                                 | pid    | job_type | scheduled | start_millis | end_millis | exec_time | start_date | end_date | success | heartbeat           |
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
| 1811 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 393299 |        0 |         0 |            0 |          0 |         0 | NULL       | NULL     |    NULL | 2026-01-29 11:34:13 |
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
1 row in set (0.00 sec)

mysql> 
mysql> select * from usage_job where end_millis=0;
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
| id   | host                                                 | pid    | job_type | scheduled | start_millis | end_millis | exec_time | start_date | end_date | success | heartbeat           |
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
| 1811 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 393299 |        0 |         0 |            0 |          0 |         0 | NULL       | NULL     |    NULL | 2026-01-29 11:35:13 |
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
1 row in set (0.00 sec)

mysql> select * from usage_job where end_millis=0;
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
| id   | host                                                 | pid    | job_type | scheduled | start_millis | end_millis | exec_time | start_date | end_date | success | heartbeat           |
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
| 1811 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 393299 |        0 |         0 |            0 |          0 |         0 | NULL       | NULL     |    NULL | 2026-01-29 11:36:13 |
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
1 row in set (0.00 sec)

mysql> select * from usage_job where end_millis=0;
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
| id   | host                                                 | pid    | job_type | scheduled | start_millis | end_millis | exec_time | start_date | end_date | success | heartbeat           |
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
| 1812 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 393299 |        0 |         0 |            0 |          0 |         0 | NULL       | NULL     |    NULL | 2026-01-29 11:37:16 |
+------+------------------------------------------------------+--------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
1 row in set (0.00 sec)

Please note that this doesn't create multiple usage_job entry, the last active usage_job is updated.
But this will queue lots of jobs to the _executor. The _executor is single threaded. So the first actual job will finish and then all the queue jobs will be run one after another, but not end up doing anything.

As a fix, Heartbeat code should check if there is a job already running and not try to schedule it again.

[abh1sar]

I could also see lot's of failed -1 second jobs as a side-effect of the above

mysql> select * from usage_job where id>3140;       
+------+------------------------------------------------------+--------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+
| id   | host                                                 | pid    | job_type | scheduled | start_millis  | end_millis    | exec_time | start_date          | end_date            | success | heartbeat           |
+------+------------------------------------------------------+--------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+
| 3141 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 393299 |        0 |         0 | 1770085920000 | 1770086219999 |    300810 | 2026-02-03 02:32:00 | 2026-02-03 02:36:59 |       1 | 2026-02-03 02:42:13 |
| 3142 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 393299 |        0 |         0 | 1770086220000 | 1770086519999 |    300870 | 2026-02-03 02:37:00 | 2026-02-03 02:41:59 |       1 | 2026-02-03 02:47:13 |
| 3143 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 393299 |        0 |         0 | 1770086520000 | 1770086819999 |    300879 | 2026-02-03 02:42:00 | 2026-02-03 02:46:59 |       1 | 2026-02-03 02:52:13 |
| 3144 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 393299 |        0 |         0 | 1770086820000 | 1770087119999 |    300785 | 2026-02-03 02:47:00 | 2026-02-03 02:51:59 |       1 | 2026-02-03 02:57:13 |
| 3146 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770087120000 | 1770087659999 |   3602755 | 2026-02-03 02:52:00 | 2026-02-03 03:00:59 |       1 | 2026-02-03 04:01:30 |
| 3147 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770087660000 | 1770091259999 |     11428 | 2026-02-03 03:01:00 | 2026-02-03 04:00:59 |       1 | 2026-02-03 04:01:33 |
| 3148 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         2 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:44 |
| 3149 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:44 |
| 3150 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:44 |
| 3151 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:44 |
| 3152 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         2 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:44 |
| 3153 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:44 |
| 3154 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         2 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3155 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3156 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3157 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3158 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3159 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3160 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3161 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3162 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3163 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3164 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3165 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3166 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3167 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3168 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3169 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3170 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         2 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3171 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3172 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         2 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3173 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3174 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3175 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3176 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3177 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3178 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         2 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3179 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3180 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3181 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3182 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         2 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3183 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3184 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3185 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3186 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3187 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         2 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3188 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3189 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:45 |
| 3190 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3191 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3192 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         2 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3193 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3194 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3195 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         2 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3196 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3197 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3198 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3199 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3200 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3201 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3202 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3203 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3204 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3205 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3206 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 | 1770091260000 | 1770091259999 |         1 | 2026-02-03 04:01:00 | 2026-02-03 04:00:59 |       0 | 2026-02-03 04:01:46 |
| 3207 | ref-trl-10702-k-Mol8-abhisar-sinha-mgmt2/10.0.35.225 | 672936 |        0 |         0 |             0 |             0 |         0 | NULL                | NULL                |    NULL | 2026-02-03 04:01:46 |
+------+------------------------------------------------------+--------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+

[DaanHoogland]

fixed in #12616