defer PostJob for a longer time when waiting on Rucio

[belforte]

We may be waiting for Rucio transfer to many days (failing the PostJob will not help since we can not resubmit jobs which failed in transfer, Rucio will not accept an old LFN with a different checksum).

In this case it is better not to run many PostJobs uselessly every 30min, since we may end up overloading the AccessPoint resources. Better to increase defer time.

The best (only?) way seems to exit immediately from dag_bootstrap.sh with exit code 4, i.e. deferral stays at 30min (hardcoded in DAG description file), but PJ duration is minimal and Python is not started, saving a lot on memory.

See
#8429 (comment)

[belforte]

one idea

• grep DEFERRING in PostJob log to find how many times we tried.
  • after 10 times exit immediately with 50% chance and log DEFERRING
  • after 20 times with 10%
  • ...

something like this

number of DEFERRING	hours since completion	probabiliyt of running PostJob	Hours between PJ runs (average)
0	0	1	0.5
8	4	1/2	1
24	12	1/6	3
48	24	1/12	6
96	48	1/24	12

which is the simple formula if Ndefer > 8, prob of runnin PJ is 1/(Ndefer/4)

I think that the "run or not run" can be done via /usr/bin/shuf see stackoverflow

[belforte]

PostJob is called by DAGMAN via this POST script macro (broken into multiple lines for readibility)

SCRIPT DEFER 4 1800 POST Job1 dag_bootstrap.sh POSTJOB $JOBID $RETURN $RETRY $MAX_RETRIES
250520_093145:belforte_crab_20250520_113141 1 
/store/temp/user/belforte.4bba3c2d14d54a938291c268ff4eba8b575b197f/GenericTTbar/StefanoTest-240524/250520_093145/0000
/store/user/belforte/GenericTTbar/Stefano-Test-240524/250520_093145/0000
cmsRun_1.log.tar.gz conventional kk_1.root

as per

CRABServer/src/python/TaskWorker/Actions/DagmanCreator.py

Line 55 in accd4c8

SCRIPT DEFER 4 1800 POST Job{count} dag_bootstrap.sh POSTJOB $JOBID $RETURN $RETRY $MAX_RETRIES {taskname} {count} {tempDest} {outputDest} cmsRun_{count}.log.tar.gz {stage} {remoteOutputFiles}

so the arguments for dag_bootstrap.sh are:

#	arg	meaning/use
1	stage	(PREJOB/PREDAG/POSTJOB)
2	$JOBID	the HTCondor job ID [ClusterId.ProcId] of the node job HTC doc, irrelevant for this
3	$RETURN	the return value of the HTCondor job HTC doc, irrelevant for this
4	$RETRY	the current job retry #, starts at 0 HTC doc
5	$MAX_RETRIES	the max number of retries, irrelevant for this
6	TaskName	irrelevant for this
7	crabId	a.k.a. the job number. toghether with $RETRY identifies the postJob log file
8	temp_dir	irrelevant for this
9	out_dir	irrelevant for this
10	log file tarball	irrelevant for this
11	stage	conventional/probe/tail... irrelevant for this
12-	output files	irrelevant for this

see also

CRABServer/src/python/TaskWorker/TaskManagerBootstrap.py

Lines 13 to 14 in accd4c8

	if command == "POSTJOB":
	return PostJob.PostJob().execute(*sys.argv[2:])

and

CRABServer/src/python/TaskWorker/Actions/PostJob.py

Lines 1575 to 1594 in accd4c8

	# Put the arguments to PostJob into class variables.
	self.dag_jobid = args[0] # = ClusterId.ProcId
	self.job_return_code = int(args[1])
	self.dag_retry = int(args[2])
	self.max_retries = int(args[3])
	# TODO: Why not get the request name from the job ad?
	# We will need to parse the job ad earlier, that's all.
	self.reqname = args[4]
	self.job_id = args[5]
	self.source_dir = args[6]
	self.dest_dir = args[7]
	self.logs_arch_file_name = args[8]
	self.stage = args[9]
	# TODO: We can get the output files from the job ad where we have
	# CRAB_EDMOutputFiles, CRAB_TFileOutputFiles and CRAB_AdditionalOutputFiles.
	# (We only need to add the job_id in the file names).
	self.output_files_names = []
	for i in range(10, len(args)):
	self.output_files_names.append(args[i])

[belforte]

this does not work well in practice. With a decreasing probability to run PJ we may and up not running it for a long time even after transfers are done like in the example below[*]. I think that we should want to make sure that waiting time is e.g. never more then twice the foreseen average in the table.
Or replace the probability with a deterministic condition on number of deferrals.

[*] from https://cmsweb.cern.ch/scheddmon/059/crabint1/260330_160719:crabint1_crab_rucio_transfers_20260330_180716/postjob.3.0.txt

/GenericTTbar/crabint1-ruciotransfers-1774886836-94ba0e06145abd65ccb1d21786dc7e1d/USER
Tue, 31 Mar 2026 08:11:55 CEST(+0200):INFO:PostJob ====== Starting to check for ASO transfers.
Tue, 31 Mar 2026 08:11:55 CEST(+0200):DEBUG:PostJob Querying ASO RDBMS database.
Tue, 31 Mar 2026 08:11:55 CEST(+0200):INFO:PostJob Got statuses: submitted; 13.0 hours since transfer submit.
Tue, 31 Mar 2026 08:11:55 CEST(+0200):INFO:PostJob Deferring the execution of the post-job.
Tue, 31 Mar 2026 08:11:55 CEST(+0200):INFO:PostJob ********************************************************************************
======== Finished post-job execution with status code 4  : DEFERRING. PostJob will run again after 30 min
Ended TaskManagerBootstrap with code 4
Tue Mar 31 08:41:58 AM CEST 2026 dag_boostrap.sh DEFERRING. PostJob will run again after 30 min
Tue Mar 31 09:12:00 AM CEST 2026 dag_boostrap.sh DEFERRING. PostJob will run again after 30 min
Tue Mar 31 09:42:03 AM CEST 2026 dag_boostrap.sh DEFERRING. PostJob will run again after 30 min
Tue Mar 31 10:12:05 AM CEST 2026 dag_boostrap.sh DEFERRING. PostJob will run again after 30 min
Tue Mar 31 10:42:08 AM CEST 2026 dag_boostrap.sh DEFERRING. PostJob will run again after 30 min
Tue Mar 31 11:12:11 AM CEST 2026 dag_boostrap.sh DEFERRING. PostJob will run again after 30 min
Tue Mar 31 11:42:13 AM CEST 2026 dag_boostrap.sh DEFERRING. PostJob will run again after 30 min
Tue Mar 31 12:12:17 PM CEST 2026 dag_boostrap.sh DEFERRING. PostJob will run again after 30 min
Tue Mar 31 12:42:19 PM CEST 2026 dag_boostrap.sh DEFERRING. PostJob will run again after 30 min
Tue Mar 31 01:12:21 PM CEST 2026 dag_boostrap.sh DEFERRING. PostJob will run again after 30 min
Tue Mar 31 01:42:22 PM CEST 2026 dag_boostrap.sh DEFERRING. PostJob will run again after 30 min
Tue Mar 31 02:12:25 PM CEST 2026 dag_boostrap.sh DEFERRING. PostJob will run again after 30 min
Tue Mar 31 02:42:27 PM CEST 2026 dag_boostrap.sh DEFERRING. PostJob will run again after 30 min
Tue Mar 31 03:12:30 PM CEST 2026 dag_boostrap.sh DEFERRING. PostJob will run again after 30 min
Tue Mar 31 03:42:33 PM CEST 2026 dag_boostrap.sh DEFERRING. PostJob will run again after 30 min
Tue Mar 31 04:12:36 PM CEST 2026 dag_boostrap.sh DEFERRING. PostJob will run again after 30 min

[belforte]

@aspiringmind-code do you feel like helping with changing the script ? Your bash skills are vastly superior to mine, and it is good to have an exchange of ideas on what is the best strategy.

CRABServer/scripts/dagman/dag_bootstrap.sh

Lines 26 to 40 in 18b9ab7

	if [[ $scriptKind == "POSTJOB" ]] && [[ $outputDest =~ "/rucio/" ]] ; then
	pjlog=postjob.$crabId.$retry.txt
	defers=`grep -c DEFERRING $pjlog` # there's one defer every 30min, so this counts time !
	prob=1
	[[ $defers -ge 8 ]] && prob=$(($defers/4)) # see https://github.com/dmwm/CRABServer/issues/9079
	test=`shuf -i 1-$prob -n 1` # a random integer in [`1...$prob]
	# the PostJob python script is only run if the random draw equals to 1
	# for the first 4 hours, $defers is < 8 and $prob is always 1 so $test is always 1
	# after 4 hours we give it a 4/defers probability to actually run the python check
	if [ $test -ne 1 ] ; then
	echo `date` "dag_boostrap.sh DEFERRING. PostJob will run again after 30 min" >> $pjlog
	exit 4 # tells Dagman to re-run me after DEFER time indicated in Dagman file
	fi
	fi