We wanted to use a Unix mount as a long term repository having multiple layers of directories and sub-directories structure, Most of the directories with different retention days. We wanted to have a script for cleanup the old files which should take care of the complex retention periods and those distinct retention periods should not be hardcoded in the script because next time if someone wants to modify the retention period or create a new sub directory with different retention period he just need to add the information in a separate file, a config file, thats it! and the code should be dynamic enough to adapt it in next execution.
To understand it, lets have a look at below directory structure then we will look at the configuration file which will store the path of various directories and retention periods in unit of days.
The configuration file which stores the directory name with full path, space or tab then retention period in unit of days.
| clScr.conf {Configuration File} |
|---|
/temp/zTmpPlGrnd1/C12/C12B/C12B4/C12B4A 40 /temp/zTmpPlGrnd1/A12/A12C 30 /temp/zTmpPlGrnd1/C12/C12B/C12B5 90 /temp/zTmpPlGrnd1/A12 15 /temp/zTmpPlGrnd1/A12/A12B 62 /temp/zTmpPlGrnd1/A12/A12C/A12C4 13 |
With a overall retention period of 7 days and above configuration file retention periods for sub-directories the diagram will be like below:
But its not complete, To get the actual view, applying the logic that the sub-directory will have the same retention period as its parent directory unless it is not mentioned in configuration file. Here is the final view :smile:
ha ha ha..... I know .... it is ....
Below is the story needs to be told to the Operating system to achieve above scenario. So you will be needing two files at your OS level, one, this script file with execution permission and second, above file named "clScr.conf" with read permission. One can always modify the duration of retention period and add or remove lines from the file. You don't have to worry about the number of blank spaces between directory name and digits plus you don't have to keep them sorted.
We are initially taking maximum retention period as 07 days and our target is to reach the free space of the file system mount greater than or equal to 50%.
In order to achive it we are decreasing the retention period from 07 to 02 days at max [You need to decide these figures as per your requirement they are already highlighted in colors in below story]
The retention period mentioned in the configuration file will not be compromised through out the script. At the end, if this entire story will not be able to free space equal to 50% or more by the deletion activity then it will send a mail for further manual consideration.
| Script |
|---|
MAXDAY=7 THRESHOLD="50" ZMOUNT="/temp" BASEDIR="${ZMOUNT}/zTmpPlGrnd1" CFGFILE="${ZMOUNT}/zTmpPlGrnd0/reaal/clScr.conf" INTERIM="${ZMOUNT}/zTmpPlGrnd0/reaal/zintrm_" LOGFILE="${ZMOUNT}/zTmpPlGrnd0/reaal/CleanSPMout.log" DELFILE="${ZMOUNT}/zTmpPlGrnd0/reaal/DeletedFile.log" rm ${ZMOUNT}/zTmpPlGrnd0/reaal/*.3nt echo "Searching for Configuation file..." echo "" if [ -s ${CFGFILE} ]; then echo "Configuration File found." else echo "Configuration File Not found. Script is terminating!" echo "" echo "Hint: Please look for ${CFGFILE} file and start the script again." exit 0 fi cat ${CFGFILE} | awk '{print $1}' | sort > ${INTERIM}CfgSrtd.3nt cat ${CFGFILE} | awk '{print $1}' | sort -u > ${INTERIM}CfgSrtdUnq.3nt echo "" echo "Scanning the entries in the configuration file..." echo "" z1Dupe=`sdiff ${INTERIM}CfgSrtd.3nt ${INTERIM}CfgSrtdUnq.3nt | grep \< | awk '{print $1}'` if [ "${z1Dupe}" == "" ]; then echo "The entries in the configuration file are consistent." echo "" else echo "Duplicate entry found in the Configuration file for : $z1Dupe" echo "Please clean it up and rerun this script." echo "" exit 0 fi find ${BASEDIR} -type d | sort -u > ${INTERIM}MasterList.3nt for iDir in `cat ${INTERIM}MasterList.3nt` do echo "${iDir} ${MAXDAY}_3vYwj1n9Tc5D" >> ${INTERIM}WorkList.3nt done cat ${CFGFILE} | sort -u > ${INTERIM}SPLLIST.3nt cat ${INTERIM}SPLLIST.3nt | awk '{print $2}' > ${INTERIM}SPLDAYS.3nt cat ${INTERIM}WorkList.3nt | awk '{print $2}' > ${INTERIM}WLDAYS.3nt n=1 for iDir in `cat ${INTERIM}CfgSrtdUnq.3nt` do splMaxDay=`head -${n} ${INTERIM}SPLDAYS.3nt | tail -1` m=1 for iMstrDir in `cat ${INTERIM}MasterList.3nt` do CheckFamily=`echo ${iMstrDir} | grep ${iDir}` WLMaxDay=`head -${m} ${INTERIM}WLDAYS.3nt | tail -1` if [ "${CheckFamily}" == "" ]; then echo "${iMstrDir} ${WLMaxDay}" >> ${INTERIM}WL2.3nt else echo "${iMstrDir} ${splMaxDay}" >> ${INTERIM}WL2.3nt fi m=$((m+1)) done mv ${INTERIM}WL2.3nt ${INTERIM}WorkList.3nt cat ${INTERIM}WorkList.3nt | awk '{print $2}' > ${INTERIM}WLDAYS.3nt n=$((n+1)) done cat ${INTERIM}WorkList.3nt | grep _3vYwj1n9Tc5D | awk '{print $1}' > ${INTERIM}CmnList.3nt while [[ $MAXDAY -gt 2 ]]; do USAGE=`df -k | grep ${ZMOUNT} | awk '{print $4}' |tr -d "%"` echo "Current FS usage is: ${USAGE}" >> $LOGFILE if(( ${USAGE} < ${THRESHOLD} )); then echo "Cleanup job finished for comman Directories." >> $LOGFILE exit 0 else echo "Clean up files older than $MAXDAY days" >> $LOGFILE for cmnDir in `cat ${INTERIM}CmnList.3nt` do find ${cmnDir} -maxdepth 1 -mtime +$MAXDAY -print >> ${DELFILE} find ${cmnDir} -maxdepth 1 -mtime +$MAXDAY -exec rm {} \; done MAXDAY=$((MAXDAY - 1)) fi done cat ${INTERIM}WorkList.3nt | grep -v _3vYwj1n9Tc5D | awk '{print $1}' > ${INTERIM}SplList.3nt cat ${INTERIM}WorkList.3nt | grep -v _3vYwj1n9Tc5D | awk '{print $2}' > ${INTERIM}SplDays.3nt n=1 for splDir in `cat ${INTERIM}SplList.3nt` do splMaxDay=`head -${n} ${INTERIM}SplDays.3nt | tail -1` find ${splDir} -maxdepth 1 -mtime +$splMaxDay -print >> ${DELFILE} find ${splDir} -maxdepth 1 -mtime +$splMaxDay -exec rm {} \; n=$((n+1)) done USAGE=`df -k | grep ${ZMOUNT} | awk '{print $4}' |tr -d "%"` echo "Current FS usage is: ${USAGE}" >> $LOGFILE echo "Cleanup job finished." >> $LOGFILE if (( ${USAGE} > ${THRESHOLD} )); then echo "Mount ${ZMOUNT} is ${USAGE}% full after cleanup on "`hostname`" -- "`date`| mail -s "${ZMOUNT} space alert" user@mailserver.com fi exit 0 rm ${ZMOUNT}/zTmpPlGrnd0/reaal/*.3nt |
I'll appreciate your suggestions/comments, and try to answer your queries. Peace!
My other Blogs, if you have time...
What's new in SAP NetWeaver 7.3 - A Basis perspective Part-I
What's new in SAP NetWeaver 7.3 - A Basis perspective Part-II
Bye bye STRUSTSSO2: New Central Certificate Administration NW7.3
Escaping tough moments of SPAM or SAINT
SAP Software Provisioning Manager : with screenshots
Multiple/Bulk transports with tp script for Unix (AIX, Solaris, HP-UX, Linux)
Holistic Basis View: BusinessObjects BI 4.0 SP 2 Installation & Configuration
How to Rename the Oracle Listener & Change Listener port for SAP
OSS1 & RFC connections SAPOSS, SAPNET_RFC, SDCC_OSS
Start/Stop SAP along with your Unix Server Start/Stop
Interrelation: SAP work process, OPS$ mechanism, oracle client & oracle shadow process
Install and configure NetWeaver PI 7.3 Decentralize Adapter part-1
Install and configure NetWeaver PI 7.3 Decentralize Adapter part-2
Holistic Basis View: BusinessObjects BI 4.0 SP 2 Installation & Configuration
List of Newly added/converted Dynamic parameter in NetWeaver 7.3
Sunset for ops$ mechanism: No more supported by Oracle & Not Used by SAP
Essential Basis for SAP (ABAP, BW, Functional) Consultants Part-I
Essential Basis for SAP (ABAP, BW, Functional) Consultants Part-II
Essential Basis for SAP (ABAP, BW, Functional) Consultants Part-III
1 Comment