chimera-dump

Release Notes

Current Release is 0.8.

Changes in release 0.8:

Changes in release 0.7:

Added exception handling for circular entries in database
Added log-file information for errors (see above), will be extended while introducing more error handling of database problems

Changes in release 0.6:

Changes in release 0.5:

Changes in release 0.4:

The configuration was put in another file (cd_dump.py) to make the code more readable and the database access easier
xml output had a minior mistake
new query for checksum added
exception caught when database not accessible

The current version is a tar.gz file containing the script chimera-dump.py and a config file cd_dump.py where the database access can be configured.

chimera-dump.py [-h] [-o output file>] [-s <pattern>] [-v <vo>] [-a]
Running chimera-dump.py without options takes the default values from for the cd_conf.py file (unaltered this means the dCache chimera default values)
and makes a bz2 xml file with all files in the SE.
With -a you can have a (faster) output in ascii.
With -s you can give a string which have to be in every path.

python chimera-dump.py -a -s /pnfs/ifh.de/data/atlas
Give a xml file with all data in Path /pnfs/ifh.de/data/atlas and there size
python chimera-dump.py -v atlas -s /atlas/userdisk/users/leffhalm --check checksum
Give a xml file with all data in Path /pnfs/ifh.de/data/atlas and there checksum
python chimera-dump.py -v atlas -s atlas/atlasuserdisk/user08/leffhalm -g
Give a xml file with all data in Path matching atlas/atlasuserdisk/user08/leffhalm
./chimera-dump.py -a --string /mc08.105003.pythia_sdiff.recon.AOD.e344_s456_r545_tid026664 --check pool
Give the pool location of all files whose path match the pattern (are in the dataset ;-) as ascii
python chimera-dump.py -a -s pool-01 --check disk
Give ascii of all files on a certain pool
python chimera-dump.py -a --check NoP -g
Give ascii of all Files not on any pool compressed with gzip

Option	Default	Description
--version		show program's version number and exit
-h, --help		show this help message and exit
-o FILEN, --output=FILEN	/tmp/chimera-dump-<date>	Name of outputfile, bz2 will be added in any case, xml when -a is not specified
-s PAT, --string=PAT		string which should always be in the Path like /data/atlas/atlasmcdisk, this will filter the output to paths where the string is contained, no wildcards used. IMPORTANT: Don;t use the / right before the filename, this is not stored and you won't find any file if you state it
-v VO, --vo=VO	None	Name of vo: just for xml file
-a, --ascii		output in text, if not present output will be xml
-c, --check	dump	<dump\|checksum\|pool> will give the size\|cheksum\|pool-location info for all files found <NoP\|Mpools\|Spools> will search for files <without any\|with multiple\|with single> location on pools <disk> will give all files on the pool matching the given PAT with -s
-g		use gzip instead of bzip2 as compression

The main query to get all files in the database takes about 2 seconds per 710000 files
Than for every new directory a select command will be started, this is very fast
Most of the time the program writes to the disk
For our site it took 100seconds for the xml-file with 710000 files