ATLAS IFIC Tier-2
Introduction
The IFIC ATLAS computing resources is built on the Tier-2 centre in Valencia. It is the corrdinator center of the Federated Spanish ATLAS Tier-2 formed by three groups: IFIC group, IFAE group and UAM group. IFIC side has126 Worker Nodes (WNs) disposed in racks specially prepared to the adaptation of communication equipment and five disk serves with a capacity of 5 TB and a tape robot with a potential capacity of 140 TB. In terms of software, resources include ATLAS offline releases as well as the Grid infrastrucure based on the EGEE middleware are available. Core services are Resource Brokers (RB), Information Index (BDII), Proxy Sever (PXY) and Virtual Organization Management (VOMS). Resources are provided by the site information Services (GIIS), Computing Element (CE) with Worker Nodes and Castor-based Storage Element (SE) as a back-end to the mass storage facility. IFIC Tier-2 facility serves the dual purpose of providing resources to the ATLAS production system and providing Spanish physicists with infrastructure to perform analysis jobs into Grid.
Resources and Accounting
Information about the present resources and accounting for the Federeated Spanish ATLAS Tier-2 can be found
here
Documentation
Getting access to the Grid
Getting an account at GOG (Grupo de Ordenadores para el GRID)
To abtain an account you have to follow the below instructions:
- Read the GOG Usage Rules
- Fill out this form
- Print it and sign
- Give it to the IFIC Secretary
- After signing you will receive your account info by e-mail.
People affiliated with IFIC institute are eligible for using the
GOG infrastracture based on the EGEE Grid middleware.
Before you can use the Grid, you have to get:
-
Grid certificate
-
Registred in a Virtual Organization (VO)
Getting a Certificate
(Old information) All the required information on how to obtain a Grid certificate is available at this
web page.
You simply have to have the IFIC afs account and follow the instructions carefully from step 1 to step 5.
Some remarks (old procedure):
- Create the $HOME/.globus directory and install the certificate files there.
- Export your Grid Certificate from PEM into PKCS12 forma (have a look at the instructions at gog web page).
- Import the P12-certificate into your browser, see here. Then test it on that page.
NEW Procedure:
To get or renew a certificate you must go to
https://pk.irisgrid.es/rat23/. This new or renew certificate is going to be in a P12 format which you have to upload into your browser. Then, you must create the KEY and PEM certificate and copy them to your $HOME/.globus directory (i.e at IFIC and at CERN).
To create the KEY and CERT files you could use the following script: ~sanchezj/public/p12toglobus.sh
How to register to the ATLAS VO
The registration to the ATLAS VO is a mandatory step for having access to grid resources. This operation is done through the
LCG User Registration. After having a valid user certificate, you must ask for a registration
here. This web form will send your request to the ATLAS VO administrator, who will then have to authorize your registration (this step requires ~1 working day). Only after approval, you will be able to do a voms-proxy-init -voms atlas (this command will look for your certificate on the server lcg-voms.cern.ch).
Grid setup
LCG setup at IFIC
From AFS account at IFIC one can get access to the LCG User Interface (UI) as follows:
To check your grid certificate have to set:
-
voms-proxy-init -voms atlas
You get something like:
Your identity: /C=ES/O=DATAGRID-ES/O=IFIC/CN=Farida Fassi
Enter GRID pass phrase:
Creating temporary proxy ..................................... Done
Contacting lcg-voms.cern.ch:15001 [/C=CH/O=CERN/OU=GRID/CN=host/lcg-voms.cern.ch] "atlas" Done
Creating proxy ............................................ Done
Your proxy is valid until Fri Jan 5 00:29:53 2007
LCG setup at LXPLUS
To enable LCG at LXPLUS source at the command line the following lines:
$ ssh lxplus.cern.ch
$ source /afs/cern.ch/project/gd/LCG-share/current/etc/profile.d/grid_env.[c]sh
$ voms-proxy-init -voms atlas
Athena setup
ATLAS CVS Browser and Software Cross Reference
- The link which provides all the documentation related to the Offline ATLAS software releases is located here
- For Cross-Referencing ATLAS Software you should have a look at this page
Setting up Athena at IFIC
The below recipe shows you how to prepare your account to use the ATLAS kit release installed at Tier-2 del IFIC.
Setp-1
1) make a directory called AthenaTestArea, and then do $ cd AthenaTestArea
2) make two other directories called "12.0.6" and "cmthome" (mkdir -p 12.0.6 cmthome). Note that your Working Area will be "AthenaTestArea/12.0.6"
Setp-2
Note that: you have to log into User Interface (ui01) for release 12.0.6 installed at IFIC Tier-2.
3) ssh ui02, and then do
$ source /opt/exp_software/atlas/software/12.0.6/cmtsite/setup.sh (you need to do this only once)
4) create the file ~/AthenaTestArea/cmthome/requirements with the following contents:
############################################################################################3
set CMTSITE STANDALONE
set SITEROOT /opt/exp_software/atlas/prod/releases/rel_12-0_2
macro ATLAS_DIST_AREA ${SITEROOT}
apply_tag projectArea
macro SITE_PROJECT_AREA ${SITEROOT}
macro EXTERNAL_PROJECT_AREA ${SITEROOT}
apply_tag simpleTest
macro ATLAS_TEST_AREA ${HOME}/AthenaTestArea/12.0.6
use AtlasLogin AtlasLogin-* $(ATLAS_DIST_AREA)
set CMTCONFIG i686-slc3-gcc323-opt
set DBRELEASE_INSTALLED 3.1.1
set CMTSTRUCTURINGSTYLE "with_version_directory"
##############################################################################################################
And then, do the next
5) cmt config
6) source setup.sh -tag=12.0.6,oneTest
Setp-3
Check out and compile the
UserAnalysis package
$ cd ~/AthenaTestArea/12.0.6, and then do:
$ export CVSROOT=:pserver:anonymous@atlas-sw.cern.ch:/atlascvs (*For kserver access readding*)
$ cmt co -r UserAnalysis-00-09-10 PhysicsAnalysis/AnalysisCommon/UserAnalysis
$ cd PhysicsAnalysis/AnalysisCommon/UserAnalysis/UserAnalysis-00-09-10/cmt/
$ cmt config
$ source setup.sh
$ cmt broadcast gmake
Sep-4
$ cd ../run
$ get_files StructuredAAN_topOptions.py
$ get_files AnalysisSkeleton_topOptions.py
Note that, you have to edit
AnalysisSkeleton_topOptions.py and specify in this line your AOD input data that you want to process:
EventSelector.InputCollections = [ "AOD.pool.root"]
$ athena.py -b
AnalysisSkeleton_topOptions.py
setting up Athena with EvenViewGroupArea? at IFIC
- Make a directory called AthenaWorkArea in which make another two directories:
$ mkdir AthenaWorkArea, and then cd AthenaWorkArea
$ mkdir -p 12.0.6 cmtdir
- Log into the IFIC UI and do the following which needs to be done only once:
$ source /opt/exp_software/atlas/software/12.0.6/cmtsite/setup.sh
$ cmt config
$ source setup.sh -tag=12.0.6,groupArea,oneTest
- To checks that the CMT path, TestArea, GroupArea is correctly defined please type the following:
echo $CMTPATH
- Something like this is printed to the screen
/afs/ific.uv.es/user/f/farida/AthenaWorkArea/12.0.6:
/afs/ific.uv.es/user/f/farida/public/EVGroupArea/EventViewGroupArea/12.0.6:
/opt/exp_software/atlas/prod/releases/rel_12-0_2/AtlasOffline/12.0.6
- Now go to your WorkArea and checkout TopView package
$ cd $TestArea
$ export CVSROOT=:pserver:anonymous@atlas-sw.cern.ch:/atlascvs (*For kserver access readding*)
$ cmt co -r TopView-00-12-13-03 PhysicsAnalysis/TopPhys/TopView
$ cd PhysicsAnalysis/TopPhys/TopView/TopView-00-12-13-03/cmt
$ cmt config
$ source setup.sh
$ gmake (Be patience, it takes a while)
- Let us check that eveything is ok, but first you have to provide alternative datasets, you might have a look at this.
$ ../run
$ athena.py -b Analysis_topOptions.py (you can copy it from the share directory)
- Note that: TopView is a runtime package, you can run your jobs from TopView-00-12-13-03/run directory. You don't need to go to UserAnalysis.
Setting up at LXPLUS
- To set up Athena at LXPLUS please check this page
- The meaning of the different settings in the requirement file is explained here
How to interact with the LFC catalog
The LFC (or LCG File Catalog) has been developed
by to resolve problems with the EDG Replica Manager File Catalog.
The File Catalog (or Replica Catalog) maps between the Logical File Name and the Storage URL (or URLs) for files managed by the Grid. More information can be found
here.
To browse through the LFC file catalog manually:
$ ssh ui02
$ voms-proxy-init -voms atlas
$ export LCG_CATALOG_TYPE=lfc
$ export LFC_HOST=lfc01.pic.es
Some examples
$ lfc-ls /grid/atlas/datafiles/csc11/recon
$ lcg-lr --vo atlas lfn:/grid/atlas/datafiles/rome/recov10/rome.004100.recov10.T1_McAtNLO_top/rome.004100.recov10.T1_McAtNLO_top._03308.AOD.pool.root
$ lcg-gt srm://castorsrm.ific.uv.es/castor/ific.uv.es/grid/atlas/csc11/csc11.005200.T1_McAtNlo_Jimmy.recon.AOD.v11004204/csc11.005200.T1_McAtNlo_Jimmy.recon.AOD.v11004204._00394.pool.root.1 rfio
$ lcg-gt sfn://castorgrid.cern.ch/castor/cern.ch/grid/atlas/datafiles/rome/recov10/rome.004100.recov10.T1_McAtNLO_top/rome.004100.recov10.T1_McAtNLO_top._03308.AOD.pool.root.2 rfio
To locate the datasets you can use lfc-ls which works exactly as a regular ls command on a unix system. For example:
$ lfc-ls /grid/atlas/dq2/misalg_csc11/
Once you choose your dataset you may use the next command to see the replica. Here is an example:
$ lcg-lr --vo atlas lfn://grid/atlas/dq2/misalg_csc11/misalg_csc11.005300.PythiaH130zz4l.recon.AOD.v12003104_tid004174/misalg_csc11.005300.PythiaH130zz4l.recon.AOD.v12003104_tid004174._00001.pool.root.7
srm://castorsrm.ific.uv.es/castor/ific.uv.es/grid/atlas/SC4DA/misalg_csc11/misalg_csc11.005300.PythiaH130zz4l.recon.AOD.v12003104_tid004174/misalg_csc11.005300.PythiaH130zz4l.recon.AOD.v12003104_tid004174._00001.pool.root.7
srm://grid002.ft.uam.es/pnfs/ft.uam.es/data/atlas/misalg_csc11/misalg_csc11.005300.PythiaH130zz4l.recon.AOD.v12003104_tid004174/misalg_csc11.005300.PythiaH130zz4l.recon.AOD.v12003104_tid004174._00001.pool.root.7
srm://ifaesrm.pic.es/nfs/atlas-data00/atlas/data/misalg_csc11/misalg_csc11.005300.PythiaH130zz4l.recon.AOD.v12003104_tid004174/misalg_csc11.005300.PythiaH130zz4l.recon.AOD.v12003104_tid004174._00001.pool.root.7
srm://srm-disk.pic.es/pnfs/pic.es/data/atlas/dq2/misalg_csc11/misalg_csc11.005300.PythiaH130zz4l.recon.AOD.v12003104_tid004174/misalg_csc11.005300.PythiaH130zz4l.recon.AOD.v12003104_tid004174._00001.pool.root.7
To copy your dataset you can use the Globus-User tool: for example
$ globus-url-copy gsiftp://castorgrid.ific.uv.es/castor/ific.uv.es/grid/atlas/SC4DA/misalg_csc11/misalg_csc11.005300.PythiaH130zz4l.recon.AOD.v12003104_tid004174/misalg_csc11.005300.PythiaH130zz4l.recon.AOD.v12003104_tid004174._00001.pool.root.7 file://$PWD/test-5300-aod.root
Datasets and Distributed Data Managment (DDM).
How to find a dataset on the Grid
the ATLAS Metadata Interface,
AMI, is a flexible database application which allows you to find a
dataset
.
Useful information concerning
how to get the CSC Files
can be found
here
Work Book DDM
Following this
link an ATLAS end-user should be able to successfully access and/or manipulate the data stored on the grid and understand the basic principles of DDM operations.
How to locate a dataset on the Grid
To find the location of a dataset you can use either:
- The PANDA DQ2 dataset browser page
- The ATLAS Production monitoring database page
- To install, configure and use the DQ2 Catalog Client follow the instruction described in this page .
- Follow the next example if you want to use the running central scripts for interacting with the dq2 client form IFIC UI:
$ voms-proxy-init -voms atlas
$ source /afs/ific.uv.es/project/atlas/software/ddm/current/dq2.sh
$ source /afs/ific.uv.es/project/atlas/software/ddm/dq2_user_client/setup.sh.IFIC
$ dq2-list-dataset-site IFICDISK
$ dq2-list-dataset-replicas csc11.005200.T1_McAtNlo_Jimmy.recon.AOD.v11004204
INCOMPLETE: ASGCDISK,BNLTAPE,CERNCAF,CNAFDISK,CSCS,DESY-ZN,FZKDISK,FZU,IFAE,IFIC,LYONDISK,PICDISK,RALDISK,SARADISK,TOKYO,UAM
COMPLETE:
$ dq2-list-files csc11.005200.T1_McAtNlo_Jimmy.recon.AOD.v11004204
Explanation:
"dq2-list-dataset-replicas" command lists the replica of a dataset on the Grid. As you can see there is one at IFIC site.
- More information on interacting with DQ2 catalog client is located here
DQ2 end user tools at IFIC
The dq2 end-user tools do not require the whole DQ2 stuff. So, to access the datasets, managed by DQ2 tools, at IFIC site you need to download the setup file (
setup.sh.dq2_0.3_lustre
), which is available
here, and then follow the instructions listed next.
$ ssh ui02
$ voms-proxy-init -voms atlas
$ chmod +x setup.sh.dq2_0.3_IFIC
$ source setup.sh.dq2_0.3_IFIC
Some remarks
- dq2_ls: List dataset and files
- dq2_get: Download a dataset
- dq2_put: Create a dataset
- dq2_poolFCjob0: Create a PoolFileCatalog to locally access data
Some examples:
Look at content of dataset in global DQ2 catalogo
$ dq2_ls -fg misalg_csc11*AOD*
Download dataset files to local filesystem from local DQ2 catalog. If files are not found locally, download from global.
$ dq2_get -rv misalg_csc11.005300.PythiaH130zz4l.recon.AOD.v12003104_tid004174
Details are located in this
page and
here
How to find a dataset at IFIC Tier-2
The datasets which are registred in the Local LFC File Catalog can be seen browsing this web
page.
From the Atlas Distributed Data Management Dashboard monitoring page one can find out which datasets are located at the IFIC Storage Element:
Distributed Analysis using Ganga
Ganga is being developed as a Grid user interface for the ATLAS and LHCb experiments.The main goal of Ganga is to provide a simple and
consistent way of preparing, organising and executing analysis tasks, allowing physicists to concentrate on the algorithmic part without having to worry
about technical details. This is achieved by providing a clean interface which can be used interactively, as a Python API or through a GUI.
Ganga has a number of ATLAS-specific features:
- Simplify configuration of applications based on the Athena framework;
- Location of input and output data using the DQ2 data management system;
- Make easy to define and run user-level productions.
Detailed information about Ganga can be found at the project's web
site
Setting up Ganga at IFIC
There is an automatic script that sets the correct environment and finds the installed release of Ganga at IFIC.
To start the last Ganga version on UI just type the following command :
$ source /afs/ific.uv.es/project/atlas/software/ganga/install/etc/setup-atlas.sh
You can add the two following lines to the .tcshrc file:
setenv GANGA_CONFIG_PATH GangaAtlas/Atlas.ini
#for the newest version installed on afs
set path = (/afs/ific.uv.es/project/atlas/software/ganga/install/4.4.2/bin/ $path)
or if you are working in a sh based shell (add the below lines to .bashrc file):
export GANGA\_CONFIG\_PATH=GangaAtlas/Atlas.ini
#for the newest version installed on afs
PATH=/afs/ific.uv.es/project/atlas/software/ganga/install/4.4.2/bin:${PATH}
The first time ganga runs, it will ask to create a configuration file $HOME/.gangarc. For that, follow the next steps:
1. $ ganga -g ( this will create $HOME/.gangarc file)
2. In the section labelled [LCG] uncomment the line:
VirtualOrganisation = atlas
3. In the section labelled [Configuration] add the line:
RUNTIME_PATH = GangaAtlas
4. In the section labelled [Athena] add the lines.
Note: You might to indicate to the path directory in the Lustre SE:
LCGOutputLocation = srm://lsrm.ific.uv.es/lustre/ific.uv.es/grid/atlas/dq2/users/
LocalOutputLocation = srm://lsrm.ific.uv.es/lustre/ific.uv.es/grid/atlas/dq2/users/
ATLAS_SOFTWARE = /opt/exp_software/atlas/prod/releases/rel_12-0_2
Remarks on Running Ganga
You can start the ganga CLI by typing ganga on the command line. This starts a python interface, where you can start defining your jobs. There are a few commands you can use to get around in ganga:
jobs: Lists all the jobs that are defined in ganga. You can get to an indivudual job by typing:
jobs[id]: where the id is listed in the second column of the jobs output.
One thing you can do with a job is view it’s status:
jobs[1].status()
This can be ’new’, ’submitted’, ’running’ or ’completed’. Once the job is completed, you can view it’s output (which is stored by default in $HOME/gangadir/workspace/Local/ jobid /output) by typing:
In [25]: jobs[0].peek()
Or look at a specific output file by typing:
In [25]: jobs[0].peek(’stderr’,’less’)
where stderr is the name of the file you want to view, and less the program to view it with. You can kill a job using the kill() method, and remove it from the jobs list with the remove() method. The most important command by far is help(). This starts the interactive help program of ganga. After typing it, you get a help> prompt. Typing index gives you a list of all possible help subjects. The explanations are rather brief, but it does help you to find methods of build-in classes of Ganga and it’s plugin. For instance, the atlas plugin defines classes like DQ2Dataset. For more info on DQ2Dataset you type DQ2Dataset at the help> prompt.
Running an athena job with Ganga at IFIC
Before statring Ganga form CLI you need to setup grid, dq2-tools and athena enviroment as follows:
- 1. For grid setup, please log into ui02.ific.uv.es and do: voms-proxy-init -voms atlas.
- 2. For dq2_tools, please download the setup.sh.dq2_0.3_lustre script and source it
- 3. For athena setup, please follow the all steps listed in here.
Remarks:
- Start Ganga from the cmt or run directory of the Athena working area that has been setup before. Thus Ganga automatically recognizes all necessary environment variables and directories to be used with the Athena plugin.
Athena job with DQ2Dataset input
The following script defines a Athena job, runs athena with dq2 input dataset and stores the output on IFIC. To save the output data into DQ2 just substitute
ATLASOutputDataset with
DQ2OutputDataset.
Edit a python file called
Athena_ganga-test.py with the following contents
# Set the logging level of ganga to DEBUG for more info on what is happening
config["Logging"]['GangaAtlas'] = 'DEBUG'
j = Job()
j.name='Test-AthenaJob-IFIC'
j.application=Athena()
j.application.exclude_from_user_area=["*.o","*.root.*","*.exe"]
j.application.prepare(athena_compile=False)
j.application.option_file='$HOME/AthenaTestArea/12.0.6/PhysicsAnalysis/AnalysisCommon/UserAnalysis/UserAnalysis-00-09-10/run/AnalysisSkeleton_topOptions.py'
j.application.max_events='10'
j.application.atlas_release='12.0.6'
j.inputdata=DQ2Dataset()
j.inputdata.dataset="trig1_misal1_mc12.005186.PythiaZmumu_pt100_fixed.recon.AOD.v12000601_tid005906"
j.inputdata.min_num_files=10
j.inputdata.type='DQ2_LOCAL'
j.outputdata=DQ2OutputDataset()
j.outputdata.outputdata=['AnalysisSkeleton.aan.root']
j.backend=LCG()
j.backend.CE='ce01.ific.uv.es:2119/jobmanager-pbs-short'
j.submit()
To submit the jobs just execute the above script form Ganga IPython prompt like the following:
execfile('$HOME/'Athena_ganga-test.py')
Explanation of the terms:
- j.Application=Athena(): Defines the job to be an Athena job. Packs the local installation of athena packages, and sends them with the job.
- j.inputdata=DQDataset(): tells the job to get the files from the DQ2 file catalogue
- j.inputdata.min_num_files=10: instead of sending the job to a random location, this first checks that a given minimum of files is present at that location
- j.outputdata.outputdata=['AnalysisSkeleton.aan.root']: gives a list of filenames that must be stored in the output dataset.
- j.backend.CE: allows you to specify which Computing Element the job should be send to.
- j.application.exclude_from_user_area = []: allows you to exclude packages that you have installed locally from inclusion in the input sandbox (the tar file containing all the files that are send with your job to the CE)
After submitting your job you can type jobs in the ganga command line, which will show something like:
Out[1]: Statistics: 13 jobs
--------------
# id status name subjobs application backend backend.actualCE
# 0 completed test-1 Athena LCG ce01.ific.uv.es:2119/jobmanager-pbs-short
# 6 completed Test-local Athena Local ui02.ific.uv.es
# 12 completed test-lcg-1 Athena LCG ce-fzk.gridka.de:2119/jobmanager-pbspro-atlas
# 13 completed test-lcg-1 Athena LCG ce01.ific.uv.es:2119/jobmanager-pbs-short
# 27 completed TestGroupArea-ific Athena LCG ce01.ific.uv.es:2119/jobmanager-pbs-short
Here you can see all the jobs, their status, the type of job, it’s name, and at which CE it is running. If you want more info, you can type jobs[id] at the commandline, and you will get the complete configuration of the job. When the status changes to completed (ganga tells you of the change of status of any job sa soon as you issue a new command), you can see any verbal output by typing, just like before:
jobs[41].peek(’stdout’,’cat’)
If the job completed succesfully, you can retrieve the outputdata by typing:
jobs.outputdata.retrieve()
The outputdata is then stored in the directory ${HOME}/gangadir/workspace/Local/
/out.
Athena TopView job with DQ2Dataset input
- As mentioned previously to run TopView you need groupArea package.
- Before starting Ganga you need to setup grid and dq2 tools as described above.
- For Athena and Topview environmnet please follow the steps presented here
- Save the below scripts in python file called Test_GroupArea.py and execute it from Ganga CLI as above (execfile(´$HOME/Test_GroupArea.py´)
# Set the logging level of ganga to DEBUG for more info on what is happening
config["Logging"]['GangaAtlas'] = 'DEBUG'
j = Job()
j.name='TestGroupArea-ific'
j.application=Athena()
j.application.exclude_from_user_area=["*.o","*.root.*","*.exe"]
j.application.prepare(athena_compile=False)
j.application.option_file='$HOME/AthenaWorkArea/12.0.6/PhysicsAnalysis/TopPhys/TopView/TopView-00-12-13-03/run/Analysis_topOptions.py'
j.application.max_events='20'
j.application.atlas_release='12.0.6'
j.inputdata=DQ2Dataset()
j.inputdata.dataset="trig1_misal1_mc12.005186.PythiaZmumu_pt100_fixed.recon.AOD.v12000601_tid005906"
j.inputdata.names=['trig1_misal1_mc12.005186.PythiaZmumu_pt100_fixed.recon.AOD.v12000601_tid005906._00225.pool.root.1']
j.inputdata.type='DQ2_LOCAL'
j.outputdata=DQ2OutputDataset()
j.outputdata.outputdata=['TopViewAANtuple.root']
j.backend=LCG()
j.backend.CE='lcg2ce.ific.uv.es:2119/jobmanager-pbs-short'
j.submit()
Tutorials with Ganga
Assuming that a Athena release area has been created and Ganga has been started as described in the previous sections, the following listed tutorial wiki pages describe how to configure, submit and retrieve output of Athena analysis jobs and/or Monte Carlo Generation jobs on LCG Grid, batch systems and the local host.
Some slides on *using Ganga for physics analysis*
Note: have a look at these useful General remarks
ATLAS Software Tutorials
Useful links to the ATLAS Software Tutorials is listed in the main web page of the Physics Analysis Tools working group
-- FaridaFassi - 18 Dec 2006