Data Management Practice Session

Exercise 1: Managing Data of Storage Elements using lcg-utils

1. Upload File Into SE:

Use your test user instead of tut28 references in this document

My file to transfer:

echo "This is a text file for the Workshop on Grid and EScience " > text_file.txt

Which is the LFC catalog asigned to my vo in the BDII?

lcg-infosites --vo vo.formacion.es-ngi.eu lfc

We prepare the catalog environment variable:

export LFC_HOST=gridlfc01.ifca.es

Test the catalog info:

lfc-ls /grid/vo.formacion.es-ngi.eu

Which are the SE availables?

lcg-infosites --vo vo.formacion.es-ngi.eu se

uploading to griddpm01.ifca.es, lcg util updates the LFC

lcg-cr --vo vo.formacion.es-ngi.eu file:./text_file.txt -l lfn:/grid/vo.formacion.es-ngi.eu/tut28/text_file.txt -d griddpm01.ifca.es

The lcg util responds whith the asigned GUID to the LFN.

Check the entry in the catalog:

lfc-ls /grid/vo.formacion.es-ngi.eu/tut28/

2. Replicating the file

Replicating the same file to another SE (e-se.iaa.es):

lcg-rep --vo vo.formacion.es-ngi.eu -d e-se.iaa.es lfn:/grid/vo.formacion.es-ngi.eu/tut28/text_file.txt

3. Get the file SURLs (Site URLs):

lcg-lr --vo vo.formacion.es-ngi.eu lfn:/grid/vo.formacion.es-ngi.eu/tut28/text_file.txt

That will display the list of replicas (SURLs) of the given lfn, or of the given GUID or SURL is also possible.

4. Alias

An alias is a LFN asociated to an existing replica. The alias is similar to the link concept in a Unix-like filesystem. For example:

Firstly, copy a sample file to a SE, without mention to the lfn (-l option), which mean that the entry will NOT be added in the LFN catalog:

lcg-cr -d griddpm01.ifca.es --vo vo.formacion.es-ngi.eu file:./text_file.txt

The command return the asigned guid to the file copy in the SE

Secondary, add an alias in the LFC catalog to the existing replica, using the GUID returned:

lcg-aa --vo vo.formacion.es-ngi.eu guid:54955ce5-d242-4c0e-882b-51eba48d0df5 lfn:/grid/vo.formacion.es-ngi.eu/tut28/alias-textfile

check the created alias:

lfc-ls -l /grid/vo.formacion.es-ngi.eu/tut28/

Have a look to the attributes asociated to the entries of the catalog (-l option for long output format), they are very similar to Unix-like filesystem attributes. For our created alias you will find something similar to:

lrwxrwxrwx 1 148 104 0 Jun 09 16:39 alias-textfile -> /grid/vo.formacion.es-ngi.eu/generated/2010-06-0/file-9fdc3dd4-ef67-497c-944d-41d9f9683ab4

Another verification can be done with:

lcg-la --vo vo.formacion.es-ngi.eu guid:54955ce5-d242-4c0e-882b-51eba48d0df5

5. Get the TURL (Transport URL):

lcg-lr --vo vo.formacion.es-ngi.eu lfn:/grid/vo.formacion.es-ngi.eu/tut28/text_file.txt

That display the list of replicas (SURLs) asociated to the lfn, in our case:

srm://e-se.iaa.es/dpm/iaa.es/home/vo.formacion.es-ngi.eu/generated/2010-06-09/fileb3285c7b-75a9-42fc-aec6-7db76da2afe2

srm://griddpm01.ifca.es/dpm/ifca.es/home/vo.formacion.es-ngi.eu/generated/2010-06-09/file11f3e1f7-dbb6-45d4bc84-d6f16a595f80

With the SURLs we can obtain the TURLs, of a specific replica in the supported Grid transfer protocols:

lcg-gt srm://e-se.iaa.es/dpm/iaa.es/home/vo.formacion.es-ngi.eu/generated/2010-06-09/fileb3285c7b-75a9-42fcaec6-7db76da2afe2 rfio

lcg-gt srm://e-se.iaa.es/dpm/iaa.es/home/vo.formacion.es-ngi.eu/generated/2010-06-09/fileb3285c7b-75a9-42fc-aec6-7db76da2afe2 gsiftp

6. Download a lfn to a localfile in the UI:

lcg-cp --vo vo.formacion.es-ngi.eu lfn:/grid/vo.formacion.es-ngi.eu/tut28/text_file.txt file:./bajado.txt

check:

cat bajado.txt

7. Delete a file from SEs and the asociated entry in the LFC catalog:

lcg-del -a --vo vo.formacion.es-ngi.eu lfn:/grid/vo.formacion.es-ngi.eu/tut28/text_file.txt

check:

lfc-ls -l /grid/vo.formacion.es-ngi.eu/tut28/

Exercise 2: Working with the catalog

Replace the tut28 references with your own tuturial user.

We prepare the catalog environment variable:

export LFC_HOST=gridlfc01.ifca.es

1. Listing and creating directories

List with -l option is lonf format:

lfc-ls -l /grid/vo.formacion.es-ngi.eu/tut28/

Create a directory for the current tutorial:

lfc-mkdir /grid/vo.formacion.es-ngi.eu/tut28/nuevoDir

2. Renaming an entry:

lfc-rename /grid/vo.formacion.es-ngi.eu/tut28/alias-textfile /grid/vo.formacion.es-ngi.eu/tut28/renamed-alias-textfile

check:

lfc-ls -l /grid/vo.formacion.es-ngi.eu/tut28/

3. Create an entry symlink and compare with alias:

lfc-ln -s /grid/vo.formacion.es-ngi.eu/tut28/renamed-alias-textfile /grid/vo.formacion.es-ngi.eu/tut28/renamed-alias-textfile-symlink

compare:

lfc-ls -l /grid/vo.formacion.es-ngi.eu/tut28/

in our case:

lrwxrwxrwx 1 148 104 0 Jun 09 16:39 renamed-alias-textfile -> /grid/vo.formacion.es-ngi.eu/generated/2010-06-09/file-9fdc3dd4-ef67-497c-944d-41d9f9683ab4

lrwxrwxrwx 1 148 104 0 Jun 09 18:06 renamed-alias-textfile-symlink -> /grid/vo.formacion.es-ngi.eu/tut28/renamed-alias-textfile

We have done a symlink of an entry which is an alias, the alias is pointing to the generated entry: file-9fdc3dd4-ef67-497c-944d-41d9f9683ab4, the symlink just point to the original entry in current directory structure: renamed-alias-textfile

Bouth have a size of 0 bytes (5th field), because they point to a source file that will have a size, something similar to:

lfc-ls -l /grid/vo.formacion.es-ngi.eu/generated/2010-06-09/file-9fdc3dd4-ef67-497c-944d-41d9f9683ab4

-rw-rw-r-- 1 148 104 58 Jun 09 16:33 /grid/vo.formacion.es-ngi.eu/generated/2010-06-09/file-9fdc3dd4-ef67-497c-944d-41d9f9683ab4

alias-> point to generated file of the catalog, of a previous file in a SE.

symlink-> point to the original entry

4. Putting a comment to an entry:

lfc-setcomment /grid/vo.formacion.es-ngi.eu/tut28/renamed-alias-textfile "This is a comment that may be used as a kindof metadata of the entry"

check:

lfc-ls --comment /grid/vo.formacion.es-ngi.eu/tut28/renamed-alias-textfile

delete the comment:

lfc-delcomment /grid/vo.formacion.es-ngi.eu/tut28/renamed-alias-textfile

check:

lfc-ls --comment /grid/vo.formacion.es-ngi.eu/tut28/renamed-alias-textfile

5. Managing rights of the entries: (symbolic modes ar not supported)

Absolute modes user, group, others:

7 rwx

6 rw-

5 r-x

4 r--

3 -wx

2 -w-

1 --x

lfc-chmod 775 /grid/vo.formacion.es-ngi.eu/generated/2010-06-09/file-9fdc3dd4-ef67-497c-944d-41d9f9683ab4

check:

lfc-ls -l /grid/vo.formacion.es-ngi.eu/generated/2010-06-09/file-9fdc3dd4-ef67-497c-944d-41d9f9683ab4

-rwxrwxr-x 1 148 104 58 Jun 09 16:33 /grid/vo.formacion.es-ngi.eu/generated/2010-06-09/file-9fdc3dd4-ef67-497c-944d-41d9f9683ab4

User Owner id of the file: 148

Group id of the file: 104

Changes on symlinks and alias, are suppeditated to the source file rights, like Unix-like filesystem.

lfc-chmod 777 /grid/vo.formacion.es-ngi.eu/tut28/renamed-alias-textfile

It will modified the original file of the alias, check:

lfc-ls -l /grid/vo.formacion.es-ngi.eu/generated/2010-06-09/file-9fdc3dd4-ef67-497c-944d-41d9f9683ab4

What can you see?

6. Managing Access Control List (ACL) for directories

In conjunction with file rights, LFC has a very powerful instrument given by access control list, which allows to a file/directory owner to grain finely access rights for that entry for any other users.

Getting the ACL:

lfc-getacl /grid/vo.formacion.es-ngi.eu/tut28/nuevoDir

# file: /grid/vo.formacion.es-ngi.eu/tut28/nuevoDir

# owner: /C=ES/O=IFCA/CN=tut28

# group: vo.formacion.es-ngi.eu

user::rwx

group::rwx #effective:rwx

other::r-x

default:user::rwx

default:group::rwx

default:other::r-x

Note that ownership is expressed through DN of certificate, in order to grant unicity and avoid conflicts, while group is expressed through VO membership.

the user group and other are ACL of the directory and the default rights, are applied for each new entry created within this directory.

Changing ACLs:

With read/write permission for user and group, and no privileges for others. The syntax we apply here is modify (-m) default (d:) for user (u::), and the same of course for group and others:

lfc-setacl -m d:u::6,d:g::6,d:o:0 /grid/vo.formacion.es-ngi.eu/tut28/nuevoDir

check:

lfc-getacl /grid/vo.formacion.es-ngi.eu/tut28/nuevoDir

ok, create a new entry in nuevoDir:

lcg-cr --vo vo.formacion.es-ngi.eu file:./text_file.txt -l lfn:/grid/vo.formacion.es-ngi.eu/tut28/nuevoDir/text_file.txt -d griddpm01.ifca.es

check default rights of new entry:

lfc-ls -l /grid/vo.formacion.es-ngi.eu/tut28/nuevoDir/

what can you see?

7. Deleting entries:

lfc-rm /grid/vo.formacion.es-ngi.eu/tut28/renamed-alias-textfile-symlink

lfc-rm removes just entry on catalog : the file on SE will continue to exist, in the case avobe, the entry is a symlink, but if the entry is a file, the file in the SE will be orphan of catalog reference, that is something to avoid, instead use lcg-del -a for a full delete in SEs and catalog.

for delete directories use lfc-rm -r and the directories have to be void before.

can you clean out all the entries of your tutorial directorie, please? deleting just the entry of catalog or also de file in SE when is needed?

Exercise 3: Working with the metadata catalog command line client

* exercise3_resources.tgz: Complementary material to do the exercise 3

USING THE command line AMGA client (mdclient)

-bash-3.2$ mdclient

Connecting to lfc01:8822...

ARDA Metadata Server 1.3.0

Query>

This is the mdclient promt

1. Basic operations with directories:

Query> pwd

Query> mkdir tut28

Instead of tut28 use your asigned tutorial user

Query> cd tut28

Query> mkdir borrame

Query> ls

Query> rmdir borrame

Query> ls

2. Handling attributes

An attribute structure is asigned to each directory.

The entries within a directory has the values for the attributes of the directory that belong to.

AMGA attributes types: int, float, varchar(n), timestamp, text, numeric(p,s)

Query> mkdir peliculas

Query> addattr /tutorial/tut28/peliculas MovieTitle^? varchar(100)

Query> addattr /tutorial/tut28/peliculas Runtime int

Query> addattr /tutorial/tut28/peliculas Sinopsis text

Query> addattr /tutorial/tut28/peliculas Year int

Query> addattr /tutorial/tut28/peliculas Director varchar(50)

Query> listattr /tutorial/tut28/peliculas/

Remove attributes is only working if the attribute to delete is not used by any entry inside the directory.

Query> removeattr /tutorial/tut28/peliculas/ Sinopsis

Query> listattr /tutorial/tut28/peliculas/

3. Managing entries

Creating an entry:

Query> addentry /tutorial/tut28/peliculas/avatar.mov MovieTitle^? Avatar Runtime 162 Year 2009

Query> addentry /tutorial/tut28/peliculas/GoneWithTheWind.mov MovieTitle^? 'Lo que el viento se llevo' Runtime 224 Year 1939

Get attributes: getattr entry_patern linst_of_attributes

Query> getattr /tutorial/tut28/peliculas/*.mov MovieTitle^? Runtime

Change attribute value:

setattr /tutorial/tut28/peliculas/GoneWithTheWind.mov Director 'Victor Fleming, George Cukor, Sam Wood'

Same, try to set James Cameron as the director of Avatar, and view all the attributes of /tutorial/tut28/peliculas/*.mov

4. Basic metadata queries

get entries of a pattern when query condition is true:

find pattern 'query_condition'

query_condition is SQL-like query and specifies a condition on attributes to filter the result set.

Common condition operators: +, -, *, /, , and, or, not, >, <=, <> or =

Find with text conditions:

Query> find /tutorial/tut28/peliculas/*.mov 'Director = like(Director,"James%")'

Now, find the entries of movies of the year between 2000 and 2010.

View attributes values: selectattr dir:attr ... query_condition

Query> selectattr .:MovieTitle .:Runtime 'Runtime > 120'

5. Integrating AMGA in the Data Management environment

Let's suppose:

echo "esto es el metraje de lo que el viento se llevó" > GoneWithTheWind^?.mov

echo "esto es el metraje de avatar" > avatar.mov

Upload the file to an SE, and include in the LFC catalog:

export LFC_HOST=gridlfc01.ifca.es

lcg-cr --vo vo.formacion.es-ngi.eu file:./avatar.mov -llfn:/grid/vo.formacion.es-ngi.eu/tut28/nuevoDir/avatar.mov -d griddpm01.ifca.es

Create an atribute to store the LFN,

mdclient

Query> addattr /tutorial/tut28/peliculas Director varchar(50)

Query> setattr /tutorial/tut28/peliculas/avatar.mov LFN 'lfn:/grid/vo.formacion.es-ngi.eu/tut28/nuevoDir/avatar.mov'

check:

Query> getattr /tutorial/tut28/peliculas LFN

6. Clean-out

Use removeentries and rmdir to clean out your directory

Exercise 4: Programing with GFAL

• exercise4_resources.tgz: Complementary material to do the exercise 4

Remote I/O using Grid File Access Library

Change tut28 references in the document with your own user

You will find complementary material in the course UIs in /tmp/DataManagement/GFAL_tutorial_resources

1. Prepare the environment variables:

export LFC_HOST=gridlfc01.ifca.es

export LCG_CATALOG_TYPE=lfc

export LCG_GFAL_VO=vo.formacion.es-ngi.eu

2. Prepare the text file:

lcg-cr --vo vo.formacion.es-ngi.eu file:./frases_de_donquijote.txt -l lfn:/grid/vo.formacion.es-ngi.eu/tut28/frases_de_donquijote.txt -d griddpm01.ifca.es

3. Compile the program:

gcc -g -o gfal_reader -I/opt/lcg/include -L/opt/lcg/lib64 -l gfal test_gfal_read.c

4. Runing the program in the UI:

./gfal_reader lfn:/grid/vo.formacion.es-ngi.eu/tut28/frases_de_donquijote.txt

5. Runing the program in the working nodes:

glite-wms-job-submit -a RemoteGFAL^?_read_test.jdl

checking the status:

glite-wms-job-status https://wms01.ific.uv.es:9000/e4sL9rJ2uopJ5pG4LSvpdw

When finished getting the output sandbox files:

glite-wms-job-output https://wms01.ific.uv.es:9000/e4sL9rJ2uopJ5pG4LSvpdw

Have a look to the output files.

6. You have to do a gfal write program running, taking the above read as an example:

6.1) edit test_gfal_write.c and find the "open" function to change to the corresponding gfal open remote file function, like in test_gfal_read.c.

6.2) change the write function to gfal remote write.

6.3) change the close function to gfal remote file close.

6.4) complie test_gfal_write.c like gfal_reader in 4.

6.5) run the program in the UI

note that gfal write function only work with an SURL argument, thus replica consistency is user dependand, use:

srm://griddpm01.ifca.es/dpm/ifca.es/home/vo.formacion.es-ngi.eu/grid/otra_frase_tut28.txt

again, where tut28 set your own tutorial username

6.6) Have a look to the writed file with the gfal_reader program

6.7) run the program in the working nodes changing the SURL and have a look to the results with gfal_reader

-- Main.vmendez - 07 Jul 2010