CDO without shell scripting
Ralf Müller
MPI Met
5. Juni 2014
Max-Planck-Institut
für Meteorologie
Ralf Müller (MPI Met)
CDO without shell scripting
5. Juni 2014
1/9
WHAT ...
... is offered
Max-Planck-Institut
für Meteorologie
Ralf Müller (MPI Met)
CDO without shell scripting
5. Juni 2014
2/9
WHAT ...
... is offered
integration of CDO into python/ruby like a native library
Max-Planck-Institut
für Meteorologie
Ralf Müller (MPI Met)
CDO without shell scripting
5. Juni 2014
2/9
WHAT ...
... is offered
integration of CDO into python/ruby like a native library
keeps CDOs main feature: operator chaining
Max-Planck-Institut
für Meteorologie
Ralf Müller (MPI Met)
CDO without shell scripting
5. Juni 2014
2/9
WHAT ...
... is offered
integration of CDO into python/ruby like a native library
keeps CDOs main feature: operator chaining
multiple types of return values:
output files, numpy arrays, masked arrays
strings for operators, which write to stdout
None on error (optional)
Max-Planck-Institut
für Meteorologie
Ralf Müller (MPI Met)
CDO without shell scripting
5. Juni 2014
2/9
WHAT ...
... is offered
integration of CDO into python/ruby like a native library
keeps CDOs main feature: operator chaining
multiple types of return values:
output files, numpy arrays, masked arrays
strings for operators, which write to stdout
None on error (optional)
access to all options
TODO: environment settings
Max-Planck-Institut
für Meteorologie
Ralf Müller (MPI Met)
CDO without shell scripting
5. Juni 2014
2/9
WHATELSE ...
... is provided
Max-Planck-Institut
für Meteorologie
Ralf Müller (MPI Met)
CDO without shell scripting
5. Juni 2014
3/9
WHATELSE ...
... is provided
prebuild debian packages: python-cdo, python3-cdo
or: installation via pip/gem
Max-Planck-Institut
für Meteorologie
Ralf Müller (MPI Met)
CDO without shell scripting
5. Juni 2014
3/9
WHATELSE ...
... is provided
prebuild debian packages: python-cdo, python3-cdo
or: installation via pip/gem
IO: provide automatic tempfile handling
IO: optional use of existing files if present
Max-Planck-Institut
für Meteorologie
Ralf Müller (MPI Met)
CDO without shell scripting
5. Juni 2014
3/9
WHATELSE ...
... is provided
prebuild debian packages: python-cdo, python3-cdo
or: installation via pip/gem
IO: provide automatic tempfile handling
IO: optional use of existing files if present
interactive help
easy way to write new operators
Max-Planck-Institut
für Meteorologie
Ralf Müller (MPI Met)
CDO without shell scripting
5. Juni 2014
3/9
WTH ... internals
cdo.{rb,py}
is a smart caller of a CDO binary (with all the pros and cons)
doesn’t need to be re-installed for a new CDO version
Max-Planck-Institut
für Meteorologie
Ralf Müller (MPI Met)
CDO without shell scripting
5. Juni 2014
4/9
WTH ... internals
cdo.{rb,py}
is a smart caller of a CDO binary (with all the pros and cons)
doesn’t need to be re-installed for a new CDO version
isn’t a shared library, which keeps everything in memory
doesn’t allow write access to files via the numpy or masked arrays
Max-Planck-Institut
für Meteorologie
Ralf Müller (MPI Met)
CDO without shell scripting
5. Juni 2014
4/9
WTH ... internals
cdo.{rb,py}
is a smart caller of a CDO binary (with all the pros and cons)
doesn’t need to be re-installed for a new CDO version
isn’t a shared library, which keeps everything in memory
doesn’t allow write access to files via the numpy or masked arrays
See homepage at https://code.zmaw.de/projects/cdo/wiki/Cdo{rbpy}
Max-Planck-Institut
für Meteorologie
Ralf Müller (MPI Met)
CDO without shell scripting
5. Juni 2014
4/9
Usage: Basic - Ruby 2.x
1
require
3
f i l e s . e a c h {| f i l e |
maskedYMeanFile =
fldmeanFile
=
rhopotFile
=
mergedFile
=
diffFile
=
initFile
=
5
7
9
11
’ cdo ’
” m a s k e d y e a r m e a n #{F i l e . basename ( f i l e )}”
” f l d m e a n #{F i l e . basename ( f i l e )}”
” r h o p o t #{F i l e . basename ( f i l e )}”
”T−S−r h o p o t #{F i l e . basename ( f i l e )}”
”T−S−r h o p o t d i f f 2 i n i t #{F i l e . basename ( f i l e )}”
” i n i t i a l #{e x p e r i m e n t }. nc ”
Cdo . d i v ( i n p u t : ” −yearmean −s e l n a m e , T , S #{ f i l e } #{m a s k F i l e}” ,
output :
maskedYMeanFile )
13
15
Cdo . r h o p o t ( 0 ,
input :
maskedYMeanFile ,
output : r h o p o t F i l e )
17
19
21
Cdo . merge ( i n p u t :
[ maskedYMeanFile , r h o p o t F i l e ] . j o i n ( ’ ’ ) ,
output : mergedFile )
Cdo . s u b ( i n p u t :
[ mergedFile , i n i t F i l e ] . j o i n ( ’ ’ ) ,
output : d i f f F i l e )
23
25
Cdo . f l d s u m ( i n p u t :
”−mul #{ d i f f F i l e } #{m a s k e d A r e a W e i g h t s}” ,
output : fldmeanFile , : options
’−r −f nc ’ )
27
o f i l e s << f l d m e a n F i l e
}
u n l e s s F i l e . e x i s t ?( o f i l e )
Cdo . c a t ( i n p u t :
f i l e s . s o r t . j o i n ( ’ ’ ) , output :
31 end
29
Ralf Müller (MPI Met)
ofile , : options
CDO without shell scripting
’−r ’ )
Max-Planck-Institut
für Meteorologie
5. Juni 2014
5/9
Usage: Basic - Python 2.7
1
from o s . p a t h i m p o r t ∗
from cdo i m p o r t ∗
3
5
7
cdo
files , ofiles , ofile
maskFile
maskedAreaWeights
initFile
=
=
=
=
=
Cdo ( )
[ ’ /home/ ram / d a t a / i c o n / o c eL o ng . nc ’ ] , [ ] , ’ f l d m e a n o c e . nc ’
cdo . s e l n a m e ( ’ w e t c ’ , i n p u t= f i l e s [ 0 ] )
cdo . g r i d a r e a ( i n p u t
= maskFile )
cdo . s e l n a m e ( ’T , S ’ , i n p u t = ’ −s e l t i m e s t e p , 1 ’ + f i l e s [ 0 ] , o u t p u t = ’ i n i t i a l . nc ’ )
9
11
13
15
f o r i , myFile i n enumerate ( f i l e s ) :
maskedYMeanFile
= ” m a s k e d y e a r m e a n ”+basename ( m y F i l e )
f l d m e a n F i l e , r h o p o t F i l e = ” f l d m e a n ”+basename ( m y F i l e ) ,
” r h o p o t ”+basename ( m y F i l e )
mergedFile , d i f f F i l e
= ”T−S−r h o p o t ”+basename ( m y F i l e ) , ”T−S−r h o p o t d i f f 2 i n i t ”+basename ( m y F i l e )
cdo . d i v ( i n p u t = ” −yearmean −s e l n a m e , T , S ” + ’ ’ . j o i n ( ( m y F i l e , m a s k F i l e ) ) ,
o u t p u t = maskedYMeanFile )
17
19
21
cdo . r h o p o t ( 0 , i n p u t = maskedYMeanFile ,
output = r h o p o t F i l e )
cdo . merge ( i n p u t = ” ” . j o i n ( ( maskedYMeanFile , r h o p o t F i l e ) ) ,
output = mergedFile )
23
25
27
cdo . s u b ( i n p u t = ” ” . j o i n ( ( m e r g e d F i l e , i n i t F i l e ) ) ,
output = d i f f F i l e )
cdo . f l d s u m ( i n p u t = ”−mul ” + ” ” . j o i n ( ( d i f f F i l e , m a s k e d A r e a W e i g h t s ) ) ,
o u t p u t = f l d m e a n F i l e , o p t i o n s = ’−r −f nc ’ )
29
o f i l e s . append ( f l d m e a n F i l e )
31
33
i f not i s f i l e ( o f i l e ) :
o f i l e s . sort ()
cdo . c a t ( i n p u t = ’ ’ . j o i n ( o f i l e s ) , o u t p u t = o f i l e , o p t i o n s = ’−r ’ )
Ralf Müller (MPI Met)
CDO without shell scripting
Max-Planck-Institut
für Meteorologie
5. Juni 2014
6/9
Usage: Parallelism with Python
2
4
6
8
10
” ” ” s p l i t i n p u t f i l e s i n t o y e a r l y f i l e s − compute yearmean i f d e s i r e d ” ” ”
def s p l i t F i l e s I n t o Y e a r s ( f i l e s O f Y e a r s , archdir , forceOutput , expInfo , procs ) :
pool
= m u l t i p r o c e s s i n g . Pool ( procs )
yearFiles = []
f o r year , f i l e s i n f i l e s O f Y e a r s . i t e r i t e m s () :
yearFile
= p o o l . a p p l y a s y n c ( g r e p Y e a r , [ f i l e s , s t r ( y e a r ) , a r c h d i r , f o r c e O u t p u t , True , e x p I n f o ] )
yearFile , yearMeanFile = getFileNamesForYears ( s t r ( year ) , archdir , expInfo )
y e a r F i l e s . append ( [ y e a r , y e a r F i l e , y e a r M e a n F i l e ] )
pool . close ()
pool . j o i n ()
12
return yearFiles
Max-Planck-Institut
für Meteorologie
Ralf Müller (MPI Met)
CDO without shell scripting
5. Juni 2014
7/9
Usage: Examples
numpy
masked arrays
netcdf handles
conditional output
return None on error
exception handling
output operators
Max-Planck-Institut
für Meteorologie
Ralf Müller (MPI Met)
CDO without shell scripting
5. Juni 2014
8/9
???
Max-Planck-Institut
für Meteorologie
Ralf Müller (MPI Met)
CDO without shell scripting
5. Juni 2014
9/9