Question using gmt batch

Q&A
1

Hi,

update : jump to it

Trying to pursue my quest to shift to GMT 6, I’m investigating gmt batch as a way to automatically plot some diagrams with a running window see this previous thread.

Adapting the example given in the doc, I have :

  • pre.sh

contains a list of fixed parameters as well as a timefile created with gmt math

gmt math -o0 -T1980-01-01T/2019-12-31T/1461d T = times.txt
  • main.sh

should contain the plot.

Here’s my “problem”. I want my plots to be from :
1 day0 to day2
2 day1 to day3
3 day2 to day4
4 … and so forth.

from what I understand BATCH_COL0 is the date extracted from times.txt at each increment.

So is batch really a good solution for me ? (compared to a for-loop)
or can I append a 2nd column to times.txt to plot from BATCH_col0 to BATCH_col1 ?

gmt math -o0 -T1980-01-01T/2015-01-01T/1y T = times_start.txt
gmt math -o0 -T1984-12-31T/2019-12-31T/1y T = times_stop.txt
awk '{print $1,$1}' times_start.txt times_stop.txt > times.txt

When the incy will be fixed

Thanks

2

Ok,

I answered my first question about BATCH_COL0 and BATCH_COL1.

Now my problem is how do I access variables declared before main.sh (or during pre.sh) ?

cat << EOF > main.sh
day1=\${BATCH_COL0}
day2=\${BATCH_COL1}
region1="-R\${lon1a}/\${lon1b}/\${day1}/\${day2}"
> -R//1980-01-01T00:00:00/1984-12-31T00:00:00

clearly lon1a and lon1b are not piped into main.sh

3

RTFM > use -Iincludefile

4

Ok … I think I’m almost done. But I still can’t produce any plot.

gmt batch main.sh -Sbpre.sh -Sfpost.sh -Ntest -Imyvars.sh -Ttimestamps.txt

main.sh :

Relatively straightforward, like said in the doc I needed to :

  1. escape all the $var : (\${var})
  2. save the called variables into an independent file (called afterward with -I flag)

-Sbpre.sh and -Sfpost.sh:

I’m not actually sure I needed it. The former generates the cpt file, the latter combines all the output into a single pdf (like the example in the doc)

-Ntest :

Explicit in the doc.

-Imyvars.sh :

It contains all the paths and variables used by both pre.sh (for -Tmin/max/inc)and main.sh (for -Rlon1/lon2/day1/day2, the path to my files …)
I’m not sure why but it has to be a script file (.sh). Also as it is called from the subfolder “test”, any relative path has to go one step further (../[...] instead of ./[...])

-Ttimestamps.txt

I was not able to use -J[…]T (absolute time), but t worked. So the file timestamps.txt becomes

1980-01-01 1984-12-31 -> 0 1826

Result : nothing

$ bash batch_test.sh 
batch [INFORMATION]: Reading Data Table from File times.txt
batch [INFORMATION]: Create parameter initiation script batch_init.sh
batch [INFORMATION]: Create preflight script batch_preflight.sh and execute it
batch [INFORMATION]: Number of main processing jobs: 3
batch [INFORMATION]: Parameter files for main processing: 3
batch [INFORMATION]: Create main batch job script batch_job.sh
batch [INFORMATION]: Create cleanup script batch_cleanup.sh
batch [INFORMATION]: 3 job product sets saved in directory: test
batch [INFORMATION]: Total jobs to process: 3
batch [INFORMATION]: Build jobs using 3 cores
batch [INFORMATION]: Execute batch job scripts in parallel
mv: rename test_1.* to /Users/guillaume/Desktop/TEST/test_1.*: No such file or directory
batch [INFORMATION]: Job 1 of 3 completed [ 33.3 %]
mv: rename test_2.* to /Users/guillaume/Desktop/TEST/test_2.*: No such file or directory
batch [INFORMATION]: Job 2 of 3 completed [ 66.7 %]
mv: rename test_0.* to /Users/guillaume/Desktop/TEST/test_0.*: No such file or directory
batch [INFORMATION]: Job 0 of 3 completed [100.0 %]
  • No plot is rendered
  • The cpt file is empty
  • I don’t think it comes from the data as the “1 shot” script works

batch_test.zip (4.5 KB)

5

Try to run batch with -Qs and examine all the scripts to see why there are no plots made. Hard to tell from here.since I have no access to data. I assume the batch man page exampes work for you?

6

It does.
So -Qs didn’t provide much but -Q did.

The pdfs are plotted in

./test/test_*/test.pdf

but once created, the script tries to access to

./test_*/test.pdf

In ./test/batch_job.sh

mkdir ${BATCH_NAME}
cd ${BATCH_NAME}
[...]
mv -f ${BATCH_NAME}.* /Users/guillaume/Desktop/TEST
cd ..

Also, the cpt file created in preflight is not used (and the file is empty). Using makecpt -H and specifying grdimage -Cfile.cpt didn’t fix it (maybe a path problem again?)

Am I doing something wrong or should I report an issue on Github ?

7

COuld you post your correct times.txt file for me, please?

8

Also not you will need to use -Cfile.cpt to all modules using it, including colorbar.

9

Do I need the -H modifier too ?

times_absolute.txt (66 Bytes) times.txt (25 Bytes)

10

Absolutely. With the -C stuff I would think your scripts will run.

11

You’d be surprised :smiley:

I added the -C../colr.cpt (cf -I myvars.sh paragraph)

  1. It indeed plot colour now. One. Red.
  2. I fall into the problem of -J[..]ct vs -J[..]cT :
    • If I use relative times.txt (0 365) : gridline behaves weirdly
    • If I use absolute times.txt (1980-01-01 1980-12-31) : I’m out of boundary for y-axis

Getting close, but not quite :grin:

12

Ok. @pwessel I might have identified a bug…

The problem doesn’t seem to come from the script but from the reading of NetCDF file :

ncdump -h $FILE
netcdf era5_eqband_1980-2019 {
dimensions:
        longitude = 201 ;
        latitude = 41 ;
        time = 14610 ;
variables:
        float longitude(longitude) ;
                longitude:standard_name = "longitude" ;
                longitude:units = "degrees east" ;
        float latitude(latitude) ;
                latitude:standard_name = "latitude" ;
                latitude:units = "degrees north" ;
        float time(time) ;
                time:standard_name = "time" ;
                time:units = "days since 1980-01-01 00:00:00.0 +0000" ;
        [...]
        float eqb_nc_sst(time, longitude) ;
        [...]
        float eqb_cl_sst(time, longitude) ;
        [...]


ncdump -v eqb_cl_sst $FILE
[...]
    -0.1943895, -0.1819219, -0.1787305, -0.173094, -0.1565612, -0.1355813, 
    -0.1194141, -0.110683, -0.108428, -0.1164616, -0.1349583, -0.1532463, 
    -0.162128, -0.1596579, -0.1486065, -0.1344378, -0.1248495, -0.1255047, 
    -0.1362992, -0.1448266, -0.1390951, -0.1221133, -0.1006322, -0.08042147, 
    -0.06641228, -0.06340005, -0.07365708, -0.08978543, -0.1017248, 
[...]

ncdump -v eqb_nc_sst $FILE
[...]
    301.3525, 301.3748, 301.4026, 301.4487, 301.4844, 301.503, 301.5208, 
    301.5542, 301.5971, 301.6189, 301.6379, 301.6664, 301.6905, 301.6994, 
    301.7058, 301.7216, 301.7384, 301.7455, 301.7495, 301.7679, 301.7834, 
    301.7922, 301.7981, 301.8046, 301.8098, 301.809, 301.8095, 301.8071, 
[...]

All the nodes in eqb_cl_sst (anomalies in Kelvin or ºC) are contained between +/- 5. However gmt reads value from 298 to 305 which correspond to eqb_nc_sst (raw in Kelvin)

gmt grdimage \${region1} \${projection1} \${FILE_ind?eqb_cl_sst} -C'./../colr.cpt' -Bxa20f10g10 -Bpyf1Og1O -Bsya1Yf1Yg1Y -BWSen --MAP_GRID_PEN_PRIMARY=thinner,grey --MAP_GRID_PEN_SECONDARY=thin,black

plots the same thing as

gmt grdimage \${region1} \${projection1} \${FILE_ind?eqb_nc_sst} -C'./../colr.cpt' -Bxa20f10g10 -Bpyf1Og1O -Bsya1Yf1Yg1Y -BWSen --MAP_GRID_PEN_PRIMARY=thinner,grey --MAP_GRID_PEN_SECONDARY=thin,black

(I also checked on Matlab and there’s no problem)

UPDATE : no matter the variable I try to access it always plot the first one having the same dimension. eqb_nc_sst here.

13

Solved : the variable need to be in quote (Thanks @pwessel for the tip)

 \${FILE_ind}"?eqb_cl_sst"
14

FYI, you may want to read this: https://github.com/GenericMappingTools/gmt/pull/4781#pullrequestreview-587133286

15

@pwessel :

I re-run the script shown in “showcase” but adding the -Ttable.txt file like this (either in include.sh or pre.sh:

option 1 : echo "1980-01-01 1984-12-31" > table.txt 
option 2 : echo "1980-01-01 1984-12-31" > table.txt 
           echo "1981-01-01 1985-12-31" >>table.txt
option 3 : gmt math -o0 -T1980-01-01T/1982-01-01T/1y T --FORMAT_CLOCK_OUT=’’ = start.txt
           gmt math -o0 -T1984-12-31T/1986-12-31T/1y T --FORMAT_CLOCK_OUT=’’ = stop
           paste -d" " start.txt stop.txt > table.txt

Options 1 2 and 3 erase the variables from -Iinclude.sh after pre.sh but before main.sh
Option 1 produce job with no number (instead of simply job_0)