# Question using gmt batch

**URL:** <https://forum.generic-mapping-tools.org/t/question-using-gmt-batch/1320>\
**Category:** Q&A\
**Created:** [February 9, 2021, 12:03am UTC](https://forum.generic-mapping-tools.org/t/question-using-gmt-batch/1320 "2021-02-09T00:03:33Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![PlanetGus](https://yyz1.discourse-cdn.com/flex047/user_avatar/forum.generic-mapping-tools.org/planetgus/32/303_2.png) [@PlanetGus](https://forum.generic-mapping-tools.org/u/PlanetGus)\
**Post date:** [February 9, 2021, 12:03am UTC](https://forum.generic-mapping-tools.org/t/question-using-gmt-batch/1320/1 "2021-02-09T00:03:33Z")

</div>

Hi,

[update : jump to it](https://flint.soest.hawaii.edu/t/question-using-gmt-batch/1320/4)

Trying to pursue my quest to shift to GMT 6, I’m investigating `gmt batch` as a way to automatically plot some diagrams with a running window [see this previous thread](https://flint.soest.hawaii.edu/t/modern-mode-should-i/1066).

Adapting the example given in the doc, I have :

- pre.sh

> contains a list of fixed parameters as well as a _timefile_ created with `gmt math`

```auto
gmt math -o0 -T1980-01-01T/2019-12-31T/1461d T = times.txt

```

- main.sh

> should contain the plot.

Here’s my “problem”. I want my plots to be from :  
1 day0 to day2  
2 day1 to day3  
3 day2 to day4  
4 … and so forth.

from what I understand `BATCH_COL0` is the date extracted from `times.txt` at each increment.

So is `batch` really a good solution for me ? (compared to a for-loop)  
or can I append a 2nd column to `times.txt` to plot from `BATCH_col0` to `BATCH_col1` ?

```auto
gmt math -o0 -T1980-01-01T/2015-01-01T/1y T = times_start.txt
gmt math -o0 -T1984-12-31T/2019-12-31T/1y T = times_stop.txt
awk '{print $1,$1}' times_start.txt times_stop.txt > times.txt

```

When the _inc_ **y** [will be fixed](https://github.com/GenericMappingTools/gmt/issues/4771)

Thanks

---

<div class="post-metadata">

**Author:** ![PlanetGus](https://yyz1.discourse-cdn.com/flex047/user_avatar/forum.generic-mapping-tools.org/planetgus/32/303_2.png) [@PlanetGus](https://forum.generic-mapping-tools.org/u/PlanetGus)\
**Post date:** [February 9, 2021, 3:41am UTC](https://forum.generic-mapping-tools.org/t/question-using-gmt-batch/1320/2 "2021-02-09T03:41:55Z")

</div>

Ok,

I answered my first question about `BATCH_COL0` and `BATCH_COL1`.

Now my problem is how do I access variables declared before main.sh (or during pre.sh) ?

```auto
cat << EOF > main.sh
day1=\${BATCH_COL0}
day2=\${BATCH_COL1}
region1="-R\${lon1a}/\${lon1b}/\${day1}/\${day2}"
> -R//1980-01-01T00:00:00/1984-12-31T00:00:00

```

clearly lon1a and lon1b are not piped into main.sh

---

<div class="post-metadata">

**Author:** ![PlanetGus](https://yyz1.discourse-cdn.com/flex047/user_avatar/forum.generic-mapping-tools.org/planetgus/32/303_2.png) [@PlanetGus](https://forum.generic-mapping-tools.org/u/PlanetGus)\
**Post date:** [February 9, 2021, 3:48am UTC](https://forum.generic-mapping-tools.org/t/question-using-gmt-batch/1320/3 "2021-02-09T03:48:22Z")

</div>

RTFM \> use **-I** _includefile_

---

<div class="post-metadata">

**Author:** ![PlanetGus](https://yyz1.discourse-cdn.com/flex047/user_avatar/forum.generic-mapping-tools.org/planetgus/32/303_2.png) [@PlanetGus](https://forum.generic-mapping-tools.org/u/PlanetGus)\
**Post date:** [February 9, 2021, 5:56am UTC](https://forum.generic-mapping-tools.org/t/question-using-gmt-batch/1320/4 "2021-02-09T05:56:18Z")

</div>

Ok … I think I’m almost done. But I still can’t produce any plot.

```auto
gmt batch main.sh -Sbpre.sh -Sfpost.sh -Ntest -Imyvars.sh -Ttimestamps.txt

```

## _main.sh_ :

Relatively straightforward, like said in the doc I needed to :

1. escape all the `$var` : (`\${var}`)
2. save the called variables into an independent file (called afterward with `-I` flag)

## -Sb_pre.sh_ and -Sf_post.sh_:

I’m not actually sure I needed it. The former generates the cpt file, the latter combines all the output into a single pdf (like the example in the doc)

## -N_test_ :

Explicit in the doc.

## -I_myvars.sh_ :

It contains all the paths and variables used by both `pre.sh` (for **-T** _min_/_max_/_inc_)and `main.sh` (for **-R** _lon1/lon2/day1/day2_, the path to my files …)  
I’m not sure why but it **has to be** a script file (`.sh`). Also as it is called from the subfolder “test”, any relative path has to go one step further (`../[...]` instead of `./[...]`)

## -T_timestamps.txt_

I was not able to use -J[…]_T_ (absolute time), but _t_ worked. So the file timestamps.txt becomes

```auto
1980-01-01 1984-12-31 -> 0 1826

```

## Result : nothing

```auto
$ bash batch_test.sh 
batch [INFORMATION]: Reading Data Table from File times.txt
batch [INFORMATION]: Create parameter initiation script batch_init.sh
batch [INFORMATION]: Create preflight script batch_preflight.sh and execute it
batch [INFORMATION]: Number of main processing jobs: 3
batch [INFORMATION]: Parameter files for main processing: 3
batch [INFORMATION]: Create main batch job script batch_job.sh
batch [INFORMATION]: Create cleanup script batch_cleanup.sh
batch [INFORMATION]: 3 job product sets saved in directory: test
batch [INFORMATION]: Total jobs to process: 3
batch [INFORMATION]: Build jobs using 3 cores
batch [INFORMATION]: Execute batch job scripts in parallel
mv: rename test_1.* to /Users/guillaume/Desktop/TEST/test_1.*: No such file or directory
batch [INFORMATION]: Job 1 of 3 completed [33.3 %]
mv: rename test_2.* to /Users/guillaume/Desktop/TEST/test_2.*: No such file or directory
batch [INFORMATION]: Job 2 of 3 completed [66.7 %]
mv: rename test_0.* to /Users/guillaume/Desktop/TEST/test_0.*: No such file or directory
batch [INFORMATION]: Job 0 of 3 completed [100.0 %]

```

- No plot is rendered
- The cpt file is empty
- I don’t think it comes from the data as the “1 shot” script works

[batch\_test.zip](https://forum.generic-mapping-tools.org/uploads/short-url/pu0cg9NxreAhBLf62Fq9Qkak7my.zip) (4.5 KB)

---

<div class="post-metadata">

**Author:** ![pwessel](https://yyz1.discourse-cdn.com/flex047/user_avatar/forum.generic-mapping-tools.org/pwessel/32/30_2.png) [@pwessel](https://forum.generic-mapping-tools.org/u/pwessel)\
**Post date:** [February 9, 2021, 7:31am UTC](https://forum.generic-mapping-tools.org/t/question-using-gmt-batch/1320/5 "2021-02-09T07:31:55Z")

</div>

Try to run batch with -Qs and examine all the scripts to see why there are no plots made. Hard to tell from here.since I have no access to data. I assume the batch man page exampes work for you?

---

<div class="post-metadata">

**Author:** ![PlanetGus](https://yyz1.discourse-cdn.com/flex047/user_avatar/forum.generic-mapping-tools.org/planetgus/32/303_2.png) [@PlanetGus](https://forum.generic-mapping-tools.org/u/PlanetGus)\
**Post date:** [February 9, 2021, 2:36pm UTC](https://forum.generic-mapping-tools.org/t/question-using-gmt-batch/1320/6 "2021-02-09T14:36:43Z")

</div>

> [@pwessel](#):
>
> I assume the batch man page exampes work for you?

It does.  
So `-Qs` didn’t provide much but `-Q` did.

The pdfs are plotted in

```auto
./test/test_*/test.pdf

```

but once created, the script tries to access to

```auto
./test_*/test.pdf

```

In `./test/batch_job.sh`

```auto
mkdir ${BATCH_NAME}
cd ${BATCH_NAME}
[...]
mv -f ${BATCH_NAME}.* /Users/guillaume/Desktop/TEST
cd ..

```

Also, the cpt file created in preflight is not used (and the file is empty). Using `makecpt -H` and specifying grdimage -C_file.cpt_ didn’t fix it (maybe a path problem again?)

Am I doing something wrong or should I report an issue on Github ?

---

<div class="post-metadata">

**Author:** ![pwessel](https://yyz1.discourse-cdn.com/flex047/user_avatar/forum.generic-mapping-tools.org/pwessel/32/30_2.png) [@pwessel](https://forum.generic-mapping-tools.org/u/pwessel)\
**Post date:** [February 9, 2021, 7:45pm UTC](https://forum.generic-mapping-tools.org/t/question-using-gmt-batch/1320/7 "2021-02-09T19:45:38Z")

</div>

COuld you post your correct times.txt file for me, please?

---

<div class="post-metadata">

**Author:** ![pwessel](https://yyz1.discourse-cdn.com/flex047/user_avatar/forum.generic-mapping-tools.org/pwessel/32/30_2.png) [@pwessel](https://forum.generic-mapping-tools.org/u/pwessel)\
**Post date:** [February 9, 2021, 7:46pm UTC](https://forum.generic-mapping-tools.org/t/question-using-gmt-batch/1320/8 "2021-02-09T19:46:17Z")

</div>

Also not you will need to use -Cfile.cpt to all modules using it, including colorbar.

---

<div class="post-metadata">

**Author:** ![PlanetGus](https://yyz1.discourse-cdn.com/flex047/user_avatar/forum.generic-mapping-tools.org/planetgus/32/303_2.png) [@PlanetGus](https://forum.generic-mapping-tools.org/u/PlanetGus)\
**Post date:** [February 9, 2021, 8:01pm UTC](https://forum.generic-mapping-tools.org/t/question-using-gmt-batch/1320/9 "2021-02-09T20:01:06Z")

</div>

> [@pwessel](#):
>
> you will need to use -Cfile.cpt

Do I need the `-H` modifier too ?

[times\_absolute.txt](https://forum.generic-mapping-tools.org/uploads/short-url/kHPaTexHdfSxoliRFFDKqYC5LVE.txt) (66 Bytes) [times.txt](https://forum.generic-mapping-tools.org/uploads/short-url/xeNVQK5MsHw3nKenhjvnA5Rn1Ua.txt) (25 Bytes)

---

<div class="post-metadata">

**Author:** ![pwessel](https://yyz1.discourse-cdn.com/flex047/user_avatar/forum.generic-mapping-tools.org/pwessel/32/30_2.png) [@pwessel](https://forum.generic-mapping-tools.org/u/pwessel)\
**Post date:** [February 9, 2021, 8:25pm UTC](https://forum.generic-mapping-tools.org/t/question-using-gmt-batch/1320/10 "2021-02-09T20:25:01Z")

</div>

Absolutely. With the -C stuff I would think your scripts will run.

---

<div class="post-metadata">

**Author:** ![PlanetGus](https://yyz1.discourse-cdn.com/flex047/user_avatar/forum.generic-mapping-tools.org/planetgus/32/303_2.png) [@PlanetGus](https://forum.generic-mapping-tools.org/u/PlanetGus)\
**Post date:** [February 9, 2021, 8:44pm UTC](https://forum.generic-mapping-tools.org/t/question-using-gmt-batch/1320/11 "2021-02-09T20:44:55Z")

</div>

You’d be surprised 😃

I added the `-C../colr.cpt` (cf [**-I** _myvars.sh_ paragraph](https://flint.soest.hawaii.edu/t/question-using-gmt-batch/1320/4))

1. It indeed plot colour now. One. Red.
2. I fall into the problem of [`-J[..]ct` vs `-J[..]cT`](https://flint.soest.hawaii.edu/t/incoherent-gridline-behaviour/1319/14) :
  - If I use relative times.txt (0 365) : gridline behaves weirdly
  - If I use absolute times.txt (1980-01-01 1980-12-31) : I’m out of boundary for y-axis

Getting close, but not quite 😁

---

<div class="post-metadata">

**Author:** ![PlanetGus](https://yyz1.discourse-cdn.com/flex047/user_avatar/forum.generic-mapping-tools.org/planetgus/32/303_2.png) [@PlanetGus](https://forum.generic-mapping-tools.org/u/PlanetGus)\
**Post date:** [February 9, 2021, 10:23pm UTC](https://forum.generic-mapping-tools.org/t/question-using-gmt-batch/1320/12 "2021-02-09T22:23:23Z")

</div>

Ok. @pwessel I might have identified a bug…

The problem doesn’t seem to come from the script but from the reading of NetCDF file :

```auto
ncdump -h $FILE
netcdf era5_eqband_1980-2019 {
dimensions:
        longitude = 201 ;
        latitude = 41 ;
        time = 14610 ;
variables:
        float longitude(longitude) ;
                longitude:standard_name = "longitude" ;
                longitude:units = "degrees east" ;
        float latitude(latitude) ;
                latitude:standard_name = "latitude" ;
                latitude:units = "degrees north" ;
        float time(time) ;
                time:standard_name = "time" ;
                time:units = "days since 1980-01-01 00:00:00.0 +0000" ;
        [...]
        float eqb_nc_sst(time, longitude) ;
        [...]
        float eqb_cl_sst(time, longitude) ;
        [...]

ncdump -v eqb_cl_sst $FILE
[...]
    -0.1943895, -0.1819219, -0.1787305, -0.173094, -0.1565612, -0.1355813, 
    -0.1194141, -0.110683, -0.108428, -0.1164616, -0.1349583, -0.1532463, 
    -0.162128, -0.1596579, -0.1486065, -0.1344378, -0.1248495, -0.1255047, 
    -0.1362992, -0.1448266, -0.1390951, -0.1221133, -0.1006322, -0.08042147, 
    -0.06641228, -0.06340005, -0.07365708, -0.08978543, -0.1017248, 
[...]

ncdump -v eqb_nc_sst $FILE
[...]
    301.3525, 301.3748, 301.4026, 301.4487, 301.4844, 301.503, 301.5208, 
    301.5542, 301.5971, 301.6189, 301.6379, 301.6664, 301.6905, 301.6994, 
    301.7058, 301.7216, 301.7384, 301.7455, 301.7495, 301.7679, 301.7834, 
    301.7922, 301.7981, 301.8046, 301.8098, 301.809, 301.8095, 301.8071, 
[...]

```

All the nodes in `eqb_cl_sst` (anomalies in Kelvin or ºC) are contained between +/- 5. However gmt reads value from 298 to 305 which correspond to `eqb_nc_sst` (raw in Kelvin)

```auto
gmt grdimage \${region1} \${projection1} \${FILE_ind?eqb_cl_sst} -C'./../colr.cpt' -Bxa20f10g10 -Bpyf1Og1O -Bsya1Yf1Yg1Y -BWSen --MAP_GRID_PEN_PRIMARY=thinner,grey --MAP_GRID_PEN_SECONDARY=thin,black

```

plots the same thing as

```auto
gmt grdimage \${region1} \${projection1} \${FILE_ind?eqb_nc_sst} -C'./../colr.cpt' -Bxa20f10g10 -Bpyf1Og1O -Bsya1Yf1Yg1Y -BWSen --MAP_GRID_PEN_PRIMARY=thinner,grey --MAP_GRID_PEN_SECONDARY=thin,black

```

(I also checked on Matlab and there’s no problem)

UPDATE : no matter the variable I try to access it always plot the first one having the same dimension. `eqb_nc_sst` here.

---

<div class="post-metadata">

**Author:** ![PlanetGus](https://yyz1.discourse-cdn.com/flex047/user_avatar/forum.generic-mapping-tools.org/planetgus/32/303_2.png) [@PlanetGus](https://forum.generic-mapping-tools.org/u/PlanetGus)\
**Post date:** [February 9, 2021, 11:08pm UTC](https://forum.generic-mapping-tools.org/t/question-using-gmt-batch/1320/13 "2021-02-09T23:08:15Z")

</div>

Solved : the variable need to be in quote (Thanks @pwessel for the tip)

```auto
 \${FILE_ind}"?eqb_cl_sst"

```

---

<div class="post-metadata">

**Author:** ![pwessel](https://yyz1.discourse-cdn.com/flex047/user_avatar/forum.generic-mapping-tools.org/pwessel/32/30_2.png) [@pwessel](https://forum.generic-mapping-tools.org/u/pwessel)\
**Post date:** [February 10, 2021, 12:15am UTC](https://forum.generic-mapping-tools.org/t/question-using-gmt-batch/1320/14 "2021-02-10T00:15:23Z")

</div>

FYI, you may want to read this: [https://github.com/GenericMappingTools/gmt/pull/4781#pullrequestreview-587133286](https://github.com/GenericMappingTools/gmt/pull/4781#pullrequestreview-587133286)

---

<div class="post-metadata">

**Author:** ![PlanetGus](https://yyz1.discourse-cdn.com/flex047/user_avatar/forum.generic-mapping-tools.org/planetgus/32/303_2.png) [@PlanetGus](https://forum.generic-mapping-tools.org/u/PlanetGus)\
**Post date:** [February 10, 2021, 6:32am UTC](https://forum.generic-mapping-tools.org/t/question-using-gmt-batch/1320/15 "2021-02-10T06:32:27Z")

</div>

@pwessel :

I re-run the script shown in “showcase” but adding the `-Ttable.txt` file like this (either in `include.sh` or `pre.sh`:

```auto
option 1 : echo "1980-01-01 1984-12-31" > table.txt 
option 2 : echo "1980-01-01 1984-12-31" > table.txt 
           echo "1981-01-01 1985-12-31" >>table.txt
option 3 : gmt math -o0 -T1980-01-01T/1982-01-01T/1y T --FORMAT_CLOCK_OUT=’’ = start.txt
           gmt math -o0 -T1984-12-31T/1986-12-31T/1y T --FORMAT_CLOCK_OUT=’’ = stop
           paste -d" " start.txt stop.txt > table.txt

```

Options 1 2 and 3 erase the variables from `-Iinclude.sh` after `pre.sh` but before `main.sh`  
Option 1 produce job with no number (instead of simply job\_0)