GRD file write error using API functions

Q&A
1

Hi

I have a code that uses GMT libraries to write to NetCDF grd files,

e.g. like so where the very last argument is the output filename without .grd, e.g. trying to write in a subdirectory and making a file tmp.grd

zcat /disk/twb3/citcom/fulltest/Ra1e8Pl1e6_ddy_151e7_10_244e9_e4_129xyz_25h/63000/v.ab.gz | abconvert 0 22 1 4 | shsyn 5 1 bin/x86_64/tmp
abconvert: reading poltor vector AB AB format in physical convention, lmax: 255
abconvert: compute divergence, switching to pol out
abconvert: output of poloidal part of expansion
shsyn: sum_mode: 1 and out_mode: 1
shsyn: readjusting nlon, nlat and xrange for FFT
shsyn: lmax=255 nlon=513 nlat=256
shsyn: E=0 dx=0.703125 W=360 S=-90 dy=0.705882 N=90
shsyn: constructing lookup tables (Legendre functions) …
shsyn: using 64.25 MB for P array, limit is set to 4200
shsyn: read 1 sets of coefficients with lmax=255
shsyn: set: 1 l: 0 pow^2: 0
shsyn: set: 1 l: 1 pow^2: 0.517226
shsyn: set: 1 l: 2 pow^2: 0.269922
shsyn: set: 1 l: 3 pow^2: 0.553164
shsyn: set: 1 l: 4 pow^2: 0.139321
shsyn: set 1: total power^2 per lmax: 0.0268946
shsyn: using FFT block method, lat= 90
shsyn: my_gmt_write_grd: written to bin/x86_64/tmp.grd
shsyn: Done.

Works fine. If I write to a different disk, mounted on NSF, I can access it

touch /disk/twb3/citcom/fulltest/Ra1e8Pl1e6_ddy_151e7_10_244e9_e4_129xyz_25h/63000/div.grd

but when I use the GMT code, I cannot write to that file, it seems
> zcat /disk/twb3/citcom/fulltest/Ra1e8Pl1e6_ddy_151e7_10_244e9_e4_129xyz_25h/63000/v.ab.gz | abconvert 0 22 1 4 | shsyn 5 1 /disk/twb3/citcom/fulltest/Ra1e8Pl1e6_ddy_151e7_10_244e9_e4_129xyz_25h/63000/div
abconvert: reading poltor vector AB AB format in physical convention, lmax: 255
shsyn: sum_mode: 1 and out_mode: 1
abconvert: compute divergence, switching to pol out
abconvert: output of poloidal part of expansion
shsyn: readjusting nlon, nlat and xrange for FFT
shsyn: lmax=255 nlon=513 nlat=256
shsyn: E=0 dx=0.703125 W=360 S=-90 dy=0.705882 N=90
shsyn: constructing lookup tables (Legendre functions) …
shsyn: using 64.25 MB for P array, limit is set to 4200
shsyn: read 1 sets of coefficients with lmax=255
shsyn: set: 1 l: 0 pow^2: 0
shsyn: set: 1 l: 1 pow^2: 0.517226
shsyn: set: 1 l: 2 pow^2: 0.269922
shsyn: set: 1 l: 3 pow^2: 0.553164
shsyn: set: 1 l: 4 pow^2: 0.139321
shsyn: set 1: total power^2 per lmax: 0.0268946
shsyn: using FFT block method, lat= 90
shsyn (gmtapi_export_grid): Could not find file [/disk/twb3/citcom/fulltest/Ra1e8Pl1e6_ddy_151e7_10_244e9_e4_129xyz_25h/63000/div.grd]
[Session shsyn (0)]: Error returned from GMT API: GMT_GRID_WRITE_ERROR (19)
[Session shsyn (0)]: Error returned from GMT API: GMT_GRID_WRITE_ERROR (19)
[Session shsyn (0)]: Error returned from GMT API: GMT_GRID_WRITE_ERROR (19)
my_gmt_write_grd: error writing grid, code 19

This is using the current GitHub GMT and seemingly worked fine just a few days ago with a slightly older GMT GitHub version. I realize that this looks like a problem on our end, and I could generate a more portable example, but I wonder if this might ring a bell?

I’ve also seen a bunch of Directory gmt.history is not locked for exclusive access. Multiple gmt processes running at once could corrupt history file. warning (though not for the runs above).

Any ideas? Some sort of file locking switch I should use during compile?

Thanks much in advance!

T

2

The only recent change that (no idea why it would interfere) touched in more deeper guts was the commit that has this hash 04c9181a64beaacb46453f09ee58c06300bfcba2 merged on the 11th. If you can go back to the sate at July 9 (I only know how to that on Windows), that would be a test to if that commit is the source of the problem or not.

3

Thank you for your quick answer. Reverting to a branch earlier than that commit seems to have fixed the problem - which is surprising, and somewhat suspicious. I still suspect something going wrong on my side, but at face value, this seems to have been the issue.

4

Wonder what this means? Basically, GMT relies on system functions to see if files exist etc.

5

There should be no issue to actually open and write to the file, I can write to it from the system (see the touch command), and copy to it. The same call works on a local file, and it used to work earlier.

Pretty strange.

6

Unclear what is going on, but it does seem like this is some issue on our side with NFS. Unclear to me why the GMT API netcdf write is seemingly the only call that triggers this issue and why it at least seemingly started with the recent Git update, but probably something correlated but not causally.

7

Can you post what gmt function call that fails in your code. and what args are passed?

Also run with -Vd for debug messages

8

The call is to
err = GMT_Write_Data (*API, GMT_IS_GRID, GMT_IS_FILE,
GMT_IS_SURFACE, GMT_CONTAINER_AND_DATA, NULL, grdfilename, G);
I’ll try to reproduce with -Vd