Conda GMT build is unusable in Julia-Windows

I know this is not the right place to report Conda issues but I’m not sure where to do it and perhaps our Py guys can redirect this to its due place.

The issue is that somehow the Conda build is screwing somewhere and in a mysterious way. Although the binaries work, they cannot be used from Julia. See, working from the Conda bin dir

julia> pwd()
"C:\\programs\\Miniconda3\\Library\\bin"

now, try a GMT, GDAL and PROJ4 dlls. Only the latter works

julia> ccall((:GMT_Create_Session, "gmt"), Ptr{Cvoid}, (Ptr{UInt8}, UInt32, UInt32, Ptr{Cvoid}), "GMT", 2, 0, C_NULL)
ERROR: could not load library "gmt"
The specified procedure could not be found.

julia> ccall((:GDALVersionInfo, "gdal302"), Cstring, (Cstring,), "--version")
ERROR: could not load library "gdal302"
The specified procedure could not be found.

julia> ccall((:proj_create, "proj_8_0"), Ptr{Cvoid}, (Ptr{Cvoid}, Cstring), C_NULL, "+proj=longlat")
Ptr{Nothing} @0x000000006faaec40

For checking let’s repeat the above but using our own GMT build

julia> ccall((:GMT_Create_Session, "gmt_w64"), Ptr{Cvoid}, (Ptr{UInt8}, UInt32, UInt32, Ptr{Cvoid}), "GMT", 2, 0, C_NULL)
Ptr{Nothing} @0x000000007e462380

julia> ccall((:GDALVersionInfo, "gdal_w64"), Cstring, (Cstring,), "--version")
Cstring(0x000000007e3ea0b0)

julia> ccall((:proj_create, "proj_w64"), Ptr{Cvoid}, (Ptr{Cvoid}, Cstring), C_NULL, "+proj=longlat")
Ptr{Nothing} @0x000000007e49d4c0

All three commands worked well.
I have now quite some experience tracking the very misleading error message The specified procedure could not be found but after quite some time investigating with the DependencyWalker I could not find the failing reason, but the point is: those binaries are useless for Julia (and very likely all other FFI interfaces) usage on Windows.

Strange that both GMT and GDAL isn’t working. The conda build recipe for GMT and GDAL on Windows is at https://github.com/conda-forge/gmt-feedstock/blob/master/recipe/bld.bat and https://github.com/conda-forge/gdal-feedstock/blob/master/recipe/set_bld_opts.bat respectively, not sure if you can spot anything that is wrong there. My next guess is that it is something to do with the %PATH% environment variable, could you print the %PATH% for the conda install and GMT build install perhaps?

Hi Weiji

No the problem should not be the PATH. See that I run those commands from within the directory where conda puts the binaries+dlls

"C:\\programs\\Miniconda3\\Library\\bin"

I don’t see anything wrong in the building recipes and it’s certainly strange that the executables run but calling the dlls fail. The build is done by VCPKG? I’ve seen it doing crazy things in building GDAL (tons of Boost shits)

BTW, another thing. Where does conda stores the GMT binaries on Linux? I’m trying to mimic what I see on Windows but the executable isn’t apparently there (never mind the .julia/conda/3 part, that’s the Conda/Julia location)

ERROR: LoadError: IOError: could not spawn `/home/runner/.julia/conda/3/Library/bin/gmt --show-library`: no such file or directory (ENOENT)

Just double checking that you ran conda activate myenvname (see Managing environments — conda 24.1.3.dev32 documentation)?

Mine is under /home/username/miniconda3/envs/pygmt/bin. There should be a $CONDA_PREFIX environment variable which points to the first part of that path (/home/username/miniconda3/envs/pygmt) you can use (once the conda env is activated) that points to the right place.

Who takes care of building GMT for Conda? Are you guys? It seems to be on different versions for different distros. While it worked in the Ubuntu20 used by the CI, it errors on a CentOS 8 with

[jluis@fct-gmt bin]$ ldd ./gmt
...
        libnsl.so.1 => not found
        librttopo.so.1 => /home/jluis/.julia/conda/3/bin/./../lib/./././librttopo.so.1 (0x00007f23368f1000)
        libcharset.so.1 => /home/jluis/.julia/conda/3/bin/./../lib/./././libcharset.so.1 (0x00007f23368ec000)
        libquadmath.so.0 => /home/jluis/.julia/conda/3/bin/./../lib/././libquadmath.so.0 (0x00007f23368b2000)

Another error that I get, this time in Ubuntu, is

PROJ4: Error During Test at /home/runner/work/GMT.jl/GMT.jl/test/test_proj4.jl:3
  Got exception outside of a @test
  could not load symbol "pj_get_spheroid_defn":
  /home/runner/.julia/conda/3/lib/././libproj.so.22: undefined symbol: pj_get_spheroid_defn

but the same test works fine when the PROJ4 lib was installed with apt-get

The builds are semi-automated. I’d ask @seisman or @leouieda who are maintainers of the repo, I just contribute to the conda gmt-feedstock once in a while.

The Linux build script is at gmt-feedstock/recipe/build.sh at main · conda-forge/gmt-feedstock · GitHub. I don’t see any mention of libnsl or libproj in that script, so maybe we need to add it into the build flag somehow?

Looking under gmt-feedstock/recipe/meta.yaml at main · conda-forge/gmt-feedstock · GitHub, PROJ isn’t actually listed as a dependency (perhaps it should be) so that could be why it isn’t being picked up. Could you please provide the contents of the test_proj4 script (preferably a GMT C equivalent), and we’ll try to add it to the test suite while fixing this issue.

libproj is a GDAL dependency (one can’t build GDAL nowadays without PROJ) but the libnsl is new to me. It’s certainly a dependency of a dependency but I’ve never seen it before.

Regarding test_proj4, it’s a Julia test that exercises wrapping libproj directly from Julia. Don’t know how to make it in other languages but it’s quite strange that it fails with that error message that means the mentioned symbol is not in the proj shared lib, which is quite suspicious.

Actually now that you mentioned it, running conda list shows proj=8.0.0 installed in my gmt conda environment (which was probably pulled in by GDAL, I have gdal=3.2.2 installed). What versions do you have installed? Not sure what to do about the undefined symbol though, maybe check the conda PROJ feedstock build scripts at GitHub - conda-forge/proj.4-feedstock: A conda-smithy repository for proj.4..

The problem to investigate this is that it happens in the CI machine and Conda doesn’t seem to install the same versions on different distros and the number of problems don’t stop and I can’t go to them all. Now is Mac that decides that otool -L gmtlib.dylib on the Conda build lib returns @rpath/libname instead of the true full name of the lib.

Unix is nuts

Apparently an error that many others have suffered too. Seems that some libc distribute it others not …

But if I do (like they say in the above link)

sudo dnf install libnsl

next other libs in the Conda install are broken due to a GlibcXXX incompatibility.

Maybe we should think about including Proj as a GMT dependency on conda-forge and setting the proper path in the cmake build. Optional dependencies are difficult with compiled code like this and GMT may be finding the system Proj instead of the conda Proj.

@Joaquim I think conda-forge ships it’s own libc specifically to avoid the incompatibility errors. So mixing system libraries with conda-forge libraries will likely not work unless you use the same libc and gcc used for the conda-forge builds.

I have no idea what libnsl is but it’s not on conda-forge. I wonder if it’s a new dependency of Proj?