COLAB example on github "Try PyGMT online" not working any more

andrebelem · June 13, 2021, 10:26pm

Hello folks,
I’m showing to my students how to work with pygmt on colab but the example on https://colab.research.google.com/github/GenericMappingTools/try-gmt/blob/master/python-demo.ipynb is not working anymore.

After installing Miniconda3, adjust sys.path and os.environ, when loading import pygmt I get:
DistributionNotFound: The ‘pygmt’ distribution was not found and is required by the application

I can see the whole package on /usr/local/lib/python3.7/site-packages/pygmt and my sys.path shows /usr/local/lib/python3.7/site-package correctly.

What I’m doing wrong ?

weiji14 · June 14, 2021, 10:34am

Hi @andrebelem, we are aware that the Colab example hasn’t been working for a few months now. See https://github.com/GenericMappingTools/try-gmt/issues/17 and Installing PyGMT on Google Colab (oh wait, I just realized you were the one who posted the error). I did try to debug this last week when GMT 6.2.0 came out by trying different Python versions and so on but couldn’t get things to work either!

In the meantime, I’d suggest using the Pangeo binder links instead if possible. If you do manage to get things to work on collab, we’d be glad if you can share the solution with us!

andrebelem · June 14, 2021, 12:17pm

Hi @weiji14.
It’s really weird because miniconda3 is correctly installed, as is python3.8 (so much so that I can start any command without any problems). I did a quick debug here and the problem shows up in the transaction between pkg_resources and _init.py (which is originally from python 3.7 !). Specifically the error is when init tries get_distribution(“pygmt”).version and gets scrambled between site-packages (3.8) and dist-packages (3.7).
Essentially this is a problem of running 2 versions of python on the same machine, and I’m trying to solve with other recipes for similar problems.

An important detail - my insistence on collab is that many of my students are just starting out in python and some have terrible computers. So the practical teaching solution is colab, for the convenience of them being able to run GMT examples without worrying about learning a lot of python. Pangeo is also very good! but I haven’t tested my class notebooks enough for an assessment. I am doing this now.

If I get a solution, I’ll post it here. Let’s leave the ticket open then.

cheers

andrebelem · June 14, 2021, 2:00pm

Just an ugdate: I did a slightly different test this time:
(1) miniconda installed
(2) !conda install gmt -c conda-forge -y (to gmt source)
(3) !conda install pygmt --channel conda-forge -y (for pygmt)

Just in case, a “!find /usr/local/lib | grep pygmt” shows pygmt correctly in /usr/local/lib/python3.8/site-packages. Just to be sure again, I did “imports;
sys.path.append(’/usr/local/lib/python3.8/site-packages/pygmt’)”. All good.

!gmt --version shows correctly the gmt installed

To test in pygmt core, I used:
“os.chdir(”/usr/local/lib/python3.8/site-packages/pygmt")
from init import *"

The error is definitely in init.py where
version = f’v{get_distribution(“pygmt”).version}’ returns a “dist” equal to “none” because it uses pkg_resources/init.py from python3.7. Even giving a symbolic link between the two repositories (3.7 and 3.8), the pygmt module cannot move forward because there is a “dist” restriction on init.py.

On the other hand, I tested another option which was to install gmt on colab (with a normal sudo apt install), and a simple “pip install pygmt” after… all good “but” - there is still no gmt6 version for apt installation and with that pygmt (which requires gmt6) does not work! The solution would be to compile from scratch inside collab but this extrapolates the “easy” flag I’m looking for my students.

Anyway, it is necessary to wait for the thing to evolve (or the colab or the gmt) so that they can talk OR we forget all this and each one lives in their “own little world”.

Just a detail answering why I’m so focused on colab and GMT: a map in lambert conic projection at high latitudes, “only and exclusively” in GMT is what is presentable. Python’s other competing solutions are horrible. Anyway, I’m just letting off steam.

At some point and with more time I will continue with other alternatives.

weiji14 · June 15, 2021, 6:45pm

Thanks for the detailed writeup! You’re right that there’s something to do with the different Python versions (3.7 and 3.8), and I can totally understand your frustration. Not to get your hopes up, but have you tried conda install gmt followed by pip install pygmt? I haven’t tested it myself but wonder if this might be a possible workaround.

Another thing you raised is that colab may have a problem with Python’s setuptools-scm used by PyGMT. That get_distribution("pygmt").version chunk of code was added in https://github.com/GenericMappingTools/pygmt/pull/695 in Nov 2020, and could be one reason why things are failing. Not easy for us to revert to non-setuptools-scm though, or debug this on colab (so I’m hoping this isn’t the actual problem).

andrebelem · June 16, 2021, 11:42am

Hello @weiji14,
yes. I tested conda install gmt followed by pip install pygmt. In fact, I was able to install everything using the python 3.7 version but the best approach was to install gmt using apt-get and THEN pip install pygmt. However, the problem then is with gmt5 version (via apt) and missing important blocks in pygmt, which crashes.

if I got it right, via apt install only gmt5 is possible for colab and pygmt needs gmt6.
The solution that seems the most viable and simple to me is (1) to create a repository for the precompiled gmt6 version, (2) install via apt, then (3) pip install pygmt and (4) correctly point out where the libraries are. I’m looking at how to create a repository to make it easier for my students to install. (I haven’t given up yet).

Cheers
PS: I studied what you told me and setuptools_scm is really an excellent tool. I looked for some alternatives on how to use it with different versions of python co-installed but that’s exactly what it prevents. So changing this is not the correct way (but detail: pygmt is correctly installed on python 3.8 but somehow it asks for python 3.7 settings and breaks).

andrebelem · June 16, 2021, 2:39pm

Worked !
The sequence was:
(1) update VM colab
!sudo apt update
!sudo apt upgrade -y
!sudo apt install -y build-essential cmake libcurl4-gnutls-dev libnetcdf-dev gdal-bin libgdal-dev libfftw3-dev libpcre3-dev liblapack-dev libblas-dev libglib2.0-dev ghostscript ghostscript-x graphicsmagick ffmpeg xdg-utils
(2) Install gmt from github
!git clone --depth 50 https://github.com/GenericMappingTools/gmt
(3) make everthing !
!cmake /content/gmt
(4) install
!cmake --build . --target install
(5) Install pygmt
!pip install pygmt
(6) use and enjoy

import pygmt
fig = pygmt.Figure()
pygmt.config(MAP_FRAME_TYPE="plain")
pygmt.config(FORMAT_GEO_MAP="DF")
fig.coast(
    shorelines="1/0.5p",
    region=[-100, -20, -70, -45],
    projection="L-60/-60/-70/-45/12c",
    land="gray",
    borders=["1/thick,black", "2/thin,black"],
    frame=['a15f5g15','WNsE']
)
fig.show()

(update - a more colorfull snapshot)
Captura de tela 2021-06-16 124517

weiji14 · June 19, 2021, 12:02pm

Wow, thanks for preservering with this @andrebelem! It seems a bit overkill to install from source though. I see using !cat /etc/*-release that Google Colab is on Ubuntu 18.04 Bionic Beaver which has GMT 5.4.3. Maybe try pulling GMT 6.1.1 from Ubuntu 21.04 Hirsute Hippo instaed? That would save on a lot of time downloading and compiling the GMT source.

Building on your answer, and taking a hint from https://askubuntu.com/questions/103320/install-packages-from-newer-release-without-building-apt-pinning, I tried this:

!echo "deb http://archive.ubuntu.com/ubuntu hirsute main restricted universe multiverse" >> /etc/apt/sources.list
!echo $'Package: *\nPin: release o=hirsute\nPin-Priority: 990' > /etc/apt/preferences
!sudo apt-get update
!sudo apt install -t hirsute gmt libgmt6 libnetcdf18 libgdal28 gdal-data libogdi4.1 libhdf5-hl-100 pip
!pip install pygmt

but things seem to crash on import pygmt and/or when I start to restart the runtime. Perhaps you can do some more magic with the above

Oh, and do feel free to submit a pull request at https://github.com/GenericMappingTools/try-gmt to update the python-demo.ipynb file. I’m sure many people will appreciate it!

andrebelem · June 19, 2021, 11:13pm

Hello @weiji14.

I’ve tried a few things today (including focal fossa’s GMT6.1.0 and Hisrut’s GMT6.1.1) but really a lot of packages get broken, mainly in the link to system dynamic libraries in addition to gmt. It’s really not a good strategy. The farther away from the base of the Bionic 18.04, the worse it gets.

Today I tried to reduce the problem to the focal with:
!sudo add-apt-repository "deb http://ubuntu.mirrors.tds.net/ubuntu focal main universe"
!sudo apt-get install libhdf5-103
!sudo apt-get install libnetcdf15
!sudo apt install gmt

This strategy worked and gmt6 was correctly installed, but conflicts appear with several python libraries and it was also a dead end, especially considering that the whole process takes 5 full minutes. If you wait 5, you can wait 8 minutes for the “build from scracht” solution.

I think at some point google will update the VM to 20.04 which is also an LTS version. We can only wait. Meanwhile, the solution I put on github works, and today was the exact time to (1) run it, (2) go to the kitchen to prepare coffee and (3) come back (8 minutes sharp).

This week I’m going to pass this recipe on to some student “newbies on colab” and see how they react with just a few instructions. If there are no major problems, this is the simplest solution, especially since it installs everything and compiles without asking the user anything - which is desirable in these cases.

Best wishes

danielchaco · August 27, 2021, 11:26pm

@andrebelem thanks for sharing the solution. I am new on this, but running the code I got a warning message:
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

So PyGMT couldn’t be installed in my case, should I try in a venv ?

Thanks,

Daniel M Chacón

seisman · December 31, 2022, 5:49am

For anyone who reads this post:

Currently, the best and simplest solution is to install conda using condacolab.

Install conda

!pip install -q condacolab
import condacolab
condacolab.install()

Install PyGMT and its depenecies
```
!mamba install pygmt
```

Please refer to https://github.com/conda-incubator/condacolab and https://github.com/GenericMappingTools/try-gmt/blob/master/python-demo.ipynb for the detailed instructions.