General Feedback and Critiques of Code and Plots:

Hi all,

I am hoping that I could get some feedback both for my code and for the plots they’ve generated. If this isn’t the correct forum for this type of post, please let me know where a better suited place would be.

For some context, these plots are showing residual GPS values. This means, for those unfamiliar, that certain loading forces have been removed from this model, such as non-tidal atmospheric and oceanic loading, seasonal loading (snow for example), hydrological loading (large bodies of (non oceanic) water that is controlled for seasonality), among others.

As for the code, I am really interested in learning how to streamline and create more efficient scripts. I’d consider myself to be a novice still, and would love to continue to learn how to be more efficient in my implementation.

For my plots, there is a little less wiggle-room for change, since my grad advisor requested the shown attributed, but I am happy to make minor changes as long as the data being presented remains easy to read.

Here is my code, and I’m happy to provide some of my data if anyone is interested:

using DelimitedFiles, Glob, GMT

# paths to various directories
imgpath = "/home/rob/Pictures/gmtpng/stack_plot/";
twosig = "/home/rob/Documents/gmt_jl/twosig_ten/";
values = "/home/rob/Documents/gmt_jl/value_match/";
lonlat = gmtread("lonlat.txt");

## Function to read each file in as a GMT dataset ##
function gmt_read(i)
    Sepoch = gmtread(i[1])
    Vepoch = gmtread(i[2])
    return Sepoch, Vepoch
end


for i in zip((glob("*.txt", twosig)), (glob("*.txt", values)))
    
    Sepoch, Vepoch = gmt_read(i)

    C = GMT.makecpt(cmap=:vik, 
        range=(-4, 4, 0.1)
    );

    G1 = GMT.gridit(Sepoch, mask = 0.3, inc = 0.1);
    G2 = GMT.gridit(Vepoch, mask = 0.3, inc = 0.1);

    base = basename(i[1])
    yr = string(base[1:8])


    GMT.subplot(grid = "2x1", savefig = imgpath * "$yr" * "_stack.png",); 
        # gridding image for 2sigma values
        GMT.grdimage(G1, region = [-130 -70 24 51],
            proj = (name = :lambertConic, center = [-100 35], parallels = [33 45]),
            cmap = C,
            colorbar = (pos = (outside = true, anchor = :MR, offset = (0, .9)), xlabel = "mm"),
            coast = (borders = ((type = 1, pen = ("thick", "black")), (type = 2, pen = ("thinner", "lightgrey"))),
                area = 500,
                shore = (:thinnest, :lightgrey)),
            panel = (1,1),
            title = "Vertical Residual greater than 2sigma | $yr",
        );

        # gridding image for values
        GMT.grdimage(G2, region = [-130 -70 24 51],
            proj = (name = :lambertConic, center = [-100 35], parallels = [33 45]),
            cmap = C,
            colorbar = (pos = (outside = true, anchor = :MR, offset = (0, .9)), xlabel = "mm"),
            coast = (borders = ((type = 1, pen = ("thick", "black")), (type = 2, pen = ("thinner", "lightgrey"))),
                area = 500,
                shore = (:thinnest, :lightgrey)),
            panel = (2,1),
            title = "Vertical Residual | $yr",
        );

            
        GMT.scatter!(lonlat, 
            marker = :c,
            markersize = 0.075,
            mc = "black",
            #figname = imgpath * "$yr" *" _stacked.png",
            panel = (1,1),
            # show = true 
        );

        GMT.scatter!(lonlat, 
            marker = :c,
            markersize = 0.075,
            mc = "black",
            #figname = imgpath * "$yr" *" _stacked.png",
            panel = (2,1),
            # show = true
        );

subplot(:end)

end

Which produced plots that look like this (I’ll post 3 here, but this loops created 1,180 plots):

As of this post, the 2002.127 is the most recent plot, and I think it will take about 3-hours for Julia to create and save all my plots up to 2005.998.

Happy to clarify any confusion and answer any questions (to the best of my ability).
Thank you for any feedback and considerations!

  • Rob

Hi Rob.

This is right place to post this.

What is the problem with your script that you want to create “more efficient scripts”? Do you want it to be faster? I am not an expert for this but for my scripts (in bash) I try to parallelize it (with gmt batch) and to write the data in binary format.

For the map, I think I would like to see the date in a another format (maybe year, month, day, or year and julian date).

1 Like

There isn’t a problem (my code is currently running and creating/saving the plots), but, I’m sure there are tips/tricks that would improve my code either syntactically, or make it “easier” to read.

I agree, that putting a yyyy/mm/dd format would be easier to read, and I think I’ll implement that in the future. Right now, since my data was provided in this decimal year format, it was quicker for me to just use the first 8 characters of the filenames rather than calculate which month and day they correspond to.

My next step is to create an animation of these plots, which, to my understanding can only be done in a shell script (bash in my case), so, perhaps learning how to write the data and compile it into binary may prove useful!

My next step is to create an animation of these plots, which, to my understanding can only be done in a shell script (bash in my case), so, perhaps learning how to write the data and compile it into binary may prove useful!

If you have all the plots, then you can make an animation with ffmpeg directly (which is what GMT movie does). There is no need to create a bash script.

1 Like

Rob,

There is nothing wrong in your commands. Right, no need to preface the commands with GMT. because they are exported but that doesn’t make any difference then than visual.

If anything from the aesthetic side I would recommend is to clip your grid to not have data over oceans. But I notice that some points are falling outside the grid image. Why’s that? Weren’t they used in the grid interpolation?

Three hours to create plots from 2002 to 2005 seems rather long. Have checked the time it takes to create each one and where most time is spent? My guess is that the grid interpolation will be culprit. Note that you can try other interpolation methods and other grid spacing’s to speed up this step without compromising the figures. Also note that being your scripts run under the modern mode, you can launch several Julia instances and run parallel scripts in them.

Very shortly a new version GMT.jl (1.16.0) will land in the servers ready to update. With it, you can put the > again in the title (under the hood the > is replaced by another that look the same but does not interfere with the shell).

And, you can easily compute dates from decimal years with

julia> doy2date(round(Int,0.021*365), 2001)
2001-01-08

Originally, I did have the values clipped over the ocean, but I was told not to do that for these interpolations. The explanation given to me is that these values I’ve generated “don’t care” if they’re over the ocean or land.

Three hours to create plots from 2002 to 2005 seems rather long

I agree, I’ll mess around with the interpolations settings and see if I can get that time down. It was taking between 4-8ish seconds for each plot to generate, and I have 1,180 plots here. If I can get it down to 1-2 seconds per plot, that would be great.

scripts run under the modern mode, you can launch several Julia instances and run parallel scripts in them.

I’m sorry, I’m not sure what this means. Like, create a separate .jl script for each block of code, and call each on in a main script??

This is great, thank you!

As always Joaquim, you’re wonderfully helpful, and I always appreciate your feedback!

I have not used FFMPEG before, but, I did try my hand at it, and I kept getting the same error message, no matter how I changed my script to try and get it to work.

Do you have any insight as to why this won’t work?

ffmpeg -y -r 30 -i *.png -c:v libx264 -preset slow -crf 22 -c:a copy stacked_anime.mp4

Which produced this error:

[png @ 0x64079fc727c0] ff_frame_thread_encoder_init failed
Error initializing output stream 583:0 -- Error while opening encoder for output stream #583:0 - maybe incorrect parameters such as bit_rate, rate, width or height
Conversion failed!

Which exclusively occurs at that output stream 583:0 and I’m not sure what is happening exactly at this iteration.

All the previous iterations seems to work?

Output #581, image2, to '2003.515_stack.png':
  Metadata:
    encoder         : Lavf58.76.100
  Stream #581:0: Video: png, rgb24(pc, progressive), 2220x2175 [SAR 11811:11811 DAR 148:145], q=2-31, 200 kb/s, 30 fps, 30 tbn
    Metadata:
      encoder         : Lavc58.134.100 png
Output #582, image2, to '2003.520_stack.png':
  Metadata:
    encoder         : Lavf58.76.100
  Stream #582:0: Video: png, rgb24(pc, progressive), 2220x2175 [SAR 11811:11811 DAR 148:145], q=2-31, 200 kb/s, 30 fps, 30 tbn
    Metadata:
      encoder         : Lavc58.134.100 png

I’ve looked up trouble-shooting via the FFMPEG documentation, tried Gemini/ChatGPT, and other forums for answers, but any variation spits out the same error. :person_shrugging:

Hi Rob

I’m affraid I don’t have much experience with ffmpeg. I usually take the code used by gmt movie (shown in the terminal).

The code looks like this (take from movie — GMT 6.6.0 documentation):

ffmpeg -loglevel warning -f image2 -framerate 24 -y -i "mydir/myimages_%04d.png" -vcodec libx264 -pix_fmt yuv420p mymovie.mp4

If I were you, I would try to modify the above command.

For example, I download your 3 figures, saved them in a directory called “Figures” and renamed the files to stack_0001.png, stack_0002.png, stack_0003.png. I try to make a movie with this command:


ffmpeg -loglevel warning -f image2 -framerate 24 -y -i "Figures/stack_%04d.png" -vcodec libx264 -pix_fmt yuv420p mymovie.mp4

I got the following error because the figures have a “height not divisible by 2 (2220x2175)”. I would need to delete one row of pixles and try again.

[libx264 @ 0x564b96121e40] height not divisible by 2 (2220x2175)
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height

I haven’t yet implemented the gmt movie module yet and I will see if I run into the same issue. I don’t know if the height figure would change due to compression from the upload or not. I’ll update when I get to it, hopefully, today.

They sure don’t. Only the readers might, but that’s my own view and if were advised not to do it, it makes it even easier.

I mean to break your main script, that I imagine lives inside a loop, in some pieces (depends on you computer capacity) and run each piece in a different Julia REPL instance. Never tried this but (in principle) it should work.

There is no need to see gmt movie. The good about gmt movie is that it creates the ffmpeg command (like the one a share) to create a video file from the images. It even takes care that the images have even number of pixels.

I managed to get the video (after I crop the image by one pixel).

I ended up changing the interpolation from:

    G1 = GMT.gridit(Sepoch, mask = 0.3, inc = 0.1);
    G2 = GMT.gridit(Vepoch, mask = 0.3, inc = 0.1);

and setting inc = 0.1 to inc = 0.2 which decreased the time by about 50%. If I used a value greater than inc = 0.2 Julia would give me the following warning:

surface [WARNING]: 12755 unusable points were supplied; these will be ignored.

So, I just stuck with inc = 0.2.

Sweet! I’m running through my loop again to make the plots. I made a few changes, like a custom :roma cpt, as well as changing the filter so that I am no longer excluding any time in the plots.

This did increase the amount of plots from 1,180 to 1,826, but the positive out of this is there won’t be any major gaps in the data.

But with the more efficient interpolation (even if I’m losing a little bit of resolution), and the faster plotting/saving of the figures, this should be done in about an hour. Then I’ll be able to try to modify the ffmpeg code your provided above.

You should set a region to your grid and not let it be guessed by the data distribution, inc, etc.

I was going to try and update the original post, but, doesn’t look like I am able to do that. Guess there is a time limit on being able to edit posts.

But, after having to crop every image by 1 pixel lol, thanks to the help of Estaban, the animation itself came together fairly quickly.

I wish it was smoother, but, that has more to do with the data than it does with the animation itself.

Can’t upload mp4 files here, so feel free to view it here on Youtube.