How to create digital elevation model from XYZ data *.txt data and module "gmt mask"

Not documented, but try doing pygmt.surface(..., M="2c", ...) to mask out areas 2 cells away from the data points. See https://docs.generic-mapping-tools.org/latest/surface.html#m