For -R you will need to first run gmt info -Itxtfile_contains_boundary_polygon to get a valid -Rstring.
If you only want data inside the polygon then you either have to clip the plot with clip (so only things inside your polygon shows) or you will need to use gmtselect -F to try to contain only points inside the polygon.