How to convert the DEM in the extension (*.dem) into (*.xyz) file?

Hmm, have you tried just using the same command? I think gmt should be able to read dem-files through gdal.