Cosmology and the galaxy-matter connection using weak gravitational lensing cross-correlations in the Dark Energy Survey