Semigroups of matrices of intermediate growth