Modular invariants for manifolds with boundary