Memory and nonlocal effects in heat transport : from diffusive to ballistic regimes