Found that if you have sharp edges in a depth map, sometimes the 3d mesh has gaps in it. Also found that if you add a small blur to the depth map, the depth blurs a bit but a lot of the gaps get resolved.