It is possible to fall down, clip the side of the middle platform slowing you down, running towards the platform with the hole then wall jumping up, but it is very dangerous. It is recommended to wait until one of the shamans builds.
The hole that belongs to you is the one opposite to the side that you spawn on. A simple solution is spawning a red-anchored plank on either platform and building to one of the holes. If you are on Divine mode, build an invisible bridge across the pit and build to either hole.