Memory stone is black, and often finely veined with white or silver threading. It was used by the Elderlings to build roads, buildings, cities and Skill-pillar. There is a stone quarry beyond the Mountain Kingdom, but there might be other sources for the stone as well. Along the Rain Wild River there are beaches of memory sand which probably are of the same origin as the memory stone.
Those proficient with the Skill feel their consciousness pulled into the stone but only if the stone has memories in it already (Fitz could not walk along the Skill road as it had been walked many times before by Elderlings, who we are to believe are very proficient in the Skill, and yet when in the quarry of Memory Stone, Fitz felt no pull)