Phantom Lake Camp got it's namesake from the nearby Phantom Lake and Cave. The lake no longer exists, since the spring water from the cave was diverted via canal system for crop irrigation during the late 1930's. But before that time, it was a spring fed natural lake that formed outside the mouth of a limestone cave. The water was crystal clear, and 78 degrees year round, and hosted several species of fish including largemouth bass.

Phantom Lake appeared on maps as early as 1893. (See below)

It was a popular fishing and swimming spot for the local and tourists who stayed at the nearby tourist camp. The water flowed from an underground aquifer, and out of the cave mouth. Swimmers cound go several hundred feet into the cave, and there are several dry caves located nearby that connect to Phantom Cave.