I’ve just come across this problem. It seems to be a bug.
If I open an old site, the Preloader Preview checkbox works as expected. I’ve just added it to a page on a new site and the preview can’t be turned off - as @scott explains.
In case it’s a clue, in the version which works, when the preview checkbox is selected, preview="true" is added to the <dmx-preloader> line; in the problem version, only preview is added. I don’t know if this is relevant, but it seems that something was changed and perhaps the bug was introduced at the same time.
Commenting out the <dmx-preloader> temporarily is probably the simplest workaround.
Perhaps this topic should be changed to a bug report.