The devil was possibly a slang term for the garboard seam, hence between the devil and the deep blue sea being an allusion to keel hauling, but a more popular version seems to be the seam between the waterway and the stanchions which would be difficult to get at, requiring a cranked caulking iron, and a restricted swing of the caulking mallet.