In the maps in the front of the books ( my copies, at least ) "Craster's Keep" is marked. So while you may be an incestuous polygamist whose sons vanish in a sinister fashion, you do have a castle. Although one made of logs and ankle deep in pig shit and rotting turnips, if you read that part.