I like to have those files set to 000 at least listed, just like Akeeba Backup does during a backup attempt.
Overall, dealing with this is a bit tricky. Making them readable often isn’t possible, the web host probably tells you otherwise, and on top of that, some needed files—like the "functions.php" in WordPress—get locked. Just blindly deleting them isn’t really the best move either.