Good question! I'd say Proust because honestly when else am I ever going to read that book, but maybe it would be hard to relate with on a desert island. For rereads, let's prioritize books with a robust value for nature, exploration, and the ties of people with their land, and for to-reads, authors that immerse without feeling mannered.

Rereads:
The Once and Future King
Les Miserables
East of Eden

To-Reads:
The Sound and the Fury
The Long Goodbye

There's no way I'd finish more than half of these and still have time for cooking, exploring etc, but I'm allowing myself some extra travel time after I get rescued from the desert island and so forth.