When layer elements are selected in the Map View, using the Invert selection tool you can select the unselected layer elements, and unselect the selected ones.

Before:

After: