Apologies for the inconvenience.
Unfortunately, it is not possible to manually set a different default color for the site when using the predefined presets. The system always loads the first color preset in the list as the default color on the frontend.
If the following setting in SP Page Builder is disabled:
https://prnt.sc/xRDmsyYtV-58
then the site will automatically display the first color preset from the preset list.
You can manage or modify the available color presets from:
SP Page Builder → Settings → Color Library
https://prnt.sc/pDJgQ9M4kIcm
From there, you can edit the existing presets or remove the ones you do not want so that only your preferred colors appear in the list.
Regarding the color switcher panel on the right side, it is controlled by the presets available in the Color Library. If it is not appearing, please ensure the color preset feature is enabled and that presets are properly configured in the Color Library.