Vendor relationships begin and end with contractual obligations.
Therefore, your service level agreements (SLAs) act as a primary starting point for measuring vendor performance. If you include specific metrics as part of your SLAs, you can measure how effective your vendor is in maintaining a secure environment.
Some questions to consider include:
How quickly do they resolve operational and administrative failures?
How often is the system unavailable?
How many times have they been breached?
How often do they update their product?
Do they incorporate continuous cyber security monitoring of their own environment and ecosystem?