MODEL DDL statements and ML functions in Spanner invoke Vertex AI endpoints and can fail due to various reasons:
| Error Code | Error Message | Possible cause | Possible solution |
|---|---|---|---|
|
CANCELLED
|
Call to Vertex AI endpoint {ENDPOINT} was cancelled due to query cancellation. | Query was cancelled by the client application. | Investigate why your client cancelled the query. |
|
DEADLINE_EXCEEDED
|
Vertex AI endpoint {ENDPOINT} exceeded the call deadline. | Query deadline is too short. | Increase query deadline on the client. |
|
-
|
- | Endpoint was too busy. | See Vertex AI monitoring and deploy more nodes. |
|
FAILED_PRECONDITION
|
Vertex AI endpoint {ENDPOINT} returned failed precondition error. | Endpoint has no models deployed. | Deploy models to the endpoint. |
|
INTERNAL
|
Unknown error when accessing Vertex AI endpoint {ENDPOINT}. | Unexpected internal error. | Use failover endpoints or open a support ticket. |
|
INVALID_ARGUMENT
|
Invalid request to Vertex AI endpoint {ENDPOINT}. Make sure that Vertex AI endpoint and Spanner model schema match. | Spanner model schema and Vertex AI endpoint schema do not match | Update Spanner model's schema. |
|
NOT_FOUND
|
Vertex AI endpoint {ENDPOINT} not found. | Endpoint was deleted. | Update Spanner model's schema. |
|
PERMISSION_DENIED
|
Access to Vertex AI endpoint {ENDPOINT} was denied. | Spanner service agent does not have permissions to access the endpoint | Grant service agent role permissions |
|
-
|
- | VPC SC error | See Vertex AI error message and follow VPC SC troubleshooting |
|
RESOURCE_EXHAUSTED
|
Vertex AI endpoint {ENDPOINT} quota has been exceeded. | Too many requests to Vertex AI. | Increase online prediction quota |
|
UNAVAILABLE
|
Could not create service agent for project {PROJECT}. | Service Usage API issue. | Create service agent manually |
|
-
|
Vertex AI endpoint {ENDPOINT} is unavailable. | Too many requests to Vertex AI. | Deploy more nodes. |
|
-
|
- | Vertex AI has a regional issue | Use failover endpoints |
