Generate all public-preview and GA PyDABs resources - #6528
Generate all public-preview and GA PyDABs resources#6528Sankalp-Mittal wants to merge 10 commits into
Conversation
class_name.lower() ran multi-word type names together (e.g. vectorsearchindex). Snake-case the type name instead. No-op for the 6 currently-wired single-word resources. Co-authored-by: Isaac <no-reply@databricks.com>
Extend the experimental marker (previously private-preview only) to public-beta fields and enums via is_experimental_stage. Beta and private preview may still change; GA and public preview are frozen. Co-authored-by: Isaac <no-reply@databricks.com>
Enum values with non-identifier characters (e.g. "amazon-bedrock") produced invalid Python member names. Replace runs of non-alphanumeric characters with an underscore. Co-authored-by: Isaac <no-reply@databricks.com>
Derive the resource set from the Resources struct in the bundle schema instead of a hardcoded allowlist, excluding denylisted resources and those below public preview. Adds 17 resources (23 total). Co-authored-by: Isaac <no-reply@databricks.com>
Integration test reportCommit: 7c96c9c
Top 6 slowest tests (at least 2 minutes):
|
…-tests-skill' into sankalp-mittal/pydabs-generate-all-resources
One acceptance/bundle/python/<plural>-support/ fixture per newly generated resource, satisfying the test_python_support_coverage guard. Each loads an instance from YAML and one from Python and runs a mutator over both. Co-authored-by: Isaac <no-reply@databricks.com>
| default_factory="dict", | ||
| create_func_default="None", | ||
| experimental=prop.stage == LaunchStage.PRIVATE_PREVIEW, | ||
| experimental=is_experimental_stage(prop.stage), |
There was a problem hiding this comment.
lots of repetition here when only a few kwargs are different in the if-elif cases. can we do (as follow-up)
fieldKwargs = dict(
field_name=field_name,
...
)
if field_type.name == "VariableOrDict":
fieldKwargs["default_factory"] = "dict"
elif:
fieldKwargs["another_field"] = "another non-default value"
return GeneratedField(**fieldKwargs)| } | ||
|
|
||
| # Only GA and public-preview resources are generated; later stages may still change. | ||
| _EXCLUDED_RESOURCE_STAGES = {"PUBLIC_BETA", "PRIVATE_PREVIEW"} |
There was a problem hiding this comment.
why the overlap with jsonschema.py's experimental stage? let's use a single source of truth
There was a problem hiding this comment.
Both of them assert different behavior, this will completely remove these resources from being generated, while the jsonschema.py experimental marker is for the fields in resources. That's why I want to keep them different
| node = config.get(type_name, {}) | ||
| if "x-databricks-launch-stage" in node: | ||
| return node["x-databricks-launch-stage"] | ||
| for option in node.get("oneOf", []): |
There was a problem hiding this comment.
this is for an enum iirc, but the function is _resource_stage - are we overloading it here?
There was a problem hiding this comment.
We are checking if the launch-stage is present in the node, and if not we iterate through its possible shapes and if launch-stage is present in any of them we return it. I don't think I understood your original question properly but this is what it does
…-tests-skill' into sankalp-mittal/pydabs-generate-all-resources
…-tests-skill' into sankalp-mittal/pydabs-generate-all-resources
Co-authored-by: Isaac <no-reply@databricks.com>
Sankalp-Mittal
changed the title
Approval status: pending
|
| @@ -81,5 +81,6 @@ def get_code(generated: GeneratedEnum) -> str: | |||
|
|
|||
| def _camel_to_upper_snake(value): | |||
There was a problem hiding this comment.
Fixes issues with changing fields with - to python variables, since they are not allowed
| @@ -61,6 +61,8 @@ class AiRuntimeTask: | |||
|
|
|||
There was a problem hiding this comment.
These are the changes that are referred to in the PR warning that change earlier normal stuff to be marked [EXPERIMENTAL]
| @@ -40,6 +40,8 @@ class IngestionPipelineDefinitionTableSpecificConfigQueryBasedConnectorConfig: | |||
|
|
|||
There was a problem hiding this comment.
Again also potentially controversial change
3 participants
Warning
One thing to keep in mind here is, some resource fields that were not marked as experimental because they were Public Beta, will now be marked as Experimental, since we are marking as experimental both on Public Beta and Private Preview.
Changes
Removes the hardcoded
RESOURCE_NAMESPACEallowlist and derives the PyDABs resource set from the bundle schema'sResourcesstruct, so every eligible resource is generated automatically.Added (17):
apps,clusters,database_catalogs,database_instances,experiments,external_locations,instance_pools,job_runs,model_serving_endpoints,models,quality_monitors,registered_models,secret_scopes,sql_warehouses,synced_database_tables,vector_search_endpoints,vector_search_indexes.(6 were already generated —
jobs,pipelines,catalogs,schemas,volumes,alerts— for 23 total.)Not added:
postgres_projects,postgres_branches,postgres_endpoints,postgres_catalogs,postgres_databases,postgres_roles,postgres_synced_tables.cluster_policies,dashboards,genie_spaces(arbitrary JSON — Gointerface{});secrets(time.Time).Why
Complete resource coverage for PyDABs.
Tests
Testing is handled by the related PRs in the stack (#6516, #6421, #6527): generated per-resource unit tests and the acceptance coverage guard that requires a fixture per resource.