Repository navigation
Add structured logs and Prometheus metrics for blobstore and packer operations - #5575
WeiQuan0605 wants to merge 2 commits into
Conversation
…lobstore and packer operations
…invocation Log one structured entry per storage-cli subprocess call on cc.blobstore.storage_cli_client with operation, bucket, resource_type, operation-specific fields (key/source_path/destination_key etc.), duration_seconds, exit_code, and success. Linked to vcap-request-id so all storage-cli calls for a CF API request are traceable in Kibana, comparable to CCDB statement logs.
stephanme
left a comment
stephanme
left a comment
There was a problem hiding this comment.
There are number of unrelated files touched in this PR.
| exit_code = resolve_exit_code(status) | ||
| success = status.success? || (allow_exit_code_three && status.exitstatus == 3) | ||
|
|
||
| logger.info('storage_cli.call.finish', |
There was a problem hiding this comment.
Would be good to have the operation and maybe the important args directly on the log msg. The complete storage-cli cmd line is logged on debug level only and this is ok (don't need every detail on info).
E.g.:
storage_cli.call.finish put local-file.txt remote-object.txt
storage_cli.call.finish exists remote-object.txt
|
|
||
| def capture_cli(command, args, start_time) | ||
| Open3.capture3(@cli_path, '-s', @storage_type, '-c', @config_file, *additional_flags, command, *args) | ||
| rescue StandardError => e |
There was a problem hiding this comment.
Is it ensured that an exceptionally failing storage-cli call gets logged? Maybe it is logged by an exception handler of the caller.
Maybe its safer to log additionally in the rescue.
| DELAYED_JOBS_TABLE = /(?:[`"]?\w+[`"]?\.)?[`"]?delayed_jobs[`"]?/i | ||
| HANDLER_COLUMN = /[`"]?handler[`"]?/i | ||
| DELAYED_JOBS_TABLE = /(?:[`"]?\w+[`"]?\.)?[`"]?delayed_jobs[`"]?(?=\W|$)/i | ||
| HANDLER_COLUMN = /\b[`"]?handler[`"]?/i |
There was a problem hiding this comment.
Why is delayed_jobs_log_redaction.rb changed in this PR?
A short explanation of the proposed change:
Add structured log entries and Prometheus metrics to blobstore and packer operations.
cp_to_blobstorenow emits cp-start, cp-finish, cp-skip, and cp-error log events with destination key, duration, and blob size.cp_r_to_blobstoreemits per-phase timing logs (blobstore.cp_r.digest, .exists, .upload). Three new Prometheus metrics track storage-cli subprocess invocations:cc_storage_cli_operations_total,cc_storage_cli_duration_seconds, andcc_storage_cli_blob_size_bytes.An explanation of the use cases your change solves
Blobstore upload failures are currently silent: operators cannot tell from logs whether a droplet/package upload was attempted, skipped, or how long it took. This change makes every upload observable: structured logs allow filtering by destination key or bucket in Kibana, and Prometheus metrics enable alerting on non-zero exit codes and latency spikes across operations (put, get, exists, delete, etc.).
I have reviewed the contributing guide
I have viewed, signed, and submitted the Contributor License Agreement
I have made this pull request to the
mainbranchI have run all the unit tests using
bundle exec rakeI have run CF Acceptance Tests