Recording Sciagraph report results in OpenTelemetry

If you’re profiling in production (this requires paying for the Team plan), it’s useful to have information about profiling in your logs. You can then search logs for slow jobs, and then used the logged information to find the profiling report.

This integration will allow you to record the job ID, download and decryption keys, and peak memory in your OpenTelemetry tracing logs. As a result, when you find a particular job whose profiling information you want, the information you need to download the profiling report will be available in the backend where you store the logs.

See this tutorial on how to use OpenTelemetry generically to find slow outliers whose profiling you may wish to check.

How to record Sciagraph information in OpenTelemetry

To hook up Sciagraph’s logging to OpenTelemetry you will need to run sciagraph.integrations.opentelemetry.record_sciagraph_job(). Make sure to run it:

1. Once for each job, at the end of your job.
2. Inside the context of an OpenTelemetry span.

For example, if your code currently looks like this:

from opentelemetry import trace

tracer = trace.get_tracer("yourjob")

@tracer.start_as_current_span("yourjob")
def main():
    do_something()
    do_something_else()
    last_thing()

You would change it to look like this:

from opentelemetry import trace
from sciagraph.integrations.opentelemetry import record_sciagraph_job

tracer = trace.get_tracer("yourjob")

@tracer.start_as_current_span("yourjob")
def main():
    do_something()
    do_something_else()
    last_thing()
    record_sciagraph_job()

If you want to record this info even if an exception is raised, use try/finally:

from opentelemetry import trace
from sciagraph.integrations.opentelemetry import record_sciagraph_job

tracer = trace.get_tracer("yourjob")

@tracer.start_as_current_span("yourjob")
def main():
    try:
        do_something()
        do_something_else()
        last_thing()
    finally:
        record_sciagraph_job()

What gets recorded

record_sciagraph_job() records an OpenTelemetry event, specific information about that point in time. It will be recorded with name "sciagraph", and with the following attributes:

• sciagraph.job_id: The job ID.
• sciagraph.download_key: The key for downloading the report.
• sciagraph.decryption_key: The key for decrypting the report.
• sciagraph.instructions: Instructions on how to download the report, intended to be read by humans.
• sciagraph.peak_memory_kb: Peak allocated memory usage, in kibibytes (==1024 bytes), as an integer.

Make sure you run record_sciagraph_job() at the end of your job, otherwise the recorded peak memory may be inaccurate if additional memory is allocated later in the job.