The extractor for BroadwayCloudPubSub v2.
A Pub/Sub message names its consume edge by its subscription, and v2 stamps it
as metadata.subscription together with metadata.attributes,
metadata.orderingKey and metadata.receivedAt, which is what a checkpoint
needs to name the edge. A message that carries no subscription - the shape
BroadwayCloudPubSub v1 produces - records no consume checkpoint, because
there is no name to report its edge under.
Consume checkpoints therefore require broadway_cloud_pub_sub 2.0.0-rc or
newer, tested with 2.0.0-rc.3, the first to stamp the receive time as
metadata.receivedAt, a %DateTime{}. On stable 1.0.0 the v1 metadata has no
:subscription, so consume/1 returns :error for every message and a v1
pipeline records nothing.
Both edges are reported as type:google-pubsub, the technology Datadog files
these streams under. Pub/Sub names its subscriptions and topics as full
resource names (projects/<project>/subscriptions/<id>); this extractor
reduces each to its short id, which is what dd-trace-java reports for Pub/Sub
(both edges). dd-trace-py reports the full resource path of the subscription
under topic: instead, so a Py consumer appears as a different node; the
pathway still connects, because the link between two nodes is the hash
carried on the message. An already-short name is reported as it is.
A Pub/Sub message metadata carries only its subscription, so the topic is read
from the dd-topic attribute the library's own producer writes next to the
pathway header. When that attribute is present the consume edge reports both
subscription:<subscription> and topic:<topic>, because Datadog derives the
queues in the middle of a pathway - data_streams.latency's visited_queues
and detailed_pathway - from the topic tag; with only a subscription the
queue between two services is missing. A message from a producer that did not
write the attribute, or from outside this library, keeps the subscription-only
tags: nothing is invented from the subscription's name, because naming is the
application's convention.
The publish time Pub/Sub stamps on a message - metadata.publishTime, a
%DateTime{} in v2 - starts the consume checkpoint of a message that starts a
pathway, so a pipeline that only consumes reports how long the message waited
in the bus instead of zero. A message that carries a pathway header keeps the
start its producer wrote, because a first-hop start is only offered to a
pathway that has not started. A publish time that is absent, is not a
%DateTime{}, or falls outside the window a clock could plausibly have
stamped is dropped rather than believed.