FOG Identifiers (Metric Ids, Submetric Ids, Ping Ids, …)

The Glean Ecosystem identifies metrics by their base identifier, a combination of the metric’s category and name. The Glean SDK further identifies “submetrics” (the metric instances that are created when you call .get(aLabel) on a labeled metric) by a combination of the base identifier and its label.

Glean identifies pings merely by their names.

FOG needs to identify instances of

across foreign function interfaces (to support JS and C++) and processes (for ipc).

For efficiency’s sake, FOG uses numerical IDs. This document explains how those IDs work to identify their objects.

Identifying a FOG Metric

Depending on the language, the process, and how a metric was defined, the way you identify a specific FOG metric instance is different.

Rust: FOG’s MetricId, BaseMetricId, and SubMetricId

To identify a specific FOG metric instance in Rust you will have either a BaseMetricId for metric instances corresponding to definitions in metrics.yaml, or a SubMetricId for submetric metric instances built on-demand when you call get(aLabel) on a labeled metric. FOG supplies enum MetricId which encapsulates whichever is appropriate and supplies methods for discrimination. (See toolkit/components/glean/api/src/private/metric_getter.rs for details.)

Both IDs have the same format:

   3                   2                   1                   0
 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| unused  | | |            FOG metric instance ID               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

  • 32 bits wide (u32)

  • Top 5 bits are unused by Rust, to align with JS.

  • Bit 26 is 1 if the FOG metric instance was created from a definition specified at runtime and thus can be found stored in factory::__jog_metric_maps, or 0 if the FOG metric instance was created from a definition in a metrics.yaml file read at compile time and can be found statically stored in the metrics::__glean_metric_maps

  • Bit 25 is 1 if the FOG metric instance was created in response to a labeled metric having get(aLabel) called on it and thus can be found in metrics::__glean_metric_maps::submetric_maps, 0 otherwise.

    • It is impossible for both bit 25 and bit 26 to be 1 as all submetric instancess are created and stored identically regardless of whether their parent metric instance was defined at compile- or run-time.

  • Lower 25 bits are for identifying a specific FOG metric instance.

Zeroes are Reserved

The values 0, 2**25, 2**26, and 2**25 + 2**26 (0 with every combination of the signal bits set to 0 or 1) are reserved. They may be used to signal “no metric found” or “no metric created” kinds of situations.

As a result of this format there is quite a bit of unreachable space in these ids. We don’t expect more than 10k metric definitions in any one product at any one time, so we’re not (at time of writing) worried.

If we run out of space for defined metrics, the build will fail due to one of several static_asserts.

If we create more than 2**25 - 1 submetrics in a given run of Firefox Desktop, things will go very weird at runtime. But that would require more than 33M submetrics, so we consider this almost impossible as it would require more than 3355 labels for each and every one of the suspected 10k upper limit of defined metrics.

C++: Just a uint32_t, actually

FOG C++ metric instances are the thinnest possible piece of behaviour layered over storage of a single metric id. Their job is to speak Firefox-Desktop-dialect C++ on the outside (toolkit/components/glean/bindings/private/), and immediately call the firefox-on-glean crate’s FFI (toolkit/components/glean/api/src/ffi) on the inside.

To call the FFI, the FOG C++ metric instance needs to be able to identify the specific FOG Rust metric instance that will need to perform the operation. To do this it stores the FOG Rust metric instance’s metric id.

The C++ layer doesn’t know or care about whether the metric id is

  • assigned during codegen from a metrics definition in a metrics.yaml file

  • assigned from a metrics definition specified at runtime

    • Though the C++ API doesn’t support runtime-defined metrics, seeing as it’s a compiled language, it does have to support JS’ support of runtime-defined metrics.

  • assigned upon first use because it’s a submetric instance created in response to calling .get(aLabel) on a labeled metric

As a result, it doesn’t put any new spin on what Rust defines.

GIFFT Cares, though

Though the C++ layer doesn’t care about what the metric id’s deal is, GIFFT has to in order to e.g. properly map a labeled_counter’s counter submetric’s operations to a keyed uint Scalar’s subscalar with the correct key.

JS: metric_entry_t, category_entry_t, and uint32_t

(Despite this being for JS, you’ll note that all these are C++ types. This is because the FOG JS API is implemented in C++, for our sins.)

The FOG JS API, unlike the FOG Rust and C++ APIs, does not statically name the identifiers of all defined metrics, assigning them their metric ids as parameters to their constructors. This is due to us not being able to find a suitably performant way to do this. (e.g. codegenning webidls was not regarded favourably at the time).

Instead, it uses NamedGetter and performs string matches on category and metric names to build FOG JS metric instances. (See toolkit/components/glean/bindings/ for details.)

FOG JS metric instances themselves contain FOG C++ metric instances to which they delegate as much work as possible. As mentioned in the section on C++, these FOG C++ metric instances are thin wrappers storing a uint32_t that maps to exactly one specific FOG Rust metric instance.

Where the complication kicks in is in how we build a JS FOG metric instance from the NamedGetter.

It boils down to needing to link:

  • the (JS conjugation of the) Glean metric’s base identifier,

    • (so when someone calls Glean.someCategory.aMetric we can locate the rest of the information)

  • the Glean metric’s type

    • (so we locate or create a JS object satisfying the correct webidl interface)

  • the FOG C++ (and, thus, the Rust) metric id

    • (so the FOG JS metric instance’s FOG C++ metric instance’s id is correct when it tries to run code over the FFI)

This is the metric_entry_t which is laid out as follows:

       6                   5                   4               3
 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                  Metric String Table Index                    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   3                   2                   1                   0
 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|   type  | | |            FOG metric instance ID               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

  • 64 bits wide (uint64_t with a typdef)

  • Top 32 bits are an index into a string table of all metrics’ base identifiers (in JS conjugation)

    • We use perfecthash to make it fast to go from Glean.someCategory.aMetric to “Here’s the metric_entry_t for that”, but perfecthash will always return a value even if the supplied string doesn’t match, so we need to check the string on retrieval.

    • We also use the entries and string table to supply a list of supported names, allowing tab completion of metric names in privileged devtools consoles.

  • Bits 31 to 27 (5 bits) denote the Glean metric type

  • Bits 26, 25, and the 25 bits below are all exactly as are used to identify FOG Rust (and, by extension, C++) metric instances.

    • (It isn’t necessary for FOG JS’ metric_entry_t to contain FOG Rust’s metric id, but it is convenient.)

We have a similar system to support categories since Glean.someCategory itself needs to return a JS object. It isn’t nearly as complicated as they’re all the same type and none of them need to correspond to any instance in C++ or Rust. We do this with category_entry_t which is laid out as follows:

   3                   2                   1                   0
 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                 Category String Table Index                   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

  • 32 bits wide (uint32_t with a typedef)

  • All 32 bits are an index into a string table of all metrics’ category names (in JS conjugation)

We maybe should have a stronger MetricId type in FOG JS

Unfortunately, due to both the FOG Rust/C++ metric ids being 32 bits wide, and the lower 32 bits of the JS metric_entry_t being useful for constructing FOG JS metric instances, in the FOG JS impl, a uint32_t aMetricId could mean either a) The equivalent of the FOG Rust/C++ metric id, suitable for use in FFI, or b) That plus 5 bits denoting the Glean metric type. So be careful.

Identifying a FOG Ping: ping_entry_t and u32/uint32_t

Luckily, Glean Pings are far less numerous and far less varied than Glean Metrics.

Unluckily, they need to support a similar level of complication when it comes to supporting string tables for the FOG JS API and supporting runtime registration.

So to identify a FOG Ping instance in Rust or C++ we use a ping id which is:

   3                   2                   1                   0
 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           unused              | |    FOG Ping instance ID     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

  • 32 bits wide (u32 or uint32_t)

  • The top 16 bits are unused, to align with JS

  • Bit 15 is the RUNTIME_PING_BIT which is 1 if the FOG ping instance was created from a definition supplied at runtime, and 0 if it corresponds to a definition supplied at compile time in a pings.yaml.

  • The bottom 15 bits identify a specific, named ping.

Zeroes are Reserved for Pings As Well

The values 0 and 2**15 (0 with or without the signal bit set) are reserved. They may be used to signal “no ping found” or “no ping created” kinds of situations.

To identify a FOG Ping instance in JS we use ping_entry_t which is:

   3                   2                   1                   0
 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|    Ping String Table Index    | |    FOG Ping instance ID     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

  • 32 bits wide (uint32_t with a typedef)

  • The top 16 bits are an index into the ping string table

  • Bit 15 and the bottom 15 bits are exactly as are used in Rust or C++ to identify a specific Ping instance

We maybe should have a stronger PingId type in FOG JS

Unfortunately, due to all of the ping ids and ping_entry_t being 32 bits wide, in the FOG JS impl a uint32_t aPingId could mean anything. It’s likely to not be a full ping_entry_t unless it uses that naming, but that’s just a typedef. So be careful.