gecko_taskgraph.transforms package

Subpackages

Submodules

gecko_taskgraph.transforms.artifact module

Apply different expiration dates to different artifacts based on a manifest file (artifacts.yml)

gecko_taskgraph.transforms.artifact.read_artifact_manifest(manifest_path)

Read the artifacts.yml manifest file and return it.

gecko_taskgraph.transforms.artifact.set_artifact_expiration(config, jobs)

Set the expiration for certain artifacts based on a manifest file.

gecko_taskgraph.transforms.attribution module

gecko_taskgraph.transforms.attribution.attribution_keyed_by(config, jobs)
gecko_taskgraph.transforms.attribution.mac_attribution(config, jobs)

Adds padding to the mac attribution data. Implicitly assumes that the attribution data is the last thing in job.run.command

gecko_taskgraph.transforms.attribution.stub_installer(config, jobs)

Not all windows builds come with a stub installer (only win32, and not on esr), so conditionally add it here based on our dependency’s stub-installer attribute.

gecko_taskgraph.transforms.balrog_submit module

Transform the per-locale balrog task into an actual task description.

gecko_taskgraph.transforms.balrog_submit.handle_keyed_by(config, jobs)

Resolve fields that can be keyed by platform, etc.

gecko_taskgraph.transforms.balrog_submit.make_task_description(config, jobs)
gecko_taskgraph.transforms.balrog_submit.remove_name(config, jobs)

gecko_taskgraph.transforms.balrog_toplevel module

Transform the beetmover task into an actual task description.

gecko_taskgraph.transforms.balrog_toplevel.generate_update_line(config, jobs)

Resolve fields that can be keyed by platform, etc.

gecko_taskgraph.transforms.beetmover module

Transform the beetmover task into an actual task description.

gecko_taskgraph.transforms.beetmover.craft_release_properties(config, job)
gecko_taskgraph.transforms.beetmover.make_task_description(config, jobs)
gecko_taskgraph.transforms.beetmover.make_task_worker(config, jobs)
gecko_taskgraph.transforms.beetmover.remove_name(config, jobs)

gecko_taskgraph.transforms.beetmover_apt module

gecko_taskgraph.transforms.beetmover_apt.batched(iterable, n)

Batch data into tuples of length n. The last batch may be shorter.

gecko_taskgraph.transforms.beetmover_apt.beetmover_apt(config, tasks)
gecko_taskgraph.transforms.beetmover_apt.filter_beetmover_apt_task(task, product)
gecko_taskgraph.transforms.beetmover_apt.filter_beetmover_apt_tasks(config, tasks, product)
gecko_taskgraph.transforms.beetmover_apt.is_not_l10n_task(task)
gecko_taskgraph.transforms.beetmover_apt.is_task_for_product(task, product)
gecko_taskgraph.transforms.beetmover_apt.is_x86_64_l10n_task(task)

gecko_taskgraph.transforms.beetmover_checksums module

Transform the checksums signing task into an actual task description.

gecko_taskgraph.transforms.beetmover_checksums.make_beetmover_checksums_description(config, jobs)
gecko_taskgraph.transforms.beetmover_checksums.make_beetmover_checksums_worker(config, jobs)
gecko_taskgraph.transforms.beetmover_checksums.remove_name(config, jobs)

gecko_taskgraph.transforms.beetmover_emefree_checksums module

Transform release-beetmover-source-checksums into an actual task description.

gecko_taskgraph.transforms.beetmover_emefree_checksums.generate_upstream_artifacts(refs, partner_path)
gecko_taskgraph.transforms.beetmover_emefree_checksums.make_beetmover_checksums_description(config, jobs)
gecko_taskgraph.transforms.beetmover_emefree_checksums.make_beetmover_checksums_worker(config, jobs)
gecko_taskgraph.transforms.beetmover_emefree_checksums.remove_name(config, jobs)

gecko_taskgraph.transforms.beetmover_geckoview module

Transform the beetmover task into an actual task description.

gecko_taskgraph.transforms.beetmover_geckoview.craft_release_properties(config, job)
gecko_taskgraph.transforms.beetmover_geckoview.make_task_description(config, jobs)
gecko_taskgraph.transforms.beetmover_geckoview.make_task_worker(config, jobs)
gecko_taskgraph.transforms.beetmover_geckoview.remove_name(config, jobs)
gecko_taskgraph.transforms.beetmover_geckoview.resolve_keys(config, jobs)
gecko_taskgraph.transforms.beetmover_geckoview.split_maven_packages(config, jobs)

gecko_taskgraph.transforms.beetmover_langpack_checksums module

Transform release-beetmover-langpack-checksums into an actual task description.

gecko_taskgraph.transforms.beetmover_langpack_checksums.make_beetmover_checksums_description(config, jobs)
gecko_taskgraph.transforms.beetmover_langpack_checksums.make_beetmover_checksums_worker(config, jobs)
gecko_taskgraph.transforms.beetmover_langpack_checksums.remove_name(config, jobs)

gecko_taskgraph.transforms.beetmover_push_to_release module

Transform the beetmover-push-to-release task into a task description.

gecko_taskgraph.transforms.beetmover_push_to_release.make_beetmover_push_to_release_description(config, jobs)
gecko_taskgraph.transforms.beetmover_push_to_release.make_beetmover_push_to_release_worker(config, jobs)

gecko_taskgraph.transforms.beetmover_repackage module

Transform the beetmover task into an actual task description.

gecko_taskgraph.transforms.beetmover_repackage.convert_deps(config, jobs)
gecko_taskgraph.transforms.beetmover_repackage.gather_required_signoffs(config, jobs)
gecko_taskgraph.transforms.beetmover_repackage.generate_partials_upstream_artifacts(job, artifacts, platform, locale=None)
gecko_taskgraph.transforms.beetmover_repackage.get_label_by_suffix(labels: List, suffix: str)

Given list of labels, returns the label with provided suffix Raises exception if more than one label is found.

Parameters:

  • labels (List) – List of labels

  • suffix (str) – Suffix for the desired label

Returns

str: The desired label

gecko_taskgraph.transforms.beetmover_repackage.make_partials_artifacts(config, jobs)
gecko_taskgraph.transforms.beetmover_repackage.make_task_description(config, jobs)
gecko_taskgraph.transforms.beetmover_repackage.make_task_worker(config, jobs)
gecko_taskgraph.transforms.beetmover_repackage.remove_name(config, jobs)
gecko_taskgraph.transforms.beetmover_repackage.strip_unwanted_langpacks_from_worker(config, jobs)

Strips out langpacks where we didn’t sign them.

This explicitly deletes langpacks from upstream artifacts and from artifact-maps. Due to limitations in declarative artifacts, doing this was our easiest way right now.

gecko_taskgraph.transforms.beetmover_repackage_l10n module

Transform the signing task into an actual task description.

gecko_taskgraph.transforms.beetmover_repackage_l10n.make_beetmover_description(config, jobs)

gecko_taskgraph.transforms.beetmover_repackage_partner module

Transform the beetmover task into an actual task description.

gecko_taskgraph.transforms.beetmover_repackage_partner.generate_upstream_artifacts(job, build_task_ref, repackage_task_ref, repackage_signing_task_ref, platform, repack_id, partner_path, repack_stub_installer=False)
gecko_taskgraph.transforms.beetmover_repackage_partner.make_task_description(config, jobs)
gecko_taskgraph.transforms.beetmover_repackage_partner.make_task_worker(config, jobs)
gecko_taskgraph.transforms.beetmover_repackage_partner.populate_scopes_and_worker_type(config, jobs)
gecko_taskgraph.transforms.beetmover_repackage_partner.remove_name(config, jobs)

gecko_taskgraph.transforms.beetmover_source module

Transform the beetmover-source task to also append build as dependency

gecko_taskgraph.transforms.beetmover_source.remove_build_dependency_in_beetmover_source(config, jobs)

gecko_taskgraph.transforms.beetmover_source_checksums module

Transform release-beetmover-source-checksums into an actual task description.

gecko_taskgraph.transforms.beetmover_source_checksums.make_beetmover_checksums_description(config, jobs)
gecko_taskgraph.transforms.beetmover_source_checksums.make_beetmover_checksums_worker(config, jobs)
gecko_taskgraph.transforms.beetmover_source_checksums.remove_name(config, jobs)

gecko_taskgraph.transforms.bootstrap module

gecko_taskgraph.transforms.bootstrap.bootstrap_tasks(config, tasks)

gecko_taskgraph.transforms.bouncer_aliases module

Add from parameters.yml into bouncer submission tasks.

gecko_taskgraph.transforms.bouncer_aliases.craft_bouncer_entries(config, job)
gecko_taskgraph.transforms.bouncer_aliases.make_task_worker(config, jobs)

gecko_taskgraph.transforms.bouncer_check module

gecko_taskgraph.transforms.bouncer_check.add_command(config, jobs)
gecko_taskgraph.transforms.bouncer_check.add_previous_versions(config, jobs)
gecko_taskgraph.transforms.bouncer_check.command_to_string(config, jobs)

Convert command to string to make it work properly with run-task

gecko_taskgraph.transforms.bouncer_check.handle_keyed_by(config, jobs)

Resolve fields that can be keyed by project, etc.

gecko_taskgraph.transforms.bouncer_locations module

gecko_taskgraph.transforms.bouncer_locations.make_task_worker(config, jobs)

gecko_taskgraph.transforms.bouncer_submission module

Add from parameters.yml into bouncer submission tasks.

class gecko_taskgraph.transforms.bouncer_submission.InvalidSubstitution(error: str)

Bases: object

gecko_taskgraph.transforms.bouncer_submission.craft_bouncer_entries(config, job)
gecko_taskgraph.transforms.bouncer_submission.craft_bouncer_product_name(product, bouncer_product, current_version, current_build_number=None, previous_version=None)
gecko_taskgraph.transforms.bouncer_submission.craft_paths_per_bouncer_platform(product, bouncer_product, bouncer_platforms, current_version, current_build_number, previous_version=None)
gecko_taskgraph.transforms.bouncer_submission.craft_ssl_only(bouncer_product, project)
gecko_taskgraph.transforms.bouncer_submission.make_task_worker(config, jobs)
gecko_taskgraph.transforms.bouncer_submission.split_build_data(version)

gecko_taskgraph.transforms.bouncer_submission_partners module

Add from parameters.yml into bouncer submission tasks.

gecko_taskgraph.transforms.bouncer_submission_partners.craft_bouncer_entries(config, job)
gecko_taskgraph.transforms.bouncer_submission_partners.craft_partner_bouncer_product_name(product, bouncer_product, current_version, partner, sub_config)
gecko_taskgraph.transforms.bouncer_submission_partners.craft_paths_per_bouncer_platform(product, bouncer_product, bouncer_platforms, current_version, partner, sub_config)
gecko_taskgraph.transforms.bouncer_submission_partners.craft_ssl_only(bouncer_product)
gecko_taskgraph.transforms.bouncer_submission_partners.make_task_worker(config, jobs)

gecko_taskgraph.transforms.build module

Apply some defaults and minor modifications to the jobs defined in the build kind.

gecko_taskgraph.transforms.build.enable_full_crashsymbols(config, jobs)

Enable full crashsymbols on jobs with ‘enable-full-crashsymbols’ set to True and on release branches, or on try

gecko_taskgraph.transforms.build.mozconfig(config, jobs)
gecko_taskgraph.transforms.build.resolve_keys(config, jobs)
gecko_taskgraph.transforms.build.resolve_shipping_product(config, jobs)
gecko_taskgraph.transforms.build.set_defaults(config, jobs)

Set defaults, including those that differ per worker implementation

gecko_taskgraph.transforms.build.set_expiry(config, jobs)
gecko_taskgraph.transforms.build.stub_installer(config, jobs)
gecko_taskgraph.transforms.build.update_channel(config, jobs)
gecko_taskgraph.transforms.build.use_artifact(config, jobs)
gecko_taskgraph.transforms.build.use_profile_data(config, jobs)

gecko_taskgraph.transforms.build_attrs module

gecko_taskgraph.transforms.build_attrs.set_build_attributes(config, jobs)

Set the build_platform and build_type attributes based on the job name. Although not all jobs using this transform are actual “builds”, the try option syntax treats them as such, and this arranges the attributes appropriately for that purpose.

gecko_taskgraph.transforms.build_fat_aar module

gecko_taskgraph.transforms.build_fat_aar.set_fetches_and_locations(config, jobs)

Set defaults, including those that differ per worker implementation

gecko_taskgraph.transforms.build_lints module

Apply some defaults and minor modifications to the jobs defined in the build kind.

gecko_taskgraph.transforms.build_lints.check_mozharness_perfherder_options(config, jobs)

Verify that multiple jobs don’t use the same perfherder bucket.

Build jobs record perfherder metrics by default. Perfherder metrics go to a bucket derived by the platform by default. The name can further be customized by the presence of “extra options” either defined in mozharness sub-configs or in an environment variable.

This linter tries to verify that no 2 jobs will send Perfherder metrics to the same bucket by looking for jobs not defining extra options when their platform or mozharness config are otherwise similar.

gecko_taskgraph.transforms.build_schedules module

gecko_taskgraph.transforms.build_schedules.set_build_schedules_optimization(config, tasks)

Set the build optimization based on the build platform.

gecko_taskgraph.transforms.build_signing module

Transform the signing task into an actual task description.

gecko_taskgraph.transforms.build_signing.add_signed_routes(config, jobs)

Add routes corresponding to the routes of the build task this corresponds to, with .signed inserted, for all gecko.v2 routes

gecko_taskgraph.transforms.build_signing.define_upstream_artifacts(config, jobs)

gecko_taskgraph.transforms.cached_tasks module

gecko_taskgraph.transforms.cached_tasks.cache_task(config, tasks)
gecko_taskgraph.transforms.cached_tasks.format_task_digest(cached_task)
gecko_taskgraph.transforms.cached_tasks.order_tasks(config, tasks)

Iterate image tasks in an order where parent tasks come first.

gecko_taskgraph.transforms.chunk_partners module

Chunk the partner repack tasks by subpartner and locale

gecko_taskgraph.transforms.chunk_partners.chunk_partners(config, jobs)

gecko_taskgraph.transforms.code_review module

Add soft dependencies and configuration to code-review tasks.

gecko_taskgraph.transforms.code_review.add_dependencies(config, jobs)
gecko_taskgraph.transforms.code_review.add_phabricator_config(config, jobs)

gecko_taskgraph.transforms.condprof module

This transform constructs tasks generate conditioned profiles from the condprof/kind.yml file

gecko_taskgraph.transforms.condprof.generate_scenarios(config, tasks)

gecko_taskgraph.transforms.cross_channel module

Build a command to run mach l10n-cross-channel.

gecko_taskgraph.transforms.cross_channel.build_command(config, jobs)
gecko_taskgraph.transforms.cross_channel.resolve_keys(config, jobs)

gecko_taskgraph.transforms.diffoscope module

This transform construct tasks to perform diffs between builds, as defined in kind.yml

gecko_taskgraph.transforms.diffoscope.fill_template(config, tasks)

gecko_taskgraph.transforms.docker_image module

gecko_taskgraph.transforms.docker_image.fill_template(config, tasks)

gecko_taskgraph.transforms.fetch module

class gecko_taskgraph.transforms.fetch.FetchBuilder(schema: Schema, builder)

Bases: object

gecko_taskgraph.transforms.fetch.configure_fetch(config, typ, name, fetch)
gecko_taskgraph.transforms.fetch.create_cft_canary_fetch_task(config, name, fetch)
gecko_taskgraph.transforms.fetch.create_chromium_fetch_task(config, name, fetch)
gecko_taskgraph.transforms.fetch.create_fetch_url_task(config, name, fetch)
gecko_taskgraph.transforms.fetch.create_git_fetch_task(config, name, fetch)
gecko_taskgraph.transforms.fetch.fetch_builder(name, schema)
gecko_taskgraph.transforms.fetch.make_task(config, jobs)
gecko_taskgraph.transforms.fetch.process_fetch_job(config, jobs)

gecko_taskgraph.transforms.final_verify module

Transform the beetmover task into an actual task description.

gecko_taskgraph.transforms.final_verify.add_command(config, tasks)

gecko_taskgraph.transforms.geckodriver_mac_notarization module

Transform the geckodriver notarization task into an actual task description.

gecko_taskgraph.transforms.geckodriver_mac_notarization.geckodriver_mac_notarization(config, jobs)
gecko_taskgraph.transforms.geckodriver_mac_notarization.remove_name(config, jobs)

gecko_taskgraph.transforms.geckodriver_signing module

Transform the repackage signing task into an actual task description.

gecko_taskgraph.transforms.geckodriver_signing.make_signing_description(config, jobs)
gecko_taskgraph.transforms.geckodriver_signing.remove_name(config, jobs)

gecko_taskgraph.transforms.hardened_signing module

Transform the signing task into an actual task description.

gecko_taskgraph.transforms.hardened_signing.add_hardened_sign_config(config, jobs)
gecko_taskgraph.transforms.hardened_signing.add_provisioning_profile_config(config, jobs)

gecko_taskgraph.transforms.l10n module

Do transforms specific to l10n kind

gecko_taskgraph.transforms.l10n.all_locales_attribute(config, jobs)
gecko_taskgraph.transforms.l10n.chunk_locales(config, jobs)

Utilizes chunking for l10n stuff

gecko_taskgraph.transforms.l10n.copy_in_useful_magic(config, jobs)
gecko_taskgraph.transforms.l10n.gather_required_signoffs(config, jobs)
gecko_taskgraph.transforms.l10n.handle_artifact_prefix(config, jobs)

Resolve artifact_prefix in env vars

gecko_taskgraph.transforms.l10n.handle_keyed_by(config, jobs)

Resolve fields that can be keyed by platform, etc.

gecko_taskgraph.transforms.l10n.make_job_description(config, jobs)
gecko_taskgraph.transforms.l10n.parse_locales_file(locales_file, platform=None)

Parse the passed locales file for a list of locales.

gecko_taskgraph.transforms.l10n.remove_repackage_dependency(config, jobs)
gecko_taskgraph.transforms.l10n.set_extra_config(config, jobs)
gecko_taskgraph.transforms.l10n.setup_name(config, jobs)
gecko_taskgraph.transforms.l10n.stub_installer(config, jobs)

gecko_taskgraph.transforms.mac_dummy module

Add dependencies to dummy macosx64 tasks.

gecko_taskgraph.transforms.mac_dummy.add_dependencies(config, jobs)

gecko_taskgraph.transforms.mac_notarization module

Transform mac notarization tasks

gecko_taskgraph.transforms.mac_notarization.only_level_3_notarization(config, jobs)

Filter out any notarization jobs that are not level 3

gecko_taskgraph.transforms.mar_signing module

Transform the {partials,mar}-signing task into an actual task description.

gecko_taskgraph.transforms.mar_signing.generate_complete_artifacts(job, kind)
gecko_taskgraph.transforms.mar_signing.generate_partials_artifacts(job, release_history, platform, locale=None)
gecko_taskgraph.transforms.mar_signing.make_task_description(config, jobs)

gecko_taskgraph.transforms.maybe_release module

gecko_taskgraph.transforms.maybe_release.make_task_description(config, jobs)

gecko_taskgraph.transforms.merge_automation module

Transform the update generation task into an actual task description.

gecko_taskgraph.transforms.merge_automation.add_payload_config(config, tasks)
gecko_taskgraph.transforms.merge_automation.handle_keyed_by(config, tasks)

Resolve fields that can be keyed by platform, etc.

gecko_taskgraph.transforms.merge_automation.update_labels(config, tasks)

gecko_taskgraph.transforms.name_sanity module

Generate labels for tasks without names, consistently. Uses attributes from primary dependency.

gecko_taskgraph.transforms.name_sanity.make_label(config, jobs)

Generate a sane label for a new task constructed from a dependency Using attributes from the dependent job and the current task kind

gecko_taskgraph.transforms.openh264 module

This transform is used to help populate mozharness options for openh264 jobs

gecko_taskgraph.transforms.openh264.set_mh_options(config, jobs)

This transform sets the ‘openh264_rev’ attribute.

gecko_taskgraph.transforms.openh264_signing module

Transform the repackage signing task into an actual task description.

gecko_taskgraph.transforms.openh264_signing.make_signing_description(config, jobs)
gecko_taskgraph.transforms.openh264_signing.remove_name(config, jobs)

gecko_taskgraph.transforms.partials module

Transform the partials task into an actual task description.

gecko_taskgraph.transforms.partials.identify_desired_signing_keys(project, product)
gecko_taskgraph.transforms.partials.make_task_description(config, jobs)

gecko_taskgraph.transforms.partner_attribution module

Transform the partner attribution task into an actual task description.

gecko_taskgraph.transforms.partner_attribution.add_command_arguments(config, tasks)

gecko_taskgraph.transforms.partner_attribution_beetmover module

Transform the beetmover task into an actual task description.

gecko_taskgraph.transforms.partner_attribution_beetmover.generate_upstream_artifacts(attribution_task, artifacts, partner_path)
gecko_taskgraph.transforms.partner_attribution_beetmover.make_task_description(config, jobs)
gecko_taskgraph.transforms.partner_attribution_beetmover.make_task_worker(config, jobs)
gecko_taskgraph.transforms.partner_attribution_beetmover.populate_scopes_and_upstream_artifacts(config, jobs)

gecko_taskgraph.transforms.partner_repack module

Transform the partner repack task into an actual task description.

gecko_taskgraph.transforms.partner_repack.add_command_arguments(config, tasks)
gecko_taskgraph.transforms.partner_repack.make_label(config, tasks)
gecko_taskgraph.transforms.partner_repack.populate_repack_manifests_url(config, tasks)
gecko_taskgraph.transforms.partner_repack.remove_mac_dependency(config, tasks)

Remove mac dependency depending on current level to accomodate for mac notarization not running on level 1

gecko_taskgraph.transforms.partner_repack.skip_unnecessary_platforms(config, tasks)

gecko_taskgraph.transforms.partner_signing module

Transform the signing task into an actual task description.

gecko_taskgraph.transforms.partner_signing.define_upstream_artifacts(config, jobs)
gecko_taskgraph.transforms.partner_signing.set_mac_label(config, jobs)

gecko_taskgraph.transforms.per_platform_dummy module

Transform the repackage task into an actual task description.

gecko_taskgraph.transforms.per_platform_dummy.one_task_per_product_and_platform(config, jobs)

gecko_taskgraph.transforms.perftest module

This transform passes options from mach perftest to the corresponding task.

gecko_taskgraph.transforms.perftest.apply_perftest_tier_optimization(config, jobs)
gecko_taskgraph.transforms.perftest.handle_keyed_by_perftest(config, jobs)
gecko_taskgraph.transforms.perftest.parse_perftest_metrics(config, jobs)

Parse the metrics into a dictionary immediately.

This way we can modify the extraOptions field (and others) entry through the transforms that come later. The metrics aren’t formatted until the end of the transforms.

gecko_taskgraph.transforms.perftest.pass_perftest_options(config, jobs)
gecko_taskgraph.transforms.perftest.setup_http3_tests(config, jobs)
gecko_taskgraph.transforms.perftest.setup_perftest_browsertime_variants(config, jobs)
gecko_taskgraph.transforms.perftest.setup_perftest_extra_options(config, jobs)
gecko_taskgraph.transforms.perftest.setup_perftest_metrics(config, jobs)
gecko_taskgraph.transforms.perftest.setup_perftest_test_date(config, jobs)
gecko_taskgraph.transforms.perftest.setup_regression_detector(config, jobs)
gecko_taskgraph.transforms.perftest.split_btime_variants(config, jobs)
gecko_taskgraph.transforms.perftest.split_perftest_variants(config, jobs)
gecko_taskgraph.transforms.perftest.split_tests(config, jobs)

gecko_taskgraph.transforms.python_update module

Transform the repo-update task into an actual task description.

gecko_taskgraph.transforms.python_update.resolve_keys(config, tasks)

gecko_taskgraph.transforms.release module

Transforms for release tasks

gecko_taskgraph.transforms.release.run_on_releases(config, jobs)

Filter out jobs with run-on-releases set, and that don’t match the release_type paramater.

gecko_taskgraph.transforms.release_beetmover_signed_addons module

Transform the beetmover task into an actual task description.

gecko_taskgraph.transforms.release_beetmover_signed_addons.make_task_description(config, jobs)
gecko_taskgraph.transforms.release_beetmover_signed_addons.make_task_worker(config, jobs)
gecko_taskgraph.transforms.release_beetmover_signed_addons.remove_name(config, jobs)
gecko_taskgraph.transforms.release_beetmover_signed_addons.resolve_keys(config, jobs)
gecko_taskgraph.transforms.release_beetmover_signed_addons.yield_all_platform_jobs(config, jobs)

gecko_taskgraph.transforms.release_deps module

Add dependencies to release tasks.

gecko_taskgraph.transforms.release_deps.add_dependencies(config, jobs)

gecko_taskgraph.transforms.release_flatpak_push module

Transform the release-flatpak-push kind into an actual task description.

gecko_taskgraph.transforms.release_flatpak_push.generate_upstream_artifacts(dependencies)
gecko_taskgraph.transforms.release_flatpak_push.make_task_description(config, jobs)

gecko_taskgraph.transforms.release_flatpak_repackage module

gecko_taskgraph.transforms.release_flatpak_repackage.add_desktop_file_url(config, tasks)

Add desktop file artifact url to task environment

gecko_taskgraph.transforms.release_flatpak_repackage.format(config, tasks)

Apply format substitution to worker.env and worker.command.

gecko_taskgraph.transforms.release_generate_checksums module

Transform the checksums task into an actual task description.

gecko_taskgraph.transforms.release_generate_checksums.handle_keyed_by(config, jobs)

Resolve fields that can be keyed by project, etc.

gecko_taskgraph.transforms.release_generate_checksums.interpolate(config, jobs)

gecko_taskgraph.transforms.release_generate_checksums_beetmover module

Transform the release-generate-checksums-beetmover task to also append build as dependency

gecko_taskgraph.transforms.release_generate_checksums_beetmover.make_task_description(config, jobs)
gecko_taskgraph.transforms.release_generate_checksums_beetmover.make_task_worker(config, jobs)
gecko_taskgraph.transforms.release_generate_checksums_beetmover.remove_name(config, jobs)

gecko_taskgraph.transforms.release_generate_checksums_signing module

Transform the release-generate-checksums-signing task into task description.

gecko_taskgraph.transforms.release_generate_checksums_signing.make_release_generate_checksums_signing_description(config, jobs)
gecko_taskgraph.transforms.release_generate_checksums_signing.remote_name(config, jobs)

gecko_taskgraph.transforms.release_mark_as_shipped module

gecko_taskgraph.transforms.release_mark_as_shipped.make_task_description(config, jobs)

gecko_taskgraph.transforms.release_msix_push module

Transform the release-msix-push kind into an actual task description.

gecko_taskgraph.transforms.release_msix_push.generate_upstream_artifacts(dependencies)
gecko_taskgraph.transforms.release_msix_push.make_task_description(config, jobs)

gecko_taskgraph.transforms.release_notifications module

Add notifications via taskcluster-notify for release tasks

class gecko_taskgraph.transforms.release_notifications.TitleCaseFormatter

Bases: Formatter

Support title formatter for strings

convert_field(value, conversion)
gecko_taskgraph.transforms.release_notifications.add_notifications(config, jobs)

gecko_taskgraph.transforms.release_sign_and_push_langpacks module

Transform the release-sign-and-push task into an actual task description.

gecko_taskgraph.transforms.release_sign_and_push_langpacks.copy_attributes(config, jobs)
gecko_taskgraph.transforms.release_sign_and_push_langpacks.filter_out_macos_jobs_but_mac_only_locales(config, jobs)
gecko_taskgraph.transforms.release_sign_and_push_langpacks.generate_upstream_artifacts(upstream_task_ref, locales)
gecko_taskgraph.transforms.release_sign_and_push_langpacks.get_upstream_task_ref(job, expected_kinds)
gecko_taskgraph.transforms.release_sign_and_push_langpacks.make_task_description(config, jobs)
gecko_taskgraph.transforms.release_sign_and_push_langpacks.make_task_worker(config, jobs)
gecko_taskgraph.transforms.release_sign_and_push_langpacks.resolve_keys(config, jobs)
gecko_taskgraph.transforms.release_sign_and_push_langpacks.set_label(config, jobs)

gecko_taskgraph.transforms.release_snap_repackage module

gecko_taskgraph.transforms.release_snap_repackage.format(config, tasks)

Apply format substitution to worker.env and worker.command.

gecko_taskgraph.transforms.release_started module

Add notifications via taskcluster-notify for release tasks

gecko_taskgraph.transforms.release_started.add_notifications(config, jobs)

gecko_taskgraph.transforms.release_update_product_channel_version module

gecko_taskgraph.transforms.release_update_product_channel_version.make_task_worker(config, jobs)

gecko_taskgraph.transforms.release_version_bump module

Transform the update generation task into an actual task description.

gecko_taskgraph.transforms.release_version_bump.handle_keyed_by(config, tasks)

Resolve fields that can be keyed by platform, etc.

gecko_taskgraph.transforms.repackage module

Transform the repackage task into an actual task description.

gecko_taskgraph.transforms.repackage.copy_in_useful_magic(config, jobs)

Copy attributes from upstream task to be used for keyed configuration.

gecko_taskgraph.transforms.repackage.handle_keyed_by(config, jobs)

Resolve fields that can be keyed by platform, etc, but not msix.* fields that can be keyed by package-format. Such fields are handled specially below.

gecko_taskgraph.transforms.repackage.make_job_description(config, jobs)
gecko_taskgraph.transforms.repackage.make_repackage_description(config, jobs)
gecko_taskgraph.transforms.repackage.remove_name(config, jobs)

gecko_taskgraph.transforms.repackage_l10n module

Transform the repackage task into an actual task description.

gecko_taskgraph.transforms.repackage_l10n.split_locales(config, jobs)

gecko_taskgraph.transforms.repackage_partner module

Transform the repackage task into an actual task description.

gecko_taskgraph.transforms.repackage_partner.copy_in_useful_magic(config, jobs)

Copy attributes from upstream task to be used for keyed configuration.

gecko_taskgraph.transforms.repackage_partner.handle_keyed_by(config, jobs)

Resolve fields that can be keyed by platform, etc.

gecko_taskgraph.transforms.repackage_partner.make_job_description(config, jobs)
gecko_taskgraph.transforms.repackage_partner.make_repackage_description(config, jobs)
gecko_taskgraph.transforms.repackage_partner.remove_name(config, jobs)

gecko_taskgraph.transforms.repackage_routes module

Add indexes to repackage kinds

gecko_taskgraph.transforms.repackage_routes.add_indexes(config, jobs)

gecko_taskgraph.transforms.repackage_set_upstream_mac_kind module

Transform mac notarization tasks

gecko_taskgraph.transforms.repackage_set_upstream_mac_kind.repackage_set_upstream_mac_kind(config, tasks)

Notarization only runs on level 3 If level < 3 then repackage the mac-signing task artifact Exception for debug builds, which will use signed build on level 3

gecko_taskgraph.transforms.repackage_signing module

Transform the repackage signing task into an actual task description.

gecko_taskgraph.transforms.repackage_signing.make_repackage_signing_description(config, jobs)
gecko_taskgraph.transforms.repackage_signing.remove_name(config, jobs)

gecko_taskgraph.transforms.repackage_signing_partner module

Transform the repackage signing task into an actual task description.

gecko_taskgraph.transforms.repackage_signing_partner.make_repackage_signing_description(config, jobs)
gecko_taskgraph.transforms.repackage_signing_partner.remove_name(config, jobs)

gecko_taskgraph.transforms.repo_update module

Transform the repo-update task into an actual task description.

gecko_taskgraph.transforms.repo_update.resolve_keys(config, tasks)

gecko_taskgraph.transforms.reprocess_symbols module

Transform the reprocess-symbols task description template, taskcluster/kinds/reprocess-symbols/job-template.yml into an actual task description.

gecko_taskgraph.transforms.reprocess_symbols.fill_template(config, tasks)
gecko_taskgraph.transforms.reprocess_symbols.gather_required_signoffs(config, jobs)

gecko_taskgraph.transforms.reverse_chunk_deps module

Adjust dependencies to not exceed MAX_DEPENDENCIES

gecko_taskgraph.transforms.reverse_chunk_deps.add_dependencies(config, jobs)
gecko_taskgraph.transforms.reverse_chunk_deps.yield_job(orig_job, deps, count)

gecko_taskgraph.transforms.run_pgo_profile module

Apply some defaults and minor modifications to the pgo jobs.

gecko_taskgraph.transforms.run_pgo_profile.run_profile_data(config, jobs)

gecko_taskgraph.transforms.scriptworker module

Transforms for adding appropriate scopes to scriptworker tasks.

gecko_taskgraph.transforms.scriptworker.add_balrog_scopes(config, jobs)

gecko_taskgraph.transforms.scriptworker_canary module

Build a command to run mach release push-scriptworker-canaries.

gecko_taskgraph.transforms.scriptworker_canary.build_command(config, jobs)

gecko_taskgraph.transforms.sentry module

gecko_taskgraph.transforms.sentry.sentry(config, tasks)

Do transforms specific to github-sync tasks.

gecko_taskgraph.transforms.shippable_l10n_signing module

Transform the signing task into an actual task description.

gecko_taskgraph.transforms.shippable_l10n_signing.define_upstream_artifacts(config, jobs)
gecko_taskgraph.transforms.shippable_l10n_signing.make_signing_description(config, jobs)

gecko_taskgraph.transforms.signing module

Transform the signing task into an actual task description.

gecko_taskgraph.transforms.signing.delete_name(config, jobs)

Delete the ‘name’ key if it exists, we don’t use it.

gecko_taskgraph.transforms.signing.get_locales_description(attributes, default)

Returns the [list] of locales for task description usage

gecko_taskgraph.transforms.signing.make_task_description(config, jobs)

gecko_taskgraph.transforms.snap_test module

gecko_taskgraph.transforms.snap_test.fill_template(config, tasks)

gecko_taskgraph.transforms.source_checksums_signing module

Transform the checksums signing task into an actual task description.

gecko_taskgraph.transforms.source_checksums_signing.make_checksums_signing_description(config, jobs)
gecko_taskgraph.transforms.source_checksums_signing.remove_name(config, jobs)

gecko_taskgraph.transforms.source_test module

Source-test jobs can run on multiple platforms. These transforms allow jobs with either platform or a list of platforms, and set the appropriate treeherder configuration and attributes for that platform.

gecko_taskgraph.transforms.source_test.add_build_dependency(config, job)

Add build dependency to the job and installer_url to env.

gecko_taskgraph.transforms.source_test.expand_platforms(config, jobs)
gecko_taskgraph.transforms.source_test.handle_platform(config, jobs)

Handle the ‘platform’ property, setting up treeherder context as well as try-related attributes.

gecko_taskgraph.transforms.source_test.handle_shell(config, jobs)

Handle the ‘shell’ property.

gecko_taskgraph.transforms.source_test.remove_optimization_on_central(config, jobs)

For pushes to mozilla-central run all source-test tasks that are enabled for code-review in order to have the code-review bot populate the DB according with the push hash.

gecko_taskgraph.transforms.source_test.set_code_review_env(config, jobs)

Add a CODE_REVIEW environment variable when running in code-review bot mode

gecko_taskgraph.transforms.source_test.set_job_name(config, jobs)
gecko_taskgraph.transforms.source_test.set_worker_exit_code(config, jobs)
gecko_taskgraph.transforms.source_test.split_jsshell(config, jobs)
gecko_taskgraph.transforms.source_test.split_python(config, jobs)

gecko_taskgraph.transforms.spidermonkey module

gecko_taskgraph.transforms.spidermonkey.handle_keyed_by(config, jobs)

Resolve fields that can be keyed by platform, etc.

gecko_taskgraph.transforms.split_by_locale module

This transform splits the jobs it receives into per-locale tasks. Locales are provided by the locales-file.

gecko_taskgraph.transforms.split_by_locale.add_command(config, jobs)

gecko_taskgraph.transforms.startup_test module

gecko_taskgraph.transforms.startup_test.add_command(config, jobs)

gecko_taskgraph.transforms.task module

These transformations take a task description and turn it into a TaskCluster task definition (along with attributes, label, etc.). The input to these transformations is generic to any kind of task, but abstracts away some of the complexities of worker implementations, scopes, and treeherder annotations.

class gecko_taskgraph.transforms.task.PayloadBuilder(schema: Schema, builder)

Bases: object

gecko_taskgraph.transforms.task.add_android_shippable_index_routes(config, task)
gecko_taskgraph.transforms.task.add_android_shippable_multi_index_routes(config, task)
gecko_taskgraph.transforms.task.add_geckoview_index_routes(config, task)
gecko_taskgraph.transforms.task.add_generic_index_routes(config, task)
gecko_taskgraph.transforms.task.add_index_routes(config, tasks)
gecko_taskgraph.transforms.task.add_l10n_index_routes(config, task, force_locale=None)
gecko_taskgraph.transforms.task.add_shippable_index_routes(config, task)
gecko_taskgraph.transforms.task.add_shippable_l10n_index_routes(config, task, force_locale=None)
gecko_taskgraph.transforms.task.add_shippable_multi_index_routes(config, task)
gecko_taskgraph.transforms.task.build_balrog_payload(config, task, task_def)
gecko_taskgraph.transforms.task.build_beetmover_maven_payload(config, task, task_def)
gecko_taskgraph.transforms.task.build_beetmover_payload(config, task, task_def)
gecko_taskgraph.transforms.task.build_beetmover_push_to_release_payload(config, task, task_def)
gecko_taskgraph.transforms.task.build_bouncer_aliases_payload(config, task, task_def)
gecko_taskgraph.transforms.task.build_bouncer_locations_payload(config, task, task_def)
gecko_taskgraph.transforms.task.build_bouncer_submission_payload(config, task, task_def)
gecko_taskgraph.transforms.task.build_docker_worker_payload(config, task, task_def)
gecko_taskgraph.transforms.task.build_dummy_payload(config, task, task_def)
gecko_taskgraph.transforms.task.build_generic_worker_payload(config, task, task_def)
gecko_taskgraph.transforms.task.build_import_from_gcs_to_artifact_registry_payload(config, task, task_def)
gecko_taskgraph.transforms.task.build_invalid_payload(config, task, task_def)
gecko_taskgraph.transforms.task.build_push_addons_payload(config, task, task_def)
gecko_taskgraph.transforms.task.build_push_flatpak_payload(config, task, task_def)
gecko_taskgraph.transforms.task.build_push_msix_payload(config, task, task_def)
gecko_taskgraph.transforms.task.build_scriptworker_signing_payload(config, task, task_def)
gecko_taskgraph.transforms.task.build_ship_it_maybe_release_payload(config, task, task_def)
gecko_taskgraph.transforms.task.build_ship_it_shipped_payload(config, task, task_def)
gecko_taskgraph.transforms.task.build_ship_it_update_product_channel_version_payload(config, task, task_def)
gecko_taskgraph.transforms.task.build_task(config, tasks)
gecko_taskgraph.transforms.task.build_treescript_payload(config, task, task_def)
gecko_taskgraph.transforms.task.chain_of_trust(config, tasks)
gecko_taskgraph.transforms.task.check_caches_are_volumes(task)

Ensures that all cache paths are defined as volumes.

Caches and volumes are the only filesystem locations whose content isn’t defined by the Docker image itself. Some caches are optional depending on the job environment. We want paths that are potentially caches to have as similar behavior regardless of whether a cache is used. To help enforce this, we require that all paths used as caches to be declared as Docker volumes. This check won’t catch all offenders. But it is better than nothing.

gecko_taskgraph.transforms.task.check_required_volumes(task)

Ensures that all paths that are required to be volumes are defined as volumes.

Performance of writing to files in poor in directories not marked as volumes, in docker. Ensure that paths that are often written to are marked as volumes.

gecko_taskgraph.transforms.task.check_run_task_caches(config, tasks)

Audit for caches requiring run-task.

run-task manages caches in certain ways. If a cache managed by run-task is used by a non run-task task, it could cause problems. So we audit for that and make sure certain cache names are exclusive to run-task.

IF YOU ARE TEMPTED TO MAKE EXCLUSIONS TO THIS POLICY, YOU ARE LIKELY CONTRIBUTING TECHNICAL DEBT AND WILL HAVE TO SOLVE MANY OF THE PROBLEMS THAT RUN-TASK ALREADY SOLVES. THINK LONG AND HARD BEFORE DOING THAT.

gecko_taskgraph.transforms.task.check_task_dependencies(config, tasks)

Ensures that tasks don’t have more than 100 dependencies.

gecko_taskgraph.transforms.task.check_task_identifiers(config, tasks)

Ensures that all tasks have well defined identifiers: ^[a-zA-Z0-9_-]{1,38}$

gecko_taskgraph.transforms.task.get_branch_repo(config)
gecko_taskgraph.transforms.task.get_branch_rev(config)
gecko_taskgraph.transforms.task.get_default_priority(graph_config, project)
gecko_taskgraph.transforms.task.index_builder(name)
gecko_taskgraph.transforms.task.payload_builder(name, schema)
gecko_taskgraph.transforms.task.set_defaults(config, tasks)
gecko_taskgraph.transforms.task.set_implementation(config, tasks)

Set the worker implementation based on the worker-type alias.

gecko_taskgraph.transforms.task.set_task_and_artifact_expiry(config, jobs)

Set the default expiry for tasks and their artifacts.

These values are read from ci/config.yml

gecko_taskgraph.transforms.task.setup_raptor(config, tasks)

Add options that are specific to raptor jobs (identified by suite=raptor).

This variant uses a separate set of transforms for manipulating the tests at the task-level. Currently only used for setting the taskcluster proxy setting and the scopes required for perftest secrets.

gecko_taskgraph.transforms.task.task_name_from_label(config, tasks)
gecko_taskgraph.transforms.task.try_task_config_chemspill_prio(config, tasks)

Increase the priority from lowest and very-low -> low, but leave others unchanged.

gecko_taskgraph.transforms.task.try_task_config_env(config, tasks)

Set environment variables in the task.

gecko_taskgraph.transforms.task.try_task_config_routes(config, tasks)

Set routes in the task.

gecko_taskgraph.transforms.task.validate(config, tasks)
gecko_taskgraph.transforms.task.validate_shipping_product(config, product)
gecko_taskgraph.transforms.task.verify_index(config, index)

gecko_taskgraph.transforms.test_apk module

Apply some defaults and minor modifications to the jobs defined in the test kinds.

gecko_taskgraph.transforms.test_apk.resolve_keys(config, tasks)

gecko_taskgraph.transforms.trigger_comm_central module

Resolve keys for the jobs defined in the trigger-comm-central kind.

gecko_taskgraph.transforms.trigger_comm_central.resolve_keys(config, jobs)

gecko_taskgraph.transforms.try_job module

gecko_taskgraph.transforms.try_job.set_job_try_name(config, jobs)

For a task which is governed by -j in try syntax, set the job_try_name attribute based on the job name.

gecko_taskgraph.transforms.update_verify module

Transform the beetmover task into an actual task description.

gecko_taskgraph.transforms.update_verify.add_command(config, tasks)

gecko_taskgraph.transforms.update_verify_config module

Transform the beetmover task into an actual task description.

gecko_taskgraph.transforms.update_verify_config.add_command(config, tasks)
gecko_taskgraph.transforms.update_verify_config.ensure_wrapped_singlequote(regexes)

Ensure that a regex (from INCLUDE_VERSION_REGEXES or MAR_CHANNEL_ID_OVERRIDE_REGEXES) is wrapper in single quotes.

gecko_taskgraph.transforms.upload_generated_sources module

Transform the upload-generated-files task description template, taskcluster/kinds/upload-generated-sources/kind.yml, into an actual task description.

gecko_taskgraph.transforms.upload_generated_sources.add_task_info(config, jobs)

gecko_taskgraph.transforms.upload_symbols module

Transform the upload-symbols task description template, taskcluster/kinds/upload-symbols/job-template.yml into an actual task description.

gecko_taskgraph.transforms.upload_symbols.check_nightlies(config, tasks)

Ensure that we upload symbols for all shippable builds, so that crash-stats can resolve any reports sent to it. Try may enable full symbols but not upload them.

Putting this check here (instead of the transforms for the build kind) lets us leverage the any not-for-build-platforms set in the update-symbols kind.

gecko_taskgraph.transforms.upload_symbols.fill_template(config, tasks)

Module contents