diff --git a/doc/languages-frameworks/gradle.section.md b/doc/languages-frameworks/gradle.section.md new file mode 100644 index 000000000000..b2058ef87491 --- /dev/null +++ b/doc/languages-frameworks/gradle.section.md @@ -0,0 +1,189 @@ +# Gradle {#gradle} + +Gradle is a popular build tool for Java/Kotlin. Gradle itself doesn't +currently provide tools to make dependency resolution reproducible, so +nixpkgs has a proxy designed for intercepting Gradle web requests to +record dependencies so they can be restored in a reproducible fashion. + +## Building a Gradle package {#building-a-gradle-package} + +Here's how a typical derivation will look like: + +```nix +stdenv.mkDerivation (finalAttrs: { + pname = "pdftk"; + version = "3.3.3"; + + src = fetchFromGitLab { + owner = "pdftk-java"; + repo = "pdftk"; + rev = "v${finalAttrs.version}"; + hash = "sha256-ciKotTHSEcITfQYKFZ6sY2LZnXGChBJy0+eno8B3YHY="; + }; + + nativeBuildInputs = [ gradle ]; + + # if the package has dependencies, mitmCache must be set + mitmCache = gradle.fetchDeps { + inherit (finalAttrs) pname; + data = ./deps.json; + }; + + # this is required for using mitm-cache on Darwin + __darwinAllowLocalNetworking = true; + + gradleFlags = [ "-Dfile.encoding=utf-8" ]; + + # defaults to "assemble" + gradleBuildTask = "shadowJar"; + + # will run the gradleCheckTask (defaults to "test") + doCheck = true; + + installPhase = '' + mkdir -p $out/{bin,share/pdftk} + cp build/libs/pdftk-all.jar $out/share/pdftk + + makeWrapper ${jre}/bin/java $out/bin/pdftk \ + --add-flags "-jar $out/share/pdftk/pdftk-all.jar" + + cp ${finalAttrs.src}/pdftk.1 $out/share/man/man1 + ''; + + meta.sourceProvenance = with lib.sourceTypes; [ + fromSource + binaryBytecode # mitm cache + ]; +}) +``` + +To update (or initialize) dependencies, run the update script via +something like `$(nix-build -A .mitmCache.updateScript)` +(`nix-build` builds the `updateScript`, `$(...)` runs the script at the +path printed by `nix-build`). + +If your package can't be evaluated using a simple `pkgs.` +expression (for example, if your package isn't located in nixpkgs, or if +you want to override some of its attributes), you will usually have to +pass `pkg` instead of `pname` to `gradle.fetchDeps`. There are two ways +of doing it. + +The first is to add the derivation arguments required for getting the +package. Using the pdftk example above: + +```nix +{ lib +, stdenv +# ... +, pdftk +}: + +stdenv.mkDerivation (finalAttrs: { + # ... + mitmCache = gradle.fetchDeps { + pkg = pdftk; + data = ./deps.json; + }; +}) +``` + +This allows you to `override` any arguments of the `pkg` used for +the update script (for example, `pkg = pdftk.override { enableSomeFlag = +true };`), so this is the preferred way. + +The second is to create a `let` binding for the package, like this: + +```nix +let self = stdenv.mkDerivation { + # ... + mitmCache = gradle.fetchDeps { + pkg = self; + data = ./deps.json; + }; +}; in self +``` + +This is useful if you can't easily pass the derivation as its own +argument, or if your `mkDerivation` call is responsible for building +multiple packages. + +In the former case, the update script will stay the same even if the +derivation is called with different arguments. In the latter case, the +update script will change depending on the derivation arguments. It's up +to you to decide which one would work best for your derivation. + +## Update Script {#gradle-update-script} + +The update script does the following: + +- Build the derivation's source via `pkgs.srcOnly` +- Enter a `nix-shell` for the derivation in a `bwrap` sandbox (the + sandbox is only used on Linux) +- Set the `IN_GRADLE_UPDATE_DEPS` environment variable to `1` +- Run the derivation's `unpackPhase`, `patchPhase`, `configurePhase` +- Run the derivation's `gradleUpdateScript` (the Gradle setup hook sets + a default value for it, which runs `preBuild`, `preGradleUpdate` + hooks, fetches the dependencies using `gradleUpdateTask`, and finally + runs the `postGradleUpdate` hook) +- Finally, store all of the fetched files' hashes in the lockfile. They + may be `.jar`/`.pom` files from Maven repositories, or they may be + files otherwise used for building the package. + +`fetchDeps` takes the following arguments: + +- `attrPath` - the path to the package in nixpkgs (for example, + `"javaPackages.openjfx22"`). Used for update script metadata. +- `pname` - an alias for `attrPath` for convenience. This is what you + will generally use instead of `pkg` or `attrPath`. +- `pkg` - the package to be used for fetching the dependencies. Defaults + to `getAttrFromPath (splitString "." attrPath) pkgs`. +- `bwrapFlags` - allows you to override bwrap flags (only relevant for + downstream, non-nixpkgs projects) +- `data` - path to the dependencies lockfile (can be relative to the + package, can be absolute). In nixpkgs, it's discouraged to have the + lockfiles be named anything other `deps.json`, consider creating + subdirectories if your package requires multiple `deps.json` files. + +## Environment {#gradle-environment} + +The Gradle setup hook accepts the following environment variables: + +- `mitmCache` - the MITM proxy cache imported using `gradle.fetchDeps` +- `gradleFlags` - command-line flags to be used for every Gradle + invocation (this simply registers a function that uses the necessary + flags). + - You can't use `gradleFlags` for flags that contain spaces, in that + case you must add `gradleFlagsArray+=("-flag with spaces")` to the + derivation's bash code instead. + - If you want to build the package using a specific Java version, you + can pass `"-Dorg.gradle.java.home=${jdk}"` as one of the flags. +- `gradleBuildTask` - the Gradle task (or tasks) to be used for building + the package. Defaults to `assemble`. +- `gradleCheckTask` - the Gradle task (or tasks) to be used for checking + the package if `doCheck` is set to `true`. Defaults to `test`. +- `gradleUpdateTask` - the Gradle task (or tasks) to be used for + fetching all of the package's dependencies in + `mitmCache.updateScript`. Defaults to `nixDownloadDeps`. +- `gradleUpdateScript` - the code to run for fetching all of the + package's dependencies in `mitmCache.updateScript`. Defaults to + running the `preBuild` and `preGradleUpdate` hooks, running the + `gradleUpdateTask`, and finally running the `postGradleUpdate` hook. +- `gradleInitScript` - path to the `--init-script` to pass to Gradle. By + default, a simple init script that enables reproducible archive + creation is used. + - Note that reproducible archives might break some builds. One example + of an error caused by it is `Could not create task ':jar'. Replacing + an existing task that may have already been used by other plugins is + not supported`. If you get such an error, the easiest "fix" is + disabling reproducible archives altogether by setting + `gradleInitScript` to something like `writeText + "empty-init-script.gradle" ""` +- `enableParallelBuilding` / `enableParallelChecking` / + `enableParallelUpdating` - pass `--parallel` to Gradle in the + build/check phase or in the update script. Defaults to true. If the + build fails for mysterious reasons, consider setting this to false. +- `dontUseGradleConfigure` / `dontUseGradleBuild` / `dontUseGradleCheck` + \- force disable the Gradle setup hook for certain phases. + - Note that if you disable the configure hook, you may face issues + such as `Failed to load native library 'libnative-platform.so'`, + because the configure hook is responsible for initializing Gradle. diff --git a/pkgs/development/tools/build-managers/gradle/README.md b/pkgs/development/tools/build-managers/gradle/README.md new file mode 100644 index 000000000000..f52779132c81 --- /dev/null +++ b/pkgs/development/tools/build-managers/gradle/README.md @@ -0,0 +1,245 @@ +# Gradle Setup Hook + +## Introduction + +Gradle build scripts are written in a DSL, computing the list of Gradle +dependencies is a turing-complete task, not just in theory but in +practice. Fetching all of the dependencies often requires building some +native code, running some commands to check the host platform, or just +fetching some files using either JVM code or commands like `curl` or +`wget`. + +This practice is widespread and isn't considered a bad practice in the +Java world, so all we can do is run Gradle to check what dependencies +end up being fetched, and allow derivation authors to apply workarounds +so they can run the code necessary for fetching the dependencies our +script doesn't fetch. + +"Run Gradle to check what dependencies end up being fetched" isn't a +straightforward task. For example, Gradle usually uses Maven +repositories, which have features such as "snapshots", a way to always +use the latest version of a dependency as opposed to a fixed version. +Obviously, this is horrible for reproducibility. Additionally, Gradle +doesn't offer a way to export the list of dependency URLs and hashes (it +does in a way, but it's far from being complete, and as such is useless +for nixpkgs). Even if did, it would be annoying to use considering +fetching non-Gradle dependendencies in Gradle scripts is commonplace. + +That's why the setup hook uses mitm-cache, a program designed for +intercepting all HTTP requests, recording all the files that were +accessed, creating a Nix derivation with all of them, and then allowing +the Gradle derivation to access these files. + +## Maven Repositories + +(Reference: [Repository +Layout](https://cwiki.apache.org/confluence/display/MAVENOLD/Repository+Layout+-+Final)) + +Most of Gradle dependencies are fetched from Maven repositories. For +each dependency, Gradle finds the first repo where it can successfully +fetch that dependency, and uses that repo for it. Different repos might +actually return different files for the same artifact because of e.g. +pom normalization. Different repos may be used for the same artifact +even across a single package (for example, if two build scripts define +repositories in a different order). + +The artifact metadata is specified in a .pom file, and the artifacts +themselves are typically .jar files. The URL format is as follows: + +`////-[-].` + +For example: + +- `https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/2.0.9/slf4j-api-2.0.9.pom` +- `https://oss.sonatype.org/content/groups/public/com/tobiasdiez/easybind/2.2.1-SNAPSHOT/easybind-2.2.1-20230117.075740-16.pom` + +Where: + +- `` is the repo base (`https://repo.maven.apache.org/maven2`) +- `` is the group ID with dots replaced with slashes + (`org.slf4j` -> `org/slf4j`) +- `` is the artifact ID (`slf4j-api`) +- `` is the artifact version (`2.0.9` for normal + artifacts, `2.2.1-SNAPSHOT` for snapshots) +- `` is the artifact version - can be either `` + or `--` (`2.0.9` for normal + artifacts, and either `2.2.1-SNAPSHOT` or `2.2.1-20230117.075740-16` + for snapshots) + - `` - `` without the `-SNAPSHOT` suffix + - `` - artifact build timestamp in the `YYYYMMDD.HHMMSS` + format (UTC) + - `` - a counter that's incremented by 1 for each new + snapshot build +- `` is an optional classifier for allowing a single .pom to + refer to multiple .jar files. .pom files don't have classifiers, as + they describe metadata. +- `` is the extension. .pom + +Note that the artifact ID can contain `-`, so you can't extract the +artifact ID and version from just the file name. + +Additionally, the files in the repository may have associated signature +files, formed by appending `.asc` to the filename, and hashsum files, +formed by appending `.md5` or `.sha1` to the filename. The signatures +are harmless, but the `.md5`/`.sha1` files are rejected. + +The reasoning is as follows - consider two files `a.jar` and `b.jar`, +that have the same hash. Gradle will fetch `a.jar.sha1`, find out that +it hasn't yet downloaded a file with this hash, and then fetch `a.jar`, +and finally download `b.jar.sha1`, locate it in its cache, and then +*not* download `b.jar`. This means `b.jar` won't be stored in the MITM +cache. Then, consider that on a later invocation, the fetching order +changed, whether it was because of a running on different system, +changed behavior after a Gradle update, or any other source of +nondeterminism - `b.jar` is fetched before `a.jar`. Gradle will first +fetch `b.jar.sha1`, not find it in its cache, attempt to fetch `b.jar`, +and fail, as the cache doesn't have that file. + +For the same reason, the proxy strips all checksum/etag headers. An +alternative would be to make the proxy remember previous checksums and +etags, but that would complicate the implementation - however, such a +feature can be implemented if necessary. Note that checksum/etag header +stripping is hardcoded, but `.md5/.sha1` file rejection is configured +via CLI arguments. + +**Caveat**: Gradle .module files also contain file hashes, in md5, sha1, +sha256, sha512 formats. It posed no problem as of yet, but it might in +the future. If it does pose problems, the deps derivation code can be +extended to find all checksums in .module files and copy existing files +there if their hash matches. + +## Snapshots + +Snapshots are a way to publish the very latest, unstable version of a +dependency that constantly changes. Any project that depends on a +snapshot will depend on this rolling version, rather than a fixed +version. It's easy to understand why this is a bad idea for reproducible +builds. Still, they can be dealt with by the logic in `gradle.fetchDeps` +and `gradle.updateDeps`. + +First, as you can see above, while normal artifacts have the same +`base-version` and `version`, for snapshots it usually (but not +necessarily) differs. + +Second, for figuring out where to download the snapshot, Gradle consults +`maven-metadata.xml`. With that in mind... + +## Maven Metadata + +(Reference: [Maven +Metadata](https://maven.apache.org/repositories/metadata.html), +[Metadata](https://maven.apache.org/ref/3.9.8/maven-repository-metadata/repository-metadata.html) + +Maven metadata files are called `maven-metadata.xml`. + +There are three levels of metadata: "G level", "A level", "V level", +representing group, artifact, or version metadata. + +G level metadata is currently unsupported. It's only used for Maven +plugins, which Gradle presumably doesn't use. + +A level metadata is used for getting the version list for an artifact. +It's an xml with the following items: + +- `` - group ID +- `` - artifact ID +- `` + - `` - the very latest base version (e.g. `2.2.1-SNAPSHOT`) + - `` - the latest non-snapshot version + - `` - the version list, each in a `` tag + - `` - the metadata update timestamp (UTC, + `YYYYMMDDHHMMSS`) + +V level metadata is used for listing the snapshot versions. It has the +following items: + +- `` - group ID +- `` - artifact ID +- `` + - `` - the metadata update timestamp (UTC, + `YYYYMMDDHHMMSS`) + - `` - info about the latest snapshot version + - `` - build timestamp (UTC, `YYYYMMDD.HHMMSS`) + - `` - build number + - `` - the list of all available snapshot file info, + each info is enclosed in a `` + - `` - classifier (optional) + - `` - file extension + - `` - snapshot version (as opposed to base version) + - `` - snapshot build timestamp (UTC, `YYYYMMDDHHMMSS`) + +## Lockfile Format + +The mitm-cache lockfile format is described in the [mitm-cache +README](https://github.com/chayleaf/mitm-cache#readme). + +The nixpkgs Gradle lockfile format is more complicated: + +```json +{ + "!comment": "This is a nixpkgs Gradle dependency lockfile. For more details, refer to the Gradle section in the nixpkgs manual.", + "!version": 1, + "https://oss.sonatype.org/content/repositories/snapshots/com/badlogicgames/gdx-controllers": { + "gdx-controllers#gdx-controllers-core/2.2.4-20231021.200112-6/SNAPSHOT": { + + "jar": "sha256-Gdz2J1IvDJFktUD2XeGNS0SIrOyym19X/+dCbbbe3/U=", + "pom": "sha256-90QW/Mtz1jbDUhKjdJ88ekhulZR2a7eCaEJoswmeny4=" + }, + "gdx-controllers-core/2.2.4-SNAPSHOT/maven-metadata": { + "xml": { + "groupId": "com.badlogicgames.gdx-controllers" + } + } + }, + "https://repo.maven.apache.org/maven2": { + "com/badlogicgames/gdx#gdx-backend-lwjgl3/1.12.1": { + "jar": "sha256-B3OwjHfBoHcJPFlyy4u2WJuRe4ZF/+tKh7gKsDg41o0=", + "module": "sha256-9O7d2ip5+E6OiwN47WWxC8XqSX/mT+b0iDioCRTTyqc=", + "pom": "sha256-IRSihaCUPC2d0QzB0MVDoOWM1DXjcisTYtnaaxR9SRo=" + } + } +} +``` + +`!comment` is a human-readable description explaining what the file is, +`!version` is the lockfile version (note that while it shares the name +with mitm-cache's `!version`, they don't actually have to be in sync and +can be bumped separately). + +The other keys are parts of a URL. Each URL is split into three parts. +They are joined like this: `/.`. + +Some URLs may have a `#` in them. In that case, the part after `#` is +parsed as `#/[/SNAPSHOT][/].` and +expanded into +`//-[-].`. + +Each URL has a value associated with it. The value may be: + +- an SRI hash (string) +- for `maven-metadata.xml` - an attrset containing the parts of the + metadata that can't be generated in Nix code (e.g. `groupId`, which is + challenging to parse from a URL because it's not always possible to + discern where the repo base ends and the group ID begins). + +`compress-deps-json.py` converts the JSON from mitm-cache format into +nixpkgs Gradle lockfile format. `fetch.nix` does the opposite. + +## Security Considerations + +Lockfiles won't be human-reviewed. They must be tampering-resistant. +That's why it's imperative that nobody can inject their own contents +into the lockfiles. + +This is achieved in a very simple way - the `deps.json` only contains +the following: + +- `maven-metadata.xml` URLs and small pieces of the contained metadata + (most of it will be generated in Nix, i.e. the area of injection is + minimal, and the parts that aren't generated in Nix are validated). +- artifact/other file URLs and associated hashes (Nix will complain if + the hash doesn't match, and Gradle won't even access the URL if it + doesn't match) + +Please be mindful of the above when working on Gradle support for +nixpkgs. diff --git a/pkgs/development/tools/build-managers/gradle/compress-deps-json.py b/pkgs/development/tools/build-managers/gradle/compress-deps-json.py new file mode 100644 index 000000000000..6397892835ad --- /dev/null +++ b/pkgs/development/tools/build-managers/gradle/compress-deps-json.py @@ -0,0 +1,163 @@ +import json +import sys + +from typing import Dict, Set + +# this compresses MITM URL lists with Gradle-specific optimizations +# specifically, it splits each url into up to 3 parts - they will be +# concatenated like part1/part2.part3 or part1.part2 +# part3 is simply always the file extension, but part1 and part2 is +# optimized using special heuristics +# additionally, if part2 ends with /a/b/{a}-{b}, the all occurences of +# /{a}/{b}/ are replaced with # +# finally, anything that ends with = is considered SHA256, anything that +# starts with http is considered a redirect URL, anything else is +# considered text + +with open(sys.argv[1], "rt") as f: + data: dict = json.load(f) + +new_data: Dict[str, Dict[str, Dict[str, dict]]] = {} + +for url, info in data.items(): + if url == "!version": + continue + ext, base = map(lambda x: x[::-1], url[::-1].split(".", 1)) + if base.endswith(".tar"): + base = base[:-4] + ext = "tar." + ext + # special logic for Maven repos + if ext in ["jar", "pom", "module"]: + comps = base.split("/") + if "-" in comps[-1]: + # convert base/name/ver/name-ver into base#name/ver + + filename = comps[-1] + name = comps[-3] + basever = comps[-2] + ver = basever + is_snapshot = ver.endswith("-SNAPSHOT") + if is_snapshot: + ver = ver.removesuffix("-SNAPSHOT") + if filename.startswith(f"{name}-{ver}"): + if is_snapshot: + if filename.startswith(f"{name}-{ver}-SNAPSHOT"): + ver += "-SNAPSHOT" + else: + ver += "-".join( + filename.removeprefix(f"{name}-{ver}").split("-")[:3] + ) + comp_end = comps[-1].removeprefix(f"{name}-{ver}") + else: + ver, name, comp_end = None, None, None + if name and ver and (not comp_end or comp_end.startswith("-")): + base = "/".join(comps[:-1]) + "/" + base = base.replace(f"/{name}/{basever}/", "#") + base += f"{name}/{ver}" + if is_snapshot: + base += "/SNAPSHOT" + if comp_end: + base += "/" + comp_end[1:] + scheme, rest = base.split("/", 1) + if scheme not in new_data.keys(): + new_data[scheme] = {} + if rest not in new_data[scheme].keys(): + new_data[scheme][rest] = {} + if "hash" in info.keys(): + new_data[scheme][rest][ext] = info["hash"] + elif "text" in info.keys() and ext == "xml": + # nix code in fetch-deps.nix will autogenerate metadata xml files groupId + # is part of the URL, but it can be tricky to parse as we don't know the + # exact repo base, so take it from the xml and pass it to nix + xml = "".join(info["text"].split()) + new_data[scheme][rest][ext] = { + "groupId": xml.split("")[1].split("")[0], + } + if "" in xml: + new_data[scheme][rest][ext]["release"] = xml.split("")[1].split( + "" + )[0] + if "" in xml: + latest = xml.split("")[1].split("")[0] + if latest != new_data[scheme][rest][ext].get("release"): + new_data[scheme][rest][ext]["latest"] = latest + if "" in xml: + new_data[scheme][rest][ext]["lastUpdated"] = xml.split("")[ + 1 + ].split("")[0] + else: + raise Exception("Unsupported key: " + repr(info)) + +# At this point, we have a map by part1 (initially the scheme), part2 (initially a +# slash-separated string without the scheme and with potential # substitution as +# seen above), extension. +# Now, push some segments from "part2" into "part1" like this: +# https # part1 +# domain1/b # part2 +# domain1/c +# domain2/a +# domain2/c +# -> +# https/domain1 # part1 +# b # part2 +# c +# https/domain2 # part1 +# a # part2 +# c +# This helps reduce the lockfile size because a Gradle project will usually use lots +# of files from a single Maven repo + +data = new_data +changed = True +while changed: + changed = False + new_data = {} + for part1, info1 in data.items(): + starts: Set[str] = set() + # by how many bytes the file size will be increased (roughly) + lose = 0 + # by how many bytes the file size will be reduced (roughly) + win = 0 + # how many different initial part2 segments there are + count = 0 + for part2, info2 in info1.items(): + if "/" not in part2: + # can't push a segment from part2 into part1 + count = 0 + break + st = part2.split("/", 1)[0] + if st not in starts: + lose += len(st) + 1 + count += 1 + starts.add(st) + win += len(st) + 1 + if count == 0: + new_data[part1] = info1 + continue + # only allow pushing part2 segments into path1 if *either*: + # - the domain isn't yet part of part1 + # - the initial part2 segment is always the same + if count != 1 and "." in part1: + new_data[part1] = info1 + continue + # some heuristics that may or may not work well (originally this was + # used when the above if wasn't here, but perhaps it's useless now) + lose += (count - 1) * max(0, len(part1) - 4) + if win > lose or ("." not in part1 and win >= lose): + changed = True + for part2, info2 in info1.items(): + st, part3 = part2.split("/", 1) + new_part1 = part1 + "/" + st + if new_part1 not in new_data.keys(): + new_data[new_part1] = {} + new_data[new_part1][part3] = info2 + else: + new_data[part1] = info1 + data = new_data + +new_data["!comment"] = "This is a nixpkgs Gradle dependency lockfile. For more details, refer to the Gradle section in the nixpkgs manual." # type: ignore +new_data["!version"] = 1 # type: ignore + +with open(sys.argv[2], "wt") as f: + json.dump(new_data, f, sort_keys=True, indent=1) + f.write("\n") diff --git a/pkgs/development/tools/build-managers/gradle/default.nix b/pkgs/development/tools/build-managers/gradle/default.nix index 32bd64d2f0e3..7567b3e3dba1 100644 --- a/pkgs/development/tools/build-managers/gradle/default.nix +++ b/pkgs/development/tools/build-managers/gradle/default.nix @@ -130,6 +130,7 @@ rec { ''; }; }; + passthru.jdk = defaultJava; meta = with lib; { inherit platforms; @@ -179,4 +180,43 @@ rec { hash = "sha256-PiQCKFON6fGHcqV06ZoLqVnoPW7zUQFDgazZYxeBOJo="; defaultJava = jdk11; }; + + wrapGradle = { + lib, callPackage, mitm-cache, substituteAll, symlinkJoin, concatTextFile, makeSetupHook + }: + gradle-unwrapped: + lib.makeOverridable (args: + let + gradle = gradle-unwrapped.override args; + in symlinkJoin { + name = "gradle-${gradle.version}"; + + paths = [ + (makeSetupHook { name = "gradle-setup-hook"; } (concatTextFile { + name = "setup-hook.sh"; + files = [ + (mitm-cache.setupHook) + (substituteAll { + src = ./setup-hook.sh; + # jdk used for keytool + inherit (gradle) jdk; + init_script = ./init-build.gradle; + }) + ]; + })) + gradle + mitm-cache + ]; + + passthru = { + fetchDeps = callPackage ./fetch-deps.nix { inherit mitm-cache; }; + inherit (gradle) jdk; + }; + + meta = gradle.meta // { + # prefer normal gradle/mitm-cache over this wrapper, this wrapper only provides the setup hook + # and passthru + priority = (gradle.meta.priority or 0) + 1; + }; + }) { }; } diff --git a/pkgs/development/tools/build-managers/gradle/fetch-deps.nix b/pkgs/development/tools/build-managers/gradle/fetch-deps.nix new file mode 100644 index 000000000000..f4485609f65f --- /dev/null +++ b/pkgs/development/tools/build-managers/gradle/fetch-deps.nix @@ -0,0 +1,222 @@ +{ mitm-cache +, lib +, pkgs +, stdenv +, callPackage +}: + +let + getPkg = attrPath: + lib.getAttrFromPath + (lib.splitString "." (toString attrPath)) + pkgs; +in +# the derivation to fetch/update deps for +{ pkg ? getPkg attrPath +, pname ? null +, attrPath ? pname +# bwrap flags for the update script (this will be put in bash as-is) +# this is relevant for downstream users +, bwrapFlags ? "--ro-bind \"$PWD\" \"$PWD\"" +# deps path (relative to the package directory, or absolute) +, data +# redirect stdout to stderr to allow the update script to be used with update script combinators +, silent ? true +, useBwrap ? stdenv.isLinux +} @ attrs: + +let + data' = builtins.removeAttrs + (if builtins.isPath data then lib.importJSON data + else if builtins.isString data then lib.importJSON "${dirOf pkg.meta.position}/${data}" + else data) + [ "!comment" "!version" ]; + + parseArtifactUrl = url: let + extension = lib.last (lib.splitString "." url); + splitUrl = lib.splitString "/" url; + artifactId = builtins.elemAt splitUrl (builtins.length splitUrl - 3); + baseVer = builtins.elemAt splitUrl (builtins.length splitUrl - 2); + filename = builtins.elemAt splitUrl (builtins.length splitUrl - 1); + filenameNoExt = lib.removeSuffix ".${extension}" filename; + verCls = lib.removePrefix "${artifactId}-" filenameNoExt; + in rec { + inherit artifactId baseVer filename extension; + isSnapshot = lib.hasSuffix "-SNAPSHOT" baseVer; + version = + if isSnapshot && !lib.hasPrefix "SNAPSHOT" verCls + then builtins.concatStringsSep "-" (lib.take 3 (lib.splitString "-" verCls)) + else baseVer; + classifier = + if verCls == version then null + else lib.removePrefix "${version}-" verCls; + # for snapshots + timestamp = builtins.elemAt (lib.splitString "-" version) 1; + buildNum = builtins.elemAt (lib.splitString "-" version) 2; + }; + + parseMetadataUrl = url: let + xmlBase = lib.removeSuffix "/maven-metadata.xml" url; + vMeta = lib.hasSuffix "-SNAPSHOT" xmlBase; + splitBase = lib.splitString "/" xmlBase; + in + if vMeta then { + vMeta = true; + baseVer = builtins.elemAt splitBase (builtins.length splitBase - 1); + artifactId = builtins.elemAt splitBase (builtins.length splitBase - 2); + } else { + vMeta = false; + baseVer = null; + artifactId = builtins.elemAt splitBase (builtins.length splitBase - 1); + }; + + extractHashArtifact = afterHash: let + nameVer = builtins.match "([^/]*)/([^/]*)(/SNAPSHOT)?(/.*)?" afterHash; + artifactId = builtins.elemAt nameVer 0; + version = builtins.elemAt nameVer 1; + isSnapshot = builtins.elemAt nameVer 2 != null; + cls = builtins.elemAt nameVer 3; + in rec { + inherit artifactId version isSnapshot; + baseVer = + if !isSnapshot then version + else builtins.head (builtins.match "(.*)-([^-]*)-([^-]*)" version) + "-SNAPSHOT"; + classifier = + if cls == null then null + else lib.removePrefix "/" cls; + clsSuf = + if classifier == null then "" + else "-${classifier}"; + }; + + # replace base#name/ver with base/name/ver/name-ver + decompressNameVer = prefix: let + splitHash = lib.splitString "#" (builtins.concatStringsSep "/" prefix); + inherit (extractHashArtifact (lib.last splitHash)) artifactId baseVer version clsSuf; + in + if builtins.length splitHash == 1 then builtins.head splitHash + else builtins.concatStringsSep "/${artifactId}/${baseVer}/" (lib.init splitHash ++ [ "${artifactId}-${version}${clsSuf}" ]); + + # `visit` all elements in attrs and merge into a set + # attrs will be passed as parent1, parent1 will be passed as parent2 + visitAttrs = parent1: prefix: attrs: + builtins.foldl' + (a: b: a // b) + {} + (lib.mapAttrsToList (visit parent1 attrs prefix) attrs); + + # convert a compressed deps.json into an uncompressed json used for mitm-cache.fetch + visit = parent2: parent1: prefix: k: v: + # groupId being present means this is a metadata xml "leaf" and we shouldn't descend further + if builtins.isAttrs v && !v?groupId + then visitAttrs parent1 (prefix ++ [k]) v + else let + url = "${decompressNameVer prefix}.${k}"; + in { + ${url} = + if builtins.isString v then { hash = v; } + else { + text = let + xmlBase = lib.removeSuffix "/maven-metadata.xml" url; + meta = parseMetadataUrl url // v; + inherit (meta) groupId vMeta artifactId baseVer; + + fileList = builtins.filter (x: lib.hasPrefix xmlBase x && x != url) (builtins.attrNames finalData); + jarPomList = map parseArtifactUrl fileList; + sortedJarPomList = + lib.sort + (a: b: lib.splitVersion a.version < lib.splitVersion b.version) + jarPomList; + + uniqueVersionFiles = + builtins.map ({ i, x }: x) + (builtins.filter ({ i, x }: i == 0 || (builtins.elemAt sortedJarPomList (i - 1)).version != x.version) + (lib.imap0 (i: x: { inherit i x; }) sortedJarPomList)); + uniqueVersions' = map (x: x.version) uniqueVersionFiles; + releaseVersions = map (x: x.version) (builtins.filter (x: !x.isSnapshot) uniqueVersionFiles); + latestVer = v.latest or v.release or (lib.last uniqueVersions'); + releaseVer = v.release or (lib.last releaseVersions); + + # The very latest version isn't necessarily used by Gradle, so it may not be present in the MITM data. + # In order to generate better metadata xml, if the latest version is known but wasn't fetched by Gradle, + # add it anyway. + uniqueVersions = + uniqueVersions' + ++ lib.optional (!builtins.elem releaseVer uniqueVersions') releaseVer + ++ lib.optional (!builtins.elem latestVer uniqueVersions' && releaseVer != latestVer) latestVer; + + lastUpdated = v.lastUpdated or + (if vMeta then builtins.replaceStrings ["."] [""] snapshotTs + else "20240101123456"); + + # the following are only used for snapshots + snapshotTsAndNum = lib.splitString "-" latestVer; + snapshotTs = builtins.elemAt snapshotTsAndNum 1; + snapshotNum = lib.last snapshotTsAndNum; + + indent = x: s: builtins.concatStringsSep "\n" (map (s: x + s) (lib.splitString "\n" s)); + containsSpecialXmlChars = s: builtins.match ''.*[<>"'&].*'' s != null; + in + # make sure all user-provided data is safe + assert lib.hasInfix "${builtins.replaceStrings ["."] ["/"] groupId}/${artifactId}" url; + assert !containsSpecialXmlChars groupId; + assert !containsSpecialXmlChars lastUpdated; + if vMeta then '' + + + ${groupId} + ${artifactId} + ${baseVer} + + + ${snapshotTs} + ${snapshotNum} + + ${lastUpdated} + + ${builtins.concatStringsSep "\n" (map (x: indent " " '' + ${ + lib.optionalString + (x.classifier != null) + "\n ${x.classifier}" + } + ${x.extension} + ${x.version} + ${builtins.replaceStrings ["."] [""] x.timestamp} + '') sortedJarPomList)} + + + + '' + else + assert !containsSpecialXmlChars latestVer; + assert !containsSpecialXmlChars releaseVer; + '' + + + ${groupId} + ${artifactId} + + ${latestVer} + ${releaseVer} + + ${builtins.concatStringsSep "\n" (map (x: " ${x}") uniqueVersions)} + + ${lastUpdated} + + + ''; + }; + }; + + finalData = visitAttrs {} [] data'; +in + mitm-cache.fetch { + name = "${pkg.pname or pkg.name}-deps"; + data = finalData // { "!version" = 1; }; + passthru = lib.optionalAttrs (!builtins.isAttrs data) { + updateScript = callPackage ./update-deps.nix { } { + inherit pkg pname attrPath bwrapFlags data silent useBwrap; + }; + }; + } diff --git a/pkgs/development/tools/build-managers/gradle/init-build.gradle b/pkgs/development/tools/build-managers/gradle/init-build.gradle new file mode 100644 index 000000000000..3c3ee8162449 --- /dev/null +++ b/pkgs/development/tools/build-managers/gradle/init-build.gradle @@ -0,0 +1,8 @@ +gradle.projectsLoaded { + rootProject.allprojects { + tasks.withType(AbstractArchiveTask) { + preserveFileTimestamps = false + reproducibleFileOrder = true + } + } +} diff --git a/pkgs/development/tools/build-managers/gradle/init-deps.gradle b/pkgs/development/tools/build-managers/gradle/init-deps.gradle new file mode 100644 index 000000000000..2cb577efaec9 --- /dev/null +++ b/pkgs/development/tools/build-managers/gradle/init-deps.gradle @@ -0,0 +1,10 @@ +gradle.projectsLoaded { + rootProject.allprojects { + task nixDownloadDeps { + doLast { + configurations.findAll{it.canBeResolved}.each{it.resolve()} + buildscript.configurations.findAll{it.canBeResolved}.each{it.resolve()} + } + } + } +} diff --git a/pkgs/development/tools/build-managers/gradle/setup-hook.sh b/pkgs/development/tools/build-managers/gradle/setup-hook.sh new file mode 100644 index 000000000000..a16742414b12 --- /dev/null +++ b/pkgs/development/tools/build-managers/gradle/setup-hook.sh @@ -0,0 +1,70 @@ +gradleConfigureHook() { + if [ -z "${GRADLE_USER_HOME-}" ]; then + GRADLE_USER_HOME="$(mktemp -d)" + fi + export GRADLE_USER_HOME + export TERM=dumb + gradleFlagsArray+=(--no-daemon --console plain --init-script "${gradleInitScript:-@init_script@}") + if [ -n "${MITM_CACHE_CA-}" ]; then + if [ -z "${MITM_CACHE_KEYSTORE-}" ]; then + MITM_CACHE_KEYSTORE="$MITM_CACHE_CERT_DIR/keystore" + MITM_CACHE_KS_PWD="$(head -c10 /dev/random | base32)" + echo y | @jdk@/bin/keytool -importcert -file "$MITM_CACHE_CA" -alias alias -keystore "$MITM_CACHE_KEYSTORE" -storepass "$MITM_CACHE_KS_PWD" + fi + gradleFlagsArray+=(-Dhttp.proxyHost="$MITM_CACHE_HOST" -Dhttp.proxyPort="$MITM_CACHE_PORT") + gradleFlagsArray+=(-Dhttps.proxyHost="$MITM_CACHE_HOST" -Dhttps.proxyPort="$MITM_CACHE_PORT") + gradleFlagsArray+=(-Djavax.net.ssl.trustStore="$MITM_CACHE_KEYSTORE" -Djavax.net.ssl.trustStorePassword="$MITM_CACHE_KS_PWD") + else + gradleFlagsArray+=(--offline) + fi + if ! [[ -v enableParallelBuilding ]]; then + enableParallelBuilding=1 + fi + if ! [[ -v enableParallelChecking ]]; then + enableParallelChecking=1 + fi + if ! [[ -v enableParallelUpdating ]]; then + enableParallelUpdating=1 + fi +} + +gradle() { + command gradle $gradleFlags "${gradleFlagsArray[@]}" "$@" +} + +gradleBuildPhase() { + runHook preBuild + + gradle ${enableParallelBuilding:+--parallel} ${gradleBuildTask:-assemble} + + runHook postBuild +} + +gradleCheckPhase() { + runHook preCheck + + gradle ${enableParallelChecking:+--parallel} ${gradleCheckTask:-test} + + runHook postCheck +} + +gradleUpdateScript() { + runHook preBuild + runHook preGradleUpdate + + gradle ${enableParallelUpdating:+--parallel} ${gradleUpdateTask:-nixDownloadDeps} + + runHook postGradleUpdate +} + +if [ -z "${dontUseGradleConfigure-}" ]; then + preConfigureHooks+=(gradleConfigureHook) +fi + +if [ -z "${dontUseGradleBuild-}" ] && [ -z "${buildPhase-}" ]; then + buildPhase=gradleBuildPhase +fi + +if [ -z "${dontUseGradleCheck-}" ] && [ -z "${checkPhase-}" ]; then + checkPhase=gradleCheckPhase +fi diff --git a/pkgs/development/tools/build-managers/gradle/update-deps.nix b/pkgs/development/tools/build-managers/gradle/update-deps.nix new file mode 100644 index 000000000000..5d6222fb11c4 --- /dev/null +++ b/pkgs/development/tools/build-managers/gradle/update-deps.nix @@ -0,0 +1,122 @@ +{ lib +, runtimeShell +, srcOnly +, writeTextFile +, writeShellScript +, path +, bubblewrap +, coreutils +, curl +, jq +, mitm-cache +, nix +, openssl +, procps +, python3 +}: + +lib.makeOverridable +({ pkg, pname, attrPath, bwrapFlags, data, silent, useBwrap }: +let + keep = [ "MITM_CACHE_HOST" "MITM_CACHE_PORT" "MITM_CACHE_ADDRESS" "MITM_CACHE_CA" "MITM_CACHE_CERT_DIR" ]; + gradleScript = writeShellScript "gradle-commands.sh" '' + set -eo pipefail + export http_proxy="$MITM_CACHE_ADDRESS" + export https_proxy="$MITM_CACHE_ADDRESS" + export SSL_CERT_FILE="$MITM_CACHE_CA" + export NIX_SSL_CERT_FILE="$MITM_CACHE_CA" + export GRADLE_USER_HOME="$(${coreutils}/bin/mktemp -d)" + export IN_GRADLE_UPDATE_DEPS=1 + trap "${coreutils}/bin/rm -rf '$GRADLE_USER_HOME'" SIGINT SIGTERM ERR EXIT + cd "$(${coreutils}/bin/mktemp -d)" + ${coreutils}/bin/mkdir out + export out="$PWD/out" + trap "${coreutils}/bin/rm -rf '$PWD'" SIGINT SIGTERM ERR EXIT + source "$stdenv/setup" + phases="''${prePhases[*]:-} unpackPhase patchPhase ''${preConfigurePhases[*]:-} configurePhase gradleUpdateScript" genericBuild + ''; + source = srcOnly (pkg.overrideAttrs (old: { + mitmCache = ""; + gradleInitScript = ./init-deps.gradle; + })); + sourceDrvPath = builtins.unsafeDiscardOutputDependency source.drvPath; + nixShellKeep = lib.concatMapStringsSep " " (x: "--keep ${x}") keep; +in +writeTextFile { + name = "fetch-deps.sh"; + executable = true; + # see pkgs/common-updater/combinators.nix + derivationArgs.passthru = + { supportedFeatures = lib.optional silent "silent"; } + // lib.optionalAttrs (attrPath != null) { inherit attrPath; }; + text = '' + #!${runtimeShell} + set -eo pipefail + export PATH="${lib.makeBinPath [ + bubblewrap coreutils curl jq mitm-cache openssl + procps python3.pkgs.ephemeral-port-reserve + ]}:$PATH" + outPath="${ + # if this is an absolute path in nix store, use path relative to the store path + if lib.hasPrefix "${builtins.storeDir}/" (toString data) + then builtins.concatStringsSep "/" (lib.drop 1 (lib.splitString "/" (lib.removePrefix "${builtins.storeDir}/" (toString data)))) + # if this is an absolute path anywhere else, just use that path + else if lib.hasPrefix "/" (toString data) + then toString data + # otherwise, use a path relative to the package + else "${dirOf pkg.meta.position}/${data}" + }" + + pushd "$(mktemp -d)" >/dev/null + MITM_CACHE_DIR="$PWD" + trap "rm -rf '$MITM_CACHE_DIR'" SIGINT SIGTERM ERR EXIT + openssl genrsa -out ca.key 2048 + openssl req -x509 -new -nodes -key ca.key -sha256 -days 1 -out ca.cer -subj "/C=AL/ST=a/L=a/O=a/OU=a/CN=example.org" + export MITM_CACHE_HOST=127.0.0.1 + export MITM_CACHE_PORT="''${mitmCachePort:-$(ephemeral-port-reserve "$MITM_CACHE_HOST")}" + export MITM_CACHE_ADDRESS="$MITM_CACHE_HOST:$MITM_CACHE_PORT" + # forget all redirects - this makes the lockfiles predictable + # not only does this strip CDN URLs, but it also improves security - since the redirects aren't + # stored in the lockfile, a malicious actor can't change the redirect URL stored in the lockfile + mitm-cache \ + -l"$MITM_CACHE_ADDRESS" \ + record \ + --reject '\.(md5|sha(1|256|512:?):?)$' \ + --forget-redirects-from '.*' \ + --record-text '/maven-metadata\.xml$' >/dev/null 2>/dev/null & + MITM_CACHE_PID="$!" + # wait for mitm-cache to fully start + for i in {0..20}; do + ps -p "$MITM_CACHE_PID" >/dev/null || (echo "Failed to start mitm-cache" && exit 1) + curl -so/dev/null "$MITM_CACHE_ADDRESS" && break + [[ "$i" -eq 20 ]] && (echo "Failed to start mitm-cache" && exit 1) + sleep 0.5 + done + trap "kill '$MITM_CACHE_PID'" SIGINT SIGTERM ERR EXIT + export MITM_CACHE_CERT_DIR="$PWD" + export MITM_CACHE_CA="$MITM_CACHE_CERT_DIR/ca.cer" + popd >/dev/null + useBwrap="''${USE_BWRAP:-${toString useBwrap}}" + if [ -n "$useBwrap" ]; then + # bwrap isn't necessary, it's only used to prevent messy build scripts from touching ~ + bwrap \ + --unshare-all --share-net --clearenv --chdir / --setenv HOME /homeless-shelter \ + --tmpfs /home --bind /tmp /tmp --ro-bind /nix /nix --ro-bind /run /run --proc /proc --dev /dev \ + --ro-bind ${toString path} ${toString path} --bind "$MITM_CACHE_CERT_DIR" "$MITM_CACHE_CERT_DIR" \ + ${builtins.concatStringsSep " " (map (x: "--setenv ${x} \"\$${x}\"") keep)} \ + --setenv NIX_BUILD_SHELL bash ${bwrapFlags} ''${BWRAP_FLAGS:-} \ + -- ${nix}/bin/nix-shell --pure --run ${gradleScript} ${nixShellKeep} ${sourceDrvPath} + else + NIX_BUILD_SHELL=bash nix-shell --pure --run ${gradleScript} ${nixShellKeep} ${sourceDrvPath} + fi${lib.optionalString silent " >&2"} + kill -s SIGINT "$MITM_CACHE_PID" + for i in {0..20}; do + # check for valid json + if jq -e 1 "$MITM_CACHE_DIR/out.json" >/dev/null 2>/dev/null; then + exec ${python3.interpreter} ${./compress-deps-json.py} "$MITM_CACHE_DIR/out.json" "$outPath" + fi + sleep 1 + done + exit 1 + ''; +}) diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix index d26f1cdcd4af..1f6c9df2fc42 100644 --- a/pkgs/top-level/all-packages.nix +++ b/pkgs/top-level/all-packages.nix @@ -18575,10 +18575,17 @@ with pkgs; inherit jdk11 jdk17 jdk21; }; gradleGen = gradle-packages.gen; - gradle_6 = callPackage gradle-packages.gradle_6 { }; - gradle_7 = callPackage gradle-packages.gradle_7 { }; - gradle_8 = callPackage gradle-packages.gradle_8 { }; - gradle = gradle_8; + wrapGradle = callPackage gradle-packages.wrapGradle { }; + + gradle_6-unwrapped = callPackage gradle-packages.gradle_6 { }; + gradle_7-unwrapped = callPackage gradle-packages.gradle_7 { }; + gradle_8-unwrapped = callPackage gradle-packages.gradle_8 { }; + gradle-unwrapped = gradle_8-unwrapped; + + gradle_6 = wrapGradle gradle_6-unwrapped; + gradle_7 = wrapGradle gradle_7-unwrapped; + gradle_8 = wrapGradle gradle_8-unwrapped; + gradle = wrapGradle gradle-unwrapped; grcov = callPackage ../development/tools/misc/grcov { };