Commit Graph

196 Commits

Author SHA1 Message Date
Weijia Wang
74d102ce50 Merge branch 'master' into staging-next 2023-11-12 19:45:20 +01:00
SharzyL
b186e2d76e nix-prefetch-git: make it quiet
Some programs (e.g. nvfetcher) parse the output of nix-prefetch-git as
JSON. However, there is a pushd/popd command in nix-prefetch-git that
outputs the directory name, breaking the structure of the result JSON.
We suppressed it with `>/dev/null`.

Passingly fixes a mixuse of tab and spaces.
2023-11-12 15:20:16 +01:00
github-actions[bot]
33248a2caf
Merge master into staging-next 2023-11-12 00:02:14 +00:00
Elliot Cameron
3a99d1b642 nix-prefetch-git: respect NETRC
This script needs to support being run both as part of a `fetchgit`
derivation and as a standalone, command-line tool.

The use of `$NIX_BUILD_TOP` only works when used in `fetchgit` but not when
invoked as a standalone tool. Instead we try to respect `$NETRC` so that
the command-line invocation behaves more like standard tools and the
`fetchgit` derivation can explicitly set `$NETRC` when `netrcPhase` is used
to avoid all ambiguity.
2023-11-11 19:34:54 +01:00
github-actions[bot]
4aad65bfdb
Merge master into staging-next 2023-11-09 06:01:11 +00:00
Shogo Takata
d6e1c7cae6
fetchgit: shallow clone for submodules (#254172) 2023-11-09 04:32:13 +02:00
Weijia Wang
32da89a1a1 Merge branch 'master' into staging-next 2023-11-06 14:33:18 +01:00
K900
fa40d6457a nix-prefetch-git: download submodules with --progress 2023-11-05 20:49:03 +03:00
github-actions[bot]
de49fc16bc
Merge staging-next into staging 2023-10-16 12:01:54 +00:00
Domen Kožar
3c6cf39435
Merge pull request #249733 from Aleksanaa/nix-prefetch-git
nix-prefetch-git: fix make_deterministic_repo in submodules
2023-10-16 11:12:11 +01:00
github-actions[bot]
81871f4c87
Merge staging-next into staging 2023-10-08 12:01:52 +00:00
Sandro Jäckel
d09cd1d7d1 fetchgit: use $NIX_BUILD_TOP instead of hardcoding /build/
Co-authored-by: Bjørn Forsman <133602+bjornfor@users.noreply.github.com>
2023-10-08 00:10:59 +02:00
github-actions[bot]
2c70ec4a27
Merge staging-next into staging 2023-10-05 18:01:34 +00:00
Sandro Jäckel
025a028c47 fetchgit: fix private fetching via netrc
This fixes a regression introduced in https://github.com/NixOS/nixpkgs/pull/256628
which broke fetching with private = true through a netrc file.
Tested locally with a really special github enterprise.
2023-10-05 13:54:00 +00:00
Maximilian Bosch
8bc5104a6e
treewide: refactor .attrs.sh detection
When specifying the `builder` attribute in `stdenv.mkDerivation`, this
will be effectively transformed into

    builtins.derivation {
      builder = stdenv.shell;
      args = [ "-e" builder ];
    }

This also means that `default-builder.sh` is never sourced and as a
result it's not guaranteed that `$NIX_ATTRS_SH_FILE` is set to a correct
location[1].

Also, we need to source `.attrs.sh` to source `$stdenv`. So, the
following is done now:

* If `$NIX_ATTRS_SH_FILE` points to a correct location, then use it.
  Directly using `.attrs.sh` is problematic for `nix-shell(1)` usage
  (see previous commit for more context), so prefer the environment
  variable if possible.

* Otherwise, if `.attrs.sh` exists, then use it. See [1] for when this
  can happen.

* If neither applies, it can be assumed that `__structuredAttrs` is
  turned off and thus nothing needs to be done.

[1] It's possible that it doesn't exist at all - in case of Nix 2.3 or
    it can point to a wrong location on older Nix versions with a bug in
    `__structuredAttrs`.
2023-10-04 18:36:57 +02:00
Alyssa Ross
7d40fbbc04
nix-prefetch-git: ignore global and user git config
nix-prefetch-git is either run as part of a build, usually sandboxed,
or outside a build, unsandboxed, to prefetch something that will later
be used in a build.  It's important that the latter use produces
hashes that can be reproduced by the former.

One way that they can differ is if the user's git config does
something that changes the result of git clone.  I ran into this,
because my global git config automatically enables git-lfs, whereas
nix-prefetch-git otherwise only uses git-lfs if specifically
requested.  This led to very confusing hash mismatches.
2023-09-22 06:59:40 +00:00
Silvan Mosberger
7cbc8215fd
Merge pull request #252865 from emily-is-my-username/fix/fetchgit-deepclone
`fetchgit`: don't shallow clone if `deepClone` is requested
2023-09-06 14:08:06 +02:00
aleksana
981754aba8 nix-prefetch-git: fix make_deterministic_repo in submodules 2023-08-31 18:25:06 +08:00
github-actions[bot]
17a15a50ab
Merge master into staging-next 2023-07-21 06:01:05 +00:00
Theodore Ni
63804228e5
fetchgit: clean up md5 references 2023-07-20 21:24:13 -07:00
PedroHLC ☭
0142902fc4
nix-prefetch-git: add sri-hash to output 2023-07-09 14:46:58 -03:00
Emily
ad94541662 fetchgit: don't shallow clone if deepClone is requested
fixes: https://github.com/NixOS/nixpkgs/issues/179788
2023-07-01 00:10:26 +02:00
Mario Rodas
922d55bf2d fetchgit: add a default NIX_BUILD_CORES variable
NIX_BUILD_CORES might not be defined at runtime, so the command would
fail.
2023-06-24 04:20:00 +00:00
Domen Kožar
4f72b6af8b fetchgit: fetch submodules in parallel 2023-06-20 10:36:10 +01:00
Rick van Schijndel
6cdf7259d0
Merge pull request #235898 from panicgh/fetchgit-sparse-checkout-failhard
fetchgit: require sparseCheckout be a list of strings
2023-06-05 22:53:19 +02:00
Nicolas Benes
52f3a1c42c fetchgit: require sparseCheckout be a list of strings
Passing a (multi-line) string was deprecated in #200082 in favour of
list of strings, but still supported (with warning). Now, enforce use of
list of strings.
2023-06-04 10:04:31 +02:00
fetsorn
29f3903f62 nix-prefetch-git: fix typo 2023-05-31 04:00:32 +04:00
Ryan Hendrickson
0f6712d45a nix-prefetch-git: redirect git lfs to stderr 2023-05-21 09:05:49 +02:00
piegames
794b05a09d
Merge pull request #158968: Make fetchFromGitHub & friends overridable 2023-05-07 21:59:36 +02:00
piegames
a205795c8a treewide: Make some fetchers overridable 2023-04-22 22:29:23 +02:00
Artturi
ddc418f3b1
Merge pull request #187210 from pinn3/fetchgit/remove-outdated-comment 2023-04-19 03:49:19 +03:00
Felix Buehler
cdb39a86e0 treewide: use optionalString 2023-02-13 21:52:34 +01:00
Artturin
fe1c7a1945 treewide: remove usages of header and stopNest
they're obsolete
2023-01-16 00:08:12 +02:00
Artturi
1121c8fcb7
Merge pull request #185005 from amjoseph-nixpkgs/pr/fetchgit/fix-comment 2023-01-14 19:50:20 +02:00
Artturin
c01f509e44 treewide: source .attrs in builders
if theres a source $stdenv then this is needed

for structuredAttrs
2022-12-08 21:09:02 +02:00
Nicolas Benes
f6b07f0e2f fetchgit: make sparseCheckout a list of strings
The `sparseCheckout` argument allows the user to specify directories or
patterns of files, which Git uses to filter files it should check-out.

Git expects a multi-line string on stdin ("newline-delimited list", see
`git-sparse-checkout(1)`), but within nixpkgs it is more consistent to
use a list of strings instead. The list elements are joined to a
multi-line string only before passing it to the builder script.

A deprecation warning is emitted if a (multi-line) string is passed to
`sparseCheckout`, but for the time being it is still accepted.
2022-11-15 19:45:33 +01:00
pinn3
bab4cdd43a fetchgit: Remove comment regarding path needing to be a string
It was changed in 2019 to be of the actual path type, and has apparently
been working since then.

Closes #143846
2022-08-18 04:50:32 +02:00
Winter
dbd18a63a7 fetchgit: allow disabling cone mode for sparse checkouts, fix test 2022-08-14 23:03:07 +02:00
Adam Joseph
c8a1838d30 fetchgit: remove "security" from comment about "security risk"
Closes #178410
2022-08-03 05:07:34 -07:00
Sebastian Jordan
783e2ef46e Revert "nix-prefetch-git: Fix inconsistency with fetchgit regarding deepClone"
This reverts commit 1dfaad73ed.
2022-06-30 11:52:36 +00:00
Maximilian Bosch
e4bf881ee0
Merge pull request #177575 from seppeljordan/nix-prefetch-git-fix
nix-prefetch-git: Fix inconsistency with fetchgit regarding deepClone
2022-06-28 14:03:41 +02:00
Sebastian Jordan
1dfaad73ed nix-prefetch-git: Fix inconsistency with fetchgit regarding deepClone
The fetchgit function in nixpkgs sets the leaveDotGit argument to true
if deepClone is set to true. nix-prefetch-git did behave differently. It
would not assume --leave-dotGit if --deepClone is specified. With this
change the inconsistency is addressed by assuming --leave-dotGit if
--deepClone is specified.
2022-06-14 00:23:16 +02:00
Adam Joseph
5ed4944130 fetchgit: allow passing allowedRequisites through to stdenv.mkDerivation
When maintainers override stages of `fetchgit' (e.g. `postPatch`) it
is very easy for them to accidentally leak the outpath-hash of their
current `stdenv` into `fetchgit''s output, and therefore into the
value they paste into `sha256`.

This is a problem, because the resulting expression will break
whenever any change is made to `stdenv` or when anybody attempts to
build the expression on a different platform than the one used by the
original maintainer.

Almost as much of a problem is the fact that CI **does not catch**
these problems.  The `fetchgit` is run only once, then its output goes
into cachix, and all future builds (hydra, CI, ofborg) pull from
cachix.

Let's offer maintainers the option to check that they aren't making
this mistake, by passing through `allowedRequisites`.  The default
value is `null`, but it might be worth changing that at some point in
the future.

It is also sometimes difficult to communicate to package maintainers
why their expression is problematic.  Having `allowedRequisites`
passed through makes it easier to do this: "look, when I switch on
`allowedRequisites` your package breaks; are you sure you meant to
hardcode the hash today's `x86_64-linux.stdenv` into your expression?`

For an example use case, see https://github.com/NixOS/nixpkgs/pull/171223

The issue above is part of a larger problem with nixpkgs infra: there
large parts of cachix cannot be reproduced easily if they are lost.
Once something ends goes into cachix, we never ever again reverify the
procedure by which it was placed into cachix.
2022-06-11 12:42:33 -07:00
Jan Tojnar
7aae279ad9 unstableGitUpdater: fix updating fetchzip-based sources
a67950f20b added `url` attribute
from `fetchurl` and therefore also from `fetchzip`.
We previously relied on `url` from fetchgit-based fetchers
to find the repo URL but now it will just return tarballs
in the case of `fetchFrom{GitHub,GitLab}`.

Let’s add an attribute to `fetch{git,FromGitHub,FromGitLab}`
to expose a repo URL consistently.
2022-05-24 19:12:12 +02:00
Robert Hensing
c071530ca5 testers.invalidateFetcherByDrvHash: Move from top-level 2022-05-05 12:08:50 +02:00
Sandro Jäckel
538a5a28b7
fetchgit: allow passing thru meta 2022-04-18 03:06:02 +02:00
Maciej Krüger
ba83787f0d
fetchgit: split off deterministic-git functions 2022-02-19 14:33:20 +01:00
Zhong Jianxin
36cc03e151 fetchgit: Add test for sparseCheckout 2022-01-24 11:23:56 +08:00
Zhong Jianxin
d03a07d5a7 fetchgit: Support sparse checkout
This allow git checkout small parts of a large repo, and avoid fetching
unnecessary blobs from server.
2022-01-24 11:23:56 +08:00
Tomas Bravo
80ab168c4a nix-prefetch-git: fix incorrect mktemp usage 2021-12-22 16:45:18 +01:00