Add a new config `hook.forceStdoutToStderr` which allows enabling
extensions.hookStdoutToStderr by default at runtime, both for new
and existing repositories.
This makes it easier for users to enable hook parallelization for
hooks like pre-push by enforcing output consistency. See previous
commit for a more in-depth explanation & alternatives considered.
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
All hooks already redirect stdout to stderr with the exception of
pre-push which has a known user who depends on the separate stdout
versus stderr outputs (the git-lfs project).
The pre-push behavior was a surprise which we found out about after
causing a regression for git-lfs. Notably, it might not be the only
exception (it's the one we know about). There might be more.
This presents a challenge because stdout_to_stderr is required for
hook parallelization, so run-command can buffer and de-interleave
the hook outputs using ungroup=0, when hook.jobs > 1.
Introduce an extension to enforce consistency: all hooks merge stdout
into stderr and can be safely parallelized. This provides a clean
separation and avoids breaking existing stdout vs stderr behavior.
When this extension is disabled, the `hook.jobs` config has no
effect for pre-push, to prevent garbled (interleaved) parallel
output, so it runs sequentially like before.
Alternatives I've considered to this extension include:
1. Allowing pre-push to run in parallel with interleaved output.
2. Always running pre-push sequentially (no parallel jobs for it).
3. Making users (only git-lfs? maybe more?) fix their hooks to read
stderr not stdout.
Out of all these alternatives, I think this extension is the most
reasonable compromise, to not break existing users, allow pre-push
parallel jobs for those who need it (with correct outputs) and also
future-proofing in case there are any more exceptions to be added.
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Add a hook.<event>.jobs count config that allows users to override the
global hook.jobs setting for specific hook events.
This allows finer-grained control over parallelism on a per-event basis.
For example, to run `post-receive` hooks with up to 4 parallel jobs
while keeping other events at their global default:
[hook]
post-receive.jobs = 4
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Expose the parallel job count as a command-line flag so callers can
request parallelism without relying only on the hook.jobs config.
Add tests covering serial/parallel execution and TTY behaviour under
-j1 vs -jN.
Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Helped-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Several hooks are known to be inherently non-parallelizable, so initialize
them with RUN_HOOKS_OPT_INIT_FORCE_SERIAL. This pins jobs=1 and overrides
any hook.jobs or runtime -j flags.
These hooks are:
applypatch-msg, pre-commit, prepare-commit-msg, commit-msg, post-commit,
post-checkout, and push-to-checkout.
Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Hooks always run in sequential order due to the hardcoded jobs == 1
passed to run_process_parallel(). Remove that hardcoding to allow
users to run hooks in parallel (opt-in).
Users need to decide which hooks to run in parallel, by specifying
"parallel = true" in the config, because git cannot know if their
specific hooks are safe to run or not in parallel (for e.g. two hooks
might write to the same file or call the same program).
Some hooks are unsafe to run in parallel by design: these will marked
in the next commit using RUN_HOOKS_OPT_INIT_FORCE_SERIAL.
The hook.jobs config specifies the default number of jobs applied to all
hooks which have parallelism enabled.
Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Helped-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The hook.jobs config is a global way to set hook parallelization for
all hooks, in the sense that it is not per-event nor per-hook.
Finer-grained configs will be added in later commits which can override
it, for e.g. via a per-event type job options. Next commits will also
add to this item's documentation.
Parse hook.jobs config key in hook_config_lookup_all() and store its
value in hook_all_config_cb.jobs, then transfer it into
hook_config_cache.jobs after the config pass completes.
This is mostly plumbing and the cached value is not yet used.
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Next commits add a 'hook.jobs' config option of type 'unsigned int',
so add a helper to parse it since the API only supports int and ulong.
An alternative is to make 'hook.jobs' an 'int' or parse it as an 'int'
then cast it to unsigned, however it's better to use proper helpers for
the type. Using 'ulong' is another option which already has helpers, but
it's a bit excessive in size for just the jobs number.
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
There is an old pre-existing memory leak in repo_init() due to failing
to call clear_repository_format() in the error case.
It went undetected because a specific bug is required to trigger it:
enable a v1 extension in a repository with format v0. Obviously this
can only happen in a development environment, so it does not trigger
in normal usage, however the memleak is real and needs fixing.
Fix it by also calling clear_repository_format() in the error case.
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* ar/config-hook-cleanups: (31 commits)
hook: show disabled hooks in "git hook list"
hook: show config scope in git hook list
hook: refactor hook_config_cache from strmap to named struct
t1800: add test to verify hook execution ordering
hook: make consistent use of friendly-name in docs
hook: replace hook_list_clear() -> string_list_clear_func()
hook: detect & emit two more bugs
hook: rename cb_data_free/alloc -> hook_data_free/alloc
hook: fix minor style issues
hook: move unsorted_string_list_remove() to string-list.[ch]
builtin/receive-pack: avoid spinning no-op sideband async threads
hook: add -z option to "git hook list"
hook: allow out-of-repo 'git hook' invocations
hook: allow event = "" to overwrite previous values
hook: allow disabling config hooks
hook: include hooks from the config
hook: add "git hook list" command
hook: run a list of hooks to prepare for multihook support
hook: add internal state alloc/free callbacks
receive-pack: convert receive hooks to hook API
...
Disabled hooks were filtered out of the cache entirely, making them
invisible to "git hook list". Keep them in the cache with a new
"disabled" flag which is propagated to the respective struct hook.
"git hook list" now shows disabled hooks annotated with "(disabled)"
in the config order. With --show-scope, it looks like:
$ git hook list --show-scope pre-commit
linter (global)
no-leaks (local, disabled)
hook from hookdir
A disabled hook without a command issues a warning instead of the
fatal "hook.X.command must be configured" error. We could also throw
an error, however it seemd a bit excessive to me in this case.
Suggested-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Users running "git hook list" can see which hooks are configured but
have no way to tell at which config scope (local, global, system...)
each hook was defined.
Store the scope from ctx->kvi->scope in the single-pass config callback,
then carry it through the cache to the hook structs, so we can expose it
to users via the "git hook list --show-scope" flag, which mirrors the
existing git config --show-scope convention.
Without the flag the output is unchanged.
Example usage:
$ git hook list --show-scope pre-commit
linter (global)
no-leaks (local)
hook from hookdir
Traditional hooks from the hookdir are unaffected by --show-scope since
the config scope concept does not apply to them.
Suggested-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Replace the raw `struct strmap *hook_config_cache` in `struct repository`
with a `struct hook_config_cache` which wraps the strmap in a named field.
Replace the bare `char *command` util pointer stored in each string_list
item with a heap-allocated `struct hook_config_cache_entry` that carries
that command string.
This is just a refactoring with no behavior changes, to give the cache
struct room to grow so it can carry the additional hook metadata we'll
be adding in the following commits.
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
There is a documented expectation that configured hooks are
run before the hook from the hookdir. Add a test for it.
While at it, I noticed that `git hook list -h` runs twice
in the `git hook usage` test, so remove one invocation.
Suggested-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Both `name` and `friendly-name` is being used. Standardize on
`friendly-name` for consistency since name is rather generic,
even when used in the hooks namespace.
Suggested-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Replace the custom function with string_list_clear_func() which
is a more common pattern for clearing a string_list.
To be able to do this, rework hook_clear() into hook_free(), so
it can be passed to string_list_clear_func().
A slight complication is the need to keep a copy of the internal
cb data free() pointer, however I think it's worth it since the
API becomes cleaner, e.g. no more calls with NULL function args
like hook_list_clear(hooks, NULL).
Suggested-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Trigger a bug when an unknown hook type is encountered while
setting up hook execution.
Also issue a bug if a configured hook is enabled without a cmd.
Mostly useful for defensive coding.
Suggested-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Rename the hook callback function types to use the hook prefix.
This is a style fix with no logic changes.
Suggested-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Fix some minor style nits pointed by Patrick and Junio:
* Use CALLOC_ARRAY instead of xcalloc.
* Init struct members during declaration.
* Simplify if condition boolean logic.
* Missing curly braces in if/else stmts.
* Unnecessary header includes.
* Capitalization in error/warn messages.
* Comment spelling: free'd -> freed.
These contain no logic changes, the code behaves the same as before.
Suggested-by: Junio C Hamano <gitster@pobox.com>
Suggested-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Move the convenience wrapper from hook to string-list since
it's a more suitable place. Add a doc comment to the header.
Suggested-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* ar/config-hooks: (21 commits)
builtin/receive-pack: avoid spinning no-op sideband async threads
hook: add -z option to "git hook list"
hook: allow out-of-repo 'git hook' invocations
hook: allow event = "" to overwrite previous values
hook: allow disabling config hooks
hook: include hooks from the config
hook: add "git hook list" command
hook: run a list of hooks to prepare for multihook support
hook: add internal state alloc/free callbacks
receive-pack: convert receive hooks to hook API
receive-pack: convert update hooks to new API
run-command: poll child input in addition to output
hook: add jobs option
reference-transaction: use hook API instead of run-command
transport: convert pre-push to hook API
hook: allow separate std[out|err] streams
hook: convert 'post-rewrite' hook in sequencer.c to hook API
hook: provide stdin via callback
run-command: add stdin callback for parallelization
run-command: add helper for pp child states
...
Map my old Gmail address to my new custom address in .mailmap.
Signed-off-by: Tian Yuchen <a3205153416@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
"fsck" iterates over packfiles and its access to pack data caused
the list to be permuted, which caused it to loop forever; the code
to access pack data by "fsck" has been updated to avoid this.
* ps/fsck-stream-from-the-right-object-instance:
pack-check: fix verification of large objects
packfile: expose function to read object stream for an offset
object-file: adapt `stream_object_signature()` to take a stream
t/helper: improve "genrandom" test helper
The core.attributesfile is intended to be set per repository, but
were kept track of by a single global variable in-core, which has
been corrected by moving it to per-repository data structure.
* ob/core-attributesfile-in-repository:
environment: move "branch.autoSetupMerge" into `struct repo_config_values`
environment: stop using core.sparseCheckout globally
environment: stop storing `core.attributesFile` globally
"git config list" is taught to show the values interpreted for
specific type with "--type=<X>" option.
* ds/config-list-with-type:
config: use an enum for type
config: restructure format_config()
config: format colors quietly
color: add color_parse_quietly()
config: format expiry dates quietly
config: format paths gently
config: format bools or strings in helper
config: format bools or ints gently
config: format bools gently
config: format int64s gently
config: make 'git config list --type=<X>' work
config: add 'gently' parameter to format_config()
config: move show_all_config()
Mark the marge-ort codebase to prevent more uses of the_repository
from getting added.
* en/merge-ort-almost-wo-the-repository:
replay: prevent the_repository from coming back
merge-ort: prevent the_repository from coming back
merge-ort: replace the_hash_algo with opt->repo->hash_algo
merge-ort: replace the_repository with opt->repo
merge-ort: pass repository to write_tree()
merge,diff: remove the_repository check before prefetching blobs
Clean-up the code around "git repo info" command.
* lo/repo-leftover-bits:
Documentation/git-repo: capitalize format descriptions
Documentation/git-repo: replace 'NUL' with '_NUL_'
t1901: adjust nul format output instead of expected value
t1900: rename t1900-repo to t1900-repo-info
repo: rename struct field to repo_info_field
repo: replace get_value_fn_for_key by get_repo_info_field
repo: rename repo_info_fields to repo_info_field
CodingGuidelines: instruct to name arrays in singular
"git maintenance" starts using the "geometric" strategy by default.
* ps/maintenance-geometric-default:
builtin/maintenance: use "geometric" strategy by default
t7900: prepare for switch of the default strategy
t6500: explicitly use "gc" strategy
t5510: explicitly use "gc" strategy
t5400: explicitly use "gc" strategy
t34xx: don't expire reflogs where it matters
t: disable maintenance where we verify object database structure
t: fix races caused by background maintenance
"git apply --directory=./un/../normalized/path" now normalizes the
given path before using it.
* jr/apply-directory-normalize:
apply: normalize path in --directory argument
The last uses of the_repository in "tree-diff.c" have been
eradicated.
* sp/tree-diff-wo-the-repository:
tree-diff: remove the usage of the_hash_algo global
API clean-up for the worktree subsystem.
* pw/no-more-NULL-means-current-worktree:
path: remove repository argument from worktree_git_path()
wt-status: avoid passing NULL worktree
Wean the mailmap code off of the_repository dependency.
* bk/mailmap-wo-the-repository:
mailmap: drop global config variables
mailmap: stop using the_repository
"gitweb" has been taught to be mobile friendly.
* rr/gitweb-mobile:
gitweb: let page header grow on mobile for long wrapped project names
gitweb: fix mobile footer overflow by wrapping text and clearing floats
gitweb: fix mobile page overflow across log/commit/blob/diff views
gitweb: prevent project search bar from overflowing on mobile
gitweb: add viewport meta tag for mobile devices
"git fetch --deepen" that tries to go beyond merged branch used to
get confused where the updated shallow points are, which has been
corrected.
* sp/shallow-deepen-relative-fix:
shallow: handling fetch relative-deepen
shallow: free local object_array allocations
Allow the directory in which reference backends store their data to
be specified.
* kn/ref-location:
refs: add GIT_REFERENCE_BACKEND to specify reference backend
refs: allow reference location in refstorage config
refs: receive and use the reference storage payload
refs: move out stub modification to generic layer
refs: extract out `refs_create_refdir_stubs()`
setup: don't modify repo in `create_reference_database()`
A prefetch call can be triggered to access a stale diff_queue entry
after diffcore-break breaks a filepair into two and freed the
original entry that is no longer used, leading to a segfault, which
has been corrected.
* hy/diff-lazy-fetch-with-break-fix:
diffcore-break: avoid segfault with freed entries
"git add -p" learned a new mode that allows the user to revisit a
file that was already dealt with.
* aa/add-p-no-auto-advance:
add-patch: allow interfile navigation when selecting hunks
add-patch: allow all-or-none application of patches
add-patch: modify patch_update_file() signature
interactive -p: add new `--auto-advance` flag
An earlier attempt to optimize "git subtree" discarded too much
relevant histories, which has been corrected.
* cs/subtree-split-fixes:
contrib/subtree: process out-of-prefix subtrees
contrib/subtree: test history depth
contrib/subtree: capture additional test-cases
Code clean-up.
* jt/object-file-use-container-of:
object-file.c: avoid container_of() of a NULL container
object-file: use `container_of()` to convert from base types
