Teach "add" to use preload-index and fscache features
to improve performance on very large repositories.
During an "add", a call is made to run_diff_files()
which calls check_remove() for each index-entry. This
calls lstat(). On Windows, the fscache code intercepts
the lstat() calls and builds a private cache using the
FindFirst/FindNext routines, which are much faster.
Somewhat independent of this, is the preload-index code
which distributes some of the start-up costs across
multiple threads.
We need to keep the call to read_cache() before parsing the
pathspecs (and hence cannot use the pathspecs to limit any preload)
because parse_pathspec() is using the index to determine whether a
pathspec is, in fact, in a submodule. If we would not read the index
first, parse_pathspec() would not error out on a path that is inside
a submodule, and t7400-submodule-basic.sh would fail with
not ok 47 - do not add files from a submodule
We still want the nice preload performance boost, though, so we simply
call read_cache_preload(&pathspecs) after parsing the pathspecs.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The purpose of this function is to stat() the files listed in the index
in a multi-threaded fashion. It is called directly after reading the
index in the read_index_preloaded() function.
However, in some cases we may want to separate the index reading from
the preloading step, e.g. in builtin/add.c, where we need to load the
index before we parse the pathspecs (which needs to error out if one of
the pathspecs refers to a path within a submodule, for which the index
must have been read already), and only then will we want to preload,
possibly limited by the just-parsed pathspecs.
So let's just export that function to allow calling it separately.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
If multiple threads access a directory that is not yet in the cache, the
directory will be loaded by each thread. Only one of the results is added
to the cache, all others are leaked. This wastes performance and memory.
On cache miss, add a future object to the cache to indicate that the
directory is currently being loaded. Subsequent threads register themselves
with the future object and wait. When the first thread has loaded the
directory, it replaces the future object with the result and notifies
waiting threads.
Signed-off-by: Karsten Blees <blees@dcon.de>
Checking the work tree status is quite slow on Windows, due to slow lstat
emulation (git calls lstat once for each file in the index). Windows
operating system APIs seem to be much better at scanning the status
of entire directories than checking single files.
Add an lstat implementation that uses a cache for lstat data. Cache misses
read the entire parent directory and add it to the cache. Subsequent lstat
calls for the same directory are served directly from the cache.
Also implement opendir / readdir / closedir so that they create and use
directory listings in the cache.
The cache doesn't track file system changes and doesn't plug into any
modifying file APIs, so it has to be explicitly enabled for git functions
that don't modify the working copy.
Note: in an earlier version of this patch, the cache was always active and
tracked file system changes via ReadDirectoryChangesW. However, this was
much more complex and had negative impact on the performance of modifying
git commands such as 'git checkout'.
Signed-off-by: Karsten Blees <blees@dcon.de>
Add a macro to mark code sections that only read from the file system,
along with a config option and documentation.
This facilitates implementation of relatively simple file system level
caches without the need to synchronize with the file system.
Enable read-only sections for 'git status' and preload_index.
Signed-off-by: Karsten Blees <blees@dcon.de>
Emulating the POSIX lstat API on Windows via GetFileAttributes[Ex] is quite
slow. Windows operating system APIs seem to be much better at scanning the
status of entire directories than checking single files. A caching
implementation may improve performance by bulk-reading entire directories
or reusing data obtained via opendir / readdir.
Make the lstat implementation pluggable so that it can be switched at
runtime, e.g. based on a config option.
Signed-off-by: Karsten Blees <blees@dcon.de>
Emulating the POSIX dirent API on Windows via FindFirstFile/FindNextFile is
pretty staightforward, however, most of the information provided in the
WIN32_FIND_DATA structure is thrown away in the process. A more
sophisticated implementation may cache this data, e.g. for later reuse in
calls to lstat.
Make the dirent implementation pluggable so that it can be switched at
runtime, e.g. based on a config option.
Define a base DIR structure with pointers to readdir/closedir that match
the opendir implementation (i.e. similar to vtable pointers in OOP).
Define readdir/closedir so that they call the function pointers in the DIR
structure. This allows to choose the opendir implementation on a
call-by-call basis.
Move the fixed sized dirent.d_name buffer to the dirent-specific DIR
structure, as d_name may be implementation specific (e.g. a caching
implementation may just set d_name to point into the cache instead of
copying the entire file name string).
Signed-off-by: Karsten Blees <blees@dcon.de>
Git for Windows ships with its own Perl interpreter, and insists on
using it, so it will most likely wreak havoc if PERL5LIB is set before
launching Git.
Let's just unset that environment variables when spawning processes.
To make this feature extensible (and overrideable), there is a new
config setting `core.unsetenvvars` that allows specifying a
comma-separated list of names to unset before spawning processes.
Reported by Gabriel Fuhrmann.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
In the Git for Windows project, we have ample precendent for config
settings that apply to Windows, and to Windows only.
Let's formalize this concept by introducing a platform_core_config()
function that can be #define'd in a platform-specific manner.
This will allow us to contain platform-specific code better, as the
corresponding variables no longer need to be exported so that they can
be defined in environment.c and be set in config.c
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This commit starts the rebase of 151f34340d to ad36dc8b4b
Sadly, v3 of the mingw-isatty-fixup branch was not picked up by upstream
Git, but v2 instead. A follow-up patch has been sent to rectify the
situation.
Also, the "Avoid a segmentation fault with renaming merges"'s latest
iteration was not picked up (the latest revision did not try to be
overzealous about warning the user when there is a warning elsewhere
already), but there are more worthy things to fight for. The other parts
of the `cherry-pick-segfault` patch series made it into `maint` unharmed,
though.
Likewise, the "mailinfo: remove misguided assert() statement" patch was
fixed differently upstream, so that patch was dropped also.
The `unc-alternates` topic branch was dropped, as it has been made
obsolete by a more general fix in `normalize_path_copy(): fix pushing to
//server/share/dir on Windows`.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Code cleanup to avoid using redundant refspecs while fetching with
the --tags option.
* jt/fetch-no-redundant-tag-fetch-map:
fetch: do not redundantly calculate tag refmap
* kh/tutorial-grammofix:
doc: omit needless "for"
doc: make the intent of sentence clearer
doc: add verb in front of command to run
doc: add articles (grammar)
When the http server gives an incomplete response to a smart-http
rpc call, it could lead to client waiting for a full response that
will never come. Teach the client side to notice this condition
and abort the transfer.
An improvement counterproposal has failed.
cf. <20161114194049.mktpsvgdhex2f4zv@sigill.intra.peff.net>
* dt/smart-http-detect-server-going-away:
upload-pack: optionally allow fetching any sha1
remote-curl: don't hang when a server dies before any output
A potential but unlikely buffer overflow in Windows port has been
fixed.
* mk/mingw-winansi-ttyname-termination-fix:
mingw: consider that UNICODE_STRING::Length counts bytes
"git p4" that tracks multile p4 paths imported a single changelist
that touches files in these multiple paths as one commit, followed
by many empty commits. This has been fixed.
* gv/p4-multi-path-commit-fix:
git-p4: fix multi-path changelist empty commits
Even though an fix was attempted in Git 2.9.3 days, but running
"git difftool --dir-diff" from a subdirectory never worked. This
has been fixed.
* jk/difftool-in-subdir:
difftool: rename variables for consistency
difftool: chdir as early as possible
difftool: sanitize $workdir as early as possible
difftool: fix dir-diff index creation when in a subdirectory
A lazy "git push" without refspec did not internally use a fully
specified refspec to perform 'current', 'simple', or 'upstream'
push, causing unnecessary "ambiguous ref" errors.
* jc/push-default-explicit:
push: test pushing ambiguously named branches
push: do not use potentially ambiguous default refspec
"git index-pack --stdin" needs an access to an existing repository,
but "git index-pack file.pack" to generate an .idx file that
corresponds to a packfile does not.
* jk/index-pack-wo-repo-from-stdin:
index-pack: skip collision check when not in repository
t: use nongit() function where applicable
index-pack: complain when --stdin is used outside of a repo
t5000: extract nongit function to test-lib-functions.sh
The function usage_msg_opt() has been updated to say "fatal:"
before the custom message programs give, when they want to die
with a message about wrong command line options followed by the
standard usage string.
* jk/parseopt-usage-msg-opt:
parse-options: print "fatal:" before usage_msg_opt()
A recent update to receive-pack to make it easier to drop garbage
objects made it clear that GIT_ALTERNATE_OBJECT_DIRECTORIES cannot
have a pathname with a colon in it (no surprise!), and this in turn
made it impossible to push into a repository at such a path. This
has been fixed by introducing a quoting mechanism used when
appending such a path to the colon-separated list.
* jk/quote-env-path-list-component:
t5615-alternate-env: double-quotes in file names do not work on Windows
t5547-push-quarantine: run the path separator test on Windows, too
tmp-objdir: quote paths we add to alternates
alternates: accept double-quoted paths
Unlike "git am --abort", "git cherry-pick --abort" moved HEAD back
to where cherry-pick started while picking multiple changes, when
the cherry-pick stopped to ask for help from the user, and the user
did "git reset --hard" to a different commit in order to re-attempt
the operation.
* sb/sequencer-abort-safety:
Revert "sequencer: remove useless get_dir() function"
sequencer: remove useless get_dir() function
sequencer: make sequencer abort safer
t3510: test that cherry-pick --abort does not unsafely change HEAD
am: change safe_to_abort()'s not rewinding error into a warning
am: fix filename in safe_to_abort() error message
The way to specify hotkeys to "xxdiff" that is used by "git
mergetool" has been modernized to match recent versions of xxdiff.
* da/mergetool-xxdiff-hotkey:
mergetools: fix xxdiff hotkeys
"git pull --rebase", when there is no new commits on our side since
we forked from the upstream, should be able to fast-forward without
invoking "git rebase", but it didn't.
* jc/pull-rebase-ff:
pull: fast-forward "pull --rebase=true"
A pathname that begins with "//" or "\\" on Windows is special but
path normalization logic was unaware of it.
* js/normalize-path-copy-ceil:
normalize_path_copy(): fix pushing to //server/share/dir on Windows
"git commit --allow-empty --only" (no pathspec) with dirty index
ought to be an acceptable way to create a new commit that does not
change any paths, but it was forbidden, perhaps because nobody
needed it so far.
* ak/commit-only-allow-empty:
commit: remove 'Clever' message for --only --amend
commit: make --only --allow-empty work without paths
"git difftool --dir-diff" had a minor regression when started from
a subdirectory, which has been fixed.
* da/difftool-dir-diff-fix:
difftool: fix dir-diff index creation when in a subdirectory
When diff.renames configuration is on (and with Git 2.9 and later,
it is enabled by default, which made it worse), "git stash"
misbehaved if a file is removed and another file with a very
similar content is added.
* jk/stash-disable-renames-internally:
stash: prefer plumbing over git-diff
Update the error messages from the dumb-http client when it fails
to obtain loose objects; we used to give sensible error message
only upon 404 but we now forbid unexpected redirects that needs to
be reported with something sensible.
* jk/http-walker-limit-redirect:
http-walker: complain about non-404 loose object errors
http: treat http-alternates like redirects
http: make redirects more obvious
remote-curl: rename shadowed options variable
http: always update the base URL for redirects
http: simplify update_url_from_redirect
Fix a corner case in merge-recursive regression that crept in
during 2.10 development cycle.
* jc/renormalize-merge-kill-safer-crlf:
convert: git cherry-pick -Xrenormalize did not work
merge-recursive: handle NULL in add_cacheinfo() correctly
cherry-pick: demonstrate a segmentation fault
"git p4" LFS support was broken when LFS stores an empty blob.
* ls/p4-empty-file-on-lfs:
git-p4: fix empty file processing for large file system backend GitLFS
mergetool.<tool>.trustExitCode configuration variable did not apply
to built-in tools, but now it does.
* da/mergetool-trust-exit-code:
mergetools/vimdiff: trust Vim's exit code
mergetool: honor mergetool.$tool.trustExitCode for built-in tools
The output from "git worktree list" was made in readdir() order,
and was unstable.
* nd/worktree-list-fixup:
worktree list: keep the list sorted
worktree.c: get_worktrees() takes a new flag argument
get_worktrees() must return main worktree as first item even on error
worktree: reorder an if statement
worktree.c: zero new 'struct worktree' on allocation
