The recent change to make fscache thread specific relied on fscache_enable()
being called first from the primary thread before being called in parallel
from worker threads. Make that more robust and protect it with a critical
section to avoid any issues.
Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Ben Peart <benpeart@microsoft.com>
Now that the fscache is single threaded, take advantage of the mem_pool as
the allocator to significantly reduce the cost of allocations and frees.
With the reduced cost of free, in future patches, we can start freeing the
fscache at the end of commands instead of just leaking it.
Signed-off-by: Ben Peart <benpeart@microsoft.com>
The threading model for fscache has been to have a single, global cache.
This puts requirements on it to be thread safe so that callers like
preload-index can call it from multiple threads. This was implemented
with a single mutex and completion events which introduces contention
between the calling threads.
Simplify the threading model by making fscache thread specific. This allows
us to remove the global mutex and synchronization events entirely and instead
associate a fscache with every thread that requests one. This works well with
the current multi-threading which divides the cache entries into blocks with
a separate thread processing each block.
At the end of each worker thread, if there is a fscache on the primary
thread, merge the cached results from the worker into the primary thread
cache. This enables us to reuse the cache later especially when scanning for
untracked files.
In testing, this reduced the time spent in preload_index() by about 25% and
also reduced the CPU utilization significantly. On a repo with ~200K files,
it reduced overall status times by ~12%.
Signed-off-by: Ben Peart <benpeart@microsoft.com>
Update enable_fscache() to take an optional initial size parameter which is
used to initialize the hashmap so that it can avoid having to rehash as
additional entries are added.
Add a separate disable_fscache() macro to make the code clearer and easier
to read.
Signed-off-by: Ben Peart <benpeart@microsoft.com>
Add tracing around initializing and discarding mempools. In discard report
on the amount of memory unused in the current block to help tune setting
the initial_size.
Signed-off-by: Ben Peart <benpeart@microsoft.com>
Track fscache hits and misses for lstat and opendir requests. Reporting of
statistics is done when the cache is disabled for the last time and freed
and is only reported if GIT_TRACE_FSCACHE is set.
Sample output is:
11:33:11.836428 compat/win32/fscache.c:433 fscache: lstat 3775, opendir 263, total requests/misses 4052/269
Signed-off-by: Ben Peart <benpeart@microsoft.com>
At the end of the status command, disable and free the fscache so that we
don't leak the memory and so that we can dump the fscache statistics.
Signed-off-by: Ben Peart <benpeart@microsoft.com>
Use FindFirstFileExW with FindExInfoBasic to avoid forcing NTFS to look up
the short name. Also switch to a larger (64K vs 4K) buffer using
FIND_FIRST_EX_LARGE_FETCH to minimize round trips to the kernel.
In a repo with ~200K files, this drops warm cache status times from 3.19
seconds to 2.67 seconds for a 16% savings.
Signed-off-by: Ben Peart <benpeart@microsoft.com>
To avoid having to play tricks as in earlier rounds, we bit the sour
apple and rebased the `builtin-stash-rebase-v3` branch thicket onto the
commit starting Git for Windows' merging-rebase.
(The merging-rebase pulls in the previous branch thicket via a "fake
merge", i.e. a merge commit that does not actually apply any changes
from the merged commit history. This has the unfortunate side effect of
confusing `merge` into thinking that any branch that was merged into an
earlier round does not need to be merged again.)
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This fixes the issue identified in
https://github.com/git-for-windows/git/issues/1498
where Git would not fall back to reading credentials from a Win32
Console when the credentials could not be read from the terminal via the
Bash hack (that is necessary to support running in a MinTTY).
Tested in a Powershell window.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This topic branch brings slightly experimental changes supporting Git
for Windows to use BusyBox-w32 to execute its shell scripts as well as
its test suite.
The test suite can be run by installing the test artifacts into a MinGit
that has busybox.exe (and using Git for Windows' SDK's Perl for now, as
the test suite requires Perl even when NO_PERL is set, go figure) by
using the `install-mingit-test-artifacts` Makefile target with the
DESTDIR variable pointing to the top-level directory of the MinGit
installation.
To facilitate running the test suite (without having `make` available,
as `make.exe` is not part of MinGit), this branch brings an experimental
patch to the `test-run-command` helper to run Git's test suite. It is
still very experimental, though: in this developer's tests it seemed
that the `poll()` emulation required for `run_parallel_processes()` to
work sometimes hiccups on Windows, causing infinite "hangs". It is also
possible that BusyBox itself has problems writing to the pipes opened by
`test-run-command` (and merging this branch will help investigate
further). Caveat emptor.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When spawning child processes, we do want them to inherit the standard
handles so that we can talk to them. We do *not* want them to inherit
any other handle, as that would hold a lock to the respective files
(preventing them from being renamed, modified or deleted), and the child
process would not know how to access that handle anyway.
Happily, there is an API to make that happen. It is supported in Windows
Vista and later, which is exactly what we promise to support in Git for
Windows for the time being.
This also means that we lift, at long last, the target Windows version
from Windows XP to Windows Vista.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This branch allows third-party tools to call `git status
--no-lock-index` to avoid lock contention with the interactive Git usage
of the actual human user.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Git for Windows supports the core.longPaths config setting to allow
writing/reading long paths via the \\?\ trick for a long time now.
However, for that support to work, it is absolutely necessary that
git_default_config() is given a chance to parse the config. Otherwise
Git will be non the wiser.
So let's make sure that as many commands that previously failed to
parse the core.* settings now do that, implicitly enabling long path
support in a lot more places.
Note: this is not a perfect solution, and it cannot be, as there is
a chicken-and-egg problem in reading the config itself...
This fixes https://github.com/git-for-windows/git/issues/1218
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This branch introduces support for reading the "Windows-wide" Git
configuration from `%PROGRAMDATA%\Git\config`. As these settings are
intended to be shared between *all* Git-related software, that config
file takes an even lower precedence than `$(prefix)/etc/gitconfig`.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
These are Git for Windows' Git GUI and gitk patches. We will have to
decide at some point what to do about them, but that's a little lower
priority (as Git GUI seems to be unmaintained for the time being, and
the gitk maintainer keeps a very low profile on the Git mailing list,
too).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This is the branch thicket of patches in Git for Windows that are
considered ready for upstream. To keep them in a ready-to-submit shape,
they are kept as close to the beginning of the branch thicket as
possible.
On a case-insensitive filesystem, such as HFS+ or NTFS, it is possible
that the idea Bash has of the current directory differs in case from
what Git thinks it is. That's totally okay, though, and we should not
expect otherwise.
Reported by Jameson Miller.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This branch adds back the scripted versions, then adds the option to
use the builtin versions of `stash` and `rebase` by setting
`stash.useBuiltin=true` and `rebase.useBuiltin=true`, respectively,
(the latter already worked for the top-level `git rebase` command and
the `--am` backend, and now it also works for the interactive backend).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This patch teaches the builtin rebase to avoid the scripted --am backend
and call `git format-patch` and `git am` directly.
Meaning: apart from the --merge and the --preserve-merges backends, `git
rebase` is now implemented in pure C, with no need to ask the Unix shell
interpreter for help.
This brings us really close to a fully builtin `git rebase`: the
--preserve-merges mode is about to be deprecated (as soon as the
--rebase-merges mode has proven stable and robust enough), and there are
plans to scrap the `git-rebase--merge` backend in favor of teaching the
interactive rebase enough tricks to run the --merge mode, too.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We recently converted both the `git rebase` and the `git rebase -i`
command from Unix shell scripts to builtins.
The former has a safety valve allowing to fall back to the scripted
`rebase`, just in case that there is a bug in the builtin `rebase`:
setting the config variable `rebase.useBuiltin` to `false` will
fall back to using the scripted version.
The latter did not have such a safety hatch.
Let's reinstate the scripted interactive rebase backend so that `rebase.useBuiltin=false` will not use the builtin interactive rebase,
just in case that an end user runs into a bug with the builtin version
and needs to get out of the fix really quickly.
This is necessary because Git for Windows wants to ship the builtin
rebase/interactive rebase earlier than core Git: Git for Windows
v2.19.0 will come with the option of a drastically faster (if a lot
less battle-tested) `git rebase`/`git rebase -i`.
As the file name `git-rebase--interactive` is already in use, let's
rename the scripted backend to `git-legacy-rebase--interactive`.
A couple of additional touch-ups are needed (such as teaching the
builtin `rebase--interactive`, which assumed the role of the
`rebase--helper`, to perform the two tricks to skip the unnecessary
picks and to generate a new todo list) to make things work again.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This simply copies the version as of v2.19.0-rc0 verbatim. As of now,
it is not hooked up (because it needs a couple more changes to work);
The next commit will use the scripted interactive rebase backend from
`git rebase` again when `rebase.useBuiltin=false`.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This merges the builtin stash.
Upstream Git did not integrate it into any stable integration branch
yet, but the performance improvements are substantial enough,
especially on Windows, that we really, really, really want to have it
early.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
While the scripted `git rebase` still has to rely on the
`git-rebase--am.sh` script to implement the glue between the `rebase`
and the `am` commands, we can go a more direct route in the builtin
rebase and avoid using a shell script altogether.
This reduces the chances of Git for Windows running into trouble due to
problems with the POSIX emulation layer (known as "MSYS2 runtime",
itself a derivative of the Cygwin runtime): when no shell script is
called, the POSIX emulation layer is avoided altogether.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We recently converted the `git stash` command from Unix shell scripts
to builtins.
Just like we have `rebase.useBuiltin` to fall back to the scripted
rebase, to give end users a way out when they discover a bug in the
builtin command, this commit adds support for `stash.useBuiltin`.
This is necessary because Git for Windows wants to ship the builtin
stash earlier than core Git: Git for Windows v2.19.0 will come with
the option of a drastically faster (if a lot less battle-tested)
`git stash`.
As the file name `git-stash` is already in use, let's rename the
scripted backend to `git-legacy-stash`.
To make the test suite pass with `stash.useBuiltin=false`, this commit
also backports rudimentary support for `-q` (but only *just* enough
to appease the test suite), and adds a super-ugly hack to force exit
code 129 for `git stash -h`.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When switching a branch *and* updating said branch to a different
revision, let's avoid a double entry by first updating the branch and
then adjusting the symbolic ref HEAD.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
