When I do git fetch, git call file stats under .git/objects for each
refs. This takes time when there are many refs.
By enabling fscache, git takes file stats by directory traversing and that
improved the speed of fetch-pack for repository having large number of
refs.
In my windows workstation, this improves the time of `git fetch` for
chromium repository like below. I took stats 3 times.
* With this patch
TotalSeconds: 9.9825165
TotalSeconds: 9.1862075
TotalSeconds: 10.1956256
Avg: 9.78811653333333
* Without this patch
TotalSeconds: 15.8406702
TotalSeconds: 15.6248053
TotalSeconds: 15.2085938
Avg: 15.5580231
Signed-off-by: Takuto Ikuta <tikuta@chromium.org>
Especially in huge code bases with fast-moving `master`, it can be
prohibitively expensive to calculate whether an upstream branch of
a local branch is ahead, behind or diverged.
This topic branch introduces a set of flags to avoid that computation
when we're not even interested in it to begin with.
This merge commit takes the feature early, therefore it is marked
experimental.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Teach long (normal) status format to respect the --no-ahead-behind
parameter and skip the possibly expensive ahead/behind computation
between the branch and the upstream.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Teach "git status --short --branch" to respect "--no-ahead-behind"
parameter to skip computing ahead/behind counts for the branch and
its upstream and just report '[different]'.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Teach "git status" and "git commit" to accept "--no-ahead-behind"
and "--ahead-behind" arguments to request quick or full ahead/behind
reporting.
When "--no-ahead-behind" is given, the existing porcelain V2 line
"branch.ab +x -y" is replaced with a new "branch.ab +? -?" line.
This indicates that the branch and its upstream are or are not equal
without the expense of computing the full ahead/behind values.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Extend stat_tracking_info() to return +1 when branches are not equal and to
take a new "enum ahead_behind_flags" argument to allow skipping the (possibly
expensive) ahead/behind computation.
This will be used in the next commit to allow "git status" to avoid full
ahead/behind calculations for performance reasons.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Once upon a time, git-clone would refuse to write into a
directory that it did not itself create. The cleanup
routines for a failed clone could therefore just remove the
git and worktree dirs completely.
In 55892d2398 (Allow cloning to an existing empty directory,
2009-01-11), we learned to write into an existing directory.
Which means that doing:
mkdir foo
git clone will-fail foo
ends up deleting foo. This isn't a huge catastrophe, since
by definition foo must be empty. But it's somewhat
confusing; we should leave the filesystem as we found it.
Because we know that the only directory we'll write into is
an empty one, we can handle this case by just passing the
KEEP_TOPLEVEL flag to our recursive delete (if we could
write into populated directories, we'd have to keep track of
what we wrote and what we did not).
Note that we need to handle the work-tree and git-dir
separately, though, as only one might exist (and the new
tests in t5600 cover all cases).
Reported-by: Stephan Janssen <sjanssen@you-get.com>
Two parts of git-clone's setup logic check whether a
directory exists, and they both call stat directly the same
scratch "struct stat" buffer. Let's pull that into a helper,
which has a few advantages:
- it makes the purpose of the stat call more obvious
- it makes it clear that we don't care about the
information in "buf" remaining valid
- if we later decide to make the check more robust (e.g.,
complaining about non-directories), we can do it in one
place
Note that we could just use file_exists() for this, which
has identical code. But we specifically care about
directories, so this future-proofs us against that function
later getting more picky about actual files.
This is an old script which could use some updating before
we add to it:
- use the standard line-breaking:
test_expect_success 'title' '
body
'
- run all code inside test_expect blocks to catch
unexpected failures in setup steps
- use "test_commit -C" instead of manually entering
sub-repo
- use test_when_finished for cleanup steps
- test_path_is_* as appropriate
Back when this test was written, git-clone could not handle
a repository without any commits. These days it works fine,
and this comment is out of date.
At first glance it seems like we could just drop this code
entirely now, but it's necessary for the final test, which
was added later. That test corrupts the repository my
temporarily removing its objects, which means we need to
have some objects to move.
This patch is no longer necessary, as we test Git for Windows on 64-bit
systems now. If at all, the test case in question has to be disabled for
32-bit systems, i.e. independent of whether the Operating System is
Windows or not.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This is to speed up git checkout for directory in very large repositories.
Taking file stats while directory traversing is faster than
stating to each files on windows.
`git checkout .` in master branch of chromium repositry, having 284659 files,
takes more than 18 seconds.
This patch improved the time to around 4 seconds on my SSD laptop.
Signed-off-by: Takuto Ikuta <tikuta@chromium.org>
Git for Windows supports the core.longPaths config setting to allow
writing/reading long paths via the \\?\ trick for a long time now.
However, for that support to work, it is absolutely necessary that
git_default_config() is given a chance to parse the config. Otherwise
Git will be non the wiser.
So let's make sure that as many commands that previously failed to
parse the core.* settings now do that, implicitly enabling long path
support in a lot more places.
Note: this is not a perfect solution, and it cannot be, as there is
a chicken-and-egg problem in reading the config itself...
This fixes https://github.com/git-for-windows/git/issues/1218
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
From the documentation of said setting:
This boolean will enable fsync() when writing object files.
This is a total waste of time and effort on a filesystem that
orders data writes properly, but can be useful for filesystems
that do not use journalling (traditional UNIX filesystems) or
that only journal metadata and not file contents (OS X’s HFS+,
or Linux ext3 with "data=writeback").
The most common file system on Windows (NTFS) does not guarantee that
order, therefore a sudden loss of power (or any other event causing an
unclean shutdown) would cause corrupt files (i.e. files filled with
NULs). Therefore we need to change the default.
Note that the documentation makes it sound as if this causes really bad
performance. In reality, writing loose objects is something that is done
only rarely, and only a handful of files at a time.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
A ton of Git commands simply do not read (or at least parse) the core.*
settings. This is not good, as Git for Windows relies on the
core.longPaths setting to be read quite early on.
So let's just make sure that all commands read the config and give
platform_core_config() a chance.
This patch teaches tons of Git commands to respect the config setting
`core.longPaths = true`, including `pack-refs`, thereby fixing
https://github.com/git-for-windows/git/issues/1218
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Fix regression described in:
https://github.com/git-for-windows/git/issues/1392
which was introduced in:
b2353379bb
Problem Symptoms
================
When the user has a .gitignore file that is a symlink, the fscache
optimization introduced above caused the stat-data from the symlink,
rather that of the target file, to be returned. Later when the ignore
file was read, the buffer length did not match the stat.st_size field
and we called die("cannot use <path> as an exclude file")
Optimization Rationale
======================
The above optimization calls lstat() before open() primarily to ask
fscache if the file exists. It gets the current stat-data as a side
effect essentially for free (since we already have it in memory).
If the file does not exist, it does not need to call open(). And
since very few directories have .gitignore files, we can greatly
reduce time spent in the filesystem.
Discussion of Fix
=================
The above optimization calls lstat() rather than stat() because the
fscache only intercepts lstat() calls. Calls to stat() stay directed
to the mingw_stat() completly bypassing fscache. Furthermore, calls
to mingw_stat() always call {open, fstat, close} so that symlinks are
properly dereferenced, which adds *additional* open/close calls on top
of what the original code in dir.c is doing.
Since the problem only manifests for symlinks, we add code to overwrite
the stat-data when the path is a symlink. This preserves the effect of
the performance gains provided by the fscache in the normal case.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
The hashmap API is just complicated enough that even at least one
long-time Git contributor has to look up how to use it every time he
finds a new use case. When that happens, it is really useful if the
provided example code is correct...
While at it, "fix a memory leak", avoid statements before variable
declarations, fix a const -> no-const cast, several %l specifiers (which
want to be %ld), avoid using an undefined constant, call scanf()
correctly, use FLEX_ALLOC_STR() where appropriate, and adjust the style
here and there.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Make fscache_enabled() function public rather than static.
Remove unneeded fscache_is_enabled() function.
Change is_fscache_enabled() macro to call fscache_enabled().
is_fscache_enabled() now takes a pathname so that the answer
is more precise and mean "is fscache enabled for this pathname",
since fscache only stores repo-relative paths and not absolute
paths, we can avoid attempting lookups for absolute paths.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
The "--format=..." option "git for-each-ref" takes learned to show
the name of the 'remote' repository and the ref at the remote side
that is affected for 'upstream' and 'push' via "%(push:remotename)"
and friends.
* js/for-each-ref-remote-name-and-ref:
for-each-ref: test :remotename and :remoteref
for-each-ref: let upstream/push report the remote ref name
for-each-ref: let upstream/push optionally report the remote name
This is the final iteration that made it into upstream Git's master
branch and that started out as 8148ee40ad (Merge branch
'mingw/ref-filter-remote-name', 2017-10-29).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The problem with not having, say, git-receive-pack.exe after a full
build is that the test suite will then happily use the *installed*
git-receive-pack.exe because it finds nothing else.
Absolutely not what we want. We want to have confidence that our test
covers the MSVC-built Git executables, and not some random stuff.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The idea of the `vcxproj` target is to generate .sln/.vcxproj files and
then commit them, to be used elsewhere. Typically, this is done in a
VSTS job whenever `master` changes. So there is little use in
initializing vcpkg and building all the dependencies: they are not
necessary here.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The `vcxproj` target does not, in fact, depend on MSVC being defined, so
let's just move it outside of that block.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We just introduced a way to build Git for Windows with MSVC on the
command line using vcpkg-generated, up-to-date dependencies. Let's bring
that convenience to the Visual Studio project, too.
(The previous method, fetching NuGet packages, is fraught with problems:
as C++ libraries have to be built for every architecture and for every
toolset, the NuGet packages which we would like to consume fell behind
and are not up-to-date with the current versions of the libraries, e.g.
cURL and OpenSSL. By using vcpkg we avoid that problem, always building
the newest dependency versions.)
The trick is to initialize the VCPKG system once, and then build Git's
dependencies using it. We do that by attaching a pre-build event to the
libgit project (which is now the base project on which all others
depend, therefore no other project is built in paralleli, side-stepping
issues with vcpkg being unprepared for being run in parallel).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The .vcxproj's text nodes do not actually need to URL-encode double
quotes. So let's not do that.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
It really does depend on libgit. It does not hurt to let it depend on
xdiff, and it makes the code simpler.
It is necessary to get this dependency chain right, because we will
introduce a change where the vcpkg system is initialized before building
libgit. The vcpkg system will then build the dependencies needed by Git
(and thereby make the include headers available):
As the vcpkg system cannot be run in parallel (it does not lock,
wreaking havoc with files being accessed and written at the same time,
letting the vcpkg processes stumble over each others' toes. We prevent
that by ensuring that only one project is built at first: libgit. And
this project's PreBuildEvent will be used to initialize vcpkg and build
all dependencies. Subsequently, the other projects can be built in
parallel.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When I run git fetch, git tries to find object for each local and remote
refs. Without specifying OBJECT_INFO_QUICK, has_object_file list up
entries in pack directory for each calls.
This patch makes git fetch fast for the repositories having large number
of refs, especially for windows because it's directory list up api is
much slower than linux.
Note: this patch was developed independently also by Jeff King:
https://public-inbox.org/git/20171120202920.7ppcwmzkxifywtoj@sigill.intra.peff.net/
Signed-off-by: Takuto Ikuta <tikuta@chromium.org>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
As we do not consume NuGet packages any longer, there is no sense to try
to point PATH to their unpacked .dll files, either.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Dependencies such as cURL and OpenSSL are necessary to build and run
Git. Previously, we obtained those dependencies by fetching NuGet
packages.
However, it is notoriously hard to keep NuGet packages of C/C++
libraries up-to-date, as the toolsets for different Visual Studio
versions are different, and the NuGet packages would have to ship them
all.
That is the reason why the NuGet packages we use are quite old, and even
insecure in the case of cURL and OpenSSL (the versions contain known
security flaws that have been addressed by later versions for which no
NuGet packages are available).
The better way to handle this situation is to use the vcpkg system:
https://github.com/Microsoft/vcpkg
The idea is that a single Git repository contains enough supporting
files to build up-to-date versions of a large number of Open Source
libraries on demand, including cURL and OpenSSL.
We integrate this system via four new .bat files to
1) initialize the vcpkg system,
2) build the packages,
4) set up Git's Makefile system to find the build artifacts, and
3) copy the artifacts into the top-level directory
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The script assumes that we're in the top-level directory of the
checkout. That does not need to be true.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
One time too many did this developer call the `generate` script passing
a `--make-out=<PATH>` option that was happily ignored (because there
should be a space, not an equal sign, between `--make-out` and the
path).
And one time too many, this script not only ignored it but did not even
complain. Let's fix that.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This not only prevents regressions, but also serves as documentation
what this new feature is expected to do.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The option is deprecated now, and we better make sure that keeps saying
so until we finally remove it.
Suggested by Kevin Willford.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
There are times when scripts want to know not only the name of the
push branch on the remote, but also the name of the branch as known
by the remote repository.
An example of this is when a tool wants to push to the very same branch
from which it would pull automatically, i.e. the `<remote>` and the `<to>`
in `git push <remote> <from>:<to>` would be provided by
`%(upstream:remotename)` and `%(upstream:remoteref)`, respectively.
This patch offers the new suffix :remoteref for the `upstream` and `push`
atoms, allowing to show exactly that. Example:
$ cat .git/config
...
[remote "origin"]
url = https://where.do.we.come/from
fetch = refs/heads/*:refs/remote/origin/*
[branch "master"]
remote = origin
merge = refs/heads/master
[branch "develop/with/topics"]
remote = origin
merge = refs/heads/develop/with/topics
...
$ git for-each-ref \
--format='%(push) %(push:remoteref)' \
refs/heads
refs/remotes/origin/master refs/heads/master
refs/remotes/origin/develop/with/topics refs/heads/develop/with/topics
Signed-off-by: J Wyman <jwyman@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Teach read_directory_recursive() and add_excludes() to
be aware of optional fscache and avoid trying to open()
and fstat() non-existant ".gitignore" files in every
directory in the worktree.
The current code in add_excludes() calls open() and then
fstat() for a ".gitignore" file in each directory present
in the worktree. Change that when fscache is enabled to
call lstat() first and if present, call open().
This seems backwards because both lstat needs to do more
work than fstat. But when fscache is enabled, fscache will
already know if the .gitignore file exists and can completely
avoid the IO calls. This works because of the lstat diversion
to mingw_lstat when fscache is enabled.
This reduced status times on a 350K file enlistment of the
Windows repo on a NVMe SSD by 0.25 seconds.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
It was a bad idea to just remove that option from Git for Windows
v2.15.0, as early users of that (still experimental) option would have
been puzzled what they are supposed to do now.
So let's reintroduce the flag, but make sure to show the user good
advice how to fix this going forward.
We'll remove this option in a more orderly fashion either in v2.16.0 or
in v2.17.0.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
There are times when e.g. scripts want to know not only the name of the
upstream branch on the remote repository, but also the name of the
remote.
This patch offers the new suffix :remotename for the upstream and for
the push atoms, allowing to show exactly that. Example:
$ cat .git/config
...
[remote "origin"]
url = https://where.do.we.come/from
fetch = refs/heads/*:refs/remote/origin/*
[remote "hello-world"]
url = https://hello.world/git
fetch = refs/heads/*:refs/remote/origin/*
pushURL = hello.world:git
push = refs/heads/*:refs/heads/*
[branch "master"]
remote = origin
pushRemote = hello-world
...
$ git for-each-ref \
--format='%(upstream) %(upstream:remotename) %(push:remotename)' \
refs/heads/master
refs/remotes/origin/master origin hello-world
The implementation chooses *not* to DWIM the push remote if no explicit
push remote was configured; The reason is that it is possible to DWIM this
by using
%(if)%(push:remotename)%(then)
%(push:remotename)
%(else)
%(upstream:remotename)
%(end)
while it would be impossible to "un-DWIM" the information in case the
caller is really only interested in explicit push remotes.
While `:remote` would be shorter, it would also be a bit more ambiguous,
and it would also shut the door e.g. for `:remoteref` (which would
obviously refer to the corresponding ref in the remote repository).
Note: the dashless, non-CamelCased form `:remotename` follows the
example of the `:trackshort` example.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
