Git's source code wants to be able to close() the same file descriptor
multiple times, ignoring the error returned by the second call (and the
ones after that), or to access the osfhandle of an already-closed stdout,
among other things that the UCRT does not like.
Simply linking invalidcontinue.obj allows such usage without resorting to
Debug Assertions (or exiting with exit code 9 in Release Mode).
Let's add a note so we don't forget, as suggested by Jeff Hostetler.
See https://msdn.microsoft.com/en-us/library/ms235330.aspx for more
details.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
In Git for Windows' SDK, there is already a script to package Git for
Windows as a NuGet package, downloading nuget.exe if needed.
Let's just fall back to using that executable (if it is there) if
nuget.exe was not found in the PATH.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When compiling with Visual Studio, the projects' names are identical to
the executables modulo the extensions. Which means that the bin-wrappers
*need* to target the .exe files lest they try to execute directories.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We do have a lovely Makefile option to state that that header file is
not available. Let's use it everywhere...
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Teach main Makefile to also delete the generated PDB files
as well as the PDB files for the various EXE files during
"make MSVC=1 clean".
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
MS Visual C suggests that the construct
condition ? (int) i : (ptrdiff_t) d
is incorrect. Let's fix this by casting to ptrdiff_t also for the
positive arm of the conditional.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When passing a command-line to call an external diff command to the
difftool, we must be prepared for paths containing special characters,
e.g. backslashes in the temporary directory's path on Windows.
This has been caught by running the test suite with an MSVC-built Git:
in contrast to the MINGW one, it does not rewrite `$TMP` to use forward
slashes instead of backslashes.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When compiling with MSVC, we rely on NuPkgs to provide the binaries of
dependencies such as libiconv. The libiconv 1.14.0.11 package available
from https://www.nuget.org/packages/libiconv seems to have a bug where
it does not set errno (when we would expect it to be E2BIG).
Let's simulate the error condition by taking less than 16 bytes
remaining in the out buffer as an indicator that we ran out of space.
While 16 might seem a bit excessive (when converting from, say, any
encoding to UTF-8, 8 bytes should be fine), it is designed to be a safe
margin.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Just like 1e2ce1d (sha1: Use OpenSSL SHA1 routines on MINGW, 2016-10-12),
we now use OpenSSL's SHA-1 routines instead of Git's own because OpenSSL
is substantially faster as of version 1.0.2: it now uses hardware
acceleration on Intel processors much more effectively.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Install required third-party DLLs next to EXEs.
Build and install release mode PDBs for git
executables allowing detailed stack traces
in the event of crash.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
In MSVC, the DEBUG constant is set automatically whenever compiling with
debug information.
This is clearly not what was intended in cache-tree.c, so let's use a less
ambiguous constant there.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
The MINGW version of the main() wrapper gets away with declaring symbols
that were intentionally not exported. However, some of these symbols do
not actually exist in MSVC's UCRT.
So let's add an MSVC version of the main() wrapper that uses wmain() and
imports the UNICODE argv and environment. While at it, we pass our UTF-8
version of ARGV to the real main -- rather than overwriting __argv as is
done in the MINGW Version.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
This special-cases various signals that are not supported on Windows,
such as SIGPIPE. These cause the UCRT to throw asserts (at least in
debug mode).
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
This adds MSVC versions of getenv() and friends. These take UTF-8
arguments and return UTF-8 values, but use the UNICODE versions
of the CRT routines. This avoids the need to write to __environ
(which is only visible if you statically link to the CRT). This
also avoids the CP_ACP conversions performed inside the CRT.
It also avoids various memory leaks and problems.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
GCC and MSVC disagree about using the GCC extension _ANONYMOUS_UNION.
Simply skip that offending keyword when compiling with MSVC.
Signed-off-by: Philip Oakley <philipoakley@iee.org>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This constant is not defined in MSVC's headers.
In UCRT's fcntl.h, _O_RDONLY, _O_WRONLY and _O_RDWR are defined as 0, 1
and 2, respectively. Yes, that means that UCRT breaks with the tradition
that O_RDWR == O_RDONLY | O_WRONLY.
It is a perfectly legal way to define those constants, though, therefore
we need to take care of defining O_ACCMODE accordingly.
This is particularly important in order to keep our "open() can set
errno to EISDIR" emulation working: it tests that (flags & O_ACCMODE) is
not identical to O_RDONLY before going on to test specifically whether
the file for which open() reported EACCES is, in fact, a directory.
Signed-off-by: Philip Oakley <philipoakley@iee.org>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We used to install into $HOME/bin/, which wreaks havoc with installed
versions of Git for Windows (because $HOME/bin is *prepended* to the
PATH, hence would override `git.exe` in Git Bash).
Let's align the MSVC case with the non-MSVC case and install into
/mingw64/bin/ (or /mingw32/bin/ in 32-bit Git for Windows SDKs) instead.
Noticed by Derrick Stolee.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Support -Z flags ("specify PDB options"), only include -l args on link
commands, and force PDBs to be created.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
This commit contains a GNU Makefile and NuGet configuration scripts to
download and install the various third-party libraries that we will need
to build/link with when using VS2015 to build Git.
The file "compat/vcbuild/README_VS2015.txt" contains instructions for
using this.
In this commit, "compat/vcbuild/Makefile" contains hard-coded version
numbers of the packages we require. These are set to the current
versions as of the time of this commit. We use "nuget restore" to
install them explicitly using a "package.config". A future improvement
would try to use some of the automatic package management functions and
eliminate the need to specify exact versions. I tried, but could not
get this to work. NuGet was happy dowload "minimum requirements" rather
than "lastest" for dependencies -- and only look at one package at a
time. For example, both curl and openssl depend upon zlib and have
different minimums. It was unclear which version of zlib would be
installed and seemed to be dependent on the order of the top-level
packages. So, I'm skipping that for now.
We need to be very precise when specifying NuGet package versions: while
nuget.exe auto-completes a version, say, 1.0.2 to 1.0.2.0, we will want
to parse packages.config ourselves, to generate the Visual Studio
solution, and there we need the exact version number to be able to
generate the exact path to the correct .targets file.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
The file compat/msvc.c includes compat/mingw.c, which means that we have
to recompile compat/msvc.o if compat/mingw.c changes.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Commit 4b623d8 (MSVC: link in invalidcontinue.obj for better
POSIX compatibility, 2014-03-29) introduced invalidcontinue.obj
into the Makefile output, which was not parsed correctly by the
buildsystem. Ignore it, as it is known to Visual Studio and,
there is no matching source file.
Only substitute filenames ending with .o when generating the
source .c filename, otherwise a .cbj file may be expected.
Split the .o and .obj processing; 'make' does not produce .obj
files.
In the future there may be source files that produce .obj files
so keep the two issues (.obj files with & without source files)
separate.
Signed-off-by: Philip Oakley <philipoakley@iee.org>
Signed-off-by: Duncan Smart <duncan.smart@gmail.com>
(cherry picked from commit d01d71fe1aed67f4e3a5ab80eeadeaf525ad0846)
... even if they may look like them.
As looking up the target of the "symbolic link" (just to see whether it
starts with `/ContainerMappedDirectories/`) is pretty expensive, we
do it when we can be *really* sure that there is a possibility that this
might be the case.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: JiSeop Moon <zcube@zcube.kr>
On file systems that support it, this can dramatically speed up operations
like add, commit, describe, rebase, reset, rm that would otherwise have to
lstat() every file to "re-match" the stat information in the index to that
of the file system.
On a synthetic repo with 1M files, "git reset" dropped from 52.02 seconds to
14.42 seconds for a savings of 72%.
Signed-off-by: Ben Peart <Ben.Peart@microsoft.com>
In preparation for making this function a bit more complicated (to allow
for special-casing the `ContainerMappedDirectories` in Windows
containers, which look like a symbolic link, but are not), let's move it
out of the header.
Signed-off-by: JiSeop Moon <zcube@zcube.kr>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
It is a known issue that a rename() can fail with an "Access denied"
error at times, when copying followed by deleting the original file
works. Let's just fall back to that behavior.
Signed-off-by: JiSeop Moon <zcube@zcube.kr>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This is retry of #1419.
I added flush_fscache macro to flush cached stats after disk writing
with tests for regression reported in #1438 and #1442.
git checkout checks each file path in sorted order, so cache flushing does not
make performance worse unless we have large number of modified files in
a directory containing many files.
Using chromium repository, I tested `git checkout .` performance when I
delete 10 files in different directories.
With this patch:
TotalSeconds: 4.307272
TotalSeconds: 4.4863595
TotalSeconds: 4.2975562
Avg: 4.36372923333333
Without this patch:
TotalSeconds: 20.9705431
TotalSeconds: 22.4867685
TotalSeconds: 18.8968292
Avg: 20.7847136
I confirmed this patch passed all tests in t/ with core_fscache=1.
Signed-off-by: Takuto Ikuta <tikuta@chromium.org>
When I do git fetch, git call file stats under .git/objects for each
refs. This takes time when there are many refs.
By enabling fscache, git takes file stats by directory traversing and that
improved the speed of fetch-pack for repository having large number of
refs.
In my windows workstation, this improves the time of `git fetch` for
chromium repository like below. I took stats 3 times.
* With this patch
TotalSeconds: 9.9825165
TotalSeconds: 9.1862075
TotalSeconds: 10.1956256
Avg: 9.78811653333333
* Without this patch
TotalSeconds: 15.8406702
TotalSeconds: 15.6248053
TotalSeconds: 15.2085938
Avg: 15.5580231
Signed-off-by: Takuto Ikuta <tikuta@chromium.org>
Fix regression described in:
https://github.com/git-for-windows/git/issues/1392
which was introduced in:
b2353379bb
Problem Symptoms
================
When the user has a .gitignore file that is a symlink, the fscache
optimization introduced above caused the stat-data from the symlink,
rather that of the target file, to be returned. Later when the ignore
file was read, the buffer length did not match the stat.st_size field
and we called die("cannot use <path> as an exclude file")
Optimization Rationale
======================
The above optimization calls lstat() before open() primarily to ask
fscache if the file exists. It gets the current stat-data as a side
effect essentially for free (since we already have it in memory).
If the file does not exist, it does not need to call open(). And
since very few directories have .gitignore files, we can greatly
reduce time spent in the filesystem.
Discussion of Fix
=================
The above optimization calls lstat() rather than stat() because the
fscache only intercepts lstat() calls. Calls to stat() stay directed
to the mingw_stat() completly bypassing fscache. Furthermore, calls
to mingw_stat() always call {open, fstat, close} so that symlinks are
properly dereferenced, which adds *additional* open/close calls on top
of what the original code in dir.c is doing.
Since the problem only manifests for symlinks, we add code to overwrite
the stat-data when the path is a symlink. This preserves the effect of
the performance gains provided by the fscache in the normal case.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Teach "add" to use preload-index and fscache features
to improve performance on very large repositories.
During an "add", a call is made to run_diff_files()
which calls check_remove() for each index-entry. This
calls lstat(). On Windows, the fscache code intercepts
the lstat() calls and builds a private cache using the
FindFirst/FindNext routines, which are much faster.
Somewhat independent of this, is the preload-index code
which distributes some of the start-up costs across
multiple threads.
We need to keep the call to read_cache() before parsing the
pathspecs (and hence cannot use the pathspecs to limit any preload)
because parse_pathspec() is using the index to determine whether a
pathspec is, in fact, in a submodule. If we would not read the index
first, parse_pathspec() would not error out on a path that is inside
a submodule, and t7400-submodule-basic.sh would fail with
not ok 47 - do not add files from a submodule
We still want the nice preload performance boost, though, so we simply
call read_cache_preload(&pathspecs) after parsing the pathspecs.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Make fscache_enabled() function public rather than static.
Remove unneeded fscache_is_enabled() function.
Change is_fscache_enabled() macro to call fscache_enabled().
is_fscache_enabled() now takes a pathname so that the answer
is more precise and mean "is fscache enabled for this pathname",
since fscache only stores repo-relative paths and not absolute
paths, we can avoid attempting lookups for absolute paths.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
The purpose of this function is to stat() the files listed in the index
in a multi-threaded fashion. It is called directly after reading the
index in the read_index_preloaded() function.
However, in some cases we may want to separate the index reading from
the preloading step, e.g. in builtin/add.c, where we need to load the
index before we parse the pathspecs (which needs to error out if one of
the pathspecs refers to a path within a submodule, for which the index
must have been read already), and only then will we want to preload,
possibly limited by the just-parsed pathspecs.
So let's just export that function to allow calling it separately.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Teach read_directory_recursive() and add_excludes() to
be aware of optional fscache and avoid trying to open()
and fstat() non-existant ".gitignore" files in every
directory in the worktree.
The current code in add_excludes() calls open() and then
fstat() for a ".gitignore" file in each directory present
in the worktree. Change that when fscache is enabled to
call lstat() first and if present, call open().
This seems backwards because both lstat needs to do more
work than fstat. But when fscache is enabled, fscache will
already know if the .gitignore file exists and can completely
avoid the IO calls. This works because of the lstat diversion
to mingw_lstat when fscache is enabled.
This reduced status times on a 350K file enlistment of the
Windows repo on a NVMe SSD by 0.25 seconds.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
A ton of Git commands simply do not read (or at least parse) the core.*
settings. This is not good, as Git for Windows relies on the
core.longPaths setting to be read quite early on.
So let's just make sure that all commands read the config and give
platform_core_config() a chance.
This patch teaches tons of Git commands to respect the config setting
`core.longPaths = true`, including `pack-refs`, thereby fixing
https://github.com/git-for-windows/git/issues/1218
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The CreateProcessW() function does not really support spaces in its
first argument, lpApplicationName. But it supports passing NULL as
lpApplicationName, which makes it figure out the application from the
(possibly quoted) first argument of lpCommandLine.
Let's use that trick (if we are certain that the first argument matches
the executable's path) to support launching programs whose path contains
spaces.
This fixes https://github.com/git-for-windows/git/issue/692
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Teach FSCACHE to remember "not found" directories.
This is a performance optimization.
FSCACHE is a performance optimization available for Windows. It
intercepts Posix-style lstat() calls into an in-memory directory
using FindFirst/FindNext. It improves performance on Windows by
catching the first lstat() call in a directory, using FindFirst/
FindNext to read the list of files (and attribute data) for the
entire directory into the cache, and short-cut subsequent lstat()
calls in the same directory. This gives a major performance
boost on Windows.
However, it does not remember "not found" directories. When STATUS
runs and there are missing directories, the lstat() interception
fails to find the parent directory and simply return ENOENT for the
file -- it does not remember that the FindFirst on the directory
failed. Thus subsequent lstat() calls in the same directory, each
re-attempt the FindFirst. This completely defeats any performance
gains.
This can be seen by doing a sparse-checkout on a large repo and
then doing a read-tree to reset the skip-worktree bits and then
running status.
This change reduced status times for my very large repo by 60%.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We no longer use any of MSVCRT's stat-functions, so there's no need to
stick to a CRT-compatible 'struct stat' either.
Define and use our own POSIX-2013-compatible 'struct stat' with nanosecond-
precision file times.
Note: Due to performance issues when using git variants with different file
time resolutions, this patch does *not* yet enable nanosecond precision in
the Makefile (use 'make USE_NSEC=1').
Signed-off-by: Karsten Blees <blees@dcon.de>
