If multiple threads access a directory that is not yet in the cache, the
directory will be loaded by each thread. Only one of the results is added
to the cache, all others are leaked. This wastes performance and memory.
On cache miss, add a future object to the cache to indicate that the
directory is currently being loaded. Subsequent threads register themselves
with the future object and wait. When the first thread has loaded the
directory, it replaces the future object with the result and notifies
waiting threads.
Signed-off-by: Karsten Blees <blees@dcon.de>
Windows paths are typically limited to MAX_PATH = 260 characters, even
though the underlying NTFS file system supports paths up to 32,767 chars.
This limitation is also evident in Windows Explorer, cmd.exe and many
other applications (including IDEs).
Particularly annoying is that most Windows APIs return bogus error codes
if a relative path only barely exceeds MAX_PATH in conjunction with the
current directory, e.g. ERROR_PATH_NOT_FOUND / ENOENT instead of the
infinitely more helpful ERROR_FILENAME_EXCED_RANGE / ENAMETOOLONG.
Many Windows wide char APIs support longer than MAX_PATH paths through the
file namespace prefix ('\\?\' or '\\?\UNC\') followed by an absolute path.
Notable exceptions include functions dealing with executables and the
current directory (CreateProcess, LoadLibrary, Get/SetCurrentDirectory) as
well as the entire shell API (ShellExecute, SHGetSpecialFolderPath...).
Introduce a handle_long_path function to check the length of a specified
path properly (and fail with ENAMETOOLONG), and to optionally expand long
paths using the '\\?\' file namespace prefix. Short paths will not be
modified, so we don't need to worry about device names (NUL, CON, AUX).
Contrary to MSDN docs, the GetFullPathNameW function doesn't seem to be
limited to MAX_PATH (at least not on Win7), so we can use it to do the
heavy lifting of the conversion (translate '/' to '\', eliminate '.' and
'..', and make an absolute path).
Add long path error checking to xutftowcs_path for APIs with hard MAX_PATH
limit.
Add a new MAX_LONG_PATH constant and xutftowcs_long_path function for APIs
that support long paths.
While improved error checking is always active, long paths support must be
explicitly enabled via 'core.longpaths' option. This is to prevent end
users to shoot themselves in the foot by checking out files that Windows
Explorer, cmd/bash or their favorite IDE cannot handle.
Test suite:
Test the case is when the full pathname length of a dir is close
to 260 (MAX_PATH).
Bug report and an original reproducer by Andrey Rogozhnikov:
https://github.com/msysgit/git/pull/122#issuecomment-43604199
Note that the test cannot rely on the presence of short names, as they
are not enabled by default except on the system drive.
[jes: adjusted test number to avoid conflicts, reinstated && chain,
adjusted test to work without short names]
Thanks-to: Martin W. Kirst <maki@bitkings.de>
Thanks-to: Doug Kelly <dougk.ff7@gmail.com>
Signed-off-by: Karsten Blees <blees@dcon.de>
Original-test-by: Andrey Rogozhnikov <rogozhnikov.andrey@gmail.com>
Signed-off-by: Stepan Kasal <kasal@ucw.cz>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Checking the work tree status is quite slow on Windows, due to slow lstat
emulation (git calls lstat once for each file in the index). Windows
operating system APIs seem to be much better at scanning the status
of entire directories than checking single files.
Add an lstat implementation that uses a cache for lstat data. Cache misses
read the entire parent directory and add it to the cache. Subsequent lstat
calls for the same directory are served directly from the cache.
Also implement opendir / readdir / closedir so that they create and use
directory listings in the cache.
The cache doesn't track file system changes and doesn't plug into any
modifying file APIs, so it has to be explicitly enabled for git functions
that don't modify the working copy.
Note: in an earlier version of this patch, the cache was always active and
tracked file system changes via ReadDirectoryChangesW. However, this was
much more complex and had negative impact on the performance of modifying
git commands such as 'git checkout'.
Signed-off-by: Karsten Blees <blees@dcon.de>
Windows paths are typically limited to MAX_PATH = 260 characters, even
though the underlying NTFS file system supports paths up to 32,767 chars.
This limitation is also evident in Windows Explorer, cmd.exe and many
other applications (including IDEs).
Particularly annoying is that most Windows APIs return bogus error codes
if a relative path only barely exceeds MAX_PATH in conjunction with the
current directory, e.g. ERROR_PATH_NOT_FOUND / ENOENT instead of the
infinitely more helpful ERROR_FILENAME_EXCED_RANGE / ENAMETOOLONG.
Many Windows wide char APIs support longer than MAX_PATH paths through the
file namespace prefix ('\\?\' or '\\?\UNC\') followed by an absolute path.
Notable exceptions include functions dealing with executables and the
current directory (CreateProcess, LoadLibrary, Get/SetCurrentDirectory) as
well as the entire shell API (ShellExecute, SHGetSpecialFolderPath...).
Introduce a handle_long_path function to check the length of a specified
path properly (and fail with ENAMETOOLONG), and to optionally expand long
paths using the '\\?\' file namespace prefix. Short paths will not be
modified, so we don't need to worry about device names (NUL, CON, AUX).
Contrary to MSDN docs, the GetFullPathNameW function doesn't seem to be
limited to MAX_PATH (at least not on Win7), so we can use it to do the
heavy lifting of the conversion (translate '/' to '\', eliminate '.' and
'..', and make an absolute path).
Add long path error checking to xutftowcs_path for APIs with hard MAX_PATH
limit.
Add a new MAX_LONG_PATH constant and xutftowcs_long_path function for APIs
that support long paths.
While improved error checking is always active, long paths support must be
explicitly enabled via 'core.longpaths' option. This is to prevent end
users to shoot themselves in the foot by checking out files that Windows
Explorer, cmd/bash or their favorite IDE cannot handle.
Test suite:
Test the case is when the full pathname length of a dir is close
to 260 (MAX_PATH).
Bug report and an original reproducer by Andrey Rogozhnikov:
https://github.com/msysgit/git/pull/122#issuecomment-43604199
[jes: adjusted test number to avoid conflicts]
Thanks-to: Martin W. Kirst <maki@bitkings.de>
Thanks-to: Doug Kelly <dougk.ff7@gmail.com>
Signed-off-by: Karsten Blees <blees@dcon.de>
Original-test-by: Andrey Rogozhnikov <rogozhnikov.andrey@gmail.com>
Signed-off-by: Stepan Kasal <kasal@ucw.cz>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Add a macro to mark code sections that only read from the file system,
along with a config option and documentation.
This facilitates implementation of relatively simple file system level
caches without the need to synchronize with the file system.
Enable read-only sections for 'git status' and preload_index.
Signed-off-by: Karsten Blees <blees@dcon.de>
[jes: adusted test number to avoid conflicts, fixed non-portable use of
the 'export' statement]
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Emulating the POSIX lstat API on Windows via GetFileAttributes[Ex] is quite
slow. Windows operating system APIs seem to be much better at scanning the
status of entire directories than checking single files. A caching
implementation may improve performance by bulk-reading entire directories
or reusing data obtained via opendir / readdir.
Make the lstat implementation pluggable so that it can be switched at
runtime, e.g. based on a config option.
Signed-off-by: Karsten Blees <blees@dcon.de>
Emulating the POSIX dirent API on Windows via FindFirstFile/FindNextFile is
pretty staightforward, however, most of the information provided in the
WIN32_FIND_DATA structure is thrown away in the process. A more
sophisticated implementation may cache this data, e.g. for later reuse in
calls to lstat.
Make the dirent implementation pluggable so that it can be switched at
runtime, e.g. based on a config option.
Define a base DIR structure with pointers to readdir/closedir that match
the opendir implementation (i.e. similar to vtable pointers in OOP).
Define readdir/closedir so that they call the function pointers in the DIR
structure. This allows to choose the opendir implementation on a
call-by-call basis.
Move the fixed sized dirent.d_name buffer to the dirent-specific DIR
structure, as d_name may be implementation specific (e.g. a caching
implementation may just set d_name to point into the cache instead of
copying the entire file name string).
Signed-off-by: Karsten Blees <blees@dcon.de>
This commit starts the rebase of 62f3116 to 23415c2
This is just an intermediate rebase, used as an excuse for a little bit
of cleanup:
- a patch was moved from win-test-fixes to remote-hg-prerequisites,
where it really belongs
- the interactive-rebase branch thicket and the status-v2-porcelain
patch series have been updated to their newest respective iterations.
- rebased jeffhostetler/vs2015 to make it easier to upstream the
patches.
- the "use OpenSSL's SHA-1 routines for a nice speedup" patch got an
updated commit message and was moved to the msys2 patches.
- the cat-file-filters branch was updated to its latest iteration.
- the mingw/default-ident patch series was moved closer to the front, so
that it can be upstreamed before the more complicated patch series.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The asciidoctor doc-tool stack does not always respect the 'tab = 8 spaces' rule
expectation, particularly for the Git-for-Windows generated html pages. This
follows on from the 'doc: fix merge-base ASCII art tab spacing' fix.
Use just spaces within the block of the ascii art.
All other *.txt ascii art containing three dashes has been checked.
Asciidoctor correctly formats the other art blocks that do contain tabs.
Signed-off-by: Philip Oakley <philipoakley@iee.org
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When passing a command-line to call an external diff command to the
difftool, we must be prepared for paths containing special characters,
e.g. backslashes in the temporary directory's path on Windows.
This has been caught by running the test suite with an MSVC-built Git:
in contrast to the MINGW one, it does not rewrite `$TMP` to use forward
slashes instead of backslashes.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When compiling with MSVC, we rely on NuPkgs to provide the binaries of
dependencies such as libiconv. The libiconv 1.14.0.11 package available
from https://www.nuget.org/packages/libiconv seems to have a bug where
it does not set errno (when we would expect it to be E2BIG).
Let's simulate the error condition by taking less than 16 bytes
remaining in the out buffer as an indicator that we ran out of space.
While 16 might seem a bit excessive (when converting from, say, any
encoding to UTF-8, 8 bytes should be fine), it is designed to be a safe
margin.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Just like 1e2ce1d (sha1: Use OpenSSL SHA1 routines on MINGW, 2016-10-12),
we now use OpenSSL's SHA-1 routines instead of Git's own because OpenSSL
is substantially faster as of version 1.0.2: it now uses hardware
acceleration on Intel processors much more effectively.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Install required third-party DLLs next to EXEs.
Build and install release mode PDBs for git
executables allowing detailed stack traces
in the event of crash.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
The VS2015 version of the CRT asserts when you
pass a zero buffer length and request line buffering.
This fix sets it to the default BUFSIZ.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
In MSVC, the DEBUG constant is set automatically whenever compiling with
debug information.
This is clearly not what was intended in cache-tree.c, so let's use a less
ambiguous constant there.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
The hack that works in MINGW does not work with MSVC's CRT. Add MSVC
versions of isatty() and swap_osfhnd().
The MINGW versions attempt to replace the underlying OS HANDLE in an
existing file descriptor (fd) by writing to some undocumented fields in
the "ioinfo" structures inside the CRT. These structures changed size
and shape with the new UCRT in VS2015. The new MSVC versions of these
routines work without touching private fields. In theory, we should be
able to replace the ming versions with
this one.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
The MINGW version of the main() wrapper gets away with declaring symbols
that were intentionally not exported. However, some of these symbols do
not actually exist in MSVC's UCRT.
So let's add an MSVC version of the main() wrapper that uses wmain() and
imports the UNICODE argv and environment. While at it, we pass our UTF-8
version of ARGV to the real main -- rather than overwriting __argv as is
done in the MINGW Version.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
This special-cases various signals that are not supported on Windows,
such as SIGPIPE. These cause the UCRT to throw asserts (at least in
debug mode).
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
This adds MSVC versions of getenv() and friends. These take UTF-8
arguments and return UTF-8 values, but use the UNICODE versions
of the CRT routines. This avoids the need to write to __environ
(which is only visible if you statically link to the CRT). This
also avoids the CP_ACP conversions performed inside the CRT.
It also avoids various memory leaks and problems.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
GCC and MSVC disagree about using the GCC extension _ANONYMOUS_UNION.
Simply skip that offending keyword when compiling with MSVC.
Signed-off-by: Philip Oakley <philipoakley@iee.org>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This constant is not defined in MSVC's headers.
In UCRT's fcntl.h, _O_RDONLY, _O_WRONLY and _O_RDWR are defined as 0, 1
and 2, respectively. Yes, that means that UCRT breaks with the tradition
that O_RDWR == O_RDONLY | O_WRONLY.
It is a perfectly legal way to define those constants, though, therefore
we need to take care of defining O_ACCMODE accordingly.
This is particularly important in order to keep our "open() can set
errno to EISDIR" emulation working: it tests that (flags & O_ACCMODE) is
not identical to O_RDONLY before going on to test specifically whether
the file for which open() reported EACCES is, in fact, a directory.
Signed-off-by: Philip Oakley <philipoakley@iee.org>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Support -Z flags ("specify PDB options"), only include -l args on link
commands, and force PDBs to be created.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
This commit contains a GNU Makefile and NuGet configuration
scripts to download and install the various third-party
libraries that we will need to build/link with when using
VS2015 to build Git.
The file "compat/vcbuild/README_VS2015.txt" contains
instructions for using this.
In this commit, "compat/vcbuild/Makefile" contains hard-coded
version numbers of the packages we require. These are set to
the current versions as of the time of this commit. We use
"nuget restore" to install them explicitly using a "package.config".
A future improvement would try to use some of the automatic
package management functions and eliminate the need to specify
exact versions. I tried, but could not get this to work.
NuGet was happy dowload "minimum requirements" rather than
"lastest" for dependencies -- and only look at one package at
a time. For example, both curl and openssl depend upon zlib
and have different minimums. It was unclear which version of
zlib would be installed and seemed to be dependent on the order
of the top-level pacakges. So, I'm skipping that for now.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
The file compat/msvc.c includes compat/mingw.c, which means that we have
to recompile compat/msvc.o if compat/mingw.c changes.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This topic branch adds the (experimental) --stdin/-z options to `git
reset`. Those patches are still under review in the upstream Git project,
but are already merged in their experimental form into Git for Windows'
`master` branch, in preparation for a MinGit-only release.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Use OpenSSL SHA1 routines rather than builtin block-sha1 routines.
This improves performance on SHA1 operations on Intel processors.
OpenSSL 1.0.2 has made considerable performance improvements and
support the Intel hardware acceleration features. See:
https://software.intel.com/en-us/articles/improving-openssl-performancehttps://software.intel.com/en-us/articles/intel-sha-extensions
To test this I added/staged a single file in a gigantic
repository having a 450MB index file. The code in read-cache.c
verifies the header SHA as it reads the index and computes a new
header SHA as it writes out the new index. Therefore, in this test
the SHA code must process 900MB of data. Testing was done on an
Intel I7-4770 CPU @ 3.40GHz (Intel64, Family 6, Model 60) CPU.
The block-sha1 version averaged 5.27 seconds.
The OpenSSL version averaged 4.50 seconds.
================================================================
$ echo xxx >> project.mk
$ time /e/blk_sha/bin/git.exe add project.mk
real 0m5.207s
user 0m0.000s
sys 0m0.250s
$ echo xxx >> project.mk
$ time /e/blk_sha/bin/git.exe add project.mk
real 0m5.362s
user 0m0.015s
sys 0m0.234s
$ echo xxx >> project.mk
$ time /e/blk_sha/bin/git.exe add project.mk
real 0m5.300s
user 0m0.016s
sys 0m0.250s
$ echo xxx >> project.mk
$ time /e/blk_sha/bin/git.exe add project.mk
real 0m5.216s
user 0m0.000s
sys 0m0.250s
================================================================
$ echo xxx >> project.mk
$ time /e/openssl/bin/git.exe add project.mk
real 0m4.431s
user 0m0.000s
sys 0m0.250s
$ echo xxx >> project.mk
$ time /e/openssl/bin/git.exe add project.mk
real 0m4.478s
user 0m0.000s
sys 0m0.265s
$ echo xxx >> project.mk
$ time /e/openssl/bin/git.exe add project.mk
real 0m4.690s
user 0m0.000s
sys 0m0.250s
$ echo xxx >> project.mk
$ time /e/openssl/bin/git.exe add project.mk
real 0m4.420s
user 0m0.000s
sys 0m0.234s
================================================================
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
This topic branch works around an out-of-memory bug when the user
specified a format via --date=format:<format> that strftime() does
not like.
Reported by Stefan Naewe.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
With the recent update in efee955 (gpg-interface: check gpg signature
creation status, 2016-06-17), we ask GPG to send all status updates to
stderr, and then catch the stderr in an strbuf.
But GPG might fail, and send error messages to stderr. And we simply
do not show them to the user.
Even worse: this swallows any interactive prompt for a passphrase. And
detaches stderr from the tty so that the passphrase cannot be read.
So while the first problem could be fixed (by printing the captured
stderr upon error), the second problem cannot be easily fixed, and
presents a major regression.
So let's just revert commit efee9553a4.
This fixes https://github.com/git-for-windows/git/issues/871
Cc: Michael J Gruber <git@drmicha.warpmail.net>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
