Commit Graph

66291 Commits

Author SHA1 Message Date
Karsten Blees
ecf90d5518 mingw: Support git_terminal_prompt with more terminals
The `git_terminal_prompt()` function expects the terminal window to be
attached to a Win32 Console. However, this is not the case with terminal
windows other than `cmd.exe`'s, e.g. with MSys2's own `mintty`.

Non-cmd terminals such as `mintty` still have to have a Win32 Console
to be proper console programs, but have to hide the Win32 Console to
be able to provide more flexibility (such as being resizeable not only
vertically but also horizontally). By writing to that Win32 Console,
`git_terminal_prompt()` manages only to send the prompt to nowhere and
to wait for input from a Console to which the user has no access.

This commit introduces a function specifically to support `mintty` -- or
other terminals that are compatible with MSys2's `/dev/tty` emulation. We
use the `TERM` environment variable as an indicator for that: if the value
starts with "xterm" (such as `mintty`'s "xterm_256color"), we prefer to
let `xterm_prompt()` handle the user interaction.

The most prominent user of `git_terminal_prompt()` is certainly
`git-remote-https.exe`. It is an interesting use case because both
`stdin` and `stdout` are redirected when Git calls said executable, yet
it still wants to access the terminal.

When running inside a `mintty`, the terminal is not accessible to the
`git-remote-https.exe` program, though, because it is a MinGW program
and the `mintty` terminal is not backed by a Win32 console.

To solve that problem, we simply call out to the shell -- which is an
*MSys2* program and can therefore access `/dev/tty`.

Helped-by: nalla <nalla@hamal.uberspace.de>
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:30:33 +01:00
nalla
e77180fe83 mingw: explicitly fflush stdout
For performance reasons `stdout` is not unbuffered by default. That leads
to problems if after printing to `stdout` a read on `stdin` is performed.

For that reason interactive commands like `git clean -i` do not function
properly anymore if the `stdout` is not flushed by `fflush(stdout)` before
trying to read from `stdin`.

In the case of `git clean -i` all reads on `stdin` were preceded by a
`fflush(stdout)` call.

Signed-off-by: nalla <nalla@hamal.uberspace.de>
2016-12-12 22:30:33 +01:00
Karsten Blees
572a0d9184 mingw: initialize HOME on startup
HOME initialization was historically duplicated in many different places,
including /etc/profile, launch scripts such as git-bash.vbs and gitk.cmd,
and (although slightly broken) in the git-wrapper.

Even unrelated projects such as GitExtensions and TortoiseGit need to
implement the same logic to be able to call git directly.

Initialize HOME in git's own startup code so that we can eventually retire
all the duplicate initialization code.

Signed-off-by: Karsten Blees <blees@dcon.de>
2016-12-12 22:30:33 +01:00
Johannes Schindelin
501bf13e2c mingw: HOT FIX: work around environment issues -- again
This developer should really, really have known better. The fact that we
are changing the environment in ways for which the MSVCRT is not
prepared for is bad enough. But then this developer followed the request
to re-enable nedmalloc -- despite the prediction that it would cause an
access violation, predicting it in the same message as the request to
re-enable nedmalloc, no less!

To paper over the issue until the time when this developer finds the
time to re-design the Unicode environment handling from scratch, let's
hope that cURL is the only library we are using that *may* set an
environment variable using MSVCRT's putenv() after we fscked the
environment up.

Note: this commit can serve as no source of pride to anyone, certainly
not yours truly. It is necessary as a quick and pragmatic stop gap,
though, to prevent worse problems.

Note: cURL manages to set the variable CHARSET when nedmalloc is *not*
enabled, without causing an access violation. In that case, it sets it
successfully to the value "cp" + GetACP() (hence it is our choice, too,
cURL may need it, Git does not):

	https://github.com/bagder/curl/blob/aa5808b5/lib/easy.c#L157-L162

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:30:33 +01:00
Karsten Blees
023f2f9b29 gettext: always use UTF-8 on native Windows
Git on native Windows exclusively uses UTF-8 for console output (both with
mintty and native console windows). Gettext uses setlocale() to determine
the output encoding for translated text, however, MSVCRT's setlocale()
doesn't support UTF-8. As a result, translated text is encoded in system
encoding (GetAPC()), and non-ASCII chars are mangled in console output.

Use gettext's bind_textdomain_codeset() to force the encoding to UTF-8 on
native Windows.

In this developers' setup, HAVE_LIBCHARSET_H is apparently defined, but
we *really* want to override the locale_charset() here.

Signed-off-by: Karsten Blees <blees@dcon.de>
2016-12-12 22:30:32 +01:00
Johannes Schindelin
19573196cd Avoid illegal filenames when building Documentation on NTFS
A '+' is not a valid part of a filename with Windows file systems (it is
reserved because the '+' operator meant file concatenation back in the
DOS days).

Let's just not use it.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:30:32 +01:00
Johannes Schindelin
2eb1a374b6 git-gui (MinGW): make use of MSys2's msgfmt
When Git for Windows was still based on MSys1, we had no gettext, ergo
no msgfmt, either. Therefore, we introduced a small and simple Tcl
script to perform the same task.

However, with MSys2, we no longer need that because we have a proper
msgfmt executable. Plus, the po2msg.sh script somehow manages to hang
when run in parallel.

Two reasons to use real msgfmt.exe instead.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:30:32 +01:00
Johannes Schindelin
269c659e44 mingw: be *very* wary about outside environment changes
The environment is modified in most surprising circumstances, and not
all of them are under Git's control. For example, calling
curl_global_init() on Windows will ensure that the CHARSET variable is
set, adding one if necessary.

While the previous commit worked around crashes triggered by such
outside changes of the environment by relaxing the requirement that the
environment be terminated by a NULL pointer, the other assumption made
by `mingw_getenv()` and `mingw_putenv()` is that the environment is
sorted, for efficient lookup via binary search.

Let's make real sure that our environment is intact before querying or
modifying it, and reinitialize our idea of the environment if necessary.

With this commit, before working on the environment we look briefly for
indicators that the environment was modified outside of our control, and
to ensure that it is terminated with a NULL pointer and sorted again in
that case.

Note: the indicators are maybe not sufficient. For example, when a
variable is removed, it will not be noticed. It might also be a problem
if outside changes to the environment result in a modified `environ`
pointer: it is unclear whether such a modification could result in a
problem when `mingw_putenv()` needs to `realloc()` the environment
buffer.

For the moment, however, the current fix works well enough, so let's
only face the potential problems when (and if!) they occur.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:30:32 +01:00
Johannes Schindelin
9b0ce1aa43 mingw: be more defensive when making the environment block
Outside of our Windows-specific code, the end of the environment can be
marked also by a pointer to a NUL character, not only by a NULL pointer
as our code assumed so far.

That led to a buffer overrun in `make_environment_block()` when running
`git-remote-https` in `mintty` (because `curl_global_init()` added the
`CHARSET` environment variable *outside* of `mingw_putenv()`, ending the
environment in a pointer to an empty string).

Side note for future debugging on Windows: when running programs in
`mintty`, the standard input/output/error is not connected to a Win32
Console, but instead is pipe()d. That means that even stderr may not be
written completely before a crash, but has to be fflush()ed explicitly.
For example, when debugging crashes, the developer should insert an
`fflush(stderr);` at the end of the `error()` function defined in
usage.c.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:30:32 +01:00
Johannes Schindelin
85adf2dde7 UTF-8 environment: be a little bit more defensive
It is unlikely that we have an empty environment, ever, but *if* we do,
when `environ_size - 1` is passed to `bsearchenv()` it is misinterpreted
as a real large integer.

To make the code truly defensive, refuse to do anything at all if the
size is negative (which should not happen, of course).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:30:32 +01:00
Johannes Schindelin
60806cf921 mingw: enable stack smashing protector
As suggested privately to Brendan Forster by some unnamed person
(suggestion for the future: use the public mailing list, or even the
public GitHub issue tracker, that is a much better place to offer such
suggestions), we should make use of gcc's stack smashing protector that
helps detect stack buffer overruns early.

Rather than using -fstack-protector, we use -fstack-protector-strong
because it strikes a better balance between how much code is affected
and the performance impact.

In a local test (time git log --grep=is -p), best of 5 timings went from
23.009s to 22.997s (i.e. the performance impact was *well* lost in the
noise).

This fixes https://github.com/git-for-windows/git/issues/501

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:30:32 +01:00
Alan Davies
e61e83247f winansi.c: Fix colourization on Cygwin pseudo terminals.
Git only colours the output and uses pagination if isatty() returns 1.
MSys and Cygwin emulate pseudo terminals via named pipes, meaning that
isatty() returns 0.

Commit 3adef8de55 fixed this for MSys
terminals, but not Cygwin.

The named pipes that Cygwin and Msys use are very similar.  MSys PTY pipes
are called 'msys-*-pty*' and Cygwin uses 'cygwin-*-pty*'.  This commit
modifies the existing check to allow both MSys and Cygwin PTY pipes to be
identified as TTYs.

Note that Pagination is still broken on Cygwin.  less.exe is spawned (as
seen in Process Explorer and using GIT_TRACE=1), but the output is not
being piped into it.

This partially fixes https://github.com/git-for-windows/git/issues/267

Signed-off-by: Alan Davies <alan.n.davies@gmail.com>
2016-12-12 22:30:32 +01:00
Cesar Eduardo Barros
9724cd4dfa mingw: Embed a manifest to trick UAC into Doing The Right Thing
On Windows >= Vista, not having an application manifest with a
requestedExecutionLevel can cause several kinds of confusing behavior.

The first and more obvious behavior is "Installer Detection", where
Windows sometimes decides (by looking at things like the file name and
even sequences of bytes within the executable) that an executable is an
installer and should run elevated (causing the well-known popup dialog
to appear). In Git's context, subcommands such as "git patch-id" or "git
update-index" fall prey to this behavior.

The second and more confusing behavior is "File Virtualization". It
means that when files are written without having write permission, it
does not fail (as expected), but they are instead redirected to
somewhere else. When the files are read, the original contents are
returned, though, not the ones that were just written somewhere else.
Even more confusing, not all write accesses are redirected; Trying to
write to write-protected .exe files, for example, will fail instead of
redirecting.

In addition to being unwanted behavior, File Virtualization causes
dramatic slowdowns in Git (see for instance
http://code.google.com/p/msysgit/issues/detail?id=320).

There are two ways to prevent those two behaviors: Either you embed an
application manifest within all your executables, or you add an external
manifest (a file with the same name followed by .manifest) to all your
executables. Since Git's builtins are hardlinked (or copied), it is
simpler and more robust to embed a manifest.

A recent enough MSVC compiler should already embed a working internal
manifest, but for MinGW you have to do so by hand.

Very lightly tested on Wine, where like on Windows XP it should not make
any difference.

References:
  - New UAC Technologies for Windows Vista
    http://msdn.microsoft.com/en-us/library/bb756960.aspx
  - Create and Embed an Application Manifest (UAC)
    http://msdn.microsoft.com/en-us/library/bb756929.aspx

[js: simplified the embedding dramatically by reusing Git for Windows'
existing Windows resource file, removed the optional (and dubious)
processorArchitecture attribute of the manifest's assemblyIdentity
section.]

Signed-off-by: Cesar Eduardo Barros <cesarb@cesarb.net>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:30:32 +01:00
Johannes Schindelin
f32e90c6d2 Tests: optionally skip redirecting stdin/stdout/stderr
There is a really useful debugging technique developed by Sverre
Rabbelier that inserts "bash &&" somewhere in the test scripts, letting
the developer interact at given points with the current state.

Another debugging technique, used a lot by this here coder, is to run
certain executables via gdb by guarding a "gdb -args" call in
bin-wrappers/git.

Both techniques were disabled by 781f76b1(test-lib: redirect stdin of
tests).

Let's reinstate the ability to run an interactive shell by making the
redirection optional: setting the TEST_NO_REDIRECT environment variable
will skip the redirection.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:30:32 +01:00
Johannes Schindelin
b1e3b87753 Error out when mingw_startup() *and* NO_UNSETENV are active
The unsetenv code has no idea to update our environ_size, therefore
causing segmentation faults when environment variables are removed
without compat/mingw.c's knowing (MinGW's optimized lookup would try
to strcmp() against NULL in such a case).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:30:31 +01:00
Johannes Schindelin
7b109f9169 Build Python stuff with MSys2
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:30:31 +01:00
Johannes Schindelin
88d6eee1b0 Help debugging with MSys2 by optionally executing bash with strace
MSys2's strace facility is very useful for debugging... With this patch,
the bash will be executed through strace if the environment variable
GIT_STRACE_COMMANDS is set, which comes in real handy when investigating
issues in the test suite.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:30:31 +01:00
Sebastian Schuberth
9e7740b77e Makefile: Set htmldir to match the default HTML docs location under MSYS
Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
2016-12-12 22:30:31 +01:00
Sebastian Schuberth
370bad7315 MinGW: Use MakeMaker to build the Perl libraries
This way the libraries get properly installed into the "site_perl"
directory and we just have to move them out of the "mingw" directory.

Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
2016-12-12 22:30:31 +01:00
Johannes Schindelin
4d80697bd6 http: also treat config options sslCert and sslKey as paths
This is a companion patch to bf9acba (http: treat config options
sslCAPath and sslCAInfo as paths, 2015-11-23).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:30:31 +01:00
Johannes Schindelin
1848244d3e mingw: handle absolute paths in expand_user_path()
On Windows, an absolute POSIX path needs to be turned into a Windows
one.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:30:31 +01:00
Jeff Hostetler
1d25f8a390 mingw: use OpenSSL's SHA-1 routines
Use OpenSSL's SHA-1 routines rather than builtin block-sha1 routines.
This improves performance on SHA1 operations on Intel processors.

OpenSSL 1.0.2 has made considerable performance improvements and
support the Intel hardware acceleration features.  See:
https://software.intel.com/en-us/articles/improving-openssl-performance
https://software.intel.com/en-us/articles/intel-sha-extensions

To test this I added/staged a single file in a gigantic
repository having a 450MB index file.  The code in read-cache.c
verifies the header SHA as it reads the index and computes a new
header SHA as it writes out the new index.  Therefore, in this test
the SHA code must process 900MB of data.  Testing was done on an
Intel I7-4770 CPU @ 3.40GHz (Intel64, Family 6, Model 60) CPU.

The block-sha1 version averaged 5.27 seconds.
The OpenSSL    version averaged 4.50 seconds.

================================================================

$ echo xxx >> project.mk
$ time /e/blk_sha/bin/git.exe add project.mk

real    0m5.207s
user    0m0.000s
sys     0m0.250s

$ echo xxx >> project.mk
$ time /e/blk_sha/bin/git.exe add project.mk

real    0m5.362s
user    0m0.015s
sys     0m0.234s

$ echo xxx >> project.mk
$ time /e/blk_sha/bin/git.exe add project.mk

real    0m5.300s
user    0m0.016s
sys     0m0.250s

$ echo xxx >> project.mk
$ time /e/blk_sha/bin/git.exe add project.mk

real    0m5.216s
user    0m0.000s
sys     0m0.250s

================================================================
$ echo xxx >> project.mk
$ time /e/openssl/bin/git.exe add project.mk

real    0m4.431s
user    0m0.000s
sys     0m0.250s

$ echo xxx >> project.mk
$ time /e/openssl/bin/git.exe add project.mk

real    0m4.478s
user    0m0.000s
sys     0m0.265s

$ echo xxx >> project.mk
$ time /e/openssl/bin/git.exe add project.mk

real    0m4.690s
user    0m0.000s
sys     0m0.250s

$ echo xxx >> project.mk
$ time /e/openssl/bin/git.exe add project.mk

real    0m4.420s
user    0m0.000s
sys     0m0.234s

================================================================

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:30:31 +01:00
Johannes Schindelin
cece61f6ac Merge pull request #93 from nalla/asciidoctor-fixes
Asciidoctor fixes

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:30:31 +01:00
Johannes Schindelin
f5c18aac8b Merge 'long-paths' into HEAD 2016-12-12 22:30:30 +01:00
Johannes Schindelin
2039e1b25f Merge 'fscache' into HEAD 2016-12-12 22:30:29 +01:00
Johannes Schindelin
3ccd3068b8 Merge branch 'perl5lib'
With this topic branch, the PERL5LIB variable is unset to avoid external
settings from interfering with Git's own Perl interpreter.

This branch also cleans up some of our Windows-only config setting code
(and this will need to be rearranged in the next merging rebase so that
the cleanup comes first, and fscache and longPaths support build on
top).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:30:29 +01:00
Johannes Schindelin
3bcfef5b41 giteveryday: unbreak rendering with AsciiDoctor
The "giteveryday" document has a callout list that contains a code
block. This is not a problem for AsciiDoc, but AsciiDoctor sadly was
explicitly designed *not* to render this correctly [*1*]. The symptom is
an unhelpful

	line 322: callout list item index: expected 1 got 12
	line 325: no callouts refer to list item 1
	line 325: callout list item index: expected 2 got 13
	line 327: no callouts refer to list item 2

In Git for Windows, we rely on the speed improvement of AsciiDoctor (on
this developer's machine, `make -j15 html` takes roughly 30 seconds with
AsciiDoctor, 70 seconds with AsciiDoc), therefore we need a way to
render this correctly.

The easiest way out is to simplify the callout list, as suggested by
AsciiDoctor's author, even while one may very well disagree with him
that a code block hath no place in a callout list.

*1*: https://github.com/asciidoctor/asciidoctor/issues/1478

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:30:27 +01:00
nalla
4518b597da asciidoctor: fix user-manual to be built by asciidoctor
The `user-manual.txt` is designed as a `book` but the `Makefile` wants
to build it as an `article`. This seems to be a problem when building
the documentation with `asciidoctor`. Furthermore the parts *Git
Glossary* and *Appendix B* had no subsections which is not allowed when
building with `asciidoctor`. So lets add a *dummy* section.

Signed-off-by: nalla <nalla@hamal.uberspace.de>
2016-12-12 22:30:27 +01:00
Karsten Blees
458deacd7f Win32: fix 'lstat("dir/")' with long paths
Use a suffciently large buffer to strip the trailing slash.

Signed-off-by: Karsten Blees <blees@dcon.de>
2016-12-12 22:30:26 +01:00
Karsten Blees
38443a00d7 Win32: support long paths
Windows paths are typically limited to MAX_PATH = 260 characters, even
though the underlying NTFS file system supports paths up to 32,767 chars.
This limitation is also evident in Windows Explorer, cmd.exe and many
other applications (including IDEs).

Particularly annoying is that most Windows APIs return bogus error codes
if a relative path only barely exceeds MAX_PATH in conjunction with the
current directory, e.g. ERROR_PATH_NOT_FOUND / ENOENT instead of the
infinitely more helpful ERROR_FILENAME_EXCED_RANGE / ENAMETOOLONG.

Many Windows wide char APIs support longer than MAX_PATH paths through the
file namespace prefix ('\\?\' or '\\?\UNC\') followed by an absolute path.
Notable exceptions include functions dealing with executables and the
current directory (CreateProcess, LoadLibrary, Get/SetCurrentDirectory) as
well as the entire shell API (ShellExecute, SHGetSpecialFolderPath...).

Introduce a handle_long_path function to check the length of a specified
path properly (and fail with ENAMETOOLONG), and to optionally expand long
paths using the '\\?\' file namespace prefix. Short paths will not be
modified, so we don't need to worry about device names (NUL, CON, AUX).

Contrary to MSDN docs, the GetFullPathNameW function doesn't seem to be
limited to MAX_PATH (at least not on Win7), so we can use it to do the
heavy lifting of the conversion (translate '/' to '\', eliminate '.' and
'..', and make an absolute path).

Add long path error checking to xutftowcs_path for APIs with hard MAX_PATH
limit.

Add a new MAX_LONG_PATH constant and xutftowcs_long_path function for APIs
that support long paths.

While improved error checking is always active, long paths support must be
explicitly enabled via 'core.longpaths' option. This is to prevent end
users to shoot themselves in the foot by checking out files that Windows
Explorer, cmd/bash or their favorite IDE cannot handle.

Test suite:
Test the case is when the full pathname length of a dir is close
to 260 (MAX_PATH).
Bug report and an original reproducer by Andrey Rogozhnikov:
https://github.com/msysgit/git/pull/122#issuecomment-43604199

Note that the test cannot rely on the presence of short names, as they
are not enabled by default except on the system drive.

[jes: adjusted test number to avoid conflicts, reinstated && chain,
adjusted test to work without short names]

Thanks-to: Martin W. Kirst <maki@bitkings.de>
Thanks-to: Doug Kelly <dougk.ff7@gmail.com>
Signed-off-by: Karsten Blees <blees@dcon.de>
Original-test-by: Andrey Rogozhnikov <rogozhnikov.andrey@gmail.com>
Signed-off-by: Stepan Kasal <kasal@ucw.cz>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:30:26 +01:00
Johannes Schindelin
60f68f7afa Win32: support long paths
Windows paths are typically limited to MAX_PATH = 260 characters, even
though the underlying NTFS file system supports paths up to 32,767 chars.
This limitation is also evident in Windows Explorer, cmd.exe and many
other applications (including IDEs).

Particularly annoying is that most Windows APIs return bogus error codes
if a relative path only barely exceeds MAX_PATH in conjunction with the
current directory, e.g. ERROR_PATH_NOT_FOUND / ENOENT instead of the
infinitely more helpful ERROR_FILENAME_EXCED_RANGE / ENAMETOOLONG.

Many Windows wide char APIs support longer than MAX_PATH paths through the
file namespace prefix ('\\?\' or '\\?\UNC\') followed by an absolute path.
Notable exceptions include functions dealing with executables and the
current directory (CreateProcess, LoadLibrary, Get/SetCurrentDirectory) as
well as the entire shell API (ShellExecute, SHGetSpecialFolderPath...).

Introduce a handle_long_path function to check the length of a specified
path properly (and fail with ENAMETOOLONG), and to optionally expand long
paths using the '\\?\' file namespace prefix. Short paths will not be
modified, so we don't need to worry about device names (NUL, CON, AUX).

Contrary to MSDN docs, the GetFullPathNameW function doesn't seem to be
limited to MAX_PATH (at least not on Win7), so we can use it to do the
heavy lifting of the conversion (translate '/' to '\', eliminate '.' and
'..', and make an absolute path).

Add long path error checking to xutftowcs_path for APIs with hard MAX_PATH
limit.

Add a new MAX_LONG_PATH constant and xutftowcs_long_path function for APIs
that support long paths.

While improved error checking is always active, long paths support must be
explicitly enabled via 'core.longpaths' option. This is to prevent end
users to shoot themselves in the foot by checking out files that Windows
Explorer, cmd/bash or their favorite IDE cannot handle.

Test suite:
Test the case is when the full pathname length of a dir is close
to 260 (MAX_PATH).
Bug report and an original reproducer by Andrey Rogozhnikov:
https://github.com/msysgit/git/pull/122#issuecomment-43604199

[jes: adjusted test number to avoid conflicts]

Thanks-to: Martin W. Kirst <maki@bitkings.de>
Thanks-to: Doug Kelly <dougk.ff7@gmail.com>
Signed-off-by: Karsten Blees <blees@dcon.de>
Original-test-by: Andrey Rogozhnikov <rogozhnikov.andrey@gmail.com>
Signed-off-by: Stepan Kasal <kasal@ucw.cz>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:30:26 +01:00
Doug Kelly
343b7da19f Add a test demonstrating a problem with long submodule paths
[jes: adusted test number to avoid conflicts, fixed non-portable use of
the 'export' statement]

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:30:26 +01:00
Karsten Blees
f54e3cd3db fscache: load directories only once
If multiple threads access a directory that is not yet in the cache, the
directory will be loaded by each thread. Only one of the results is added
to the cache, all others are leaked. This wastes performance and memory.

On cache miss, add a future object to the cache to indicate that the
directory is currently being loaded. Subsequent threads register themselves
with the future object and wait. When the first thread has loaded the
directory, it replaces the future object with the result and notifies
waiting threads.

Signed-off-by: Karsten Blees <blees@dcon.de>
2016-12-12 22:30:25 +01:00
Karsten Blees
478f52349a Win32: add a cache below mingw's lstat and dirent implementations
Checking the work tree status is quite slow on Windows, due to slow lstat
emulation (git calls lstat once for each file in the index). Windows
operating system APIs seem to be much better at scanning the status
of entire directories than checking single files.

Add an lstat implementation that uses a cache for lstat data. Cache misses
read the entire parent directory and add it to the cache. Subsequent lstat
calls for the same directory are served directly from the cache.

Also implement opendir / readdir / closedir so that they create and use
directory listings in the cache.

The cache doesn't track file system changes and doesn't plug into any
modifying file APIs, so it has to be explicitly enabled for git functions
that don't modify the working copy.

Note: in an earlier version of this patch, the cache was always active and
tracked file system changes via ReadDirectoryChangesW. However, this was
much more complex and had negative impact on the performance of modifying
git commands such as 'git checkout'.

Signed-off-by: Karsten Blees <blees@dcon.de>
2016-12-12 22:30:25 +01:00
Karsten Blees
b98f8dc535 add infrastructure for read-only file system level caches
Add a macro to mark code sections that only read from the file system,
along with a config option and documentation.

This facilitates implementation of relatively simple file system level
caches without the need to synchronize with the file system.

Enable read-only sections for 'git status' and preload_index.

Signed-off-by: Karsten Blees <blees@dcon.de>
2016-12-12 22:30:25 +01:00
Karsten Blees
7d4f4fe070 Win32: make the lstat implementation pluggable
Emulating the POSIX lstat API on Windows via GetFileAttributes[Ex] is quite
slow. Windows operating system APIs seem to be much better at scanning the
status of entire directories than checking single files. A caching
implementation may improve performance by bulk-reading entire directories
or reusing data obtained via opendir / readdir.

Make the lstat implementation pluggable so that it can be switched at
runtime, e.g. based on a config option.

Signed-off-by: Karsten Blees <blees@dcon.de>
2016-12-12 22:30:25 +01:00
Karsten Blees
831025d027 Win32: Make the dirent implementation pluggable
Emulating the POSIX dirent API on Windows via FindFirstFile/FindNextFile is
pretty staightforward, however, most of the information provided in the
WIN32_FIND_DATA structure is thrown away in the process. A more
sophisticated implementation may cache this data, e.g. for later reuse in
calls to lstat.

Make the dirent implementation pluggable so that it can be switched at
runtime, e.g. based on a config option.

Define a base DIR structure with pointers to readdir/closedir that match
the opendir implementation (i.e. similar to vtable pointers in OOP).
Define readdir/closedir so that they call the function pointers in the DIR
structure. This allows to choose the opendir implementation on a
call-by-call basis.

Move the fixed sized dirent.d_name buffer to the dirent-specific DIR
structure, as d_name may be implementation specific (e.g. a caching
implementation may just set d_name to point into the cache instead of
copying the entire file name string).

Signed-off-by: Karsten Blees <blees@dcon.de>
2016-12-12 22:30:25 +01:00
Karsten Blees
cffb362830 Win32: dirent.c: Move opendir down
Move opendir down in preparation for the next patch.

Signed-off-by: Karsten Blees <blees@dcon.de>
2016-12-12 22:30:25 +01:00
Karsten Blees
4cfbcc315e Win32: make FILETIME conversion functions public
Signed-off-by: Karsten Blees <blees@dcon.de>
2016-12-12 22:30:25 +01:00
Johannes Schindelin
b76723d910 mingw: unset PERL5LIB by default
Git for Windows ships with its own Perl interpreter, and insists on
using it, so it will most likely wreak havoc if PERL5LIB is set before
launching Git.

Let's just unset that environment variables when spawning processes.

To make this feature extensible (and overrideable), there is a new
config setting `core.unsetenvvars` that allows specifying a
comma-separated list of names to unset before spawning processes.

Reported by Gabriel Fuhrmann.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:30:24 +01:00
Johannes Schindelin
f7099c44ad Move Windows-specific config settings into compat/mingw.c
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:30:24 +01:00
Johannes Schindelin
2a89f281b1 Allow for platform-specific core.* config settings
In the Git for Windows project, we have ample precendent for config
settings that apply to Windows, and to Windows only.

Let's formalize this concept by introducing a platform_core_config()
function that can be #define'd in a platform-specific manner.

This will allow us to contain platform-specific code better, as the
corresponding variables no longer need to be exported so that they can
be defined in environment.c and be set in config.c

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:30:24 +01:00
Johannes Schindelin
3bf90d5376 mingw: include the full version information in the resources
This fixes https://github.com/git-for-windows/git/issues/723

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:30:23 +01:00
Johannes Schindelin
58e7864ffb Start the merging-rebase to junio/maint
The changes in upstream/maint are actually not really that crucial for Git
for Windows, but we can use it as an excuse to fix the branch thicket: the
`visual-studio` branch clearly wants to be based on the
`jeffhostetler/vs2015` branch, as the latter contains important compile
fixes for Visual C, required to build inside Visual Studio.

While at it, we also move the Vagrant patches farther to the end, as they
are unlikely to be accepted upstream, so we'll try only after getting the
MSVC/VS patches in.

Also, let's reorder a couple other patches into more logical places.

This commit starts the rebase of 4638a925c0 to a274e0a036

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 22:29:21 +01:00
Johannes Schindelin
6119d9b81b Merge pull request #991 from jeffhostetler/jeffhostetler/string_list_realloc
string-list: use ALLOC_GROW macro when reallocing string_list
2016-12-12 22:06:20 +01:00
Johannes Schindelin
e62496e150 mingw: intercept isatty() to handle /dev/null as Git expects it
When Git's source code calls isatty(), it really asks whether the
respective file descriptor is connected to an interactive terminal.

Windows' _isatty() function, however, determines whether the file
descriptor is associated with a character device. And NUL, Windows'
equivalent of /dev/null, is a character device.

Which means that for years, Git mistakenly detected an associated
interactive terminal when being run through the test suite, which
almost always redirects stdin, stdout and stderr to /dev/null.

This bug only became obvious, and painfully so, when the new
bisect--helper entered the `pu` branch and made the automatic build & test
time out because t6030 was waiting for an answer.

For details, see

	https://msdn.microsoft.com/en-us/library/f4s0ddew.aspx

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 21:51:27 +01:00
Johannes Schindelin
518cd0cece Merge tag 'prerelease-v2.11.0.windows.1.1'
This merges the "Pre-release to test fixes for #981 and #987".

These commits were actually meant to land in `master` before merging the
Pull Requests; Better late than never.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-12 21:49:20 +01:00
Jeff Hostetler
642d8280a4 string-list: use ALLOC_GROW macro when reallocing string_list
Use ALLOC_GROW() macro when reallocing a string_list array
rather than simply increasing it by 32.  This is a performance
optimization.

During status on a very large repo and there are many changes,
a significant percentage of the total run time was spent
reallocing the wt_status.changes array.

This change decreased the time in wt_status_collect_changes_worktree()
from 125 seconds to 45 seconds on my very large repository.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
2016-12-12 12:49:46 -05:00
Johannes Schindelin
13d7269108 Merge pull request #988 from jeffhostetler/jeffhostetler/quick_add_index_entry
read-cache: speed up add_index_entry during checkout
2016-12-12 18:21:56 +01:00
Johannes Schindelin
c480725840 version --build-options: report commit, too, if possible
In particular when local tags are used (or tags that are pushed to some
fork) to build Git, it is very hard to figure out from which particular
revision a particular Git executable was built.

Let's just report that in our build options.

We need to be careful, though, to report when the current commit cannot be
determined, e.g. when building from a tarball without any associated Git
repository.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2016-12-09 17:53:07 +01:00