MSys2's strace facility is very useful for debugging... With this patch,
the bash will be executed through strace if the environment variable
GIT_STRACE_COMMANDS is set, which comes in real handy when investigating
issues in the test suite.
Also support passing a path to a log file via GIT_STRACE_COMMANDS to
force Git to call strace.exe with the `-o <path>` argument, i.e. to log
into a file rather than print the log directly.
That comes in handy when the output would otherwise misinterpreted by a
calling process as part of Git's output.
Note: the values "1", "yes" or "true" are *not* specifying paths, but
tell Git to let strace.exe log directly to the console.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Windows paths are typically limited to MAX_PATH = 260 characters, even
though the underlying NTFS file system supports paths up to 32,767 chars.
This limitation is also evident in Windows Explorer, cmd.exe and many
other applications (including IDEs).
Particularly annoying is that most Windows APIs return bogus error codes
if a relative path only barely exceeds MAX_PATH in conjunction with the
current directory, e.g. ERROR_PATH_NOT_FOUND / ENOENT instead of the
infinitely more helpful ERROR_FILENAME_EXCED_RANGE / ENAMETOOLONG.
Many Windows wide char APIs support longer than MAX_PATH paths through the
file namespace prefix ('\\?\' or '\\?\UNC\') followed by an absolute path.
Notable exceptions include functions dealing with executables and the
current directory (CreateProcess, LoadLibrary, Get/SetCurrentDirectory) as
well as the entire shell API (ShellExecute, SHGetSpecialFolderPath...).
Introduce a handle_long_path function to check the length of a specified
path properly (and fail with ENAMETOOLONG), and to optionally expand long
paths using the '\\?\' file namespace prefix. Short paths will not be
modified, so we don't need to worry about device names (NUL, CON, AUX).
Contrary to MSDN docs, the GetFullPathNameW function doesn't seem to be
limited to MAX_PATH (at least not on Win7), so we can use it to do the
heavy lifting of the conversion (translate '/' to '\', eliminate '.' and
'..', and make an absolute path).
Add long path error checking to xutftowcs_path for APIs with hard MAX_PATH
limit.
Add a new MAX_LONG_PATH constant and xutftowcs_long_path function for APIs
that support long paths.
While improved error checking is always active, long paths support must be
explicitly enabled via 'core.longpaths' option. This is to prevent end
users to shoot themselves in the foot by checking out files that Windows
Explorer, cmd/bash or their favorite IDE cannot handle.
Test suite:
Test the case is when the full pathname length of a dir is close
to 260 (MAX_PATH).
Bug report and an original reproducer by Andrey Rogozhnikov:
https://github.com/msysgit/git/pull/122#issuecomment-43604199
[jes: adjusted test number to avoid conflicts, added support for
chdir(), etc]
Thanks-to: Martin W. Kirst <maki@bitkings.de>
Thanks-to: Doug Kelly <dougk.ff7@gmail.com>
Signed-off-by: Karsten Blees <blees@dcon.de>
Original-test-by: Andrey Rogozhnikov <rogozhnikov.andrey@gmail.com>
Signed-off-by: Stepan Kasal <kasal@ucw.cz>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
There is a problem in the way 9ac3f0e5b3 (pack-objects: fix
performance issues on packing large deltas, 2018-07-22) initializes that
mutex in the `packing_data` struct. The problem manifests in a
segmentation fault on Windows, when a mutex (AKA critical section) is
accessed without being initialized. (With pthreads, you apparently do
not really have to initialize them?)
This was reported in https://github.com/git-for-windows/git/issues/1839.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
If multiple threads access a directory that is not yet in the cache, the
directory will be loaded by each thread. Only one of the results is added
to the cache, all others are leaked. This wastes performance and memory.
On cache miss, add a future object to the cache to indicate that the
directory is currently being loaded. Subsequent threads register themselves
with the future object and wait. When the first thread has loaded the
directory, it replaces the future object with the result and notifies
waiting threads.
Signed-off-by: Karsten Blees <blees@dcon.de>
Checking the work tree status is quite slow on Windows, due to slow lstat
emulation (git calls lstat once for each file in the index). Windows
operating system APIs seem to be much better at scanning the status
of entire directories than checking single files.
Add an lstat implementation that uses a cache for lstat data. Cache misses
read the entire parent directory and add it to the cache. Subsequent lstat
calls for the same directory are served directly from the cache.
Also implement opendir / readdir / closedir so that they create and use
directory listings in the cache.
The cache doesn't track file system changes and doesn't plug into any
modifying file APIs, so it has to be explicitly enabled for git functions
that don't modify the working copy.
Note: in an earlier version of this patch, the cache was always active and
tracked file system changes via ReadDirectoryChangesW. However, this was
much more complex and had negative impact on the performance of modifying
git commands such as 'git checkout'.
Signed-off-by: Karsten Blees <blees@dcon.de>
Add a macro to mark code sections that only read from the file system,
along with a config option and documentation.
This facilitates implementation of relatively simple file system level
caches without the need to synchronize with the file system.
Enable read-only sections for 'git status' and preload_index.
Signed-off-by: Karsten Blees <blees@dcon.de>
Emulating the POSIX lstat API on Windows via GetFileAttributes[Ex] is quite
slow. Windows operating system APIs seem to be much better at scanning the
status of entire directories than checking single files. A caching
implementation may improve performance by bulk-reading entire directories
or reusing data obtained via opendir / readdir.
Make the lstat implementation pluggable so that it can be switched at
runtime, e.g. based on a config option.
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Emulating the POSIX dirent API on Windows via FindFirstFile/FindNextFile is
pretty staightforward, however, most of the information provided in the
WIN32_FIND_DATA structure is thrown away in the process. A more
sophisticated implementation may cache this data, e.g. for later reuse in
calls to lstat.
Make the dirent implementation pluggable so that it can be switched at
runtime, e.g. based on a config option.
Define a base DIR structure with pointers to readdir/closedir that match
the opendir implementation (i.e. similar to vtable pointers in OOP).
Define readdir/closedir so that they call the function pointers in the DIR
structure. This allows to choose the opendir implementation on a
call-by-call basis.
Move the fixed sized dirent.d_name buffer to the dirent-specific DIR
structure, as d_name may be implementation specific (e.g. a caching
implementation may just set d_name to point into the cache instead of
copying the entire file name string).
Signed-off-by: Karsten Blees <blees@dcon.de>
We will use them in the upcoming "FSCache" patches (to accelerate
sequential lstat() calls).
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This is one of the few places where Git violates its own deprecation of
the dashed form. It is not necessary, either.
As of 595d59e2b5 (git.c: ignore pager.* when launching builtin as
dashed external, 2017-08-02), Git wants to ignore the pager.* config
setting when expanding aliases. So let's strip out the
check_pager_config(<command-name>) call from the copy-edited code.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
It is a real old anachronism from the Cogito days to have a
.git/branches/ directory. And to have tests that ensure that Cogito
users can migrate away from using that directory.
But so be it, let's continue testing it.
Let's make sure, however, that git init does not need to create that
directory.
This bug was noticed when testing with templates that had been
pre-committed, skipping the empty branches/ directory of course because
Git does not track empty directories.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When compiling with Visual Studio, the projects' names are identical to
the executables modulo the extensions. Read: there will exist both a
directory called `git` as well as an executable called `git.exe` in the
end. Which means that the bin-wrappers *need* to target the `.exe` files
lest they try to execute directories.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Add the Microsoft .manifest pattern, and do not anchor the 'Debug'
and 'Release' entries at the top-level directory, to allow for
multiple projects (one per target).
Signed-off-by: Philip Oakley <philipoakley@iee.org>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The problem with not having, say, git-receive-pack.exe after a full
build is that the test suite will then happily use the *installed*
git-receive-pack.exe because it finds nothing else.
Absolutely not what we want. We want to have confidence that our test
covers the MSVC-built Git executables, and not some random stuff.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The entire idea of generating the VS solution makes only sense if we
generate it via Continuous Integration; otherwise potential users would
still have to download the entire Git for Windows SDK.
So let's just add a target in the Makefile that can be used to generate
said solution; The generated files will then be committed so that they
can be pushed to a branch ready to check out by Visual Studio users.
To make things even more useful, we also generate and commit other files
that are required to run the test suite, such as templates and
bin-wrappers: with this, developers can run the test suite in a regular
Git Bash (that is part of a regular Git for Windows installation) after
building the solution in Visual Studio.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Based on the previous patch series to be able to compile Git using
Visual C++ from the command-line, this patch offers to generate project
definitions for Visual Studio, so that Git can be developed in a modern
IDE.
Based on the generator for Visual Studio versions <= 2008 (which used
.sln/.vcproj files), this patch copy-edits the generator of the .vcproj
files to a new generator that produces .vcxproj files ready for Visual
Studio 2010 and later (or MSBuild).
As the vcpkg system (which is used to build Git's dependencies) cannot
run in parallel (it does not lock, wreaking havoc with files being
accessed and written at the same time, letting the vcpkg processes
stumble over each others' toes), we make libgit the root of the project
dependency tree and initialize the vcpkg system in this project's
PreBuildEvent.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
One time too many did this developer call the `generate` script passing
a `--make-out=<PATH>` option that was happily ignored (because there
should be a space, not an equal sign, between `--make-out` and the
path).
And one time too many, this script not only ignored it but did not even
complain. Let's fix that.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
With the recent changes to allow building with MSVC=1, we now pass the
/OPT:REF option to the compiler. This confuses the parser that wants to
turn the output of a dry run into project definitions for QMake and Visual
Studio:
Unhandled link option @ line 213: /OPT:REF at [...]
Let's just extend the code that passes through options that start with a
dash, so that it passes through options that start with a slash, too.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Git's test suite shows tons of breakages unless Git is compiled
*without* NO_ICONV. That means, in turn, that we need to generate
build definitions *with* libiconv, which in turn implies that we
have to handle the -liconv option properly.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Upon seeing the '-lcurl' option, point to the libcurl.lib.
While there, fix the elsif indentation.
Signed-off-by: Philip Oakley <philipoakley@iee.org>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Add an option for capturing the output of the make dry-run used in
determining the msvc-build structure for easy debugging.
You can use the output of `--make-out <path>` in subsequent runs via the
`--in <path>` option.
Signed-off-by: Philip Oakley <philipoakley@iee.org>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Rather than swallowing the errors, it is better to have them in a file.
To make it obvious what this is about, use the file name
'msvc-build-makedryerrors.txt'.
Further, if the output is empty, simply delete that file. As we target
Git for Windows' SDK (which, unlike its predecessor msysGit, offers Perl
versions newer than 5.8), we can use the quite readable syntax `if -f -z
$ErrsFile` (available in Perl >=5.10).
Note that the file will contain the new values of the GIT_VERSION and
GITGUI_VERSION if they were generated by the make file. They are omitted
if the release is tagged and indentically defined in their respective
GIT_VERSION_GEN file DEF_VER variables.
Signed-off-by: Philip Oakley <philipoakley@iee.org>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Git's build contains steps to handle internationalization. This caused
hiccups in the parser used to generate QMake/Visual Studio project files.
As those steps are irrelevant in this context, let's just ignore them.
Signed-off-by: Philip Oakley <philipoakley@iee.org>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The engine.pl script expects file names not to contain spaces. However,
paths with spaces are quite prevalent on Windows. Use shellwords() rather
than split() to parse them correctly.
Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Philip Oakley <philipoakley@iee.org>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
For regular debugging, it is pretty helpful when a debug assertion in a
running application triggers a window that offers to start the debugger.
However, when running the test suite, it is not so helpful, in
particular when the debug assertions are then suppressed anyway because
we disable the invalid parameter checking (via invalidcontinue.obj, see
the comment in config.mak.uname about that object for more information).
So let's simply disable that window in Debug Mode (it is already
disabled in Release Mode).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The error message talked about a "lib option", but it clearly referred
to a link option.
Signed-off-by: Philip Oakley <philipoakley@iee.org>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
With this patch, Git can be built using the Microsoft toolchain, via:
make MSVC=1 [DEBUG=1]
Third party libraries are built from source using the open source
"vcpkg" tool set. See https://github.com/Microsoft/vcpkg
On a first build, the vcpkg tools and the third party libraries are
automatically downloaded and built. DLLs for the third party libraries
are copied to the top-level (and t/helper) directory to facilitate
debugging. See compat/vcbuild/README.
A series of .bat files are invoked by the Makefile to find the location
of the installed version of Visual Studio and the associated compiler
tools (essentially replicating the environment setup performed by a
"Developer Command Prompt"). This should find the most recent VS2015 or
VS2017 installation. Output from these scripts are used by the Makefile
to define compiler and linker pathnames and -I and -L arguments.
The build produces .pdb files for both debug and release builds.
Note: This commit was squashed from an organic series of commits
developed between 2016 and 2018 in Git for Windows' `master` branch.
This combined commit eliminates the obsolete commits related to fetching
NuGet packages for third party libraries. It is difficult to use NuGet
packages for C/C++ sources because they may be built by earlier versions
of the MSVC compiler and have CRT version and linking issues.
Additionally, the C/C++ NuGet packages that were using tended to not be
updated concurrently with the sources. And in the case of cURL and
OpenSSL, this could expose us to security issues.
Helped-by: Yue Lin Ho <b8732003@student.nsysu.edu.tw>
Helped-by: Philip Oakley <philipoakley@iee.org>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The Generators/ directory can contain spurious files such as editors'
backup files. Even worse, there could be .swp files which are not even
valid Perl scripts.
Let's just ignore anything but .pm files in said directory.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This special-cases various signals that are not supported on Windows,
such as SIGPIPE. These cause the UCRT to throw asserts (at least in
debug mode).
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Since 4b623d8 (MSVC: link in invalidcontinue.obj for better POSIX
compatibility, 2014-03-29), invalidcontinue.obj is linked in the MSVC
build, but it was not parsed correctly by the buildsystem. Ignore it, as
it is known to Visual Studio and will be handled elsewhere.
Also only substitute filenames ending with .o when generating the
source .c filename, otherwise we would start to expect .cbj files to
generate .obj files (which are not generated by our build)...
In the future there may be source files that produce .obj files
so keep the two issues (.obj files with & without source files)
separate.
Signed-off-by: Philip Oakley <philipoakley@iee.org>
Signed-off-by: Duncan Smart <duncan.smart@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Visual Studio takes the first listed application/library as the default
startup project [1].
Detect the 'git' project and place it the head of the apps list, rather
than the tail.
Export the apps list before libs list for both the projects and global
structures of the .sln file.
[1] http://stackoverflow.com/questions/1238553/
vs2008-where-is-the-startup-project-setting-stored-for-a-solution
"In the solution file, there are a list of pseudo-XML "Project"
entries. It turns out that whatever is the first one ends up as
the Startup Project, unless it’s overridden in the suo file. Argh.
I just rearranged the order in the file and it’s good."
"just moving the pseudo-xml isn't enough. You also have to move the
group of entries in the "GlobalSection(ProjectConfigurationPlatforms)
= postSolution" group that has the GUID of the project you moved to
the top. So there are two places to move lines."
Signed-off-by: Philip Oakley <philipoakley@iee.org>
We ran out GUIDs. Again. But there is no need to: we can generate them
semi-randomly from the target file name of the project.
Note: the Vcproj generator is probably only interesting for historical
reasons; nevertheless, the upcoming Vcxproj generator (to support modern
Visual Studio versions) is based on the Vcproj generator and it is
better to fix this here first.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This constant is not defined in MSVC's headers.
In UCRT's fcntl.h, _O_RDONLY, _O_WRONLY and _O_RDWR are defined as 0, 1
and 2, respectively. Yes, that means that UCRT breaks with the tradition
that O_RDWR == O_RDONLY | O_WRONLY.
It is a perfectly legal way to define those constants, though, therefore
we need to take care of defining O_ACCMODE accordingly.
This is particularly important in order to keep our "open() can set
errno to EISDIR" emulation working: it tests that (flags & O_ACCMODE) is
not identical to O_RDONLY before going on to test specifically whether
the file for which open() reported EACCES is, in fact, a directory.
Signed-off-by: Philip Oakley <philipoakley@iee.org>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
