Use OpenSSL's SHA-1 routines rather than builtin block-sha1 routines.
This improves performance on SHA1 operations on Intel processors.
OpenSSL 1.0.2 has made considerable performance improvements and
support the Intel hardware acceleration features. See:
https://software.intel.com/en-us/articles/improving-openssl-performancehttps://software.intel.com/en-us/articles/intel-sha-extensions
To test this I added/staged a single file in a gigantic
repository having a 450MB index file. The code in read-cache.c
verifies the header SHA as it reads the index and computes a new
header SHA as it writes out the new index. Therefore, in this test
the SHA code must process 900MB of data. Testing was done on an
Intel I7-4770 CPU @ 3.40GHz (Intel64, Family 6, Model 60) CPU.
The block-sha1 version averaged 5.27 seconds.
The OpenSSL version averaged 4.50 seconds.
================================================================
$ echo xxx >> project.mk
$ time /e/blk_sha/bin/git.exe add project.mk
real 0m5.207s
user 0m0.000s
sys 0m0.250s
$ echo xxx >> project.mk
$ time /e/blk_sha/bin/git.exe add project.mk
real 0m5.362s
user 0m0.015s
sys 0m0.234s
$ echo xxx >> project.mk
$ time /e/blk_sha/bin/git.exe add project.mk
real 0m5.300s
user 0m0.016s
sys 0m0.250s
$ echo xxx >> project.mk
$ time /e/blk_sha/bin/git.exe add project.mk
real 0m5.216s
user 0m0.000s
sys 0m0.250s
================================================================
$ echo xxx >> project.mk
$ time /e/openssl/bin/git.exe add project.mk
real 0m4.431s
user 0m0.000s
sys 0m0.250s
$ echo xxx >> project.mk
$ time /e/openssl/bin/git.exe add project.mk
real 0m4.478s
user 0m0.000s
sys 0m0.265s
$ echo xxx >> project.mk
$ time /e/openssl/bin/git.exe add project.mk
real 0m4.690s
user 0m0.000s
sys 0m0.250s
$ echo xxx >> project.mk
$ time /e/openssl/bin/git.exe add project.mk
real 0m4.420s
user 0m0.000s
sys 0m0.234s
================================================================
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
With this topic branch, the PERL5LIB variable is unset to avoid external
settings from interfering with Git's own Perl interpreter.
This branch also cleans up some of our Windows-only config setting code
(and this will need to be rearranged in the next merging rebase so that
the cleanup comes first, and fscache and longPaths support build on
top).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The "giteveryday" document has a callout list that contains a code
block. This is not a problem for AsciiDoc, but AsciiDoctor sadly was
explicitly designed *not* to render this correctly [*1*]. The symptom is
an unhelpful
line 322: callout list item index: expected 1 got 12
line 325: no callouts refer to list item 1
line 325: callout list item index: expected 2 got 13
line 327: no callouts refer to list item 2
In Git for Windows, we rely on the speed improvement of AsciiDoctor (on
this developer's machine, `make -j15 html` takes roughly 30 seconds with
AsciiDoctor, 70 seconds with AsciiDoc), therefore we need a way to
render this correctly.
The easiest way out is to simplify the callout list, as suggested by
AsciiDoctor's author, even while one may very well disagree with him
that a code block hath no place in a callout list.
*1*: https://github.com/asciidoctor/asciidoctor/issues/1478
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The `user-manual.txt` is designed as a `book` but the `Makefile` wants
to build it as an `article`. This seems to be a problem when building
the documentation with `asciidoctor`. Furthermore the parts *Git
Glossary* and *Appendix B* had no subsections which is not allowed when
building with `asciidoctor`. So lets add a *dummy* section.
Signed-off-by: nalla <nalla@hamal.uberspace.de>
Windows paths are typically limited to MAX_PATH = 260 characters, even
though the underlying NTFS file system supports paths up to 32,767 chars.
This limitation is also evident in Windows Explorer, cmd.exe and many
other applications (including IDEs).
Particularly annoying is that most Windows APIs return bogus error codes
if a relative path only barely exceeds MAX_PATH in conjunction with the
current directory, e.g. ERROR_PATH_NOT_FOUND / ENOENT instead of the
infinitely more helpful ERROR_FILENAME_EXCED_RANGE / ENAMETOOLONG.
Many Windows wide char APIs support longer than MAX_PATH paths through the
file namespace prefix ('\\?\' or '\\?\UNC\') followed by an absolute path.
Notable exceptions include functions dealing with executables and the
current directory (CreateProcess, LoadLibrary, Get/SetCurrentDirectory) as
well as the entire shell API (ShellExecute, SHGetSpecialFolderPath...).
Introduce a handle_long_path function to check the length of a specified
path properly (and fail with ENAMETOOLONG), and to optionally expand long
paths using the '\\?\' file namespace prefix. Short paths will not be
modified, so we don't need to worry about device names (NUL, CON, AUX).
Contrary to MSDN docs, the GetFullPathNameW function doesn't seem to be
limited to MAX_PATH (at least not on Win7), so we can use it to do the
heavy lifting of the conversion (translate '/' to '\', eliminate '.' and
'..', and make an absolute path).
Add long path error checking to xutftowcs_path for APIs with hard MAX_PATH
limit.
Add a new MAX_LONG_PATH constant and xutftowcs_long_path function for APIs
that support long paths.
While improved error checking is always active, long paths support must be
explicitly enabled via 'core.longpaths' option. This is to prevent end
users to shoot themselves in the foot by checking out files that Windows
Explorer, cmd/bash or their favorite IDE cannot handle.
Test suite:
Test the case is when the full pathname length of a dir is close
to 260 (MAX_PATH).
Bug report and an original reproducer by Andrey Rogozhnikov:
https://github.com/msysgit/git/pull/122#issuecomment-43604199
Note that the test cannot rely on the presence of short names, as they
are not enabled by default except on the system drive.
[jes: adjusted test number to avoid conflicts, reinstated && chain,
adjusted test to work without short names]
Thanks-to: Martin W. Kirst <maki@bitkings.de>
Thanks-to: Doug Kelly <dougk.ff7@gmail.com>
Signed-off-by: Karsten Blees <blees@dcon.de>
Original-test-by: Andrey Rogozhnikov <rogozhnikov.andrey@gmail.com>
Signed-off-by: Stepan Kasal <kasal@ucw.cz>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Windows paths are typically limited to MAX_PATH = 260 characters, even
though the underlying NTFS file system supports paths up to 32,767 chars.
This limitation is also evident in Windows Explorer, cmd.exe and many
other applications (including IDEs).
Particularly annoying is that most Windows APIs return bogus error codes
if a relative path only barely exceeds MAX_PATH in conjunction with the
current directory, e.g. ERROR_PATH_NOT_FOUND / ENOENT instead of the
infinitely more helpful ERROR_FILENAME_EXCED_RANGE / ENAMETOOLONG.
Many Windows wide char APIs support longer than MAX_PATH paths through the
file namespace prefix ('\\?\' or '\\?\UNC\') followed by an absolute path.
Notable exceptions include functions dealing with executables and the
current directory (CreateProcess, LoadLibrary, Get/SetCurrentDirectory) as
well as the entire shell API (ShellExecute, SHGetSpecialFolderPath...).
Introduce a handle_long_path function to check the length of a specified
path properly (and fail with ENAMETOOLONG), and to optionally expand long
paths using the '\\?\' file namespace prefix. Short paths will not be
modified, so we don't need to worry about device names (NUL, CON, AUX).
Contrary to MSDN docs, the GetFullPathNameW function doesn't seem to be
limited to MAX_PATH (at least not on Win7), so we can use it to do the
heavy lifting of the conversion (translate '/' to '\', eliminate '.' and
'..', and make an absolute path).
Add long path error checking to xutftowcs_path for APIs with hard MAX_PATH
limit.
Add a new MAX_LONG_PATH constant and xutftowcs_long_path function for APIs
that support long paths.
While improved error checking is always active, long paths support must be
explicitly enabled via 'core.longpaths' option. This is to prevent end
users to shoot themselves in the foot by checking out files that Windows
Explorer, cmd/bash or their favorite IDE cannot handle.
Test suite:
Test the case is when the full pathname length of a dir is close
to 260 (MAX_PATH).
Bug report and an original reproducer by Andrey Rogozhnikov:
https://github.com/msysgit/git/pull/122#issuecomment-43604199
[jes: adjusted test number to avoid conflicts]
Thanks-to: Martin W. Kirst <maki@bitkings.de>
Thanks-to: Doug Kelly <dougk.ff7@gmail.com>
Signed-off-by: Karsten Blees <blees@dcon.de>
Original-test-by: Andrey Rogozhnikov <rogozhnikov.andrey@gmail.com>
Signed-off-by: Stepan Kasal <kasal@ucw.cz>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
[jes: adusted test number to avoid conflicts, fixed non-portable use of
the 'export' statement]
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
If multiple threads access a directory that is not yet in the cache, the
directory will be loaded by each thread. Only one of the results is added
to the cache, all others are leaked. This wastes performance and memory.
On cache miss, add a future object to the cache to indicate that the
directory is currently being loaded. Subsequent threads register themselves
with the future object and wait. When the first thread has loaded the
directory, it replaces the future object with the result and notifies
waiting threads.
Signed-off-by: Karsten Blees <blees@dcon.de>
Checking the work tree status is quite slow on Windows, due to slow lstat
emulation (git calls lstat once for each file in the index). Windows
operating system APIs seem to be much better at scanning the status
of entire directories than checking single files.
Add an lstat implementation that uses a cache for lstat data. Cache misses
read the entire parent directory and add it to the cache. Subsequent lstat
calls for the same directory are served directly from the cache.
Also implement opendir / readdir / closedir so that they create and use
directory listings in the cache.
The cache doesn't track file system changes and doesn't plug into any
modifying file APIs, so it has to be explicitly enabled for git functions
that don't modify the working copy.
Note: in an earlier version of this patch, the cache was always active and
tracked file system changes via ReadDirectoryChangesW. However, this was
much more complex and had negative impact on the performance of modifying
git commands such as 'git checkout'.
Signed-off-by: Karsten Blees <blees@dcon.de>
Add a macro to mark code sections that only read from the file system,
along with a config option and documentation.
This facilitates implementation of relatively simple file system level
caches without the need to synchronize with the file system.
Enable read-only sections for 'git status' and preload_index.
Signed-off-by: Karsten Blees <blees@dcon.de>
Emulating the POSIX lstat API on Windows via GetFileAttributes[Ex] is quite
slow. Windows operating system APIs seem to be much better at scanning the
status of entire directories than checking single files. A caching
implementation may improve performance by bulk-reading entire directories
or reusing data obtained via opendir / readdir.
Make the lstat implementation pluggable so that it can be switched at
runtime, e.g. based on a config option.
Signed-off-by: Karsten Blees <blees@dcon.de>
Emulating the POSIX dirent API on Windows via FindFirstFile/FindNextFile is
pretty staightforward, however, most of the information provided in the
WIN32_FIND_DATA structure is thrown away in the process. A more
sophisticated implementation may cache this data, e.g. for later reuse in
calls to lstat.
Make the dirent implementation pluggable so that it can be switched at
runtime, e.g. based on a config option.
Define a base DIR structure with pointers to readdir/closedir that match
the opendir implementation (i.e. similar to vtable pointers in OOP).
Define readdir/closedir so that they call the function pointers in the DIR
structure. This allows to choose the opendir implementation on a
call-by-call basis.
Move the fixed sized dirent.d_name buffer to the dirent-specific DIR
structure, as d_name may be implementation specific (e.g. a caching
implementation may just set d_name to point into the cache instead of
copying the entire file name string).
Signed-off-by: Karsten Blees <blees@dcon.de>
Git for Windows ships with its own Perl interpreter, and insists on
using it, so it will most likely wreak havoc if PERL5LIB is set before
launching Git.
Let's just unset that environment variables when spawning processes.
To make this feature extensible (and overrideable), there is a new
config setting `core.unsetenvvars` that allows specifying a
comma-separated list of names to unset before spawning processes.
Reported by Gabriel Fuhrmann.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
In the Git for Windows project, we have ample precendent for config
settings that apply to Windows, and to Windows only.
Let's formalize this concept by introducing a platform_core_config()
function that can be #define'd in a platform-specific manner.
This will allow us to contain platform-specific code better, as the
corresponding variables no longer need to be exported so that they can
be defined in environment.c and be set in config.c
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This matches up with the targets git-%, git-http-fetch, git-http-push
and git-remote-testsvn. It must be done this way in Cygwin else lcrypto
cannot find lgdi32 and lws2_32.
Signed-off-by: Steven Penny <svnpenn@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This commit starts the rebase of 5c76d3ceb2 to b32fe956d0
While at it, a couple of patches were moved around, Philip Oakley's
changes to the issue reporting template were squashed into Brendan
Forster's original patch, and the GIT_SSH_COMMAND patch series was updated
to the latest (v4).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Code cleanup.
* js/exec-path-coverity-workaround:
git_exec_path: do not return the result of getenv()
git_exec_path: avoid Coverity warning about unfree()d result
Tighten a test to avoid mistaking an extended ERE regexp engine as
a PRE regexp engine.
* jk/grep-e-could-be-extended-beyond-posix:
t7810: avoid assumption about invalid regex syntax
"git <cmd> @{push}" on a detached HEAD used to segfault; it has
been corrected to error out with a message.
* km/branch-get-push-while-detached:
branch_get_push: do not segfault when HEAD is detached
"git rebase -i" with a recent update started showing an incorrect
count when squashing more than 10 commits.
* jk/rebase-i-squash-count-fix:
rebase--interactive: count squash commits above 10 correctly
"git blame --porcelain" misidentified the "previous" <commit, path>
pair (aka "source") when contents came from two or more files.
* jk/blame-fixes:
blame: output porcelain "previous" header for each file
blame: handle --no-abbrev
blame: fix alignment with --abbrev=40
"git archive" did not read the standard configuration files, and
failed to notice a file that is marked as binary via the userdiff
driver configuration.
* jk/archive-zip-userdiff-config:
archive-zip: load userdiff config
It is natural that "git gc --auto" may not attempt to pack
everything into a single pack, and there is no point in warning
when the user has configured the system to use the pack bitmap,
leading to disabling further "gc".
* dt/disable-bitmap-in-auto-gc:
repack: die on incremental + write-bitmap-index
auto gc: don't write bitmaps for incremental repacks
Leakage of lockfiles in the config subsystem has been fixed.
* nd/config-misc-fixes:
config.c: handle lock file in error case in git_config_rename_...
config.c: rename label unlock_and_out
config.c: handle error case for fstat() calls
Recent update to the default abbreviation length that auto-scales
lacked documentation update, which has been corrected.
* jc/abbrev-autoscale-config:
config.abbrev: document the new default that auto-scales
"git fast-import" sometimes mishandled while rebalancing notes
tree, which has been fixed.
* mh/fast-import-notes-fix-new:
fast-import: properly fanout notes when tree is imported
Compression setting for producing packfiles were spread across
three codepaths, one of which did not honor any configuration.
Unify these so that all of them honor core.compression and
pack.compression variables the same way.
* jc/compression-config:
compression: unify pack.compression configuration parsing
Update the procedure to generate "tags" for developer support.
* jk/make-tags-find-sources-tweak:
Makefile: exclude contrib from FIND_SOURCE_FILES
Makefile: match shell scripts in FIND_SOURCE_FILES
Makefile: exclude test cruft from FIND_SOURCE_FILES
Makefile: reformat FIND_SOURCE_FILES
Some platforms no longer understand "latin-1" that is still seen in
the wild in e-mail headers; replace them with "iso-8859-1" that is
more widely known when conversion fails from/to it.
* jc/latin-1:
utf8: accept "latin-1" as ISO-8859-1
utf8: refactor code to decide fallback encoding
The _TODO_ entries in the Issues_template are easily misunderstood
as residual developer actions, rather than a place holder for user
data entry.
Replace the _TODO_ with positively directed actions for the user
("you/your") to perform. Ease off on the tone toward the end.
Signed-off-by: Philip Oakley <philipoakley@iee.org>
