Changes opendir/readdir to use Windows Unicode APIs and convert between
UTF-8/UTF-16.
Removes parameter checks that are already covered by xutftowcs_path. This
changes detection of ENAMETOOLONG from MAX_PATH - 2 to MAX_PATH (matching
is_dir_empty in mingw.c). If name + "/*" or the resulting absolute path is
too long, FindFirstFile fails and errno is set through err_win_to_posix.
Increases the size of dirent.d_name to accommodate the full
WIN32_FIND_DATA.cFileName converted to UTF-8 (UTF-16 to UTF-8 conversion
may grow by factor three in the worst case).
Signed-off-by: Karsten Blees <blees@dcon.de>
Replaces Windows "ANSI" APIs dealing with file- or path names with their
Unicode equivalent, adding UTF-8/UTF-16LE conversion as necessary.
The dirent API (opendir/readdir/closedir) is updated in a separate commit.
Adds trivial wrappers for access, chmod and chdir.
Adds wrapper for mktemp (needed for both mkstemp and mkdtemp).
The simplest way to convert a repository with legacy-encoded (e.g. Cp1252)
file names to UTF-8 ist to checkout with an old msysgit version and
"git add --all & git commit" with the new version.
Signed-off-by: Karsten Blees <blees@dcon.de>
Add Unicode conversion functions to convert between Windows native UTF-16LE
encoding to UTF-8 and back.
To support repositories with legacy-encoded file names, the UTF-8 to UTF-16
conversion function tries to create valid, unique file names even for
invalid UTF-8 byte sequences, so that these repositories can be checked out
without error.
The current implementation leaves invalid UTF-8 bytes in range 0xa0 - 0xff
as is (producing printable Unicode chars \u00a0 - \u00ff, equivalent to
ISO-8859-1), and converts 0x80 - 0x9f to hex-code (\u0080 - \u009f are
control chars).
The Windows MultiByteToWideChar API was not used as it either drops invalid
UTF-8 sequences (on Win2k/XP; producing non-unique or even empty file
names) or converts them to the replacement char \ufffd (Vista/7; causing
ERROR_INVALID_NAME in subsequent calls to file system APIs).
Signed-off-by: Karsten Blees <blees@dcon.de>
Winansi.c has many static variables that are accessed and modified from
the [v][f]printf / fputs functions overridden in the file. This may cause
multi threaded git commands that print to the console to produce corrupted
output or even crash.
Additionally, winansi.c doesn't override all functions that can be used to
print to the console (e.g. fwrite, write, fputc are missing), so that ANSI
escapes don't work properly for some git commands (e.g. git-grep).
Instead of doing ANSI emulation in just a few wrapped functions on top of
the IO API, let's plug into the IO system and take advantage of the thread
safety inherent to the IO system.
Redirect stdout and stderr to a pipe if they point to the console. A
background thread reads from the pipe, handles ANSI escape sequences and
UTF-8 to UTF-16 conversion, then writes to the console.
The pipe-based stdout and stderr replacements must be set to unbuffered, as
MSVCRT doesn't support line buffering and fully buffered streams are
inappropriate for console output.
Due to the byte-oriented pipe, ANSI escape sequences and multi-byte UTF-8
sequences can no longer be expected to arrive in one piece. Replace the
string-based ansi_emulate() with a simple stateful parser (this also fixes
colored diff hunk headers, which were broken as of commit 2efcc977).
Override isatty to return true for the pipes redirecting to the console.
Exec/spawn obtain the original console handle to pass to the next process
via winansi_get_osfhandle().
All other overrides are gone, the default stdio implementations work as
expected with the piped stdout/stderr descriptors.
Global variables are either initialized on startup (single threaded) or
exclusively modified by the background thread. Threads communicate through
the pipe, no further synchronization is necessary.
The background thread is terminated by disonnecting the pipe after flushing
the stdio and pipe buffers. This doesn't work for anonymous pipes (created
via CreatePipe), as DisconnectNamedPipe only works on the read end, which
discards remaining data. Thus we have to setup the pipe manually, with the
write end beeing the server (opened with CreateNamedPipe) and the read end
the client (opened with CreateFile).
Limitations: doesn't track reopened or duped file descriptors, i.e.:
- fdopen(1/2) returns fully buffered streams
- dup(1/2), dup2(1/2) returns normal pipe descriptors (i.e. isatty() =
false, winansi_get_osfhandle won't return the original console handle)
Currently, only the git-format-patch command uses xfdopen(xdup(1)) (see
"realstdout" in builtin/log.c), but works well with these limitations.
Many thanks to Atsushi Nakagawa <atnak@chejz.com> for suggesting and
reviewing the thread-exit-mechanism.
Signed-off-by: Karsten Blees <blees@dcon.de>
Some constants (such as LF_FACESIZE) are undefined with -DNOGDI (set in the
Makefile), and CONSOLE_FONT_INFOEX is available in MSVC, but not in MinGW.
Cast FARPROC to PGETCURRENTCONSOLEFONTEX to suppress MSVC compiler warning.
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Git requires the TERM environment variable to be set for all color*
settings. Simulate the TERM variable if it is not set (default on Windows).
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This reverts commit df599e9612.
As of 5e9637c6 "i18n: add infrastructure for translating Git with gettext",
eval_gettext uses MinGW envsubst.exe instead of git-sh-i18n--envsubst.exe
for variable substitution. This breaks git-submodule.sh messages and tests,
as envsubst.exe doesn't support case-sensitive environment lookup (the same
is true for almost everything on Windows, including MSys and Cygwin tools).
30a615ac "Windows/i18n: rename $path to prevent clashes with $PATH" renames
the conflicting variable in git-submodule.sh, so that it works on Windows
(i.e. with case-insensitive environment, regardless of the toolset).
Revert to the documented behaviour of case-insensitive environment on
Windows.
Signed-off-by: Karsten Blees <blees@dcon.de>
GetCurrentConsoleFontEx in an atexit routine doesn't work because git
closes stdout before exit (which also closes the console handle). Check
the console font when we first encounter a non-ascii character and only
schedule the warning message to be printed at exit (warnings go to stderr,
which is not closed by git).
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Erik Faye-Lund <kusmabite@gmail.com>
MingwRT listens to _CRT_glob to decide if __getmainargs should
perform globbing, with the default being that it should.
Unfortunately, __getmainargs globbing is sub-par; for instance
patterns like "*.c" will only match c-sources in the current
directory.
Disable __getmainargs' command line wildcard expansion, so these
patterns will be left untouched, and handled by Git's superior
built-in globbing instead.
MSVC defaults to no globbing, so we don't need to do anything
in that case.
This fixes t5505 and t7810.
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Erik Faye-Lund <kusmabite@gmail.com>
The code in the MinGW main macro is getting more and more complex, move to
a separate initialization function for readabiliy and extensibility.
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Erik Faye-Lund <kusmabite@gmail.com>
Improve the dirent implementation by removing the relics that were once
necessary to plug into the now unused MinGW runtime, in preparation for
Unicode file name support.
Move FindFirstFile to opendir, and FindClose to closedir, with the
following implications:
- DIR.dd_name is no longer needed
- chdir(one); opendir(relative); chdir(two); readdir() works as expected
(i.e. lists one/relative instead of two/relative)
- DIR.dd_handle is a valid handle for the entire lifetime of the DIR struct
- thus, all checks for dd_handle == INVALID_HANDLE_VALUE and dd_handle == 0
have been removed
- the special case that the directory has been fully read (which was
previously explicitly tracked with dd_handle == INVALID_HANDLE_VALUE &&
dd_stat != 0) is now handled implicitly by the FindNextFile error
handling code (if a client continues to call readdir after receiving
NULL, FindNextFile will continue to fail with ERROR_NO_MORE_FILES, to
the same effect)
- extracting dirent data from WIN32_FIND_DATA is needed in two places, so
moved to its own method
- GetFileAttributes is no longer needed. The same information can be
obtained from the FindFirstFile error code, which is ERROR_DIRECTORY if
the name is NOT a directory (-> ENOTDIR), otherwise we can use
err_win_to_posix (e.g. ERROR_PATH_NOT_FOUND -> ENOENT). The
ERROR_DIRECTORY case could be fixed in err_win_to_posix, but this
probably breaks other functionality.
Removes the ERROR_NO_MORE_FILES check after FindFirstFile (this was
fortunately a NOOP (searching for '*' always finds '.' and '..'),
otherwise the subsequent code would have copied data from an uninitialized
buffer).
Changes malloc to git support function xmalloc, so opendir will die() if
out of memory, rather than failing with ENOMEM and letting git work on
incomplete directory listings (error handling in dir.c is quite sparse).
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Erik Faye-Lund <kusmabite@gmail.com>
Git-compat-util.h is two dirs up, and already includes <dirent.h> (which
is the same as "dirent.h" due to -Icompat/win32 in the Makefile).
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Erik Faye-Lund <kusmabite@gmail.com>
FILENAME_MAX and MAX_PATH are both 260 on Windows, however, MAX_PATH is
used throughout the other Win32 code in Git, and also defines the length
of file name buffers in the Win32 API (e.g. WIN32_FIND_DATA.cFileName,
from which we're copying the dirent data).
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Erik Faye-Lund <kusmabite@gmail.com>
Remove the union around dirent.d_type and the unused dirent.d_reclen member
(which was necessary for compatibility with the MinGW dirent runtime, which
is no longer used).
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Erik Faye-Lund <kusmabite@gmail.com>
There are no proper inodes on Windows, so remove dirent.d_ino and #define
NO_D_INO_IN_DIRENT in the Makefile (this skips e.g. an ineffective qsort in
fsck.c).
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Erik Faye-Lund <kusmabite@gmail.com>
Unicode console output won't display correctly with default settings
because the default console font ("Terminal") only supports the system's
OEM charset. Unfortunately, this is a user specific setting, so it cannot
be easily fixed by e.g. some registry tricks in the setup program.
This change prints a warning on exit if console output contained non-ascii
characters and the console font is supposedly not a TrueType font (which
usually have decent Unicode support).
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
GetStdHandle(STD_OUTPUT_HANDLE) doesn't work for stderr if stdout is
redirected. Use _get_osfhandle of the FILE* instead.
_isatty() is true for all character devices (including parallel and serial
ports). Check return value of GetConsoleScreenBufferInfo instead to
reliably detect console handles (also don't initialize internal state from
an uninitialized CONSOLE_SCREEN_BUFFER_INFO structure if the function
fails).
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
WriteConsoleW seems to be the only way to reliably print unicode to the
console (without weird code page conversions).
Also redirects vfprintf to the winansi.c version.
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The third maintenance release for Git 1.9; contains all the fixes
that are scheduled to appear in Git 2.0 since 1.9.2.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Since commit 0c499ea60f the send-pack builtin uses the side-band-64k
capability if advertised by the server.
Unfortunately this breaks pushing over the dump git protocol if used
over a network connection.
The detailed reasons for this breakage are (by courtesy of Jeff Preshing,
quoted from ttps://groups.google.com/d/msg/msysgit/at8D7J-h7mw/eaLujILGUWoJ):
----------------------------------------------------------------------------
MinGW wraps Windows sockets in CRT file descriptors in order to mimic the
functionality of POSIX sockets. This causes msvcrt.dll to treat sockets as
Installable File System (IFS) handles, calling ReadFile, WriteFile,
DuplicateHandle and CloseHandle on them. This approach works well in simple
cases on recent versions of Windows, but does not support all usage patterns.
In particular, using this approach, any attempt to read & write concurrently
on the same socket (from one or more processes) will deadlock in a scenario
where the read waits for a response from the server which is only invoked after
the write. This is what send_pack currently attempts to do in the use_sideband
codepath.
----------------------------------------------------------------------------
The new config option "sendpack.sideband" allows to override the side-band-64k
capability of the server, and thus makes the dump git protocol work.
Other transportation methods like ssh and http/https still benefit from
the sideband channel, therefore the default value of "sendpack.sideband"
is still true.
Signed-off-by: Thomas Braun <thomas.braun@byte-physics.de>
The git-shell(1) manpage says
EXAMPLE
To disable interactive logins, displaying a greeting
instead:
+
$ chsh -s /usr/bin/git-shell
$ mkdir $HOME/git-shell-commands
[...]
The stray "+" has been there ever since the example was added in
v1.8.3-rc0~210^2 (shell: new no-interactive-login command to print a
custom message, 2013-03-09). The "+" sign between paragraphs is
needed in asciidoc to attach extra paragraphs to a list item but here
it is not needed and ends up rendered as a literal "+". Remove it.
A quick search with "grep -e '<p>+' /usr/share/doc/git/html/*.html"
doesn't find any other instances of this problem.
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
"git p4" dealing with changes in binary files were broken by a
change in 1.9 release.
* cl/p4-use-diff-tree:
git-p4: format-patch to diff-tree change breaks binary patches
The shell prompt script (in contrib/), when using the PROMPT_COMMAND
interface, used an unsafe construct when showing the branch name in
$PS1.
* rh/prompt-pcmode-avoid-eval-on-refname:
git-prompt.sh: don't put unsanitized branch names in $PS1
"git rebase" used a POSIX shell construct FreeBSD /bin/sh does not
work well with.
* km/avoid-non-function-return-in-rebase:
Revert "rebase: fix run_specific_rebase's use of "return" on FreeBSD"
rebase: avoid non-function use of "return" on FreeBSD
Some more Unicode codepoints defined in Unicode 6.3 as having zero
width have been taught to our display column counting logic.
* tb/unicode-6.3-zero-width:
utf8.c: partially update to version 6.3
When applying binary patches a full index is required. format-patch
already handles this, but diff-tree needs '--full-index' argument
to always output full index. When git-p4 runs git-apply to test
the patch, git-apply rejects the patch due to abbreviated blob
object names. This is the error message git-apply emits in this
case:
error: cannot apply binary patch to '<filename>' without full index line
error: <filename>: patch does not apply
Signed-off-by: Tolga Ceylan <tolga.ceylan@gmail.com>
Acked-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
MinGW builds of cURL does not ship with curl-config unless built
with the autoconf based build system, which is not the practice
recommended by the documentation. MsysGit has had issues with
binaries of that sort, so it has switched away from autoconf-based
cURL-builds.
Unfortunately, broke pushing over WebDAV on Windows, because
http-push.c depends on cURL's multi-threaded API, which we could
not determine the presence of any more.
Since troublesome curl-versions are ancient, and not even present
in RedHat 5, let's just assume cURL is capable instead of doing a
non-robust check.
Instead, add a check for curl_multi_init to our configure-script,
for those on ancient system. They probably already need to do the
configure-dance anyway.
Signed-off-by: Erik Faye-Lund <kusmabite@gmail.com>
Both bash and zsh subject the value of PS1 to parameter expansion,
command substitution, and arithmetic expansion. Rather than include
the raw, unescaped branch name in PS1 when running in two- or
three-argument mode, construct PS1 to reference a variable that holds
the branch name. Because the shells do not recursively expand, this
avoids arbitrary code execution by specially-crafted branch names such
as '$(IFS=_;cmd=sudo_rm_-rf_/;$cmd)'.
Signed-off-by: Richard Hansen <rhansen@bbn.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This reverts commit 99855ddf4b.
The workaround 99855ddf introduced to deal with problematic
"return" statements in scripts run by "dot" commands located
inside functions only handles one part of the problem. The
issue has now been addressed by not using "return" statements
in this way in the git-rebase--*.sh scripts.
This workaround is therefore no longer necessary, so clean
up the code by reverting it.
Signed-off-by: Kyle J. McKay <mackyle@gmail.com>
Acked-by: Matthieu Moy <Matthieu.Moy@imag.fr>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Since a1549e10, 15d4bf2e and 01a1e646 (first appearing in v1.8.4)
the git-rebase--*.sh scripts have used a "return" to stop execution
of the dot-sourced file and return to the "dot" command that
dot-sourced it. The /bin/sh utility on FreeBSD however behaves
poorly under some circumstances when such a "return" is executed.
In particular, if the "dot" command is contained within a function,
then when a "return" is executed by the script it runs (that is not
itself inside a function), control will return from the function
that contains the "dot" command skipping any statements that might
follow the dot command inside that function. Commit 99855ddf (first
appearing in v1.8.4.1) addresses this by making the "dot" command
the last line in the function.
Unfortunately the FreeBSD /bin/sh may also execute some statements
in the script run by the "dot" command that appear after the
troublesome "return". The fix in 99855ddf does not address this
problem.
For example, if you have script1.sh with these contents:
run_script2() {
. "$(dirname -- "$0")/script2.sh"
_e=$?
echo only this line should show
[ $_e -eq 5 ] || echo expected status 5 got $_e
return 3
}
run_script2
e=$?
[ $e -eq 3 ] || { echo expected status 3 got $e; exit 1; }
And script2.sh with these contents:
if [ 5 -gt 3 ]; then
return 5
fi
case bad in *)
echo always shows
esac
echo should not get here
! :
When running script1.sh (e.g. '/bin/sh script1.sh' or './script1.sh'
after making it executable), the expected output from a POSIX shell
is simply the single line:
only this line should show
However, when run using FreeBSD's /bin/sh, the following output
appears instead:
should not get here
expected status 3 got 1
Not only did the lines following the "dot" command in the run_script2
function in script1.sh get skipped, but additional lines in script2.sh
following the "return" got executed -- but not all of them (e.g. the
"echo always shows" line did not run).
These issues can be avoided by not using a top-level "return" in
script2.sh. If script2.sh is changed to this:
main() {
if [ 5 -gt 3 ]; then
return 5
fi
case bad in *)
echo always shows
esac
echo should not get here
! :
}
main
Then it behaves the same when using FreeBSD's /bin/sh as when using
other more POSIX compliant /bin/sh implementations.
We fix the git-rebase--*.sh scripts in a similar fashion by moving
the top-level code that contains "return" statements into its own
function and then calling that as the last line in the script.
Signed-off-by: Kyle J. McKay <mackyle@gmail.com>
Acked-by: Matthieu Moy <Matthieu.Moy@imag.fr>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This commit replaces the Git for Windows-only patch by the patch
accepted upstream instead.
It was performed by the following shell commands:
git revert -n 3c87ce5^2 &&
git cherry-pick -n 06bdc23 &&
git commit -s --fixup 3c87ce5
(and amending the commit with this commit message)
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This topic branch addresses out-of-memory errors in particular on
Windows, where the default stack space is not very large.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
