Win32: support long paths

Windows paths are typically limited to MAX_PATH = 260 characters, even
though the underlying NTFS file system supports paths up to 32,767 chars.
This limitation is also evident in Windows Explorer, cmd.exe and many
other applications (including IDEs).

Particularly annoying is that most Windows APIs return bogus error codes
if a relative path only barely exceeds MAX_PATH in conjunction with the
current directory, e.g. ERROR_PATH_NOT_FOUND / ENOENT instead of the
infinitely more helpful ERROR_FILENAME_EXCED_RANGE / ENAMETOOLONG.

Many Windows wide char APIs support longer than MAX_PATH paths through the
file namespace prefix ('\\?\' or '\\?\UNC\') followed by an absolute path.
Notable exceptions include functions dealing with executables and the
current directory (CreateProcess, LoadLibrary, Get/SetCurrentDirectory) as
well as the entire shell API (ShellExecute, SHGetSpecialFolderPath...).

Introduce a handle_long_path function to check the length of a specified
path properly (and fail with ENAMETOOLONG), and to optionally expand long
paths using the '\\?\' file namespace prefix. Short paths will not be
modified, so we don't need to worry about device names (NUL, CON, AUX).

Contrary to MSDN docs, the GetFullPathNameW function doesn't seem to be
limited to MAX_PATH (at least not on Win7), so we can use it to do the
heavy lifting of the conversion (translate '/' to '\', eliminate '.' and
'..', and make an absolute path).

Add long path error checking to xutftowcs_path for APIs with hard MAX_PATH
limit.

Add a new MAX_LONG_PATH constant and xutftowcs_long_path function for APIs
that support long paths.

While improved error checking is always active, long paths support must be
explicitly enabled via 'core.longpaths' option. This is to prevent end
users to shoot themselves in the foot by checking out files that Windows
Explorer, cmd/bash or their favorite IDE cannot handle.

Test suite:
Test the case is when the full pathname length of a dir is close
to 260 (MAX_PATH).
Bug report and an original reproducer by Andrey Rogozhnikov:
https://github.com/msysgit/git/pull/122#issuecomment-43604199

[jes: adjusted test number to avoid conflicts]

Thanks-to: Martin W. Kirst <maki@bitkings.de>
Thanks-to: Doug Kelly <dougk.ff7@gmail.com>
Signed-off-by: Karsten Blees <blees@dcon.de>
Original-test-by: Andrey Rogozhnikov <rogozhnikov.andrey@gmail.com>
Signed-off-by: Stepan Kasal <kasal@ucw.cz>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

t/t2027-checkout-long-paths.sh

@@ -0,0 +1,99 @@
+#!/bin/sh
+
+test_description='checkout long paths on Windows
+
+Ensures that Git for Windows can deal with long paths (>260) enabled via core.longpaths'
+
+. ./test-lib.sh
+
+if test_have_prereq !MINGW
+then
+	skip_all='skipping MINGW specific long paths test'
+	test_done
+fi
+
+test_expect_success setup '
+	p=longpathxx && # -> 10
+	p=$p$p$p$p$p && # -> 50
+	p=$p$p$p$p$p && # -> 250
+
+	path=${p}/longtestfile && # -> 263 (MAX_PATH = 260)
+
+	blob=$(echo foobar | git hash-object -w --stdin) &&
+
+	printf "100644 %s 0\t%s\n" "$blob" "$path" |
+	git update-index --add --index-info &&
+	git commit -m initial -q
+'
+
+test_expect_success 'checkout of long paths without core.longpaths fails' '
+	git config core.longpaths false &&
+	test_must_fail git checkout -f 2>error &&
+	grep -q "Filename too long" error &&
+	test_path_is_missing longpa~1/longtestfile
+'
+
+test_expect_success 'checkout of long paths with core.longpaths works' '
+	git config core.longpaths true &&
+	git checkout -f &&
+	test_path_is_file longpa~1/longtestfile
+'
+
+test_expect_success 'update of long paths' '
+	echo frotz >> longpa~1/longtestfile &&
+	echo $path > expect &&
+	git ls-files -m > actual &&
+	test_cmp expect actual &&
+	git add $path &&
+	git commit -m second &&
+	git grep "frotz" HEAD -- $path
+'
+
+test_expect_success cleanup '
+	# bash cannot delete the trash dir if it contains a long path
+	# lets help cleaning up (unless in debug mode)
+	test ! -z "$debug" || rm -rf longpa~1
+'
+
+# check that the template used in the test won't be too long:
+abspath="$(pwd -W)"/testdir
+test ${#abspath} -gt 230 ||
+test_set_prereq SHORTABSPATH
+
+test_expect_success SHORTABSPATH 'clean up path close to MAX_PATH' '
+	p=/123456789abcdef/123456789abcdef/123456789abcdef/123456789abc/ef &&
+	p=y$p$p$p$p &&
+	subdir="x$(echo "$p" | tail -c $((253 - ${#abspath})) - )" &&
+	# Now, $abspath/$subdir has exactly 254 characters, and is inside CWD
+	p2="$abspath/$subdir" &&
+	test 254 = ${#p2} &&
+
+	# Be careful to overcome path limitations of the MSys tools and split
+	# the $subdir into two parts. ($subdir2 has to contain 16 chars and a
+	# slash somewhere following; that is why we asked for abspath <= 230 and
+	# why we placed a slash near the end of the $subdir template.)
+	subdir2=${subdir#????????????????*/} &&
+	subdir1=testdir/${subdir%/$subdir2} &&
+	mkdir -p "$subdir1" &&
+	i=0 &&
+	# The most important case is when absolute path is 258 characters long,
+	# and that will be when i == 4.
+	while test $i -le 7
+	do
+		mkdir -p $subdir2 &&
+		touch $subdir2/one-file &&
+		mv ${subdir2%%/*} "$subdir1/" &&
+		subdir2=z${subdir2} &&
+		i=$(($i+1)) ||
+		exit 1
+	done &&
+
+	# now check that git is able to clear the tree:
+	(cd testdir &&
+	 git init &&
+	 git config core.longpaths yes &&
+	 git clean -fdx) &&
+	test ! -d "$subdir1"
+'
+
+test_done

t/t7414-submodule-long-path.sh

@@ -11,15 +11,20 @@ This test verifies that "git submodule" initialization, update and clones work,
 TEST_NO_CREATE_REPO=1
 . ./test-lib.sh
 
-longpath=""
-for (( i=0; i<4; i++ )); do
-	longpath="0123456789abcdefghijklmnopqrstuvwxyz$longpath"
-done
-# Pick a substring maximum of 90 characters
-# This should be good, since we'll add on a lot for temp directories
-longpath=${longpath:0:90}; export longpath
+# cloning a submodule calls is_git_directory("$path/../.git/modules/$path"),
+# which effectively limits the maximum length to PATH_MAX / 2 minus some
+# overhead; start with 3 * 36 = 108 chars (test 2 fails if >= 110)
+longpath36=0123456789abcdefghijklmnopqrstuvwxyz
+longpath180=$longpath36$longpath36$longpath36$longpath36$longpath36
 
-test_expect_failure 'submodule with a long path' '
+# the git database must fit within PATH_MAX, which limits the submodule name
+# to PATH_MAX - len(pwd) - ~90 (= len("/objects//") + 40-byte sha1 + some
+# overhead from the test case)
+pwd=$(pwd)
+pwdlen=$(echo "$pwd" | wc -c)
+longpath=$(echo $longpath180 | cut -c 1-$((170-$pwdlen)))
+
+test_expect_success 'submodule with a long path' '
 	git init --bare remote &&
 	test_create_repo bundle1 &&
 	(
@@ -56,7 +61,7 @@ test_expect_failure 'submodule with a long path' '
 	)
 '
 
-test_expect_failure 'recursive submodule with a long path' '
+test_expect_success 'recursive submodule with a long path' '
 	git init --bare super &&
 	test_create_repo child &&
 	(
@@ -96,6 +101,5 @@ test_expect_failure 'recursive submodule with a long path' '
 		)
 	)
 '
-unset longpath
 
 test_done