GitHub Actions Sparse Checkout

Sparse Checkout Not Supported in actions/checkout@v2

At the time of writing this post, GitHub's built-in actions/checkout@v2 does not support sparse (partial) checkout, so I decided to just manually perform the checkout with sparse enabled. As a reference, there is an open issue about this on GitHub, with some comments offering some suggestions.

Note that the script I'm using is PowerShell, since the project I'm working on needs Windows to build.

A working demo can be found at my GitHub repo. The rest of the post goes through some of my findings.

Authentication

One tricky part was authentication. A simple way would be just adding the token to the repo URL:

git remote add origin https://${Env:GITHUB_ACTOR}:${{secrets.GITHUB_TOKEN}}@github.com/$Env:GITHUB_REPOSITORY

But that's not what actions/checkout does. Also, it's not a good idea to add authentication into the URL since it maybe logged by the server (though in this case it might not be that critical since it's the GitHub server itself and the auth token expires after each run). So to follow what actions/checkout does, I tried to use the header. According the log of when I used actions/checkout, it looked like it's just the token itself, since it's masked out:

"C:\Program Files\Git\cmd\git.exe" config --local http.https://github.com/.extraheader "AUTHORIZATION: basic ***"

But when I tried it, it didn't work. After looking at the actions/checkout code, found out it actually wraps the token with additional data. Here's the snippet of the job for handling this:

- name: Git - Setup auth extraheaders config
  run: |
	$authToken="x-access-token:${{ secrets.GITHUB_TOKEN }}"
	$bytes=[System.Text.Encoding]::UTF8.GetBytes($authToken)
	$encodedAuthToken=[Convert]::ToBase64String($bytes)

	git config --local http.https://github.com/.extraheader "AUTHORIZATION: basic ${encodedAuthToken}"

Fake Log Output..?

Well, I suppose technically it's not fake, but one very interesting thing that I noticed in the actions/checkout code is that the log masking is explicitly set, and not delegated to the general token masking feature of the log output. It executes git config command with the mask as the actual parameter. Afterwards, it changes the config file with the correct value, to avoid the OS from capturing the command line with the base64 token on it. So to people just looking at the log output, it looks as if it executed the git config command with the token value, but in reality, it didn't.

In addition to the regular token value, GitHub actions will also automatically mask the encodedAuthToken above.

LFS

To handle sparse checkout for LFS, I looked at how Jenkins' git plug-in does it, by looking at the log file first from the old Jenkins job I've had setup prior to moving to GitHub, and skimmed through the code to confirm, and which led me to using lfs.fetchinclude:

git config lfs.fetchinclude folder/subfolder

LFS Pull?

Since git lfs is included in the git installation of the host runner, the checkout will actually go through the filters and download the LFS files. But I've seen some cases where it missed a few, so I decided to add git lfs pull origin. I can probably just enable skip-smudge on the checkout and keep lfs pull for possibly better performance.

Further Improvements

As mentioned above, just like the actions/checkout, the job script can be modified to write the authentication in the config file instead of executing the git config command so that the OS will not capture it.

The job only handles main/master branch builds. Probably should parse the ref to find out the branch or use explicit sha.

If you're running this on a self-hosted runner, you should also do some clean up before checkout. For GitHub runners, it's probably not necessary since every new run starts with a clean slate.

Self-Hosted Runner

One thing I want to mention about GitHub actions – compared to Azure DevOps, I like that GitHub actions' self-hosted runner is free. So I can add multiple self-hosted runners for a repo without worrying about cost. Azure DevOps charges $15 per month for each runner if you want more than one for your organization.

SVN to Git Migration

Recently, we decided to migrate one of our "legacy" product in SVN repository to git that will be hosted on Bitbucket. It had to have the full history maintained. While researching this topic, I was surprised to find out that git has built-in support for SVN!

Steps

Note that I'm only migrating the trunk in this post, but you can also migrate the full SVN structure including branches and tags.

1. Get the list of users that have committed to SVN (from PowerShell): PS C:\MySVNRepo> svn log --quiet | ? { $_ -notlike '-*' } | % { "{0} = {0} <{0}>" -f ($_ -split ' \| ')[1] } | Select-Object -Unique | Out-File 'authors-transform.txt' -Encoding utf8
2. Open the authors-transform.txt generated from above and add the email address of the users. This step is actually optional, it's because git includes email addresses in commits and will help map out the users later when pushed to remote.
3. "Clone" the SVN repo as a git repo using the git svn clone command, e.g. in bash: git svn clone http://<svn repo URL>/trunk --prefix=svn/ --no-metadata --authors-file "authors-transform.txt" /c/repos/migrated-repo --username <svn user name>. (It will prompt for the SVN password.)
4. Create an empty repository on the git server, such as Azure Repos/Bitbucket/GitHub, and get the clone URL.
5. Add the clone URL as origin in the git repo: git remote add origin <clone URL>
6. Push to the git server: git push -u origin master

If you were to use LFS, you can run LFS migrate between steps 3 and 4.

Migration..?

Now, as you might know already, when you hear the word "migration", it never goes smoothly. Below are some of the issues that I had to deal with.

Git 2.27.0

So at the time, I was using git 2.27.0, and of course, git svn is broken in that release. Used 2.28 RC since it wasn't officially out yet. I suppose I could've downgraded as well.

Author Not Defined

In the middle of the process, it kicked out with a message, Author: [user] not defined in authors-transform.txt file. The [user] was the first user listed in the file. I've used the -Encoding utf8 option in the PowerShell command, and git didn't like the BOM. So opened the file in Notepad++ and converted to UTF-8 without BOM, and that did the trick.

Hanging, Timeouts and Other Errors

The SVN repo had a lot of revisions, riddled with large binaries. After getting some revisions, it hanged – no activity for a while. In the Task Manager, perl was taking up about 50% of the CPU, and the folder was not growing. So after canceling the command, cd'ed into the folder and ran git svn fetch --authors-file "../authors-transform.txt" as per this article, and that seemed to have made it continue from where it left off – it detected that the last retrieved revision was not complete, and started over again from that revision. All errors below were resolved the same way:

• Connection timed out: Connection timed out at C:/Program Files/Git/mingw64/share/perl5/Git/SVN/Ra.pm line 312
• 1 [main] perl 44975 cygwin_exception: Dumping stack trace to perl.exe.stackdump
• Failed to commit, invalid old:
• Name or service not known at C:/Program Files/Git/mingw64/share/per15/Git/SVN/Ra.pm line 312.
  (This can happen if your DNS goes down, yes, it happened.)

Checksum mismatch

This was a tricky one, and it occurred on random files. After many trials, I was able to avoid this error by running the git svn clone on the SVN server machine itself, e.g.:

git svn clone file:///c/SVNData/MyProject/trunk --prefix=svn/ --no-metadata --authors-file "authors-transform.txt" /c/gitrepo/my-project

I really don't know what the root cause is, maybe our internal network is unstable, or the SVN webserver (CollabNet) had some issues. It was a large SVN repo though, took about 24 hours to complete.

New Changes in SVN

For this migration, I didn't have to worry about two way support, since we were planning to retire the SVN after migrating to git. But I did have to get some new changes from SVN after the initial migration to git.

I used git svn fetch as above to get the latest SVN changes into git. This is not enough, though, since now I have the master branch, so I needed to bring in those changes into the master branch. This SO answer seems incorrect – running git svn rebase -l gave the following error message after running for a while:

$ git svn rebase -l
Unable to determine upstream SVN information from working tree history

Just simply doing git merge remotes/svn/git-svn worked.

LFS

If you run LFS migrate, you may no longer be able to run git svn fetch again and do a merge to master to bring it up to the latest:

$ git merge remotes/svn/git-svn
fatal: refusing to merge unrelated histories

LFS migrate rewrites commits, hence new hashes will be created for commits and git will think the master branch and the svn remote branch are completely separate since they won't share a common ancestor (remember, LFS rewrites the very first commit to add .gitattributes file). There are ways around it, such as specifying --allow-unreleated-histories option, but it may get ugly since all commits are technically different, and git will warn you about merging binaries as well:

$ git merge remotes/svn/git-svn --allow-unrelated-histories
warning: Cannot merge binary files: Libraries/MyLibrary.dll (HEAD vs. remotes/svn/git-svn)
warning: Cannot merge binary files: Libraries/MyLibrary.exe (HEAD vs. remotes/svn/git-svn)
...
...

One way to handle this would be by making backups before each step, and then you can go back to the point right before you ran LFS migrate, run git svn fetch, merge to master, then run LFS migrate again.

More on Git LFS

This is a follow up to the Git LFS Basics post – some additional notes on LFS.

Partial Clone

If the storage space is not a concern (e.g., Bitbucket's 2 GB hard limit), then partial clone and sparse checkout may replace the need for LFS. They are still in early stages, though.

Case Sensitivity

Windows file system is case insensitive, but git is case sensitive, so there may be problems when specifying the tracking pattern. According to this open issue (which was based on an older issue), you can use regex patterns, e.g., git lfs migrate import --include="*.[dD][lL][lL], *.[eE][xX][eE], [Bb]in/". Specifying it as "*.dll, *.DLL" doesn't work as expected.

LFS File Size Report

LFS has a built-in feature that will go through the history and report on the file types and sizes.

$ git lfs migrate info
migrate: Fetching remote refs: ..., done.
migrate: Sorting commits: ..., done.
migrate: Examining commits: 100% (1681/1681), done.
*.dll   5.6 GB  2013/2013 files(s)      100%
*.exe   3.6 GB    546/546 files(s)      100%
*.dat   1.9 GB        2/2 files(s)      100%
*.zip   1.7 GB      16/16 files(s)      100%
*.war   595 MB      17/17 files(s)      100%

The thing is, it defaults to showing only the top five. To show more, use the --top option, e.g., git lfs migrate info --top=100

Create .gitattributes Before or After Migrate?

Per migrate import documentation, it will create .gitattributes for you (except on certain cases based on options passed in). One thing it doesn't tell you is where it's added - it adds it to the very first commit in the history – it rewrites the very first commit (which is not surprising since rewriting history is one of the main tasks for LFS).

Find Lingering Large Files After the Migration

After migrating to LFS, if you find that the repo is still too large, you may want to run git lfs migrate info again, but it won't show any files. Instead, you can use git ls-tree to find large files in the repository. For example: git ls-tree -r -l --abbrev --full-name HEAD | sort -n -r -k 4 | head -n 10

Viewing Differences

There's no git lfs diff, and git log will show differences in "pointer" files, not the content. But in some cases, it might be useful to be able to view the differences. One way to do it is by using the external diff tool, e.g., git difftool HEAD^ HEAD Document.pdf (assuming you have difftool already configured). Note that in Bitbucket, if you browse to the source file tracked as LFS, it won't let you view the differences in the UI even if it's a "text" file, it just allows you to download the file.

Git LFS Basics

In most cases, it's not the best practice to store large files in git. It's designed to store source code, not large files. However, for some legacy projects and processes where you don't want to invest a lot time and effort, or perhaps a game project with large assets, or an ML project with a large set of training data, or you're using Bitbucket Cloud which has a hard 2 GB repo size limit, it may be necessary to do so, and Git LFS allows you to "store" large files with whatever git workflow you're using, without git's inherent "inefficiencies" with large files.

Beware that if you decide to go with LFS, you'll lose some distributed-ness of git, since the content of some files have been moved to the LFS server and not part of the repo anymore, and when you clone the repo, it will only pull down the files that's needed for checking out the master branch. Be sure to have a backup policy in place.

Here's a high level animation of how Git LFS works:

As for what LFS does, the man page explains it pretty well:

$ git lfs
Git LFS is a system for managing and versioning large files in
association with a Git repository.  Instead of storing the large files
within the Git repository as blobs, Git LFS stores special "pointer
files" in the repository, while storing the actual file contents on a
Git LFS server.  The contents of the large file are downloaded
automatically when needed, for example when a Git branch containing
the large file is checked out.
...
...

Let's go over some of the basics of Git LFS by doing the same operation with, and without, LFS.

Installation

If you have an older version of git and can't upgrade for some reason, you'll need to install Git LFS first. If you're on Windows, as of version 2.12.0, Git LFS is already bundled with Git for Windows.

Initialize and Configure Git LFS

Let's start from two brand new repos in the remote (such as Azure Repos, Bitbucket, GitHub), and clone them into local folders. In the first repo, we'll enable LFS. In the second repo, we'll leave it alone so that we can compare normal git with LFS.

In the first local repo, let's initialize and configure Git LFS:

$ git lfs install
Updated git hooks.
Git LFS initialized.

As you can see from the message, it uses git hooks to perform LFS operations. It modifies 4 hook files – post-checkout, post-commit, post-merge, and pre-push. Here's what pre-push looks like:

#!/bin/sh
command -v git-lfs >/dev/null 2>&1 || { echo >&2 "\nThis repository is configured for Git LFS but 'git-lfs' was not found on your path. If you no longer wish to use Git LFS, remove this hook by deleting .git/hooks/pre-push.\n"; exit 2; }
git lfs pre-push "$@"

Next, let's assume we have large DLL files that we need to keep in our repo, so we'll tell LFS to handle, or "track", DLL files.

$ git lfs track '*.dll'
Tracking "*.dll"

(Note that we're using quotes around *.dll to prevent the shell from expanding the actual files matching the pattern.)

Above will create .gitattributes with the following content:

$ cat .gitattributes
*.dll filter=lfs diff=lfs merge=lfs -text

According to Git LFS documentation, the pattern follows gitignore rules.

Commit and push the changes we made above.

Adding a Large File

Now, let's add a DLL to both repos, say a DLL called MyLibrary.dll which is about 1.6 MB. At this point, the two repos are about the same size, except for a few extra bytes to account for files such as .gitattributes.

	With LFS	Without LFS
Folder Size:	~1.6 MB	~1.6 MB
count-objects	count: 6 size: 861 bytes	count: 3 size: 540 bytes

(The Folder Size is from Windows Explorer, and count-objects is from git count-objects -vH command.)

Stage the DLL file with git add.

	With LFS	Without LFS
Folder Size:	~3.22 MB	~2.37 MB
count-objects	count: 7 size: 986 bytes	count: 4 size: 774.87 KiB

Now things are getting interesting...

• So why is the LFS repo bigger? Git compresses objects, but LFS doesn't. An issue has been opened to address this in 2015, but still no implementation yet. It's still on their roadmap. (In a way it makes sense, since if you have large files, maybe they are compressed already, so you might not want to waste time and resources compressing it again as it may even yield a bigger file, so perhaps a more granular control is needed here.)
• Git's count-object size is much smaller in LFS. This is the size of the push.
• The 774 KiB for the non-LFS repo is about the size of the DLL compressed with zlib.
• The file object is stored in .git/lfs/objects for LFS, not in .git/objects.

Let's commit and push to remote.

With LFS
$ git push Uploading LFS objects: 100% (1/1), 1.7 MB | 89 KB/s, done. Enumerating objects: 4, done. Counting objects: 100% (4/4), done. Delta compression using up to 4 threads Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 437 bytes | 218.00 KiB/s, done. Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 To https://bitbucket.org/[organization]/lfs-demo.git 09e450f..753cc3d master -> master
Without LFS
$ git push Enumerating objects: 4, done. Counting objects: 100% (4/4), done. Delta compression using up to 4 threads Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 717.60 KiB | 6.71 MiB/s, done. Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 To https://bitbucket.org/[organization]/without-lfs-demo.git 23d4243..b21091d master -> master

Note that there's an extra step for uploading LFS objects in the LFS repo, and the push size to remote is much smaller in the LFS repo as we've seen from count-objects before.

Most git providers such as Azure Repos, Bitbucket, and GitHub have built-in support for LFS. You can also use a separate LFS server.

Cloning LFS Repo

One benefit of using LFS is that you can use git without the need to download the full git history data, as it will download LFS files as needed. In git, the LFS files it stores are actually reference, or "pointer" files to the LFS objects. Let's see a bit of how that works.

Let's overwrite the DLL with a much small one, say ~49 KB, then commit and push.

With LFS
$ git commit -m "Replaced with smaller DLL." [master 7c9b2ab] Replaced with smaller DLL. 1 file changed, 2 insertions(+), 2 deletions(-)
Without LFS
$ git commit -m "Replaced with smaller DLL" [master 2286636] Replaced with smaller DLL 1 file changed, 0 insertions(+), 0 deletions(-) rewrite MyLibrary.dll (99%)

Note that the messages are a bit different, and that's because as far as git is concerned for the LFS, the file that changed is the pointer file. If you run git show, you will see something like below, which shows you the content of the pointer file and how it changed. The hash is how it tracks the file between git and LFS object storage.

diff --git a/MyLibrary.dll b/MyLibrary.dll
index 91c5966..d8b338d 100644
--- a/MyLibrary.dll
+++ b/MyLibrary.dll
@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3dce36d583ba1c741e95df1a265e47f0de581bef77ab48165dd67266be7a42ef
-size 1677824
+oid sha256:2b615798c36b1996093d44e77eb5306b4db9260546ce5aa2d3f7dde23476586b
+size 49664

Now, let's clone the repositories to new folders, and see what the sizes are.

	With LFS	Without LFS
Folder Size:	~122 KB	~872 KB
count-objects	count: 12 size: 1.68 KiB	count: 9 size: 801.93 KiB

So there you have it, Git LFS is much smaller, as it only downloaded the latest commit of the DLL file.

At this point, if you look at .git/lfs/objects/, there should be one folder, in my case 2b, and inside that is 61 folder. If you open that folder, there is a file, in my case: 2b615798c36b1996093d44e77eb5306b4db9260546ce5aa2d3f7dde23476586b, sitting in at 49KB. This is the actual MyLibrary.DLL file stored by LFS with git. Note that how the folder names match the beginning of the file name, which is the hash from the pointer file.

To view which hash the file corresponds to, use git lfs ls-files:

$ git lfs ls-files
2b615798c3 * MyLibrary.dll

What would happen if we checkout the previous commit, the one that had the larger DLL?

In my case, the previous commit's hash is a4febdc, so if we checkout that commit with git checkout a4febdc, the folder size gets larger, about 3.27 MB. There's a new folder under the .git/lfs/objects folder, storing the DLL of this commit. In my case, .git/lfs/objects/3d/ce/3dce36d5...., at 1.6 MB.

If we go back to the latest commit that has the smaller DLL, will LFS delete the old one from .git/lfs/objects? No, but you can run git lfs prune, which will delete the file from .git/lfs/objects/3d/ce (though it doesn't seem to delete the folders, just the file).

$ git lfs prune
prune: 2 local object(s), 1 retained, done.
prune: Deleting objects: 100% (1/1), done.

Now you may wonder, when I cloned the repo that had LFS, do I need to run git lfs install again? The answer is no, because I'm running git version 2.27. As per this commit, LFS clone support is built into git as of git version 2.15 (released in October 2017), so it will update the hook files and such. Note that the documentation for git lfs clone has not been updated with this information.

Some Concerns

Azure Repos LFS Interface

In Azure Repos, there doesn't seem to be a UI to view and manage LFS objects as of now. There is a suggestion, but it's closed.

Bitbucket Cloud offers a dedicated UI to manage LFS objects, such as deleting them:

Maximum File Size

So LFS can support large files, but there might be a limit on the maximum size of a single file:

• GitHub enforces 4 GiB size limit on Team plan, and 5 GiB on Enterprise.
• Azure Repos LFS doesn't seem to have a documented limit.
• Bitbucket doesn't have a limit, as long as you pay for storage capacity ($10 per 100 GB increments). As per their documentation: "Note that there's no limit on the LFS file size you can push to Bitbucket Cloud."
• Note that in most cases the storage limit is across the organization/account, not per repo.

Push Size Limit

• Some remotes may have push limit size...? If you encounter errors while pushing to remote, may need to update some configuration, such as running git config http.version HTTP/1.1.

Pipelines

• Some remotes may require special instructions when using LFS from a Pipeline process/builds, such as in Azure Pipelines.

Deleting LFS Objects from Remote

Since LFS is built to support git's workflow where all history is stored, it probably makes sense that you need to use caution if you want to delete LFS objects. For GitHub, and also for Azure Repos and Bitbucket, to reclaim LFS storage, you'll need to delete the entire repo. In Bitbucket, as seen above, you can use the UI to delete individual objects, but this will break your git unless you also clean up the git accordingly.

Other Limitations

• Bitbucket doesn't support Git LFS file locking.
• On Windows, it may have issues with files larger than 4 GB.

Additional notes can be found on the follow up post.

Enforcing Commit Message Format Using Husky

TL;DR: Github repo of the demo.

In the past, I've implemented githooks to check for the commit message format – mainly that it had to begin with a Jira ID. It works well, but the initial setup is a bit clunky – I had a separate repo to store githooks, and a batch file in the main repo to set core.hooksPath to the path of that other repo. Each developer had to pull down the githooks repo in a well-known folder path relative to the main repo and run the batch file...

I've recently introduced Husky to force linting locally before developers can commit (and push) their changes. It meant that I couldn't use the githooks I had for the commit message check, since Husky overwrites them during its installation, so I created a simple script to check for the commit message format and then added as a Husky task.

The demo repo is in Github. I could've used a shell script, or a PowerShell script, but since my team members have Windows and Mac, decided to write it in JavaScript and run it via node, since each front-end developer is guaranteed to have it for running npm. The script uses terminal color codes to highlight errors so that developers can quickly notice it.

  /*
    Assumptions:
    1. Only working with commit messages (tiny commit message files), so no need to worry about large file handling such as using a buffer.
    2. Assumed utf8 format, which is the git default unless changed explicitly (https://git-scm.com/docs/git-commit/#_discussion).
  */

  // Used for console color coding.
  const colorReset = "\x1b[0m";
  const colorBrightRed = "\x1b[1;31m";
  const colorGreen = "\x1b[32m";

  const fs = require("fs");

  // Customize the regex pattern below to match your desired commit message format.  The example pattern below checks
  // that commit messages begin with a Jira ID where the Jira Project Key is "JIRA".
  const pattern = /^JIRA-[0-9]+: .*/;

  // The environment variable is set by Husky.  The value is: .git/COMMIT_EDITMSG
  // Refer to https://git-scm.com/docs/githooks#_hooks for information on how git changes the working directory before
  // executing hooks.
  fs.readFile(process.env.HUSKY_GIT_PARAMS, "utf8", (err, data) => {
    if (err) throw err;

    if (data.match(pattern)) {
      console.log(colorGreen + "check-commit-msg: Passed." + colorReset);
    } else {
      console.log(
        colorBrightRed +
          'check-commit-msg: Failed - please begin your commit message with a Jira ID, followed by ":" and space e.g., "JIRA-123: Added commit message check."' +
          colorReset
      );
      process.exit(1);
    }
  });

One thing I've noticed is that the commit message hook runs after the pre-commit hook. Since I have pre-commit hook that does the linting, it means I have to wait for the linting to finish, then the commit message hook will kick in to let me know whether the commit message is good or not. The official reference doesn't specifically call out the order of hooks, though it looks like it's documented in a top-down order. The official book does say "The pre-commit hook is run first, before you even type in a commit message." This is probably due to how git works internally, since the commit message is actually part of the commit object and not some metadata (as in other traditional source control systems).

Of course, this husky solution only works for front-end (JavaScript) projects. I'll be keeping the standard githooks for other back-end projects (C#, C++, etc.) we have for now.

Also, while working on this, found commitlint. Might be worth a look...

Git Flow Diagram

We've been using git flow with some slight changes here and there. I wanted to document the process for future reference. View the post for the full size image.

Git Case Sensitivity on Branch Name

I was trying to push a local branch to remote, but kept getting this error (simplified example):

git push -u origin Bugfix/Foobar
fatal: Bugfix/Foobar cannot be resolved to branch

After googling a bit, came across this StackOverflow question, and that got me thinking about case sensitivity, so I changed the "B" to "b":

git push -u origin bugfix/Foobar

And that worked. Yes, I was pushing to a *nix based server from my Windows machine, though perhaps the message given by git could've been more clear. Other people had already pushed branches with the lower case "bugfix" before, so my guess is that git on the remote server couldn't create the folder using "Bugfix"...