Showing posts with label Azure DevOps. Show all posts
Showing posts with label Azure DevOps. Show all posts

July 19, 2020

Git LFS Basics

In most cases, it's not the best practice to store large files in git. It's designed to store source code, not large files. However, for some legacy projects and processes where you don't want to invest a lot time and effort, or perhaps a game project with large assets, or an ML project with a large set of training data, or you're using Bitbucket Cloud which has a hard 2 GB repo size limit, it may be necessary to do so, and Git LFS allows you to "store" large files with whatever git workflow you're using, without git's inherent "inefficiencies" with large files.

Beware that if you decide to go with LFS, you'll lose some distributed-ness of git, since the content of some files have been moved to the LFS server and not part of the repo anymore, and when you clone the repo, it will only pull down the files that's needed for checking out the master branch. Be sure to have a backup policy in place.

Here's a high level animation of how Git LFS works:

As for what LFS does, the man page explains it pretty well:

$ git lfs
Git LFS is a system for managing and versioning large files in
association with a Git repository.  Instead of storing the large files
within the Git repository as blobs, Git LFS stores special "pointer
files" in the repository, while storing the actual file contents on a
Git LFS server.  The contents of the large file are downloaded
automatically when needed, for example when a Git branch containing
the large file is checked out.
...
...

Let's go over some of the basics of Git LFS by doing the same operation with, and without, LFS.

Installation

If you have an older version of git and can't upgrade for some reason, you'll need to install Git LFS first. If you're on Windows, as of version 2.12.0, Git LFS is already bundled with Git for Windows.

Initialize and Configure Git LFS

Let's start from two brand new repos in the remote (such as Azure Repos, Bitbucket, GitHub), and clone them into local folders. In the first repo, we'll enable LFS. In the second repo, we'll leave it alone so that we can compare normal git with LFS.

In the first local repo, let's initialize and configure Git LFS:

$ git lfs install
Updated git hooks.
Git LFS initialized.

As you can see from the message, it uses git hooks to perform LFS operations. It modifies 4 hook files – post-checkout, post-commit, post-merge, and pre-push. Here's what pre-push looks like:

#!/bin/sh
command -v git-lfs >/dev/null 2>&1 || { echo >&2 "\nThis repository is configured for Git LFS but 'git-lfs' was not found on your path. If you no longer wish to use Git LFS, remove this hook by deleting .git/hooks/pre-push.\n"; exit 2; }
git lfs pre-push "$@"

Next, let's assume we have large DLL files that we need to keep in our repo, so we'll tell LFS to handle, or "track", DLL files.

$ git lfs track '*.dll'
Tracking "*.dll"

(Note that we're using quotes around *.dll to prevent the shell from expanding the actual files matching the pattern.)

Above will create .gitattributes with the following content:

$ cat .gitattributes
*.dll filter=lfs diff=lfs merge=lfs -text

According to Git LFS documentation, the pattern follows gitignore rules.

Commit and push the changes we made above.

Adding a Large File

Now, let's add a DLL to both repos, say a DLL called MyLibrary.dll which is about 1.6 MB. At this point, the two repos are about the same size, except for a few extra bytes to account for files such as .gitattributes.

  With LFS Without LFS
Folder Size: ~1.6 MB ~1.6 MB
count-objects
count: 6
size: 861 bytes
count: 3
size: 540 bytes

(The Folder Size is from Windows Explorer, and count-objects is from git count-objects -vH command.)

Stage the DLL file with git add.

  With LFS Without LFS
Folder Size: ~3.22 MB ~2.37 MB
count-objects
count: 7
size: 986 bytes
count: 4
size: 774.87 KiB

Now things are getting interesting...

  • So why is the LFS repo bigger? Git compresses objects, but LFS doesn't. An issue has been opened to address this in 2015, but still no implementation yet. It's still on their roadmap. (In a way it makes sense, since if you have large files, maybe they are compressed already, so you might not want to waste time and resources compressing it again as it may even yield a bigger file, so perhaps a more granular control is needed here.)
  • Git's count-object size is much smaller in LFS. This is the size of the push.
  • The 774 KiB for the non-LFS repo is about the size of the DLL compressed with zlib.
  • The file object is stored in .git/lfs/objects for LFS, not in .git/objects.

Let's commit and push to remote.

With LFS
$ git push
Uploading LFS objects: 100% (1/1), 1.7 MB | 89 KB/s, done.
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 4 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 437 bytes | 218.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
To https://bitbucket.org/[organization]/lfs-demo.git
   09e450f..753cc3d  master -> master
Without LFS
$ git push
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 4 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 717.60 KiB | 6.71 MiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
To https://bitbucket.org/[organization]/without-lfs-demo.git
   23d4243..b21091d  master -> master

Note that there's an extra step for uploading LFS objects in the LFS repo, and the push size to remote is much smaller in the LFS repo as we've seen from count-objects before.

Most git providers such as Azure Repos, Bitbucket, and GitHub have built-in support for LFS. You can also use a separate LFS server.

Cloning LFS Repo

One benefit of using LFS is that you can use git without the need to download the full git history data, as it will download LFS files as needed. In git, the LFS files it stores are actually reference, or "pointer" files to the LFS objects. Let's see a bit of how that works.

Let's overwrite the DLL with a much small one, say ~49 KB, then commit and push.

With LFS
$ git commit -m "Replaced with smaller DLL."
[master 7c9b2ab] Replaced with smaller DLL.
 1 file changed, 2 insertions(+), 2 deletions(-)
Without LFS
$ git commit -m "Replaced with smaller DLL"
[master 2286636] Replaced with smaller DLL
 1 file changed, 0 insertions(+), 0 deletions(-)
 rewrite MyLibrary.dll (99%)

Note that the messages are a bit different, and that's because as far as git is concerned for the LFS, the file that changed is the pointer file. If you run git show, you will see something like below, which shows you the content of the pointer file and how it changed. The hash is how it tracks the file between git and LFS object storage.

diff --git a/MyLibrary.dll b/MyLibrary.dll
index 91c5966..d8b338d 100644
--- a/MyLibrary.dll
+++ b/MyLibrary.dll
@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3dce36d583ba1c741e95df1a265e47f0de581bef77ab48165dd67266be7a42ef
-size 1677824
+oid sha256:2b615798c36b1996093d44e77eb5306b4db9260546ce5aa2d3f7dde23476586b
+size 49664

Now, let's clone the repositories to new folders, and see what the sizes are.

  With LFS Without LFS
Folder Size: ~122 KB ~872 KB
count-objects
count: 12
size: 1.68 KiB
count: 9
size: 801.93 KiB

So there you have it, Git LFS is much smaller, as it only downloaded the latest commit of the DLL file.

At this point, if you look at .git/lfs/objects/, there should be one folder, in my case 2b, and inside that is 61 folder. If you open that folder, there is a file, in my case: 2b615798c36b1996093d44e77eb5306b4db9260546ce5aa2d3f7dde23476586b, sitting in at 49KB. This is the actual MyLibrary.DLL file stored by LFS with git. Note that how the folder names match the beginning of the file name, which is the hash from the pointer file.

To view which hash the file corresponds to, use git lfs ls-files:

$ git lfs ls-files
2b615798c3 * MyLibrary.dll

What would happen if we checkout the previous commit, the one that had the larger DLL?

In my case, the previous commit's hash is a4febdc, so if we checkout that commit with git checkout a4febdc, the folder size gets larger, about 3.27 MB. There's a new folder under the .git/lfs/objects folder, storing the DLL of this commit. In my case, .git/lfs/objects/3d/ce/3dce36d5...., at 1.6 MB.

If we go back to the latest commit that has the smaller DLL, will LFS delete the old one from .git/lfs/objects? No, but you can run git lfs prune, which will delete the file from .git/lfs/objects/3d/ce (though it doesn't seem to delete the folders, just the file).

$ git lfs prune
prune: 2 local object(s), 1 retained, done.
prune: Deleting objects: 100% (1/1), done.

Now you may wonder, when I cloned the repo that had LFS, do I need to run git lfs install again? The answer is no, because I'm running git version 2.27. As per this commit, LFS clone support is built into git as of git version 2.15 (released in October 2017), so it will update the hook files and such. Note that the documentation for git lfs clone has not been updated with this information.

Some Concerns

Azure Repos LFS Interface

In Azure Repos, there doesn't seem to be a UI to view and manage LFS objects as of now. There is a suggestion, but it's closed.

Bitbucket Cloud offers a dedicated UI to manage LFS objects, such as deleting them:

Maximum File Size

So LFS can support large files, but there might be a limit on the maximum size of a single file:

  • GitHub enforces 4 GiB size limit on Team plan, and 5 GiB on Enterprise.
  • Azure Repos LFS doesn't seem to have a documented limit.
  • Bitbucket doesn't have a limit, as long as you pay for storage capacity ($10 per 100 GB increments). As per their documentation: "Note that there's no limit on the LFS file size you can push to Bitbucket Cloud."
  • Note that in most cases the storage limit is across the organization/account, not per repo.
Push Size Limit
  • Some remotes may have push limit size...? If you encounter errors while pushing to remote, may need to update some configuration, such as running git config http.version HTTP/1.1.
Pipelines
  • Some remotes may require special instructions when using LFS from a Pipeline process/builds, such as in Azure Pipelines.
Deleting LFS Objects from Remote

Since LFS is built to support git's workflow where all history is stored, it probably makes sense that you need to use caution if you want to delete LFS objects. For GitHub, and also for Azure Repos and Bitbucket, to reclaim LFS storage, you'll need to delete the entire repo. In Bitbucket, as seen above, you can use the UI to delete individual objects, but this will break your git unless you also clean up the git accordingly.

Other Limitations

Additional notes can be found on the follow up post.

February 18, 2020

MkDocs with Docker - Part 2: Azure Pipelines and ACR

In Part 1, we created a Docker image with MkDocs, and used the container to execute MkDocs. In this post, we will:

  • Improve the Docker image with additional MkDocs features.
  • Publish the image to Azure Container Registry.
  • Use the container in Azure Pipelines.
  • Use Azure Pipelines to publish the compiled documentation to Azure App Service.

Making Improvements

I'm not fond of MkDoc's default theme, and after trying out various themes, I've settled on Material for MkDocs. It seems to be the most feature-complete, with 3K GitHub stars and lots of extension support. (I also like Read-the-Docs theme, but it seems there are some bugs, such as code blocks rendering in a single line, and even though there's a pull-request for the fix, the developer has not merged it. That project has also been inactive for a couple of years.)

Let's change the Dockerfile to integrate the Material theme, and also to reduce some steps in preparation for Azure Pipelines. Refer to the sample GitHub repo for the project folder structure.

FROM python:3.8.1-alpine3.11

EXPOSE 8000

# Git is required for the git-revision-date plugin.
RUN apk add git

RUN pip install --no-cache-dir mkdocs && \
    pip install --no-cache-dir mkdocs-material && \
    pip install --no-cache-dir mkdocs-git-revision-date-localized-plugin && \
    pip install --no-cache-dir pymdown-extensions

# Note that /mnt/repo should be the git root folder, not the content folder,
# otherwise the git-revision plugin will complain during build.
CMD ["/bin/sh", "-c", "cd /mnt/repo/content && mkdocs build"]

I'm relatively new to Docker and python, so it's probably not the optimal Dockerfile, so will need to explore further, such as using multi-stage builds. Also should look into requirements.txt file for pip.

Note the CMD command in the Dockerfile – it assumes certain conventions for mounting the host directory path, so we'll need to follow it when we want to run the container in Azure Pipelines. If we don't want to follow the convention, it can be overwritten in docker run command.

Just out of curiosity, what shell does Docker use when it's executing the RUN command? Looks like it's /bin/sh.

Publish to Azure Container Registry

We can publish the docker image to Docker Hub, but we need to keep things private, so for this demo, let's set up an Azure Container Registry to host the docker images. Follow the quick start guide to setup an Azure Container Registry to push the image we've built above.

Enable the admin user functionality, as we'll use that later in Azure Pipelines below. Note that enabling the admin user is not recommended, but good enough for this demo. Service principals is used in production.

Azure Pipelines

To setup the pipeline, first, add a secret variable to store the Azure Container Registry's admin user password, as ACR-SECRET. Then add the following to azure-pipelines.yml:

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

steps:
- script: docker login -u <acr-name> -p $(ACR-SECRET) <acr-name>.azurecr.io
  displayName: 'Login to Azure Container Registry'

- script: docker pull <acr-name>.azurecr.io/tools/mkdocs:0.1
  displayName: 'Pull MkDocs docker image'

- script: docker run --rm --mount type=bind,source=$(pwd),target=/mnt/repo <acr-name>.azurecr.io/tools/mkdocs:0.1
  displayName: Run MkDocs build through the docker container.

The above pipeline will login to our container registry, pull down the image, and spin up a container of that image to execute MkDocs to build the documentation. One thing that may need to be investigated is if there's a way to cache the docker image so we don't have to pull it down every time. We'll need to weigh which will be more cost effective, since pulling from the ACR may incur cost. Also note that the Azure Pipelines Build Agents already have common docker images preinstalled, so we should try to use those as base when creating the Dockerfile. We'll also need to weigh the speed of the build, since loading the cache in the pipeline can also take some time.

Now, of course, you don't have to use the docker container here. You can just install MkDocs and its plugins in the pipeline... However, since all of my team members will be using the same docker container to build and test their documentation, we can be sure that it will work the same way on this build. And yes, there are other ways of enforcing those standards even if we didn't use Docker, but let's leave it at that for now.

Publish to Azure App Service

So the pipeline above does the build but doesn't actually do anything with the output of the build. Let's publish the compiled documentation site to Azure App Service. Since the site is static, I can also do Azure Storage Static Hosting, but it seems it's always public access. With the App Service, I'll put it behind our Active Directory authentication.

Follow the documentation to create the site. The documentation doesn't go into details on how to setup a static HTML site using the Portal, so I ended up just creating a Windows .Net Framework site for this demo.

Add the following to the azure-pipelines.yml from above:

- task: ArchiveFiles@2
  inputs:
    rootFolderOrFile: 'content/site'
    includeRootFolder: false
    archiveType: 'zip'
    archiveFile: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip'
    replaceExistingArchive: true

- task: AzureRmWebAppDeployment@4
  inputs:
    ConnectionType: 'AzureRM'
    azureSubscription: '<Your Subscription Here>'
    appType: 'webApp'
    WebAppName: '<app-site-name>'
    packageForLinux: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip'

There are two additional tasks added – one to zip up the content of MkDocs build output, and the other to publish that zip file to Azure App Service.

We could also use Release Pipelines, but for this demo, it's enough to just let the build pipeline handle the deployment.

Now whenever the master branch is updated, the pipeline will do the build and publish the site.

Misc. Notes

The git-revision-date-localized plugin requires git to be installed in the container, since it uses git when MkDocs builds the site. It takes the date of the commit and adds it to the compiled document, and is supported by the Material theme.

Note that there are two sections for configuring MkDocs – plugins and markdown extensions. The git-revision-date-localized is configured under the plugin section. Refer to the demo GitHub repo.

It's slightly concerning that MkDocs hasn't had a release since September 7th, 2018. There are some issues, such as favicon not working, where the fix has been merged but hasn't been released yet, though a workaround does exist — overwriting it after the site has been built, which I've implemented in the azure-pipelines.yml in the demo repo.

One of the big gripes when working with markdown is handling images — you can't simply paste in a screenshot, which we often need to do when creating technical documentation. Fortunately, Visual Studio Code has an extension, Markdown Paste, that seems to work well.