Git LFS — A 2GB .git Folder Broke CI for Everyone

Git is designed for text files. Every version of every file lives in .git forever. A 100MB design file modified 50 times generates 5GB of .git storage — and everyone who clones the repo downloads all 5GB. Git LFS solves this by replacing large files with text pointers and storing the actual blobs on a remote server. The trade-off: LFS files aren't in the object store, so git bisect, git log -p, and git archive may not include LFS content. This guide covers setup, the incident that broke CI with a 2GB .git folder, and how to migrate existing history to LFS.

Why Git LFS Exists: The Problem with Large Files in Git

Git stores every version of every file in its object database. For text files, this is efficient because Git compresses and diffs them. But binary files—like images, audio, videos, or compiled binaries—don't diff well. Every commit that changes a 100 MB binary file adds another 100 MB to the repository. Over time, this bloats the repository, slows down clones and fetches, and makes git push time out. Git LFS (Large File Storage) solves this by replacing large files in the repository with lightweight text pointers, while storing the actual file content on a remote server (e.g., GitHub, GitLab, or a self-hosted LFS server). Only the pointer is versioned in Git; the blob is downloaded on demand during checkout. This keeps the Git history lean and operations fast. LFS is not a replacement for Git—it's a complementary extension that hooks into Git's smudge/clean filters to transparently manage large files.

# Install Git LFS (macOS with Homebrew)
brew install git-lfs

# Or on Ubuntu/Debian
sudo apt install git-lfs

# Initialize LFS for your user account
git lfs install

# Verify installation
git lfs version
# Output: git-lfs/3.4.0 (Git v2.39.0)

Installing and Configuring Git LFS

Installation is straightforward: download the package for your OS from git-lfs.com or use a package manager. After installation, run git lfs install once per user account to set up the required Git hooks and filters. This modifies your global Git config to register the LFS smudge and clean filters. Next, in your repository, you need to tell LFS which file patterns to track. Use git lfs track ".psd" to track all Photoshop files, or git lfs track "assets/*" for a directory. This creates or updates a .gitattributes file at the root of your repo. Commit this file to share the tracking rules with your team. Without .gitattributes, each developer would have to run git lfs track manually. Always commit .gitattributes early. You can also configure LFS storage backend (e.g., GitHub, GitLab, or custom S3) via git config lfs.url. For GitHub, it's automatic; for self-hosted, set lfs.url to your LFS server endpoint.

# Inside your repository
git lfs track "*.psd"
git lfs track "*.zip"
git lfs track "assets/videos/*.mp4"

# Check what's tracked
git lfs track
# Output:
# Listing tracked patterns
#     *.psd (.gitattributes)
#     *.zip (.gitattributes)
#     assets/videos/*.mp4 (.gitattributes)

# Commit the .gitattributes file
git add .gitattributes
git commit -m "Configure Git LFS for large file types"

How Git LFS Works Under the Hood

Git LFS operates through two Git filters: smudge and clean. When you add a file matching a tracked pattern, the clean filter replaces the file content with a pointer file. This pointer is a small text file (about 100 bytes) containing the SHA-256 hash of the original file and the LFS server URL. The actual file is uploaded to the LFS server. When you checkout a commit, the smudge filter reads the pointer and downloads the real file from the server. This is transparent: you see the actual file in your working directory, but Git only stores the pointer. The pointer file is what gets committed and pushed to the remote Git repository. The LFS server stores the blobs separately. This separation means that cloning a repo with LFS files is fast—you only download the pointers. The actual files are fetched lazily on checkout. You can also prefetch all LFS files with git lfs fetch --all if needed. LFS also supports file locking to prevent merge conflicts on binary files.

# Create a large file and track it
echo "This is a large file content" > large.bin
git lfs track "*.bin"
git add large.bin
git commit -m "Add large file via LFS"

# View the pointer file stored in Git
git show HEAD:large.bin
# Output:
# version https://git-lfs.github.com/spec/v1
# oid sha256:abc123...
# size 28

Migrating an Existing Repository to Git LFS

If you already have large files committed in your Git history, you can migrate them to LFS using git lfs migrate. This command rewrites history, replacing the large files in all commits with LFS pointers. It's a destructive operation—it changes commit SHAs. Therefore, coordinate with your team: everyone must clone the new history after migration. The most common command is git lfs migrate import --include=".psd,.zip" --everything. This scans all branches and tags, imports matching files into LFS, and rewrites the commits. You can also use --include-ref to limit to specific branches. After migration, run git push --force to update the remote. Be aware that git lfs migrate does not remove the old blobs from the Git object database; you need to run git reflog expire --expire=now --all && git gc --prune=now --aggressive to reclaim space. For safety, always backup your repo before migration.

# Backup your repo first!
cp -r my-repo my-repo-backup

# Migrate all .psd and .zip files to LFS
git lfs migrate import --include="*.psd,*.zip" --everything

# Force push the rewritten history
git push --force origin --all
git push --force origin --tags

# Clean up old objects
git reflog expire --expire=now --all
git gc --prune=now --aggressive

Working with Git LFS in a Team: Locking and Pull Requests

Binary files can't be merged like text files. If two developers modify the same image, Git will show a conflict in the pointer file (different OIDs), but you can't merge the actual content. Git LFS provides file locking to prevent this. Use git lfs lock path/to/file to lock a file exclusively. Other developers will see a warning if they try to lock or modify it. Locks are stored on the server and can be viewed with git lfs locks. Unlock with git lfs unlock path/to/file. In pull requests, LFS files are handled transparently: the diff shows the pointer change. Reviewers should verify the file was updated appropriately, but they can't see the actual binary diff. Some platforms like GitHub show a preview for images. For CI/CD, ensure your runner has LFS installed and configured. Use GIT_LFS_SKIP_SMUDGE=1 in CI to avoid downloading LFS files if not needed (e.g., for linting).

# Lock a file
git lfs lock assets/logo.psd
# Output: Locked assets/logo.psd

# View all locks
git lfs locks
# Output:
# assets/logo.psd  alice  ID:123

# Unlock
git lfs unlock assets/logo.psd

# In CI, skip smudge to avoid downloading LFS files
export GIT_LFS_SKIP_SMUDGE=1
git clone repo.git
# Only pointers are downloaded; actual files are not fetched.

Git LFS Storage and Bandwidth Limits: What You Need to Know

Git LFS is not free. Hosting providers like GitHub, GitLab, and Bitbucket impose storage and bandwidth quotas. GitHub, for example, offers 1 GB of LFS storage and 1 GB of bandwidth per month for free accounts. Beyond that, you pay for data packs. Bandwidth is consumed when someone clones or fetches LFS files. If you have a large repository with many LFS files, a single clone can eat your monthly bandwidth. To reduce bandwidth, use git lfs prune to delete local copies of LFS files that are not referenced by your current checkout. Also, consider using a CDN or self-hosted LFS server for large-scale usage. Monitor your usage via your provider's dashboard. For self-hosted LFS, you can use an S3 bucket as the backend, which gives you control over costs. Be aware that LFS traffic can be significant if you have many developers or CI pipelines cloning frequently.

# Prune local LFS files not needed for current checkout
git lfs prune
# Output:
# ✔ 3 local objects, 2 retained (1 removed)

# Check LFS disk usage
du -sh .git/lfs/objects
# Output: 2.3G    .git/lfs/objects

# View LFS logs for debugging
git lfs logs last

Troubleshooting Common Git LFS Issues

Common issues include: (1) git lfs install not working—ensure the binary is in PATH. (2) Files not being tracked—check .gitattributes is committed and patterns match. (3) Push failures due to LFS auth—use git config lfs.https://github.com/.username YOUR_USERNAME and git config lfs.https://github.com/.password YOUR_TOKEN. (4) Corrupted LFS objects—run git lfs fsck to check integrity. (5) Merge conflicts on pointer files—resolve by choosing the correct OID or re-downloading the file. (6) LFS files not downloading on checkout—check network connectivity and LFS server URL. Use GIT_TRACE=1 git checkout to debug. For persistent issues, check git lfs env to see configuration. If you accidentally commit a large file without LFS, use git lfs migrate to fix it, but be prepared for history rewrite.

# Check LFS configuration
git lfs env
# Output includes: git-lfs version, endpoint, filters, etc.

# Verify LFS objects integrity
git lfs fsck
# Output: OK or list of missing/corrupt objects

# Debug a checkout
export GIT_TRACE=1
git checkout main
# Look for smudge filter calls

# Force re-download of LFS files
git lfs pull
# Or for a specific file
git lfs pull --include="*.psd"

Git LFS Alternatives and When to Use Them

Git LFS is not the only solution for large files. Alternatives include: (1) Git Annex—manages files without storing them in Git at all, but requires a separate daemon. (2) DVC (Data Version Control)—designed for ML models and datasets, integrates with cloud storage. (3) Submodules—can reference external repositories, but adds complexity. (4) Artifact repositories like Nexus or Artifactory—store binaries separately and reference them via URLs in code. (5) Simple workaround: don't version large files at all; use a script to download them from a CDN. Choose LFS when you need seamless Git integration and your team is comfortable with the overhead. For ML workflows, DVC is often better because it handles large datasets and model versioning natively. For very large repositories (>10 GB), consider splitting into multiple repos or using a monorepo tool like Buck or Bazel that handles large binaries natively.

# DVC example: track a dataset
dvc init
dvc add data/dataset.csv
git add data/dataset.csv.dvc .gitignore
git commit -m "Add dataset via DVC"

# Push to remote storage
dvc push

# Compare with LFS: LFS tracks the file directly, DVC tracks a .dvc file.
# LFS stores on LFS server; DVC stores on S3, GCS, etc.

Best Practices for Git LFS in Production

1. Set up LFS before the first commit. Retroactive migration is painful. 2. Keep .gitattributes in the repo root and commit it early. 3. Use specific patterns, not broad ones like *. 4. Lock binary files before editing. 5. In CI, set GIT_LFS_SKIP_SMUDGE=1 and only pull LFS files when needed. 6. Run git lfs prune periodically to free local storage. 7. Monitor LFS storage and bandwidth usage via provider dashboards. 8. Use git lfs fsck in CI to detect corruption. 9. For large teams, consider a self-hosted LFS server to control costs and latency. 10. Educate your team: LFS is not magic—it requires discipline. Avoid committing large files without LFS by adding a pre-commit hook that checks file sizes. Example: use git-lfs-pre-commit hook or a custom script.

#!/bin/bash
# .git/hooks/pre-commit: Reject files > 10 MB not tracked by LFS

THRESHOLD=10485760  # 10 MB

for file in $(git diff --cached --name-only); do
  size=$(stat -f%z "$file" 2>/dev/null || stat -c%s "$file" 2>/dev/null)
  if [ "$size" -gt "$THRESHOLD" ]; then
    if ! git check-attr filter "$file" | grep -q "filter: lfs"; then
      echo "Error: $file is larger than 10 MB and not tracked by Git LFS."
      echo "Use 'git lfs track \"$file\"' and re-add."
      exit 1
    fi
  fi
done

Git LFS Performance: Optimizing Clone and Fetch Times

Cloning a repo with LFS can be slow if all LFS files are downloaded. By default, git clone triggers the smudge filter, downloading all LFS files for the checked-out branch. To speed up clones, use GIT_LFS_SKIP_SMUDGE=1 to clone only pointers, then selectively pull LFS files with git lfs pull --include="*.psd". For shallow clones, use --depth=1 to limit history. For large repos, consider using git clone --filter=blob:none (partial clone) to avoid downloading blobs at clone time. LFS also supports parallel downloads via git config lfs.concurrenttransfers 8. Increase this number for faster downloads on high-bandwidth connections. On the server side, use a CDN or edge caching for LFS objects. For self-hosted LFS, place the storage backend close to your developers geographically.

# Fast clone: skip LFS download
export GIT_LFS_SKIP_SMUDGE=1
git clone --depth=1 https://github.com/user/repo.git
cd repo

# Pull only the LFS files you need
git lfs pull --include="*.psd"

# Increase parallel transfers
git config lfs.concurrenttransfers 8

# Use partial clone (Git 2.19+)
git clone --filter=blob:none https://github.com/user/repo.git

Git LFS and CI/CD: Integration Patterns

Integrating LFS with CI/CD requires careful handling. Most CI runners have Git LFS pre-installed, but you may need to install it manually. The key decision: when to download LFS files. For build jobs that need the binaries, run git lfs pull after checkout. For linting or static analysis, skip LFS entirely with GIT_LFS_SKIP_SMUDGE=1. For deployment, you might need to pull LFS files on the target server. Use git lfs fetch --all to prefetch all LFS objects for the entire history (useful for mirroring). Be aware of bandwidth limits: if your CI runs many builds, consider caching LFS objects between runs. On GitHub Actions, use actions/cache to cache .git/lfs/objects. For self-hosted runners, use a shared network drive. Also, ensure your CI has proper authentication: set GIT_LFS_SKIP_SMUDGE and then use git lfs pull with a token stored in secrets.

name: CI
on: [push]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          lfs: true  # This installs LFS and pulls files
      - name: Build
        run: make build

  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          lfs: false  # Skip LFS to save bandwidth
      - name: Lint
        run: make lint

Git LFS Security: Authentication and Access Control

LFS authentication is separate from Git authentication. Most providers (GitHub, GitLab) use the same credentials, but you can configure custom auth. For self-hosted LFS, you can use basic auth, SSH, or OAuth. The LFS endpoint URL is typically https://hostname/path/to/repo.git/info/lfs. You can set credentials via git config lfs.https://hostname/.username and git config lfs.https://hostname/.password. For tokens, use git config lfs.https://hostname/.access-token. Be careful: storing passwords in plain text is insecure. Use credential helpers like git-credential-oauth or git-credential-manager. For access control, LFS respects repository permissions: if a user can read the repo, they can download LFS files; if they can write, they can upload. For fine-grained control, use a self-hosted LFS server with custom middleware. Also, LFS files are stored as blobs with their original content; ensure sensitive files are not accidentally tracked. Use .gitattributes to exclude sensitive patterns.

# Set credentials for a self-hosted LFS server
git config lfs.https://lfs.example.com/.username myuser
git config lfs.https://lfs.example.com/.password mytoken

# Use a credential helper (recommended)
git config --global credential.helper osxkeychain  # macOS
# or
# git config --global credential.helper manager  # Windows

# Verify authentication
git lfs fetch --all