GIT - HowTo's about "repositories"

How to extract part of a Git repository into a new independent repository ?

Situation :

I have a Git repository like :
.git
dir1
dir2
dir3
This was fine until today, but now I'd like dir2 to live as a distinct repository.

Details :

The concept is to :
  1. clone the existing repository into a new place. This will become the new dir2 repository
  2. remove everything from it which is not related to the contents of the dir2 directory

Solution :

  1. find the root of the repository
  2. clone it :
    existingRepoRootDir='/root/of/existing/repository'; newRepoDir='/path/to/new/repository'; mkdir -p "$newRepoDir" && git clone "$existingRepoRootDir" "$newRepoDir"
    Cloning into '/path/to/new/repository'...
    done.
  3. then enter the clone and keep only the dir2 stuff :
    Rewrite dd39933a75594ac29d780de568e872fa22e1a480 (45/62) (1 seconds passed, remaining 0 predicted)
    Ref 'refs/heads/master' was rewritten
    Ref 'refs/remotes/origin/master' was rewritten
    WARNING: Ref 'refs/remotes/origin/master' is unchanged
  4. The 2 previous steps as a big one-liner :
    existingRepoRootDir='/root/of/existing/repository'; relativePathToDirectoryToKeep='dir2'; newRepoDir='/path/to/new/repository'; mkdir -p "$newRepoDir" && git clone "$existingRepoRootDir" "$newRepoDir" && cd "$newRepoDir" && git filter-branch --subdirectory-filter "$relativePathToDirectoryToKeep" --prune-empty -- --all
  5. Don't forget the final clean up :
    • in this new repository : reset the remotes
    • in the source repository : delete stuff you just extracted (or you'll have duplicate contents and it'll become a nightmare )

How to "get" all branches from a remote ?

Using the vague word "get" on purpose because I still don't know whether its a matter of pull, fetch, or anything else .

  • when you git clone a repository, you get everything, including branches.
  • they are not listed by git branch until you explicitly git checkout myBranch
  • to get further changes made on the original repository, you'll have to explicitly checkout + pull every branch (view short example).

Full demonstration :

  1. create a repository with several branches :
    baseDir='/run/shm/playingWithGit'; repoDir="$baseDir/repo"; mkdir -p "$repoDir"; cd "$repoDir"; git init; echo 'hello world' > myFile; git add myFile; git commit myFile -m 'initial commit (on master)'; for i in {1..3}; do branchName="branch$i"; git checkout -b "$branchName"; echo "hello from '$branchName'" >> myFile; git commit myFile -m "commit from '$branchName'"; git checkout master; done; git branch
    Initialized empty Git repository in /dev/shm/playingWithGit/repo/.git/
    [master (root-commit) 2d79b06] initial commit (on master)
     1 file changed, 1 insertion(+)
     create mode 100644 myFile
    Switched to a new branch 'branch1'
    [branch1 407687e] commit from 'branch1'
     1 file changed, 1 insertion(+)
    Switched to branch 'master'
    Switched to a new branch 'branch2'
    [branch2 8680357] commit from 'branch2'
     1 file changed, 1 insertion(+)
    Switched to branch 'master'
    Switched to a new branch 'branch3'
    [branch3 4188c2c] commit from 'branch3'
     1 file changed, 1 insertion(+)
    Switched to branch 'master'
      branch1
      branch2
      branch3
    * master
  2. view branches content :
    for branch in $(git branch | tr -d '* '); do git checkout "$branch"; cat myFile; echo; done
    Switched to branch 'branch1'
    hello world
    hello from 'branch1'
    
    Switched to branch 'branch2'
    hello world
    hello from 'branch2'
    
    Switched to branch 'branch3'
    hello world
    hello from 'branch3'
    
    Switched to branch 'master'
    hello world
  3. create a clone and try to get everything from the remote :
    cloneDir="$baseDir/clone"; mkdir -p "$cloneDir"; cd "$cloneDir"; git clone "$repoDir" .
    Cloning into '.'...
    done.
    git branch; cat myFile
    * master
    hello world
    for branch in branch1 branch2 branch3; do git checkout "$branch"; cat myFile; echo; done
    Branch 'branch1' set up to track remote branch 'branch1' from 'origin'.
    Switched to a new branch 'branch1'
    hello world
    hello from 'branch1'
    
    Branch 'branch2' set up to track remote branch 'branch2' from 'origin'.
    Switched to a new branch 'branch2'
    hello world
    hello from 'branch2'
    
    Branch 'branch3' set up to track remote branch 'branch3' from 'origin'.
    Switched to a new branch 'branch3'
    hello world
    hello from 'branch3'
    git branch
      branch1
      branch2
    * branch3
      master
    All branches are actually there .
  4. make some changes on the remote repository :
    cd "$repoDir" && for branch in master branch1 branch2 branch3; do git checkout "$branch"; echo "I made this commit on branch '$branch' after the 'clone'" >> myFile; git add myFile; git commit -m "another commit on '$branch'"; done
    Already on 'master'
    [master 28b51d1] another commit on 'master'
     1 file changed, 1 insertion(+)
    Switched to branch 'branch1'
    [branch1 0ff4301] another commit on 'branch1'
     1 file changed, 1 insertion(+)
    Switched to branch 'branch2'
    [branch2 fcab109] another commit on 'branch2'
     1 file changed, 1 insertion(+)
    Switched to branch 'branch3'
    [branch3 fcf5830] another commit on 'branch3'
     1 file changed, 1 insertion(+)
  5. view branches content :
    Switched to branch 'branch1'
    hello world
    hello from 'branch1'
    I made this commit on branch 'branch1' after the 'clone'
    
    Switched to branch 'branch2'
    hello world
    hello from 'branch2'
    I made this commit on branch 'branch2' after the 'clone'
    
    Switched to branch 'branch3'
    hello world
    hello from 'branch3'
    I made this commit on branch 'branch3' after the 'clone'
    
    Switched to branch 'master'
    hello world
    I made this commit on branch 'master' after the 'clone'
  6. will I get these ?
    cd "$cloneDir" && git fetch
    remote: Enumerating objects: 20, done.
    remote: Counting objects: 100% (20/20), done.
    remote: Compressing objects: 100% (8/8), done.
    remote: Total 12 (delta 0), reused 0 (delta 0)
    Unpacking objects: 100% (12/12), done.
    From /run/shm/playingWithGit/repo
    	4188c2c..fcf5830  branch3	-> origin/branch3
    	407687e..0ff4301  branch1	-> origin/branch1
    	8680357..fcab109  branch2	-> origin/branch2
    	2d79b06..28b51d1  master	-> origin/master
    git branch
      branch1
      branch2
    * branch3
      master
  7. view branches content again :
    Switched to branch 'branch1'
    Your branch is behind 'origin/branch1' by 1 commit, and can be fast-forwarded.
      (use "git pull" to update your local branch)			thank you, Git 
    hello world
    hello from 'branch1'
    
    Switched to branch 'branch2'
    Your branch is behind 'origin/branch2' by 1 commit, and can be fast-forwarded.
      (use "git pull" to update your local branch)
    hello world
    hello from 'branch2'
    
    Switched to branch 'branch3'
    Your branch is behind 'origin/branch3' by 1 commit, and can be fast-forwarded.
      (use "git pull" to update your local branch)
    hello world
    hello from 'branch3'
    
    Switched to branch 'master'
    Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
      (use "git pull" to update your local branch)
    hello world
  8. git branch; git pull; cat myFile
      branch1
      branch2
      branch3
    * master
    Updating 5784923..5ccb801
    Fast-forward
     myFile | 1 +
     1 file changed, 1 insertion(+)
    Current branch master is up to date.
    hello world
    I made this commit on branch 'master' after the 'clone'
  9. Looks like there is no Git command to do this automatically for all branches at once. Instead :
    git checkout branch1
    Switched to branch 'branch1'
    Your branch is behind 'origin/branch1' by 1 commit, and can be fast-forwarded.	As seen above
      (use "git pull" to update your local branch)						Ok, I'll do this !
    git pull
    Updating b5e84db..6f16d51
    Fast-forward
     myFile | 1 +
     1 file changed, 1 insertion(+)
    Current branch branch1 is up to date.
    cat myFile
    hello world
    hello from 'branch1'
    I made this commit on branch 'branch1' after the 'clone'
  10. the generic command is then :
    for branch in $(git branch | tr -d '* '); do git checkout "$branch"; git pull; cat myFile; echo; done
    Already on 'branch1'
    Your branch is up to date with 'origin/branch1'.
    Already up to date.
    Current branch branch1 is up to date.
    hello world
    hello from 'branch1'
    I made this commit on branch 'branch1' after the 'clone'
    
    Switched to branch 'branch2'
    Your branch is behind 'origin/branch2' by 1 commit, and can be fast-forwarded.
      (use "git pull" to update your local branch)
    Updating 1e4eef9..56d7474
    Fast-forward
     myFile | 1 +
     1 file changed, 1 insertion(+)
    Current branch branch2 is up to date.
    hello world
    hello from 'branch2'
    I made this commit on branch 'branch2' after the 'clone'
    
    Switched to branch 'branch3'
    Your branch is behind 'origin/branch3' by 1 commit, and can be fast-forwarded.
      (use "git pull" to update your local branch)
    Updating 545c032..ab21a4f
    Fast-forward
     myFile | 1 +
     1 file changed, 1 insertion(+)
    Current branch branch3 is up to date.
    hello world
    hello from 'branch3'
    I made this commit on branch 'branch3' after the 'clone'
    
    Switched to branch 'master'
    Your branch is up to date with 'origin/master'.
    Already up to date.
    Current branch master is up to date.
    hello world
    I made this commit on branch 'master' after the 'clone'
  11. (maybe I'll find further stuff to test / question / investigate / show... ;-)
  12. clean everything before leaving :
    for dirToDelete in "$repoDir" "$cloneDir"; do cd "$baseDir" && [ -d "$dirToDelete" ] && rm -rf "$dirToDelete"; done; cd .. && rmdir "$baseDir"

Replay the full demo with the final git checkout + git pull :

  1. create a repository with several branches
  2. clone it
  3. make some changes on the remote repository
  4. get changes on the cloned repository :
    cd "$cloneDir"; for branch in master branch1 branch2 branch3; do git checkout "$branch"; git pull; cat myFile; done
  5. clean everything before leaving

How to find the root directory of a Git repository ?

git rev-parse --show-toplevel

Why doesn't git status show whether I'm up-to-date with my remote counterpart ?

Situation :

git status
On branch master
nothing to commit (use -u to show untracked files)
I expected something like :
On branch master
Your branch is ahead of 'origin/master' by 2 commits.

Details :

Try these :
  1. git branch -vv
    * master 6f3d70e (a very interesting commit message)
    should be :
    * master 35ca599 [origin/master] (another highly interesting commit message)
  2. git rev-parse --symbolic-full-name "@{u}"
    fatal: No upstream configured for branch 'master'
    should be :
    refs/remotes/origin/master

This is because the master local branch has no upstream origin/master remote branch to be compared to.

The presence or absence of an upstream setting mainly affects whether git status can tell you if you are ahead or behind, and whether git merge and git rebase can do their job with no additional parameters. So it's basically just a convenience setting.

Solution :

  1. Define an upstream for the current branch :
    git branch --set-upstream-to origin/master
    Branch master set up to track remote branch master from origin.
  2. Retry the commands above :
    1. git branch -vv
      * master 6f3d70e [origin/master: ahead 7] (a very interesting commit message)
    2. git rev-parse --symbolic-full-name "@{u}"
      refs/remotes/origin/master
  3. And :
    git status
    On branch master
    Your branch is ahead of 'origin/master' by 7 commits.
    	
  4. Should you need to unset the upstream :
    git branch --unset-upstream

How to untrack a file ... ?

... without deleting it from the local filesystem :
git rm --cached file
... and delete it from the local filesystem :
git rm file, then commit file
To untrack an entire directory and delete it :
  1. git rm -r path/to/directory/*
  2. then make a simple commit : git commit -m 'cleaning'

How to shrink the .git directory ?

git gc --aggressive --prune

Resulted in a ~10% volume decrease.

How to hide untracked files from git status ?

Once this is defined as an alias in ~/.gitconfig, you may list untracked files with : git status --untracked-files

How may I push to a non-bare repository ?

This is dirty1000 and is actually not recommended.

Situation :

Before things are configured : can not push to non-bare repo

git push origin master
Counting objects: 6, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 697 bytes | 0 bytes/s, done.
Total 6 (delta 4), reused 0 (delta 0)
remote: error: refusing to update checked out branch: refs/heads/master
remote: error: By default, updating the current branch in a non-bare repository				the error
remote: error: is denied, because it will make the index and work tree inconsistent
remote: error: with what you pushed, and will require 'git reset --hard' to match
remote: error: the work tree to HEAD.
remote: error:
remote: error: You can set 'receive.denyCurrentBranch' configuration variable to
remote: error: 'ignore' or 'warn' in the remote repository to allow pushing into			the solution
remote: error: its current branch; however, this is not recommended unless you
remote: error: arranged to update its work tree to match what you pushed in some
remote: error: other way.
remote: error:
remote: error: To squelch this message and still keep the default behavior, set				a workaround
remote: error: 'receive.denyCurrentBranch' configuration variable to 'refuse'.
To gitServer:/path/to/repository
 ! [remote rejected] master -> master (branch is currently checked out)
error: failed to push some refs to 'stuart@gitServer:/path/to/repository'

Solution :

Get the initial configuration value

The error message says that, on the gitServer side, we just have to change the value of the receive.denyCurrentBranch directive to ignore or warn. Alright but -just in case- what's the original value of this Git configuration directive ?
git config --get receive.denyCurrentBranch
(nothing but UNIX FAILURE exit code)
This directive is currently unset.

Change the configuration value (source) :

  1. git config receive.denyCurrentBranch ignore
  2. check :
    git config --get receive.denyCurrentBranch
    ignore

With Git 2.3.0+, it is possible to safely push to a non-bare repository with :

git config receive.denyCurrentBranch updateInstead

Let's try to push again :

git push origin master
Counting objects: 6, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 697 bytes | 0 bytes/s, done.
Total 6 (delta 4), reused 0 (delta 0)
To gitServer:/path/to/repository
	f5c8ce7..629a87e	master -> master

Hack you have to do after each push to a non-bare repository :

On the gitServer side :
git reset --hard
  • As for why you have to do so, refer to the large Git error message above.
  • On gitServer side, since the index and the working tree are out-of-sync, "addition changes" made by a git push will be reported as "remove changes" by git status (hence the need of the git reset).
  • If there are some uncommitted changes on gitServer, there is a risk of data loss.

How to remove sensitive data (such as passwords, private keys, ...) from the Git history ?

Keep in mind that any private information that has been committed to a public repository MUST BE CONSIDERED AS COMPROMISED. So, clean up your mess, but forget about your passwords, keys, ... and generate new ones !!!

  1. Navigate to the repository's working directory :
    cd home/of/my/git/repository
  2. Run git filter-branch, forcing (--force) Git to process -but not check out (--index-filter)- the entire history of every branch and tag (--tag-name-filter cat -- --all), removing the specified file ('git rm --cached --ignore-unmatch path/to/fileContainingSensitiveData) and any empty commits generated as a result (--prune-empty) :
    git filter-branch --force --index-filter 'git rm --cached --ignore-unmatch path/to/fileContainingSensitiveData' --prune-empty --tag-name-filter cat -- --all
    Specify path/to/fileContainingSensitiveData, not just the file name.
    Example:
    git filter-branch --force --index-filter 'git rm --cached --ignore-unmatch ./test' --prune-empty --tag-name-filter cat -- --all
    Rewrite de1291f7f10da919f611358539e3d2e0acfd632f (1/9)rm 'test'
    Rewrite 023a969f60328055b6cab2e1a82e307760b7a308 (2/9)rm 'test'
    Rewrite aaffedfe16c297d02c2fee5eb72b740c91cae942 (3/9)rm 'test'
    Rewrite 8a766a9226e360ba0f147ae9c890cec200f958d7 (4/9)rm 'test'
    Rewrite 1f9bd643aa3495f46ca9470c021a56f8a0765c62 (5/9)rm 'test'
    Rewrite 2e22a397f2283f398038d9a44f451914c0a58f68 (6/9)rm 'test'
    Rewrite 15f247b46d558b057ccead7f7980c7f24b58879c (7/9)rm 'test'
    Rewrite e54ff82df52d8e7d02749b17e4fac4aa9386122e (8/9)rm 'test'
    Rewrite 5bbcaf42adac870f13295b2bbe2673807bf1e059 (9/9)rm 'test'
    
    Ref 'refs/heads/master' was rewritten
    Ref 'refs/remotes/origin/master' was rewritten
  3. If fileContainingSensitiveData used to exist with a different path or name (git mv ), repeat the previous step for each previous path or name.
  4. Add fileContainingSensitiveData to .gitignore to prevent committing again by mistake :
    echo fileContainingSensitiveData >> .gitignore; git add .gitignore; git commit -m "I won't commit fileContainingSensitiveData again."
  5. Make sure your history is clean :
    git log --name-status
    http://stackoverflow.com/questions/1338728/delete-commits-from-a-branch-in-git
    git reset --hard commitId	WARNING, this deletes all commits from the current point back to commitId. Use with care !!!
  6. Force-push your local changes to overwrite your public repository, as well as all the branches you've pushed up :
    git push origin --force --all
  7. In order to remove the sensitive file from your tagged releases, you'll also need to force-push against your Git tags :
    git push origin --force --tags
  8. Ask others to rebase instead of merging to avoid sending back the compromised data
  9. After a while, once sure we've not broken anything (and with Git ≥ 1.8.5) :
    git for-each-ref --format='delete %(refname)' refs/original | git update-ref --stdin
    git reflog expire --expire=now --all
    git gc --prune=now
==========================================8<=========================================================
IN CASE OF SENSITIVE DATA PUSHED TO GITHUB :

SOLUTION 1 :
	untrack sensitive file from local git (see above)

	delete sensitive file from local git history
		http://help.github.com/remove-sensitive-data/
		git filter-branch --index-filter 'git rm --cached --ignore-unmatch hosts.csv' --prune-empty -- --all

		==> Doesn't seem to work on directory containing uncommitted modifications.

		If error message :
			Cannot create a new backup. A previous backup already exists in refs/original/
			==> git filter-branch -f --index-filter 'git rm --cached --ignore-unmatch output/hosts.cfg' --prune-empty -- --all

	commit to local repo
	commit to GitHub


	<to be checked>

	==> sensitive file still visible in history 

	[Grok mode] delete GitHub repo + recreate it (within web UI) + RE-SETUP above

	then push local repo to GitHub :
		git push -u origin master --force


	[Normal mode]
	force push local repo to GitHub :
		git push -u origin master --force

	</to be checked>

How to import an existing Git repository into another ?

Let's consider 2 repositories : To do so :

Before starting this procedure, make sure there are no uncommitted changes on both repositories. commit or stash them accordingly.

  1. cd path/to/bigRepo
  2. git remote add small path/to/smallRepo
  3. git fetch small
  4. git checkout -b small small/master
    At this step, all files are imported into the root of "bigRepo". See the "git ls-files | xargs mv + commit" hack below
  5. git checkout master
  6. git merge small --allow-unrelated-histories
  7. git remote rm small
  8. git branch -d small

pathToBigRepo='/path/to/bigRepo'; pathToSmallRepo='/path/to/smallRepo'; pathToSmallRepoWithinBigRepo="$pathToBigRepo/smallRepo"; mkdir -p "$pathToSmallRepoWithinBigRepo"; cd "$pathToBigRepo"; git remote add remoteRepository "$pathToSmallRepo"; git fetch remoteRepository; git checkout -b newLocalBranchForRemoteData remoteRepository/master; git ls-files | xargs -I {} git mv {} "$pathToSmallRepoWithinBigRepo"; git co -am "Moved into '$pathToSmallRepoWithinBigRepo'"; git checkout master; git merge newLocalBranchForRemoteData --allow-unrelated-histories; git remote rm remoteRepository; git branch -d newLocalBranchForRemoteData

How to get a Git configuration value ?

How to contribute / perform a pull request ?

  1. Fork source project from GitHub UI
  2. Get the URL of your forked version of the project : https://github.com/myGitHubLogin/projectName.git
  3. Get a local copy of your forked repository : git clone https://github.com/myGitHubLogin/projectName.git
  4. Enter your local work directory : cd projectName
  5. Register the remote repo (named upstream by GitHub) : git remote add upstream https://github.com/projectOwner/projectName.git
  6. git fetch upstream
  7. Code, and commit to the local repository. Then push to your GitHub repository : git push origin master
  8. On GitHub UI, open your fork's webpage, then Pull-request
  9. Add title and details to the pull request, then submit.
  10. Done !
Stuff below could be obsolete / wrong

git clone https://github.com/sni/Thruk.git
git remote add --track master upstream git://github.com/Httqm/Thruk.git
git branch newFeature
git checkout newFeature
[do some code modifications, commit]
git push upstream newFeature (?)

How to checkout / clone an existing project ?

checkout is the SVN command to get a local copy of a project from its main repository (when joining a team, for instance).
Git also has a checkout command, but it has a different meaning.
The Git equivalent of SVN checkout is git clone.

clone from a remote host over SSH (I can log in there with ssh stuart@server) :
git clone stuart@server:/path/to/repository /local/path
  • on server, there is /path/to/repository/.git/
  • locally, I'll get /local/path/.git/
clone over SSH and "git checkout" the myDevelopmentBranch branch :
git clone git@host:/path/to/myGreatProject.git -b myDevelopmentBranch /path/to/local/directory
If /path/to/local/directory is not specified, the files will be extracted as ./myGreatProject/*