Adding a New Repository to Git/Gitosis
It’s been a while since I used my own git server. I’m pulling together a bunch of management scripts into a toolbox to take with me from server to server and wanted to make a new repository.
Except I had no idea what to do to make a new one and I don’t seem to have written it down last time since it seemed so obvious at the time.
So, here’s the scoop…
To create new repositories:
- Add the project to the gitosis configuration
- Make them writable for the user you want to push as
- Push up the gitosis-admin configuration
- Create a new home for the project
- Initialize it as a git repository
- Copy all your project files into the new repository directory
- Add them all to the git repository
- Commit everything
- Push to the server
For example: let’s assume your username is ‘jdoe’ and you want to create a repository ‘myproject’.
In your clone of your server’s gitosis-admin, edit ‘gitosis.conf’ and add::
[group exampleproject] members = jdoe writable = myproject
Commit that change and push the changed gitosis configuration.
# git commit -a -m "Added new group 'exampleproject' and project 'myproject'"
# git push
Then create the local repository:
# mkdir myproject
# cd mypyroject
# git init
# ...copy all your project files in...
# git commit -a -m "First commit"
Set it up to push to the remote:
# git remote add myserver gitosis_user@yourserver.com:myproject.git
Do some work, add, commit everything then push it up:
# git commit -a -m "Putting away before first push to server"
# git push myserver master:refs/heads/master
That’s it. You now have a new project up on your git server and every time you do a git push it’ll go up.
Moving a Gitosis Setup
We’ve done a server shuffle here and now that the dust has settled a bit, I’ve got to get out Gitosis repository back online.
To make things more interesting, I also changed the name of the user under which the whole setup is running. Long story short; create a user in cPanel to be your Gitosis user rather than creating just a unix user with useradd so that the user can be moved etc. using the handy cPanel tools. My original user almost got stranded on the old server since he didn’t show up in any of the cPanel tools I was using to move hosting accounts.
So, on the new server (as per the old post :
# cd /usr/local/src # git clone git://eagain.net/gitosis.git # cd gitosis # python setup.py install Traceback (most recent call last): File "setup.py", line 2, in from setuptools import setup, find_packages ImportError: No module named setuptools
I downloaded ez_setup.py from the Peak website and than try to remember anything about how to set it up, I just took the two lines at the top of ez_setup.py and stuck’em into the gitosis setup.py per the instructions at the top of ez_setup.py and reran setup.py.
When I moved servers, I created a cPanel user with a new name, git_new, different from the one on the old server and copied all the Gitosis stuff into their home directory.
The trick is to get the gitosis setup to recognize everything in its new home and getting at it from my local machine. I’m not comfortable manually twiddling any of the files in the git setup since I’m not 100% sure what’s what so I’ll start by trying to checkout the Gitosis configuration to my local work area.
Since I’ve already created the user in cPanel, I’ll have to figure out how to make sure they’re not able to login by password but for now, I’m just going to get gitosis installed and initialized.
Before I do anything crazy, I’m going to make a quick copy of my original repositories. They’re in /home/git_new/repositories so I just quickly make a backup with:
# cp -r /home/git_new/repositories /home/git_new/repositories.2009.03-06.bak
I always put the date in backups so that I know when I was messing with something. File dates are unreliable since file changes change dates and it’s not easy to see when something was created. This keeps things straight with very little effort.
I copied my id_rsa.pub key from my main machine up to the /tmp directory on the new server and ran:
# sudo -H -u git gitosis-init < /tmp/id_rsa.pub
I got an error doing this because the /home/git_new/gitosis directory wasn’t owned by the git_new user but after fixing that up with:
cd /home/git_new chown -R git_new:git_new gitosis repositories
everything went fine. I fixed the user on /home/git_new/repositories while I was at it, btw.
Now rerunning the gitosis-init works just fine printing out the “Reinitializing…” message twice as documented.
Unfortunately, the original instructions used the –disabled-password switch on adduser which can’t be used here since the user was created through cPanel and neither cPanel’s Password Modification or the command line passwd utility can set a disabled password.
After poking around for a while, I finally just dumped the contents of /etc/shadow and saw that, for all the usually disabled accounts, the shadow password was set to !!. I manually edited /etc/shadow, poked in that password, and off we went.
This leaves the user unable to login via password, but they can still login using an ssh key which is exactly what we want. According to shadow(5),
If the password field contains some string that is not valid result of
crypt(3), for instance ! or *, the user will not be able to use a unix
password to log in, subject to pam(7). There are some intricacies and stupidity in the exact configuration and who can login which way but we don’t need to go there for this purpose.
NOTE: You can also globally disable password authentication in the ssh server but that’s not what we needed here either.
The next step, following my own instructions, is to attempt to clone the configuration repository with:
# git clone git_new@my.hostname.com:gitosis-admin
resulting in:
fatal: protocol error: bad line length character
Fooey.
Googling around lead to the Git FAQ entry on that error.
Following the instructions there, I just tried ssh’ing in as the user to execute a simple command with:
# ssh git_new@my.hostname.com echo testing commands Shell access is not enabled on your account!
I had forgotten to enable shell access on the account (it’s off by default in my cPanel setup since it’s a security risk and so few hosting clients actually need it). If I had gotten stucker I would have used ssh’s -v parameter to get verbose output to see where things were falling down.
I went into cPanel and enabled shell access for the account and voilà, out came my stuff!
Since I left my repositories in place in the /home/git_new/repositories directory I was hoping to just be able to check them out normally.
I always create a test repository, available to everyone who has any privileges on any repository, so that we can quickly establish connectivity to the server before messing around with specific privileges on any particular project.
So, the quickest test for a new repository is to just try:
# git clone git_new@my.hostname.com:test
And, since it hadn’t clobbered any of my original settings when I reinitialized gitosis, I’m right back in business!
NOTE: Just for fun, I tried to use some of my checkouts from the old repositories, just to see how hard it would be. A simple git pull or git push didn’t work and failed in a strange and not worth repeating sort of way.
Just edit the checkout’s .git/config and change the [remote "origin"] section’s url variable to the current user name and repository hostname and everything seems to work perfectly.
Git Subprojects
Subprojects with Git
I’m working on a fairly complex piece of OS X software right now; lots of moving parts.
I also have several elves working on these various parts. The parts are very well isolated from the main project in a plug-in-y sort of way.
The main project has some kinky dependencies that I really don’t want to have to help someone set up and there’s no real reason to. In the final build, it’s all there.
I’m paying them to help with the parts they’re good at, not dick around with installation and configuration and such.
This also means that each independent piece has to have its own completely separate test application/suite that demonstrates the component’s functionality while remaining completely uncoupled from the main app.
All is as it should be.
Except for the version control part.
Each component has its own Git repository, and every developer working on the component can check out the whole component’s repository. No problem.
Except that I’ve also got to be able to check out their work into my build tree to make the whole product.
Git Subprojects to the Rescue
I just picked up The Pragmatic Programmers’ Pragmatic Version Control Using Git. Haven’t had much time to get into it yet but I did jump on the chance to celebrate their 5th anniversary with a 30% off coupon. Congratulations to them, they really do publish some fine books and they deserve all their success.
Chapter 8, “Organizing Your Repository” has a section on using Git’s submodules feature to track external dependencies. These will be familiar to Subversion users as svn:externals.
Creating a Submodule
To illustrate, let’s just create a whole new structure starting with the root of the project.
# cd ~ # mkdir projectRoot # cd projectRoot # git init
Pretty straightforward. Make a subdirectory, initialize the git repository in it.
I’m going to follow the book here since there’s already a public repository out there and it’s not like a million people are going to read this drivel and rush out and knock github over.
# git submodule add \ git://github.com/tswicegood/hocus.git \ hocus
This creates the submodule hocus under projectRoot. What is not particularly clear is that this initializes the submodule to track the current HEAD of the external repository. It stores the revision number of the head in a configuration file called .gitmodules which you’ll add and track with the repository.
Unlike subversion, the submodule is bound to a particular revision within the external repository and will not follow the repository when modifications are made to it
Running the git submodule command will give a hint of that but it’s not that obvious, especially if you’re used to Subversion’s behaviour.
# git submodule -20cc9ddc65b5f3ea3b871480c1e6d8085db48457 hocus
This shows that there’s a single submodule and will show a ‘-’ sign next to it in the listing since it’s not been initialized.
To get it fired up:
# git submodule init hocus
That will register the hocus submodule.
To actually pull the contents of hocus, at the revision at which you ran submodule add, into the hocus subdirectory:
# git submodule update hocus
It is possible to change the commit to which the submodule is bound but that’s for another day (or buy the book!).
There’s a really good (and fast!) overview of this process and the intricacies of checking out a new copy of a project with submodules at the Rubaidh Ltd. blog.
Why git is Better Than Whatever You’re Using Now
Just found this interesting link:
Answers a lot of questions about why Git is taking over the world.
Gitosis: Adding Users
Adding Users to our Secure Gitosis setup
Now that we’ve got our gitosis based git server set up (see Previous Installment), it’s time to add users and repositories.
First, users.
Here’s what you’ve got to do:
Generate a public key
Here’s what to tell your users to have them generate a public key that you can use to give them access to your repositories.
Go to a terminal.
Unless you’ve changed something, you’ll be at your home directory. Just in case you’re not for some reason, type:
# cd
That should bring you to your home directory. Now, create a key with:
# ssh-keygen -t rsa
The key will be generated in the ~/.ssh hidden subdirectory as ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub
Have the user send you id_rsa.pub by whatever means.
Add their public key to the gitosis-admin/keydir
Copy the “id_rsa.pub” that the user sent you to the “keydir” subdirectory of your “gitosis-admin” checkout, renaming it on the way to the username that you can find as the last bit of the id_rsa.pub key.
There might be a way of renaming this so that you can use a shorter name than blah.machinename.local but I haven’t had time with a user to figure this out. Let me know in the comments if you know how to do this.
Jeff sent me his id_rsa.pub, his local user name was jeff.JeffsHappyMachine.local, so I copied his id_rsa.pub to the gitosis-admin/keydir as jeff.JeffsHappyMachine.local.pub
# cp ~/id_rsa.pub ~/gitosis-admin/keydir/jeff.JeffsHappyMachine.local.pub
Now I have jeff.JeffsHappyMachine.local as an available user. As I said, there must be a way to shorten this all up but I haven’t had time to explore this further.
To get this all up to the server, first commit:
# git commit -a -m "Added jeff's key to keydir"
Then push to the shared server with a simple:
# git push
Repeat as necessary with any other users. Obviously you don’t have to commit between each user.
Next, onto creating projects and allowing users to access them.
Installing Secure Git with Gitosis
Gitosis — making hosting multiple Git repositories manageable
Gitosis is a tool for managing hosted git repositories.
There’s a bit of a blurb here since the main page doesn’t really say anything about what it’s for &c.
To quote:
gitosis aims to make hosting git repos easier and safer. It manages multiple repositories under one user account, using SSH keys to identify users. End users do not need shell accounts on the server, they will talk to one shared account that will not let them run arbitrary commands.
It is written in Python, not that that really matters but if I do want to jump in and fix/enhance something I’ll be in familiar territory.
Installing Gitosis
On all of my systems, I keep all the stuff I install from source in /usr/local/src so, to get gitosis installed on my Linux server. You may need to sudo some of these depending on your setup:
# cd /usr/local/src # git clone git://eagain.net/gitosis.git # cd gitosis # python setup.py install
So far, so good.
Now, somewhat following the instructions at Garry Dolley’s Blog…
First snag is that, on the version of Linux I’m running on this particular server (CentOS 4), the long forms of the options to adduser don’t exist so I ended up using:
# sudo adduser \ -s /bin/sh \ -d /home/git \ git
Just to make sure I wasn’t adding a user who could log in with no password, I tried ssh-ing into the server as the ‘gut’ user. The ‘git’ user could not log in. So far so good.
On my main machine (Mac), I ran
# cd ~ # ssh-keygen -t rsa
To generate an rsa key (I only had dsa flavored key) and copied it up to the /tmp directory on my Linux server.
Then I ran:
# sudo -H -u git gitosis-init < /tmp/id_rsa.pub
This creates the gitosis configuration repository, gitosis-admin.git, in /home/git/repositories/.
The -H sets the HOME environment variable to the home of the user (/home/git, in this case) so that gitosis-init creates its configuration repository in the right place.
The -u makes the command get run as the supplied user (git).
As per the instructions, I also ran:
# sudo chmod 755 /home/git/repositories/gitosis-admin.git/hooks/post-update
Then, to get the ball rolling, I fetched the configuration repository to my local machine — I put it in my home directory, for now.
# cd ~ # git clone git@MY_SERVER_HOSTNAME:gitosis-admin # cd gitosis-admin
There’s not much there yet, just gitosis.conf and the keydir.
This is where the fun begins. By creating entries for new repositories in the gitosis.conf and adding the public keys for other users in the keydir, then pushing those changes to the server, you can control access to the repositories managed by gitosis all from your local copy.
More on that in my next post…
Bootstrapping Git Onto CentOS
I’m working on getting Git set up on the same server I’ve been using for my Subversion (SVN) server.
The SVN files are in /var/www/svn.
Just to keep things clean, I’m going to put the git stuff in its own directory /var/www/git.
Seems logical.
Since I’ve got no ‘git’ on this server at all, first, I have to “get git”, as it were.
On the home page of git at : Git
There’s a box in the middle of the right hand side of the page titled “Download GIT source package” which leads to:
So, at the command prompt:
[/usr/local/src]# wget http://kernel.org/pub/software/scm/git/git-1.6.0.5.tar.gz
[/usr/local/src]# tar zxvf git-1.6.0.5.tar.gz
[/usr/local/src]# cd git-1.6.0.5
[/usr/local/src]# ./configure
[/usr/local/src]# make
[/usr/local/src]# make install
[/usr/local/src]# git
Shows the help for the git command line.
Git installed, yay!
Git, SSH, How To Serve Git Securely
I’ve been using Git for a while now on my local projects. It’s quick to set up, fast, and I’ve always got my full revision history with me.
I’ve been using it locally, and backing up my projects with my normal methods but now I’ve got to share a repository.
I set up Subversion as per my previous couple of posts but, after doing so, realized that I really didn’t want to deal with Subversion any more.
In my opinion, Subversion’s conflict resolution absolutely sucks. You get three files, yours, the revision you started from, and the newest revision in files named ‘filename.r26′, ‘filename.r35′, and ‘filename.mine’ which is what the file you edited was before it spews conflicts right into your original source file to be cleaned up by hand. No thanks.
I haven’t found a tool that would make it easy to merge them and, because of that, I was always afraid to branch; way too much effort when trying to merge things back again.
We used Perforce on a project and its conflict handling was much better but at $900/seat, it’s a little hard to justify.
Git is supposedly much better than Subversion at conflict resolution and such and, as a leap of faith (and because I’ve found Subversion’s merging virtually unusable) I’m just going to set up the new project under Git and see how things go.
Into the breach!
I’m looking at the following references as I begin to figure out how to serve Git securely.
Perforce — IDC, PAY for Version Control?!
Update: please see my Git rants to see where we’re at now.
We’re starting a new software project here at IDC and some of the developers we’ve hired have strongly recommended using use Perforce. They also have recommended ExtraView for bug tracking which integrates with Perforce.
While it’s not against my religion to use expensive, proprietary software for business critical functions, especially server type functions, and especially when pricing is per user, it’s pretty damn close.
We’ve been using Subversion around here for a few years and, frankly, other than for storing revisions, it pretty much sucks.
The way it handles conflicts is a complete pain in the ass that always leaves me thinking “there’s got to be a better way”, pulling out older versions always sends me back to the manual, the monolithic numbering scheme has always given me agita, and God help you if you’ve got to merge a branch (or branches!) back to the trunk…the list goes on and on.
So, much as it pains me to face an $800/user licensing fee, the 2 user, 5 client workspaces limitation on the free, unlicensed version of Perforce has sucked me in.
ExtraView also has a limited, hosted version that will allow us to do this project and see how things go.
Who knows? We may actually have to pay for these things if they work well enough.
Next entry — my trials and tribulations getting it set up on one of our public-facing servers.
