How to find open files on a volume

This shows what files are open on volumename.

Handy if you’re trying to unmount volumename and it won’t unmount due to open files.

	# sudo lsof | grep /Volumes/volumename

Moving a Gitosis Setup

We’ve done a server shuffle here and now that the dust has settled a bit, I’ve got to get out Gitosis repository back online.

To make things more interesting, I also changed the name of the user under which the whole setup is running. Long story short; create a user in cPanel to be your Gitosis user rather than creating just a unix user with useradd so that the user can be moved etc. using the handy cPanel tools. My original user almost got stranded on the old server since he didn’t show up in any of the cPanel tools I was using to move hosting accounts.

So, on the new server (as per the old post :

	# cd /usr/local/src
	# git clone git://eagain.net/gitosis.git
	# cd gitosis
	# python setup.py install
	Traceback (most recent call last):
	  File "setup.py", line 2, in
	    from setuptools import setup, find_packages
	ImportError: No module named setuptools

I downloaded ez_setup.py from the Peak website and than try to remember anything about how to set it up, I just took the two lines at the top of ez_setup.py and stuck’em into the gitosis setup.py per the instructions at the top of ez_setup.py and reran setup.py.

When I moved servers, I created a cPanel user with a new name, git_new, different from the one on the old server and copied all the Gitosis stuff into their home directory.

The trick is to get the gitosis setup to recognize everything in its new home and getting at it from my local machine. I’m not comfortable manually twiddling any of the files in the git setup since I’m not 100% sure what’s what so I’ll start by trying to checkout the Gitosis configuration to my local work area.

Since I’ve already created the user in cPanel, I’ll have to figure out how to make sure they’re not able to login by password but for now, I’m just going to get gitosis installed and initialized.

Before I do anything crazy, I’m going to make a quick copy of my original repositories. They’re in /home/git_new/repositories so I just quickly make a backup with:

	# cp -r /home/git_new/repositories /home/git_new/repositories.2009.03-06.bak

I always put the date in backups so that I know when I was messing with something. File dates are unreliable since file changes change dates and it’s not easy to see when something was created. This keeps things straight with very little effort.

I copied my id_rsa.pub key from my main machine up to the /tmp directory on the new server and ran:

	# sudo -H -u git gitosis-init < /tmp/id_rsa.pub

I got an error doing this because the /home/git_new/gitosis directory wasn’t owned by the git_new user but after fixing that up with:

	cd /home/git_new
	chown -R git_new:git_new gitosis repositories

everything went fine. I fixed the user on /home/git_new/repositories while I was at it, btw.

Now rerunning the gitosis-init works just fine printing out the “Reinitializing…” message twice as documented.

Unfortunately, the original instructions used the –disabled-password switch on adduser which can’t be used here since the user was created through cPanel and neither cPanel’s Password Modification or the command line passwd utility can set a disabled password.

After poking around for a while, I finally just dumped the contents of /etc/shadow and saw that, for all the usually disabled accounts, the shadow password was set to !!. I manually edited /etc/shadow, poked in that password, and off we went.

This leaves the user unable to login via password, but they can still login using an ssh key which is exactly what we want. According to shadow(5),
If the password field contains some string that is not valid result of
crypt(3), for instance ! or *, the user will not be able to use a unix
password to log in, subject to pam(7). There are some intricacies and stupidity in the exact configuration and who can login which way but we don’t need to go there for this purpose.

NOTE: You can also globally disable password authentication in the ssh server but that’s not what we needed here either.

The next step, following my own instructions, is to attempt to clone the configuration repository with:

	# git clone git_new@my.hostname.com:gitosis-admin

resulting in:

	fatal: protocol error: bad line length character

Fooey.

Googling around lead to the Git FAQ entry on that error.

Following the instructions there, I just tried ssh’ing in as the user to execute a simple command with:

	# ssh git_new@my.hostname.com echo testing commands
	Shell access is not enabled on your account!

I had forgotten to enable shell access on the account (it’s off by default in my cPanel setup since it’s a security risk and so few hosting clients actually need it). If I had gotten stucker I would have used ssh’s -v parameter to get verbose output to see where things were falling down.

I went into cPanel and enabled shell access for the account and voilà, out came my stuff!

Since I left my repositories in place in the /home/git_new/repositories directory I was hoping to just be able to check them out normally.

I always create a test repository, available to everyone who has any privileges on any repository, so that we can quickly establish connectivity to the server before messing around with specific privileges on any particular project.

So, the quickest test for a new repository is to just try:

	# git clone git_new@my.hostname.com:test

And, since it hadn’t clobbered any of my original settings when I reinitialized gitosis, I’m right back in business!

NOTE: Just for fun, I tried to use some of my checkouts from the old repositories, just to see how hard it would be. A simple git pull or git push didn’t work and failed in a strange and not worth repeating sort of way.

Just edit the checkout’s .git/config and change the [remote "origin"] section’s url variable to the current user name and repository hostname and everything seems to work perfectly.

Python Path and .pth files

So…I have a django app I’ve been running on one of our old servers for a while and it’s time to move the client off to one of the new servers.

cPanel’d the account over, no problem, application moved with it, no problem, only the app won’t run.

To make a long story short, couldn’t figure out why sys.path() wasn’t returning my current directory whereas it was on the old server.

Turns out I had added a .pth file to /usr/local/lib/python2.4/site-packages/.

This time, however, I left myself a note here and in the application directory so I wouldn’t forget.

Oh, also remember to unblock the port in the firewall and restart the firewall.

Installing Secure Git with Gitosis

Gitosis — making hosting multiple Git repositories manageable

Gitosis is a tool for managing hosted git repositories.

There’s a bit of a blurb here since the main page doesn’t really say anything about what it’s for &c.

To quote:

gitosis aims to make hosting git repos easier and safer. It manages multiple repositories under one user account, using SSH keys to identify users. End users do not need shell accounts on the server, they will talk to one shared account that will not let them run arbitrary commands.

It is written in Python, not that that really matters but if I do want to jump in and fix/enhance something I’ll be in familiar territory.

Installing Gitosis

On all of my systems, I keep all the stuff I install from source in /usr/local/src so, to get gitosis installed on my Linux server. You may need to sudo some of these depending on your setup:

	# cd /usr/local/src
	# git clone git://eagain.net/gitosis.git
	# cd gitosis
	# python setup.py install

So far, so good.

Now, somewhat following the instructions at Garry Dolley’s Blog…

First snag is that, on the version of Linux I’m running on this particular server (CentOS 4), the long forms of the options to adduser don’t exist so I ended up using:

	# sudo adduser \
   		 -s /bin/sh \
	    -d /home/git \
	    git

Just to make sure I wasn’t adding a user who could log in with no password, I tried ssh-ing into the server as the ‘gut’ user. The ‘git’ user could not log in. So far so good.

On my main machine (Mac), I ran

	# cd ~
	# ssh-keygen -t rsa

To generate an rsa key (I only had dsa flavored key) and copied it up to the /tmp directory on my Linux server.

Then I ran:

	# sudo -H -u git gitosis-init < /tmp/id_rsa.pub

This creates the gitosis configuration repository, gitosis-admin.git, in /home/git/repositories/.

The -H sets the HOME environment variable to the home of the user (/home/git, in this case) so that gitosis-init creates its configuration repository in the right place.

The -u makes the command get run as the supplied user (git).

As per the instructions, I also ran:

	# sudo chmod 755 /home/git/repositories/gitosis-admin.git/hooks/post-update

Then, to get the ball rolling, I fetched the configuration repository to my local machine — I put it in my home directory, for now.

	# cd ~
	# git clone git@MY_SERVER_HOSTNAME:gitosis-admin
	# cd gitosis-admin

There’s not much there yet, just gitosis.conf and the keydir.

This is where the fun begins. By creating entries for new repositories in the gitosis.conf and adding the public keys for other users in the keydir, then pushing those changes to the server, you can control access to the repositories managed by gitosis all from your local copy.

More on that in my next post…

SVN, HTTPS, and Firewalls

The server on which I had SVN 1.4.xxx running had a custom Apache setup that ran the service securely on port 445 instead of the usual 443 to try to make it harder for hackers to find the service. Most hacker robots are stupid and only check ‘the usual’ since most sysadmins use ‘the usual’ since it’s easiest.

I hadn’t used the SVN server for a while since I have been using git for most of my current development.

As per my last post, I upgraded SVN to 1.5.x, restarted Apache, etc.

Still no joy connecting to it using the same settings that worked last time I used it.

Turned out that, somewhere in the “keeping the server up to date” upgrade process, my most excellent apf firewall got a rule rewritten to it that excluded the port I had chosen.

Turns out that port, 445, is hackable on Windows machines so I switched to a different port, made sure it was unblocked, restarted apf, and was then able to access my SVN repositories.

Fun? No.

Working? Yes.

How to restart cPanel manually

I frequently get e-mails when servers restart that ‘cpservd’ failed to restart.

You can have the cpanel services restarted if you logon to the server via ssh; use the command:

/etc/init.d/cpanel restart

along with the cpanel please make sure that you do restart the cpanel monitor service as:

/etc/init.d/chkservd restart

Installing Perforce on Red Hat Linux — The Gory Details

Do I Have A Pain In My Ass Yet?ere does not seem to be an RPMish installer for Perforce for Red Hat Linux.

I started following the instructions at Installing PerForce on Ubuntu but ran into some incompatibilities (apt-get not existing on Red Hat, for example).

Also, as I was following along, modifying the instructions along the way to adapt to Red Hat, I realized that some of the steps are out of order.

I’m just going to create my own instructions for Red Hat, based somewhat on the Ubuntu ones, but with my own improvements and adaptations. Hopefully it will be of some use to someone in the same boat down the road (maybe even me!). I have modified the original instructions enough that I’m not going to reference them or comment on the differences.

You just want to get Perforce installed on Red Hat Linux, right? Good, me too!

There was an installation script in the original post that I’ve saved and if I ever have to do this again, I’ll modify it and post a completed script that will work on Red Hat. For now, this is enough; I have work to do!

Configure Perforce

This information is a reprise of the Admin Guide along with some hard-won details and improvements.

1. Download the `daemon` utility. This utility allows `p4d` to be run by the `perforce` user instead of `root`. I use /usr/local/src for the source of anything I install on the server so here’s what I did. If you’re not logged in as root, put a `sudo` ahead of each of the commands below or `su’ in first.

cd /usr/local/src

wget http://libslack.org/daemon/download/daemon-0.6.3.tar.gz
tar zxvf daemon-0.6.3.tar.gz
cd daemon-0.6.3.tar.gz
./configure
make install


2. Download the PerForce daemon file `p4d` and client `p4` files directly into /usr/local/bin. These files are for Linux installs using the 2.6 kernel version, adjust if you’re still running a 2.4.x kernel:

cd /usr/local/bin
wget http://www.perforce.com/downloads/perforce/r07.3/bin.linux26x86/p4d
wget http://www.perforce.com/downloads/perforce/r07.3/bin.linux26x86/p4


3. Make the `p4d` and `p4` files executable.

chmod +x p4d p4


4. Create a group for perforce files and a user for admin work:

groupadd p4admin
useradd -m -g p4admin perforce


5. Use `visudo` to give the perforce user account the ability to use `sudo`. Add the following line at the end o f the file. NOTE: visudo on Red Hat doesn’t actually use `vi` to do the work, it uses pico so don’t expect `vi` commands to work. Stupid Red Hat. Stupid pico.

perforce ALL = ALL

6. Log off your default user account.
7. Log in using the `perforce` account.
8. Create a directory to hold the repository. I based mine in /var/www.

sudo mkdir /var/www/perforce
sudo chown perforce:p4admin /var/perforce


9. Create a directory to hold Perforce log files under /var/log.

sudo mkdir /var/log/perforce
sudo chown perforce:p4admin /var/log/perforce


A. Add the following lines to the end of /etc/profile. These settings will be used by local client programs run on the Linux server – not by the Perforce server.

# Perforce Settings
export P4JOURNAL=/var/log/perforce/journal
export P4LOG=/var/log/perforce/p4err
export P4PORT=localhost:1666
export P4ROOT=/var/www/perforce
export P4USER=perforce


B. Load the Perforce settings.

source /etc/profile


Setup Perforce As Bootup Service

1. Change to the initialization control directory.

cd /etc/init.d


2. Create the Perforce control script using `sudo vi perforce`.

#!/bin/sh -e

export P4JOURNAL=/var/log/perforce/journal
export P4LOG=/var/log/perforce/p4err
export P4ROOT=/perforce_depot
export P4PORT=1666

PATH=”/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin”

p4start=”p4d -d”
p4stop=”p4 admin stop”
p4user=perforce

case “$1″ in
start)

# NOTE: in original, haven’t found RedHat equiv or source to include
# Haven’t found . /lib/lsb/init-functions
# log_action_begin_msg “Starting Perforce Server”
daemon -u $p4user $p4start;
;;

stop)
# see above
# log_action_begin_msg “Stopping Perforce Server”
daemon -u $p4user $p4stop;
;;

restart)
daemon -u $p4user $p4stop
daemon -u $p4user $p4start
;;

*)
echo “Usage: /etc/init.d/perforce (start|stop|restart)”
exit 1
;;

esac

exit 0

3. Goto your home directory then use the control script to start the Perforce server.

cd ~
sudo /etc/init.d/perforce start


4. Make sure it actually started

ps aux | grep perforce

I get:
userx@wonky [~]# ps aux | grep perforce
perforce 3669 0.0 0.0 1736 536 ? Ss 20:27 0:00 daemon -u perforce p4d -d
perforce 3672 0.0 0.0 5076 1492 ? S 20:27 0:00 p4d
userx 3689 0.0 0.0 1584 492 pts/0 S+ 20:27 0:00 grep perforce
14. Create a Perforce group to limit resource usage using:

sudo p4 group developers

This, on Red Hat, opens `pico` where you’re editing a text file to set the following values. See the Perforce site for more information.

Group: developers
MaxResults: 50000
MaxScanRows: 250000
MaxLocktime: 30000
Timeout: 4320
Subgroups:
Users: developer


To which perforce responds:
userx@wonky [~]# sudo p4 group developers
Group developers created.

5. Use the control script to restart the Perforce server.

sudo /etc/init.d/perforce restart

Housekeeping

Before Perforce can be used by a team there are two housekeeping task that need to be done – creating the journal and closing a security hole.

If you’ve been following along, the p4d server should already be started and all of your environment variables should be set properly.

1. Create the journal.
sudo p4d –jc
userx@wonky [~]# p4d -jc
Checkpointing to checkpoint.1...
Rotating /var/log/perforce/journal to journal.0...

2. Open the firewall.
We use the `apf` firewall, so the only thing left to do is to open the firewall for our chosen port. Edit /etc/apf/conf.apf and modify EG_TCP_CPORTS and IG_TCP_CPORTS to add our chosen port:

# Common egress (outbound) TCP ports
EG_TCP_CPORTS=”…,1666“

# Common ingress (inbound) TCP ports
IG_TCP_CPORTS=”…,1666,…:”

Then restart the apf server, flushing and reloading firewall rules:

apf -rl --restart


3. NOTE: the original instructions suggest running p4 protect.
I’m not doing that since I don’t know what it does and I’m delegating further Perforce setup to others on my team.

4. Here’s my TODO list:

• Is this secure? Are passwords etc. sent in the clear?
• If not, is there a way to make it so?

5. According to the original blogger, these items remained on his “TODO” list. I leave them here in case I blog them later when I find out what they mean .
TODO

• Journalling
• checkpointing
• backups

Resources

• Perforce’s public depot
• http://kb.perforce.com/AdminTasks/InstallAndUpgrade/RunningAPerf..InetdOnUnix How to start `p4d` via `inetd` instead of a startup script.

Installing Perforce

One good resource, though since I’m not installing on the same Linux version, the instructions are coming up a bit short:

Installing PerForce on Ubuntu

I’m installing on RedHat and so, `sudo apt-get daemon` doesn’t work.

Since ‘daemon’ is such a common word in the Google lexicon, finding the app to which they were referring was a bitch.

Sorry, I don’t know the search that I used to find it — I was in search mode, not blogging mode.

Anyway, the utility is “daemon”.

It allows you to daemonize any process while specifying the user under which to run and various other things. Since p4d itself has no support for running as a specific user, and since they specifically recommend running p4d as an ‘unpriviliged’ user, something had to be done.

More tomorrow when I finish the setup.

Perforce — IDC, PAY for Version Control?!

Update: please see my Git rants to see where we’re at now.

We’re starting a new software project here at IDC and some of the developers we’ve hired have strongly recommended using use Perforce. They also have recommended ExtraView for bug tracking which integrates with Perforce.

While it’s not against my religion to use expensive, proprietary software for business critical functions, especially server type functions, and especially when pricing is per user, it’s pretty damn close.

We’ve been using Subversion around here for a few years and, frankly, other than for storing revisions, it pretty much sucks.

The way it handles conflicts is a complete pain in the ass that always leaves me thinking “there’s got to be a better way”, pulling out older versions always sends me back to the manual, the monolithic numbering scheme has always given me agita, and God help you if you’ve got to merge a branch (or branches!) back to the trunk…the list goes on and on.

So, much as it pains me to face an $800/user licensing fee, the 2 user, 5 client workspaces limitation on the free, unlicensed version of Perforce has sucked me in.

ExtraView also has a limited, hosted version that will allow us to do this project and see how things go.

Who knows? We may actually have to pay for these things if they work well enough.

Next entry — my trials and tribulations getting it set up on one of our public-facing servers.