Linux

Open Source Bridge Proposal Deadline Coming

Submitted by justin on

Look, I know this isn't our normal type of post.  No code snippets, no real world examples, etc.  Barely even a trace of wit and/or style.

Ok, maybe the lack of style thing really is in keeping with our normal posts.  Whatever.

Point is that this is Important Stuff and, therefore, you need to know:

The deadline for submitting proposals for Open Source Bridge is fast approaching, as in you-need-to-get-yours-in-in-under-20-days-as-of-this-writing fast.  Or, more specifically, by March 31.

So what, exactly, is Open Source Bridge?  Well, my under-rock dwelling friend, it is:

 

Open Source Bridge is a new conference for developers working with open source technologies. It will take place June 17-19 in Portland, OR, with five tracks connecting people across projects, languages, and backgrounds to explore how we do our work, and why we participate in open source. The conference structure is designed to provide developers with an opportunity to learn from people they might not connect with at other events.

 

Them's their words, of course.  Me?  I like to think of it as The OSCON Replacement That Will No Doubt Be Better Than Its Predecessor, So San Jose Can Just Keep That Bloody Thing Because We Don't Need Them In Portland, Anyway, Conference. 

Granted, "Open Source Bridge" rolls off the tongue a little easier.

So go to the Open Source Bridge site.  Submit a proposal!  Volunteer!  Sponsor!  

And if you can't do any of those things, just make sure to register and get your bums to Portland in JUNE. 

 

References: 
Open Source Bridge
Silicon Florist - OSCON? Gone

SQL:

OSs:

Miscellaneous:

Upgrading from Drupal 5.x to Acquia Drupal 1.0.3-ISR

Submitted by justin on

My God, we've been behind the 8 ball.

Seriously, did you realize that there have been something like 1.8 million Drupal releases since we last upgraded? Here we are sitting on Drupal 5.2 when the rest of the world is runnin' 6.6. Clearly, time to upgrade.

This afternoon, we decided to jump from Drupal 5.2 to Acquia Drupal, and this here article is the first of two that will document the process.

In this post, we're just talking about the actual Drupal upgrade process. No coding, no theming. Just updating Drupal to a version created in the last, oh, decade or so. Later, Jeremy will tackle updating your 5.x themes to work with 6.x.

Ok, so first thing's first. Before you do squat, there are a couple of files you'll want read.

First, check out the UPGRADE.txt file in your root Drupal directory. That bad boy is the Bible of your upgrade process, and you should have a pretty good idea of what's in there before you get going.

Likewise, if you're installing the Acquia flavor, get familiar with the
Acquia Getting Started with Acquia Drupal doc.

There are steps in there that aren't covered by a normal Drupal upgrade and, while this here post will talk about some of them, your own installation and upgrade experience could be drastically different. This post is just a supplement and, frankly, might not apply to certain installations.

We're also going a slightly different route in our migration. For instance, we're skipping the "Place the site in 'Off-line' mode" step because we're doing a copy-upgrade-rename thing rather than upgrading the live version of the site in place. That works for us, since we're a low traffic site and our comments are few and far between. You popular kids (God, I hate you guys...) might be better served putting the live site in Off-Line mode anyway, just so you don't lose any comments made while you're fiddling around with the copy.

Oh, and before you even start, make sure that there are 6.x versions of any modules you're using before you get too far into this. Wouldn't it suck to update your page only to find out that your favorite, my-site-depends-on-it module doesn't have a 6.x flavor? Yeah. It would.

So, without further ado, let's get rollin'.

Like I mentioned, we aren't going to just update our site in place. Frankly, the idea scares me. Instead, we're going to make a copy of our site and database and update that. After we've finished, we'll swap the copied-and-updated site for the live one, then crack open a cold one to toast our success.

And here's how we do it:

  1. Make a copy of current Drupal install directory.

    cp -R <drupal directory> <copy directory>

2.   Make a copy of the database for the upgrade.
      Easy as pie:

  • Create a new database using whatever means you normally would.  My provider has a little control panel tool I use.  Maybe you do it from the command line, maybe you use phpMyAdmin.  Whatever.  Just create a blank database.
  • Create a new MySQL user and give it the necessary rights to the new database.  Or grant those priveleges to your old user.  Whatever makes you happy (I went with Option B, just because I like to minimize the number of users I have to manage.  Up to you.) 
  • Export the data from your live database.  I used phpMyAdmin for this.  Just logon and click the Export button.  Select your database and choose the SQL format.  I left all the other options at their default values.
  • Modify the export file to point to the new database.  In my case, I changed the "USE" statement on line 22 of my dump to point to the new file.  I also deleted the "CREATE DATABASE" line because my host makes me use their aforementioned tool for the process.  Your mileage may vary.
  • Import the data file.  With phpMyAdmin, you just click the Import tab, browse to the file you want to import (the one you just exported/modified) and click Go.  Done.

        That's it.  You should now have a copy of your DB.  Of course, there are other methods.  Feel free to share your fav in the comments.
     

3. Update the settings.php file in the new directory to point to the new database.

IMPORTANT.  If you don't go to your settings.php file (/sites/default/settings.php) and edit the $db_url line, any changes you make from this point forward will happen on your live database.  That could be very, very bad if things blow up on you.  Edit that $db_url line to point to the new database name you created in the last step.  If you created a new user, change that, too.
    
4.  Confirm that you're using the new database, etc.  

Ok, I'm anal about this kind of thing.  I want to be 100% certain I'm about to jack with my copy rather than the real site.  In my case, I created a new subdomain that points to the new directory, so when I browse to "http://nerdtest.nerdliness.com," Apache serves up the files in my ./nerdtest directory instead of my original ./nerdliness one.  Make sense?

How you do this will depend on your host.  In my case, I get a handy little control panel with a "create subdomain" link that takes care of the Apache and DNS changes.  If you're hosting yourself, you might have to make your own httpd.conf and DNS record changes.  Or if you have another host, you'll need to check with them.  Sorry, but there are as many different steps for this part as there are hosting providers. 

Anyway.
    
After creating the subdomain, browse to your site copy and make a small change.  Maybe leave yourself a comment or make a test post.  Whatever you do, make sure it only appears on the subdomain and not on the original.

5.  Get your modules in order.
Ok, we've been bad.  We had a ton of modules chillin' in our /modules directory instead of in the /sites/all/modules 'hood where they belong.  The longest part of this process involved moving those user contributed, third party modules into their correct location.

The Acquia guide has a good list of core Drupal modules that need to stay in the /modules directory, but it isn't all-inclusive.  Before you start moving all the modules from /modules to /sites/all/modules, make sure you're moving the right ones.  Check out the docs online for the full list.  In my case, I just had one window opened to my Drupal Admin page and made sure I didn't touch any modules that were listed under the Core - Required or Core - Optional headings.

Make sure you check out your site after making all thes moves, just to be safe.  Would hate to try to update after you've already moved out some core module that you shouldn't have touched only to find that nothing works right.

6.  More module maintenance...
Acquia includes fancy-pants versions of some important modules, so you need to 86 any versions you have installed already before you do The Upgrade.  Make sure you deactive and remove them before you continue.  There's a full list of modules in the Acquia Getting Started guide, starting on Page 24, and it includes stuff you're almost certainly using (CCK, anyone? How 'bout Views?)

Note:  there's a special case around the Printer-Friendly Pages and Filefield Meta modules. Didn't apply to us, might apply to you.  Make sure you RTFAcquiaGettingStartedGuide, page 25 ("Special Cases:  Printer-Friendly Pages Module and Filefield Meta Module").

7.  Download/unpack the Acquia Drupal goods.
Browse to http://acquia.com/downloads and download the current "Update Existing" version.  Copy that bad boy over to your web server and untar it.

8.  Update!
Now the real fun starts.

First, since we downloaded the "Update Existing" version of Acquia, the directory created when we untarred that file is missing some key elements from our actual site.  We need to copy those over before we do anything else.  Basically, we need to copy our .htaccess and robots.txt files, our entire sites directory (and its contents), as well as any other customized files that live outside those folders.

    cp <copy directory>/.htaccess <acquia directory>/
    cp <copy directory>/robots.txt <acquia directory>/
    cp -R <copy directory>/sites <acquia directory>/

9.  Rename your directories
Now that our customized files are in the Acquia untar directory, we just need to rename our copy directory to something new, then rename the Acquia directory to the same name our copy directory used to have. 

Good God, that doesn't make much sense, does it?

Ok, so say we untarred the Acquia files to a directory called "acquia," and our old working Drupal directory was "nerdtest."  Now, we're going to rename "nerdtest" something like "old_nerdtest," then rename "acquia" to "nerdtest."  Got it?
    
    mv <copy directory> <new name for copy directory>
    mv <acquia directory> <copy directory>

10.  Run the update.php file
Just browse to the /update.php file on your site and do what it says.

11.  Pray.

12.  Check for errors
You should get a progress report immediately after the update that mentions any specific errors.  Also check the Status Report page (/admin/reports/status).  Make sure you follow any advice given on this update results page.

In our case, we had a couple issues.  First, the Status Report mentioned that the "files" directory didn't exist.  Sure enough, we'd forgotten to copy that over from our previous install back in step 8.  Easily fixed with a little "cp -R <new name for copy directory>/files <copy directory>" action.

Second, we noticed this blurb:

Updates for CCK-related modules are not run until the modules are enabled on the administer modules page. When you enable them, you'll need to return to update.php and run the remaining updates.

Piece of cake.  All we had to do at that point was go into the Administer Modules page, enable those CCK modules, then rerun the /update.php script.  No problems.

13.  Update and reactivate modules
You'll probably find that several modules you used with 5.x have big red X's next to them in the Admin page now.  Yeah.  Sucks, huh?  Probably should have checked for newer versions of those modules ahead of time, right?  Like we warned you about before you even started? 

Anyway, now's the time to download and install those newer versions.  You know how.  And this time, remember to put them in /sites/all/modules.

15.  Bring the site "On-Line"
We didn't put ours in Off-Line mode, but if YOU did, make sure you bring it back up.

16.  Test.
Browse around, check everything out.  If it all looks good, fantastic!  If not, well...  Can't really help you there.  Our's went just fine.  :)

Ok, maybe "fine" is a bit too...  um... Pollyanna of a word. We had some issues with some permissions (anonymous users couldn't see content for some reason) and formatting (former header/footer blocks ended up in the sidebar), etc.  But all of those were things that were pretty easy fixes.  Nothing major.

If YOU have something major come up though, leave comments.  We (or someone reading this) might be able to help.

17.  Go live!
If everything looks good, it's time to bring it live. 

Remember that we did this with a copy of the site, so we need to change a couple directory names.  It's really the same exact process as in step 9, just with different directories:

    mv <drupal directory> <some other name>
    mv <copy directory> <drupal directory>
 

Also, if you put yours in Maintenance Mode, remember to disable that.

And that's pretty much it.  Really not too difficult.  I'm sure we'll continue to find little pieces here and there that need some massaging every now and then, but overall the process was relatively painless.

Now if we could just get Jeremy to update our old theme...  Not like he's got a full time job and a newborn daughter keeping him busy or anything...

SQL:

OSs:

Miscellaneous:

Creating Simple Ubuntu/Debian Packages

Submitted by luke on
Creating static content Debian package

Occasionally there comes a time when you need to create your own Debian packages. Whether it be for deployment or needing to customize an install to your needs or whatever other reason you may have.

Personally, in my work, I deploy configurations of various sorts to hundreds of Ubuntu systems and have found that automating the creation of debian packages to be very useful. It allows me to track revisions, easily install, or remove the packages that I have made with scripting hooks that are all part of the package. In this example, we are going to construct a static debian package. This packages only purpose will be to copy files to the OS, and make any permission changes needed. It will also show a simple example of pre-install and post-install scripting and how/where those scripts go.

First, let's disect a simple debian package. In this example, the package we are going to make will be called 'mypackage'. We'll have a parent directory named 'mypackage' Inside the 'mypackage' directory will be a sub-directory called 'DEBIAN' and also contain a simple OS directory replica of where you want to place the file or files on the OS.

mypackage/
----------/DEBIAN/
----------/etc/mycoolfile

In the above example, we are placing 'mycoolfile' in the /etc directory. The real magic in all of this lies in what is actually in the DEBIAN dir.

mypackage/DEBIAN/
----------------control
----------------postrm
----------------preinst
----------------postinst
----------------prerm

 

Inside mypackage/DEBIAN/ you'll place a file describing your package called 'control', and any scripts that you want to be ran at various points during the installation and/or removal of your package. Here is an example 'control' file

Package: mypackage
Version: 1
Section: base
Priority: optional
Architecture: all
Depends: bash
Maintainer: luke@nerdliness.com
Description: My super cool package

 

Here is a short description of the other files you may find in the mypackage/DEBIAN/ directory.

postrm = postremove
preinst = preinstall postinst = postinstall prerm = preremove

All the scripts are run at the named times, meaning 'preinstall' is run before the contents of your package are installed. Here is an example 'postinst' script. Keep in mind that these could just be very simple shell scripts.

#!/bin/sh

ln -s /etc/mycoolfile /home/user/

If for whatever reason you want your file to be linked into your home directory putting the above into the mypackage/DEBIAN/postinst file would do it. I would definitely experiment and play around with exactly what you can do with these scripts. One thing to keep in mind, you aren't just limited to shell scripting in these scripts. Personally, I've used ruby as well.

Now, the only thing left is to build our package. We're going to use the 'dpkg' command to do this. From the parent directory that your 'mypackage' folder is in run

dpkg --build mypackage ./

Command Breakdown: dpkg is the debian package manager. In this case we are asking it to build a package, hence the --build parameter. The name of our package is 'mypackage' from that name it will also look for a folder in the parent directory called 'mypackage' and decend into that. It will then look for the DEBIAN directory, and parse the 'control' file for the various options. NOTE: if you didn't create the control file in the right fashion, you will receive errors letting you know this. We are also using ./ to tell dpkg too build the package in the present directory.

If all went well with the above command, you should end up with a freshly created package in your working directory called 'mypackage_1_all.deb' and it should be ready to install. This is a very easy process to automate if your content changes. Happy packaging!

References: 

OSs:

Coding:

SSH with no password

Submitted by luke on
Logging into *nix based OS's with no password prompt

If like me, you have a lot of *nix boxes that you login to from one machine, typing your password over and over can be a pain. Also, if like me, you automate jobs on remote machines, SSH can be a great way to go. However you probably don't want to hang out until 1AM when no one is around to type in the password to execute that job via SSH. It is quite simple to have SSH automatically login for you and run your job, or just log you in so you can work at the console.

I've used this on various *nix platforms and OSX with great success.

The first thing we need to do is setup some keys to use. We're going to generate the public and private key pair for SSH to use on your local machine or the machine you want to initiate the SSH session from.

ssh-keygen -b 1024 -t dsa

Command breakdown: ssh-keygen is the command we are using. The "-b" option specifies the number of bits used in the key. The "-t" option specifies the type of key pair that we are going to create. In this example we are using DSA.

NOTE: There are a variety of different types of key pairs that we could use and there is quite a discussion we could get into about this, but that definitely goes above and beyond the scope of this how-to.

When you run the above command you will receive the following promps:

Generating public/private dsa key pair.
Enter file in which to save the key (/home/username/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:

Pressing "enter" at all the above prompts will work fine. One thing to keep in mind would be that if you do enter a passphrase you will be required to enter that passphrase before you can use your key.

After our keys are generated, use your preferred method of transferring your public key (in this example the public key is called id_dsa.pub and is kept in /home/user/.ssh/) to the remote machine. Rsync, scp, whatever you prefer...

Once the key exists on the remote machine login there and get to a command prompt. We are now going to copy the public key from the orinal machine into a place that lets this machine know that it is an accepted host. Run the following command to do so.

cat id_dsa.pub >> /home/user/.ssh/authorized_keys


Command breakdown: Cat, probably no explanation needed there... In this case, the user needs to be the user you are going to login as and we are appending the contents to the authorized_keys file.

As a final step let's make sure the permissions on this users .ssh directory are correct.

chmod 700 ~/.ssh
chmod 600 ~/.ssh/*


That is it! You should be able to login to this machine from the original with no password prompt and you should also be able to run commands on this machine from the origina via SSH with no password needed.

References: 

OSs:

Coding:

SSH Port Forwarding

Submitted by luke on
SSH Tunnels made easy

SSH is a wonderful tool. Not only does it allow secure remote access to your *nix box console, but it can also allow secure access to any of the services that box has to offer remotely even if the machine you are connecting from is behind a firewall.

First, The basics. Forwarding a port:

ssh -l username -L 9090:localhost:3306 remotemachine

The above command will forward port 3306 (MySQL) on your remote machine to local machine on port 9090 (or whatever random unprivileged port that you choose). You would then be able to login the remote mysql server from your local machine all over SSH.

The command break down is simple. Replace "username" with the username you are going to login to the remote machine with.

The "-L" option specifies that you are forwarding a Local port. In the above example, you are forwarding the port to port 9090 on the localhost.

The machine you are connecting to is "remotemachine" This can be an IP address, or a resolvable domain name.

The above example will open an SSH session on the remote machine and it will stay open until you close it or logout. This can easily be changes to forward your smtp, http, or any other port you choose to over SSH.

This is all great, however scripting an interaction with that remote machine isn't overly pleasant with that SSH session hanging around waiting to be closed by you. In that case, let's have it connect, run our command, then close without us telling it to.

In the example below, we going to update a Debian/Ubuntu based machine over the internet. However the server we are getting our updates from only accepts SSH traffic publicly and all other ports are blocked by a firewall. Debian/Ubuntu repositories are usually served by Apache or another webserver so they tend to answer on port 80 only.

NOTE: I know this sounds like a nutty example, but I have actually been asked this before. Also it provides easy ghetto authentication to your package repository.

ssh -l username -f -L 9090:localhost:80 remotemachine sleep 5
apt-get update
apt-get upgrade
exit 0

The above command/script is broken down like this:
We are connecting to the remote machine as "username."

Instead of having that pesky prompt stay open, we are going to fork this session into the background with "-f". From my experience, this has been the simplest way to get these tunnels to die once your command has completed.

The remainder of the command is the same as above except for the "sleep 5". The Sleep command is being run on the remote machine. This allows our local machine the ability to run the various apt commands in the remainder of the script that require it to interact with the remote machines.

Now for these particular commands to work, you'll need to setup your apt.sources file to update from your local machine instead of the remote machine. This is just an example, and that's another post for another time. Happy tunneling!

References: 

OSs:

Coding:

Subscribe to RSS - Linux