Its probably better to just use $this->setBase(''); in the template you're using, (i.e. /templates/]/index.php, somewhere close to the top of the file) This is safer, because if you upgrade Joomla, /libraries/joomla/document/html/renderer/head.php will be overridden, but your template wont be.

I had problems getting the images display correctly in IE6 (argh ), this was because joomla always used the actual path as href parameter for the base-tag. So if you want to remove the automatic generation and add it manually in your template, do the following:

Open  <YourJoomlaInstallation>/libraries/joomla/document/html/renderer/head.php in a text-editor of your choice.
Around line 64 you'll find

// Generate base tag (need to happen first)
$base = $document->getBase();
if(!empty($base)) {
$strHtml .= $tab.''.$lnEnd;
}

Change it to(comment it out):

// Generate base tag (need to happen first)
//    $base = $document->getBase();
//    if(!empty($base)) {
//        $strHtml .= $tab.'<base href="'.$document->getBase().'" />'.$lnEnd;
//    }

Now you're ready to edit your template and add the base-tag manually.