Monthly Archives: September 2010

Scripts for Gibsons: TimePropagate

This boat's in front of that boat and my boat's the last boat.

As the privileged few users of my FTP server can attest, I like to keep it tidy. Of course, in administrating a 5TB fileserver, scripts emerge to help keep things organized. Below is one such script that a least a couple of you readers could use. I made it and it’s no masterpiece, but it is LGPL, so this is my first potentially actually useful foray into open-source. What was originally an 8-line bash script has grown into a couple hundred of PHP (don’t you dare judge me) for readability and maintainability.

Read the comment at the top of the code to understand what timepropagate actually does. Usage is also in the script.

Sample output

As run on my ftp (with extra stats enabled)

Processed 69780 files (9 links) and 5439 directories, 2 out of date (0 touchable).
Maximum directory depth seen was 8.
The newest modified date seen was September 12, 2010, 8:55 pm.
Extension counts:
        mp3 - 45882
        avi - 6872
        wav - 3636
        jpg - 3521
        m4a - 2373
        aif - 1054
        mkv - 646
        mp4 - 428
        txt - 309
        bmp - 270
        flac - 264
        zip - 239
        mpg - 230
        gif - 224
        exe - 220
        amd - 185
        ogm - 185
        pdf - 180
        slp - 126
        srt - 124

Code

Download TimePropagate here.

#!/usr/bin/php
<?php

/*
 * TimePropagate 
 * An uncreatively-named script to breathe life into directory modified times
 *
 * As users of my ftp server might know, most linux filesystems that I know of 
 * don't propagate the modified-by date of directories. So, if I have a folder
 * called 30 Rock, with subdirectories for each season, adding a new episode
 * to "Season 5" isn't going to produce the desired results when viewing the
 * full listing of TV Series sorted by modified times.
 * 
 * Long story short, sorting by modified date is more useful when you run this
 * command, usually. If you can come up with some scenarios where the default
 * behavior is actually useful (and not just a performance bonus), let me know.
 *
 * In the meantime, run it with the -pv flags in the directory of your choice
 * to see how it works for you.
 *
 * All this is licensed LGPL.
 */

 // for easily-glanced-at log lines
 define('GLANCE_OK', 'OK');
 define('GLANCE_NOTE', '**');
 define('GLANCE_ERROR', '!!');
 define('GLANCE_TITLE_WIDTH', '19');
 
$options = parse_arguments($argv);
$mdc = new ModifiedDateController($options);

if($mdc->help) { // if user asks for help/usage
	print_usage($argv[0]);
	exit();
}

$continue = true;
if(!$mdc->force) { // if not force
	$continue = ask($mdc->target_dir);
}

if(!$continue) {
	echo "You and me are done.\n";
} else {
	$mdc->process();
}


function print_usage($command) {
	echo "\nModified Time Propagate v1 - GPL 2010 Mahmoud Hashemi\n\n";
	echo "Usage: ", $command, " [-h] [-vpsfLN] [-m ] [-d ]\n\n";
	
	echo " -h \t- Print this message and exit.\n";
	echo " -v \t- Enable verbose output.\n";
	echo " -p \t- Pretend (does not write any dates; use this with verbose).\n";
	echo " -s \t- Gather and show some miscellaneous statistics.\n";
	echo " -f \t- Suppress confirmation.\n";
	echo " -L \t- Follow links (careful, linked directories can greatly affect results).\n";
	echo " -N \t- Suppress recursion; only do the current directory.\n";
	echo " -m #\t- Max depth to recurse (default: 30).\n";
	echo " -d dir\t- Target directory (default: current directory).\n";
}

function ask($target_dir) {
	$stdin = $fh = fopen('php://stdin', 'r');

	echo 'Propagating file modification times for ', $target_dir,"\n";
	echo "Continue? (y/n)  ";

	$response = '';
	while(trim($response) === '') {
		$response = trim(strtolower(fgets($stdin, 8)));
	}
	if($response === null || ($response != 'y' && $response != 'yes')) {
		return false;
	} else {
		return true;
	}
}

class ModifiedDateController {
	public $current_uid = '';
	
	public $help = false;
	public $target_dir = '';
	public $follow = false;
	public $no_recurse = false;
	public $verbose = false;
	public $more_stats = false;
	public $pretend = false;
	public $force = false;
	public $max_depth = 30;
	
	public $files_seen = 0;
	public $links_seen = 0;
	public $folders_seen = 0;
	public $touchable_folders_seen = 0;
	public $max_depth_seen = 0;
	public $folders_updated = 0;
	public $extensions = array();
	
	function __construct($options=null) {
		if($options !== null) {
			if($options['h']) {
				$this->help = true;
			}
			if($options['L']) {
				$this->follow = true;
			}
			if($options['v']) {
				$this->verbose = true;
			}
			if($options['p']) {
				$this->pretend = true;
			}
			if($options['f']) {
				$this->force = true;
			}
			if(isset($options['D']) && $options['D'] !== '') {
				$this->max_depth = intval($options['D']);
				
				if($this->max_depth no_recurse = true;
				$this->max_depth = 0;
			}
			if($options['s']) {
				$this->more_stats = true;
			}
			if(isset($options['d']) && $options['d'] !== '') {
				$this->target_dir = $options['d'];
				
				if(!is_dir($this->target_dir)) {
					echo "Specify a valid target directory (-d).\n";
					exit(2);
				}
			} else {
				$this->target_dir = getcwd();
			}
		}
	}
	
	function process() {
		$this->current_uid = getmyuid();
		$newest_mtime = $this->r_filemtime($this->target_dir);
		echo "\n";
		echo "Processed ",$this->files_seen," files (", $this->links_seen," links";
		if($this->misc_files_seen > 0) {
			echo ", ",$this->misc_files_seen, " miscellaneous";
		}
		echo ") ";
		echo "and ",$this->folders_seen," directories, ",$this->folders_updated," out of date (";
		echo $this->touchable_folders_seen, " touchable).\n";
		echo "Maximum directory depth seen was ", $this->max_depth_seen,".\n";
		echo "The newest modified date seen was ",date("F j, Y, g:i a", $newest_mtime),".\n";
		if($this->more_stats) {
			arsort($this->extensions);
			echo "Extension counts:\n";
			foreach($this->extensions as $ext=>$count) {
				echo "\t",$ext," - ",$count,"\n";
			}
		}
	}
	
	// call this on a directory path.
	function r_filemtime($path, $depth=0) {
		if($depth > $this->max_depth) {
			if($this->verbose) {
				echo get_glanceline("Depth limit (".$this->max_depth.")",GLANCE_NOTE,$path);
			}
			return FALSE;
		}
		if(!file_exists($path)) {
			echo get_glanceline('No such path', GLANCE_ERROR, $path);
			return FALSE;
		}
		
		$path_mtime = filemtime($path);
		if($path_mtime === FALSE) {
			echo get_glanceline('Access denied', GLANCE_ERROR, $path);
			return FALSE;
		}
		++$this->folders_seen;
		if($depth > $this->max_depth_seen) {
			$this->max_depth_seen = $depth;
		}
		
		$max_mtime = 0;
		
		foreach(glob($path."/*") as $file) {
			$filetype = filetype($file);
			if($filetype === 'link') {
				++$this->links_seen;
				if($this->follow) {
					$link_target = realpath($file);
					if($link_target == FALSE) {
						echo get_glanceline("Dead link", GLANCE_ERROR, $link_target); 
						continue;
					} else {
						$file = $link_target;
						$filetype = filetype($link_target);
					}
				} else { //if we're not following links
					continue;
				}
			}
			
			if($filetype === 'dir') {
				$cur_mtime = $this->r_filemtime($file, $depth+1);
			} else if($filetype === 'file') {
				++$this->files_seen;
				if($this->more_stats) {
					 $ext = strtolower(end(explode('.',end(explode('/', $file)))));
					 if($ext === '' || strlen($ext) > 5) { //arbitrary 5 char ext limit
						continue;
					 } else if(isset($this->extensions[$ext])) {
						++$this->extensions[$ext];
					 } else {
						$this->extensions[$ext] = 1;
					 }
				}
				$cur_mtime = filemtime($file);
			} else {
				++$this->files_seen;
				++$this->misc_files_seen;
			}
			if ($cur_mtime > $max_mtime) {  
				$max_mtime = $cur_mtime;   
			}
		}
		if($max_mtime > $path_mtime) {
			$touchable = (fileowner($path) == $this->current_uid);
			if($touchable) { 
				++$this->touchable_folders_seen;
			}
			if(!$this->pretend) {
				if($touchable) {
					touch($path, $max_mtime);
				} else {
					echo get_glanceline("Can't touch this",GLANCE_ERROR,$path);
					++$this->touch_errors;
				}
			}
			++$this->folders_updated;
			if($this->verbose) {
				echo get_glanceline(date("M d, Y, H:i", $max_mtime),GLANCE_NOTE,$path);
			}
		} else {
			$max_mtime = $path_mtime;
			if($this->verbose) {
				echo get_glanceline(date("M d, Y, H:i", $max_mtime),GLANCE_OK,$path);
			}
		}
		
		return $max_mtime;
	}
}

function get_glanceline($title, $blip, $detail) {
	$title_width = GLANCE_TITLE_WIDTH;
	$title = str_pad($title, $title_width, ' ', STR_PAD_BOTH);
	return $title.' - [ '.$blip.' ] - '.$detail."\n";
}

function parse_arguments($argv) { 
    $arguments = array(); 
	array_shift($argv);
	$args = explode('-', implode(' ', $argv));
	foreach($args as $arg) {
		$arg = trim($arg);
		if($arg == '')
			continue;
		$arg_pair = explode(' ',$arg);
		if($arg_pair[1] == '') {
			$argarray = preg_split('//', $arg_pair[0], -1, PREG_SPLIT_NO_EMPTY);
			foreach($argarray as $char) {
				$arguments[$char] = true;
			}
		} else
			$arguments[trim($arg_pair[0])] = trim($arg_pair[1]);
	}
    return $arguments; 
} 

Some Words and Dedications

There you have it. Feel free to suggest features or make modifications or give me props. Also, thanks to Alan for offering some guidance/sane defaults pre-release, and Ben for getting me Chinese food as I was polishing this off.