(Re)Structuring Upload Directories with PHP

Posted on April 18, 2015

Years ago, a younger version of myself made the silly decision to store user-created images in a simple two-level directory (/cards/thumbs & /cards/full). Thousands up uploads later, I had a major problem on my hands. Or 142,000 problems, to be more specific. I needed these files organized, both for readability and easier future cleanup (expired files, storage limitations). Here is how I made the move.

My site expires content on a monthly/yearly basis, so I decided to use a YEAR/MONTH/DAY/ folder structure moving forward. All of the current uploads had time-stamps associated with their matching database records, so I had the data necessary to make the move.

First, I needed something to move the files with relation to the stored data. I created a FileMover class which took a result set from my database to loop over the results, either moving or failing each item. I began by verifying both old files exist (main and thumbnail). Next, I created the new date path from the item's create_date, and while doing so create the directories necessary to store the file in the correct location. Finally, I rename each file path to the new location, verifying no errors occurred in the move.

Helper methods:

private function getPathFromDate($prefix, $date){
        $date_info = date_parse($date);
        $year = $date_info['year'];
        $month = $date_info['month'];
        $day = $date_info['day'];

        $this->check_dir($prefix, $year);
        $this->check_dir($prefix.$year."/", $month);
        $this->check_dir($prefix.$year."/".$month."/", $day);
        return $prefix.$year."/".$month."/".$day."/";
}
private function check_dir($prefix, $folder){
    if(!file_exists($prefix.$folder)){
        $old_umask = umask(0);
        mkdir($prefix.$folder,0775);
        umask($old_umask);
    }
}
private function moveFile($prev_loc, $new_loc){
    if(file_exists($prev_loc)){
        return rename($prev_loc, $new_loc);
    }
    return false;
}

Main Method:

public function move($list){
    $failed = array();
    foreach($list as $card){
        $image = $card->image;
        $ext = ".png";
        $old_thumb_file = ROOT."/view/cards/thumbs/".$image.$ext;
        $old_full_file = ROOT."/view/cards/".$image.$ext;

        if(file_exists($old_thumb_file) && file_exists($old_full_file)){
            $thumb_path = $this->getPathFromDate(ROOT."/view/images/thumbs/", $card->create_date);
            $full_path = $this->getPathFromDate(ROOT."/view/images/full/", $card->create_date);
            $thumb_file = $thumb_path.$image.$ext;
            $full_file = $full_path.$image.$ext;

            //move the file
            $success=true;
            if(!$this->moveFile($old_thumb_file, $thumb_file)){
                $success = false;
            }
            if(!$this->moveFile($old_full_file, $full_file)){
                $success = false;
            }
        }else{
            $success = false;
        }

        if(!$success){
            $failed[] = $card->id;
        }
    }
    return $failed;
}

This worked pretty well, I ended up moving 138,000 files, with a total 32 missing/incomplete file pairs. (Turns out I have a few thousand "dead" files just sitting in the directory without a database entry - this bug was fixed during this project as well!) During migration I was concerned with server load and memory usage, so I broke up the file list by year, and then months for more recent (heavier) images. Also, the helper methods above were reused to properly look up directories for images and save new images to the correct date-based locations (in other classes).

I hope this helps anyone else who finds themselves in a similar bind, or at least gets you started on options for migration. Cheers!