Have you ever needed to copy or move a ton of files from one website to another? Have you ever run across the issue where only 1998 or 7998 files can be viewed in a directory even though you know there’s more in that folder?
We ran across both issues this week. The issue stemmed from when the hosting company for one of our esteemed clients moved our stage server to the new live server location. When they did so, they missed copying over a lot of the images for the shopping cart. When we contacted customer support about this, they indicated that they did. From a cursory look at the folder in WinSCP and FileZilla, it appeared that way since the stage site and live site were both reporting the same 7998 files. However, on the stage site, we were able to access many more images by simply browsing the site. So the files were there but just weren’t being reported to FTP.
What could cause this? As it turns out, there is an apache setting that limits how many files that can be listed. The FTP client trusts this number and so even copying an entire folder of images will only copy the first 7998! We contacted the hosting company and they were able to increase this number to 39998 files but that still wasn’t enough.
I contacted the hosting company a few times (4 times!) to request moving these files for us (since they missed them in the first go-round after all!). After bungling my request a few times, they finally told me that it would be 4-5 days. Mind you, this is for an eCommerce site so not showing the product images is somewhat short of acceptable. So of course, we needed a quicker solution so I was able to transfer all 260,000 images (2.4GB) within about 14 hours. Here’s what I did:
- I enabled SSH/Shell access through the control panel on both hosting accounts.
- I then generated public/private keys through the CP. They had an option to generate a ppk but this option didn’t work in their CP so I had to generate a new private key .ppk through PuttyGen.exe.
- Using Putty.exe, I then connected to the server through SSH to get the bash screen.
- From here, I navigated to the parent folder for these 260,000 images and compressed the whole folder in tar format, with this command: tar –zcf file.tar foldername/
This took about an hour to compress. - I then downloaded this 2.4GB file to my computer and uploaded it to the other website hosting. This took about 7 hours just pure file transfer time.
- After uploading the tar file to the parent folder of the new hosting, I just extracted the archive with the following command: tar -zcf file.tar.gz foldername/
Hopefully this quick guide will make it a little easier for you and you can avoid some of the frustration! Happy coding!
