My first big 2.0 gallery

[devenh]

I've had 2.0 running for over a month now having up upgraded from v1.5. My galleries consist of over 30,000 images, the vast majority of these having been uploaded in v1.5.

This week I uploaded my first large EE 2 gallery of over 600 images. The process went smoothly but slowly. In most cases EE was not able to upload batches of 100+ images without an interruption that would require manual intervention. The recovery from interruptions was handled very gracefully by EE -- EE would just pick up where it left off -- but you did have to keep an eye on it.

My main concern is speed. EE v1.5 was much faster because it did not refresh the browser as *each* images was uploaded. I think the design if v2 is great if you're uploading 10 images and you want to check each one, but this approach is a real impediment when uploading hundreds of photos.

So anything that would streamline uploading large numbers of photos would be appreciated.

Deven

[mlucek]

Do you have a gallery link we can check out ?

[devenh]

This is the link customers use:

http://www.biglens.com/skiphotos.htm#SBST%20Intersquad%201

Here is a link directly into the gallery:

http://www.biglens.com/gallery/list.php?exhibition=211

Deven

[chriz]

I notice the same slowness regarding uploading images.
It must be possible to change a timer variable ?
Or maybe add the previous uploadscript so we can choose?
In 1.5 I was able to upload a few hundred pictures per batch without timeout problems very quickly.

Thanks,
Chriz

[devenh]

I've now added two more LARGE galleries, one with almost 900 photos and the other with over 600. So the combined number of photos was about 1500. Here are links:

http://www.biglens.com/gallery/list.php?exhibition=214
http://www.biglens.com/gallery/list.php?exhibition=215

Let me first say that I uploaded the photos from a location where the internet connection is flakey -- sometimes it works great and at other times it crawls or stop completely.

FTP'ing the files to my server went more smoothly and faster than uploading the files to EE. The 900 photo upload to EE took nearly 4 hours. If it was 4 hours where you could start the process and ignore it, that would be one thing, but you have to keep an eye on it becuase it frequently needs restarting. Some of the restarting was due to my internet connection, but some due to EE.

Before I went to bed last night I started the 600 photo upload. I woke up and EE stopped after uploading about 100 photos. I restarted the process and the remaining 500 photos were processed in 1.5 hours and required one restart. This was the fastest I've seen EE go and I had a good internet connection during this period.

If we assume EE can upload 300 photos an hour, a 600 photo gallery will take two hours to process and the process needs to be monitored. This is much, much longer than it took with EE 1.5.

Deven

[chriz]

Is this maybe a solution for slow uploads?
http://photography-on-the.net/forum/showthread.php?t=223332

Chriz

[Pekka]

Let me first say that I uploaded the photos from a location where the internet connection is flakey -- sometimes it works great and at other times it crawls or stop completely.

This is the major problem you are having if you want speed. You need to have
a) server that processes images fast (Imagemagick, RAM, CPU)
b) fast connection (ftp, http) without dropouts.

Connect errors will most likely cause EE to halt because it waits until you get a PHP timeout or sockets die.

FTP'ing the files to my server went more smoothly and faster than uploading the files to EE.

EE upload to server function uses http which should be basically same speed as protocol as ftp. In fact EE does not upload photos with http, the browser you use does. Of course server settings and load may affect speed.

The 900 photo upload to EE took nearly 4 hours. If it was 4 hours where you could start the process and ignore it, that would be one thing, but you have to keep an eye on it becuase it frequently needs restarting. Some of the restarting was due to my internet connection, but some due to EE.

I'm sorry but before you say "due to EE" I would need some evidence. I have been on your server and its ftp implementation is flakey (unpredictable) which is the reason for halts and connect errors. I have tried to code workarounds but they worked only to next server setup change. Now you also say you use imperfect net connection, too. Where do I start to debug EE?

- If ftp server is rebooted/restarted e.g. by hourly cron EE can not do anything about it.
- If server runs e.g. backup script cron while you upload, eating all resources, EE can not do anything about it.
- If internet connection fails or drops intermittently, EE is helpless.

The actual upload page can be made lighter, this is what I work on and there will be optional lighter interface(s) with less data to update, just like I did with photo size checker. Upload consists also of procedures that can not be omitted, like resize and ftp - those can be sped up by server and its settings.

[chriz]

Hey Pekka,

As far as myself is concerned, I have my own server with 2Gig Ram, an 8Mb upload connection, so that is no problem.
The new upload script works well for me in the sense that you can select as many files as you want and it will happily upload everything while I can enjoy other things :-)
But it is indeed slower, I understand why though.
The only thing I was wondering is whether there is a config-file where you can set time per action, but there probably isn't heh?

Anyways, as always, thank you for this software,

Christian

[devenh]

Pekka,

I understand that some (or most??) of the slowness is due to my server's configuration and the internet connection I was using (but sometimes photographers are uploading from remote locations...).

My point is that 1.5 worked much more predictably and faster under the same conditions. 2.0 requires signficantly more communication with the client during upload and this makes it vulnerable to all sorts of things. In short, the design if 2.0 is more fragile than 1.5.

I think that the 'lighter' interface you mention may make a significant difference and look forward to using it.

Let me ask another question: What upload speeds (photos per minute) to a remote server have people experienced with EE 2?

My best is 5-6 per minute so I'd like to hear if this is typical.

Deven

[devenh]

I found a work-around for a less than perfect internet connection. This won't work for everyone, but for those it does help it helps a lot.

To recap, when I am away from home I often don't have the best internet connection and this really impacts the EE upload process. What I do now is remotely connect to another PC that has a reliable connection and use this remote PC to perform the EE upload. Once I get the process started on the remote PC, I can disconnect and check back periodically. Works great (but I would still like to see EE have a bulk upload option).

Deven

[wkitty42]

using zip files for the upload doesn't work for you? or are you speaking of the inserting to database portion??

[devenh]

I am talking about inserting into the database.

[wkitty42]

ahh, yes... zip files won't work there... i guess it is/was the terminology that confused me... i see "upload" as sending to the server... "importing" would be more my style of terminology for depicting adding to the database...

[devenh]

I agree with you. I used the term "upload" because in EE the option is "UPLOAD TO DATABASE." Maybe it should be "IMPORT TO DATABASE."

[puffa]

devenh, thats one nice shopping cart. was it hard to set up?

[devenh]

Yes it was, for many reasons.

First, I don't use Javascript very much, so writing the JS code took a while. Then it takes time to test the code under several different browsers.

Second, I had to tweak the PHP code in the EE pages and figure out how to gain access to all the EE fields I needed. Some of the solutions I had to use work, but are far from elegant. It appears the EE page generation PHP code heavily mixes data routines and page presentation routines, and this makes understanding what is going on very difficult.

Third, I am not happy with where I had to place the "Add to cart" and "View cart" buttons. I would rather have them at the top of the page, but all my attempts to do so generated other CSS problems, so I gave up.

Fourth, I would have liked to use the EE CSS in my cart pages so that they had the same visual style as my EE pages, but again I gave up as the EE CSS was too complex for me to figure out.

So, for me, I found that the EE PHP and CSS structure very difficult to modify.

Deven