1.5RC4 causes double bandwidth counting

[stevehof]

In the aftermath of the worm attack and inspection of logs I noticed that my own fixed IP is getting charged with bandwidth in a nearly equal amount to each visitor. So, I now have close to double the bandwidth usage that I had with EE 1.22. This problem has only been present since I installed EE 1.5RC4. I tried adding the lines noted below to my .htaccess file but EE won't load the photo on the photo page with this code in the .htaccess file. EE will still show thumbnails on the gallery viewer page. Pekka, do you have any suggestions? See my server config below code lines.

Thanks, Steve

order allow,deny
deny from 66.(my IP#)
allow from all

My Server:
Apache version 1.3.33 (Unix)
PHP version 4.3.10
MySQL version 4.0.22-standard-log

[Pekka]

There is some difference what size html pages EE 1.22 or 1.5 creates, 1.5 should create 40% smaller html because of several html code optimizations.

EE consumes server bandwidth by serving html code and images to visitors. The bandwidth used comes from html code (usually 30kb per photo page but zgipped to 5KB which is used for bandwith count) and images (microthumbs + one big size photo on photo page). If your bandwidth gets higher it means either that you have more page loads and/or your image filesizes are bigger (thumbs and large sizes).

I'm not sure what you try to do with Apache directives. Are you trying to block your IP out or are you trying to block everyone else out?

One thing I noticed in your site. It seems that your thumbnails were created with some EXIF data intact. This makes them much larger (3-8 times larger) than needed and that will increase bandwith usage.

Go to graphics modules editor
Select Imagemagick (that is what you are using, right?)
Click "DO THE SELECTED TEST".

Now see what is the filesize of the thumb created (it's printed next to the result thumb). If it is over 5KB then go to bottom of that page and change another profile command setting and click "Update".

Now do the Imagemagick test again. If filesize is about 3-4KB then you're ok. If not, try the third profile command and try again.

Note that with Imagemagick 6.01 or so they had a bug with default compression setting, so you''ll need to do the size test in "resize methods /thumbs and Microthumbs" editor (75% is the default JPEG compression) and right-click to check the size. With 6.03 this bug is fixed.

Now, when you get the profile command setting ok and the test thumb is small you should recreate all thumbs (rethumbnail editor) for the new exhibitions. Note that rethumbnail needs to have the paths set with resize methods and ftp (and working ftp), so if you have some older size and thumb paths from 1.22 you'll need to set those up before rethumbnail can work with them.

[stevehof]

Pekka,
I am still trying to figure out where the problem actually lies. My web host says that the PHP code is causing the server to double count the actual bandwidth being used. Their tech support can't pinpoint the exact code so I suppose they are guessing too. The bottom line is that my own fixed IP number now shows up in my web stats as a user of approximately half the bandwidth attributed to sphoto.com. This phenomenon is new since I installed EE 1.5RC4. Under EE 1.22 my IP didn't list on the user's page of my web stats. The photos and thumbnails are exactly the same file size as they were under 1.22. I don’t see how 1.5RC4 could be using twice the bandwidth just in html code?

The apache handler trick was supposed to remove my own IP from the statistics just in case the PHP code was somehow loading the image once for my IP and once for the visitor's IP. As mentioned, this didn't work since disallowing my IP caused the photos to not load on the photo page. So, there may some validity to my web host tech support's claim that the PHP code may be contributing to this problem. This post was meant to be an FYI as well as ‘possible’ bug report. I can live with this if I have too….
Thanks, Steve

[stevehof]

PS. Imagemagik and FTP are not installed on my server. I don't mind since I'd rather prepare my own images. I have been careless in watching whether or not I save the JPEGs with EXIF info or not....;^)

[Pekka]

PHP code can not affect bandwith because it is processed in server. The html produced by it is the only factor here.

MySQL server can increase bandwith usage if they give you a MySQL server which is not in your IP (localhost). That should not be counted in bandwith you use, though, as it is data moved inside network. If the MySQL server is outside ISP network the it is counted as normal bandwidth.

Only think I can think of which may cause this is that you have "misc settings/When image file is not found" at 2 or 3. That would make PHP check image existance and that could be added to bandwith (max 1/3 datasize per image only). But in your server it looks like above settings is at 1.

My guess is that their bandwith measurement tool is broken.

my own fixed IP number now shows up in my web stats as a user of approximately half the bandwidth attributed to sphoto.com. This phenomenon is new since I installed EE 1.5RC4. Under EE 1.22 my IP didn't list on the user's page of my web stats.
If your site sphoto.com is in different IP than you connect it to from home, then only explanation is upload/download with ftp from your home to the site, or using topmost upload form via EE's Upload page.

Do you have Webalizer or such on your server to examine detailed site usage?

[stevehof]

My server's control panel has both Webalizer and AW Stats. That's where I'm getting this information. The only thing I don't understand is if I disallow my own IP in the .htaccess file, everything works fine except that the photos don't load on the photo page only. That seems strange to me. Why does EE require access to my static IP number to display the photo on the photo page? EE will display the thumbnails on the gallery browser page even with the IP denied. All image file sizes are hosted onsite just one folder down from the root of my web site.
Thanks, Steve

[stevehof]

My web host tech support's comments regarding the .htaccess file that disallowed my own fixed IP.

"I would try contacting the creator of the script first. I have never seen an instance where just loading a page causes a request from the server itself. And because it is only happening on the 'main' image of the page (and not the dozens of filler images in the header, etc.) It has to be caused by something the script is doing, there's just no other logical explanation. I would try to figure out what is going on there, maybe it has happening to someone else before and a simple configuration change somewhere would fix it."

[Pekka]

My server's control panel has both Webalizer and AW Stats. That's where I'm getting this information. The only thing I don't understand is if I disallow my own IP in the .htaccess file, everything works fine except that the photos don't load on the photo page only. That seems strange to me. Why does EE require access to my static IP number to display the photo on the photo page? EE will display the thumbnails on the gallery browser page even with the IP denied. All image file sizes are hosted onsite just one folder down from the root of my web site.
Thanks, Steve

Perhaps you are downloading pages and images (thumbs) from browser cache.

Here is how EE page is received:

1. Your browser requests EE page from URL
2. Apache httpd process is started
3. Apache checks from its config if client IP is allowed
4. if ok, it sees the requested URL is php, so it uses PHP module (Apache PHP module or CGI)
5. In this PHP environment, EE builds html
6. PHP module passes the complete html data to Apache
7. html is served by Apache to your browser

Depending on Apache settings and version, Apache may serve also images from same "process", often it opens a new process for each "item" on page (graph, image, flash, sound). Every time permissions are checked.

The paths in EE, where the images are loaded from, are set up in path editor. The IMG tags that end up in html contain the http://something/somesize/someimage.jpg address for your browser. In case image does not load, you can take IMG URL's from html source code and paste it in your browser's address line to check if they give a specific error message from server.

If you want to filter your server IP from logs, it should not be done by denying server IP in Apache - this will definitely cause problems as most processes on server communicate with Apache as localhost. Webalizer has settings to ignore certain IP's, use that.

[Pekka]

My web host tech support's comments regarding the .htaccess file that disallowed my own fixed IP.

"I would try contacting the creator of the script first. I have never seen an instance where just loading a page causes a request from the server itself. And because it is only happening on the 'main' image of the page (and not the dozens of filler images in the header, etc.) It has to be caused by something the script is doing, there's just no other logical explanation. I would try to figure out what is going on there, maybe it has happening to someone else before and a simple configuration change somewhere would fix it."

Well. They have Apache access logs. Why not check there what happens when EE photo page is requested from a browser that is not cached?

[stevehof]

Pekka,

I forwarded your information and comments on the logs to my web host tech support.

Thanks for your time....

[Pekka]

Here is what I get in my server log when I open page

http://photography-on-the.net/gallery/photo.php?photo=9&u=33-157

with Internet Explorer ALL CACHE's OFF. XXX.XXX.XXX.XXX is the IP I browse from (i.e. home Helsinki/Finland to USA where the server is):

(command grep XXX.XXX.XXX.XXX /path/to/logs/access_log)

XXX.XXX.XXX.XXX - - [28/Dec/2004:13:12:09 -0600] "GET /gallery/photo.php?photo=9&u=33-157 HTTP/1.1" 200 5943
XXX.XXX.XXX.XXX - - [28/Dec/2004:13:12:11 -0600] "GET /gallery/graphs/x.gif HTTP/1.1" 304 -
XXX.XXX.XXX.XXX - - [28/Dec/2004:13:12:12 -0600] "GET /gallery/graphs/separator.gif HTTP/1.1" 304 -
XXX.XXX.XXX.XXX - - [28/Dec/2004:13:12:12 -0600] "GET /gallery/D30_photos/microthumb/CRW_1964_00001.jpg HTTP/1.1" 304 -
XXX.XXX.XXX.XXX - - [28/Dec/2004:13:12:12 -0600] "GET /gallery/styles/dark_2/microthumbseparator.gif HTTP/1.1" 200 812
XXX.XXX.XXX.XXX - - [28/Dec/2004:13:12:12 -0600] "GET /gallery/D30_photos/microthumb/CRW_1187_00001.jpg HTTP/1.1" 200 1032
XXX.XXX.XXX.XXX - - [28/Dec/2004:13:12:12 -0600] "GET /gallery/D30_photos/microthumb/CRW_2331_00001.jpg HTTP/1.1" 304 -
XXX.XXX.XXX.XXX - - [28/Dec/2004:13:12:12 -0600] "GET /gallery/D30_photos/microthumb/CRW_8057_00001.jpg HTTP/1.1" 200 972
XXX.XXX.XXX.XXX - - [28/Dec/2004:13:12:12 -0600] "GET /gallery/D30_photos/microthumb/CRW_7370_00001.jpg HTTP/1.1" 304 -
XXX.XXX.XXX.XXX - - [28/Dec/2004:13:12:12 -0600] "GET /gallery/D30_photos/microthumb/CRW_0126_00001.jpg HTTP/1.1" 304 -
XXX.XXX.XXX.XXX - - [28/Dec/2004:13:12:13 -0600] "GET /gallery/D30_photos/microthumb/CRW_3603_00001.jpg HTTP/1.1" 200 1037
XXX.XXX.XXX.XXX - - [28/Dec/2004:13:12:13 -0600] "GET /gallery/styles/dark_2/shadow_1/fill.gif HTTP/1.1" 200 825
XXX.XXX.XXX.XXX - - [28/Dec/2004:13:12:13 -0600] "GET /gallery/styles/dark_2/shadow_1/right.jpg HTTP/1.1" 200 3823
XXX.XXX.XXX.XXX - - [28/Dec/2004:13:12:13 -0600] "GET /gallery/styles/dark_2/shadow_1/b_r_corner.jpg HTTP/1.1" 200 341
XXX.XXX.XXX.XXX - - [28/Dec/2004:13:12:13 -0600] "GET /gallery/D30_photos/medium/CRW_8057_00001.jpg HTTP/1.1" 200 47663
XXX.XXX.XXX.XXX - - [28/Dec/2004:13:12:13 -0600] "GET /gallery/styles/dark_2/shadow_1/bottom.jpg HTTP/1.1" 200 2023
XXX.XXX.XXX.XXX - - [28/Dec/2004:13:12:14 -0600] "GET /gallery/styles/dark_2/left.gif HTTP/1.1" 200 594
XXX.XXX.XXX.XXX - - [28/Dec/2004:13:12:14 -0600] "GET /gallery/styles/dark_2/right.gif HTTP/1.1" 200 601
XXX.XXX.XXX.XXX - - [28/Dec/2004:13:12:14 -0600] "GET /gallery/styles/dark_2/tab.jpg HTTP/1.1" 200 3230

if I do grep 28/Dec/2004:13:12 /path/to/logs/access_log I get surrounding events from any IP and there are no events on that path not to mention another GET /gallery/D30_photos/medium/CRW_8057_00001.jpg HTTP/1.1" 200 47663

The I copied the URL to Firefox (which has cache on) and it opens, then I click leftmost microthumb on top and get URL http://photography-on-the.net/gallery/photo.php?photo=21&u=33-154 mostly from cache, only these new items are asked by browser from Apache:

XXX.XXX.XXX.XXX - - [28/Dec/2004:13:15:46 -0600] "GET /gallery/D30_photos/microthumb/CRW_2059_00001.jpg HTTP/1.1" 200 844
XXX.XXX.XXX.XXX - - [28/Dec/2004:13:15:46 -0600] "GET /gallery/D30_photos/medium/CRW_1964_00001.jpg HTTP/1.1" 200 76078

All this is EXACTLY as it is supposed to be. Not double imageloading or other anomalies.