Hello,
I'm the admin/owner of fatechan.net, and I run /moe/ and /cw/ on iichan. When I first started my boards, I had my own box I had coloed, and everything ran fine. Due to some monetary issues tho, I was forced to cancel my colo and move to shared hosting.
Problem is, my account has now been suspended several times for excessive cpu usage. I'm hosted with cirtexhosting.com, and I run kareha. I keep my thread count around 6 pages (I manually clean out the boards), but according to my host, I'm still eating 14% of cpu time.
Any ideas for hosts that would be more lenient with their cpu usages? Dreamhost is not an option, I'm just waiting for them to collapse from overselling. I use kareha for the custom tripcode catches, so if there's another imageboard soft out there that can do that, I could switch.
Any ideas at all? I'd rather not have to pay more for a semi-dedicated just for the imageboard.
Turn off captcha. Kareha uses next to no CPU power without it.
Captcha is off, but according to my host, here's my load numbers.
fatennet fatechan.net 6.77 7.26 0.0
Top Process %CPU 14.0 /usr/bin/perl kareha.pl
Top Process %CPU 11.9 httpd [www.fatechan.net] [/moe/src/1154017429681.jpg]
Top Process %CPU 10.7 httpd [www.fatechan.net] [/moe/thumb/1158840292216s.jpg]
I cut down the page count down to 6 for both, it was at like 30ish, since I didn't like to clean out the board, and I really didn't have a need to.
Top Process %CPU 11.9 httpd [www.fatechan.net] [/moe/src/1154017429681.jpg]
Does that mean that 11.9% of CPU usage is taken up by serving an image? What the hell?
I have no idea what's going on, but the host is threatening to cancel my account, and they've suspended me twice for it.
This is perhaps obvious enough that you have already checked it, but are the pics being leeched? Googling reveals fatechan.net/moe is linked to from all over the web. All it takes is some thoughtless idiot doing image links on some really popular forum and there you go. Though how a thumb could eat up that much CPU is beyond me. Some forums allow people to link to sites for their avatar pics so maybe somebody is doing that?
BTW down to 6 pages now? Gosh, goodbye Arisa and Suzuka, goodbye Saya, goodbye ... Ayu? Uguuuuu!! I'm feeling torn. On the one hand I've been feeling guilty about bumping big threads but now I feel like I should rush in and bump the heck out of them as soon as /moe/ and /cw/ are back. But the latter might not be advisable in this situation. Uguu. Perhaps if you set up a public image archive for your fav pics and then let threads autosage at 111 pictures?
Before I cleaned out the boards, I backed it all up, so I still have all the images. If I can solve the cpu problems, I'll look into bringing back the threads I deleted.
As for leeching, I have entries in my .htaccess to prevent leeching.
RewriteEngine On
RewriteCond %{HTTP_REFERER} !^http://(.+\.)?fatechan\.net/ [NC]
RewriteCond %{HTTP_REFERER} !^$
RewriteRule .*\.(jpe?g|gif|bmp|png)$ /hotlinking.gif [L]
The image leechers are referred to is a 1x1 transparent gif btw.
Well, I think I found the issue. I noticed on my mediawiki install, thumbnails weren't working because it couldn't find imagemagick "convert" in the path. I'm wondering if the huge cpu time is from kareha having to do the resizing itself with the perl imagemagick module.
I've put the absolute paths to the convert command in my config.pl, gonna see if that helps.
Yay, it's back to 8 pages. But seriously, you don't mind me adding to those big threads? Your wallet and host can handle the traffic?
Feel free to post and make the big threads even bigger. I have over 2tb/month in bandwidth, and you guys are nowhere near pushing it
Only thing that worried me was cpu usage. If cpu usage stays within bounds, I'll look into remerging the old threads back into the boards. I might look into making an archive board, and setup a script, where if the thread count gets too large, it'll automatically move it to a read-only store. Serving static files uses no resources, just bandwidth, and I have more than enough to burn.
This is getting rediculous, they're continuing to suspend me for excessive cpu usage. I'm on vanilla kareha, no modifications, and imagemagick is working. Captchas are off.
>>14 perhaps you could recommend another host that is not dreamhost?
age because fatechan still needs a host recommendation
the current plan is to hack kareha so it can host images on a separate server, then run it on a VPS (with miniscule bandwidth allowance) that TSO also has, but cirtex is somewhat unreliable so we'd like to switch anyway
>>16
I am extremely sad because your site is down. Please please bring it back!
did you not read the thread?
also i'm not TSO, just a friend
i love fatechan and am sad to see it go. I'm not a rich fellow by any means seeing that i'm in college but presuming donations are acceptable would money from users fix the problem and how much would presumably be needed?
The CPU thing sounds like bullshit. They could renice your tasks just the same as anything else. I mean, Kareha pretty much caches everything out the ass already, updating things mostly just when the relevant pages have changed or not been updated since last change.
It's not like /cw/ or /moe/ got enough traffic for that to happen or anything. (Could there be a hidden processor time sink thing in Kareha, for boards that've got a bit older? I remember 4ch's DQN used to have a huge post delay before the long tail was archived.)
>>21 He's running wakaba, though. I can't fathom what the problem might be.
No, I'm on Kareha. I'm still taking recommendations for friendlier shared hosting, but I'm prepared to hack kareha to run the script and db itself on a vps, then use shared hosting to host the images on a much higher bandwidth connection.
As for donations, I thank you guys for the offer, and please get in touch with me via my email (tso at fatechan.net) and I'll arrange paypal or something.
>>21 It's not like /cw/ or /moe/ got enough traffic for that to happen or anything. (Could there be a hidden processor time sink thing in Kareha, for boards that've got a bit older? I remember 4ch's DQN used to have a huge post delay before the long tail was archived.)
fatechan.net actually pulls about 500-600gb/month of traffic, which is surprising. I didn't realize how popular my boards were, but it still baffles me how kareha could be taking up that much cpu time. When I hosted it on my own server, it took up nothing.
PS, it's she ;)
Oh, I forgot to mention. It seems shared hosting is useless, due to very restrictive cpu usages. VPS plans give way too little bandwidth and hd space, and semi-dedicated/dedicated seems to be overkill, and is wayyyy out of my price range.
With donations, semi-dedicated/dedicated might be feasible. I'm still willing to do a major kareha hack, to run the main script on a VPS, and then do the actual image storage/hosting on a shared host with much higher space and bandwidth. Doing that would also allow possible load-balancing and multiple hosts, but that'd take me quite a bit of time and effort to run properly.
And to be perfectly honest, I'm a student too. I really don't have time to sit here and code something major, especially something I've never really done. The semi-dedicated/dedicated route would be a lot easier, and a lot quicker, but would cost a little more (well, it might not, the price of a capable vps and shared hosting might be around the same).
Up to you guys, I'm doing this for you guys.
(The other thing is, there's already kei.iichan's /2/ board. Which is much the same thing as your two boards.)
Based on the forum names it might appear as if there's a lot of overlap but /2/ feels like most other iichan boards, while /moe/ and /cw/ have developed their own distinct dynamics. In any case, it's idlechan, the more boards the merrier.
There's a little backhistory of that. When my boards were added to the iichan list, they were brought on under the condition that /2/ would eventually be dropped, and my boards would replace them.
That's why the main page says it's the replacement for /2/, that was the original intention. I originally started out with just /moe/, but to take up the slack of /2/ I started /cw/ to help divide things out. But I guess Blackmage forgot :T
As far as I can tell, it's impossible to do load balancing on kareha as it is. Uploaded images would have to go to the high bandwidth servers somehow, after posting, and their names would have to be the same. I sadly suggest you run dreamhost, hell gurochan is running there and has not had any problems.
Not impossible at all. DNS round-robin setup. www.fatechan.net would point to the vps, and img.fatechan.net would point to a list of image hosts. All I have todo is write a wrapper script for karena that is called when images are uploaded to push the image to the hosts.
Hell, I don't even have to do the DNS magic. I can just have the wrapper script do that as well. EX. Kareha wants to link to an image. It thinks the images are in /src/wrapper.php/imagename.whatever. What really happens is with that url, the image name is passed to the wrapper script, the script goes "hay, let me go through my list of hosts, and randomly pick one, and pass that onto the person requesting it."
The hard part for me is not the loadbalancing for the image displaying, it's figuring out how to reliably process and push uploaded images to the hosts, and keep them in sync in realtime. I've already got a basic framework going, I just need to do a little more testing and some security setup and then burn it in.
I plan on GPL'ing my abstraction hacks/scripts, to make it easy for other image boards to deal with scaling/bandwidth issues.
As for dreamhost, I'm not worried about their level of service, I'm terrified I'll give them money, then they'll finally collapse from such drastic overselling. And from what I've experienced, dreamhosts are very serious cpu nazis, and if kareha is giving me fits on a far less stressed host, I can't imagine what dreamhost is going to do to me.
Any other recommendations? I was looking at LunarPages or Servint right now.
> how to reliably process and push uploaded images to the hosts, and keep them in sync in realtime
Maybe you could just set up a cronjob on the image hosts to check for new images in a directory on the VPS every hour or so, and then a fallback for the wrapper script to fetch the image locally if it couldn't be found on the image hosts. It'd consume some extra bandwidth on the VPS, but nowhere near as much as having it all on one host.
Also, why not make your stuff public domain like Wakaba/Kareha themselves? No one's going to make a profit selling imageboard software, and you'll make the copyleft haters happier.
Per lunarpages: http://www.bearcityweb.com/lunarpages/index.html
Didn't realize kareha was public domain, I just assumed gpl/some sort of bsd license.
As for that review, that was made over a year ago, it seems they've shapen up a lot since then, and so far I've read a lot of good things.
>> Maybe you could just set up a cronjob on the image hosts to check for new images in a directory on the VPS every hour or so, and then a fallback for the wrapper script to fetch the image locally if it couldn't be found on the image hosts. It'd consume some extra bandwidth on the VPS, but nowhere near as much as having it all on one host.
Was actually just thinking about just letting the script push it itself, with ftp or possibly another catch script on the other site. Then make a cronjob do an actual rsync or whatever to keep all the mirrors in line, just in case one was down when an image was first released.
I'm trying to force the load of the actual thread/db management and syncing on the vps, and just let the image hosts be dumb hosts that just serve the images. So I want to leave the task of actually doing all the cpu intensive and actual footwork to the vps. Just make it so that anyone with extra space on a host just needs to either give me an ftp account, or drop a script and make a few directories in order to become a mirror.
I'm also trying to code an archiving system, that takes the last page on the active board, makes a static page, including threads, and moves it to an archive. This will reduce strain on the active board as it rebuilds the thread tree after each new post, and static pages need very very little resources to be hosted.
>>31
Well, sounds complicated (i.e. impossible if you don't have enough free time) And I'm a bit fuzzy on "pushing the images". This puts same load on both servers CPU wise during transfer, and double CPU on the host receiving the upload.
Dreamhost stopped overselling 3 months ago. They noticed it seems. Their prices are higher and bandwidth less. I still would ask you to contact chiisai on the site i mentioned in my post, he seems to run a ~1tb site without problems on dreamhost on kareha.
I don't know, but I was in love with your site, and I'm willing to help financially or psychologically, or technically as far as I can.
If you can pull off the load balancing scripts, I'm sure guys running imageboards will fall in love with you.
But that load is only once, during the upload cycle. The main bulk of the CPU usage is in the actual thread sorting/generation and display, not the image thumbnailing or uploading. The image upload only occurs once, and doesn't actually take up that much in resources. It's the thread sorting, rearrangement, and cache creation every time a post is made that takes up the most time, and happens every single time you post. That image is only uploaded and processed once, and that's it.
Why not switch to wakaba?
Kareha's CPU use is far higher (I vehemently disagree with >>2, because it just ain't so). Also, wakaba already has a load balancer, so no need to reinvent a wheel.
Just to nail the point home:
From secchan.net, which has several Kareha installs:
_Process CPU seconds user machine count average
kareha.pl 2958.5100 88.537% 12.327% 17311 0.171From wakachan.org, which is all wakaba, yet gets twice as many hits:
wakaba.pl 64.8100 33.006% 0.270% 127 0.510The numbers speak for themselves. I disagree with WAHa's extending Kareha to do what Wakaba does, rather than the other way around. This is why.