Waka chan structure
I started working on a script that would let people “seed” a image board much like the way people seed a (Bit)Torrent. People would navigate to the board, and the server would of course share out the texts of the message board and the thumbnails. But when a user would click on a thumbnail to view the full image, instead of having the central server share it out, the server would navigate the user to a “seeder” who was sharing the full image. I was thinking that this would cut down majorly on the amount of bandwidth the central server uses.
But someone told me that WakaChan already does this. This is what I’m confused at. Because as I browse around on WakaChan, yeah, I notice that different sections are hosted by different domains, but each one is only hosted by one at the most. WakaChan doesn’t have the ability to be seeded by multiple home users? I saw the Winny thing, but that’s not HTTP based and looks more like a P2P app similar to EMule (Napster copy-cats).
So like… I’m a little confused, what did this guy mean when he said that people “seed” this network like the way people seed BitTorrents?
Well, I’ve finally broken into a field I have wanted to get into for a while now. I made it so the ImageBoard-Traker can now detect the bandwidth of the seeders. The way it works is by giving a JavaScript to the user as he/she is being redirected to a target image (on a seeder), this JavaScript will then time how long it takes for the server (seeder) to respond, and then time how long it takes to download the target image. This is a pretty cool thing as this information will give the tracker the ability to throttle more/less users to a particular seeder automatically based on the ping time and estimated bandwidth as noted in static logs.
I have a working example here follow both links:
1) Goto http://70.187.193.222:178/ImageBoardTracker/SeedSupport.php?ReDirect=1125470996492.jpg
2) Goto http://70.187.193:178/ImageBoardTracker/ImgBrdCache/GlobalStaticsLog.txt
The first link redirects you to an image (while giving you the JavaScript I talked about), the second link shows you the tracker-statics log, you should see your report in the log after navigating to the first link.
Sorry, the 2nd link should be:
http://70.187.193.222:178/ImageBoardTracker/ImgBrdCache/GlobalStaticsLog.txt
typo :S
Well, seeing as all that really needs to be done is changing a few small things in the HTML code, up to now, I have depended on FireFox extensions. But now I’m starting another method. I made a bot that clones the HTML pages on a board makes the changes. The bot send HEAD requests to the source server and basically, remain updated. I setup a beta test working with two sections of WakaChan (Cosplay & SuperNatural). Check it out;
http://70.187.193.222:178/BoardClone/mnc.rokcorp.net/ib/ghost/0.html
http://70.187.193.222:178/BoardClone/imageboards.4-ch.net/cosplay/0.html (note: the thumbnails may be screwed up but that’s because they are screwed up at the source)
You should notice that each thumbnail redirects though a (beta version of a) ImageBoard-Tracker I setup. The Tracker should redirect users to a dummy-seeder I setup with some space on the SourceForge.net domain I have. The bot I made also changes the HTML so that all URLs within it are NOT relative, this is important because it means you can still use the form on that page to upload/post content onto the board. Of course, if you do so, the cloned board will not be updated until the bot notices there has been a change (may take up to 20 mins).
Anyways, this project has come a long way, I’m going to continue to do work on it though the free-time in my life has been cut in half now that I’m taking classes again full time cry so I might be going a little slow at this. Though on the bright side, I’m almost done with this project (in terms of my original plans for this) All that’s left on the ToDo list is:
- Finnish making the ImageBoard-Tracker be able to be seeded by seeders (very little if NO more code to do here)
- Finnish making the .php script that lets people seed (about 4/5 done)
- Make a <u>program</u> that will let people seed, something which can be used INSTEAD of the .php script (I haven’t started building this yet, but once I finish the .php, it should be easy and fast to build as I will just make an app to emulate the .php)
- Some people seemed to show some interest in the "extras" in the FireFox extension I made (the support for Avatars, Spoiler tags, and Quote tags), this was not really part of this project, but I’ll just quickly reshape it into JavaScript for people to download or include in their .html pages which will in turn do all that editing crap.
Though that seems like allot, most of the coding and pain in the ass work is put into the ImageBoard-side Tracker. The reason this is, is because the seeders are in a way, just dumb terminals. They are like this because you can’t really make the seeders the over-powering commanders of the boards because they can be easily edited by the actual (human) seeders. So obviously, you can’t trust them, and therefore the overwhelming majority of security features has to be put into the ImageBoard-side Tracker. This is a big chunk of coding, and that’s just in addition to all the other crap like the code needed for sorting, tracking, managing of files, URL pointers, etc - Which of course I have finished (you can see it working in the beta tests above).
> - Make a program that will let people seed, something which can be used INSTEAD of the .php script
I take this will replace Apache too? I think the main problem keeping people from running as seeds is that it's a rare animal that runs an HTTP server at home.
If you had a simple click'n'run proggie, that would open up a far larger potential pool of seeders. People won't help unless it's dirt-simple to do so.
Ah, I don't think the software is quite ready yet. I've been playing with the /cosplay/ mirror a bit, and had the following issues:
Having said that, it's a neat idea. I wonder what board maintainers will think.
The feeling I get is that most people have no need for seeders yet. But if this site ever gets popular it might be useful.
What's the projected latency on retrieving images?
> Links on main page to seeded images lead to 127.0.0.1
WaaaaaaAAA!!~ l0l, can’t believe I forgot to make such an obvious change - fixed
> Some pages don't display any images (blocked due to referrer, I suspect)
If you are referring to the Cosplay board, the thumbnails DO display, but it seems imageboards.4-ch.net has... lost their thumbnails? Well, see for yourself:
http://imageboards.4-ch.net/cosplay/ (the source)
and then go to:
http://70.187.193.222:178/BoardClone/imageboards.4-ch.net/cosplay/0.html (the mirror)
You will see they are the same. Also note that for some reason, the thumbs on the lower half of page 2 are working. See: http://70.187.193.222:178/BoardClone/imageboards.4-ch.net/cosplay/2.html and http://imageboards.4-ch.net/cosplay/2.html
I probably shouldn’t of made a beta test from a board that is having problems, but I decided to use them due to their very small demand for space. Current I’m taking a class in web development, and I’m going to use this project as one of my assignments. In doing this I’m hoping I can talk to my teacher about the space and bandwidth the university gives me. I doubt he will care, so he will probably give me an ‘ok’ terms of using.... more. When that happens I will make a mirror of all work-safe sections on WakaChan and 4Chan.
> If you had a simple click'n'run proggie, that would open up a far larger potential pool of seeders. People won't help unless it's dirt-simple to do
> I take this will replace Apache too?
Yeah, I know. That’s what I was referring to as something to build. It’s not really hard to do at all. And as far as replacing Apache, Apache is to large of a project to reinvent, but a simple HTTP server isn’t. All a HTTP server does is listen, and sever. I won’t even need to do the work myself as there are open source examples I have seen out there that are just basically really simple HTTP servers. And then I just need to handle the functions needed to be a seeder, not to hard... depending on how I am going to deal with it.
> The feeling I get is that most people have no need for seeders yet
> What's the projected latency on retrieving images?
O_O ..........no need for seeders eh? Well, maybe WakaChan. And maybe they don’t NEED it, but it would certainly help. IIChan was pushing two terabytes a month, and other board such as 4Chan keep dying due to their constant lack of money to pay for their bandwidth. If 4Chan was to die again, I’m sure that the people who normally go there would come here, I know this because I’ve seen it happen before (between the last death and resurrection of 4Chan). If/when this happens the current bandwidth used by WakaChan would probably double (if not more). I highly recommend you watch this flash:
http://127.0.0.1:178/Misc/Flash/unix.swf
That flash concerns Japanese ImageBoard(s), and it really shows that this is an occurring problem, not just with English board, but just about all of them (who are regularly used).
> http://127.0.0.1:178/Misc/Flash/unix.swf
oh lol @ url
If it's the flash from this: http://4-ch.net/tech/kareha.pl/1100254230
then it's not about imageboards. 2channel is not an imageboard.
No, squeeks referer-blocks on all of 4-ch.net, so you can't link the images from another page. Shift-reload to get the thumbnails back on the real page.
> but it would certainly help.
It's best to prepare for the future. :)
>>64
Well shit, they did that even for thumbnails? sighs m'kay, I'll have to make the bot DL a clone directory for the thumbnails then.... Gonna have to be carefull though, I was auto-banned from WakaChan when using the bot with the waka domain, gues it looks like a site-ripper, even though it conservs bandwidth by using http-head requests... :/
gets to work
I don't know what dmpk2k does on wakachan.org, but on wakaba.c3.cx I've got a hidden link on the front page, that is listed as forbidden in robots.txt, that bans you if you hit it. Well-behaved site rippers will check robots.txt and not hit it, and don't get banned. Spam harvesting spiders don't care about robots.txt, or look at it to find likely targets. Those get banned real quick.
~ Crys ~
Well for the amount of work I have done (in the past few days) I feel like I walked a ways, and then took several steps backwards. I wanted to make a mirror of WakaChan’s anime section, but as I started to set it up, it suddenly occurred to me how much space this was going to require (space isn’t an issue on my home comp but it is on the dummy seeder I setup). So I somehow got the idea in my head that I should make the bot that sets up the mirroring limit the number of pages (and threads) it mirrors. So instead of just automatically ssh-ing into SourceForge and posting WakaChan content up there, I had it only posting so much, and then when a thread aged to the point where it moved to page 4 or 5 or whatever, the bot ssh-ed in and deleted it. ~~ BIG MISTAKE ~~ Rather than explaining what was wrong with that plan, I think it’s better to show you:
http://sourceforge.net/project/stats/detail.php?group_id=143055&ugn=seeder-chan&type=prweb&mode=week
Take a good look at the lower graph, and then look at the table. I first setup the thing to limit pages for Oct 19th… l0l, and then the bandwidth skyrocketed. I normally wouldn’t care about the bandwidth consumption, but when saw the number of people who actually USED that much BW for viewing purposes (which is shown @ http://70.187.193.222:178/ImageBoardTracker/ImgBrdCache/GlobalStaticsLog.txt ) is not very high, I was just like, WTF!
sigh well, only now did I finally realize that when people bump a thread from beyond page 4 it caused the bot to re-upload the file contents (ALL images in the thread) to SF, hence the bandwidth spike, and hence why I feel like I “took sever steps backwards,” because it’s clear now that I will have to turn the page-limiter off. Or to put it in more simple words, the programming I did a few nights ago is now for nothing ;_;
So, if anyone is reading this, learn from my mistakes and don’t lack in foresight... :＼
Dude. You're solving a problem that doesn't exist, I believe. dmpk2k already has a script for this.
Hm? You mean to mirror? Or the hole ImageBoard-Tracker thing?
…No The Mirror is just a quick thing I'm making to test a project that will give people the ability to seed ImageBoards, similar to the way people seed BitTorrent. Because I can't edit the pages directly on the ImageBoard servers (the project is still experimental, I wouldn't want to yet), I'm making a mirror of those pages and making the few small changes to them needed. The Primary change is taking every thumbnail link and changing them to something like;
http://70.187.193.222:178/ImageBoardTracker/SeedSupport.php?ReDirect=http://imageboards.4-ch.net/cosplay/src/1129534730252.jpg
(Adding the ImageBoard-Tracker's address before the target URL)
If you navigate to that URL, and you right click on the image and go to properties, you will notice that the image shown to you is NOT from imageboards.4-ch.net. It redirects to one of the “seeders” of the ImageBoard. Of course, this project is still in the experimental stage, so there are no real seeders, but I set one up with my SF space and edited the settings files so they see SF as a seeder.
Go to http://seeder-chan.sourceforge.net/ and read up on it there.
~crys~
Why what? I thought the ImageBoard-Tracker project was a good idea. It was originally intended for 4Chan as they kept dieing, and when they started to ask for donations to keep them selves afloat, (some) thought that people would be more likely to donate bandwidth than money as it was the main principle behind BitTorrent....
I think what he's doing is a good idea. Quit beating the guy over it. A simple mirror isn't that useful, but if that's what's needed in order to help develop his seeder concept, so be it.
I'm actually getting a little tired over people griping over others developing software. You should have seen the reaction of posters on 4chan to this, the bunch of idiots they be.
Then there's the whole Albright story. He develops something useful and people hound him. Fine, the first version was horribly ugly, but after he fixes it people still moan and groan. They come up with stupid shit like, "You're forcing people on wakachan to use it." I think it's amazing Albright hasn't walked away from the community in disgust.
Yes, I think Wakaba's the best, but I also think it should stand on its own merits. As for SWG's efforts, what he's slowly building could come in very useful one day.
Quit being a bunch of killjoys on development.
THANK YOU!~ I couldn't of better worded it then how you just did. I don't know what the "Albright story" is, but it in terms of the lack of respect people get for taking time out of their own lives to help in development, I totally know what you mean. It amazes me that people act like such assholes when we are doing this for them.
>>76
I would be overjoyed if Albright left iichan. It would be the greatest thing he's ever done for us.
And why is that?
> They come up with stupid shit like, "You're forcing people on wakachan to use it."
I came up with that first and I think it's perfectly valid criticism, esp. considering the fact that many people didn't like Thorn to begin with.
>I would be overjoyed if Albright left iichan. It would be the greatest thing he's ever done for us.
What ISP keeps putting in connections under bridges?
Must be AOL.
(That's not to say I myself am never guilty of trolling now and then, of course.)
> And why is that?
He's caused nothing but trouble since he started adminning.
About the only trouble I can see is the drama drama surrounding Thorn, and he seems quite civil about it considering the flames. Franky, it seems like much ado about nothing.
The existence of Thorn can only benefit the community. Cross-polination of ideas, you know?
Yeah, I see nothing wrong with the current Thorn interface, though I admit I thought the last one was crap. (Perhaps just becuase I'm used to seeing futaba-like imageboards.) And the ability to post multiple images is definitely a plus.
It would look better with Waha's CSS, but I really have no problem with the current look.
...I feel like wrote this before.
Reposting from the frontpage cache, which wasn't included in the restoration
94 Name: SomeWhiteGuy !obkj4fb50I : 2006-01-22 13:40 ID:55WL3xNm [Del]
L0L, Aahahahahah, good idea. Maybe I'm a pervert because that totally slipped my mind. Ok, just made one and posted it.
95 Name: Albright!LC/IWhc3yc : 2006-01-23 10:27 ID:Py9K8nis [Del]
Across all boards, I seem to be getting more "Fatal Error: The ImageBaord-Tracker's URL-source-list for that file seems to either deleted, or it's contents seems to have been erased" than actual images. I even tried uploading my own image and opening it, and got that error. What's causing this?
Seems like there's still a few glitches to iron out...
96 Name: SoeWhiteGuy !obkj4fb50I : 2006-01-29 15:53 ID:55WL3xNm [Del]
Well this is a bit overdue, but I finally got to making the Tracker able to install it self with Kareha and Thorn! During installation, it will inject code into Kareha/Thorn files if you give it the directory path, and it will auto update the templates(s). I setup some examples of them working on TestChan; http://somewhiteguy.homelinux.org:178/TestChan/
The /a/ section is using Kareha, and the /w/ section is using Thorn.
Right now I don’t have it working with Wakaba because I have to figure out how to get Perl to get access to MySQL, I think it needs something called DBI? But when I try to download it, it seems like it’s giving me files for Linux, ack, man just getting the freeking permission on Apache set to allow CGI scripts to work was painful enough (enabling Perl was), I need a break before I retry the DBI installation… unless anyone knows of any guides out there for it… which I couldn’t seem to find on Google.
97 Name: !WAHa.06x36 : 2006-01-29 16:10 ID:Heaven [Del]
On windows, use the ppd command to install it.
On Unix, use perl -MCPAN -e shell or perl -MCPAN -e "install DBI"
98 Name: Albright!LC/IWhc3yc : 2006-01-29 20:55 ID:Py9K8nis [Del]
Your Thorn installation seems pretty wonky; it's trying to link to stuff on 127.0.0.1. Please open up config.php and edit the THurl value to something sane.
99 Name: SomeWhiteGuy !obkj4fb50I : 2006-01-30 11:35 ID:55WL3xNm [Del]
>>97
O_O Wow, you guys need to tell me these things sooner. I had no idea it was so easy. Before I was trying to unzip contents off some package I found and trying to get settings files and environment paths set correctly... that ppm fucker did all the shit needed with one command line! :o
>>98
WWWWAAAAAAAAAAAA!!!~ Holy shit, that blind-sided me.... for like, the third time o_o Ugh... Ok, I think it should be fixed now, I think the problem was during installation I told it (either Thorn or the Tracker, I can't tell which one) to see itself as installed on 127~ instead of the host-name... blah
100 Name: !WAHa.06x36 : 2006-01-30 14:31 ID:Heaven [Del]
Shouldn't you be able to use SCRIPT_NAME to avoid needing an option like that?
101 Name: SomeWhiteGuy !obkj4fb50I : 2006-01-30 17:12 ID:55WL3xNm [Del]
>>100
Hm... This is pretty interesting, but I just noticed (from reading up at http://us3.php.net/reserved.variables ), that one can use this code (in PHP):
extract($_SERVER);
echo "http://" . $HTTP_HOST . $REQUEST_URI;
That should say public URL to the script... And for the local path, I think you could just use $PATH_TRANSLATED (from $_Server)
Pretty interesting, I just found that out myself, before when I needed the public (URL) and local paths I just had the user plug it in, and when it came time for them to do that I had a JavaScript on the user's browser plug it in automatically from looking at document.location
102 Name: Albright!LC/IWhc3yc : 2006-01-30 21:50 ID:Py9K8nis [Del]
>>100: Upon installation, Thorn makes a guess as to its own URL (and path) using the methods in >>101, but it also gives the installer the opportunity to change it if it guesses wrong. Either Thorn guessed wrong, or SWG changed it.
>>99: Hmm, image links are working now, but it seems to still be trying to get the thumbnails, CSS and JS code from 127... Try deleting the cache...
103 Name: SWG !obkj4fb50I : 2006-01-31 01:42 ID:55WL3xNm [Del]
Ok, fixed it, thought I fixed the THurl var, but guess I closed the text pad and didn't save it before because it was still going to 127~
Gah, this is annoying, and it's al because of my crappy LinkSys router (it's an older one). I dunno why the f**k it does this, but it seems to give me a "Connection Reset by Peer" error on all connections when I try to connect to my external IP address. It does this every so often anyways, but I have noticed it does this immediately after trying my external IP. If I hit it, within 10 seconds I get a CRbP error over IRC, and then other connections seem to break/reset to. Ugh....
So yeah, that's the cause of all of this bull s**t, otherwise I would use my external IP to connect to my server and there would never be any problems. Supposedly (I head from someone) that it's doing this as a security feature which was programmed a little to trigger happy... Blah I dunno, whatever, RRrrghh....
Funny, this thread jumped a post. Oh well, some breakage is to be expected, I guess.
dont u love how maericans always screw everything up? I can honestly say I hate them and I am one, god dam, this is just more proof of how completly retarded america is, I wish I lived in canada
>>96
is this a broken spambot, or is some idiot really going around bumping threads from 3 years ago to make wildly off-topic posts?
>>97
I wouldn't be surprised if it was the latter. And my has the quality of SOC really gone downhill as of late.
