Waka chan structure
I started working on a script that would let people “seed” a image board much like the way people seed a (Bit)Torrent. People would navigate to the board, and the server would of course share out the texts of the message board and the thumbnails. But when a user would click on a thumbnail to view the full image, instead of having the central server share it out, the server would navigate the user to a “seeder” who was sharing the full image. I was thinking that this would cut down majorly on the amount of bandwidth the central server uses.
But someone told me that WakaChan already does this. This is what I’m confused at. Because as I browse around on WakaChan, yeah, I notice that different sections are hosted by different domains, but each one is only hosted by one at the most. WakaChan doesn’t have the ability to be seeded by multiple home users? I saw the Winny thing, but that’s not HTTP based and looks more like a P2P app similar to EMule (Napster copy-cats).
So like… I’m a little confused, what did this guy mean when he said that people “seed” this network like the way people seed BitTorrents?
It doesn't exist. Maybe the person you spoke to was referring to "DistWakaba", an idea that was floated back in October when the original IIchan died. A spec already exists, if you want it.
The problem is that it's a hard problem. First off, if you just use one board and farm out src/ requests, you still have one point of failure, and people are less likely to donate space just for a src.
The moment you have multiple boards with the same images though, you end up with hairy problems. For example, how do you keep post numbers synchronized? What happens if one host drops off and reappears later? How do you keep threads synchronized? How do you keep this responsive? What happens if there's a race condition and you lose? How do you keep one broken or hacked board from bringing down the rest? How do you plan to allow the balancing algorithm to scale? Issues of calling perl instead of static html and server load. Other issues related to security. Etc.
It can be done. As I said, there's a spec that lays out how to do it. But you're going to be working on it a looong time. I'd love it if you did though. I'd even help.
Actually, I have been working on it for a while now. And to answer a few questions..
“For example, how do you keep post numbers synchronized?”
I was thinking a main board, like <Whatever>Chan, would host the board. The board would basically handle the posts. So it would be central-server based to an extent, but “seeders” would be the ones serving full images. When a user would post an image, the board would upload the image to the seeders.“What happens if one host drops off and reappears later?”
Seeders would sign in to the <Whatever>Chan. They would goto the Chan and say, this is my IP address, I’m an serving images <here>, to upload a file to me, posit it to <this> php file, and whatnot. The server would assume the seeder is online, and when navigating a user to the full image hosted by a seeder, the user will be given a JavaScript programmed with a list of seeders who have the same file. If the image fails to load on one seeder, the JavaScript will navigate to one of the other seeder.The rest of your questions seem to assume that all the seeders are the ones hosting the board entirely. With the plan and beta I made, <Whatever>Chan would be a centralized server hosting text for threads (and maybe thumbnails) only. The seeders would host the full images.
Seeders would sign in and say their IP address and the public location of there PHPkernel.php file. When a user made a post on <Whatver>Chan with a pic attached, the <Whatever> server would save the thread (maybe make a thumbnail to serv), and then post the attached image to…
PHPkernel.php?Cmd=upload (to manually do it you would have to goto PHPkernel.php?Cmd=showupform)
…for every seeder. Every seeder would make public what they are sharing by PHPkernel.php?Cmd=JavaScriptDirListing
Well, that probably sounds confusing. But if you look at the beta version I made, you will probably understand. Basically, PHPkernel.php is the main object used for sharing. I made a .html file to interact with it you can see it here:
http://70.187.193.222:178/imgchan.htm
The .html file is not like a <Whatever>Chan because you can’t talk on it. It’s just used to interact with one or multiple PHPkernel.php files at once. You will see it work as every thumbnail source goes through it, and when trying to upload it redirects you to it (with the Cmd= showupform as I mentioned above), and to initialize it downloads a list of what every seeder is sharing (like the Cmd= JavaScriptDirListing as I was talking about above).
…it might be a little to confusing by words, but if you navigate to the link above maybe you will understand what I mean a little better.
No, I know what you're driving at. All the more power to you.
Keep us updated.
Interesting. But how do you deal with issues like:
Ooh... just looking at the site, I see two very interesting features there. First of all, I simply love the distributed idea. Certainly there are issues to be solved, and while the security and robustness should NOT be tacked on later, some functionality upfront is a good focus, IMHO. I don't quite yet know how you'll "call" the seeders to keep things up-to-date, but I think the idea is solid, and could be great fun.
Second is categorisation. As the proprieter of meidokon, I'm of course very interested in any such topic. I've literally only just loaded the front page, and am excited about what I hope to soon see...
People grabbing a seed and uploading their own images there and using the bandwidth for themselves?
Indeed, a refer check might help. Also, if a person was seeding to <whatever>Chan, building the .php to only accept incoming files from the IP or host-mask from <whatever>Chan might also do the trick.People running a fake seed that contains nothing but goatse.
You mean “ghosts”? Well, the way I was thinking this was gonna work was to have a .php on the seeder, and a .php on the <whatever>Chan that acts as an extension to the <whatever>Chan (Wakaba/Futallaby) board. When a seeder tells the <whatever>Chan’s .php that they are going to seed, they will tell their IP address, and the location of their .php file used to seed. This will need to be done anyways as the <whatever>Chan will obviously need to know the IP address of the seeders to rout users to the seeder. Anyways, the seeder’s .php would return a file-list of all files they are sharing upon request. I guess to take care of what you asked, the <whatever>Chans’s .php could check the file list. It would test random files on the file list. PHP supports the code like:$s =implode(“”, file(“http://filezilla.sourceforge.net/misc/ip.php”));
This means you can get the file contents of another website and domain, (the above example will return your external IP address into a string $s. I guess I could make the <whatever>Chan’s .php test this using the above code, only replacing the above URL with a seeders shared file.
Latency issues? If a seed goes down, the connection there will most likely take a long time before it timeouts, making it hard to know when to try a new one.
Hm… that’s a hard one to deal with. Before I talked about sending a JavaScript to a user who is being redirected to a seeder, and having the JavaScript redirect the user to another seeder if that first target seeder times out or doesn’t have the file. I guess, to deal with this issue, I could make the JavaScript do the following in response to a seeder timing out:
- Open a new window
- Redirect the user to another seeder (not the new window)
- Have the newly opened window redirect to <whatever>Chan’s .php with a parameter like: whatever.php?Cmd=ReportSeedFail&SeedIP=555.555.555.555
- Have the whatever.php on the <whatever>Chan’s side send a java script in return that will try to close the newly opened window after registering that a seeder has timed out.
- Now that the <whatever>Chan has been giving a report that a seeder has timed out, it can test for it self, verify the seeder is lagging, and…. I dunno do something, like maybe not redirect as many users to that seeder, or even ignore the seeder completely.
Though your question really is hard to deal with because FireFox users probably won’t open a new window (that’s a built in function of FireFox). Though if it was a hidden frame, that would be doable. Dunno… I’ll have to think about that, like where I would put the frame, I would have to plan in advance on where the frame would go, and how exactly it would even be given to the user machine.
Furi, thanks, that really means allot to me. Just keep commin to my page about once a week or so, I usually try to post news at least once every other day, and I try to do work on this project every night. I think within two weeks time I should be done with the seeder-side .php that will manage shared files. Dunno about the <whatever>Chan's .php that will manage the seeders though. I have had ideas, and I have been taking notes on what it will need to do, but I have not yet made a beta version of one yet.
Thanks for your support, it really means allot to me.
No problem. Seems we both have fun projects being developed. :)
I'm not 100% sure I understand all of what you're getting at, but my rough idea was this. The main server has a list of files that have been uploaded, and this can only be updated by posting/uploading them at the main server. Thumbnails are also generated and stored on the main server so they can't be corrupted (that's not to say that the seeder actually storing the real image can't put replace everything with goatse, but it's a start).
FYI, "goatse" as it's called, is a nasty piece of work that has become a sort of net-cult icon. It's an image (which was part of a series) of a guy doing nasty things to his anus. Looking at it is like watching a train wreck in slow-motion. (ie. You can't look, but you just can't look away). Anyway...
A client (don't know how, but I think PHP would be the easiest way, IMO) would connect to the main server (aka. tracker) and register its identity and a list of images that it's holding. Only those that were uploaded to the main server could be used, of course.
A request to the main server for an image simply picks one of the registered seeders and redirects the request to that IP address. It's far too much load to check which seeders are online for every request for an image, so perhaps have the server check which seeders are online every... 5 mins?
Biggest issue I see with this:
My idea requires the seeders to be running PHP, and by extension, Apache. That's not a trivial amount of work for most users, and it's a non-trivial security risk as well, if you don't know what you're doing.
I could set up a clone of the script that accepts uploads from an image board, and whenever the image board connects back, I return those images. However, if any other IP address connects, I return goatse, or a browser crasher.
I guess this is a pretty hard problem to solve. You might have to trust the seeds in the end, which I am not sure is a good idea on something like this, where script kiddies abound. You might need some sort of approval system for seeds...
(Also, don't put space or tabs at the start of comment lines, it makes the script put in <pre> tags that mess up your text.)
Actully that problem is a little easy to solve....
Mabey I can have a support page on <whatever>Chan that will be made for reports on such stuff. If people report a seeder is returning a file that does not either...
a) Look like the thumb-nail (a dead give-away)
b) Sound like it's the image being talked about on the board-thread
...then someone can make query on the support board. It can be automated, accepting the thread ID and Seeder ID/IP and whatnot. Then, once submitted, this message can be either forwarded to the admin, or a script that will auto-ban people for doing such things. Yeh, people might try to abuse this, claming that a seeder is seeding false files for the wrong thumbnails, but if they do so the admin can double check and ban people who try to do this just to be an ass. Obviously most <whatver>Chans seem to have more than one admin/mod, and their IPs are all different. So maybe they can do a final check before banning a seeder's IP when it's needed.
Ah, fair enough. Yeah, multiple trusted admins isn't a bad way to go. And of course, a trusted admin can always have the final word and ban/delete stuff without further approval/checking.
I'm actually thinking of some over-the-top cryptographic methods to ensure that the image data can't be tampered with, but it's probably a bit much for our purposes here.
...
Hmm, on further thought, I don't think it's possible without some cryptography, which would be nasty (and as I've said, inappropriate for our situation). If you're vaguely interested by cryptography then read on.
In short, the only way you can ensure someone is sending you good data without directly trusting them is to have a hash of the data that you know is true. But if you hash the whole image, it means you have to download the entire image before you can check it. Which of course defeats the entire point of the exercise (you've just wasted bandwidth on what may or may not be goatse).
Of course, we need not get all of it at once. It would place some load on the server, but it'd be very small. The server could send a series of hashes of sequential chunks of the file. That way we could request chunks of the real file from the seeder and verify small blocks at a time. Not really efficient, but not unworkable. The moment a block is received from the seeder that doesn't hash as the server says it should, we just assume the file is corrupt.
I figure you could do this easily and efficiently enough, but I don't know how you'd implement it. As far as I know, Javascript can't be used to open connection streams and get chunks of data across the network. The space saving is cool, though. Assuming you hash 4kilobyte blocks, you could use 128bit MD5 and get a space saving of 32x. A 2meg file would need only 64KiB of upload from the main server to verify it. You can also balance hash size against chunk size to fit whatever is convenient for your situation.
But I'm losing myself here. This is quite impractical...
It's impractical, but an interesting theoretical exercise.
Doing constant-sized blocks is not really necessary here. The header is more likely to be different for an image file anyway. You could just keep checksums for a range of power-of-two sizes. The first 256 bytes, the first 512 bytes, the first 1024 bytes, the first 2kbytes, and so on. Would take up negligable space for files of any size.
Javascript can receive data just fine though, using XMLHttpRequest. Combined with the data: protocol, you could probably make Javascript that downloads the image and checksums it, and then creates an <img> tag with a data: URL containing the image data. This would, however, be slow, and it would also not work on IE (IE doesn't allow null bytes in Javascript strings).
Not that's a useful idea! Yes, even very large files can be effectively hashed in chunks like that. Lemme see... a one gig file will only need 13 hashes in such a manner, which is positively inconsequential for server load.
Blah. I guess it really comes down to "do we give a damn about IE, then?" :)
Seems Javascript has had some changes since I last looked...
nevermind me
i am just unbreaking the frontpage's table
Well I FINALLY got a beta version out, and released graphical explanations of what I was talking about.
About half way down through http://seeder-chan.sourceforge.net/Welcome/SeedSetup2.htm I start talking about the HTML (that holds JavaScripts) that I made. It doesn’t do anything significant though. The REAL major thing put in use is the PHPkernel.php
I also released documentation about how it works here:
http://seeder-chan.sourceforge.net/Welcome/Docs_PHPKernel.htm
I made a beta of this and released it, you can DL it here:
http://seeder-chan.sourceforge.net/SeedChan.zip
So far all I have out is the thing that lets people share files out, and the ImgChan.htm that acts something like <whatever>Chan emulator in place of the real thin, but it doesn’t have the ability for people to post text. People can post images, but cannot chat. I have an example of ImgChan.htm working with my PHPkernel.php up at:
http://70.187.193.222:178/ImgChan.htm
To make a working version of what I have on your computer, all you need is to download the zip file above, and then follow the instructions file inside it.
As for what we discussed above, I haven’t really got to those yet. I haven’t even built the php that will sit on <whatever>Chan’s server. Give me some more time.
Two things bother me. First, would you please say "imageboard" instead of "<whatever>Chan"? It's horribly awkward, and the tradition of naming imageboard something-chan is only western thing. The vast majority of image boards aren't named like that. The "chan" comes from "2channel" anyway, which has nothing to do with images.
Second, and this is just a pet peeve of mine, the words "hentai" and "ecchi". It's really only westerners who use these words to refer to porn. There's really no need to mangle the japanese language like that. We laugh at the japanese for their engrish, but it works the other way too. Using them as categorizations for how explicit porn is extra silly.
Am i the only one getting very many 404 errors on the test board?
>>19
/me apologises for using a category on meidokon called "not ecchi or hentai"
(excuse me while I ignore the fact that we're talking about usage to gauge explicitness)
Well, I don't know about you, but I feel such a term is at least better than "worksafe".
<tirade>
If you're looking at my stuff, none of it is worksafe! Why are you wasting time at work looking at this drivel?!
> "not ecchi or hentai"
> "worksafe"
"not pornographic"? No need to mince words.
>>20
Sorry, my comp hasn’t always been on. And then I lost DMZ access for a while on my router. But I'm back up now
>>19
O_o arn't we being a little picky? hentai means porn and ecchi means perverted. Regardless of weather that means the same in jap or not, that's what we westerners understand. Trying to change that all of a sudden will just confuse people more.
> Trying to change that all of a sudden will just confuse people more.
I don't see what's wrong with that. Being confused is the first sign of coming to a better understanding by realizing prior misunderstanding.
> Regardless of weather
PS: It's "whether", not "weather" - in regard to the contextual right meaning in ami
As I said, it's just a pet peeve of mine. There's no need to have the words mean anything else - I just avoid using them alltogether. A lot of japanese people browse Wakachan, and it's only common courtesy to not mangle their language in ways that make them giggle. (Japanese people do giggle at westerners saying "hentai" just as much as we giggle at their Engrish.)
They giggle because it means "pervert".
Let them know who they are, I'd say! They are in denial! :D
> hentai means porn
I think it's closer to "perverted psychopath" than "porn".
LET'S DANCING!
>>22
But what I'm trying to cover here (at least on my own board) is stuff that could be what we think of as "ecchi" (eg. a girl seductively flipping her skirt up, or touching her body in a non-explicit and suggestive manner) and stuff that's "hentai" (more explicit nudity, little artistic redeeming merit, etc.).
I definitely don't think pornographic/non-pornographic is applicable here. Heck, by my own values, I don't really want "pornography" on my board (eg. girl using fingers to expose her genitals, sexual intercourse, stuff the Japanese usually need to mosaic-censor). It just doesn't do anything for me.
>>26
So what do the Japanese say?
> So what do the Japanese say?
It depends, really. Porn movies are called AV (Adult Videos), "pink" is often applied to words to hint that they have to do with erotica ("pink doujin") but mostly, they seem to apply the prefix "ero" to stuff ("eroge" (for "erotic game"), "erodoujin","eroneta", "ero AA" etc.)
Futhermore, what we'd call hentai manga is usually tagged as "成人コミック" on the filesharing networks - literally, "adult comic" (and actually "comic" as a loan word, not "manga"). There's also "18禁", "Over 18 only". Both of these are more formal, obviously.
Also, lots of people translate "ero" as "erotic", and while that is technically where it originates, it doesn't really have the right connotations. The meaning is pretty much exactly the same as the word "porn". "Porn game", "porn doujin", etc.
"erotic" would be a good substitute for the way you're using "ecchi", though.
>>32
Hmm, definitely worth considering a change to.
http://4-ch.net/nihongo/kareha.pl/1121923289/ - People who say "hentai" are perverts!
I got curious and Googled a bit for further discussion on this topic, and found this (on site that is apparently now dead, so only the Google cache exists):
The part I found most interesting were the quotes that person had gathered and translated. I'll cut-and-paste them here to preserve them. Many thanks to Akadai Shirou for doing this work:
What Japanese think about this
These are quotes straight from the Japanese, translated, pulled from numerous sources.
どうやら、日本語の「変態」の意味のPervertやKinkyという意味は少なくて、単にHなアニメとかをいうみたいです。
It seems they rarely mean the Japanese meaning of 変態, Pervert or Kinky, but rather they are using it just to mean any H anime
---
外国人に｢HENTAI｣と言う言葉を広めた奴を捜しています！！！懸賞金＄1,000,000−
I'm looking for the guy that spread the word 'HENTAI' among the foreigners!!! $1,000,000 reward
---
海外のオタク系が日本製のアダルトアニメやゲームを表す言葉。
Definition: Word used by foreign fanboys to mean Japanese adult animation and games
---
ちなみにエロゲーはHENTAI GAMEとか書いてあった。・・・・・・・・・・・・・・・・・・・・・・・・・なんかやなカンジ。
By the way, [in U.S. video game magazines] they call [Japanese] adult games HENTAI GAME...................... that sounds awful
---
HENTAIとは成年コミックや１８禁美少女ゲーム、アダルトアニメをさす
HENTAI is used [by foreigners] to indicate adult comics, games, and animation
---
外国の HENTAI サイトをうろうろと見て歩いていたら・・・なんだか気分が悪くなってしまった（苦笑）。
なんて言うか・・・独特の雰囲気があるよね。
ちなみに HENTAI というのは、日本の漫画、アニメ、ゲームなどのえっちなメディアを指すらしい。
日本人の俺としては、やっぱり抵抗があるなぁ（笑）。
他人が描いたＣＧや市販ゲームのＣＧを無断で転載して、ギャラリーで御座い・・・というサイトが実に多い。
うーん、この辺は基本的な考え方が違うのかな。
節操の無い連中だ・・・という印象を強く感じるね（笑）。
I went looking around at those foreign HENTAI sites... and kinda I just got sick.
What can I say...they have a 'unique' atmosphere.
'HENTAI' is what they use to indicate Japanese manga, animation, games, etc. when they have etchi content.
Speaking as a Japanese person, I have to say that's pretty disturbing.(laugh)
There's a ton of sites that just copy other people's illustrations and rip images out of games, then display them in a gallery.
I guess their entire way of thinking is different from ours?
I get a strong impression of a total lack of civility and morals. (laugh)
---
Ｈ（えっち）
「性交」を意味する若者用語。
実は「ＨＥＮＴＡＩ（変態）」の頭文字から来ている。
海外の濃いＯＴＡＫＵたちの間ではＨではなくＥＣＣＨＩと表記される。
それに対して事情通のＯＴＡＫＵが「ＥＣＣＨＩはＨＥＮＴＡＩのイニシャルなんだから、Ｈと書くのが正しい！」と啓蒙運動を行っている。
その様な日本人も知らないような知識を、どこから得たのか非常に謎である。
てゆーか、正しかろうと間違ってようと、あんまそんな日本語を世界に広めないで欲しい。
H (etchi)
Word used by young people to mean sexual intercourse.
It actually comes from the first letter in the word HENTAI(変態).
Among the hard-core foreign OTAKU, they write it 'ECCHI' instead of 'H'.
Acting against this are a group of other OTAKU familiar with the etymology of the word who go around saying "ECCHI is just the initial for HENTAI, so you're supposed to write it H!" in an attempt to enlighten everyone.
How they learned that sort of trivia that not even Japanese people know is a complete mystery.
More importantly than that, regardless of whether they're getting it right or not, I'd rather they didn't go spreading the usage of that sort of Japanese
---
お名前：天狼星
＜・#65310;BAKAとかHENTAIとかOTAKUって言葉を教え
てるのは誰ですか？ 知人じゃないことを祈ります（苦笑） いや・・ HENTAIは
すでに日本語じゃなくなってるか？ 日本製の萌え系エロ絵なんかをさす言葉だし；
Name: Tenrousei
[...]Who taught these people to say BAKA and HENTAI and OTAKU and such?
I'm praying it wasn't anybody I know. (pained smile) Actually, it seems like HENTAI
isn't even Japanese anymore, you know? The use it to mean Japanese drawn cute erotic illustrations and all
---
さて、このポルノアニメというジャンルには、animeよりももっとこの世界で一般的になってしまっているらしい驚くべき名称がつけられている。 HENTAI、というのだ・・！ 「エッチ」という語がそもそもなぜ使われるようになったのか、常識だと思っていたら案外知らない人もいるらしいので、余計なことかもしれないが言っておけば、この「変態HENTAI」の頭文字なのである。初期に日本の「エッチアニメ」を取り入れた人が妙に律儀だったらしく、略語をちゃんとオリジナルに展開して使用した模様である。売り文句には「JAPANESE KAWAII HENTAI」とあったりする。可愛い変態、である。やれやれ。
Now, this genre of porno animation is known to the world not by the word ANIME but by the surprising name 'HENTAI.' The source of the word 'etchi,' -- while I thought most people knew it, apparently some don't, so I'll go ahead and say it here -- is the first letter in the word 変態 (HENTAI). Early on, when Japan's etchi anime was first taken in by foreigners, they apparently were trying to be weirdly faithful linguistically and restored the abbreviation to its original form before using it. You can see the phrase 'JAPANESE KAWAII HENTAI' used in advertising phrases. Cute perversion! Man...
> I'm looking for the guy that spread the word 'HENTAI' among the foreigners!!! $1,000,000 reward
LOL
:o
TheGuyThatSpreadTheWordHentai runs and hides
Hey all. Well, it’s been along time since anyone has continued this thread. Hm… As for my project, it’s been going along. For anyone who doesn’t remember, I’m the guy trying to make an ImageBoard Add-on script that will allow ImageBoards to be seeded much like how torrents are seeded with BitTorrent. In my lasts posts (here) I had the seeder-side scripts built, and now, FINALLY I have a stable build of the ImageBoard-side script. At this moment (time I am posting this message), I have not yet released the script yet though, but I do have it built and working! Sorry it took so long, it’s just that it was a major pain in the ass because I had to write that entire system to keep track of information in so many different places, AND THEN I had to actually take the time to figure out how to the Futaba script worked inside-out. The reason being because obviously when someone uploads a file to you ImageBoard, you want the ImageBoard-Tracker (the add-on script which I made) to auto-keep track of the file and make it available for ImageBoard-seeders. (NOTE: Incase you didn’t know the script I am making is an add-on/support-script, it is NOT an ImageBoard system by itself.) So I had to look in the code of Futaba and find out where to insert certain lines. In the end, it wasn’t so bad, you only need to insert lines where:
• Files get uploaded
• File links are shown to users
• Files get deleted (even though I couldn’t find it)
• Insert include statement on load of the ImageBoard script (The lines I’m referring to inserting is simply calls to functions that exist in the script I am making. So yeah, one would need to use include(MyScript) or require(MyScript)
So uhh… yeah, one question I do have for anyone out there, is where does a file/post get deleted in Futaba scripts (due to old age or whatever)? I couldn’t find it.
A second question I have is about Wakaba. You see, I want my add-on script to be able to work with Futaba and Wakaba (and anything else really), but whenever I go to the Wakaba site, I am directed to http://wakaba.c3.cx/releases/ and all the files in the zips I see there are non-php files…. Am I missing something? I was under the impression Wakaba was programmed in PHP, was I wrong?
Well, anyways, I continued the project and kept going. But the ImageBoard-side script I made can’t really install itself (in terms of integrating with the ImageBoard that already exists on server), because I thought it would be too time-consuming on my part to make PHP code to scan another ImageBoard script, and do the inserting of lines automatically. To make such code would require to know the ins and outs of many different ImageBoard scripts and have the code ready for handling any situation that comes by it. sigh So in the end I write a really long 2 page file about how to insert 4 lines of code into a Futaba script o_o;
Well, enough. I know I’m boring you all with this, so I’ll show you what I got!
I have a website now that explain how the system works. It doesn’t explain everything, but it’s a start. You can see the website here:
http://seeder-chan.sourceforge.net/
And the web site really is good, I have an FAQ guide there, and soon I’m gonna be done with the walkthroughs.
Also, though I don’t have the ImageBoard-side script released yet, I DO HAVE AN EXAMLE RUNNING! Now I have to warn you it doesn’t LOOK like much, but you need to remember I am programming an ImageBoard-Support system, not a GUI upgrade. Anyways, see the running example here:
http://70.187.193.222:178/ExampleChan/futaba.htm
AND READ THE BOX ON THE TOP OF THAT PAGE, it explains a little bit on how the script is integrated. As of right now, I’m not letting people upload to that board because I want to use it as an example to help admins with the installation of the ImageBoard-side script. But if you click on a link you will see how it forwards you to another location which may be off the domain the board is on. And if you click the same image a few times (like more then 5 times or so within 10 seconds), you will notice that it WILL be hosting the full images from the main domain. This is a good example of flow control, and how the script makes sure that not too many users of the board are routed to one seeder. Read the main page (on the website near the bottom) to learn more about flow control.
Anyways, I just waned to l you all know I’m still alive. And yeah, I would really appreciate anyone would could answer those questions for me;
1) Where do files/posts get deleted in Futaba script?
2) Where is the Wakaba PHPs?
I also have a question to dmpk2k, you said something about “DistWakaba” on post #2. Where is the spec on this? I tried to Google it, but the only result I got oddly enough was this very thread on this very board o_o;
Wakaba and Kareha are written in Perl, not PHP.
>>40 He's working with the hacked futuba code from thatdog over at 1chan, it seems.
And I'm not just working with that one script. I want to make an add-on/support script that works with all ImageBoard scripts. I also have a running example of the add-on script working with the English MySQL version here:http://66.252.18.113:178/ExampleChan2/imgboard.htm
I also intend to make this work with Wakaba, I just didn’t recognize what language it was programmed in. Thanks btw >>40
> Where is the spec on this?
The only people who have ever seen it are me, WAHa, and perhaps Lain. I probably have one of the versions collecting dust somewhere on my drive.
BTW, if you're planning on modifying Wakaba, you might want to wait until the release of the next version. It has rudimentary support for a system somewhat similar to yours, so modifying it to run your semi-P2P system will be a lot easier.
Hardly. Most of the proposed new code has been running in /os/, /mai/, and now /miz/ for several weeks now.
There's been a testing board for a bit over two months: http://wakaba.c3.cx/sup/kareha.pl/1119971465/
http://coralcdn.org/ would allow to serve each image file only once a day, it caches things automagically, just rewrite the image url a bit.
What I don't understand is how or why they do that. Pure research?
Bandwidth doesn't grow on trees, yet they'll let a site use up to 250GB a day.
Back when iichan was running on just the one server, we briefly considered using the Coral cache to save on bandwidth. But around that time, they were proudly announcing that they had just transfered one terabyte of data in total. iichan was pushing two terabytes a month, so we figured they weren't quite up to the task.
I'm sure they've grown since, but it would be nice to know what they can handle.
Ok time for an update ~
As I was working on the ImageBoard-Tracker system I started posting in 4Chan /dis/ section to see if 4Chan was interested in my idea seeing as they have died in the past and seem like they are going to die again... Moot blew me off but as people responded, someone gave me a good idea. An idea that would let this system make it easier to support WakaChan and in addition still support 4Chan.
Someone told me to make a FireFox extension. As I looked into this I realized that FireFox extensions could alter the HTML of files given to a user. It sparked in my head that this could take care of an issue with WakaChan using this support system. The issue I noticed a long time ago was that WakaChan is divided into many separate domains - domains that I assume are owned by different people. This became an issue as the ImageBoard-Tracker I was building was designed to sit on a central server where the ImageBoard was being hosted, this obviously can’t happen easily as WakaChan is not central-servers based for all sections. Another side issue that I noticed was that getting each person who is hosting their section of WakaChan to edit their script could become troublesome.
Thus the idea of the FireFox extension was like an explosion in my mind because rather than troubling all admins, the extension could just detect a certain ImageBoard, and route the user to the correct ImageBoard-Tracker when need be. This also takes care of another MAJOR issue, which is testing. One can load the extension, and if it totally fucks up, then the extension can just be unloaded, thus there is no need to do things to the server which would affect everyone.
So, now all my plans on the architecture have changed... kinda, right now I don’t really know (1) how should this project be continued, and (2) should I even bother...
But, just for kicks, I did some "rewiring" and made an ImageBoard-Tracker that is external to WakaChan, and built a quick program to watch WakaChan and update the tracker when need be. I wanted to set it up to host and track all of WakaChan, but I began to realize just how much space that would require... something which isn’t a big deal to me, but became one when I was setting up dummy seeders (hosted by my university, which has no idea what I’m doing with my space --- and I would like to keep it that way)
Anyways, I have the FireFox extension build at http://seeder-chan.sourceforge.net/pubdownloads/4ChanGUIAddon.xpi After you load it up, try going to the "Cosplay" (@ http://imageboards.4-ch.net/cosplay/ ) section, and the "supernatural" (@ http://mnc.rokcorp.net/ib/ghost/index.htm ) section. Upon clicking thumbnails you should be redirected to the tracker that should redirect you to one of two seeders I set it. One setup on my SourceForge space, the other at my university.
I expect it's been mentioned before that not everyone uses, wishes to use or can use Firefox. This can't hurt, though.
>>53
Hey, it's an interesting idea. Give it a chance and we'll see what happens.
Well, I’ve finally broken into a field I have wanted to get into for a while now. I made it so the ImageBoard-Traker can now detect the bandwidth of the seeders. The way it works is by giving a JavaScript to the user as he/she is being redirected to a target image (on a seeder), this JavaScript will then time how long it takes for the server (seeder) to respond, and then time how long it takes to download the target image. This is a pretty cool thing as this information will give the tracker the ability to throttle more/less users to a particular seeder automatically based on the ping time and estimated bandwidth as noted in static logs.
I have a working example here follow both links:
1) Goto http://70.187.193.222:178/ImageBoardTracker/SeedSupport.php?ReDirect=1125470996492.jpg
2) Goto http://70.187.193:178/ImageBoardTracker/ImgBrdCache/GlobalStaticsLog.txt
The first link redirects you to an image (while giving you the JavaScript I talked about), the second link shows you the tracker-statics log, you should see your report in the log after navigating to the first link.
Sorry, the 2nd link should be:
http://70.187.193.222:178/ImageBoardTracker/ImgBrdCache/GlobalStaticsLog.txt
typo :S
Well, seeing as all that really needs to be done is changing a few small things in the HTML code, up to now, I have depended on FireFox extensions. But now I’m starting another method. I made a bot that clones the HTML pages on a board makes the changes. The bot send HEAD requests to the source server and basically, remain updated. I setup a beta test working with two sections of WakaChan (Cosplay & SuperNatural). Check it out;
http://70.187.193.222:178/BoardClone/mnc.rokcorp.net/ib/ghost/0.html
http://70.187.193.222:178/BoardClone/imageboards.4-ch.net/cosplay/0.html (note: the thumbnails may be screwed up but that’s because they are screwed up at the source)
You should notice that each thumbnail redirects though a (beta version of a) ImageBoard-Tracker I setup. The Tracker should redirect users to a dummy-seeder I setup with some space on the SourceForge.net domain I have. The bot I made also changes the HTML so that all URLs within it are NOT relative, this is important because it means you can still use the form on that page to upload/post content onto the board. Of course, if you do so, the cloned board will not be updated until the bot notices there has been a change (may take up to 20 mins).
Anyways, this project has come a long way, I’m going to continue to do work on it though the free-time in my life has been cut in half now that I’m taking classes again full time cry so I might be going a little slow at this. Though on the bright side, I’m almost done with this project (in terms of my original plans for this) All that’s left on the ToDo list is:
- Finnish making the ImageBoard-Tracker be able to be seeded by seeders (very little if NO more code to do here)
- Finnish making the .php script that lets people seed (about 4/5 done)
- Make a <u>program</u> that will let people seed, something which can be used INSTEAD of the .php script (I haven’t started building this yet, but once I finish the .php, it should be easy and fast to build as I will just make an app to emulate the .php)
- Some people seemed to show some interest in the "extras" in the FireFox extension I made (the support for Avatars, Spoiler tags, and Quote tags), this was not really part of this project, but I’ll just quickly reshape it into JavaScript for people to download or include in their .html pages which will in turn do all that editing crap.
Though that seems like allot, most of the coding and pain in the ass work is put into the ImageBoard-side Tracker. The reason this is, is because the seeders are in a way, just dumb terminals. They are like this because you can’t really make the seeders the over-powering commanders of the boards because they can be easily edited by the actual (human) seeders. So obviously, you can’t trust them, and therefore the overwhelming majority of security features has to be put into the ImageBoard-side Tracker. This is a big chunk of coding, and that’s just in addition to all the other crap like the code needed for sorting, tracking, managing of files, URL pointers, etc - Which of course I have finished (you can see it working in the beta tests above).
> - Make a program that will let people seed, something which can be used INSTEAD of the .php script
I take this will replace Apache too? I think the main problem keeping people from running as seeds is that it's a rare animal that runs an HTTP server at home.
If you had a simple click'n'run proggie, that would open up a far larger potential pool of seeders. People won't help unless it's dirt-simple to do so.
Ah, I don't think the software is quite ready yet. I've been playing with the /cosplay/ mirror a bit, and had the following issues:
Having said that, it's a neat idea. I wonder what board maintainers will think.
The feeling I get is that most people have no need for seeders yet. But if this site ever gets popular it might be useful.
What's the projected latency on retrieving images?
> Links on main page to seeded images lead to 127.0.0.1
WaaaaaaAAA!!~ l0l, can’t believe I forgot to make such an obvious change - fixed
> Some pages don't display any images (blocked due to referrer, I suspect)
If you are referring to the Cosplay board, the thumbnails DO display, but it seems imageboards.4-ch.net has... lost their thumbnails? Well, see for yourself:
http://imageboards.4-ch.net/cosplay/ (the source)
and then go to:
http://70.187.193.222:178/BoardClone/imageboards.4-ch.net/cosplay/0.html (the mirror)
You will see they are the same. Also note that for some reason, the thumbs on the lower half of page 2 are working. See: http://70.187.193.222:178/BoardClone/imageboards.4-ch.net/cosplay/2.html and http://imageboards.4-ch.net/cosplay/2.html
I probably shouldn’t of made a beta test from a board that is having problems, but I decided to use them due to their very small demand for space. Current I’m taking a class in web development, and I’m going to use this project as one of my assignments. In doing this I’m hoping I can talk to my teacher about the space and bandwidth the university gives me. I doubt he will care, so he will probably give me an ‘ok’ terms of using.... more. When that happens I will make a mirror of all work-safe sections on WakaChan and 4Chan.
> If you had a simple click'n'run proggie, that would open up a far larger potential pool of seeders. People won't help unless it's dirt-simple to do
> I take this will replace Apache too?
Yeah, I know. That’s what I was referring to as something to build. It’s not really hard to do at all. And as far as replacing Apache, Apache is to large of a project to reinvent, but a simple HTTP server isn’t. All a HTTP server does is listen, and sever. I won’t even need to do the work myself as there are open source examples I have seen out there that are just basically really simple HTTP servers. And then I just need to handle the functions needed to be a seeder, not to hard... depending on how I am going to deal with it.
> The feeling I get is that most people have no need for seeders yet
> What's the projected latency on retrieving images?
O_O ..........no need for seeders eh? Well, maybe WakaChan. And maybe they don’t NEED it, but it would certainly help. IIChan was pushing two terabytes a month, and other board such as 4Chan keep dying due to their constant lack of money to pay for their bandwidth. If 4Chan was to die again, I’m sure that the people who normally go there would come here, I know this because I’ve seen it happen before (between the last death and resurrection of 4Chan). If/when this happens the current bandwidth used by WakaChan would probably double (if not more). I highly recommend you watch this flash:
http://127.0.0.1:178/Misc/Flash/unix.swf
That flash concerns Japanese ImageBoard(s), and it really shows that this is an occurring problem, not just with English board, but just about all of them (who are regularly used).
> http://127.0.0.1:178/Misc/Flash/unix.swf
oh lol @ url
If it's the flash from this: http://4-ch.net/tech/kareha.pl/1100254230
then it's not about imageboards. 2channel is not an imageboard.
No, squeeks referer-blocks on all of 4-ch.net, so you can't link the images from another page. Shift-reload to get the thumbnails back on the real page.
> but it would certainly help.
It's best to prepare for the future. :)
>>64
Well shit, they did that even for thumbnails? sighs m'kay, I'll have to make the bot DL a clone directory for the thumbnails then.... Gonna have to be carefull though, I was auto-banned from WakaChan when using the bot with the waka domain, gues it looks like a site-ripper, even though it conservs bandwidth by using http-head requests... :/
gets to work
I don't know what dmpk2k does on wakachan.org, but on wakaba.c3.cx I've got a hidden link on the front page, that is listed as forbidden in robots.txt, that bans you if you hit it. Well-behaved site rippers will check robots.txt and not hit it, and don't get banned. Spam harvesting spiders don't care about robots.txt, or look at it to find likely targets. Those get banned real quick.
~ Crys ~
Well for the amount of work I have done (in the past few days) I feel like I walked a ways, and then took several steps backwards. I wanted to make a mirror of WakaChan’s anime section, but as I started to set it up, it suddenly occurred to me how much space this was going to require (space isn’t an issue on my home comp but it is on the dummy seeder I setup). So I somehow got the idea in my head that I should make the bot that sets up the mirroring limit the number of pages (and threads) it mirrors. So instead of just automatically ssh-ing into SourceForge and posting WakaChan content up there, I had it only posting so much, and then when a thread aged to the point where it moved to page 4 or 5 or whatever, the bot ssh-ed in and deleted it. ~~ BIG MISTAKE ~~ Rather than explaining what was wrong with that plan, I think it’s better to show you:
http://sourceforge.net/project/stats/detail.php?group_id=143055&ugn=seeder-chan&type=prweb&mode=week
Take a good look at the lower graph, and then look at the table. I first setup the thing to limit pages for Oct 19th… l0l, and then the bandwidth skyrocketed. I normally wouldn’t care about the bandwidth consumption, but when saw the number of people who actually USED that much BW for viewing purposes (which is shown @ http://70.187.193.222:178/ImageBoardTracker/ImgBrdCache/GlobalStaticsLog.txt ) is not very high, I was just like, WTF!
sigh well, only now did I finally realize that when people bump a thread from beyond page 4 it caused the bot to re-upload the file contents (ALL images in the thread) to SF, hence the bandwidth spike, and hence why I feel like I “took sever steps backwards,” because it’s clear now that I will have to turn the page-limiter off. Or to put it in more simple words, the programming I did a few nights ago is now for nothing ;_;
So, if anyone is reading this, learn from my mistakes and don’t lack in foresight... :＼
Dude. You're solving a problem that doesn't exist, I believe. dmpk2k already has a script for this.
Hm? You mean to mirror? Or the hole ImageBoard-Tracker thing?
…No The Mirror is just a quick thing I'm making to test a project that will give people the ability to seed ImageBoards, similar to the way people seed BitTorrent. Because I can't edit the pages directly on the ImageBoard servers (the project is still experimental, I wouldn't want to yet), I'm making a mirror of those pages and making the few small changes to them needed. The Primary change is taking every thumbnail link and changing them to something like;
http://70.187.193.222:178/ImageBoardTracker/SeedSupport.php?ReDirect=http://imageboards.4-ch.net/cosplay/src/1129534730252.jpg
(Adding the ImageBoard-Tracker's address before the target URL)
If you navigate to that URL, and you right click on the image and go to properties, you will notice that the image shown to you is NOT from imageboards.4-ch.net. It redirects to one of the “seeders” of the ImageBoard. Of course, this project is still in the experimental stage, so there are no real seeders, but I set one up with my SF space and edited the settings files so they see SF as a seeder.
Go to http://seeder-chan.sourceforge.net/ and read up on it there.
~crys~
Why what? I thought the ImageBoard-Tracker project was a good idea. It was originally intended for 4Chan as they kept dieing, and when they started to ask for donations to keep them selves afloat, (some) thought that people would be more likely to donate bandwidth than money as it was the main principle behind BitTorrent....
I think what he's doing is a good idea. Quit beating the guy over it. A simple mirror isn't that useful, but if that's what's needed in order to help develop his seeder concept, so be it.
I'm actually getting a little tired over people griping over others developing software. You should have seen the reaction of posters on 4chan to this, the bunch of idiots they be.
Then there's the whole Albright story. He develops something useful and people hound him. Fine, the first version was horribly ugly, but after he fixes it people still moan and groan. They come up with stupid shit like, "You're forcing people on wakachan to use it." I think it's amazing Albright hasn't walked away from the community in disgust.
Yes, I think Wakaba's the best, but I also think it should stand on its own merits. As for SWG's efforts, what he's slowly building could come in very useful one day.
Quit being a bunch of killjoys on development.
THANK YOU!~ I couldn't of better worded it then how you just did. I don't know what the "Albright story" is, but it in terms of the lack of respect people get for taking time out of their own lives to help in development, I totally know what you mean. It amazes me that people act like such assholes when we are doing this for them.
>>76
I would be overjoyed if Albright left iichan. It would be the greatest thing he's ever done for us.
And why is that?
> They come up with stupid shit like, "You're forcing people on wakachan to use it."
I came up with that first and I think it's perfectly valid criticism, esp. considering the fact that many people didn't like Thorn to begin with.
>I would be overjoyed if Albright left iichan. It would be the greatest thing he's ever done for us.
What ISP keeps putting in connections under bridges?
Must be AOL.
(That's not to say I myself am never guilty of trolling now and then, of course.)
> And why is that?
He's caused nothing but trouble since he started adminning.
About the only trouble I can see is the drama drama surrounding Thorn, and he seems quite civil about it considering the flames. Franky, it seems like much ado about nothing.
The existence of Thorn can only benefit the community. Cross-polination of ideas, you know?
Yeah, I see nothing wrong with the current Thorn interface, though I admit I thought the last one was crap. (Perhaps just becuase I'm used to seeing futaba-like imageboards.) And the ability to post multiple images is definitely a plus.
It would look better with Waha's CSS, but I really have no problem with the current look.
...I feel like wrote this before.
Reposting from the frontpage cache, which wasn't included in the restoration
94 Name: SomeWhiteGuy !obkj4fb50I : 2006-01-22 13:40 ID:55WL3xNm [Del]
L0L, Aahahahahah, good idea. Maybe I'm a pervert because that totally slipped my mind. Ok, just made one and posted it.
95 Name: Albright!LC/IWhc3yc : 2006-01-23 10:27 ID:Py9K8nis [Del]
Across all boards, I seem to be getting more "Fatal Error: The ImageBaord-Tracker's URL-source-list for that file seems to either deleted, or it's contents seems to have been erased" than actual images. I even tried uploading my own image and opening it, and got that error. What's causing this?
Seems like there's still a few glitches to iron out...
96 Name: SoeWhiteGuy !obkj4fb50I : 2006-01-29 15:53 ID:55WL3xNm [Del]
Well this is a bit overdue, but I finally got to making the Tracker able to install it self with Kareha and Thorn! During installation, it will inject code into Kareha/Thorn files if you give it the directory path, and it will auto update the templates(s). I setup some examples of them working on TestChan; http://somewhiteguy.homelinux.org:178/TestChan/
The /a/ section is using Kareha, and the /w/ section is using Thorn.
Right now I don’t have it working with Wakaba because I have to figure out how to get Perl to get access to MySQL, I think it needs something called DBI? But when I try to download it, it seems like it’s giving me files for Linux, ack, man just getting the freeking permission on Apache set to allow CGI scripts to work was painful enough (enabling Perl was), I need a break before I retry the DBI installation… unless anyone knows of any guides out there for it… which I couldn’t seem to find on Google.
97 Name: !WAHa.06x36 : 2006-01-29 16:10 ID:Heaven [Del]
On windows, use the ppd command to install it.
On Unix, use perl -MCPAN -e shell or perl -MCPAN -e "install DBI"
98 Name: Albright!LC/IWhc3yc : 2006-01-29 20:55 ID:Py9K8nis [Del]
Your Thorn installation seems pretty wonky; it's trying to link to stuff on 127.0.0.1. Please open up config.php and edit the THurl value to something sane.
99 Name: SomeWhiteGuy !obkj4fb50I : 2006-01-30 11:35 ID:55WL3xNm [Del]
>>97
O_O Wow, you guys need to tell me these things sooner. I had no idea it was so easy. Before I was trying to unzip contents off some package I found and trying to get settings files and environment paths set correctly... that ppm fucker did all the shit needed with one command line! :o
>>98
WWWWAAAAAAAAAAAA!!!~ Holy shit, that blind-sided me.... for like, the third time o_o Ugh... Ok, I think it should be fixed now, I think the problem was during installation I told it (either Thorn or the Tracker, I can't tell which one) to see itself as installed on 127~ instead of the host-name... blah
100 Name: !WAHa.06x36 : 2006-01-30 14:31 ID:Heaven [Del]
Shouldn't you be able to use SCRIPT_NAME to avoid needing an option like that?
101 Name: SomeWhiteGuy !obkj4fb50I : 2006-01-30 17:12 ID:55WL3xNm [Del]
>>100
Hm... This is pretty interesting, but I just noticed (from reading up at http://us3.php.net/reserved.variables ), that one can use this code (in PHP):
extract($_SERVER);
echo "http://" . $HTTP_HOST . $REQUEST_URI;
That should say public URL to the script... And for the local path, I think you could just use $PATH_TRANSLATED (from $_Server)
Pretty interesting, I just found that out myself, before when I needed the public (URL) and local paths I just had the user plug it in, and when it came time for them to do that I had a JavaScript on the user's browser plug it in automatically from looking at document.location
102 Name: Albright!LC/IWhc3yc : 2006-01-30 21:50 ID:Py9K8nis [Del]
>>100: Upon installation, Thorn makes a guess as to its own URL (and path) using the methods in >>101, but it also gives the installer the opportunity to change it if it guesses wrong. Either Thorn guessed wrong, or SWG changed it.
>>99: Hmm, image links are working now, but it seems to still be trying to get the thumbnails, CSS and JS code from 127... Try deleting the cache...
103 Name: SWG !obkj4fb50I : 2006-01-31 01:42 ID:55WL3xNm [Del]
Ok, fixed it, thought I fixed the THurl var, but guess I closed the text pad and didn't save it before because it was still going to 127~
Gah, this is annoying, and it's al because of my crappy LinkSys router (it's an older one). I dunno why the f**k it does this, but it seems to give me a "Connection Reset by Peer" error on all connections when I try to connect to my external IP address. It does this every so often anyways, but I have noticed it does this immediately after trying my external IP. If I hit it, within 10 seconds I get a CRbP error over IRC, and then other connections seem to break/reset to. Ugh....
So yeah, that's the cause of all of this bull s**t, otherwise I would use my external IP to connect to my server and there would never be any problems. Supposedly (I head from someone) that it's doing this as a security feature which was programmed a little to trigger happy... Blah I dunno, whatever, RRrrghh....
Funny, this thread jumped a post. Oh well, some breakage is to be expected, I guess.
dont u love how maericans always screw everything up? I can honestly say I hate them and I am one, god dam, this is just more proof of how completly retarded america is, I wish I lived in canada
>>96
is this a broken spambot, or is some idiot really going around bumping threads from 3 years ago to make wildly off-topic posts?
>>97
I wouldn't be surprised if it was the latter. And my has the quality of SOC really gone downhill as of late.
