Hi guys. This isn't so much an image board I'm doing, but a repository where you categorise and organise stuff.
Backstory: I've often thought at times that it's nice to have an organised image collection, but it's hard work. Do I sort them all by series, and then by other categories? What about this image that's both ecchi, but also has maids? What about a situation where you've got multiple characters where it belongs to no single series.
What I've devised isn't a perfect solution, but I think it could be practical and very nifty, if it's not too cumbersome. In short, you import a bunch of images into the system, and you can set flags for different attributes. There's about 20 such flags in the system as it stands, but that's extensible as far as your imagination wants to go. As you'll be able to see, I think I've got a fair number of "useful" ones covered, such as maid, miko, megane-ko, pantsu, gothic, etc. There's scope for a lot more, too.
Of course, all this tagging is of no use unless you can find what you want. So naturally you can just check a couple of boxes and say what you want. Example, I can immediately tell you that I have no miko+megane-ko images, but there are two maid+megane-ko images (and incidentally, they'll both show up if you check the box for "bondage" as well ^_^).
Another nice thing is that you can select great swathes of images and set flags across them. I mean, when you've just imported the latest volume of Muragami Suigun's F-ism collection, you don't want to sit there and tag each image manually, do you?
Anyway, I'd like to offer this to people to see what they think. There's no great restrictions on what you can do at the moment, so you can see all the features at work. You can add and delete categories, you can drop images, you can set flags en masse. And by all means do so. Naturally, I've taken a simple backup so I can fix things if people want to be silly buggers.
http://furinkan.ath.cx:8000/imgdb/
There's the database, please feel free to mess about with it. In the interests of saving peoples' sanity, I've limited the main page to display no more than 50 images at a time. My outgoing bandwidth sucks, and I'm hosting this at home. The thumbnails are small (~4K each) , but things will take long enough as it is.
What I'm looking for:
Any suggestions for improvement will be welcome. I know I should add lots of categories, and I'll do that as I think of them. One thing I'd like to do (and will need to figure out) is to have a hierarchy. Example: when you check "pantsu", the options for striped, spotted, etc should be available. Implementing this will be a matter of consistency, as having a sub-flag set must ensure that the higher up is set, and so on. I'm still learning, so you'll have to bear with me. Also being able to tag an image to series/serieses would be very nice, if difficult. I'll see...
Also, if anyone has an idea for a prettier interface, you could knock up a real simple screenshot of what you have in mind and perhaps link to it from here. I'm a competant web-coder, but I have practically zero aesthetic sense.
Please excuse the bits of SQL lying around everywhere. They're to help me figure out what I'm doing, and whether my selects are working.
For the technically inclined, the database uses some fairly simple PHP to do the frontend, and the backend is PostgreSQL. I was using MySQL before, but I felt I should be using a "real" DBMS for this. It's something new for me to learn as well. Also, I understand that Postgres will allow me to do some stuff that isn't supported in MySQL, such as subselects, and that would be invaluable/necessary when I implement queries like "megane-ko AND maid WITHOUT bondage". It's all running on a box called shirayuki, which you can see if you go up a level from the DB.
Have fun!
It's difficult to hold an open beta when it's that slow.
Ehe, sorry, there's not a lot I can do about it. The hosting that I do have access to is limited (ie. I can't make another table to run this thing, and that's a MySQL server anyway), and is something someone has offered me out of gratitude, so this is pretty much all I have.
FYI, I live in Australia. Our ex-Minister for Communications was the guy who proclaimed that broadband takeup was fuelled by demand for pornography, and essentially relegated us into being a technological backwater as far as internet connectivity. Stupid retard.
Anyway, I'll change shirayuki's bandwidth limiter to allow you a bit more outgoing bandwidth. Hopefully it'll help a little. My upload here is a pathetic 128kbps, and I need to courteously share this line with my family. I can't use all the upload, otherwise it'll slow them to a crawl. I'll adjust the limit on the number of results returned even lower if anyone thinks it'll help.
> FYI, I live in Australia. Our ex-Minister for Communications was the guy who proclaimed that broadband takeup was fuelled by demand for pornography,
Way to prove him wrong! Kidding, kidding.
Anyway, some thoughts, in no particular order:
Further expanding on the virtual filesystem idea, if you then decide to add uploading of images, you can let a user upload a file into /megane/nekomimi/, and the script will automatically set these tags on the new image.
Minister for Comms: Etto... well, he's right in some sense...
SQLite: Hmm, I might take a look at that. One of my priorities though is that it should be scalable and fast. I understand there's a certain elegance in (what I assume is) flat-files for database, but (correct me if I'm wrong) I understand they won't scale well. I'm willing to incur some performance overheads on the small scale if it'll stay relatively consistent up to a larger scale.
Yeah, I dunno what exactly to do about the mid-size images at the moment. They are an ugly hack, and it's especially nasty on wallpaper++ sized images.
Input: Yep. I'm generally pretty concious of security, and I know cross-site and SQL scripting is some of the worst stuff you can have happen. I don't think anyone's entirely immune, but we can do a lot to prevent getting shot in the foot. I'm pretty new to PHP and SQL, so this'll be something I have to learn about.
Path info: I must say, I like the sound of this a lot. Useless? Perhaps, but it could make peoples' life easier. The SQL queries will no doubt be equally ugly, but it makes for a nice and accessable interface. Could you tell me a little more about this thing? I haven't yet ascertained if it's a PHP or SQL thing, or how it'd work...
Thanks for the comments. I'll keep trying to tune and update things.
>>5
Funky! Yeah, I'll add upload support sometime, when I figure out how (and how to do so safely). I don't need people trying to exploit gd.
＿＿＿_∧∧ ／￣￣￣￣￣￣￣￣￣￣￣￣￣￣￣￣
～' ＿＿__（ ﾟДﾟ）＜ I DEMAND MORE PORNOGRAPHY!
ＵU Ｕ U ＼＿＿＿＿＿＿＿＿＿＿＿＿＿＿＿＿
You want more? Yeesh, okay... lemme just crack out F-ism 1-9 and dump them in. :)
Or if you want, I'll give you an FTP account to dump in. :P
Furi notices the new and tasty category added (presumably) by Waha
Arg, the blinking! My eyes!!
At least make it CSS compliant... ^_^
SQLite is pretty fast. You're not going to hit any performance problems with it unless you run a REALLY huge site. It's faster than mysql at a lot of things, too.
Also, a good first step for security is to run htmlspecialchars() on all input strings, always.
PATH_INFO is an environement variable that gets set by the web server whenever someone accesses your script like .../script.php/stuff/etc/. I forget exactly how to access environment variables in PHP, but I think it's something along the lines of $_SERVER["PATH_INFO"], which would in that example evaluate to /stuff/etc. It's then up to you to figure out what to do with it. In this case, parse the keywords, and construct a query from those.
Use htmlspecialchars() with the ENT_QUOTES parameter to remove single quotes that can mess up SQL queries and double-quotes that can mess up HTML generation. Right now, doing anything with the "Broken queries?" category fails because of a ' in the tagName.
>>3
Yeah. It's a cool idea. I just can't stand using the thing right now.
Where's the next page button?
You should make sure magic quotes are disabled to avoid really weird bugs showing up.
>>11 Interesting. I guess I should give it more serious consideration.
htmlspecialchars: Ah, so that's the one I want. Yeah, another consideration is that if/when I allow uploads, it has to ensure that those filenames are clean. I don't want to "accidentally" run sha1sum bleh.jpg;cat /etc/passwd or something equally insidious. I think I can cover that, though.
Path Info: Ah, cool. Thanks for the tip. That'll be a bit tougher to perfect, but it's nice.
Next page button? Etto, I'll hack one up for you... it'll be broken, but will have the basic functionality.
Magic quotes: I'll learn about that.
Thanks for your help, everyone. I'll try and fix these nasty bits sometime today.
Magic Quotes is one of the most wonderfully shitty ideas I've ever seen. Not only does it apply strange transformation to all sorts of text that doesn't need it, but even if you wanted to use it, you can't rely on it being turned on anyway.
In that specific case, you shouldn't be calling out to system() ever. Read the file yourself and use sha1().
>>17 Gah, just my luck, it's there under my nose. Hmm, now to see if there's a safe/r 'ls' in there...
Thanks!
This is a lot sexier then the filename hack/shell script I'm using to sort my images. I really want to set up one when the code gets released, except Dreamhost runs MySQL not Postgre. :( SQLite I can't find anything in the knowledge base about, but they have PHP, so it ought to work... Please switch! (or offer a choice between the two)
In the way of actual constructive commentary, the ability to let users rename files might be nice.
>>19 Man, this is all just a matter of knowing where to look... thanks, mate.
>>20 Hehe, in it's current incarnantion it could run on MySQL with minimal changes. I was doing that a little while ago before moving to something better, but I may be able to reconsider that if stuff like nested queries and stuff work in MySQL (I just have to figure out how).
As for the renaming, one of the issues I have at the moment is deciding how to treat the images. It should be possible to handle it if you throw a folder in the source directory, but I'm not sure how to do it yet. Every file will (should) hash uniquely, but if I move every file to a single directory, filename clashes will be possible. I can rename each file uniquely, but this would perhaps mess things up for users. If I find a way to handle folders, there's then an issue with duplicates. If you have a duplicate, it'll be ignored during import. Still, this'd only be a problem if you're keeping manga in there or something.
In short, the system is quite nice for images you've saved off 4chan/wakachan. ow to deal with limitations and further possibilities.
here's an idea. rather than making fields for a limited number of categories that you'll undoubtedly want to change/add to later, make each pic have a variable-length KEYWORD list (take a look at the 'SET' column data type in mySQL).
although, like the fixed number of categories, this has some inherent problems, such as people submitting things with mispelled or variant keywords. (there are a few ways to help fix that, like display a list of similarly spelled keywords, or all keywords, or make some filters.)
Just to let everyone know that progress is being made, but it's more on the backend of things. Things that are much better, like not using exec() when there are PHP functions already there for me. :)
Now with less faggotry.
Let's get this thread back on course, then.
lol at someone administratively losing it. :)
∧＿∧ ／￣￣￣
（ ´∀｀）＜ Look who's talking!
（ ） ＼＿＿＿
｜ ｜ |
（_＿）＿）
So sorry to have ruined your little wankfest, but this thread actually has a purpose, which was discussing image organization scripts. You are welcome to join in! Let me give you an example of how this works:
"Hi! I've made a similar script! You can see it at http://domain.com/path/. What do you think?"
>>31
you are aware that i did exactly that?
i posted a link to tma.pl, explained some of what it does?
well, of course you deleted the posts where i did that in your little private war against someone who is not me.
so, just to repeat myself ...
TMA does technical image matching by comparing the sha1 of the first 4kb of each image (to do truncation resolving) and by distance comparisons between 48dimensional idvector-locations assigned to each image by scaling it to 4x4 pixel and exporting it as RGB.
(perl, cgi, mysql, imagemagick, fuse)
http://baka.no-ip.biz/src/lootie/pubweb/tma.pl
if this makes any sense to you, feel free to join #tma on irc.freenode.net
Are you there, >>33? I have to let you know, there is a fool who has fallen in love with you.
It is me.
I am not a fag as you might think. I have no interest in homo things.
However, something began to swell greatly in me after I saw your posts.
Without even thinking, I came across one of your posts while I fapped.
Ah, even to me it was unpleasant. However, on a certain day I read one of your posts and started fapping.
From the beginning to the end I thought of you.
74 times... this is the number of times I have come reading your posts.
Perhaps, you think that you will reject me. But remember this: I will always be. near you
When you feel someone's hot breath on your neck on a crowded train, it is me.
When you notice someone staring at you in a strange way on the other side of the aracade, it is me.
When you pick up the phone and can only hear panting, it is me.
While you read this, I had already started...
>>33
Where can I see a demonstration of this script with relevant details for some existing pictures?
Hmm, this is most interesting... I've thought about fuzzy image-matching things before, purely for getting rid of "almost-the-same" duplicates. This is a rather cool idea, and certainly far better than doing some sort of edge-detection then matching up lines.
I can see this'd be useful for archiving a place like 4chan. Far too much duplicate crap to worry about...
Why are you using RGB searches to match images that are probably JPEG? Use YUV 4:2:0.
There's a lot that could improve on that. Using YUV, using the distance from the average instead of absolute values, normalizing...
>>41
actualy, "normalizing" is the problem that makes the first about 200k images in the DB less than good. at least what ImageMagicks idea of "normalize()" is. i ended up with idvectors distorted in different ways, depending on the input file format. (no, i dont get that either. but then again, IM is about as readable as my code)
using relative instead of absolute values for comparison is something i have been thinking about, same goes for greyscale of higher resolution than 4x4 or 8bit depth (would be very useful for matching manga/doujin pages).
i dont really understand what the advantage of YUV would be though, mostly because i dont really know what the difference would be.
currently i am holding back on dumping about 60-65 GB more images that are in a holding partition into the DB while i am working on the image-processor, stuff like alternate id-vectors is on the list of things i would like to figure out to avoid reprocessing even more than i have to by now already.
another is somehow converting embedded data like EXIF records into something usable...
ah, so much to do, so little time.
With "normalize" I wasn't talking about anything you'd do with ImageMagick.
Either way, I'm not really interested in helping out with this, seeing as how you're such an annoying and arrogant asshole.
>>43
Normalize
transform image to span the full range of color valuesEither way, I'm not really interested in your "help" there, seeing as how you're such an annoying "i dont know what it means but let me spew some random jargon i picked up somewhere anyways" asshole.
You want to turn this into an academic pissing contest, or what? Yes, I know that ImageMagick has a "Normalize" function. What I was talking about was normalizing in a much more mathematical sense. In your case, for instance, normalizing the distances from the average for each channel against the square root of the sum of the squares of the distances. There are probably other normalizations that might be useful too, that's just the first one to come to mind.
Also, you want to use YUV because it gives much more useful information in each channel, and you can weight the channels depending on whether you are more interested in brightness or colour differences.
> i dont really understand what the advantage of YUV would be though, mostly because i dont really know what the difference would be.
What >>45 said. Also, JPEG is already YUV so you don't have to waste time doing a colorspace conversion.
>>46
actualy, doing some more reading on the topic, it seems that JPEG uses a YUV-like system, but with a full 8bit-depth instead of the clamped value-range of real YUV.
so i end up converting all JPEGs to YUV anyways, or converting anything else to JFIF-YUV.
i mostly stay away from the video colorspaces since i have much less actual experience with those than i have with RGB.
also, a small "waste of time" that happens once per image while i have them open/loaded for rescaling anyways seems to be acceptable there. even more so when i see how making imagemagick not give me a "free" RGB conversion while opening anything might require more of my time than converting a lot of images does require _cpu_time.
otoh, the luminance part of YUV (or jfif-yuv) seems to be a step towards the "greyscale matching for manga pages" thing. but i wouldnt need any color values for that, just something like a 7x7 grid of 8bit depth (so i end up with a comparable amount of work/data as with the 4x4 RGB).
so considering that i can also do channel-based weighting on RGB just fine, i still dont see much point in using a YUVish colorspace for the comparisons.
To threadstarter: Any progress on your program? Code release, compatibility with MySQL/SQLite?
I told a bunch of people I'd be putting my image collection on the web and then discovered ochiba doesn't do what I want, so 困っている. orz
>>49
Eh, well, in a rather embarrassing turn of events, I was going to work on it a few nights ago, and managed to perform an 'rm -fr *' on the damned directory it was in. :(
I'll just have to code it from scratch again, which won't actually be too bad. It was relatively simple, and it'll be faster to code the second time 'round.
Ah, I was wondering why the site went down all of a sudden.
This is one of my reasons for releasing things as open source, as a matter of fact.
>>52
:P I hear you, alright. Tell you what, I'll try and package the php files that I've worked on every now and then, so if anyone's interested, they can attempt to decipher my awful code (I'm a php n00b, but that's why I'm learning).
I've been working on it, and the code's coming back slowly. If anyone wants to check it again, it's still there at furinkan.ath.cx:8000/imgdb/
I've tightened the checking on stuff like user-data insertion (eg. category names), so if anyone happens to break something, I'd love to know. I'll try and get some src up when I next work on it.
Now, something I'd like to ask people who might actually want to use it. I've used a bit of PEAR so far, so as to make the code a little "nicer". I'm also considering making more of the code OO, for the hell of it, but I'm concernded it'll break under PHP 4.x, which is probably what most webhosts would have installed (I don't know if you people just run servers at home, but if you pay a 3rd party host, I assume they have PHP4, MySQL and goddess-knows-what installed.
So. Can I use PEAR::DB, or do you want "hardcoded" SQL statements? I've also made use of PEAR::HTTP_Upload. Naughty? If so, I assume I can just copy and paste the code in from the package, it's just not as "nice" as having it installed with PEAR.
In short, if you can give me an idea of what sort of setup it needs to run on, I'll try and work around that. On my machine it runs with PHP 4.3.10 and Postgres 7.4.7. I'd like to use inheritence in Postgres, but I bet MySQL will break badly even thinking about stuff. I know people want some SQLite love, so I'll try and look at that too, but it'll be much less robust than a "proper" RDBMS; be warned.
Anyhow, any suggestions/criticisms are welcome. Apologies for the fact that as I write this, there's no "tagging" support (I spent a lot of time researching issues and making sure users can't do stupid/malicious things, as best I can). Oh, and there's no thumbnails yet either (that's my next step, once I can make it robust), so be prepared to wait a while if you want to see images (it's been resized on-the-fly in the HTML to 100x100).
Speaking of that, one last thing? GD or ImageMagick? What's supported by most people, and are there any compelling reasons for one over the other?
Thanks, guys.
I've noticed GD tends to make some ugly-ass thumbnails, at least if you're not careful. Maybe one can work around that, though.
Hmm, I've heard such things too. Well, they're thumbnails. It doesn't really matter if they're arse-ugly. That said, if you open more full-size images because you can't judge the thumbnail, that's a nono.
I've heard it may have something to do with the gd library version being >=2. Will have to investigate.
I went to that site, but there was only this place holder page. Nothing else would show up. Are you having domain problems?
>>57
Umm, if you read the post, you''ll see he was berating another user for being an idiot, and then provided an example of how one should post...
Perhaps this is more appropriate, for various reasons:
http://www.example.com/path/
Why doesn't that one work? See here: http://www.example.com/
It's odd using AOL - http://www.example.com is DISALLOWED FOR AOL MEMBERS....
After spending some time hammering away at the code, I think I've finally got this thing back to how it should be. I'm also an impatient bastard, so I paid to get a domain and hosted service. Now I can attempt to show this to people without things slowing to a crawl. I think. ^_^;
Things feel a bit slow at my end, but that's likely just a matter of geography (the server is located in Chicago, USA).
Significant changes are that now search criteria can be inserted via the URL. This isn't bad, and it's something I implemented after suggestion from Waha(?). Somewhere earlier in this thread. Anyway, it seems good, but it tends to break stuff. Want to use the add/delete links at the top of the page? You'd better not have anything after index.php on the URL...
Seriously, working around that stuff is hard. I might just go back to GET queries. It's really quite infuriating not knowing where you really are all the time (which is why you need a clean URL before you click one of the add/delete links).
Apologies for the lack of actual images there at the moment. I've pretty much only just got the hosting and stuff setup this evening, then I had to hack the PHP so it'd work at the new location.
Things I'll add next includes the ability to mass-tag (which is arguably the most useful feature of the damned thing) and characters/series. That way you can do searches for all images with Aria from Sister Princess. sigh of pleasure
Have fun (I hope):
http://www.meidokon.net/index.php
------------------------------
"Meidokon?"
Good question. While I can't find the little essay I wrote about this and "soft", the long and the short is that it's like "lolicon". Thus, Maid Complex. Usage is the same as for lolicon, incorrect as it is.
A site full of meidokon --> Correct!
I am a meidokon --> Correct!
He has a meidokon --> Never!
If you're easily amused, you can go and whois the domain for some rego info. ^_^
Another update:
Stuff has been cleaned up somewhat and some of the suck has been removed. The searching should be pretty damned intuitive now.
It's back to GET queries for searching, because they're nowhere near as messy to handle as PATH_INFO strings. My sanity just breathed a sigh of relief.
What can I say..? There's a terse changelog on the site, and I'd almost call the damned thing usable now. New step is characters and series.
Are you going to rename uploaded file names? Or truncate them, perhaps?
>>64
I'm considering it, yeah. I might do that if the length is greater than n characters, maybe n=20.
And when this is done, what then? This doesn't seem like it could be run on people's computers, purely a web-based script.
>>66
Well, that would depend on what kind of person you are. My machines are already set up to serve a web-based script. :)
Really, it's best suited to an open forum situation where people can add images. Sort of. The reason I made it web-based was mainly because I can show it off, and because it was fairly easy for me to do. I'll admit I've never developed real applications for a desktop before.
If you're interested, there's a piece of software I've heard about but not tried for myself, it's called ImgSeek. you throw images at it and I think you can put them in groups, a bit like my system. One of the cool features it has is heuristic query by example. The example they show you is a picture of a sunset. The system then goes and finds all pictures that it thinks look close to the source you supply. I think that's pretty cool, and it's similar to what someone described earlier on this thread. Anyway, it's free software, and it seems to work.
Main reason I stopped by was to drop another quick update. There's a few more entires in the changelog, and it now supports series marking. In a clunky fashion. I'll have to overhaul the interface part sometime.
You can set the series for a single image after setting its attributes. You can set the series for a bunch of images by marking them, clicking the button as usual, but then ignoring the attribute table. It's ugly, but it works. I'll overhaul the interface and split attrib/series tagging apart in the future.
I also got bored, so the default page is now index.html instead of php. You get a disclaimer thing and a cookie I hacked up. Wai~! I've never used cookies before. They're so easy! :)
Oh, and there's source now. Ehe, I meant to say that, but then I noticed anon had written something. Yes, there's a small link on the splash page where you can grab all of the ugly code I've written in a gzipped tarball. Have fun, suggestions are welcome.