Hi guys. This isn't so much an image board I'm doing, but a repository where you categorise and organise stuff.
Backstory: I've often thought at times that it's nice to have an organised image collection, but it's hard work. Do I sort them all by series, and then by other categories? What about this image that's both ecchi, but also has maids? What about a situation where you've got multiple characters where it belongs to no single series.
What I've devised isn't a perfect solution, but I think it could be practical and very nifty, if it's not too cumbersome. In short, you import a bunch of images into the system, and you can set flags for different attributes. There's about 20 such flags in the system as it stands, but that's extensible as far as your imagination wants to go. As you'll be able to see, I think I've got a fair number of "useful" ones covered, such as maid, miko, megane-ko, pantsu, gothic, etc. There's scope for a lot more, too.
Of course, all this tagging is of no use unless you can find what you want. So naturally you can just check a couple of boxes and say what you want. Example, I can immediately tell you that I have no miko+megane-ko images, but there are two maid+megane-ko images (and incidentally, they'll both show up if you check the box for "bondage" as well ^_^).
Another nice thing is that you can select great swathes of images and set flags across them. I mean, when you've just imported the latest volume of Muragami Suigun's F-ism collection, you don't want to sit there and tag each image manually, do you?
Anyway, I'd like to offer this to people to see what they think. There's no great restrictions on what you can do at the moment, so you can see all the features at work. You can add and delete categories, you can drop images, you can set flags en masse. And by all means do so. Naturally, I've taken a simple backup so I can fix things if people want to be silly buggers.
http://furinkan.ath.cx:8000/imgdb/
There's the database, please feel free to mess about with it. In the interests of saving peoples' sanity, I've limited the main page to display no more than 50 images at a time. My outgoing bandwidth sucks, and I'm hosting this at home. The thumbnails are small (~4K each) , but things will take long enough as it is.
What I'm looking for:
Any suggestions for improvement will be welcome. I know I should add lots of categories, and I'll do that as I think of them. One thing I'd like to do (and will need to figure out) is to have a hierarchy. Example: when you check "pantsu", the options for striped, spotted, etc should be available. Implementing this will be a matter of consistency, as having a sub-flag set must ensure that the higher up is set, and so on. I'm still learning, so you'll have to bear with me. Also being able to tag an image to series/serieses would be very nice, if difficult. I'll see...
Also, if anyone has an idea for a prettier interface, you could knock up a real simple screenshot of what you have in mind and perhaps link to it from here. I'm a competant web-coder, but I have practically zero aesthetic sense.
Please excuse the bits of SQL lying around everywhere. They're to help me figure out what I'm doing, and whether my selects are working.
For the technically inclined, the database uses some fairly simple PHP to do the frontend, and the backend is PostgreSQL. I was using MySQL before, but I felt I should be using a "real" DBMS for this. It's something new for me to learn as well. Also, I understand that Postgres will allow me to do some stuff that isn't supported in MySQL, such as subselects, and that would be invaluable/necessary when I implement queries like "megane-ko AND maid WITHOUT bondage". It's all running on a box called shirayuki, which you can see if you go up a level from the DB.
Have fun!
Categories for everything is a win. If you take a look at how major Digital Asset Management systems work, you'll note they use categories for everything, as well as ITCP/EXIF/XMP metadata per individual image (or video or audio asset).
Righty-o then. Categories it is. When I get some more time to myself I'll shove it all over to the unified system. At the moment the code is a horrible mess of SELECT queries to both the old and new linkings. One thing that gets me is that I can't foreign key anymore. It really shits me, and I can't see any way around it. Dunno if I can create a view of the seperate "targets" by doing a UNION of selects of all the tables for each metacategory. Seems ugly, though. I might be able to solve it with CHECK constraints, but that won't be possible unless I suddenly discover I can use if clauses or something. (something like CHECK that "data" is a member of table X when type is "series")
Well, maybe you'll want to try out a category system yourself first, see if you like it. There's a free one, named Picasa2, that google is releasing for free. You'll need windows for it.
To be honest, Picasa2 is piss poor as a DAM, but it'll give a good idea how the more advanced ones do it.
Now, an online website DAM would be amazing. If you add categories, then maybe versioning one day in the distant future, you'd have that all locked up. OTOH, I really doubt an online DAM for the public would need versioning.
The great thing about such a system is it makes categorizing things easier. Instead of spending days categorizing images yourself, let the monkeysvisitors do it. Well, we hope they will, but I digress.
Yeah, I've heard about Picasa. I guess I should try it sometime. Won't know otherwise.
How do you define DAMs in terms of functioanlity. Anything key features that make a DAM a DAM? Hmm, versioning... it could be done. nastily.
(just noticed that I typo'ed DAM and got FSM. Heh, finite state machine...)
The only real requirement for a DAM is that there's a robust cataloging/tagging/identifying/whatever mechanism. Asset Management is exactly as it sounds.
The versioning is just gravy, although it's crucial for professionals. In the context of an online DAM, I suppose it could be used to stack edits together. For example, all the edits that pass through IItran and /os/, and maybe even originals images and derived wallpapers.
There's also colour management, but that'd be a useless feature for an web-based DAM.
Anyway, I do recommend giving Picasa a spin. I don't like it (I use iView MediaPro), but it's easy to use and leaves no doubts where I'm coming from. It'll also give you ideas on what you may want to implement in your own system, once you start banging up against Picasa's numerous limitations. If you have a Mac, try iPhoto.
The no.1 thing I like about yours is that the metadata is all in an open format, namely SQL. Whatever you do, don't give that up. Professional systems that also export SQL cost thousands to license.
Robust, eh? Well, that's exactly what I want to make it, so we're on the right track. The things I've been keeping in mind so far:
A degree of portability - It ran on MySQL first, because that's what my textbook is. :) Then it moved to Postgres, partly because I could, but also because I have this idea in my mind that it's a Real Database. Let's face it, when the default table type for your DB is MyISAM, you've got issues. MySQL alledgedly has better performance, but it's a tradeoff for other more useful features. Either way, it's SQL. Which is just sane, right? I'm an activist, but I'm one of the last people you'll see using something proprietary or with strings attached. How could I ask people to use it otherwise? It should also be usable with SQLite, with some minor changes to the PEAR_DB, but I don't know if SQLite offers the necessary robustness.
Robustness - This basically means paranoia everywhere I can put it. I figure that anything could happen, and I know it's a Bad Thing to make assumptions. Using an ACID backend is of course a Good Thing. The system should recover gracefully from errors/dropouts at any point.
Performance - Okay, there's a performance hit from using Postgres, but that's a necessity. Doesn't mean I'll write bad code if I can avoid it.
Usability - One of the reasons I've not made the UI pretty is because that would require artistic talent. It's also because it'll probably use Javascript and other stuff that's inconsistent across browsers. Likewise, I'm not using cookies or logins because they're a pain in the arse. (the no-mature-content cookie is a triviality, and the site works perfectly without it)
Modularity - I'm somewhat unsatisfied with how this part is going, but I'd like to imagine that blocks of code can be drop-in replaced to enhance existing sections. I understand that not everyone has ImageMagick available and whatnot.
The idea of versioning sounds interesting, but I'm not sure how it'd apply to a situation like mine. Perhaps I just don't know how the DB could be used in such circumstances.
Well, while I quite agree with using PostgreSQL (ask WAHa about his hate-hate relationship with MySQL), by robust I meant a category system that's sufficiently flexible and extensible that it can handle a near-arbitrary number of classifications on any single image. Conceptually this is pretty easy:
Main
Ref # Category Image etc...
1 2 1 ...
2 2 2 ...
3 2 3 ...
4 3 1 ...
5 1 4 ...Have Category serve as a foreign key to the Category table, and Image serve as foreign key to the Image table. Ref is the primary key, Image is not, etc. In order to find all images in a category, you'd do something like a select * from main where category = 2, and to find all categories for a particular image you're do a select * from main where image = 1.
Attributes that are specific to an image, or other image-specific metadata can be stored in the Image table. Same with Categories, etc.
Now, optimizing this may be a bit more complicated. The Main table could get quite big if you're dealing with many images and categories.
Versioning is pretty simple, actually, if you decide to go the everything-is-a-category route. Use hidden categories that group versions of the same image together. They're exactly the same as user-created categories, except only the system uses it.
Forgot to mention: while you'd use hidden categories to group versions of the same image together, you'd put the version ID in an attribute in the Image table if you're dealing with explicit versioning. It's not enough to know a bunch of images are versions of each other, but also in which order they were made.
New Robust: Ah, I already do that. :)
The table file_to_cat looks exactly like you describe, except that there's no artificial primary key. The file and cat column together make a primary key, and this has worked nicely. As for optimisation, well, it's been nice so far. I had an absolutely mind-blowing moment some weeks ago when a friend told me how to make a hellishly elegant query finding multiple entries of img X with category A,B,C set. I think it might even scale linearly, but it's not something I've investigated. It's nice, though.
Versioning: Ah, I see. That's simpler than I thought it might be.
Quick update. The backend has been generalised a lot, meaning that with a little more work, it could easily be adapted to most situations. As it is, it's using a fixed set of meta-categories. Handling a variable number of meta-categories will require extra work, and is actually somewhat undesirable in these circumstances. Characters have ties to series, and a more "vanilla" implementation wouldn't have such linking (I think).
Now that an image can belong to multiple series, it'll take a little more work to figure out how to deal with characters. I'd display all possible characters on the index page, but it's huge and ugly. I'll... work some interface thing out.
ok, so SQLite decided to eat my ochiba database. Looks like I'll have to redo all the categorizing; before that, are there any other image repository scripts in the woodwork I could consider switching to? I'd really like one with the ability to search for multiple keywords, and all the 'PHP sucks' talk on 4-ch is getting to me.
http://danbooru.donmai.us/ would work beautifully, if only dreamhost offered PostgreSQL.
>>111
there are two tagging/categorization systems that can be found by visiting irc://irc.freenode.net/tma ... but i guess that was already mentioned here.
and no, i do not mean TMA itself, that one is still just working with auto-generated low-quality metadata. but the authors of other image-db things hang out there too ...
How badly do you need Postgres? Depending on functionality, danbooru might be able to be converted to MySQL. Meidokon could use MySQL with some work, I know that much. You could switch to another host that supports Postgres, but it'd probably be more expensive. We like Postgres just because it's more robust and "proper" about what it does.
Hello, I am a member of Team Danbooru.
If I recall, there are only two places where danbooru relies on pgsql features: the rules on posts_tags (to automatically increment tags.post_count) and the pgplsql stored procedures to perform related post count. The former can easily be moved into the app and the latter you don't really need, you can just rely on tags.post_count instead.
There are plans to offer a SQLite version in the future, but right now we're busy whoring Postgres for all she's worth.
but right now we're busy whoring Postgres for all she's worth.
Amen, brother.