Experimental image "board"-type thing (118)

1 Name: Furi!EuK0M02kkg 2005-04-05 03:21 ID:Heaven [Del]

Hi guys. This isn't so much an image board I'm doing, but a repository where you categorise and organise stuff.

Backstory: I've often thought at times that it's nice to have an organised image collection, but it's hard work. Do I sort them all by series, and then by other categories? What about this image that's both ecchi, but also has maids? What about a situation where you've got multiple characters where it belongs to no single series.

What I've devised isn't a perfect solution, but I think it could be practical and very nifty, if it's not too cumbersome. In short, you import a bunch of images into the system, and you can set flags for different attributes. There's about 20 such flags in the system as it stands, but that's extensible as far as your imagination wants to go. As you'll be able to see, I think I've got a fair number of "useful" ones covered, such as maid, miko, megane-ko, pantsu, gothic, etc. There's scope for a lot more, too.

Of course, all this tagging is of no use unless you can find what you want. So naturally you can just check a couple of boxes and say what you want. Example, I can immediately tell you that I have no miko+megane-ko images, but there are two maid+megane-ko images (and incidentally, they'll both show up if you check the box for "bondage" as well ^_^).

Another nice thing is that you can select great swathes of images and set flags across them. I mean, when you've just imported the latest volume of Muragami Suigun's F-ism collection, you don't want to sit there and tag each image manually, do you?

Anyway, I'd like to offer this to people to see what they think. There's no great restrictions on what you can do at the moment, so you can see all the features at work. You can add and delete categories, you can drop images, you can set flags en masse. And by all means do so. Naturally, I've taken a simple backup so I can fix things if people want to be silly buggers.

http://furinkan.ath.cx:8000/imgdb/

There's the database, please feel free to mess about with it. In the interests of saving peoples' sanity, I've limited the main page to display no more than 50 images at a time. My outgoing bandwidth sucks, and I'm hosting this at home. The thumbnails are small (~4K each) , but things will take long enough as it is.

What I'm looking for:

Any suggestions for improvement will be welcome. I know I should add lots of categories, and I'll do that as I think of them. One thing I'd like to do (and will need to figure out) is to have a hierarchy. Example: when you check "pantsu", the options for striped, spotted, etc should be available. Implementing this will be a matter of consistency, as having a sub-flag set must ensure that the higher up is set, and so on. I'm still learning, so you'll have to bear with me. Also being able to tag an image to series/serieses would be very nice, if difficult. I'll see...

Also, if anyone has an idea for a prettier interface, you could knock up a real simple screenshot of what you have in mind and perhaps link to it from here. I'm a competant web-coder, but I have practically zero aesthetic sense.

Please excuse the bits of SQL lying around everywhere. They're to help me figure out what I'm doing, and whether my selects are working.

For the technically inclined, the database uses some fairly simple PHP to do the frontend, and the backend is PostgreSQL. I was using MySQL before, but I felt I should be using a "real" DBMS for this. It's something new for me to learn as well. Also, I understand that Postgres will allow me to do some stuff that isn't supported in MySQL, such as subselects, and that would be invaluable/necessary when I implement queries like "megane-ko AND maid WITHOUT bondage". It's all running on a box called shirayuki, which you can see if you go up a level from the DB.

Have fun!

69 Name: 743 2005-06-23 04:47 ID:XHSIYtC4 [Del]

There are some categories that mean the same thing but have different content and names.

Examples:
Loli/lolita
Catgirl/nekomimi

Should that be fixed?

70 Name: 743 2005-06-23 04:49 ID:XHSIYtC4 [Del]

about >>69
I was talking about meidokon's categories in case you didn't know.
I could fix the problem.

71 Name: 743 2005-06-23 04:56 ID:XHSIYtC4 [Del]

I think I'll just fix it...

72 Name: Furi!EuK0M02kkg 2005-06-23 17:27 ID:Heaven [Del]

Yeah, it's something that's been considered. Loli and Lolita have been planned for merging, as can be seen on the to-do list. Catgirl and Nekomimi are different things, though, but would be better handled in a hierarchical fashion. I'd accept mathematically that "catgirl->nekomimi". Ie. catgirl practically enforces nekomimi, but NOT the other way around. How you want to handle it, though, is open to argument.

I see you got rid of Nekomimi. Fair enough, I just pray to Goddess that:
1) You gave all the "nekomimi" pictures the "catgirl" tag first
2) Made good use of the mass-tagging feature

As for your comment in >>69, the categories have different content and names, but they certainly don't mean the same thing. Anyone who is confusing Loli and Lolita is a fool. I did put rollover info on them as well with <abbr>, but I can accept that there are browsers that won't render it. That said, I'll merge Loli and Lolita because they're very hard to effectively categorise data with.

Nekomimi and Catgirl aren't the same thing either. Hazuki (Tsukuyomi ~Moon Phase~) wears cat ears. But they're plainly not real. She's not a catgirl, she's a vampire. Hence the different categories.

73 Name: Anonymous 2005-06-24 14:20 ID:Heaven [Del]

>I'd accept mathematically that "catgirl->nekomimi".
>Nekomimi and Catgirl aren't the same thing either. Hazuki (Tsukuyomi ~Moon Phase~) wears cat ears. But they're plainly not real. She's not a catgirl, she's a vampire. Hence the different categories.

Sounds like "nekomimi->catgirl" is truer rather than the other way around, as it can include both natural and costume ears.

74 Name: Anonymous 2005-06-25 04:49 ID:Heaven [Del]

-> (or =>) is implies, that's different from > or contains. => catgirl->nekomimi is correct, while nekomimi->catgirl is not.

75 Name: Anonymous 2005-06-26 12:12 ID:Heaven [Del]

>>74

Uh... "implies" IS "contains". We're talking sets here, right?

76 Name: Furi!EuK0M02kkg 2005-06-26 18:47 ID:Heaven [Del]

Actually, no, it's the opposite. (I hated this Uni maths as well).
It's not mathematical sets, this is to do with logic (but we'll deal with sets in a moment). The use of implies in "A implies B" means that If A, then B, but it's also important that Not-B does not imply Not-A.

I like this analogy someone once gave me. A implies B:
If you study for the test, you can pass. True.
If you don't study for the test, you can't pass. True.
If you don't study for the test, you can pass. True (fluke/cheat)
If you study for the test, you can't pass. False.

So in our case,
You're a catgirl, you have nekomimi.
You're not a catgirl, you might not have nekomimi.
You're not a catgirl, you might still have nekomimi.
You're a catgirl, you don't have nekomimi. WRONG!

As for sets, yes, it's logical to think of catgirl as a subset of nekomimi, and if/when I implement such things, it may involve set theory instead of logic (well, it's all logic, really). The problem here is twofold.

  1. I've forgotten all my set notation.
  2. I probably couldn't show the symbols here even if I could remember the stuff I needed.

77 Name: Anonymous 2005-06-26 23:12 ID:zV95B+ZR [Del]

The use of implies in "A implies B" means that If A, then B, but it's also important that Not-B DOES imply Not-A.

78 Post deleted by user.

79 Name: Anonymous 2005-06-27 03:23 ID:Heaven [Del]

A  B
T T T
T F F
F T T
F T F

is the same as

~B  ~A
F T F
T F F
F T T
T T T

80 Name: Anonymous 2005-06-27 03:24 ID:Heaven [Del]

Also, "logics" is too broad a term and is not neccessarily contrary, equal or contradictory to set theory.

81 Post deleted by user.

82 Name: Furi!EuK0M02kkg 2005-06-27 07:29 ID:Heaven [Del]

>>77
Ah yes, true/T/1. :P
I can't remember what that's called. The converse or the inverse...

>>79
Er, wha? In the first table I see two rows where A=F and B=T.
The second table also has two rows like that.
This is a useful reference:
http://www.omnifarious.org/~eljay/info_ttable.html

>>80
Agreed. Apologies for my poor knowledge of the nomenclature.
Anyway, the point is here that the setting/usage of certain categories may affect the use of other categories. It needs to be dealt with in some way.

83 Name: Anonymous 2005-06-27 07:59 ID:Heaven [Del]

> Er, wha? In the first table I see two rows where A=F and B=T.

It doesn't matter. What matters is the truth value characteristic of the expression which is identical for A B and ~B ~A ( = T F T T).

84 Name: Anonymous 2005-06-27 08:01 ID:Heaven [Del]

PS: In truth tables, you look at the columns for matters of truth or falsity of the given operator or atomic sentence particle. The rows only matter insofar as you want to test matters of validity if the given expression is an argument (if all premises are true in one case (row) then the conclusion must be true as well, otherwise the argument is invalid).

85 Post deleted by user.

86 Name: Furi!EuK0M02kkg 2005-06-27 10:44 ID:Heaven [Del]

Say what? Are you trying to tell me that your result columns are listing possible outcomes? That's the only way I can see it at the moment.

A truth table is meant to show you the result of an operator acting on inputs. As your table is, there are two distinct rows with identical inputs and a different output in each. This implies inconsistency and/or an undefined nature. In short, it looks broken to me.

But anyway, we're somewhat off topic. In other news, meidokon is still adding features, and that support for characters is nearing completion. I'll just have to sit down sometime and hammer it out. There's probably a better way to handle it, but the source is really ugly and it goes through a tree of possibilities to formulate the SQL query string (are there any category constraints, is there a series specified? If so, have they specified characters from that series?)

A friend suggested a "fun" idea whereby meidokon boards could talk to one another to pass img metadata. An image added to the DB on one board could pull the tags and other metadata across on request, saving effort, presumably. And it'd just be kinda cool.

87 Name: Anonymous 2005-06-27 21:42 ID:Heaven [Del]

> there are two distinct rows with identical inputs and a different output in each.

I think the "output" is the middle column, not the third column.

Are you going to add "artist" categories as well?

88 Name: Anonymous 2005-06-28 00:16 ID:Heaven [Del]

> A truth table is meant to show you the result of an operator acting on inputs.

Which is the column underneath the operator - which is the same for the operator defining the meaning of the expressions itself.

89 Name: Furi!EuK0M02kkg 2005-06-28 06:19 ID:Heaven [Del]

Okay, then, my apologies. I've always had truth tables as A,B,Out.

Artist: It's something I've been considering, yes. I'll finish work on the characters first, though. For the moment, an artist can be a category unto themself.

90 Name: Anonymous 2005-06-29 00:58 ID:Heaven [Del]

Wait... are you assuming there's only one "series" per image? That's not always the case.

91 Name: Furi!EuK0M02kkg 2005-06-29 05:29 ID:Heaven [Del]

I know, but it'll make life very difficult otherwise. It's something I've considered and may revise in the future.

92 Name: !WAHa.06x36 2005-06-29 05:51 ID:YBfFGBQ7 [Del]

Why wouldn't you want to just use categories for characters, series, and artists? That seems much more flexible to me, and less work. What you want instead is meta-categories: A way to group categories under different labels. So you can just say that category so-and-so represents an artist, and category so-and-so a character. Then you just group them appropriately in the interface.

93 Name: !WAHa.06x36 2005-06-29 06:00 ID:YBfFGBQ7 [Del]

Also, you might want to turn those category lists into some sort of popups at some point, because it's annoying to have to scroll past them at every page.

94 Name: !WAHa.06x36 2005-06-29 06:02 ID:YBfFGBQ7 [Del]

>>92 would also make it easier to use the software for other things than strictly anime art. You can just set up categories and meta-categories in suitable ways for whatever you want to store.

95 Name: Anonymous 2005-06-29 07:59 ID:Heaven [Del]

XML GET!

96 Name: Furi!EuK0M02kkg 2005-06-29 08:23 ID:Heaven [Del]

True. You reckon it's good to have everything as a category? I'm just a little worried about consistency, but that's more of a vague worry in my head than anything I could substantiate at the moment. I can already see that artists as a category is suitable. I suppose I can chuck everything in like that. I'm just worried things will get "messy" if it's not enforced the way it is at the moment. Ah, hell with it... means I have to get off my arse and make hierarchy (meta-categories as you called them) work. Still, no better time than the present. I was about to go and make characters "work".

This could be interesting, actually. Forming data chains whereby an entry can have a pointer to another entry. This sounds fun already! Rawr!

>>93 Haihai~, yeah, they're a pain in the arse as well. I've been thinking some fancy javascript sometime to tuck them away. My focus so far has been on compatibility, so we'll see how it goes (just because it's useless doesn't mean it shouldn't work in links, dammit!!).

>>95 Oh, you want XML now? Go on, I'm listening.

97 Name: !WAHa.06x36 2005-06-29 10:20 ID:Zmmu0d7B [Del]

>>96

You don't need Javascript to make hidden lists - CSS will do just fine. A quick google search should find you several tutorials on how to do it. The trick is using :hover selectors and display:none.

Of course, you can create more fancy stuff with Javascript, but then you're limiting usability for people who have it turned off.

98 Name: !WAHa.06x36 2005-06-29 11:44 ID:Zmmu0d7B [Del]

Also, I've been playing around with making a similar interface, but using the OS X Spotlight metadata database as the backend. We'll see how that turns out though.

99 Name: Furi!EuK0M02kkg 2005-06-29 11:57 ID:Heaven [Del]

I was thinking CSS too, but support for that can be shakier than JS. I think that IE7 script helps that, though.

There's been some discussion about Spotlight. Something to do with smart folders as well. But, I'm not a mac person, so I don't know anything about it.

Just been augmenting my code and the DB. The DB now has a table called file_to_target that holds all the metadata links. There's three columns, file,type,data. File is the sha1 as always, type is now a small set of metacategories (foreign-keyed to another small table). Data can now hold whatever you want, which in theory is bad because you can't foreign key it to anything you control (you'd have to foreign key to a union of table data). Just have to use PHP checks to do constraint checking, I guess.

The code I just fixed means it should be updating metadata in the new and old locations. Once I have it all nice, I can switch exclusively to the new unified table and dump the rest.

Right now, however, is time for sleep (0400). If I'm doing anything stupidly wrong, just let me know. :)

100 Name: Anonymous 2005-06-29 13:44 ID:Heaven [Del]

Covertly got 100

101 Name: dmpk2k!hinhT6kz2E 2005-06-30 02:25 ID:P6nsARJb [Del]

Categories for everything is a win. If you take a look at how major Digital Asset Management systems work, you'll note they use categories for everything, as well as ITCP/EXIF/XMP metadata per individual image (or video or audio asset).

102 Name: Furi!EuK0M02kkg 2005-06-30 03:59 ID:Heaven [Del]

Righty-o then. Categories it is. When I get some more time to myself I'll shove it all over to the unified system. At the moment the code is a horrible mess of SELECT queries to both the old and new linkings. One thing that gets me is that I can't foreign key anymore. It really shits me, and I can't see any way around it. Dunno if I can create a view of the seperate "targets" by doing a UNION of selects of all the tables for each metacategory. Seems ugly, though. I might be able to solve it with CHECK constraints, but that won't be possible unless I suddenly discover I can use if clauses or something. (something like CHECK that "data" is a member of table X when type is "series")

103 Name: dmpk2k!hinhT6kz2E 2005-06-30 05:35 ID:KRr1OanE [Del]

Well, maybe you'll want to try out a category system yourself first, see if you like it. There's a free one, named Picasa2, that google is releasing for free. You'll need windows for it.

To be honest, Picasa2 is piss poor as a DAM, but it'll give a good idea how the more advanced ones do it.

Now, an online website DAM would be amazing. If you add categories, then maybe versioning one day in the distant future, you'd have that all locked up. OTOH, I really doubt an online DAM for the public would need versioning.

The great thing about such a system is it makes categorizing things easier. Instead of spending days categorizing images yourself, let the monkeysvisitors do it. Well, we hope they will, but I digress.

104 Name: Furi!EuK0M02kkg 2005-06-30 09:24 ID:Heaven [Del]

Yeah, I've heard about Picasa. I guess I should try it sometime. Won't know otherwise.

How do you define DAMs in terms of functioanlity. Anything key features that make a DAM a DAM? Hmm, versioning... it could be done. nastily.

(just noticed that I typo'ed DAM and got FSM. Heh, finite state machine...)

105 Name: dmpk2k!hinhT6kz2E 2005-06-30 19:48 ID:k7OpDZBS [Del]

The only real requirement for a DAM is that there's a robust cataloging/tagging/identifying/whatever mechanism. Asset Management is exactly as it sounds.

The versioning is just gravy, although it's crucial for professionals. In the context of an online DAM, I suppose it could be used to stack edits together. For example, all the edits that pass through IItran and /os/, and maybe even originals images and derived wallpapers.

There's also colour management, but that'd be a useless feature for an web-based DAM.

Anyway, I do recommend giving Picasa a spin. I don't like it (I use iView MediaPro), but it's easy to use and leaves no doubts where I'm coming from. It'll also give you ideas on what you may want to implement in your own system, once you start banging up against Picasa's numerous limitations. If you have a Mac, try iPhoto.

The no.1 thing I like about yours is that the metadata is all in an open format, namely SQL. Whatever you do, don't give that up. Professional systems that also export SQL cost thousands to license.

106 Name: Furi!EuK0M02kkg 2005-07-01 07:48 ID:Heaven [Del]

Robust, eh? Well, that's exactly what I want to make it, so we're on the right track. The things I've been keeping in mind so far:

A degree of portability - It ran on MySQL first, because that's what my textbook is. :) Then it moved to Postgres, partly because I could, but also because I have this idea in my mind that it's a Real Database. Let's face it, when the default table type for your DB is MyISAM, you've got issues. MySQL alledgedly has better performance, but it's a tradeoff for other more useful features. Either way, it's SQL. Which is just sane, right? I'm an activist, but I'm one of the last people you'll see using something proprietary or with strings attached. How could I ask people to use it otherwise? It should also be usable with SQLite, with some minor changes to the PEAR_DB, but I don't know if SQLite offers the necessary robustness.

Robustness - This basically means paranoia everywhere I can put it. I figure that anything could happen, and I know it's a Bad Thing to make assumptions. Using an ACID backend is of course a Good Thing. The system should recover gracefully from errors/dropouts at any point.

Performance - Okay, there's a performance hit from using Postgres, but that's a necessity. Doesn't mean I'll write bad code if I can avoid it.

Usability - One of the reasons I've not made the UI pretty is because that would require artistic talent. It's also because it'll probably use Javascript and other stuff that's inconsistent across browsers. Likewise, I'm not using cookies or logins because they're a pain in the arse. (the no-mature-content cookie is a triviality, and the site works perfectly without it)

Modularity - I'm somewhat unsatisfied with how this part is going, but I'd like to imagine that blocks of code can be drop-in replaced to enhance existing sections. I understand that not everyone has ImageMagick available and whatnot.

The idea of versioning sounds interesting, but I'm not sure how it'd apply to a situation like mine. Perhaps I just don't know how the DB could be used in such circumstances.

107 Name: dmpk2k!hinhT6kz2E 2005-07-01 08:39 ID:K6oPXZVS [Del]

Well, while I quite agree with using PostgreSQL (ask WAHa about his hate-hate relationship with MySQL), by robust I meant a category system that's sufficiently flexible and extensible that it can handle a near-arbitrary number of classifications on any single image. Conceptually this is pretty easy:

Main
Ref # Category Image etc...
1 2 1 ...
2 2 2 ...
3 2 3 ...
4 3 1 ...
5 1 4 ...

Have Category serve as a foreign key to the Category table, and Image serve as foreign key to the Image table. Ref is the primary key, Image is not, etc. In order to find all images in a category, you'd do something like a select * from main where category = 2, and to find all categories for a particular image you're do a select * from main where image = 1.

Attributes that are specific to an image, or other image-specific metadata can be stored in the Image table. Same with Categories, etc.

Now, optimizing this may be a bit more complicated. The Main table could get quite big if you're dealing with many images and categories.

Versioning is pretty simple, actually, if you decide to go the everything-is-a-category route. Use hidden categories that group versions of the same image together. They're exactly the same as user-created categories, except only the system uses it.

108 Name: dmpk2k!hinhT6kz2E 2005-07-01 08:43 ID:K6oPXZVS [Del]

Forgot to mention: while you'd use hidden categories to group versions of the same image together, you'd put the version ID in an attribute in the Image table if you're dealing with explicit versioning. It's not enough to know a bunch of images are versions of each other, but also in which order they were made.

109 Name: Furi!EuK0M02kkg 2005-07-01 10:21 ID:Heaven [Del]

New Robust: Ah, I already do that. :)
The table file_to_cat looks exactly like you describe, except that there's no artificial primary key. The file and cat column together make a primary key, and this has worked nicely. As for optimisation, well, it's been nice so far. I had an absolutely mind-blowing moment some weeks ago when a friend told me how to make a hellishly elegant query finding multiple entries of img X with category A,B,C set. I think it might even scale linearly, but it's not something I've investigated. It's nice, though.

Versioning: Ah, I see. That's simpler than I thought it might be.

110 Name: Furi!EuK0M02kkg 2005-07-03 08:23 ID:Heaven [Del]

Quick update. The backend has been generalised a lot, meaning that with a little more work, it could easily be adapted to most situations. As it is, it's using a fixed set of meta-categories. Handling a variable number of meta-categories will require extra work, and is actually somewhat undesirable in these circumstances. Characters have ties to series, and a more "vanilla" implementation wouldn't have such linking (I think).

Now that an image can belong to multiple series, it'll take a little more work to figure out how to deal with characters. I'd display all possible characters on the index page, but it's huge and ugly. I'll... work some interface thing out.

111 Name: Anonymous 2005-08-30 11:42 ID:Z2sjoNhI [Del]

ok, so SQLite decided to eat my ochiba database. Looks like I'll have to redo all the categorizing; before that, are there any other image repository scripts in the woodwork I could consider switching to? I'd really like one with the ability to search for multiple keywords, and all the 'PHP sucks' talk on 4-ch is getting to me.

http://danbooru.donmai.us/ would work beautifully, if only dreamhost offered PostgreSQL.

112 Name: Anonymous 2005-09-01 07:36 ID:foma1TVm [Del]

>>111
there are two tagging/categorization systems that can be found by visiting irc://irc.freenode.net/tma ... but i guess that was already mentioned here.

and no, i do not mean TMA itself, that one is still just working with auto-generated low-quality metadata. but the authors of other image-db things hang out there too ...

113 Name: Furi!EuK0M02kkg 2005-09-02 19:44 ID:Heaven [Del]

How badly do you need Postgres? Depending on functionality, danbooru might be able to be converted to MySQL. Meidokon could use MySQL with some work, I know that much. You could switch to another host that supports Postgres, but it'd probably be more expensive. We like Postgres just because it's more robust and "proper" about what it does.

114 Name: albert 2005-09-03 07:19 ID:dgdnyBeg [Del]

Hello, I am a member of Team Danbooru.

If I recall, there are only two places where danbooru relies on pgsql features: the rules on posts_tags (to automatically increment tags.post_count) and the pgplsql stored procedures to perform related post count. The former can easily be moved into the app and the latter you don't really need, you can just rely on tags.post_count instead.

There are plans to offer a SQLite version in the future, but right now we're busy whoring Postgres for all she's worth.

115 Name: Furi!EuK0M02kkg 2005-09-05 02:28 ID:Heaven [Del]

but right now we're busy whoring Postgres for all she's worth.

Amen, brother.

116 Post deleted by moderator.

117 Post deleted by moderator.

118 Name: Celine Purse : 2012-08-03 11:17 ID:iv6+lTbs [Del]

Name: Link:
Leave these fields empty (spam trap):
More options...
Verification: