Doranwen

In case anyone's curious, all my other projects, except for a few curriculum-related ones, have been put on hold until I get the metadata sorting done and all tagging tabs prepped. I've not written any fic since last year, and probably won't for a long while. Even my fic db is on pause - I've got WIPs to update wordcount for, but YG has pretty much taken over my life, lol.

However, I have some neat stats to report. As of this evening, it's up to 12.39% sorted and 0.99% tagged.

Available tabs:

English: 386
Spanish: 8
Portuguese: 4
Italian: 83
German: 1
French: 1
Chinese: 1
Indonesian: 24
Arabic: 11
Persian: 4
Turkish: 11
Romanian: 2
Unknown: 44
Spam: 3

You'll notice some big languages (German, French, Chinese) have virtually no tabs so far, and Spanish and Portuguese still have very few, whereas Italian's got dozens and dozens. That's because I'm going in order of the cat_ids, and all the Italian categories were stacked at the beginning, followed by tons of English. I haven't yet gotten to the categories for any of the other languages. Indonesian, Arabic, and Turkish didn't have separate category structures for them, so they've been sprinkled throughout other categories, mainly English ones.

There are a lot more languages I've got heaps for (over 300 Dutch groups, for instance), but they're so few that I don't know how many tabs I'd get out of them by the end, so they are all shoved onto a single tab for each language for now, which won't be available until I'm done sorting.

The Unknown is, in case anyone's curious who doesn't know, a mix of groups which are pretending not to be spam (but which have obvious patterns in their descriptions), groups which have so little or confusing info that someone will have to actually look inside their messages to get an idea of what they were, and groups where the info was unclear as to what the primary language or languages for the group was/were, so they also need looking inside.


Languages

As far as total languages seen, besides English and the languages listed above as having tabs ready to tag, I've seen just around 85 so far (listed in the order I found the first group in that language, by broad region):

European - Esperanto, Albanian, Bosnian, Hungarian, Polish, Swedish, Greek, Danish, Croatian, Dutch, Estonian, Lithuanian, Norwegian, Finnish, Slovak, Czech, Slovenian, Latvian, Catalan, Icelandic, Welsh, Basque

Cyrillic (separated out because of the alphabet): Russian, Ukrainian, Bulgarian

Oceania - Māori

North American - Haitian Creole

African: Swahili, Afrikaans, Somali

Asian - Hindi/Urdu, Vietnamese, Filipino, Hebrew, Azerbaijani, Javanese, Tamil, Mongolian, Thai, Korean, Marathi, Japanese, Sundanese, Kurdish

Not counted in the above list were the nine different languages from the Kuki-Chin language family - a branch of the Tibeto-Burman one. (Telling *those* apart - given most aren't in Google Translate - has been a real exercise in pattern recognition and detective work.)


Then there were the languages I have only one or two groups max for (when it hits three, then I give them their own tab, just in case):

European: Breton, Ido, Interlingua Romanica, Luxembourgish, Aromanian, Belarusian, Piedmontese, Middelsprake

Cyrillic: Macedonian, Tatar

Asian: Tausug, Tetun Dili, Malayalam, Kyrgyz, Armenian, Batak Toba, Meitei

African: Ukwuani, Kinyarwanda


Names for languages have generally been taken from Google Translate, or Wikipedia when that fails. If it turns out one of the language names is incorrect, it can be easily fixed once everything's into the database.


Volunteers

A big call for volunteers won't go out until I have all tabs prepped and ready to go, but if you or someone you know is detail-oriented and likes the idea of helping out, limited numbers of volunteers can be used right now, as indeed a couple of them have been hard at work for months tagging tabs and fleshing out any bugs in the tagging process.

In addition to business, computers, and mixed tabs of a whole jumble of things (some of which include fandom groups interspersed throughout), there are already fandom-only tabs available in the following areas:

- a few specific pop groups (BSB, Britney Spears, and NSYNC)
- celebrities (mainly actors)
- anime & manga
- comics
- cartoons
- Disney

Currently I'm working on fashion models, and when finished with those, will move on to the humanities categories - which almost immediately start into books and authors. :D

If you're interested and haven't already joined, you can find the Discord server here: https://discord.gg/UyJdffhw2b

At last, the database of Yahoo Groups metadata has reached the point of readiness so that tagging can begin in earnest - and done properly this time, unlike the attempt last summer (which I have kept the info for, so the efforts were not in vain).

Big pluses this time:

- Tabs are organized by one or more cat_id numbers, each of which is unique to the combination of category path, category id, and category name. This means that, on the whole, the groups on a tab should be roughly similar in theme/content, and in some cases, tabs can be limited to a specific fandom (generally super popular fandoms of the early 2000s such as LOTR, HP, Buffy, Sailor Moon, or Backstreet Boys).

- Tabs are separated by language, so the average volunteer doesn't have to deal with identifying languages at all, nor do they have to tag groups in a language other than English if they don't want to. (Of course, that means we are definitely eager for volunteers who ARE able and willing to tag groups in other languages!)

- Nonfandom tagging is done alongside the fandom tagging by the same volunteer, so there's no second pass that has to be done later. (There's a preset list of nonfandom categories to copy/paste from, to make it simple for taggers.)

- Each tab only needs to be looked at twice (once for initial tagging, once for checking), but checkers have a slightly different set of tasks, and should be able to breeze through most tabs more quickly than the initial tagging.


If you're interested in helping, here are the relevant links:

Tagging guidelines: https://docs.google.com/document/d/1AWFSmXLH-KsVU7N1EGkmbrLyv1N_fYoRlWLCEWLxtX4/edit?usp=sharing
Category list with cat_id and groups count: https://mega.nz/file/EZ9xCY4b#N8l9_LTJ-mV4KsMRO0DT4J--ws1fackYRcLZHPJ370o
Discord server (Save Yahoo Groups): https://discord.gg/fqsNqdpF7r

You do NOT have to be on the server, if you don't want to do Discord. It is helpful, as I update the last processed cat_id so you know which ones are already on tabs and can be requested, but it's not necessary.


Especially needed:

- anyone who can read a language other than English (the first 1000+ cat_ids are all Italian category paths, for instance, and there are tons of groups later on in Spanish, French, German, Portuguese, Chinese, Turkish, Indonesian, Arabic, some Hebrew - and we will particularly need to find someone who can read Persian written with the Latin alphabet)

- anyone with specialist knowledge in particular areas, whether it's a specific fandom or general area of fandom you know well, or a nonfandom area like computers, biology, or various cultures

- anyone willing to download and import mbox files in order to identify language and/or fandom/category for groups on the "unknown" tabs (groups where the metadata is not sufficient for tagging); we've got a visual tutorial for a lightweight free software program, so it's not hard!

This last week or so has seen a major increase in my geek skills.

A week ago yesterday, I wrote my first Python script (to create a macro using the library pyAutoGui). Then I wrote several shell scripts to automate some processing of things for that project.

And then it worked out that arcadianmaggie did not have the time to set up the next stage in dealing with Yahoo Groups data (identifying fandoms and setting up a database to put the info into it)… so I learned how to use postgreSQL and pgAdmin4 and quite a bit more SQL, and I now have all of the spreadsheets I created (the Yahoo Groups Metadata that was uploaded to the IA) into a database I can work with. (I also found a few errors I need to fix in that upload at some point - they're fixed in the db.)

So now I'm currently prepping a spreadsheet of tasks that volunteers can do, using a SQL query to pull the necessary info on all the groups from the database. I have a few tabs to get ready before I can link it publicly…

Between the ArchiveTeam volunteers and the Yahoo Groups Fandom Rescue Project, we saved 960,233 unique groups through Yahoo's GetMyData method (not counting what was saved through the Python script and PGOffline). Wow! And hopefully some more is yet to come as there are a few people who have promised to send us their GMDs but haven't yet. We will always accept it, even if it's just as a "please don't share this but hold onto it for safekeeping" measure, so if you know anyone who never got around to sending data to us (no matter what method they saved with), we'll still be happy to get a copy!

How much of that is fandom? At this point, all I can estimate is "somewhere around 300,000?" It is really impossible to know yet. I finished working on the metadata and will be uploading it soon, and sometime next spring (I hope!) we might begin on the more finely-detailed analysis that would let us identify individual fandoms and all of that.

Also, there's a really awesome script that will convert pg4 files (which can be exported from PGOffline) to mbox files like all of the GMD ones, and those can be imported into any email client to read, including Sylpheed as shown in the visual tutorial I put together. So if anyone can't run a Python script but would like that, I will be happy to do it; I already did for any group I saved with PGOffline which had attachments, as Yahoo often did not save or include those with the GMDs. PG4 is definitely the best export method for PGOffline as it lets one convert to multiple other formats. HTML is better than nothing, but it's awfully limited in what you can do with it easily. And when I have the chance, I'm going to experiment with Hypermail and see if I can't figure out how to convert any mbox to html files easily enough that way, as I know some owners of (mostly non-fandom) groups are interested in uploading the HTML files to a website for viewing.

I've been working on processing Yahoo Groups metadata for ages (something like seven months now), and probably few of you actually care what all that has involved, but I decided to take a break from it this evening to type up the process for any data processing geeks out there.


( Cut so you don't have to read this if you don't want to )

Before I start, I'm giving full permission to anyone to link to this post without asking (and quote, if they like, as long as they link to the original post so my words can be seen in context).

One of the other members of our small Yahoo Groups Fandom Rescue Project was talking to me about how past archival teams didn't get enough non-Western media and a lot of people were very upset by this. They were concerned our project was going to have the same problem.

So let me explain where things stand.

We have literally thousands of groups we're trying to save. There are about half a dozen of us regulars, if that. More volunteers come, join one or two tabs of 250 groups apiece, and then are done and leave. Some don't even get to half a tab and say they can't finish. (For perspective, I can join a whole tab in several focused hours.)

At this point, I've been told not to bother running more searches because we can't save all that we've already discovered. So which fandoms have we prioritized? A combination of the very biggest ones that we're aware of, and the ones we love the most. (If we're going to spend all this time saving, you bet we're going to make sure our beloved fandoms are archived.)

We would love for people to show up and go "hey, can I get a list of groups for these fandoms?" so they can save them. But the people aren't coming. No one wants to spend the time saving. (I understand that not everyone can, but if the fandoms are that big, one would think someone would be working on them!)

I can hope that there's parallel projects to ours in other countries, saving the fandom groups from their media, but I haven't heard of any. So Eastern media is likely to be underrepresented. It's not out of unwillingness to save, but we can't save what we don't know of, and we can't save what we don't have the time for.

So here's the final call: If you want more groups saved, come help. If you want a list of groups for YOU to save, join our Discord and ask. (Link found here.) I can run a search of a partial (it's impossible to get a complete list out there - we're having to do all manner of processing data to get what we've got) list of groups for any keyword or keyword combination, or search specific categories for more groups than can be found in the directory itself, and I'm happy to do so.

But please don't show up later and wonder why we didn't save your favorite fandom, Western or non-Western.

As I write this, I've backed up 245* groups (with another currently downloading) and have a queue of 91 more (not counting the tabs of potential groups to join and backup). I started with fandoms I'm into, moved to fandoms where I've seen part of the show, and added to that ones where I know my mom has seen it, even, lol. (If I have time) I'll be sending out a message to all the groups I've backed up (unless they're already discussing it or I don't have access to post).

* See edit note at end.

All content backed up is going to a Mega folder where it can be held securely in trust for any owners/moderators who want it, and backed up by Open Doors if groups are entirely inactive (or unless requested not to - note that they will not publicly share any data for a private group for any reason). If you're a group owner or moderator of one of these below, once you've approved me, please post to your group requesting the links of the data sent to you, and I'll be glad to email them to you.

What concerns me most are the many groups that I haven't been approved for, and cannot backup (many of which have thousands of messages and fics). If none of their members back them up, all messages and fic will vanish when Yahoo deletes the content. (You can see here which ones have been backed up and which are still waiting for someone to back them up. Note that a team is working on backing up a large number of groups which don't require approval to join. It's the ones that require approval that won't be in that number.)

If you are in any fan spaces for these groups, or can post it somewhere else - a DW comm, a Tumblr, a Facebook post, wherever that fans would see it - please do so!

I'm looking for a member of each of these groups to do one of the following:

• ask the moderator/owner to approve my membership so I can backup the group.

or

• should the moderator/owner be MIA, backup the messages, files, photos, and links yourself.

or

• communicate with me privately to give me two cookies from your own login and I can run the backup for you without logging in as you. (You're welcome to ask anyone that knows me for a reference; I will absolutely not share your data with anyone and will not compromise the security of your account in any way.)

In nearly all cases I'm interested in the content for its own sake, and would love if the backup files could be shared with me so I can read, but even if you're just backing it up privately and making sure someone else has a copy (because one place saved is precariously vulnerable), that's better than leaving it to be deleted. (In one or two cases, the groups have been backed up by someone else, but I very much want to read them myself.)

Below are the groups I'm waiting approval for, by fandom.


( Multifandom )


( JAG )


( Law & Order & offshoots )


( Lois & Clark: TNAOS )


( LOTR )


( NCIS & offshoots )


( Remington Steele )


( Smallville )


( The Pretender )


( X-Men )


( Miscellaneous other fandoms )

EDIT: Updated numbers are 524 groups backed up (or at least looked at - about four of those were entirely spam and not archived), if you're curious. At this point I've probably found all the groups for the fandoms I'm interested in, and joined or applied to them. Only four—not on this list anymore—have outright denied my application, which is sort of sad, because I want to see them for my own self—why deny me? I'm not a spammer or bot, and I definitely wouldn't share the files if they wanted them kept private. But oh well, they didn't bother to tell me why.

For anyone who hasn't heard, Yahoo Groups is closing; in less than two weeks all ability to add/post new content will be locked and in December all data will be deleted. (Groups will be converted to mailing lists only, ugh.) If you were a member of any Yahoo Groups and want to back up your groups' messages, files, or other content, take a look at some of the options and discussion mentioned here.

I'm using this Windows program (on a computer I rarely use, but I didn't have the time or want to deal with trying to create a VM that would have the requisite NET framework, so working on native Windows was the best option here) to back up all of the groups I'm a part of. After that I plan on joining as many fandom-related groups (particularly fanfic ones) as possible before the deadline and backing them up.

Unfortunately, I don't currently have the $$ to pay to register the program in order to be able to export the backups into human-readable HTML, but it looks like SQLiteStudio should do the job - and for free! (Plus, it's got a Linux version, so I can move the database to my main computer and export there if I want.)

I hate it when sites close like this! I wasn't in the position to do anything about archival when Geocities went poof, but at least this time I can do something.

LINKS TO MORE INFO:
https://archiveofourown.org/admin_posts/14117
https://morgandawn.dreamwidth.org/1678027.html

The step-by-step guide under the second link contains a link to a spreadsheet and Google Form coordinating group archival for fandom-related groups.