Evilrooster Crows: Abi Sutherland's Blog

Run down the Jolly Roger, Run up the Union Jack

Posted by Abi on Mon 5 May 2008 at 23:24 | Comments (4) |

Making Light is back up, substantially populated with the lost data. Our saintly datameisters are still filling in the cracks, but we have active threads again.

So thank you, everyone, for behaving so nicely here, but let's move the Making Light discussions back to their natural home. I'll sweep up and fold up the guest beds, and restore normal evilrooster-type behavior here over the next few days.

Rebuilding the threads

Posted by Patrick Nielsen Hayden on Sun 4 May 2008 at 21:50 | Comments (27) |

Here's what I've got for Making Light since March 1: original post date, abbreviated name of the post, and the number of comments I have actual copies of.

3/1		Who's surprised?	66
3/3		All come singing	69
3/3		Can you read this	53
3/4		Greyhawk	253
3/11		Collect underpants	265
3/13		Open thread 103	936
3/16		Just do it	38
3/16		Literary divination	106
3/18		Arthur C. Clarke	177
3/20		Bigger laser	174
3/28		Divided by errors	34
3/28		Open thread 104	931
3/30		London photograph	204
3/31		Deep value	434
4/1		Amsterdam	70
4/2		Pity the Times	167
4/4		Forty years gone	70
4/6		Heads they win	320
4/6		Employ the scythe	126
4/9		SFWA deadline	25
4/11		Future of publishing	32
4/12		Book by its cover	37
4/13		Bury my acorns	87
4/13		Goose-stepping (actually 469)	468
4/14		Open thread 105	906
4/16		Housekeeping	7
4/16		Newsweek	245
4/17		Little Brother	180
4/22		Penn for Hillary	124
4/23		Font game	125
4/23		TNH in San Francisco	18
4/25		Indistinguishable from parody	186
4/26		Clapton	107
4/26		Feeling the heat	31
4/26		SFWA election	45
4/26		TNH in the Observer	105
4/27		Open thread 106	288

In addition, I have a 131-comment version of the Clay Shirky post, but in fact I know there were at least 254 comments; if anyone has a 254-comment version, please do send it on. (Apologies if you did already. I may have lost it. Processing all this stuff on the fly has been a challenge.)

I also have the recently-posted comments to old threads "Worldcongoing," "New Magics," and "Abi on catz." I do not to have the recently-posted comments to "Darwin fish found"; the same apologies apply as in the previous paragraph.

Who's been saved

Posted by Teresa Nielsen Hayden on Sun 4 May 2008 at 15:30 | Comments (36) |

Teresa here. I've figured out how to make a Google doc universally readable. Please understand that vast amounts of what we've been sent is still being logged. This just means you get to see it happening.

There are now two spreadsheets. One is the general spreadsheet Abi built yesterday. I can't make it visible because Abi owns it, and I didn't ask her about it before she left for the day. The other is a new one I just put together. It lists individual users known to have posted on or later than 01 March 2008, and notes whether their comments have been saved, and by whom. (If you're working on that project and want to be able to enter data, write and ask. Our addresses are in their usual spot.)

Here's the link:

Individual commenters: saved or unsaved.

It's too bad Mike Ford isn't here to write "Sinners in the Hands of an Angry Google."

Needed: Movable Type mavens

Posted by Patrick Nielsen Hayden on Sun 4 May 2008 at 13:57 | Comments (17) |

Patrick here, again. It's Sunday morning in Brooklyn.

Last night, Hosting Matters set us up with a new server (larger and more powerful, and all to ourselves, I gather), and we uploaded the complete March 1 backup of our home directory. This is why there's been a March 1 version of Making Light's home page visible for the last twelve hours or so at the usual location. Up until a few minutes ago, this was a version of the front page that looked all right, except that links to individual-post pages-with-comments didn't work, and the Sidelights and Particle sections were empty, along with every other sidebar section that works by pulling data in from a file external to the main index page.

We also uploaded the March 1 export from the site's MySQL database, and while we slept, Hosting Matters verified that the database export seemed to be okay, and re-created the database and the user. They suggested that we do a full "rebuild" from inside Movable Type, which should recreate all the internal links.

I've just now been trying to do that. The first thing I tried was a full site rebuild, but it crapped out and reported "internal server error" somewhere in the midst of rebuilding the second hundred individual-post pages. (A full site rebuild does individual-post pages first, then monthly archive pages, then index pages last.) The next thing I tried was an "index pages only" rebuild, which yielded the version of the front page that's now visible, with the entire center column empty. I then tried an "individual archive pages only" rebuild, which once again failed in the middle of the second hundred pages. Finally, I made another attempt at a full site rebuild, and it failed ("Internal Server Error - The server encountered an internal error or misconfiguration and was unable to complete your request") just after the 300-pages-rebuilt mark.

Rebuilds that fail to complete have been a constant bane of our Movable Type installation, but in the past we've dealt with them by simply trying again another day. Does anyone have a better idea that will get the site (at least, the March 1 instantiation of the site) up and functional now?

(Noted and of interest: You can get at the individual archive pages that are linked from the "recent comments" sidebar; you can even load the "last 4000 comments" page and get at lots and lots of individual archive pages from that. But as far as I know, until I get a full rebuild to happen, the middle column of the front page is going to stay empty.)

What you need to do *Right Now*; also, an interim status report

Posted by Teresa Nielsen Hayden on Sun 4 May 2008 at 04:20 | Comments (74) |

Hi, everybody. Teresa here. You may have already read earlier, incomplete versions of this post, which I published prematurely in order to get the THINGS TO DO RIGHT NOW part out fast. I said in them that you'd know you'd seen the complete version when the last word in it was dinosaur. That word is there now.

Onward.

First: apologies if you've written to us and haven't heard back. We've been wrestling with the task of getting the pre-March 2008 part of Making Light back online. For a bit there, it looked like we might have lost the whole thing, not just March and April. That was exciting. Fortunately, Patrick found his alternate backup of the MySQL database that's the difference between losing two months and losing seven years. As Erik Olson said at that point,

Good. We're now at "this really sucks."
Not "well, fuck, Maude, better sell the cattle."

Thanks to many people who shall receive fulsome thanks once this settles down, we've reconstituted all the front-page entries, plus Sidelights and Particles. What we don't have all of are the comments.

THIS IS IMPORTANT: collect your own comments NOW from Google or other search engines, then help collect others. Do it as soon as you can, because Google has already overwritten some caches with versions that end at 01 March. If you find this has happened to the Google cache of your own comments--for instance, Xopher, Niall McAuley, CHip, and NC Hanger have already gotten nailed--try the caches at Yahoo, MSN Live Search, or other search engines.

If you've been reading the comments at Making Light via RSS feed, please check and see what you have cached. It's possible you have the only surviving version of some of our missing comments. We're particularly on the prowl for new comments that got posted to old threads. We know to look for complete runs of comments from (f.i.) "All come singing" or "The rather difficult font game"; what's far less obvious is the need to collect recent runs of comments in old threads like "Introduction to New Magics".

Abi's chart of what we do and don't have isn't up to date. That's because Abi went to sleep (she's six time zones east of Plymouth Rock), and Patrick and I are still figuring out exactly what we have on hand. She'll be gone all day tomorrow, but we figure we'll have the chart updated by tomorrow morning EST. If you're doing intensive work salvaging comments and need more up-to-date information, send us your email address and we'll try to reset the permissions so you can see our working spreadsheet in Google docs.

I've finally gotten in touch with Jim Macdonald. He'd been away most of today, and had no idea what was going on. The bits of Making Light's database we have least hope of recovering are unfinished articles the editors had saved as draft posts in Movable Type. We've all lost some, but Jim had been working on the big final post in his Trauma and You series. He took the news very calmly, though it's possible he was simply too tired to get upset. He says he'll just have to rewrite the article from memory:

I'd gotten through traumatic amputations, degloving, and avulsions, and was about to start on incisions and lacerations.

When last seen, Jim was running searches on other people's names. His suggested search string is "Comments posted to Making Light by [name of poster]". If that doesn't work, try something else. If that works, come back and tell us what it was.

(UPDATE: Jim has done as much comment-salvaging as he can do tonight. In the comment thread for this entry you'll find his list of people he knows to have posted comments during the lost months, but whose body of comments he hasn't salvaged. If you have time, please consider running some of those searches. If you know of other names, post a list of your own--Jim's list is by no means complete. While you may wind up duplicating someone else's efforts, you may also save the comments of people who haven't gotten the word in time to do it themselves. Also, read the whole thread. There's useful information in it. End of update: tnh, 0300 EDT.)

Salvaging the data is only the first part of the project. Once we've collected it, it'll have to be reprocessed into proper MySQL format and grafted back onto the main database. We've had several offers of help, but if you want to add to them, please feel free.

...

In the midst of all this effort to salvage the missing months, we're feeling awed and humble about the amount of help we're getting. We've said all along that its readers are the best thing about Making Light, but that's never seemed more true than it does right now.

("Hey, look! It's a dinosaur!")

Hello, I'm back

Posted by Patrick Nielsen Hayden on Sat 3 May 2008 at 23:07 | Comments (12) |

[Patrick here, not Abi.]

As luck would have it, I had to run off to band practice very shortly after the crisis began. I'm back. And I've had a life-giving, brain-restoring sandwich. As soon as I post this I'm going to start reading the email, and comments below, from our incredibly generous and helpful readers.

As mentioned previously, we do have a complete backup, downloaded with wget, of the site as of March 1, 2008. Hosting Matters has now set up new server space for us, and presently I'll be trying to upload the backup; if everything works, Making-Light-as-of-March-1 will appear at the usual URL. Much more likely, however, is that I'll have a zillion finicky permission problems that I'll need someone's help with before we get to that point.

At this point we have HTML of all the missing posts, and quite a lot of the missing comments, but many more comments remain to be excavated. Please, if you have been using an RSS reader to follow ML comments--generally or in specific threads--consider that you may actually have copies of comments from the last two months in your reader's cache folders. Please take a look.

More to come. My actual suspicion is that once we get the March 1 version of ML restored, the best way to re-attach the missing material is going to be by editing and manipulating the MySQL database in the guts of the site. I know very little about this, and the small amount that's been explained to me by helpful readers is pretty much worn away by time, but it seems to me that in principle it should be possible to automate, or at least semi-automate, the processes of:

(1) stripping away extraneous cruft from various saves of ML content in HTML and XML form, and--

(2) --wedging it all into the appropriate fields and tables in the MySQL database, so that--

(3) --all of ML reappears, right up to the present, along with all the--

[...]books we bought in college and sold for half-price unread
And sacks and sacks of earring backs lost under someone's bed
And baseball cards and army men and model planes galore
And every tiny plastic high-heel Barbie ever wore

(Thank you, Austin Lounge Lizards. Music here, if you're willing to sign up for a free trial of Rhapsody.)

Of course, knowing that something is theoretically possible doesn't mean it's practical, but I bring it up for discussion nonetheless.

Making Light Entries: The Master List

Posted by Abi on Sat 3 May 2008 at 20:19 | Comments (21) |

Below is an initial list of postings that have been active recently, to the best of my memory. I have noted caches that I have URLs for, kindly provided by people in these threads. I am digging through my own caches. If you have better caches of data, please send them to me (abi at the domain we're on), Patrick, and Teresa (their initials at panix.com). We will update this as we go.

(Why, yes, there is a spreadsheet!)

Making Light

Post No	Date	Title	Highest Known No	Latest Complete Comment	Savior
		April Archives (all April posts)			Doctor Science
010186	Apr-27	Where do people find the time?	251	#251 ::: Serge ::: (view all by) ::: May 02, 03:54 PM:	Nick Fagerlund
010184	Apr-27	Open thread 106	11		Abi has the email notifications
010178	Apr-26	Eric Clapton, White Power enthusiast	73	#10 ::: will shetterly ::: (view all by) ::: April 26, 11:18 PM:	Doctor Science
010177	Apr-26	Teresa in the Observer	54	#13 ::: Angelle ::: (view all by) ::: April 26, 10:53 PM:	Doctor Science
010176	Apr-26	Feeling the Heat	26
010175	Apr-26	SFWA election results	45	#45 ::: Dave Bell ::: (view all by) ::: April 30, 02:51 AM:	Doctor Science
010174	Apr-25	Indistinguishable from parody	186	#186 ::: Clifton Royston ::: (view all by) ::: May 02, 06:52 PM:	Doctor Science
010173	Apr-24	The Rather Difficult Font Game	123	#123 ::: David Goldfarb ::: (view all by) ::: April 29, 06:46 AM:	Doctor Science
010168	Apr-23	Live in San Francisco, it's TNH!	18	#18 ::: pat greene ::: (view all by) ::: April 24, 06:54 PM:	Doctor Science
010167	Apr-22	NBC News calls Penn for Hillary	124	#124 ::: Matthew Austern ::: (view all by) ::: April 28, 11:44 PM:	Doctor Science
010157	Apr-17	Little Brother	180	#180 ::: Bruce Cohen (SpeakerToManagers) ::: (view all by) ::: April 30, 06:13 AM:	Doctor Science
010154	Apr-16	Newsweek invents an alarming trend	245	#235 ::: Ginger ::: (view all by) ::: April 29, 05:02 PM:	Doctor Science
010151	Apr-16	Housekeeping	7	#7 ::: paul ::: (view all by) ::: April 17, 03:23 PM:	Doctor Science
010146	Apr-14	Open thread 105	906	#215 ::: Bruce Cohen (SpeakerToManagers) ::: (view all by) ::: April 18, 01:48 AM:	Doctor Science
010143	Apr-13	Could lead to goose-stepping	469	#150 ::: Lee ::: (view all by) ::: April 15, 02:18 AM:	Doctor Science
010142	Apr-13	Bury my acorns at Wounded Knee	87	#87 ::: Stefan Jones ::: (view all by) ::: April 21, 2008, 12:24 AM:	Abi
010134	Apr-12	A book by its cover	37	#37 ::: Claire ::: (view all by) ::: April 19, 2008, 11:34 PM:	Abi
010133	Apr-11	Future of Publishing, Part 5,271,009	32	#32 ::: Dave Bell ::: (view all by) ::: April 17, 2008, 09:17 AM:	Abi
010129	Apr-09	Don't Miss the Deadline	25	#25 ::: Greg London ::: (view all by) ::: April 11, 2008, 11:13 PM:	Abi
010125	Apr-06	Heads they win; tails we lose	320	175	Carol Witt
010123	Apr-06	Some must employ the scythe	126	#126 ::: abi ::: (view all by) ::: April 15, 2008, 03:53 PM:	Abi
010115	Apr-04	Pity the Times	167	#167 ::: Terry Karney ::: (view all by) ::: April 16, 2008, 12:22 PM:	Abi
010113	Apr-04	Forty years gone	70	#70 ::: rea ::: (view all by) ::: April 08, 2008, 07:20 PM:	Abi
010108	Apr-01	Amsterdam	70	#70 ::: Bill Higgins finds more spam ::: (view all by) ::: April 25, 2008, 08:48 AM:	Abi
		March Archives (all March posts)			Doctor Science
010104	Mar-31	Deep Value	434	165	Carol Witt
010103	Mar-30	The photograph that terrorized London	204	203	Carol Witt
010097	Mar-28	Open thread 104	931	239	Carol Witt
010101	Mar-28	Divided by common errors	34	#34 ::: Edward Oleander ::: (view all by) ::: April 05, 2008, 01:39 AM:	Abi
010083	Mar-20	Going to need a bigger laser	174	#174 ::: Paul A. ::: (view all by) ::: March 31, 2008, 08:08 PM:	Abi
010077	Mar-18	Arthur C. Clarke, 1917-2008	177	#177 ::: Dave Bell ::: (view all by) ::: April 06, 2008, 08:14 PM:	Abi
010069	Mar-16	Just do it	38	#38 ::: Fragano Ledgister ::: (view all by) ::: April 10, 2008, 06:36 AM:	Abi
010066	Mar-16	Literary Divination, A Parlour Game	106	#106 ::: heresiarch sees, um, spam? ::: (view all by) ::: March 26, 2008, 03:25 AM:	abi
010046	Mar-13	Open thread 103	936	222	Carol Witt
010044	Mar-11	Phase one: collect underpants	265	#265 ::: Laurie ::: (view all by) ::: April 21, 2008, 08:18 AM:	Abi
010028	Mar-04	Greyhawk's flags at half-staff	253	200	Carol Witt
010025	Mar-03	Can you read this?	53	#53 ::: Robert N Stephenson ::: (view all by) ::: March 07, 2008, 08:21 PM:	Abi
010012	Mar-03	All come singing	69	#69 ::: The Constructivist ::: (view all by) ::: March 21, 2008, 04:37 PM:	Abi
007399		Darwin fish found	386	204
005451		Worldcongoing	326	183
009050		Abi Sutherland, on Catz	574	293

Particles

Teresa has been looking at this and will report

Sidelights

Vide supra

Nielsenhayden.com

Placeholder for anything from this page, in particular the new narcolepsy postings

Techie Thread: Making Light is Down

Posted by Abi on Sat 3 May 2008 at 17:44 | Comments (76) |

See Patrick's LiveJournal post for details, but the short version is that we've lost everything from March 1.

This thread is for co-ordinating solutions to the problem. What we need:

1. Any caches of Making Light since March 1. If you have tabs open with ML, please save it, and send to Patrick and Teresa (their initials @panix.com), please.
2. Any other saved versions: please post what you have here
3. Offers of assistance in processing these saves
4. Wisdom in the ways of LJ feeds: apparently there is a makinglight one, which has data back to April 13th
5. Movable Type and blogging gods and gurus: suggestions, ideas
6. Time machines you can loan us
7. Suppliers for magical pixie dust