Monday, August 15, 2005

(Updated) Cleared, 8/15 Journals Problems

Hi folks...we should be better now. Most of the problems with truncated
or missing entries and sidebar information should be fixed.



Please note: Some of you who posted during the time when we first
started having problems (around 4AM ET) to when the fix went in (about
10AM ET) may still be missing those entries; those are being restored;
I will let you know when I get more info on that.



Please check your blog (refresh and clear your browser's cache if necessary) and see if you are still having problems.



I will provide more information in this entry when I get it.



Update: 8/15, 8:03PM: Okay folks, we had two separate (but
possibly related) problems today, one in the morning and one in the
afternoon. Not your typical Monday:



1. This morning, starting around 4AM, the tech folks were moving
the Journals database to some new servers (or hosts). This happens
periodically to replace or add new equipment, etc. It's typically done
in the early morning hours, when the fewest users are online to be
affected.



There was a problem with the database after the move; this caused a few
days of Journal entries to not show up, and for older entries to get
cut off at 101 characters.  It also caused sidebar links (like
Favorite Sites) to not show up.



The tech folks fixed things at around 10AM, though posts made in the
intervening 6-hour period needed to be restored. (I am not certain if
this has happened yet.)



2. Then, at around 4PM today, there was another problem with
some database keys. (I am not well-versed in database workings, so
please excuse the generalization.)  This started causing problems
for everybody, to the point where everyone was getting an error
message.  This was fixed about 30 minutes later.



The tech folks are monitoring the database and related systems; if you
see any flakiness, please leave a comment or send me an e-mail.



For the afternoon outage, I was listening in on the conference call,
where the different tech folks huddle up to work the problem. Most of
it was over my head, but it was pretty interesting to hear all the
different teams working together to diagnose and solve the problem.



Thanks, sorry for the inconvenience. -- Joe

9 comments:

Anonymous said...

The feeds (RSS and Atom) were affected too. Truncated entries in Bloglines for the last 10 or 11 updates with all the Journals I'm subscribed to. All the error truncated entries show as newly updated entries in Bloglines which for me means I've lost those updates in bloglines :-( It's how Bloglines handles updates to previously published feeds. [sigh]

Updated feeds since the issue was fixed appear normal in Bloglines.

Anonymous said...

Joe thanks for the update.  I had the upmost confidence we wouldn't have lost anything.  
Betty

Anonymous said...

This is minor and not affecting anything, but there seems to be a glitch in the widget on the Journals main page.  My journals show twice on there.  Other than that, since restoration after the outage yesterday, I haven't experienced any problems with the AOL Journals product.

Anonymous said...

h

Anonymous said...

h

Anonymous said...

Dear Joe,
Please check the last entry of Sunnyside 46 (Marti) (MidlifeMusings)becasue try as I did I still have not been able to leave acomment on her comment page

Anonymous said...

Looks like the hit counters on the journals aren't working, the images don't seem to be loading of the few I've checked.

Anonymous said...

The TEXT and links are, of course, what I was trying to say was missing.
-Paul

Anonymous said...

Hi Joe,
  I didn't notice anything wrong with my journal until late last night (around midnight) when I noticed there is missing data from my about me section. The picture is showing up, but the test and links are gone completely. Still missing, as of 3:30pm today.
-Paul
http://journals.aol.ca/plittle/AuroraWalkingVacation/