29 January 2020, 03:57:27 *

Login with username, password and session length
Welcome to War and Tactics!    War and Tactics Forum is currently undergoing some modifications that might disable features you are used to. This is unabvoidable as we have to update the forum engine to a new structure that is incompatible with many of the features we had used so far. The good news: WaT will be more secure and stable, and most of the features we uninstalled will be a natural part of the new structure anyway. For the rest we will be looking for solutions. (APR 23, 2018)
   
  Home   Forum   Help ! Forum Rules ! Search Calendar Donations Login Register Chat  
Pages: [1]   Go Down
  Print  
Share this topic on Del.icio.usShare this topic on DiggShare this topic on FacebookShare this topic on GoogleShare this topic on MySpaceShare this topic on RedditShare this topic on StumbleUponShare this topic on TechnoratiShare this topic on TwitterShare this topic on Yahoo
Author Topic: JUL 19 - Forum Down TA AAR (Login Denied To All) and Lessons Learned  (Read 3424 times)
TechAdmin
Administrator

*

Offline Offline

Germany

Location: Planet Earth - sometimes...
Posts: 1002




View Profile WWW
« on: 20 July 2009, 15:49:44 »
ReplyReply

(copied this from the tech room to make accesible for external viewing when I call in help)

This is the TA AAR on the events that disrupted the forum today and that up to now do not yet have an explanation or an established cause, nor are they fully understood.

Part 1: Chain of Events

On JUL 19, 2009 we for the first time had serious problems that were out of my tech capabilities:

As today and upcoming week I won´t have time I had worked rather frantically the last days to get us to the desired (functional forum endstate and reached that goal on Friday night GMT, wherupon I proceed to publish us to Google on Saturday, JUL 18.

When I got up sunday morning at 0030 GMT (0230 Local) due to age related bed flight syndrom I

- logged on as TechAdmin
- created an XML sitemap using the sitemap link at the bottom of the pages
- submitted it too Google and checked Googles sites entries about us: I found out that the pages they displayed had the right titles, but spam gibberish in their content summary (we still had a lot of infected files from the KB virus attack which I suspected to be the source).

Next step, as it was Sunday and I had time on my hand:

- I started (in cPanel) manually cleaning files of the first line malicious code, managed til 0430 GMT to have all 600+ files we functionally need (except chat, I leave those 300 to Koen Smiley ) cleaned manually.

Job done, wanted to log out. Did not work: Came up with a blank page and a seesionID URL (like http://www.warandtactics.com/smf/index.php?action=logout;sesc=7cd3f45956a7a080609f5a6247406a70), reloading the page showed me still logged in (fully functional).

Cleared cache, same story.

Tried other forum features, all worked smoothely (but could not get back to admin section to write a notification because of the same problem: After one hour you need to specifically log in to Admin section again and my password entry was ignored), it was just that I could not log out.

Tried to log in as Rattler from other browser, same problem, no login (but, of cause, other URL: http://www.warandtactics.com/smf/index.php?PHPSESSID=3b4829553bc1ed249ab94fae1084a7a9&action=login2).

Next decided to force my logout as admin in two steps:

- First killed the session cookie (sessionID), the resulting logout try gave me an error message: "Session ID could not be verified" (this means that no new cookie was transmitted to the browser, interesting find and physically the primary source of the prob)

- 2nd killed the normal SMF80 cookie, logout worked.


Part 2: Measures taken

- went to sleep (important!) as I anticipated that I would need a clear head to try and remember what *exactly* I had done in those 4 hours, and as I was exhausted and close to panic

- When up, sat down and wrote a memory protocol of what steps I had taken in the 4 hours to extablish whether I had touched any files for other reasons than virus cleaning. Made a list of file groups that I remembered Ihad cleaned. I could definitely establish that I had *not* touched ./Sources/login php (which is the routine that handles the login together with login-template and the theme template), as I had previously cleaned all Source files up to  the lettter "M".

- ran a quick scan on (the few: 20) files of this list that could have to do with the problem, all looked ok at first glance

- Informed Forum Owner and Co-Admin via email about the problem and explained that I was out of my depth and probably needed external help (forgot to ask him to send an email to all members explaining the situation). I layed out the worst case scenario which would have meant a clean re-install and about a month´s work to reach same functionality as now (but w/o any data lost we already have generated up to now)

- Koen confirmed the problem with IE on Vista and XP, he could not log on either (but we both could log on into the chatbox on TinyPortal).

- Established various possible scenarios and started researching in the free SMF support forae, found similar but not the same problem

Then my time ran out for today, on the list were in this order:

- file a support request explaining the problem in detail (as I am doing now)
- go over all 600 edited files to see whether I accidentally had deleted more than the virus code (one weeks work)


Part 3: Miraculous Solution of the Problem

Just before going to bed (and on the hunch that such things are possible) decided to look at 1200 GMT at the forum again, found I could log in and access Admin area without problem (I need to reload every page to actually see it, but the URLs are ok, so maybe it is just a display prob that I can solve).

I frankly have no clue why we are suddenly able to log on again (and error logging was not enabled during the events), but I guess - and it is a wild guess - (as I had not done anything yet to resolve the issue) that the problem is maybe on the host side (server) and not related to my software handling. Bad thing is that the host never informed about this (on my Spanish host I would have accessed hosts blog and found why my stuff didnt work if he was the cause).

Put up the notification, now writing the AAR.

On the to-do list:

- backup db again, also all files (now, during session)

- run maintenance cycle

- try to log out, and go to sleep. More tomorrow.


Part 4: Lessons Learned

- Admins, when anticipating that they work longer than one hour, disable Admin Security temporarily til end of job, in this case it would have allowed me at least to put up a notification to guests and members of the problem

- When doing *any* admin work, enable error logging (was disabled, so now that we can log on again I have no error log entries to see what was happening)

- If we ever go and make a serious job out of this (or even money) we need to subscribe (50$ the year) to the SMF express support where the guys actually handle your files if needed: "Charter Membership". THe forae are great (and great and helpful people), but in the paid version they actually do installs, talk to your host, etc. when you as TA are out of your depth.

- A host that does not inform about problems on his servers or that has no support via ticket is not suitable for us. If I had had the possiblity to communicate via email with the host we couldhave cleared much earlier where the problem lay

- *Always* back up the database (incl table structure) when entering admin section, whatever other pressing stuff you might have to do: If everything had gone haywire permanently and a new clean install would have been necessary, we would not have lost a lot: The last back up of db was just 6 hours old, meaning 2 posts lost and all members and profiles as well as the forum setup would have been restorable.

TechAdmin
Logged



"Smile, tomorrow will be worse!"  Murphy
Koen
Poster

****

Offline Offline

Belgium

Location: Belgium
Posts: 4247




View Profile
« Reply #1 on: 21 July 2009, 10:45:14 »
ReplyReply

copy from my other post:

latest report from me but forgive me that maybe I'm not so easy to read as our TechAdmin...

*when you try to log in you'll be directed to a blank page with an URL like this one:
http://www.warandtactics.com/smf/index.php?PHPSESSID=aa424a97b48559a2daa39c9ac1f17c20&action=login2

delete the action=login2 at the end and hit the refresh button of your browser, this should log you in!

The same procedure works for Internet Explorer and Firefox !

Please report back
Logged
TechAdmin
Administrator

*

Offline Offline

Germany

Location: Planet Earth - sometimes...
Posts: 1002




View Profile WWW
« Reply #2 on: 22 July 2009, 01:31:47 »
ReplyReply

All FIXED (who wants to see what had happened: http://www.simplemachines.org/community/index.php?topic=325760.msg2170286#msg2170286), Forum back to normal...

TA
Logged



"Smile, tomorrow will be worse!"  Murphy
Tanker
Sergeant Major
**

Offline Offline

United States

Posts: 221


View Profile
« Reply #3 on: 22 July 2009, 03:59:21 »
ReplyReply

Time for some well earned sleep eh TA? iconclap
Logged
TechAdmin
Administrator

*

Offline Offline

Germany

Location: Planet Earth - sometimes...
Posts: 1002




View Profile WWW
« Reply #4 on: 22 July 2009, 11:22:47 »
ReplyReply

...indeed, and it was short (I start work at 0600J and went to bed at 0245J)...

TA
Logged



"Smile, tomorrow will be worse!"  Murphy
Tanker
Sergeant Major
**

Offline Offline

United States

Posts: 221


View Profile
« Reply #5 on: 23 July 2009, 05:15:12 »
ReplyReply

Ouch.
Logged
Pages: [1]   Go Up
  Print  
 
Jump to:  

Unique Hits: 27678718 | Sitemap
Powered by MySQL Powered by PHP Powered by SMF 1.1.16 | SMF © 2011, Simple Machines
TinyPortal v0.9.8 © Bloc
Valid XHTML 1.0! Valid CSS!


Google visited last this page 20 November 2019, 17:54:02