Development Madness during Upgrade

[#1]

Hello World,

please hear me out before demanding to drive a stake through my heart.

Because I know very well that latest during step "Preprocessing" of the upgrade process (EHP5ERP6 -> EHP7ERP6) *no* development should take place, except absolute emergencies.

Problem was, when we designed the upgrade strategy for our systems 4-6 weeks ago, I "forgot" to narrow down the definition of "emergency" to our developers to the bare essentials - believing the term "emergency" was self-explanatory 😞

Now they come to me almost twice a day with another "let's push this one one in quickly, too ..."

By now the system is done with Preprocessing and we await the maintenance window for step "Execution" - which unfortunately is still two days in the making.

And guess what...they'd like for me to import one more "emergency transport" into the system.

Its a 'Y' report, so the logic goes, "its in the customer name space, so what could possibly happen?"

My big worry is, that *if* something happens, the Production system will be down!

No SE38, no STMS, no SE80 and worst of all, no Prod system and the whole factory is waiting for this thing to come back online - and yes, ours is a 24x7 operation.

Has anyone got any experience with pushing in transports, within the customer namespace, so late in the upgrade process?

PS: I know this whole procedure is a "no, no", but I am one Basis guy here, being outnumbered 5 to 1 by the developer team 😮

[Reagan]

As a BASIS chap, why would I want to import a transport request and interrupt the upgrade activity just beacause the developer says so ?

Once I pass the lock development phase (Yes, I do inform the team) there are no requests accepted. The problem looks like the company you work for is not willing to co-operate in situations like these.

Now they come to me almost twice a day with another "let's push this one one in quickly, too ..."

Puch them back or ignore them.

If you keep accepting requests then you will end up in situations like this.

http://scn.sap.com/thread/3594806

PS: I know this whole procedure is a "no, no", but I am one Basis guy here, being outnumbered 5 to 1 by the developer team :-O

So what? Michael Jordan played (individually) against the big teams. Play like that.

What matters is the result and not personal relations.

Regards

RB

[george_credland]

As a BASIS chap, why would I want to import a transport request and interrupt the upgrade activity just beacause the developer says so ?

You don't. It isn't "just because the developer says so", the request is coming from the business teams to keep their operations running. "Don't shoot the messenger" comes to mind.

Push them back or ignore them.

Whatever you do NEVER ignore a request.

Yes speak up and ask what the changes are, the risk assessment and recovery plan (i.e. Change Management), but if you think your view is more important than the rest of the team you're headed for a rude awakening.

In the scenario given the changes were to keep a production line running. You ignore the request and it fails. The business now has direct costs as a result of the outage.

Incident review:

Production line supervisor spotted a problem and raised it in advance. Y

Development team investigated the problem as priority quickly identifying the solution. They make an impact assessment and believe it won't impact on anything else as its purely custom code. They follow due process and request the code to move through the landscape.

You ignored the request. The upgrade project is on track but the system is production system is compromised/unusable in the mean time.

You can pass the buck and say the supervisor should have planned ahead better, or the developers and business teams should have tested things better, but ultimately something needs to be done.

Most likely outcome management will take a view that the transports need to go through despite the objections. Maybe the supervisor gets fired, maybe not. What you're likely to face is disciplinary action for neglect. If it transpires that the problem could have been fixed quickly without fuss and instead was ignored leading to serious disruption you'll also be in the firing line.

Also consider what will happen in future when you make a mistake and need other people to help?

I head our combined devops team which means everyone is on the same side when it comes to working through problems rather than people looking out for themselves.

[Former Member]

Reagan Benjamin wrote:

As a BASIS chap, why would I want to import a transport request and interrupt the upgrade activity just beacause the developer says so ?
Once I pass the lock development phase (Yes, I do inform the team) there are no requests accepted. The problem looks like the company you work for is not willing to co-operate in situations like these.

Now they come to me almost twice a day with another "let's push this one one in quickly, too ..."

Puch them back or ignore them.
If you keep accepting requests then you will end up in situations like this.
http://scn.sap.com/thread/3594806

PS: I know this whole procedure is a "no, no", but I am one Basis guy here, being outnumbered 5 to 1 by the developer team :-O

So what? Michael Jordan played (individually) against the big teams. Play like that.
What matters is the result and not personal relations.

Regards
RB

Sorry, but you seem to get an awful many points here just plain wrong.

First, its not my choice whether or not to put the whole factory on hold, because of the upgrade.

Priorities matter, and in the end they brought me here to keep the systems running, *not* to take over the company.

The company doesn't have to "co-operate" with me for squat - they are the ones I gotta beg to sign my time sheets.

They pay the bills, they get to dictate the tune.

If the manager of a whole production line comes in and says, "transport stop? Buahaha, nobody told me and if they did that was yesterday. I need that fix now or my machines are gonna run out of material, my finished parts won't be registered and all hell is gonna break loose on the shop floor soon thereafter ..."

Then you don't tell them, "I am the big, bad Basis Admin and I am now the ultimate power inside your Universe..."

They big chief, me little Indian - those are the rules of the game.

Michael Jordan??? Pleaase, I can make jokes and spiffy remarks in here (even so some already find that objectionable), but "out there" its cold hard win or loose reality.

When millions of $ are at stake, nobody cares about the bygone glories of a former basket ball player.

And there is no I(ndividual) in Team, an old, ugly saying, that still holds very much true in SAP live today.

You gotta go with the flow and top priority must never be your own ego, not even to keep the systems running well - but to keep the company operating !

That was my top concern, not to mess up the system, but not to mess it up during DOWNTIME.

When errors happen in uptime, they may be fixable w/o too much business impact.

But when the system is in error *and* down, then white collared lynch-mobs start roaming the hallways, in search of someone to blame.

PS: Simply ignoring any manager's complaint is *never* an option, if you're the Basis guy.

[Former Member]

Dear George, thank you, thank you and thank you.

Couldn't have said it any better myself.

In here I might be b1tching (a lot) amongst fellow Basis sufferers, but "out there" I get paid to listen to people's requests, not ignore them.

[Reagan]

Activities like these are planned and possibly with a roadmap. All activities will be mentioned in the roadmap and actions assigned to every unit. We clearly mention that no transports will be allowed after the lock phase. Before you start the lock development phase you inform the development team and the business about this and if there are some transports to be imported you would wait until they are imported. Like I said before activities like these are planned and in the planning phase you consider the "dual maintenance" option if there is a need to do transports during an upgrade and if you are upgrading the production then this is not considered. Here is what the upgrade guide says and passed on to all those involved.

Impact on Development

At a certain point in time during the upgrade, the development environment of the system that is upgraded is locked. Consider this development freeze in your project plan and inform your developers about it. To avoid the development system being unavailable during the upgrade, we recommend that you add a temporary copy of your development system to your system landscape. This temporary development system can supply your production system with emergency corrections or support a phased development go-live after you have upgraded the original development system. You then have to make any corrections in the original development system as well as in the temporary development system. Make sure that your developers are notified about the dual maintenance.

I might be wrong but what I have told you is based on my experience. I haven't come across so called "emergency requests" all these years and I am sure it could be due to the fact that we plan the activities and communicate well (I am not saying you or the others don't do that) and if I start an activity without proper planning then I will get such requests.

If the manager of a whole production line comes in and says, "transport stop? Buahaha, nobody told me and if they did that was yesterday.

Don't you think this is a joke ? If the manager says why "transport stop?" then it makes me feel that he is not reading his mails or he is not aware of the upgrade or EHP update or there is no communication done (Read no planning).

You gotta go with the flow and top priority must never be your own ego, not even to keep the systems running well - but to keep the company operating !

I am not showing any ego. What I've said is how it should be done. Based on what you wrote it makes me feel that the development team is not co-operating during the upgrade. For me the top priority is the successful completion of the activity and that has got nothing to do with my ego.

Most likely outcome management will take a view that the transports need to go through despite the objections. Maybe the supervisor gets fired, maybe not. What you're likely to face is disciplinary action for neglect. If it transpires that the problem could have been fixed quickly without fuss and instead was ignored leading to serious disruption you'll also be in the firing line.

As long as you have a genuine reason why should someone get fired when there is a reason not to import transports during an upgrade ?

If the transport is going to fix an issue which is blocking the upgrade then yes you terminate the upgrade, unlock the system and bring in the transport because there is no other choice.

If the transport request is related to "Development team investigated the problem as priority quickly identifying the solution" then this can be considered and yes you do inform the business that this is not a recommended practice. Yes requests like these are understandable but requests one after the other makes me feel that there is some serious talk required.

Terminating the upgrade and unlocking the system to import transport requests is not an easy thing and we had a discussion about this sometime ago.

To conclude everyone has their own way of working. What I have said is my opinion based on my point of view and I am sure everyone has their own opinions, some agree and some don't and nothing personal.

Regards

RB

[Former Member]

Reagan B. wrote >For me the top priority is the successful completion of the activity and that has got nothing to do with my ego.<

Reagan, with that first name you ought to know that the first priority in any business of size is to keep the business going - and quiet often that's the only priority.

If I had that single minded attitude, they'd kicked my rear end outta here some time ago, and rightfully so.

'Nuff said, both DEV & PRD made it to the gates of Execution, so just a few more hours of anxiety and can go on to the next pile of headaches in my "career".

[george_credland]

For what its worth we communicate the project timeline well in advance and follow up closer to the time. If you have a large complex landscape with thousands of users it only takes one key person to get something wrong, forget to do something etc.

Yes you can point the finger at their poor planning, but ulimately you still have to be flexible to deal with the problem. As you acknowledge you would actually apply the transport rather than ignore the situation and incurr multi-million losses.

Based on what you wrote it makes me feel that the development team is not co-operating during the upgrade.

Line of communication. Business->Developers->BASIS. BASIS guy sees the requests from the developers so sees them as the problem. Developers don't make changes in isolation, it always

comes in response to business demands. In this instance it appears they're being pushed for last minute changes to keep things running from the business side. Blame the business users, blame the developers, it doesn't really matter you still have to deal with whatever arises.

I wish you continued success, but in over 21 years experience I've yet to meet anyone who didn't have an unexpected production problem at some point in time, and if this co-incides with an upgrade a wider view of the risks is needed of living with the problem or taking action.

[Reagan]

In this instance it appears they're being pushed for last minute changes to keep things running from the business side. Blame the business users, blame the developers, it doesn't really matter you still have to deal with whatever arises.

Won't it help all to release and transport all the requests before the "lock" phase? The developers can ask the BASIS to wait until the transports are imported and then continue with the lock. Again, I agree to do a transport in an emergency situation but stopping the update and importing requests is just a bad practice.

I wish you continued success, but in over 21 years experience I've yet to meet anyone who didn't have an unexpected production problem at some point in time, and if this co-incides with an upgrade a wider view of the risks is needed of living with the problem or taking action.

Thanks and yes I have had plenty of issues but they were upgrade related.

Regards

RB