h1. *Introduction*
I have made this
article based mainly on proposal I have created last summer after
finished tests on prototype systems (detailed document available on my LinkedIn profile), and additional tests and unofficial benchmarking data will follow later in new posts. This text is is mainly intended for those who own SAP on Itanium and Windows, and/or for those interested in SAP, Itanium and virtualization. Many things happened in the
mean time, one of which is the Red Hat's decision (by the end of last
December) to discontinue support for Itanium (or IPF, Intel's Itanium
Processor Family) in the announced future release RHEL 6. This was
not so unexpected, and RH states that this decision is made because
of the small number of Itanium systems sold in the recent period, and
which doesn't tend to grow. But this story is not likely to end that
fast (RH is fully supporting Itanium on RHEL 5 to the end of it's
life in March 2014, while giving extended support until year 2017
through OEMs), Oracle bought Sun who has SPARC architecture and other
hardware (there was a failed attempt to port Solaris to Itanium),
enters the Xen advisory board (
http://xenbits.xensource.com) with
people involved in Open Source contributions, Oracle Enterprise Linux
and Xen IA64 (Itanium) ports. RH6 is to be supported on IBMs POWER
architecture, and so are other "traditional" RISC/CISC CPU
architectures, x86 and x86_64
(
http://kbase.redhat.com/faq/docs/DOC-23403).
There are no clear indications from other software and hardware
vendors about Itanium's future (at least not the ones I deal with
here having their support: HP, SAP and Microsoft), but there are
current benchmarking results and research studies with predictions.
Back in 2003 my
company bought HP EVA 5000 storage and OpenVMS cluster with two Alpha
ES47 nodes to support Oracle 9i RAC database for our legacy
application, along with a rack of Xeon based Proliants (it's a kind
of tradition, before them we had Alpha and VAX systems). A year after
or more, it was decided that we should migrate to SAP. It was an
ambitious project: we had 9-12 months from start to going live,
implementing BW3.5, ERP 2004 with MM, WM (having MFC automated
warehouse), FI/CO, SD, PP, QM, PM, TM, logistics and procurement, etc
(maybe I have omitted something), everything that we had in the
legacy Oracle application but HR, CRM and some remote locations
(which were covered in later roll-out and optimization projects). We
also implemented Solution Manager as project base and maintenance
base, having all the necessary strict Q&A and GMP procedures. All
this had to be supported with appropriate hardware and
infrastructure. The first choice was (among others) was HP's offer
with 10 Integrity (Itanium based) servers and HP-UX. Somewhere in the
early beginnings, our project management decided to change OS
platform to - Windows !!! Today, five years later, we have at least
twice larger system (1TB database in size in production, number of
users - up to 800 with expected growth, additional affiliate
production plants, automated warehouses, end users, etc) working
quite good with only minor changes (added RAM up to 16-24G in
production, same number of CPUS: 2x mx2 1.1GHz per node), and some
infrastructure improvements. We never had any serious unplanned
downtime or system failures, performance, reliability, availability
and stability was predictable (apart from some OS problems with
Microsoft MSCS and one short storage outage).
But, in the past
year we had a large roll-out and optimization project involving ERP
upgrade from ECC5.0 to ECC6.0, and it showed that project members,
developers, IT staff and key users always needed (and still need)
additional sandbox systems, IDES demo systems, system copies and
similar for different purposes (from experimenting and unofficial
testing, to training and Q&A, project preparation and operation,
etc). This is showing the need for new level of flexibility which
could be only obtained with virtualization. There were many options,
from buying new physcial machines (very expensive even for less
expensive architectures), manual consolidation (MCOS and MCOD, with
or without Microsoft WSRM - Windows System Resource Manager), thin OS
(single kernel) virtualization (like VZ, Parallels Virtuozzo, but as
we have strict security standards and procedures which conflict with
OS patch level for VZ, this wasn't convenient even for testing), to
full virtualization platforms. It turns out that with Itanium you can
only have HP-UX VSE / Itanium VM or RHEL if you want official SAP
support (as explained later). There is also a very interesting and
sophisticated SAP solution - Adaptive Controller, but for now, Xen on
RHEL is doing a very good job. I have two HP BL870c hosts with 6
active sandbox systems (homogeneous copies of ERP development, Q&A
and production, BI/SEM IDES with Java instance, ERP IDES, etc) and
they all work properly and very stable. From pefromance standpoint,
HP-UX with Windows guests doesn't offer more at all (without AVIO
drivers, similar to PV guest drivers on Xen which don't exist only
for Itanium), the only thing I miss at the moment is Live Migration
on RHEL/Xen (which Integrity VM supports, as it also has some other
nice-to-have features). But I am able to move virtual machines to
different hosts manually without a problem (multipathing and shared
storage does it's work).
SAP supports
virtual guests, without responsibility about their performance, and
database on production systems should be on physical servers if
maximum performance is needed - otherwise, there aren't important
reasons not to do it even with database. We use Oracle 10g at the
moment (10.2.0.4) on Windows, which works fine on guest systems - but
because of Windows, it has to be HVM, full virtualization. For
optimum performance (and many other good reasons), the best option
would be migration to RHEL, including production systems (all the
tests show that it would be a smooth transition). There is a general
trend about migration of SAP systems from Unix to Linux (for all the
good reasons), while migration from Windows is less popular. Thing is
that 60% of all recently sold Itanium servers are Unix (read: HP-UX),
35% are Linux (RHEL and SLES), and only 5% are Windows
(first link among references).
Microsoft has no intention to introduce Hyper-V on Itanium (as Citrix
also doesn't currently, because the Xen code branch they bought
didn't cover ia64), but it is supported in all important flavours on
Windows Server 2008. Migration to a different hardware platform is
not an option, just as migration to HP-UX at the moment (current
number of Itanium servers does not justify that risk and expense).
But, the old aging hardware should be either upgraded (but it costs –
additional CPU costs almost as one BL860c, and though it's partly
consequence of local vendor's clumsiness, it is not very
reasonable), or replaced with new servers – or, used for
non-production (until dead), which is aligned very good with
virtualization (RHEL/Xen paravirtualization is available for old
Madison cores, but HVM isn't). There are better reasons to migrate to
RHEL beside this (money saving driver) – decision makers can decide
to change CPU architecture at one point, but with RHEL we can already
have the level of flexibility we need – far better than in any
other option: HP-UX is too expensive, and Windows doesn't offer it.
One important aspect is coming from GMP and other compliance issues,
so we need same platform both on test (Q&A) systems and in
production, and this justifies even more migration to RHEL5.
So,
the starting point was: SAP ERP and BI systems on Windows Server 2003
on HP Integrity servers and HP EVA storage virtualization solution, and the current stop is: some of those
systems working on Xen fully virtualized guest machines on Windows
Server 2003 and on HP Integrity blade servers, and some working on
paravirtualized guest machines on RHEL5. Next stop would be more
systems involved, and finally – everything migrated to RHEL, either
on Xen virtual machines or bare metal. Also, I find that Integrity blade servers are far more affordable (comparable to Xeon based Proliants) than expensive mid-range cell based andother Integrity flavours.
!
https://weblogs.sdn.sap.com/weblogs/images/251885800/rh04.jpg|height=504|alt=image|width=665|src=htt...!
*Current
environment and Itanium platform*
Currently, in the environment I am describing here, IPF (Itanium Processor
Family) architecture is being used exclusively for the needs of SAP
systems in my company. There are 18 such servers at the moment (10х HP rx4640 + 2х
rx2620 with Madison cores, 2х rx640 and 2х BL860c with Itanium
dual-core Montecito cores, 2х BL870c with Montvale cores), beside
additional x86 HP Proliant DL360 G4/5 servers (4x without additional
servers for support SAP routers and Web Dispatchers which are in
Microsoft Cluster for production just as SAP central services on
production and test systems, or on VMWare platform for support and
other purposes). Beside VMWare as main VTI for x86/x86_64 processor
families, HP EVA Storage nodes are used as form of FC SAN Storage
virtualization (also using MSA2012 for VMWare, additional EVA 8400
obtained, but not yet ready for production).
SAP landscape currently consists of:
BI
systems (development, test (Q&A) and production), BI7.0 SP18
CEN
system (transport domain controller, CUA, partly central
monitoring), NW04
Solution
Manager (MOPZ, EWA, monitoring, project roadmaps, ticketing, and
other)
SAP
routers (for end users, and for support and external access)
SAP
Web Dispatchers (BI portal, WebGUI)
network
printing servers with SAPSprint, SAP Web Console for some RF
terminals
different
sandbox and other systems: homogeneous system copies, traning
systems, IDES systems, etc. - all working as Xen guests at the
moment
h2. *Equipment
has life time (it is aging)
*
Among these 18 servers, 12 servers are entering
5th year of usage: rx4640/2620 servers on which lies the main SAP
landscape (it consists of ERD/Q/P, BWD/Q/P systems). All these
servers are very reliable and stable – there never was any serious
hardware failure or issue on production servers (or even any incident
as a consequence) until this day ! But, with hardware aging, support
becomes more expensive, additional components or spare parts also
become very expensive, and also vendor desupport dates are getting
closer and closer for equipment and different functionalities (usual
practice is to have maintenance renewal periods and IT equipment
amortization within 3-5 years, but sometimes it makes sense to extend
it). There are at least two possible roads: continued use and
planning about Itanium platform, or migration to another processor
architecture (as mentioned). If such migration is planned, then all
other technological aspects and existing options must be taken into
account (some less demanding and ”painful”, some are not, but
offer different advantages and overall total cost – just as
changing the OS platform, changing the server vendor, storage vendor,
etc. should be considered as well). Some kind of trade-in model or
amortization were not practice up to now as they are not offered by
all available vendors currently (used or refurbished equipment is
also not used in production systems in critical business environment like this) and probably it also
will not be in respective future. There is an option to stay on
Itanium platform by sole migration to Itanium blade servers which
has for consequence mandatory need for additional servers based on
current requirements for the main landscape, but without all the
other additional systems (this is not justified and will be explained
further on).
*Itanium
servers – unused brute force*
Main argument for Itanium system application is
usage within OLTP systems and databases – an example: for CPU patch
19 for Oracle 10.2.0.4 based on standard README instructions,
downtime lasts about half an hour (given for some internal Oracle
tests for a database with 2000 views and 4000 objects, while our
production system has at least twice as more), on production systems (old 2p mx2 rx4640 1.1GHz) it lasts about 5 minutes. But, except for biggest loads
during the day (peak is usually 13:00-15:00 on work days and in the
end of the month), this power mostly remains unused during the rest
of the day (average CPU load on our production does not go over 10%
on central instances with database and central service nodes, and up
to 35% on dialog instances, having at least 400 active users among
800 users) and this is expected state. Consolidated CPU usage and
other server resources usage can be realized by installing additional
SAP or DB instances on the same OS instance (operating system
instance, or in general OE, Operating Environment) on the same
server, allowed by SAP MCOD or MCOS installations – but, all there
many possible problems of coexistence of such instances during their
work, usage and maintenance life-cycle. That is one of the reasons
why consolidated OS environments should be isolated and that is
usually done through some form of virtualization. It can be justified
above all simply by using non-productive and productive systems and
their total cost – but, completely virtualized production
environments have became a standard experience today and also a need
for many business environments (if it is possible to scale resources
and estimate bottom lines, then it is just necessary to take into
account the “price” for virtualization).
Existing solutions for virtualization which were
considered while preparing this text: Citrix
XenServer, Hyper-V
/ Win2k8, HPUX
VSE/VM, vPars, nPars, Parallels
Virtuozzo / OpenVZ, Novell
Suse / Xen, Oracle
VM, Fedora
/ Xen, FreeBSD
/ Xen, Red
Hat Enterprise Linux (RHEL), and so is Centos / EPEL / xenbits, Sun
Virtual Box, QEMU/KV, VMWae - there
were also other solutions which were not considered for obvious
reasons - some solutions were eliminated from the start
because those are not available for IPF and are not even planned to
be supported and working on Itanium any time near (Hyper-V, VMWare,
XenServer, Virtual Box, and Oracle VM up to some point) but only
x86/x86_64 (I am dealing with SAP environment which is mostly based on IPF platforms and Windows
OS, apart from some remaining OpenVMS and Alpha systems). Following
criteria are applied on remaining solutions:
- only
full virtualization (hardware based, HVM) or paravirtualization
(PVM, more favoured) with hypervisor is supported (solutions which provide full
isolation) – HP vPars, Virtuozzo, OpenVZ represend OS / “Single Kernel“ virtualization which is not supported (by SAP and other
vendors) on production systems
all
other official requirements given by SAP as a software vendor for
our production systems, and by all other software and hardware vendors about platform support
fully
working installation prototype which confirms solution feasibility
in our system environment, e.g. boot from SAN
For HP-UX it is
also mandatory to have additional expenses for HP installation and
maintenance support including additional preparations, training and
similar activities which are not needed for RHEL (though, HP-UX
systems offer high level of availability, manageability and more,
they represent top of the business and industrial standard, but it is
very questionable if this is really needed). RHEL Xen still does not
support many advanced features (like Live Migration, PCI
passthrough, paravirtualized Windows drivers – HP-UX doesn't
support those drivers too, etc) which are easily enabled on other
platforms (x86, x86_64) or which HP-UX supports, not likely that all
of the will be, but some might be.
Furthermore, HP BL870c can support almost 4
completely isolated active guest systems (even on physically
different processors, CPU partitions avoiding kind of context switching – HP nPars can not go
in granularity under the physically available resources, including
CPU, Integrity VM can), and without a significant performance loss it
can support up to 7-15 such guest systems (having 16 HT cores)
compared to 1-core based bare metal systems, and more if high-end
performance and load is not expected. Of course, number of inactive
guest systems is almost unlimited – ready to be started and used
when needed at any time if resources are available, without
additional reinstallation, side-effects and server setup.
*Business
requirements*
Business requirements, coming from business case which emerges here during
development, testing and usage of all SAP systems, are putting high
demands about fast changes and mechanisms which enable such
flexibility having all the existing business standards and SAP usage
standards preserved (GMP and validation practice, QA,
availability/reliability SLAs above all, etc). Specific requirements
which arise from these are:
one
or more system copies in a given time (usually ASAP), homogeneous or
heterogeneous – as part of the strategy for testing and/or
development (but not for Q&A needs), and by given requirement or
expressed need (testing of new or risky features, system
types/usages and functionalities)
change
of the existing architecture / SAP Landscape – for instance, the
problem of the peak loads on the Q system during testing periods
(systems were never sized for these needs), Support Pipeline (to
circumvent transport change and validation requirements in order to
speed up cut-over after system upgrade, roll-out and similar),
training and sandbox systems – last two examples show in our
practice that system copies are far more efficient and usable than
any other solution (aligned with storage split-mirror technologies
like snapshots and snapclones) compared to unjustified client copies
or exports on systems with already more than 1 TB in DB size, which
can not support frequent requests for Q system data refreshments.
One temporary solution is also to combine those two procedures (as
used in R/3 system upgrade to ECC6.0 here during year 2009), and even
more efficient would be to slightly change validation practice
(which in essence remains same, but using far more efficient system
copies). This is aligned with SAP note
432027 - Strategy for using SAP Support Packages, which
describes additional sandbox and test system usage (e.g. sandbox in
Upgrade Master Guide) and as part of the official landscape.
*A
step further*
(a solution and a problem) would be involvement of additional
development and Q&A systems, but also using more efficient
change management and automated test procedures using SAP Solution
Manager and eCATT tools which we already have available at hand
without additional investments (but there also many, many other
useful tools) except for additional planned effort by validation and
different SAP teams. Every testing of changes during any of the SAP
projects is a convenient opportunity for preparation and
implementation of such tools and solutions.
Changes
during system patching or system upgrade which require alternate
system landscapes (as during the former mentioned upgrade in year 2009),
*Solution* - Pool of Servers
The strategy which enables the fastest possible
response on large number of requests: certain number of servers is
prepared like an “on hold“ template (firmware / hardware, FC /
network, SAN storage and other needed resources, then the OS
installation, Windows domain name reservation and AD infrastructure
settings, antivirus / firewall / proxy, account specifications and
all other OS settings, Oracle RDBMS and SAP s/w installation), and
then by a well defined procedure a copy is made using HP Storageworks
Business Copy technology in a shortest time according to SAP
standards (homogeneous copy based on database copy, with additional
post-installation activities which last not more than an hour).
Server in the pool can be activated in a very short time, deactivated
or renewed with fresh data (just by making database copy and
post-installation tasks again).
The
only problem with this solution (as mentioned about the missing
features for Xen on Itanium) is a very poor disk I/O. This is the
consequence of the needed full virtualization (HVM) for Windows
guests (3-6 times slower than on bare metal, while network I/O is
less deteriorated). This makes critical impact on database
performance in some cases (though parsing is done faster than on
older bare metal systems), which makes the whole ERT system in
average slower (e.g. closing the financial period on guest system takes 4
hours in batch instead of 2.5 hours as in production system). There are many workarounds and solutions for this
problem - using paravirtualized I/O drivers if it was possible,
iSCSI/NFS storage approach (not top performance, again), etc - but
the best thing would be moving database to physical RHEL machine, using
MCOD wherever possible. Database nodes wouldn't be easily
reallocated, but one of the possible solutions in future would also
involve adaptive infrastructure (SAP ACC, with Solution Manager -
Adaptive Computing Controller) as part of the SAP strategy for system
usage and consolidation:
all
SAP systems can adapt to business needs very fast and in a flexible
manner
dynamic
load balancing according to system load (physical and virtual)
easier
SAP landscape maintenance
Optimal
solution would involve ACC because not only that it avoids better
possible human errors, but it also follows vendor recommendations and
standards better given with SAP Solution Manager as the base
technical tool which is really useful in all system maintenance and
management tasks according to GMP / ITIL / ITSM, and which is free of
charge (unless additional Service Desk Extended component and
functionalities are needed for non-SAP systems and Service Desk /
Help Desk, and similar). Therefore it makes sense to make additional
effort and improve this system and it's usage in our environment.
!
https://weblogs.sdn.sap.com/weblogs/images/251885800/vnc.jpg|alt=image|src=https://weblogs.sdn.sap.c...!
*Realization options*
These are the only existing usable IPF
consolidation platforms for pool of servers and their perspectives (having Windows on Itanium systems):
acquisition
of new physical servers (without hypervisor and virtualization)
Pros:
most efficient (but not optimal) use of existing knowledge and
resources
Cons:
price of 1 Itanium server is between $5К and $20К in average, and
more (without support agreements and other hidden maintenance costs,
human labor)
HP-UX
Integrity VM hypervisor:
HP
Integrity VM (software virtualization) as part of VSE (Virtual
Server Environment) offers also full isolation, as opposed to vPars
(similar to OpenVZ and appropriate Parallels Virtuozzo solution)
HP
nPars as part of VSE (cell based virtualization, similar to IBM
LPARs, HP-UX as hypervisor is only controlling them), demanding
specific cell-based (NUMA) hardware (rx7640 or stronger, like Superdome)
Pros:
a
highly stable platform which represents on of the leading industry
standards
Cons:
the
licenses only for MC OE (DCOE, VSEOE now) are about $10000 per CPU
(TCOE
as a possible minumum is more than $2000 per CPU), performance factor
Xen
with RHEL5 as hypervisor
Good
side: supported
by SAP and HP, Open Source, quite stable and tested, in-house
knowledge available.
Cons:
does
not support all the features other options have
(at least some of the
features like live migration), performance factor
All these solution are stable and robust, but maintenance and support for RHEL / Xen infrastructure is about 1200 euros per server for RHEL Advance Platform (unlimited sockets, unlimited number of guests, cluster suite and GFS included) and that makes it most optimal for system consolidation.
*Realization steps*
In general, following implementation gradual
phases and steps are proposed in a short overview:
- Prototype
preparation and testing (in process).
- Virtualization
of all IA64 sandbox (and test) systems and preparation of pool of
servers (for system copies), Solution Manager (SOD), dialog
instances of development systems.
- In
parallel, all 32-bit SAP servers on VMWare platform should be
virtualized (old Solution Manager, IDES, CEN; even SAP routers, but
not those for end users, for a start).
- Complete
virtualization of development systems.
- Test
drive virtualization for Q&A and production systems (they must
be aligned, dialog instances first).
- Complete
virtualization of the whole SAP landscape.
Taking Q&A servers together with productive
servers is necessary because of the nature of their intended usage.
The last phase is not yet planned (if it ever will be), at least not
with a final date or specific requirements at the moment (higher
reliability (disaster recovery), greater security and scalability).
This might justify HP-UX as an option as it's implementation has
references nearby, with full implementation and integration support
from HP in a critical environment (with given dates).
For the successful implementation of these phases,
good preparation is crucial, specially about planning needed
resources:
storage
space
available
network and optical ports and switches, settings on firewall, etc.
needed
total number of servers in the landscape (categorized according to
usage types and associated with possible hosts)
this
provides information needed to estimate people-days needed for
implementation of the pool of servers in each of the phases, and
also physical resources needed for the requested service levels
according to user requirements.
Much
of storage space is saved using HP BC Snapshot functionalities (as
during upgrade), but these also have some level of impact on EVA
performance and puts constraints on original VLUN, and makes storage
maintenance more complex. Price of the EVA FC disks is about 15-20
eur/G which also has to be noted (snapshot grows up to about 10% of
the original disk size in a normal usage period, which gives around
1.5-2 eur/G) – there are other storage options, but there also
other parts of the TCO structure
backup
(about 7 eur/G in averga, but varies from 1 to 15eur)
network
equipment and security (antivirus, firewall, LAN, FC), licenses
guest
OS licenses and support agreements
infrastructure
in the data center room (space, air conditioning, power supply)
system
maintenance (human resources, licenses for some monitoring
solutions)
other
hidden expenses ...
Therefore, it is of the utmost importance to have
business requirements carefully prepared and estimated, because all
real technical requirements (like additional hardware or licenses,
memory, servers, storage). For instance, if large enough number of
servers is hosted (which grows very easily with virtualization), so
grows the I/O bandwidth on SAN or the number of storage spindles.
*Details
about implementation*
Installation of current environment is divided in three groups:
installation
of hosts (Dom0s)
installation
of HVM (fully virtualized) Windows guests
installation
of PV (paravirtualized) RHEL guests
While the installation of hosts and PV guests has
many similar steps (network setup – I am using trunk utility as
local NTLM proxy for http/ftp, RHN registration, ntfs-3g uitilities
installation, etc), there are things to be set only on hosts (xend
service and boot parameters, HP support pack installation, etc).
For
guests I am generally proposing using copies of the template systems
for many different reasons, one of which is using good features of
EVA storage (snapshots and snapclones) or LVM on local disks. If
making guest from the scratch (or new template), it is always best to
use physical (phy:/) drives instead of files – they have better
performance, enabling moving to another host (with shared storage),
and always use multipathd.
I am giving the following hints for people who
have experience with Windows installations and SAP administration on
Itanium (and who have some basic knowledge and awareness of things
about Linux) – many things are similar, but there are also many
misleading details.
*Host Installation *
- RHEL installation is quite simple and straightforward – no EFI boot
driver is needed (Windows installation needs it), usual server
preparation, firmware update if needed, boot device configuration in
EBSU setup disk – boot from SAN is used, so HBA adapater should be
configured there and a VLUN prepared, after booting from installation
disk, installer should be started manually by entering “linux
mpath vnc=1”
(vnc is optional, GUI is then available later through vnc client on
screen 1, or http://that-host:5901)
- installation number can be entered after the installation (using
rhn_register), Virtualization group should be chosen if available
(beside Development, Web Server and others) - or later, packages to
be installed are: xen, kernel-xen, xen-ia64-guest-firmware,
virt-manager, libvirt and virt-viewer, or just “yum groupinstall
Virtualization” (if doing it manually with rpm -ivh in VT folder of
the installation disk: libvirt-python, python-virtinst,
Virtualization-en-US, perl-Sys-Virt, xen-devel ../Server/bridge-utils
../Server/xen-libs ../Server/gnome-python2-gnomekeyring
../Server/iscsi-initiator-utils ../Server/gtk-vnc
../Server/gtk-vns-python ../Server/cyrus-sasl-md5)...
installation lasts up to 1 hour – all setup parameters (like
network address, gateway, etc) should be prepared
...
check if /boot/efi/efi/redhat/elilo.conf is set correctly before
restarting
...
/etc/resolv.conf should contain at least one line "nameserver
DNS_IP_address" and one "domain group.hemofarm.com",
enabling and disabling network interface is done with ifup eth0 /
ifdown eth0
- on
some bad terminal emulation/clients after restart, firstboot gets
stuck (additional setup wizard), which shuts down automatically after
about 10 minutes, and after logging in it can be disabled with
“chkconfig firstboot off”
- trunk (local NTLM proxy, [http://ntlmaps.sourceforge.net | http://ntlmaps.sourceforge.net/])
after unpacking with tar -xzf has to be configured
(NT_DOMAIN:HEMONET, USER:proxyuser, PASSWORD:somepass,
PARENT_PROXY:proxy_server) and started manually with “scripts/ntlmaps
&”
(or put into /etc/rc.local to start automatically after boot), then
it is possible to start “rhn_register
–proxy=http://127.0.0.1:5865”
- also, for other utilities accessing internet, like yum (package
installer), edit ~/.bash_profile (using gedit in GUI console, or vi
editor):export
http_proxy=http://127.0.0.1:5865
export
HTTP_PROXY=http://127.0.0.1:5865
export
ftp_proxy=http://127.0.0.1:5865
export
FTP_PROXY=http://127.0.0.1:5865
...
update can be started with “yum update” after setting up software
channels and other additional settings on RHN side (if needed)
NOTE:
LVM/multipath is not good is not good with FAT partitions, so before
each kernel update/installation it is needed to mount manually
/boot/efi (e.g. with “pvs” one can get sytsem disk device for
system VG LogVol00, like /dev/sd..., for instance /dev/sdh2, and then
EFI partition is mounted with: “mount -t vfat /dev/sdh1 /boot/efi”)
using Windows (cifs) shares: e.g.
serverx$ can be mounted on
/mnt/tmp with:
mount
-t cifs -o dom=HEMONET,username=someuser //server/x$ /mnt/tmp
...
or for simple copying smbclient is useful (like ftp)
- ntfs-3g installation (http://www.ntfs-3g.org):cp
/mnt/tmp/rhel/ntfs-3g*.tar
tar
xvf ntfs-3g....
cd
ntfs-3g...
./configure
make
make
install
yum
install ntfsprogs
- firewall should be configured (opening needed ports,
system-config-firewall)
and selinux (for a start and troubleshooting, “setenforce
0”
and SELINUX=permissive in /etc/selinux/config)
- Xen
Dom0 configuration for several network adapters, create
/etc/xen/scritps/network-multi-bridge:#!/bin/sh
dir=$(dirname
"$0")
"$dir/network-bridge"
"$@" vifnum=0 bridge=xenbr0 netdev=eth0
"$dir/network-bridge"
"$@" vifnum=1 bridge=xenbr1 netdev=eth1
"$dir/network-bridge"
"$@" vifnum=2 bridge=xenbr2 netdev=eth2
"$dir/network-bridge"
"$@" vifnum=3 bridge=xenbr3 netdev=eth3
...
and then: “chmod
+x /etc/xen/scritps/network-multi-bridge”
and edit /etc/xen/xend-config.sxp, putting “(network-script
network-multi-bridge)”
instead of “(network-script
network-bridge)”
...
also, put there (dom0-min-mem 2048) ... and dom0_mem=2G into
elilo.conf to reserve 2G for Dom0 and avoid memory ballooning (2G
should be enough) – elilo.conf example:
...
image=vmlinuz-2.6.18-164.10.1.el5xen
vmm=xen.gz-2.6.18-164.10.1.el5
label=2.6.18-164.10.1.el5xen
initrd=initrd-2.6.18-164.10.1.el5xen.img
read-only
root=/dev/VolGroup00/LogVol00
append="dom0_mem=4224M
-- palhalt rhgb quiet"
...
- vncserver setup (/etc/sysconfig/vncservers), vncpasswd should be set
for the running account, and initial vncserver start (“chkconfig
vncserver on” for the service, too)
- hp
support pack installation:yum
install net-snmp net-snmp-libs
./install.sh
(options:
1,2,3,8)
yum
install tog-pegasus
service
tog-pegasus start
domainname=group.hemofarm.com
is needed in /etc/sysconfig/network - in case of additional problems
with hpsmhd start, in /etc/hosts should be put:
host-IP-address
host.group.hemofarm.com host
...
initial configuration “/etc/init.d/hpima
sample ”
(or better, reconfiguration, especially after each kernel update):
/etc/init.d/hpmgmtbase
reconfigure
...
(and additional restart of services: tog-pegasus, hpmgmtbase, hpima,
hplip, hmsmhd)
/etc/init.d/hpima
reconfigure
/etc/init.d/hpima
start
...
kernel parameters and related settings and recommendations (by
Oracle, SAP):
/etc/rc.local:
ulimit
-u 16384 -n 65536
modprobe
hangcheck-timer hangcheck_tick=30 hangcheck_margin=18
/etc/sysctl.conf:
kernel.msgmni=1024
kernel.sem=1250 256000 100 1024
vm.max_map_count=300000
net.core.rmem_max=1048576
net.core.rmem_default=1048576
net.core.wmem_max=262144
net.core.wmem_default=262144
- old
A/P storages should have excluesively Failover (not failback, not
None) in the prefered path in the presentation options (mixed environments like I have at the moment are very problematic - upgrade to A/A is more than recommended, it is just about uggrading firmware but needs planning as a potential risk)
- multipath.conf example (starting the service and setting chkconfig
also is needed) - example content of /etc/multipath.conf:
blacklist {
devnode "
(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"<br /> devnode "(hd|xvd|vd)[a-z]
"<br /> wwid ""
}
# Make sure our multipath devices are enabled.
defaults {
udev_dir /dev
polling_interval 10
selector "round-robin 0"
path_grouping_policy group_by_prio
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
prio_callout "/sbin/mpath_prio_alua /dev/%n"
path_checker tur
rr_min_io 100
rr_weight uniform
failback immediate
max_fds 65536
no_path_retry 12
user_friendly_name yes
}
blacklist_exceptions {
wwid "3600508b4000129f70002900002030000"
wwid "36005
"<br /> wwid "36001"
}
multipaths {
multipath { /* for newer A/A storages, from EVA VCS4.0
/<br /> wwid 3600508b4000e302d0000a00001200000<br /> alias sap-trn-sys<br /> }<br /> multipath { / for A/P storage, up to EVA VCS4.0
/<br /> wwid 3600508b4000129f70002900000cb0000<br /> path_grouping_policy multibus<br /> path_checker readsector0<br /> prio_callout /bin/true<br /> no_path_retry queue<br /> rr_weight priorities<br /> alias sap-bisem-sys<br /> }<br /> ...<br />}<br /><br />devices {<br /> device {<br /> / EVA 3000/5000 with new firmware, EVA 4000/6000/8000, EVA 4400
/<br /> vendor "(COMPAQ|HP)"<br /> product "HSV1[01][01]|HSV2[01]0|HSV300|HSV450"<br /> getuid_callout "/sbin/scsi_id -g -u -s /block/%n"<br /> prio_callout "/sbin/mpath_prio_alua /dev/%n"<br /> features "0"<br /> hardware_handler "0"<br /> path_selector "round-robin 0"<br /> path_grouping_policy group_by_prio<br /> failback immediate<br /> rr_weight uniform<br /> no_path_retry 12<br /> rr_min_io 100<br /> path_checker tur<br /> }<br /> / MSA */
device
{
vendor "HP"
product "MSA2[02]12fc|MSA2012i"
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
features "0"
hardware_handler "0"
path_selector "round-robin 0"
path_grouping_policy multibus
failback immediate
rr_weight uniform
no_path_retry 18
rr_min_io 100
path_checker tur
}
}
- Data Protector installation (MUST be started from the PA-RISC disk,
e.g. B6960-15008-01.tar.gz or downloaded from ITRC SUM):yum
install xinetd
(if not installed)
...
add port 5555 in iptables firewall (/etc/sysconfig/iptables "-A
INPUT -m state --state NEW -m tcp -p tcp --dport 5555 -j ACCEPT")
./omnisetup
-source /root/hp/B6960-15008-01/ -server dprotector.group.hemofarm.com -install da,ma,cc
chkconfig
--list | grep omni
...
if no service found, edit manually /etc/xinted.d/omni:
service
omni
{
socket_type
= stream
protocol
= tcp
wait
= no
user
= root
server
= /opt/omni/lbin/inet
server_args
= inet -log /var/opt/omni/log/inet.log
disable
= no
}
...
INFO: linux rescue is a method of disaster recovery, by booting
installation disk with “linux rescue” or better, “linux rescue
mpath”, and it is also possilbe to do a reinstallation or upgrade
over the existing OS file system (as a repair method) by choosing any
of these options, guided by the installer dialogues
*!!!!!
NOTE: always, always make backup of everything that is important ...
</p><br />
*h2.
PV Guests
sqlplus
/ as sysdba ...
<br /><br />- SAP:
su -ertadm
startsap r3
stopsap r3
(using
all instead of r3 if Java stack is present, as it is with BI systems)
- shutting down:shutdown
-h 0
(as with other guest or bare metal systems, there is always a risk
with forced shutting down of damaging system disk or something more –
linux rescue boot with prepared sap-rescue configuration with
affected system disk would help, or restore from a backup ...) or
init
0
- guests can be backed up just like any other machine using Data Protector or other usual tools (no FC tape medi, though, unless using PCI passthrough which is not available on Itanium even though it has VT-d hardware support)
HVM Guests<br />
- It
is enough to copy existing template configuration (e.g..
/etc/xen/sap-test) for a new guest, having at least changed
name=new_machine (instead of name=sap-test), and also
/usr/lib/xen/boot/nvram_sap-test into
/usr/lib/xen/boot/nvram_new_machine(this
is instead of nvrboot import from the backup on the EFI partition –
HVM guests have their own EFI environment, just as any bare metal
machine)
- for
new guest disks created either as snaphots / snapclones or CA
replicas of the template disks, check the parted /dev/disk/by-id/...
print before proceeding – if any I/O errors occur, try changing in
EVA Command View the Presentation / Preferred path/mode on old EVA
5000 (A or B, when upgraded to new A/A VCS4.xx firmware this will no
longer be a problem).It
recommended to use for each VLUN it's wwid set accordingly in
/etc/multipath.conf with appropriate /dev/mapper/alias ...
With
ntfsls utility one can even check the M$ partition without any
mounting
- for
each new disk (which is not a copy/replication, not having gpt label)
initialization must be done: parted /dev/disk/by-id/... mklabel gpt
(it is also possible as usual in EFI shell) ...
- after Windows boot and administrator login, first the IP address
should be set in the GUI console (vnc) and remote desktop enabled
(for a newly installed machine, it should be already set in the
template) and then continue working with RDP (rdesktop)
- the
next step is changing the name to the machine and adding it to Winows
domain: newsid /a novoime (restart should be done only from
virt-manager or with xm create, anything other will fail to bring up
network correctly)
- machine preparation is the same as with any other Windows machine
(Oracle and SAP installation, mind installing Montecito Oracle
patches if needed)
- e.g. SAP
ERT 30 instance in the template with preinstalled ERT instance can be
used only for the same machine, which comes down to same MAC address,
having same database schema and licenses preserved with (otherwise,
additional setup is needed):... on
the old system:
exp file=lic.dmp userid='/ as sysdba'
tables=(SAPERP.SAPLIKEY)
... on the newly copied system:
imp file=lic.dmp userid='/ as sysdba' fromuser=SAPERP touser=SAPERP
- SAPDATA_HOME and oraarch folders (or whatever set in sapinst for the
template) are best to be kept on C: - HVM guests have that
unfortunate constraint to have maximum four drives
*!!!!!
NOTE: always, always make backup of everything that is important ...*
*KNOWN
PROBLEMS*
I
had several problems during during all these tests, and these
are the most significant issues:
EFI
environment can slow down vnc console and virt-manager refresh
terribly for unknown reasons (xm commands are usable, but also a bit
slower, while the guest domains and Dom0 work perfectly normal) –
a RH service request is open on this one
ORA-07445
occurs intermittently on Dom0s and DomUs (but does not on normal
kernel) – a metalink SR is open on this one (bug 8868468) - the
only viable workarounds are not to use Xen for Oracle, or to somehow
prepare database without Xen, and then use it under Xen (this is not
causing serious issues, but it would be unacceptable in producion)
References
Note 962334 - Linux: SAP on Xen virtual machine
Note
1048303 - Red Hat Enterprise Linux 5.x: Installation and upgrade
Note
958253 - SUSE LINUX Enterprise Server 10: Installation notes
Note
941735 - SAP memory management for 64-bit Linux systems
Note
964705 - Linux IPF: "floating-point assist fault" in Linux
syslog
Note
784391 - SAP support terms and 3rd-party Linux kernel drivers
Note 527843 - Oracle RAC support in the SAP environment
Note
592393 - FAQ: Oracle
http://www.infoworld.com/d/open-source/red-hat-will-drop-itanium-support-in-enterprise-linux-6-083
http://en.wikipedia.org/wiki/Xen
Linux Supported platforms
Supported Platforms
homogeneous copy x64 to ia64
Re: System copy from SuSe Linux-IA64_64 Ent to SuSe Linux-X8_64 Ent
SAP on Linux:
SAP on Linux
http://www.sdn.sap.com/irj/sdn/index?rid=/webcontent/uuid/e338fb57-0d01-0010-2cb8-e728f66b715f
http://wiki.sdn.sap.com/wiki/display/HOME/SAPonLinuxNotes
NovellXenSetup:
http://wiki.sdn.sap.com/wiki/display/HOME/NovellXenSetup
http://www.tpc.org/tpcc/results/tpcc_perf_results.asp
http://www.tpc.org/tpcc/results/tpcc_price_perf_results.asp
http://ts.fujitsu.com/ps2/press/read/news_details.aspx?id=2202
http://www.oracle.com/corporate/press/1524171.html
http://www.nec.co.jp/press/en/0802/2101.html
http://en.wikipedia.org/wiki/Rate_of_return
http://en.wikipedia.org/wiki/Total_cost_of_ownership
SAPS
http://www.sap.com/solutions/benchmark/measuring/index.epx
benchmarks:
http://www.sap.com/solutions/benchmark/index.epx
http://service.sap.com/benchmark
http://www.sap.com/solutions/benchmark/sd3tier.epx?num=200
http://www.sap.com/solutions/benchmark/sd2tier.epx