vCO - bug discovered on initializing variables

After examing Deric's question in VMware community for vCO I was a little bit confused of my testing results - could that be? - is it a feature or a bug.
Now we have a first statement, it is definitly a bug. Read more at http://communities.vmware.com/thread/296963.

Regards, Andreas

vCO - cluster AAM compliance check

Normal productive clusters are spread over two or more fire compartments. But only maximum five hosts in a cluster are primary hosts for AAM. What happens if all primary hosts reside in one section and this section fails?!  Check your cluster AAM fire compartment compliance:

The following code is compressed in one task for better viewing. For production use split it up in action & workflow(s), depending on sections. It will check, if minmum 2 host in each cell (fire compartment) is a primary host. Adapt these values for your use.

Parameters:

• (input) Cluster [VcClusterComputeResource]: the cluster to be checked
• (input) Cell1, Cell2 [Array/VcHostSystem]: an array containing the hosts in each fire compartment (e.g. use vCO configurations)
• (output) compliant [boolean]: the cluster compliance
• (output) Cell1Primary, Cell2Primary [Array/VcHostSystem]: primary hosts found for cell

Depending on compliance state you can use the both arrays in a next step email workflow to inform about the cell compliance.

If you are running vSphere 4.1 you can use DRS groups to get already defined host lists:

<Cluster>.configurationEx.group contains the group listing (array).
Cluster.configurationEx.group[x].host constains the host list of group x (array). You can use this to feed the parameters Cell1 & Cell2 dynamically with values from vSphere.

Regards, Andreas

X-Mas - a little gift: MCS-ViewHA

How to see the primary Nodes in a VMware HA Cluster?

As you know, VMware vSphere has really nice features. One of then is the VMware HA or VMware High Availbility feature. With this, it is possible to build a high availbility Cluster for virtual maschines. Any virtual Machine in that cluster are protected for Hosts crashes.

The HA Feature monitors all ESX-Hosts in the Cluster and restart any virtual maschine on other ESX-Server if one or more ESX-Servers are crashing. So the virtual machines are back and running in a few minutes.

A VMware Cluster can be build with at least max. 32 ESX-Hosts. You must configure it by the vCenter Server, but pretty nice is, that the VMware HA Feature works without the vCenter Management Server. This means that the ESX-Host is montoring itself and all other ESX-Hosts over heartbeats.

All Server do it?... Not all!

In a VMware HA Cluster are max. five ESX-Server primary Hosts. All other ESX-Hosts will be secondary Hosts. Only the primary Hosts will be have Information about the whole Cluster. The secondary Host does not have that Information. But in a kind of failure, a secondary Hosts can be promoted to a primary Node by any primary Node.

In the normal Szenario, if one primary Host goes down, the other primary Node realize that over the heartbeat and promote a secondary Hosts to a primary Host.

But what happend, if all primary Hosts goes down at the same time? Now the Cluster has a problem, that no ESX-Hosts has Information about the whole Cluster and can´t promote any Hosts to a primary state. Did you think that this szenario are infrequent? We have seen this really often.

One Example is, when you build a Cluster over two locations. The only secured way for that szenario is, to build up a Cluster with max. of 8 ESX-Hosts, so you can sure, that at least one Host per location is a primary host.

But what can you do if you want to build a cluster with more than 8 Hosts? Are you sure that you have primary Nodes on all Locations? Where can you seen this? The vCenter Client dosn´t show you this Information.

In many Project we are asked which ESX-Server in a VMware Cluster are active (primary) and which one was passive (secondary). So we build a little Tool that display the primary Nodes in a VMware Cluster what we named MCS-ViewHA. With that, you can see which ESX-Node is a primary Node.




You can Download this free Tool from our Website.

http://www.mightycare.de/downloads/mcs_viewha

vCloud Director - installation part I

Based on the actual evolutions in the VMware product environment, I decide to installl the new vCloud Director in my home-lab. There are two things which i had to get before starting:

• Oracle DB 11g for Windows
• RHEL 5 64bit

After several register processes and hours of downloads I decide to start with the oracle server based on windows. In the future I think i will run it under RHEL also.

Starting with the Oracle DB server there are several things which need my attention, because after the installation the Enterprise Manager (http://localhost:1521/em) was available to me but the listener only reacts on the 127.0.0.1 address. In my opinion there was no way to change it while installing.

First thing i try was diabling the Windows firewall :-) , but without success. Next thing i try was a special command:

tnsping oracle.cjo.local

which reports an error while connecting. After that i try to get an status of the actual configuration with:

lsnrctl status

which shows several TNS-xxxx errors (for example: TNS-03505: Failed to resolve name). After reading some support statements i stopped all services in the server manager.

Then i go to the dbhome_1 directory and edit the tnsnames.ora and listener.ora (configuration files for the service) and change all localhosts to the right DNS name.

tnsnames.ora

listener.ora




After that i start all stopped services (OracleDBConsoleoracle, OracleMTSRecoveryService, OracleOraDb11g_home1TNSListener and OracleServiceORACLE) and check the status via:

lsnrctl status

which shows the right connection name.

Also i try another

tnsping oracle.cjo.local

which shows the following:



Next part now is to install the vCloud Director and try if the connection works. An update will follow...

vCO - get ESX SSL thumbprint

To add a new ESX host to virtual center by vCO you need the SSL thumbprint of this host.
The following scriptable task shows a simple solution. To keep the example as simple as possible, most parameters are defined local. For production use declare these parameters (port, userName, ...) as input parameters.

For example, you can modify the VMware workflow 'add host to cluster' as shown below, to add a host with self signed or unknown CA to your cluster:

Feel free to leave comments.

Regards, Andreas

vCO - get performance data from VM & build graph

Retrieving performance data from an entity is a little bit complex not only because of the nested parameters. To reduce code and make it easier for starters to step in, we just grab the average CPU in MHz from a VM about the last hour and build a graph like this:

For this example the code is placed in one scriptable task. For production purpose it is mor sensible to split it in several actions & workflows.

• Input Parameter: VM [VcVirtualMachine]

Part I - retrieve performance data

setting interval:

var end = new Date(); // now
var start = new Date();
start.setTime(end.getTime() - 3600000); // 1h before end
System.log (end.toUTCString());
System.log (start.toUTCString());

Look at the logged time stamps. They are in UTC and later on also in graph. If you want to adjust this to client time, you have to recalc the time stamp for CSV using Date.getTimezoneOffset().

create querySpec (here for only one VM)

var querySpec = new Array();
querySpec.push(new VcPerfQuerySpec());
querySpec[0].entity = VM.reference;
querySpec[0].startTime = start;
querySpec[0].endTime = end;
querySpec[0].intervalId = 20; //or use 300 for 5 minute stepping

create perfMetricId for one metric (CPU average in MHz) and call perfManager

var PM = new VcPerfMetricId();
PM.counterId = 6; //6 = cpu.usagemhz.average
PM.instance = ""; // no instances
var arrPM = new Array();
arrPM.push(PM);
querySpec[0].metricId = arrPM; //assign PerfMetric to querySpec
querySpec[0].format = "csv";

var CSVs = VM.sdkConnection.perfManager.queryPerf(querySpec);

Now the array CSVs contains one VcPerfEntityMetricCSV object - we only called one - nevertheless i will iterate over CSVs so you can reuse the code

Part II - join data and time stamps in a CSV file

for (i in CSVs)
{
    var CSV = CSVs[i];
    var Temp = CSV.sampleInfoCSV.split(",");
    var Sample = Array();
    for (j in Temp)
    {
        if (j % 2 != 0)
        //only use odd entries, they contain the sample time - even ones contain interval
        {
            Sample.push(Temp[j]);
        }
    }
    var Values = CSV.value[0].value.split(","); // the MHz values

    var BaseName = "C:/Test/" + workflow.id;
    var CSVname = BaseName + ".csv";
    var ControlName = BaseName +  ".control";
    var PNGname = BaseName + ".png";
    var FW = new FileWriter(CSVname);
    FW.open();
    FW.lineEndType = 1;
    for (j in Sample)
    {
           FW.writeLine(Sample[j] + "," + Values[j]);
    }
    FW.close;

Using workflow.id to build the file names makes them individual. So you can call the workflow parallel without having duplicate file names.

Part III - generate graph

To do this there are some requirements:

• enable local execution for vCO
• download gnuplot and unzip - in this example it is unzipped to C:\test\gnuplot

First we have to build the control file for gnuplot for manipulate graph rendering. I've changed only some basic parameter - if you are familiar with gnuplot, just add more parameters to get a better look. At the end we just call gnuplot with our control file.
Pay attention on the single quotation marks. We have to mix them up to get the double ones in control file.

    var FW = new FileWriter (ControlName);
    FW.open();
    FW.lineEndType = 1;
    FW.writeLine ('set datafile separator ","');
    FW.writeLine ('unset key');
    FW.writeLine ('set title "performance data ' + VM.id + ' [' + VM.name + ']' + '"');
    FW.writeLine ("set terminal png");
    FW.writeLine ('set output "' + PNGname + '"');
    FW.writeLine ("set xdata time");
    FW.writeLine ('set timefmt "%Y-%m-%dT%H:%M:%SZ"');
    FW.writeLine ('set format x "%H:%M:%S"');
    FW.writeLine ("set xtics rotate");
    FW.writeLine ('set ylabel "MHz"');
    FW.writeLine ('plot "' + CSVname + '" using 1:2 wi li');
    FW.writeLine ("quit");
    FW.close;
    var cmd = "cmd.exe /c C:\\Test\\gnuplot\\binary\\gnuplot.exe " + ControlName;
    System.log (cmd);
    var CMD = new Command(cmd) ;
    var Result = CMD.execute(true);
    System.log ("GNUplot: " + Result);
}

That's all - feel free to leave a comment - regards, Andreas

vCO - rights management for WebViews

Yesterday i try to publish a WebView which should be used by only one user group of my Active-Dirctory. After a few attempts i decide to describe the whole process with some pictures. So the goal for todays article is to get only access to the workflow based under "Customer2".

At first you have to define the rights at the root object (Edit access rights....):




Because of the rights heredity you have to enable minimum the "View" right for all objects.



 

In my case a set a view more. After setting the rights for my user group: "Benutzer" which is an Active-Directory group every folder in my hierachy inherits the rights. If you log in to the WebService portal for example, every user in "Benutzer" can view, execute and inspect all folder.

When setting the rigths at the root object is done you have to edit the access rights for the folders you whish to hide. Similar to the steps at the root object you have to select "Edit access rights..." on the folder you want to hide. As you can see the folder has inherited its rights from the parent object (root). Now you have to set the rights, or better the restriction to the folder.

Restrictions in child objects are set by deselecting the rights (cruel sentence...). So deselect all rights and choose the same user group "Benutzer" as before.

After that you can verify the settings and press "Save and Close". Now do the same step for alle folders you want to hide.

In my example the "Customer2" folder is an child of "Customer". Regarding this my parent folder "Customer" needs all the rights set in the root object. If you change the rights here it will affect the child folders! Next we hide my "Customer1" folder because my users should only see workflows in "Customer2". You can do this exactly the same way as for the other folders.

  

As done before we "Edit access rights..." and deactivate all rights for the "Customer1" folder.

After that the child object has no rights and prevails to the parent object. On the "Customer2" folder you have nothing to change (if the parent rights in root are the right ones) because it is visible and the workflows can be executed and inspected.

Now you can logon at the WebViews portal with a user from the Active-Directory group you have enabled ("Benutzer").

In my case the user "Raketen RJ. Joe" can now create a simple virtual machine with his user rights in my vCenter Orchestrator WebViews :-) #

I hope this simple instruction helps you to design a rights management for you administration or user team.

ESXi - esxtop/resxtop and perfmon

Last week I had an appointment with one of our customer who wants to know more about esxtop/resxtop. So I decide to write some, in my eyes, important things to know. Because of the simple display esxtop wasn´t the tool for me to troubleshoot in the past. But with the enhancements in vSphere 4.1 there are some useful possibilities.

With this article iI want to show how to use esxtop in batch mode and how to evaluate the collected data in perfmon.

First we start with the ssh connection through the host (don´t forget to enable it on the security settings in the host configuration screen).



After that we start esxtop and create an new configuration file which only collects cpu metrics. You can do this easily by pressing "W" and name the new configuration file //.esxtop_cpu in my case.



After the configuration file is written you can close the esxtop screen by pressing "q". Now you can start the data collector with a simple command:

esxtop -b -c //.esxtop_cpu -n 10 > testcpu_10intervals.csv

The "-b" switch is for the batch mode, the "-c" switch is for the configuration file, the "-n" switch is for the intervals (be careful with more because of the disk usage) and the > pipes the data into the .csv file.



After the execution is finished there should be a .csv file with the collected data. This file we copy to a windows workstation and open the perfmon (in my case a Windows 7). Inside the performance monitor there is a button called "View Log Data" which allows you to import the .csv file.



After adding the .csv file you have to add the data (data tab)...



and the display options (Graph tab) in the performance monitor.



After the performance metrics (in my case: physical cpu (0,1)) are choosen the graph will be showed in the performance monitor. Now you are able to look the ESXi performance data in a historical graph.

vCO - XML post with pre-authentication

If you want to post XML data to webservices wich need pre-authentication you need a workaround, because vCO does not support this feature.

In my case, I had to post XML data to CA unicenter without activated SOAP interface, only authenticated post was available. So I wrote this little app XMLpost and call it from vCO.

Read more on original post here: http://communities.vmware.com/docs/DOC-14024

VMware VCAP-DXA exam experience

Today was the day! After travelling the long way to Frankfurt and reading every guide from VMware at the train i arrived at the VUE test center around 1:00 pm. After the normal check-in procedure i take my seat in front of an old DELL client. After 10 questions in the survey the exam begins...

After 3 hours and 30 minutes my eyes were dry and red and my head begans to burn. I think it was one of the hardest exams i had made in the last years. So i try to give you an overview what drives me mad:

1.building performance reports and statistics
I think nearly every question has to do with performance metrics, performance reports, utilization reports and so on. Also there are several questions to build custom reports and performance graphs like: build up an DRS Cluster, put the hosts into it and design a performance chart for all memory operations (average, used etc.). Sometimes the description doesn´t match the really available options...

2.vMA administration
Another strong part were the vMA questions. Several tricky things are asked: delay in the vMA input via vSphere Client console, esxupdate with NIC drivers (did not work in my test environment!), verifying an ks.cfg file (was not available on my hosts, should be available under /tmp on one of them!), esxcli to create an SATP ALUA rule (New Array does not work as name, because of the space between!)

3.PowerCLI
In my case there was only on script to build: Find all VMs with a CDROM attached and write the list into a file.

4.Standard operations
The most things i had to do where standard vSphere operations: build up a vDS with several port groups, change Uplink orders in vDS port groups, build up an vApp and a resource pool with explicit configuration tasks (start order, reservation under 25%, 5VMs start without expandable resources).

In my case there were no question about the vShield Zones, the vCenter Orchestrator or the Linked Mode, but i really missed them because the vMA questions were really hard! I think i will need a second attempt to get the certification :-(

UPDATE: Today the certification@vmware.com mail arrives and what should i say: i passed the exam!!!

vCO - dvPortGroup, uplink or not

I had to determine if a distributedVirtualPortGroup (dvPG) is an uplink or not. E.g. to filter uplinks from datacenter network list.

I found nothing suitable, so I wrote this action:




You can find the original post here.

HDS HNAS Platform 3x00 - undocumented NFS feature

If you are using a HNAS from BluARC(HDS) and a vSphere environment which supports Thin-Provisioning there is a possible problem: While deploying a virtual machine with thin-provioned disks (thin provisioning is displayed as supported for the datastore) the deployed virtual machine shows up thick virtual disks after creation.



If this happens, there is an undocumented feature for the NFS server parameters:

enable_sparse_block_counts

After the counts are enabled the documentation of the provisoned space into the vSphere environment is correct.

white paper - vCO and secure ldap

Based on the problems at one of our enterprise customers, i decide to releas a white paper which describes the possibilities to implement the vCenter Orchestrator in a secure ldap environment.



The download is available at our Mightycare website:
http://www.mightycare.de/system/files/download/MCS-Whitepaper_vCO_and_ldaps_v0.5.pdf

Please note that the described solution is a workaround, i prefer the more secure PKI based solution.

vCO - secure ldap configuration - UPDATE

Today i had a configuration problem with a larger vCO environment with several domains... After starting the ldap configuration an error code 8 occurs and i had to change to secure ldap (SSL). After this change the vCenter Orchestrator want a new CA certificate for authentification. No problem, if you had an CA and export the CA request (.csr). But in my case i thought they had a "normal" ldap on port 389 and i create an simple self-signed CA.



The problem is: After creating a self-signed Certificate there is no way to create a new request with the company defaults! Deleting all cert* files and restart the configuration server does not work at all.

So you had to install the vCO again... :-( and can not import a configuration, because of the self-signed certificate.

UPDATE #1:
After consulting the vCO installation guide i found out that the certificate is stored in the Database: vmo_keystore. So after cleaning it up you are able to create a new certficate.



With this new certificate, based on the customer data i export the .csr (request) file and send it to the central certificate instance.

UPDATE #2:
Several days after my last attempt my fellow Andreas try to implement the AD SSL/TLS certificates and identifies another problem: the customer doesn´t use certificates in his AD! So after a few mails the basic problem is the NTLMv2 authorization, which is not supported in the ldap configuration.

So we try to find a fix for that and will inform you guys shortly.

UPDATE #3:
Yes, we found the solution!!! After hours of attempts, thinking about it and several coffee cups a simple GPO (thank you Microsoft!) was the cause for all the trouble.



Because the Java engine uses simple bind the connection handshake with the domain controller fails. After turning of the ldap signing request option everything works fine. The second problem was the reverse DNS wich wasn´t right.

So, lessons learned and beer earned!

vCO - new plug-in released: VIX

VMware released a new vCenter Orchestrator plug-in this week: the vCO VIX plug-in. With this integration it is possible to start operations direct in the virtual machine, like file copy, starting a service or check for directories.



The plugin can be downloaded at VMware Labs: http://labs.vmware.com/flings/vix-vco
Within the .zip file are several components:

o11nplugin-vix.dar - The vCO VIX plug-in (binary)
vCO41-VIX-Plugin-Guide.pdf - The documentation guide
VMware-vix-x64-1.10.1-266898.zip - The Windows 64-bit VIX DLL
VSOSDK-vix-src.zip - The vCO VIX plug-in (project and source code)

The .dar file can be installed like every other plug-in in the configuration interface of the vCenter Orchestrator.

vCO - Input Dependency - using Presentation in VMware Orchestrator

A vCO Starter asked how to prefill DropDow-Boxes (or other fields) as input in dependency of an other input. Here a simple guide to use an action in Presentation to do this.

Assuming there is a workflow with two input parameters, Input1 [string] for free input, Input2 [string] in dependency of Input1 wich should offer following values:

• Input1 = A --> A1, A2, A3
• Input1 = B --> B1, B2, B3
• Input1 all other values --> C1, C2, C3

Define both strings as input parameter. Then go to the presentation tab.

The Mandatory Property forces an input for Parameter Input1. You can not proceed the Workflow without a value for Input1.

Before looking at Input2, create an action:

This action will return an array of string in dependency of the input parameter MySelection described at the beginning.

Now use this action to define the second parameter:

Add a "Predefined list of elements" property. 

Use action sign (the puzzle) to bind the action on this property. Click the "string" to select the propriate input parameter or our action. After this the presentation look like this.

Execute the workflow and test the behavior.

vCO - mount NFS Datastore with VMware Orchestrator

Here a little action to mount a NFS export to a Host (assuming all export settings are checked for ESX).

Input parameter

• Host [VcHostSystem] - Host to mount the NFS-Export
• NfsServer [string] - IP or Name of Server exporting the NFS
• NfsPath [string] - path to mount point from export, e.g. /mnt/export/NFS
• DatastoreName [string] - Datastore name on Host 

Output parameter

• Datastore [VcDatastore] - Datastore object of attached NFS-Store

vCO - setting DRS mode in VMware Orchestrator

To set DRS automation level is easy with vCO. But how to set the DRS mode (manual, partially automated and fully automated)?

The above example has Cluster [VcClusterComputeResource] and DRSmode [string] as input parameter. Output is Task as VcTask, so you can continue with action vim3WaitTaskEnd.

DRSmode has 3 possible values:

• manual
• partiallyAutomated
• fullyAutomated

vCO - functions in VMware Orchestrator

Sometimes you need a function to calculate a result and you don't want to use an action. Here we have an (sense free) example which shows us how a function is build up and used. Also the scope of used variables is demonstrated:

This example will log (in extra lines): 3, 4, 0, 7, 3, 4, 99

This show us the scope of defined variables. The variables a and b are defined in outer and inner (function) scope. The changed values (set to 99) will only affect the inner a and b. Otherwise c - there is no c defined inside the function or passed to it. So the outer c will be used.

After calling Add(a,b) the value of c is set to 99.

Regardless of the scope a function can return a value using the keyword return. There is no type definition. The function will return an object.

Warning - it is not recommended to manipulate outer variables from inside a function. It works, but making your code difficult to debug if any error or misbehavior occurs.

vCO - sorting arrays in VMware Orchestrator

The integrated sort function for array Array.sort() will sort the array ascending. This works good for scalar types like number. But how to sort descending (not using Array.reverse()) or sorting arrays containing complex types like virtual machines? Let's have a look.

The example above will log this:


The following example is sorting descending:


The example will log this:
The function "desc" returns a boolean to decide to sort or not. It works as comparator for the sorting algorithm inside sort(). I don't know this algorithm, but this is not necessary. Just use ist as sorting comparator. So if returned true the pairs will change place in array - like quicksort or so.

This offers us an option to sort non scalar types like virtual machine or other complex types.
Have a look at a workflow with VMs [Array/VirtualMachine] as input parameter - in this example my array contains 3 virtual machines:

The (partial) log shows this:


As you can see, the array is sorted ascending by the id of each VM. If you want the array sorted ascending by VM.name, try this (there are now 4 VMs in array):

And the log shows this:Be careful with your self created sort function. Use it only on homogen arrays. Because all elements must support the properties you use in your function. On heterogen arrays adapt your function to match all possible elements.

SQL 2008 Express Management

Somtimes the day begins as the last day ends... in this morning my fellow here at mightycare solutions tried to install the new vCenter Orchestrator for an enterprise customer. Because of the limited functionality (it is an proof-of-concept) he decides to use SQL 2008 Express with the SQL Express 2008 Management Studio. At frist we would install the SQL Server (like known from SQL deployments) and afterwards we decide to install the Management Studio.

After the installation of several features (.NET, .NET update, and so on) we install the SQL Server 2008 Express and start the installation of the Management Studio Express. But what is that? After choosing the "Add features to an existing instance..." we can not select the "Management Tools - Basic" option!



After a few moments of investigating we try to use the other, in my eyes senseless, option "Perform a new installation of SQL Server 2008" and what did my swollen eyes see? An option called "Management Tools - Basic"!

So i think this is another example of user experience at Microsoft...