Search This Blog

Monday, April 22, 2013

Murky Waters - When Xenserver pools go bad.

Our XenServer (version 6.0) pool went bad. The word 'bad' - while not precise is the best way to describe it.
The planned configuration was:
  • 3 servers, each with 3 nics - runing XenServer v6.0
  • 2 iSCSI shared storage servers
  • 2 of the NICS bonded providing the management interface and added to each guest (VM) system.
  • 1 if the NIC for a dedicated storage network.
  • A dedicated storage network (addresses 192.168.40.x)
We had removed a server from the pool to upgrade the hardware (specifically to swap the hard drive for a 60Gb SSD drive).

The suggested procedure is to remove the server from the pool, upgrade the hardware, reinstall XenServer and then add it back to the pool.
We then reinstalled exactly the same version of Xenserver. When it started, it could see the 3 network intefaces.

When we added it back to the pool the new server couldn't see all of the networks.
Either the primary interface wasn't working reliably after a reboot, or the storage interface was not available.
On other attempts the primary interface would connect ok, but the 3rd storage interface would not.
We tried various combinations of first bonding the first 2 NICs before adding it to the pool, or adding it without any configuration.

I tested this process in a test system mentioned below (with 2 servers and 1 storage array) and never had any problems removing and adding back a server to the pool.
This was annoying as nothing else was changed (particularly with the network interfaces), just a new hard disk and a fresh install of XenServer.

After much research the decision was made to recreate the XenServer pool.
To summarise this involves:
  • backing up the VM metadata.
  • though the VM data on the storage array should be unaffected, ensure it is all is backed up.
  • for each server - remove it from the pool, install XenServer
  • create a new pool, and
  • restore the storage repositories and VM metadata.
This post details this procedure.

1. Preparation.

The pool was running, but in a degraded state - with 2 of the 3 servers operational  As this is an 'elective' procedure we were able to carefully plan, and in particular the procedure was practised on a test system.
The test system had 2 servers, 1 storage array, each server having 2 NICs; supporting had 3 VMs (Centos systems each with 8Gb of storage).
It was set up much as per this post, there wasn't much difference between v6.0 and v6.1. 

1.1 Testing.

The testing went as expected with one problem.

When we attempted to restore the virtual machine metadata for all of the servers using the console (xsconsole) Restore Virtual Machine Metadata command, 1 of the 3 VMs would not restore.
In hindsight I believe this VM had a drive attached to local storage on one of the servers.

What I decided to do is to save the metadata for the VMs individually using cli vm-export command in addition to the xsconsole command that backs up the metadata to disk on the storage repository.
The vm export command:

xe vm-export filename=vm_testserv2 uuid=ac7fc085-aa66-826d-5a69-599c5af20118 metadata=true

This prove to be invaluable later.

As part of the testing we upgraded the test pool from v6.0 to v6.1 (skipping v6.0.2) - the same version of Xenserver as the live pool. I wanted to be sure that there wouldn't be any problem accessing the storage repositories and the VM metadata backups and restarting the virtual machines themselves.

1.2 Documentation.

We checked and documented the following:
  • VM storage configuration.
  • VM (in this case Centos) kernel levels, Centos versions older than 5.4 can have problems with XenServer versiosn 6.02 and 6.1
  • Ejected any mounted virtual machine iso images.
  • Storage repository configuration - ip addresses, names and connection details.
  • Storage repository configuration - as screenshots from XenCenter, if under stress later a pictorial representation later can be helpful. We had 7 virtual machines, so taking the screenshots was manageable.
  • Saved lists from xen command line utilities, this included: xe sr-list; xe vm-list
  • Backup (dump) the pool ( pool-dump-database)  a metadata backup
  • Run a server status report and upload it to https://taas.citrix.com/ - Citrix Autosupport
Example Server disk screenshot.












One thing we didn't record specifically was ip addresses for the VMs, during the recovery process we did need this information. We retrieved it from other records.

2. The Process.

The following steps were done:
  1. Check no disks are on any of the local storage arrays. Sometimes templates - if created when the servers were first built can remain on local storage.
  2. Recheck and eject any mounted VM CD drives (as mentioned above).
  3. Do another backup of the virtual machines to a storage repository, also backup each vm individually:
    xe vm-export filename=vm_custweb uuid=ac7fc085-aa66-826d-5a69-599c5af20118 metadata=true
  4. Remove one server from the pool (or in our case the newly installed server that we couldn't rejoin to the pool).
  5. Install Xenserver 6.1, give it the same networking details (ip address, gateway, dns settings).
  6. On the old servers in the running pool, stop all virtual machines and detach - disconnect) from all of the shared storage repositories. For now we left them running.
  7. On the new server configure the networking (note however I didn't create the bonded network interface at this point).
  8. On the new server, connect to the storage repositories.
  9. Restore the virtual machines - (xs console, see screenshot below)
  10. Start one of the VMs.
  11. Then shutdown the old Xenservers, install Xenserver 6.1 and add them to the new pool.
VM Metadata restore






3. Problems.

VM Metadata Restoration.

Of our 7 virtual machines, one would not restore. It was known to not have any local storage. Apart from the fact it was one of the earlier (oldest) systems it didn't have any differences from the other VMs.
In this case it was able to be easily restore from the vm metadata export.

xe vm-import filename=vm_custweb metadata=true

This restore was able to work without any problem.

VM Networking.

One mistake we did do was to not fully reconfigure the networking before doing the VM metadata restore.
When we restored the VMs, they were not attached to a network because I hadn't recreated the bonded interface they were expecting.

I tried the configuration as it was with a test server - when it started the Centos system didn't have any network interfaces, later even after I had recreated the bonded interface, the other systems also didn't have any network interfaces. If I had configured it before doing the metadata restore I believe the restore would have been complete.

Our test system only had 2 NICs - the primary interface for the VMs was not bonded. If it had we would have picked up this problem in testing.

What I had to do - after configuring the networking, is to check each virtual machine had a network interface, and when the guest OS had started, reconfigure the networking - configure the IP address/netmask and default gateway. The dns configuration - not being tied to a particular interface was still in place.

This was manageable for 7 VMs of which only 1 had more than one interface. For a larger number I would have dropped them and re-restored them. Reconfiguring a larger number of VMs manually would have been error prone.


Network Configuration

Unreable Guest Server Attached Disk.

This was another error that wasn't anticipated and doesn't have a known cause.

One of the VMs when starting was unable to mount a partition. An fsck run from the single user command prompt also failed.

We edited the /etc/fstab file and the system started without the drive.
[ To edit the fstab in this situation, you need to remount the root partition re-write, i.e. from single user mode
 # mount -o remount,rw /dev/sda1 /  ]

We did have a copy of the data but I investigated first.
The virtual machine restore of the metadata did include the drive however a listing from fdisk showed:

[root@webtest ~]# fdisk -l
.... 
Disk /dev/xvdc: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/xvdc doesn't contain a valid partition table

I first tried the rescue command in parted:
parted /dev/xvdc

rescue
       Using /dev/xvdc
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p

Model: Xen Virtual Block Device (xvd)
Disk /dev/xvdc: 42.9GB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number  Start   End     Size    File system  Flags
 1      0.00kB  42.9GB  42.9GB  ext3

This returned the expected partition - a single ext2/ext3 partition for the entire disk. However a fsck still failed.
Once again there was no obvious reason why this partition should have problems. It hadn't been resized or any other changes. One possbility is that it may have been moved from another storage repository - as opposed to being created and formatting on its current repository.

I then attempted to recover the partition using gpart, which did work.
The full trace of the restore session is below, I had first run it in read mode before specifying write mode (-W <device>)

[root@webtest ~]# gpart -W /dev/xvdc /dev/xvdc

Begin scan...
Possible partition(Linux ext2), size(40954mb), offset(0mb)
End scan.

Checking partitions...
Partition(Linux ext2 filesystem): primary
Ok.

Guessed primary partition table:
Primary partition(1)
   type: 131(0x83)(Linux ext2 filesystem)
   size: 40954mb #s(83875296) s(63-83875358)
   chs:  (0/1/1)-(1023/254/63)d (0/1/1)-(5220/254/57)r

Primary partition(2)
   type: 000(0x00)(unused)
   size: 0mb #s(0) s(0-0)
   chs:  (0/0/0)-(0/0/0)d (0/0/0)-(0/0/0)r

Primary partition(3)
   type: 000(0x00)(unused)
   size: 0mb #s(0) s(0-0)
   chs:  (0/0/0)-(0/0/0)d (0/0/0)-(0/0/0)r

Primary partition(4)
   type: 000(0x00)(unused)
   size: 0mb #s(0) s(0-0)
   chs:  (0/0/0)-(0/0/0)d (0/0/0)-(0/0/0)r

Edit this table (y,n) : y

Edit which partition (1..4, q to quit) : 1

Warning: entered values will not be checked; enter at your own risk!
1 - Absolute start sector (          63)
2 - Absolute sector count (    83875296)
3 - Partition type        (         131)(Linux ext2 filesystem)

Edit which value (1..3, q to quit) : 1
Enter value for 1 : 63

Warning: entered values will not be checked; enter at your own risk!
1 - Absolute start sector (          63)
2 - Absolute sector count (    83875296)
3 - Partition type        (         131)(Linux ext2 filesystem)

Edit which value (1..3, q to quit) : 3
Enter value for 3 : 131

Warning: entered values will not be checked; enter at your own risk!
1 - Absolute start sector (          63)
2 - Absolute sector count (    83875296)
3 - Partition type        (         131)(Linux ext2 filesystem)

Edit which value (1..3, q to quit) : q
Primary partition(1)
   type: 131(0x83)(Linux ext2 filesystem)
   size: 40954mb #s(83875296) s(63-83875358)
   chs:  (0/1/1)-(1023/254/63)d (0/1/1)-(5220/254/57)r

Primary partition(2)
   type: 000(0x00)(unused)
   size: 0mb #s(0) s(0-0)
   chs:  (0/0/0)-(0/0/0)d (0/0/0)-(0/0/0)r

Primary partition(3)
   type: 000(0x00)(unused)
   size: 0mb #s(0) s(0-0)
   chs:  (0/0/0)-(0/0/0)d (0/0/0)-(0/0/0)r

Primary partition(4)
   type: 000(0x00)(unused)
   size: 0mb #s(0) s(0-0)
   chs:  (0/0/0)-(0/0/0)d (0/0/0)-(0/0/0)r

Edit which partition (1..4, q to quit) : q
Activate which partition (1..4, q to quit) : 1
Write this partition table (y,n) : y


After this I was able to check and mount the partition:
[root@webtest ~]# fsck.ext3 /dev/xvdc1
e2fsck 1.39 (29-May-2006)
/dev/xvdc1: clean, 1394/5242880 files, 8291901/10484412 blocks


While this process worked ok there is no substitution for having a complete backup of the system.

Conclusion.

To conclude the process went well and the pool is now in an improved state -with all 3 servers in it.
The exercise was a valuable one in terms of practising a disaster recovery scenario and improving our knowledge and confidence with Xenserver.

I would recommend the procedure - done on a test system -to anyone responsible for managing a virtual server pool.


Other References.

This post Reinstalling Xenserver 6.0/5.6 and Preserving Virtual Machines on the Local Disk, is interesting as it documents a procedure to keep the local storage repositories intact. We didn't follow this procedure as we were not using the local disks, however the material in it is very useful.
The documents:

Another XenServer - iscsi gotcha - problems connecting to an QNAP iSCSI target

Xenserver can decide (ie some how be set) to use Chap authentication on discovery
Chris Keim document this at 1bitatatime.blogspot.com.au.
I had this problem with a new Qnap 870 NAS I could connect using a NFS share, but not using iSCSI.

I wanted to use iSCSI as detailed in this post - as a particular disk with a lot of small files would generate errors and hang the Xenserver.

To diagnose
1) connect from another iSCSI initiator (e.g. Windows). Note you can connect, discover targets and connect to a LUN.
2) attempt from Xenserver - both XenCenter and the command line.
3) attempt to connect using iscsadm


[root@xennet2 ~]# iscsiadm  --mode discovery --type sendtargets --portal 192.168.40.2
iscsiadm: Login failed to authenticate with target
iscsiadm: discovery login to 192.168.40.2 rejected: initiator error (02/01), n                                         on-retryable, giving up
iscsiadm: Could not perform SendTargets discovery.

The clue is the error - "failed to authenticate with target". The Qnap NAS doesn't like the authentication passed during the discovery.

To fix - as documented in  Chris' blog, edit the st_config file - in /etc/iscsi/send_targets/192.168.40.2,3260
remove the lines
discovery.sendtargets.auth.username = <user>
discovery.sendtargets.auth.username = <password>

and add:
discovery.sendtargets.authmethod = none

To be persistent (after reboots) make sure /etc/iscsid.conf also has auth.method set to none for discovery.sendtargets (or is commented out).

Saturday, April 20, 2013

Symfony Project SfServermon - Post 8 Notification

Having got the server status in the previous post, we now handle any notifications resulting.


The notify() method in UpdaterStatus is called from ServerUpdate.
It can use the Entity Manager and configuration references created when UpdaterStatus was initalised.
The notification in phpServerMon and sfServermon support:
  • Email.
  • Logging.
  • SMS - implemented as a seperate bundle.
These are enabled (and checked for here, in the application configuration).
Also each server can be specified as supporting any of the 3 methods to notify:
  • Always - everytime the status is checked - as a 'heartbeat'.
  • If server is offline.
  • If server status has changed.
These are checked for first:
public function notify() {
if ($this->config['email_status'] == false && ! $this->config['config_status'] == false && $this->config['log_status'] == false) {
//nothing to log;
return;
}
$notify = false;
$logWarning = false;

switch($this->config['alert_type']) {
case 'always':
if($this->status_new == 'off') {
// server is offline. we are in error state.
$notify = true;
}
break;
case 'offline':
// only send a notification if the server goes down for the first time!
if($this->status_new == 'off' && $this->status_org == 'on') {
$notify = true;
}
break;
case 'status':
if($this->status_new != $this->status_org) {
// status has been changed!
$notify = true;
}
break;
}

if(!$notify && !$logWarning) {
return false;
}

Log Notification:

Log notification is simply created a new log entity and saving it to the database:
if($this->config['log_status']) {
$log = new MonitorLog();
$log->setServer($this->server);
$log->setType('status');
$log->setMessage('status');
$log->setDateTime(new \DateTime());
$log->setuserId(0);
$log->setMessage($this->server->parseMessage($this->status_new, 'sms'
, array('label'=>$this->server->getLabel()
, 'ip'=>$this->server->getIp()
, 'port'=>$this->server->getPort()
, 'error'=>$this->server->getError())
));
$this->em->persist($log);
$this->em->flush();

phpServerMon had the parseMessage in a utility class, however I have moved it to the server entity, as I thought it made sense for an entity to know how to generate a status message for itself.
The parseMessage method:
public function parseMessage($status, $type, $vars)
{
    $notifications = array(
      'off_sms' => 'Server \'%LABEL%\' is DOWN: ip=%IP%, port=%PORT%. Error=%ERROR%',
      'off_email_subject' => 'IMPORTANT: Server \'%LABEL%\' is DOWN',
      'off_email_body' => "Failed to connect to the following server:<br/><br/>Server: %LABEL%<br/>IP: %IP%<br/>Port: %PORT%<br/>Error: %ERROR%<br/>Date: %DATE%",
      'on_sms' => 'Server \'%LABEL%\' is RUNNING: ip=%IP%, port=%PORT%',
      'on_email_subject' => 'IMPORTANT: Server \'%LABEL%\' is RUNNING',
      'on_email_body' => "Server '%LABEL%' is running again:<br/><br/>Server: %LABEL%<br/>IP: %IP%<br/>Port: %PORT%<br/>Date: %DATE%",
      'warn_sms' => 'Server \'%LABEL%\' needs ATTENTION: ip=%IP%, port=%PORT%. Error=%ERROR%',
      'warn_email_subject' => 'IMPORTANT: Server \'%LABEL%\' need ATTENTION',
      'warn_email_body' => "check server load or disk usage:<br/><br/>Server: %LABEL%<br/>IP: %IP%<br/>Port: %PORT%<br/>Error: %ERROR%<br/>Date: %DATE%",
    );


    $message = '';
  
    //$message = sm_get_lang('notifications', $status . '_' . $type);
    $message=$notifications[$status . '_' . $type];

    if(!$message) {
      return $message;
    }
    $vars['date'] = date('Y-m-d H:i:s');
  
    foreach($vars as $k => $v) {
      $message = str_replace('%' . strtoupper($k) . '%', $v, $message);
    }
  
    return $this->message = $message;
  }

If course the vars associative array should be generated within the class, so the method becomes:


public function parseMessage($status, $type)
{
$vars = array('label'=>$this->getLabel()
, 'ip'=>$this->getIp()
, 'port'=>$this->getPort()
, 'error'=>$this->getError());

$message = '';

//$message = sm_get_lang('notifications', $status . '_' . $type);
$message=$this->notifications[$status . '_' . $type];

if(!$message) {
return $message;
}

$vars = array('label'=>$this->getLabel()
, 'ip'=>$this->getIp()
, 'port'=>$this->getPort()
, 'error'=>$this->getError());

$vars['date'] = date('Y-m-d H:i:s');

foreach($vars as $k => $v) {
$message = str_replace('%' . strtoupper($k) . '%', $v, $message);
}

return $this->message = $message;
}

Note - this is an initial version of the parseMessage method will be recoded as part of a future post to add internationalisation support.

Email Notification.

For email notification we will use Swiftmailerbundle - available as a common addition to Symfony installations.

The code for this:
 protected function notifyByEmail() {

      $message = \Swift_Message::newInstance();
   $message->setSubject($this->server->parseMessage($this->status_new, 'email_subject'))
      ->setFrom($this->config['email_from_email'])
      ->setTo('johnr@reidyint.com')
      ->setBody($this->server->parseMessage($this->status_new, 'email_body'));
       
   $this->mailer->send($message);
}


This version has a hard coded recipient email address, the users to notify are specified in the users table - which is document in a future post in this series.
This needs a reference to the mailer. One was added to the service configuration for ServerUpdate:

services:
  server_update:
    class: JMPR\ServerMonBundle\Update\ServerUpdate
    arguments: [ @doctrine.orm.entity_manager, @mailer ]

And the constructor was updated:
class ServerUpdate
{
protected $em;
protected $mailer;

 public function __construct($em, $mailer)
   {
       $this->em = $em;
       $this->mailer = $mailer;
   }

A reference is then passed to the constructor for the UpdaterStatus class.


SMS Notification - SMS Messages.

The third type of notification supported are SMS text messages. This functionality is implemented as a seperate bundle and documented in the next post.

Friday, April 12, 2013

Reading Exported SugarCRM Modules

I recently had the need to re-create exported SugarCRM modules.
I had exported the modules from a Sugar Online instance with the plan to import them into a hosted SugarCRM instance.
However more time than I expected had passed and version details changed, so the modules would not import.
The modules were not heavily customised, so I decided to look at the exported module files and re-create them.
I wrote a php cli script that give the path to an exported extract module, will list the fields:

John-Reidys-MacBook-2:$ php dumpfields.php Jobs2012_12_19_190139
Modules found:
JOB_Jobs
read module JOB_Jobs: 
module: JOB_Jobs
fields count: 13
    date client_advised_date                    
checkbox completed                             1
    date completed_date                         
 varchar estimated_time                 255
 varchar estimated_time_code             20
currency invoice_amount                  26
currency currency_id                     36
    date invoice_date                           
dropdown job_status                     
[complete->"complete" domain_not_ready->"domain not ready" in_production->"in production" incomplete->"incomplete" on_hold->"on hold" ready->"ready" ]
    date loaded_to_search_date                  
    date loaded_to_server_date                  
    date start_date                             
     int job_number                             


The code does the following:
Checks the arguments passed:

if ($argc == 1 ) {
echo "Usage: {$argv[0]} <module-path> <module-name>\n";
exit(1);
}
$modulePath=$argv[1];
if (!file_exists($modulePath) || is_file($modulePath) ) {
echo "{$argv[0]}: bad module-path {$modulePath}\n";
exit(1);
}

$basedir = $modulePath.'/SugarModules/';
$moduleDir=$basedir.'/modules/';

if (!file_exists($basedir) || is_file($basedir) 
     || !file_exists($moduleDir) || is_file($moduleDir) ) {
echo "{$argv[0]}: bad structure cannot find SugarModules or modules sub directories  module-path: {$modulePath}\n";
exit(1);
}

This is an example of a cli php script - which can be very useful for php programers who need to do scripting.

Argument Handling.

1 or 2 arguments are expected - the path to the module package and the module itself.
The module path is checked and then the expected sub folders in the module.
Then if the module name is not specified, they are listed and the user asked if they want to read the first module:

$moduleName='';
if ($argc == 2) {
  echo "Modules found:\n";
  $handle=opendir($moduleDir);
while (false !== ($entry = readdir($handle))) {
   if (!is_file($entry) && $entry != '.' && $entry != '..') {
    echo "$entry\n";
    if (! $moduleName) $moduleName = $entry;
   }
}
if ($moduleName) {
echo "read module {$moduleName}: ";
 $handle = fopen ("php://stdin","r");
 $line = fgets($handle);

 if(trim($line) != 'yes' && trim($line) != 'y' && trim($line) !=''){
    echo "ABORTING!\n";
    exit(0);
 }

} else {
  exit(1);
}
}
Entering 'y', 'yes' or enter will continue.

Read the Vardefs

Next the variable definitions for the module itself are read.

require $basedir.'modules/'.$moduleName.'/vardefs.php';
require $basedir.'modules/'.$moduleName.'/language/en_us.lang.php';
require $basedir.'language/application/en_us.lang.php';

class VardefManager{
    static $custom_disabled_modules = array();
    static $linkFields;

    /**
     * this method is called within a vardefs.php file which extends from a SugarObject.
     * It is meant to load the vardefs from the SugarObject.
     */
    static function createVardef($module, $object, $templates = array('default'), $object_name = false)
    {
}
}




The included class and static method - createVarDef is invoked by the vardefs.php, this currently does nothing, however the following code could be moved inside it.

Print the Fields

The fields and some of the details are then printed:


$fields = $dictionary[$moduleName]['fields'];
printf("module: %s\n", $moduleName);
$maxfieldlen=0;
foreach($fields as $fieldname => $fielddef) $maxfieldlen = strlen($fieldname) > $maxfieldlen ? strlen($fieldname) : $maxfieldlen;
echo "fields count: ". count($fields)."\n";
$fieldNameFmt="-{$maxfieldlen}.{$maxfieldlen}";
foreach($fields as $fieldname => $fielddef) {
switch($fielddef['type']) {
case 'date':
case 'datetime':
case 'bool':
case 'int':
  printf("%8.8s %{$fieldNameFmt}s %8.8s %8.8s", $fielddef['type']=='bool'?'checkbox':$fielddef['type'], $fieldname, $fielddef['required'], isset($fielddef['default'])?isset($fielddef['default']) : '');
break;
case 'enum':
  printf("%8.8s %{$fieldNameFmt}s %8.8s ", 'dropdown', $fieldname, $fielddef['required']);
  $enumList = $app_list_strings[$fielddef['options']];
$maxEnumValuelen=$maxEnumLablelen=0;
foreach($enumList as $enumValue => $enumLabel) {
$maxEnumValuelen = strlen($enumValue) > $maxEnumValuelen ? strlen($enumValue) : $maxEnumValuelen;
$maxEnumLablelen = strlen($enumLabel) > $maxEnumLablelen ? strlen($enumLabel) : $maxEnumLablelen;
  }
  printf("\n\t[");
$enumValueFmt=""; //"{$maxEnumValuelen}.{$maxEnumValuelen}";
$enumLableFmt="-";//"-{$maxEnumLablelen}.{$maxEnumLablelen}";
  foreach($enumList as $enumValue => $enumLabel) {
    $enumLabelpp = "\"${enumLabel}\"";
printf("%{$enumValueFmt}s->%{$enumLableFmt}s ", $enumValue, $enumLabelpp);
}
  printf("]");
  
break;
default:
  printf("%8.8s %{$fieldNameFmt}s %8.8s %3.3d", $fielddef['type'], $fieldname, $fielddef['required'],isset($fielddef['len'])?$fielddef['len']:$fielddef['size']);
break;
}

echo "\n";
}


This could be extended and made smarter to print more information, however for my purposes it was enough.

The complete script.



<?php

if ($argc == 1 ) {
echo "Usage: {$argv[0]} <module-path> <module-name>\n";
exit(1);
}
$modulePath=$argv[1];
if (!file_exists($modulePath) || is_file($modulePath) ) {
echo "{$argv[0]}: bad module-path {$modulePath}\n";
exit(1);
}

$basedir = $modulePath.'/SugarModules/';
$moduleDir=$basedir.'/modules/';

if (!file_exists($basedir) || is_file($basedir) 
     || !file_exists($moduleDir) || is_file($moduleDir) ) {
echo "{$argv[0]}: bad structure cannot find SugarModules or modules sub directories  module-path: {$modulePath}\n";
exit(1);
}

$moduleName='';
if ($argc == 2) {
  echo "Modules found:\n";
  $handle=opendir($moduleDir);
while (false !== ($entry = readdir($handle))) {
   if (!is_file($entry) && $entry != '.' && $entry != '..') {
    echo "$entry\n";
    if (! $moduleName) $moduleName = $entry;
   }
}
if ($moduleName) {
echo "read module {$moduleName}: ";
 $handle = fopen ("php://stdin","r");
 $line = fgets($handle);

 if(trim($line) != 'yes' && trim($line) != 'y' && trim($line) !=''){
    echo "ABORTING!\n";
    exit(0);
 }

} else {
  exit(1);
}
}


require $basedir.'modules/'.$moduleName.'/vardefs.php';
require $basedir.'modules/'.$moduleName.'/language/en_us.lang.php';
require $basedir.'language/application/en_us.lang.php';

class VardefManager{
    static $custom_disabled_modules = array();
    static $linkFields;

    /**
     * this method is called within a vardefs.php file which extends from a SugarObject.
     * It is meant to load the vardefs from the SugarObject.
     */
    static function createVardef($module, $object, $templates = array('default'), $object_name = false)
    {
}
}



$fields = $dictionary[$moduleName]['fields'];
printf("module: %s\n", $moduleName);
$maxfieldlen=0;
foreach($fields as $fieldname => $fielddef) $maxfieldlen = strlen($fieldname) > $maxfieldlen ? strlen($fieldname) : $maxfieldlen;
echo "fields count: ". count($fields)."\n";
$fieldNameFmt="-{$maxfieldlen}.{$maxfieldlen}";
foreach($fields as $fieldname => $fielddef) {
switch($fielddef['type']) {
case 'date':
case 'datetime':
case 'bool':
case 'int':
  printf("%8.8s %{$fieldNameFmt}s %8.8s %8.8s", $fielddef['type']=='bool'?'checkbox':$fielddef['type'], $fieldname, $fielddef['required'], isset($fielddef['default'])?isset($fielddef['default']) : '');
break;
case 'enum':
  printf("%8.8s %{$fieldNameFmt}s %8.8s ", 'dropdown', $fieldname, $fielddef['required']);
  $enumList = $app_list_strings[$fielddef['options']];
$maxEnumValuelen=$maxEnumLablelen=0;
foreach($enumList as $enumValue => $enumLabel) {
$maxEnumValuelen = strlen($enumValue) > $maxEnumValuelen ? strlen($enumValue) : $maxEnumValuelen;
$maxEnumLablelen = strlen($enumLabel) > $maxEnumLablelen ? strlen($enumLabel) : $maxEnumLablelen;
  }
  printf("\n\t[");
$enumValueFmt=""; //"{$maxEnumValuelen}.{$maxEnumValuelen}";
$enumLableFmt="-";//"-{$maxEnumLablelen}.{$maxEnumLablelen}";
  foreach($enumList as $enumValue => $enumLabel) {
    $enumLabelpp = "\"${enumLabel}\"";
printf("%{$enumValueFmt}s->%{$enumLableFmt}s ", $enumValue, $enumLabelpp);
}
  printf("]");
  
break;
default:
  printf("%8.8s %{$fieldNameFmt}s %8.8s %3.3d", $fielddef['type'], $fieldname, $fielddef['required'],isset($fielddef['len'])?$fielddef['len']:$fielddef['size']);
break;
}

echo "\n";
}
?>