Skip to content

CategorySystems

Executable Extension

Want to run a script in the command console that is not of type EXE, COM, BAT by just typing in the name of the script? Have a script in Perl or Python that you have that you want to run without typing the name of the interpreter with switches? It’s pretty easy to do in Windows.

  1. Open up System Properties and click on Environment Variables:

  2. Select the PATHEXT System variable and click on the Edit button.

  3. Add the extension you want. In this case, let’s add the extension for .PL so we can run Perl scripts without typing in the name of the Perl interpreter:

  4. Hit OK on the Windows.
  5. In an Explorer Window, Go to Tools -> Folders

  6. In the Folder Options Window, click on the File Types tab, and then click New:

  7. Type .PL for the File Extension

  8. Click OK then Click on the Advanced button to open up the Edit File Type and click on New…

  9. Enter the name of the interpreter:

  10. Hit OK for all Windows.
  11. You’re set and good to go.

Now let’s test it. Make a .pl file under c:\ called "add.pl". Make add.pl be:

$sum = $ARGV[0] + $ARGV[1];
print "\nSum of the numbers is: $sum\n\n";

 

Now let’s call it via the console by typing:

add 5 5

9

ANT Script to Deploy

A friend of mine asked me to post a simple script to show the power of ANT. Here it is with comments that show some of the things you can do with it. To read how to set up ANT and run scripts, please read my earlier post on doing so.

This script gets the latest from an SVN repo, deploys using FTP, then sends an email using a Gmail account. Here’s the ant script (deploy.ant), followed by a properties file (i.e. the config file, deploy.properties), which the ANT script pulls from:

<?xml version="1.0"?>
<!-- ======================================================== -->
<!-- Sample build file used to retrieve svnant's sources -->
<!-- ======================================================== -->

<!-- TODO: Mail if the script failed --> 
  
<project name="STAGING-Deploy" basedir="." default="empty">
  
  <property file="STAGING-Deploy.properties" />            
  <!-- Import SVNAnt tasks from jar files.  -->
  <typedef resource="org/tigris/subversion/svnant/svnantlib.xml" />
  
  <target name="Deploy"> 
    <!-- Step 1: Export the latest files from the repo to local machine. -->
    <antcall target="export" />
    
    <!-- Step 2: FTP the files from the local machine to STAGING. -->
    <antcall target="ftp_files" />
    
    <!-- Step 3: Cleanup. -->
    <antcall target="cleanup" />  
  </target>  
    
  
  <!-- SVN Checkout (Latest) -->
  <target name="checkout_latest">
    <svn username="${svnant.repository.user}" password="${svnant.repository.passwd}">
      <checkout url="${svnant.projectrunk.url}" revision="${svnant.checkout.revision}" destPath="${svnant.checkout.destpath}" />
    </svn>    
  </target>   
  
  <!-- SVN Update -->
  <target name="update_latest">
    <svn username="${svnant.repository.user}" password="${svnant.repository.passwd}">
      <update dir="${svnant.update.directory}" recurse="${svnant.update.recurse}" revision="${svnant.update.revision}" />
    </svn>    
  </target>  
  
  <!-- SVN Export  -->
  <target name="export">
    <svn username="${svnant.repository.user}" password="${svnant.repository.passwd}">
      <export srcUrl="${svnant.export.srcurl}" destPath="${svnant.export.destpath}" force="${svnant.export.force}" />
    </svn>    
  </target>
  
  <!-- FTP the export of the latest revision to STAGING. -->  
  <target name="ftp_files">    
    <!-- Import FTP tasks from jar files.  -->
    <taskdef classname="org.apache.tools.ant.taskdefs.optional.net.FTP" name="ftp" />
    
    <ftp server="${ftp.server}" port="${ftp.port}" remotedir="${ftp.destination}" userid="${ftp.userid}" password="${ftp.password}" 
         passive="${ftp.passive}" depends="${ftp.depends}" binary="${ftp.binary}" retriesAllowed="${ftp.retriesallowed}" verbose="${ftp.verbose}">
      
      <fileset dir="${svnant.export.destpath}" />
      
    </ftp>    
  </target>  
  
  <!-- Cleans Up: Remove temp info, etc. -->
  <target name="cleanup">  
    <delete includeEmptyDirs="true">
      <fileset dir="${svnant.export.destpath}"/>      
    </delete>
    
    <!-- Email Notification -->
    <mail mailhost="${mail.host}" mailport="${mail.port}" subject="${mail.subjectalert}" ssl="${mail.ssl}" user="${mail.username}" password="${mail.password}">
      <from address="${mail.from}"/>      
      <to address="${mail.to}"/>
      <message>The nightly build has completed.</message>
    </mail>
    
    <echo>Cleanup complete. </echo>
  </target>
  
  <target name="empty">    
    <echo>Pass in a default target to call </echo>    
  </target>

</project>

Now let’s check out the properties file (deploy.properties):

# -----------------------------------------------------------------------------
# STAGING-DEPLOY
# Variables are initialized here.
# -----------------------------------------------------------------------------
#
# NOTE: Backslashes have to be escaped with another backslash (on Windows)
#

# === INIT =====================================================================
svnant.projectrunk.url=svn://192.168.3.142/TESTING/trunk
svnant.repository.user=YOURUSERNAME
svnant.repository.passwd=YOURPASSWORD

# === CHECKOUT (INITIAL) =======================================================
# Check out a working copy of the latest to a local directory.
svnant.checkout.destpath=C:\\testing123
svnant.checkout.revision=HEAD

# === UPDATE TO LATEST =========================================================
# Update the working copy of your local directory to latest revision
svnant.update.directory=C:\\Experiments\\Firehawk\\src_latest
svnant.update.revision=HEAD
svnant.update.recurse=true


# === EXPORT ===================================================================
# Copies From Source Repo:
svnant.export.srcurl=${svnant.projectrunk.url}

# Into a temporary directory in your ant home (that will be FTP'd):
svnant.export.destpath=${ant.home}\\export-temp

# Overwrite (needs to in case the file changed)? 
svnant.export.force=true

PBNStagingFTP
# == FTP INFO ==================================================================
# FTP what was exported (latest version) to this location on staging:
ftp.destination=/testing/
ftp.server=TESTING
ftp.port=21
ftp.userid=USERNAME
ftp.password=PASSWORD
ftp.passive=yes
ftp.depends=yes
ftp.binary=yes
ftp.retriesallowed=5
ftp.verbose=yes

# == MAIL INFO ==================================================================
# Currently using GMAIL for now.
mail.host=smtp.gmail.com
mail.port=465
mail.subjectalert=[STAGING ANT Build] Test build
mail.ssl=yes
mail.username=USERNAME
mail.password=PASSWORD
mail.from=someone@somewhere.com
mail.to=someone@somewhere.com

Unlock System Files

Ever had such an annoying message given by Windows?

It has many other flavors:

Cannot delete file: Access is denied
There has been a sharing violation.
The source or destination file may be in use.
The file is in use by another program or user.
Make sure the disk is not full or write-protected and that the file is not currently in use.

Now you have an option to unlock it using Unlocker. Now, sometimes it’s theoretically impossible to unlock all files, especially some DLLs that are burned into Windows, but for most applications that you run that are not part of Windows, you either have the choice of unlock, rename, and move. This program is free too and it tries to do it without actually killing processes.

Here are some reviews for Unlocker.

Automate FTP Uploads

I thought this might be useful for those who are constantly uploading the same files. Whether you need to deploy a project live or constantly upload the same group of files, you can do this by creating a batch file in Windows using ftp.exe.

First of all, if you’re not familiar with FTP.exe, it’s just a command console way to upload/download files from an FTP server. You can run this program and start typing away…. OR, you can load a text file filled with FTP commands in FTP.exe. For example, the following are commands in a file called upload.bs:

open western.dreamhost.com
username
password
binary
hash
send index.htm
send frame.htm
send car.jpg
bye

The above should be straightforward. The “open” command is used to connect to the FTP server. The “username” should be your usename and “password” should be your actual password. “binary” is how your data should be handled on transfer – if it’s not text, it’s binary data, and should be “binary.” “hash” makes the console display pound signs (#) to serve as a progress bar as it’s uploading to the server. The “send” commands are used to upload the files. “bye” is exits the FTP session.

That was just a FTP script session file. You now need to call that file from FTP.exe on WindowsXP. You do that as follows:

@%windir%\system32\ftp.exe -s:upload.bs

You can include that call in a file called “upload.bat” – To learn more about FTP scripting and how to merge the two files into just one batch file, you can check out this site. Also, there’s more information about this found here: tinyapps.org.

Outlook Emails/Contacts in “To” Dropdown

Needed to backup my contacts in Outlook, but it would not save the emails in the email “To” dropdown field, when autocomplete triggers. This is because it’s stored in Outlook’s NK2 cache file. This file is not in readable form, but you if you have the patience, you can view the emails if you don’t mind looking through funky ASCII characters of binary data. In the process of looking for and trying various tools, I realized that not all tools were able to extract all contacts. Keep this in mind.

There’s a few tools out there that you can use to extract a list of emails saved in the NK2 file. Here they are.

Ingressor – is the best program, I was able to extract the most amount of emails/contacts, but you have to pay for it. I think it extracts everything, but you can’t save anything or copy anything. Use OCR software?

Debunk2– It’s free, but it missed 1 (could possibly miss more on different NK2 files) contact.

NK2.info– It’s free, but it missed 2 contacts, and it doesn’t show the names accurately, like Debung2 and Ingressor.

Run Program, Resize it, Relocate it

My work often changes. I could be doing programming, then systems administration, then quality assurance. I often have to change from different toolset to toolset and I find it annoying sometimes having to rerun programs and moving them to the same locations. Then resizing. Eech. This is especially frustrating when I’m doing testing and I have to debug and read requirements all at the same time. One handy thing to do is to use AutoHotkey, the great scripting tool.

Here’s a handy script I put together that opens up two different programs, moves them on the screen and resizes the Window. Just copy and paste the same block of code for each program to open as may apps as you want with whatever on-screen configuration you want.

; Run Note.exe
; WinMove, WindowTitle, WindowText, X-Axis, Y-Axis, [, Width, Height]
Run, note.exe
WinWait, Untitled - Notepad2
WinGetPos,,, Width, Height, Untitled - Notepad2
WinMove, Untitled - Notepad2,, 500, 500, 200, 200

; Run your browser with the following URL
Run, www.google.com/search?source=ig&hl=en&rlz=1G1GGLQ_ENUS313&=&q=tigers&aq=f&oq=&aqi=g5g-z1g4
WinWait, tigers - Google Search - Mozilla Firefox
WinGetPos,,, Width, Height, tigers - Google Search - Mozilla Firefox
WinMove, tigers - Google Search - Mozilla Firefox,, 900, 200, 300, 400

Remove Comments from IIS Logs

If you think that Log Parser is a bit on the slow side (i.e. if you’re dealing with big IIS logs) and you want to bulk import your logs into SQL Server, then you’ll have to remove # comments from the log files. Microsoft has the PrepWebLog Utility to do this, but it seems to choke for files that are > 100 MB. Also, you’ll have to write this as a batch file so it goes through a whole directory of files.

I wrote a Perl script that’s relatively fast (faster than PrepWebLog) and it can crawl folders/subfolders recursively. Here it is:

# parse.pl
# example: 
#   parse c:\temp\logs\logs*\*.log
#
# Requirement: no spaces in the directory names and file names.
# This gets called via run.bat. 


sub getFileList 
{    
    # This function returns an array of file list based on filter
    # This is the filter they can put in.       
    # Returns a file with full path. 
    # Example of filters: getFileList ( "*.log" );
    @files = ;
    return @files;    
}


sub remove_comments
{
  # Remove # pound sign comments from files. 
  # @_[0] = filename
  
  open (my $in, "", "@_[0].txt") 
      or die "out: @_[0]";

  while( my $line = )
  {
      print $out $line
          unless $line =~ /^#/;
  }

  close $in;
  close $out;
}


########## MAIN #############
$arg = @ARGV[0];

# Location of root directory of logs files
#$arg = 'c:\temp\logs\logs*\*.log';

# Replace slashes
$arg =~ s/\\/\\\\/g;

# Loop through all the log files. 
for $file (getFileList ($arg))
{  
  print ( "Processing file $file ... \n" );    
  remove_comments( $file );  
}

The Perl script gets called via run.bat:

REM No spaces in directory and file names.
perl Parse.pl D:\statesites\W3SVC*\*.log
pause

IIS Logs Scripts

While working with some IIS logs, I decided to start practicing my Python. I put together some handy Python functions to work with IIS Log files. These will come in handy. On a 3GB, 2.5GHz, running WinXP machine, these functions take about 3 seconds to process a 180MB Text file. Python code could be optimized to be faster if you’re dealing with larger sized files.

#!/usr/bin/env python

# An IIS log file can have various log properties. Everytime you add new columns to log for
# in IIS, it creates a new row full of columns.
import re
import os

MainLogDelimiter = "#Software: Microsoft Internet Information Services 6.0"
TestFile         = "C:\\Dan\\IIS-Log-Import\\Logs\\not-the-same.txt"
BigTestFile      = "C:\\Dan\\IIS-Log-Import\\Logs\\ex090914\\ex090914.log"
LogsDir          = "C:\\Dan\\IIS-Log-Import\\Logs"

def SearchForFile( rootpath, searchfor, includepath = 0 ):
  
  # Search for a file recursively from a root directory.
  #  rootpath  = root directory to start searching from.
  #  searchfor = regexp to search for, e.g.:
  #                 search for *.jpg : \.exe$                     
  #  includepath = appends the full path to the file
  #                this attribute is optional
  # Returns a list of filenames that can be used to loop
  # through.
  #
  # TODO: Use the glob module instead. Could be faster.  
  names = []
  append = ""
  for root, dirs, files in os.walk( rootpath ): 
    for name in files:
      if re.search( searchfor, name ):
        if includepath == 0:
          root = ""          
        else:          
          append = "\\"
        names.append( root + append + name )        
  return names  


def isSameLogProperties( FILE ):
  # Tests to see if a log file has the same number of columns throughout
  # This is in case new column properties were added/subtracted in the course
  # of the log file.
  FILE.seek( 0, 0 )
  SubLogs = FILE.read().split( MainLogDelimiter )
  
  # SubLogs[0] Stores the number of different log variations in the log file  
  SubLogs[0] = len( SubLogs ) - 1    
  
  # Grab the column names from the log file, separated by space
  columns = re.search( "^#Fields:\s([\w\-()\s]+)$", SubLogs[1], re.IGNORECASE | re.MULTILINE ).group(1)   
  LogSameProperties = True
  
  for i in range( 2, SubLogs[0] + 1 ):
    # If there are columns
    if ( len( columns ) > 0 ):    
      if ( columns != re.search( "^#Fields:\s([\w\-()\s]+)$", SubLogs[i], re.IGNORECASE | re.MULTILINE ).group(1) ):        
        LogSameProperties = False
        break  
    
  return LogSameProperties
  

def getFirstColumn( FILE ):
  # This gets the columns from a log file. It returns only the first columns, and ignores another column
  # row that may exist in case new columns were added/subtracted in IIS. 
  # input: FILE
  # output: 1 single element List
  FILE.seek( 0, 0 )
  names = []
  # Grab the column names from the log file, separated by space
  names.append( re.search( "^#Fields:\s([\w\-()\s]+)$", FILE.read().split( MainLogDelimiter )[1], re.IGNORECASE | re.MULTILINE ).group(1).strip() )
  return names
  

def getAllColumns( FILE ):
  # This gets all the columns from a log file. 
  # input: FILE
  # output: List
  FILE.seek( 0, 0 )  
  names = []
  SubLogs = FILE.read().split( MainLogDelimiter )    
  # SubLogs[0] Stores the number of different log variations in the log file  
  SubLogs[0] = len( SubLogs ) - 1        
  for i in range( 1, SubLogs[0] + 1 ):        
    names.append( re.search( "^#Fields:\s([\w\-()\s]+)$", SubLogs[i], re.IGNORECASE | re.MULTILINE ).group(1).strip() )  
  return names  


# EXAMPLE:
# Loop through all the IIS log files in the directory
# for file in SearchForFile( LogsDir, "\.txt$", 1 ):  
LogFile = open( file, "r" )
if ( isSameLogProperties( LogFile ) ):
  print file, "the same"
else:
  print file, "not the same"
LogFile.close()

Handling Images from the Command Line

Recently, I needed to do some work from the Windows Command Line, and I needed to deal with a few images. Along the way, I found some great tools. All these tools are free. Can come in useful when automating.

Manipulation

ImageMagick – This is a collection of command line tools. You can do image conversion, view properties, transform, transparency, join, overlay, add special special effects, and tons more. Also has APIs for C, C++, Java, .NET, Perl, PHP, Python, Ruby, and others. Highly recommend it.

Screen Capture


CmdCapture – Takes a screenshot of your desktop from the command line.

IECapt – Capture Internet Explorer’s rendering of a web page into a BMP, JPEG or PNG image file.

Cutycapt – Capture WebKit’s rendering of a web page into a variety of vector and bitmap formats, including SVG, PDF, PS, PNG, JPEG, TIFF, GIF, and BMP.

wkhtmltopdf – Convert HTML to PDF using the Webkit rendering engine, and Qt.

Reprogram Your Keyboard Keys

I recently got a new Microsoft keyboard without the Right-Windows-Key. I use this all the time ’cause I’m a shortcut fanatic, and I couldn’t get used to using the one on the left. Not only that, but I use a CTRL/SHIFT + INSERT a lot to copy/paste and the keys were way different from my past keyboard…

along comes AutoHotKey….

This program rocks. With it, you can create a script that runs and stays in memory. Remap keys and buttons on your keyboard, joystick, and mouse. In any case, with this tool, I was able to remap the menu key (the one usually between the Right-Windows-Key and CTRL key) that I never use to a windows key. That was close enough to me. Once installed, I was able to create a script with this command:

AppsKey::RWin

And that’s it! That mapped the menu key to the windows key! I’ll be experimenting with this program a little more and trying to get more shortcuts. You can also compile your scripts into a program that runs without an installation – in case you want your shortcuts / remapping run on a different machine.

The URL is: http://www.autohotkey.com/

Shopping Cart Basics

There are actually two parts to a shopping cart application. The first consists of the actual shopping cart (where the users search and buy items from) and the other is the administrative portion where tasks such as setting price, adding/deleting items to the database are done.

Shopping Cart

An Internet shopping cart actually consists of many subparts. These are the basics:

  • Catalog
  • Session Tracking
  • Search Engine
  • Security
  • Payment
  • Return/Exchanges

Catalog

A catalog is information about goods. Customers can browse the catalog (web pages) and see what there is in stock. Each good will most likely be composed of the name, picture, and price. Rather than assigning an item to each web page, the items can be generated dynamically from a database on to one page. Instead of listing all items on one page, a customer can browse through page by page. (All the items will be browsed page by page. For example, if there are 20 items, then it’s broken down into 4 pages. If there are 100 items, it will be broken down into 25 pages, with links that say Next/Back so they can go to the next page. In a catalog, it’s preferable that items be listed in categories.

Session Tracking

While a customer is browsing a catalog and looking for items, there has to be a way to keep track of what items he/she clicks on (so it gets sent to the customer’s shopping bag). This is done through session tracking. On check out, the check out process will tell them to register (fill in their information and pick a username and password) before checking out. The information, after they register, is saved in the database so they can login next time they enter the site and not have to reenter it again (all except credit card information).

Search Engine

This is closely related to the catalog. Rather than browsing and clicking links to find the item the customer wants, they can go to a search textbox and search for an item. This will be located on a visible spot on each page. After the customer submits the search, it will display the results like a catalog with Next/Back links.

Security

This is vital when processing credit cards and keeping customer information safe. Information submitted by the customer will be encrypted and stored in the database. Also, when submitting information so credit cards get processed, it will be submitted under an SSL secure connection. A logo can be shown by whomever company handles the SSL to make it shown that there is in fact security.

Payment

Directly connected with session tracking and security. Before checking out, customers will have to submit valid information about themselves. They will be sent to a form page under SSL where they must put valid information. After they submit, the data will be validated then sent off to a gateway where the funds will be sent to a merchant account. All done securely. An automatic e-mail will be sent to the user when this process is done.

Return/Exchanges
Done as secure as Payment. The customer will pick which item he wants to return, put all the necessary information from the invoice they received, and the return/exchange will be processed.

Administrative

The administrative portion consists of two basic subparts:

  • Content Management System (CMS)
  • Data Inventory

Content Management System (CMS)
There will be pages on the web site that have significant importance. For example, the home page and the catalog pages. These are pages that will constantly be changing. (The home page can contain items on sale, popular items, “just in,” etc. It might also be a good idea to include some of this home page information on the catalog pages because they are constantly browsed.)

Instead of changing the pages manually (opening up the page, changing the source code of each page), it will be faster and safer (changing the source code has to be done carefully so no errors occur) to do it through an administrative feature. You can go into an administrative panel, and just change that portion of the page that you want to update. There would never be a need to look at any code, just a click, type, and submit. Everything will be done through options. This is what a CMS does. Manages the content of pages.

Data Inventory

There will be a section under the administrative feature where company employees can insert/update/delete information from the database. This information can be customer, item, purchase information. They can also check out a queue of the items customers have purchased.

Shopping Cart Application Add-ons


Contingency Plan

A plan that’s executed if something fails or is performing poorly. For example, if the host goes down, there’s an error in the application, or there’s a hacker break-in.

Automation

Schedule content through the content manager. Can assign by date when something will be on sale, for example. When the date comes, it will display the new information on the page automatically.

Forum/Message Board

This will create an online community.

Chat

Where people can interact in real time.

Rating

Each item can have its own rating. For example, 1 out of 5 stars.

Reviews
Users can critique each item. Contains language filter.

Reports

Stats can be generated according to customer’s habits. What they click on, what the buy, etc. Reports can be generated according to this information. As an option, automated actions can be taken upon reporting information. For example, if a customer has bought purely shoes in the past from the site, then show shoes on the home page.

Advanced Search

Search not only by item name, but by price range, date range, and any other related information.

Data Export

Export information from the database as text files, XML, Access and Excel.

Mailing Lists

Send customers e-mails with news on prices, offers, new items, etc.

Portals

A page that contains links and other related information about other sites.

Gift Certificate

Gift Certificates can be bought online so it can be used in-store or online.

Site Map

Will contain a list or a map of all the sections and pages of the web site for quick navigation.

Other Online Marketing Options:

  • Affiliates
  • Web Rings
  • Forums
  • Search Engine Submission
  • METATAG creation