Author Archives: Alex

TXF Converter for 2010 ThinkOrSwim and Ameritrade Excel to CSV Files

After attempting to import my equity trades into Intuit’s TurboTax 2010 Home and Business edition, I discovered that  support for importing CSV files has been removed. So, I decided to roll my own program.

You can download the source here or download the Windows executable here. Please note, the .NET 4.0 framework is required. After executing the program, you need to enter in the full file path [directory and file name] in the file text box and the full path for the directory where you want the file to be created. A ‘TXF’ file should be created afterwards in the specified location.

You can import the TXF file in TurboTax 2010 by opening your current tax return and selecting File -> Import -> From Accounting Software. Check the details before finishing the import to make sure everything looks right.

If you want more details about the TXF file format or the CSV reader I used in the program. Please check out the following links.

TXF Format
LumenWorks CSV Reader

 I’ve only tested the TXF files from this program on Turbo Tax 2010 Business and Home edition. This software is provided without warranty or support.

IIS 6 Optimizations: Connection Timeout, Application Pools, Threads, Connection Strings, and File Compression.

One of the sites I maintain was unable to serve pages due to the server timing out. At first, I suspected it was a bandwidth issue, since traffic was significantly elevated due to a recent ad campaign that was initiated. On close examination, I noticed the system’s resources were barely taxed at all. Hard disk I/0, CPU usage, bandwidth, and memory were either normal or barely used.

I then remembered a similar incident with another site that showed the same symptoms, except the site was hosted on an Apache server. The problem was that Apache was consuming too much memory and was unable to fulfill requests fast enough due to having a high connection timeout setting. The fix was to limit the amount of processes spawned and reduce the connection timeout setting.

Since memory and CPU usage was not an issue, the only other suspect was to check the connection timeout setting. The Connection Timeout setting is located under:

  1. Open IIS Manager

  2. Expand websites

  3. Select target website’s properties.

  4. Select Website tab

On default, it is set to 120 or 900 seconds – a very long time, especially if a web server has a queue of 1000+ objects to serve. My recommendation is to set the Connection Timeout setting to 2-5 seconds. This will free up the amount of connections available to users immensely. In addition, there are other optimizations to threading and compression that aren’t enabled by default.

The following tweaks will help manage processes and threads for serving pages. In the %WINDIR%\Microsoft.NET\Framework\v1.1.4322\CONFIG\machine.config, set the following:

Setting
Default “Optimized”
maxconnection 2 12 * #CPUs
maxIoThreads 20 100
maxWorkerThreads 20 100
minFreeThreads 8 88 * #CPUs
minLocalRequestFreeThreads 4 76 * #CPUs

Enabling file compression can free up a significant amount of bandwidth and increase the number of objects served at a time. First, give the folder where the compressed files will be stored write permissions to the IIS User. By default, this is “%windir%\IIS Temporary Compressed Files”. The IUSR_{machinename} will need the write permission to the folder.

Next, you need to enable Compression in IIS.

  1. Open IIS Manager

  2. Select the websites folder

  3. Right click and select properties.

  4. Select the Service’s tab

  5. Enable Compress application files

  6. Enable Compress static files

  7. Set max size to something sensible.

Then enable the direct Metabase editing. This allows you to edit the metabase while IIS is running.

  1. Open IIS Manager

  2. Right click the computer icon

  3. select properties

  4. Check the checkbox ‘Enable Direct Metabase Edit’

Afterwards, you will need to open the IIS metabase by editing the file, %WINDIR%\system32\inetsrv\metabase.xml. Then find the attribute HcDynamicCompressionLevel and change the setting from 0 to 9, zero is the default setting. This will compress the pages as much possible without straining the CPU too much. There will two HcDynamicCompressionLevel attributes, each one under IIsCompressionScheme element that set deflate and gzip behaviors. Save the file.

Next entails adding the file extensions for static and dynamic pages that will be compressed. To do this, open the command prompt and open the directory C:\InetPub\AdminScripts\. Then execute the following to enable compression for the most commonly served files.

cscript.exe adsutil.vbs set W3Svc/Filters/Compression/GZIP/HcFileExtensions "htm" "html" "txt" "ppt" "xls" "xml" "pdf" "xslt" "doc" "xsl" "htc" "js" "css"

cscript.exe adsutil.vbs set W3Svc/Filters/Compression/DEFLATE/HcFileExtensions "htm" "html" "txt" "ppt" "xls" "xml" "pdf" "xslt" "doc" "xsl" "htc" "js" "css"

cscript.exe adsutil.vbs set W3Svc/Filters/Compression/GZIP/HcScriptFileExtensions "asp" "dll" "exe" "aspx" "asmx" "ashx"

cscript.exe adsutil.vbs set W3Svc/Filters/Compression/DEFLATE/HcScriptFileExtensions "asp" "dll" "exe" "aspx" "asmx"

IISreset.exe /restart

Do not compress images, as they are already compressed [jpg, gif, tiff, etc.]

Adjusting the IIS Application Pool can help resource utilization. The settings I changed were:

  • ‘Limit Request Queues’ setting under the Performance tab. Depending on how many CPUs, Worker Processes, and memory available, this setting can increase performance. The machine I was tweaking this setting on had 4 CPUs and 8 GB of RAM. I increased the limit from 4000 to 10,000 with only 1 worker process enabled.
  • Another setting I disabled in the tab was the ‘Shutdown worker processes after being idle for..’. I noticed that starting worker processes was extremely expensive in terms of CPU usage.

If your site is database driven, you may want to turn off Connection Pooling and set a low connection timeout if you’re confident that your server will not have resources issues when a large number of database connections are opened. In ASP.NET, add the Connect Timeout and Pooling settings to the connection string:

Data Source=SERVERNAME;Database=DATABASENAME;Integrated Security=SSPI;Pooling=false;Connect Timeout=5;

For references, please consult the following Links

Import a MySQL 5 database into SQL Server 2005

*Update* 8-28-2009

After trying this out, I’ve found that this does not work… at all. MSSQL compatibility mode must of been built back before 2000. I will continue searching the internet for a solution, but my feeling is that I will have manually create the tables and then add the insert statements into the database.

This is how I imported a database using MySQL 5 running on Ubuntu 8 LTS to MS SQL Server 2005.

First export the database by typing in a Ubuntu shell:

‘mysqldump -p –compatible=mssql databasename > mssql.sql’

The ‘-p’ switch prompts for a password, while the ‘mssql.sql’ file is the exported database file. If you need to specify a username, add ‘-u username’ as the first parameter or before the -p switch.

Transfer the file over to the Windows machine and  create a database for the import. Make sure you have permissions to transfer data via Windows Authentication. If you don’t have permissions, you can specify credentials using the -U and -P switches.

Run the following command line:

“C:\Program Files\Microsoft SQL Server\90\Tools\Binn\sqlcmd.exe” -S localhost -d databasename -i “mssql.sql”

Before running, verify that the ‘sqlcmd.exe’ file path is correct. The -S switch is the host name, server name, or location. You may have to use the machine name depending on the name of your instance. The -d switch specifies the database to import the data.

ASP.NET: Modifying DetailsView Select and Updating Events

Recently I was tasked with encrypting/decrypting some data that could be edited using ASP.NET’s DetailsView control.  Along with the details view control, the legacy code used a SQL Data Source to select and update everything. This means the data access layer was bypassed completely, so any business logic that dealt with encryption on this page had to use a different execution path. So, how would you encrypt and decrypt data?

The problem could be divided into two tasks:

  1. Create a filter to decrypt data in any bound columns.
  2. Edit the DetailsView event, ItemUpdating and figure out which columns needed to be updated during the event.

The first task was the easier of the two, as it just involved referencing the encryption library from the code behind page and calling the function inline in the aspx page. So, the method in the code behind page would look like something like this:

protected string Decrypt(object input)
{
return EncryptionHelper.Decrypt(input.ToString());
}

While calling the decrypt function in the DetailsView child  Fields tags on the aspx page:

<asp:templatefield headertext="Decrypted Data" sortexpression="EncryptedData>
<edititemtemplate>
<asp:TextBox ID="TextBox1" runat="Server" Text='<%#Decrypt(Eval("EncryptedData"))%>' />
</edititemtemplate>
<itemtemplate>
<asp:label runat="server" text='<%# Decrypt(Eval("EncryptedData")) %>' id="Label1"></asp:label>
 </itemtemplate>
</asp:templatefield>

However, decrypting the data using a templatefield tag posed another problem, extracting new values from the ‘TextBox1’ control. This will be solved when calling the ItemUpdating event.

After setting the DetailsView’s ItemUpdating method, you will need to modify the SQL Data Source’s UpdateCommand, UpdateCommandType, and UpdateParameters. The legacy code in the SQL data source used a update statement with parameters. Normally the details view control automatically detects whether to pass a null in the parameter. Instead when  setting the UpdateParameters in the codebehind, the control will update the database with the parameter names entered. For example ‘@Parameter’, will be sent in the update to the record instead of null. To solve this, you would need to automatically detect whether a value is null and then add it to the UpdateParameters list. This was done using the ‘IsFieldNull’ method [shown below].

Initializing the SQL data source also had another problem – because certain data needed to be re-encrypted before being sent back to the database. You will need to single out any columns that need to be encrypted. On top of that, the table being updated had over 50 columns, so typing in 50+ lines just to initialize the parameters was required, but I decided to do something else instead.

The parameters being sent to the stored procedure were identically named to the column names. Using this protocol, I decided to simply grab all the field names, which were identical to the column names and add the ‘@’ character next during initialization. So, the method to get the column names looked like the following:

private List<string> GetColumnNameList()
{
List<string> Ret = new List<string>();

for (int i = 0; i < DetailsView.Rows.Count; i++)
{
if (!String.IsNullOrEmpty(DetailsView.Rows[i].Cells[0].Text))
{
Ret.Add(DetailsView.Rows[i].Cells[0].Text);
}
}
return Ret;
}

Initializing the update parameters looked like the following:

private void InitSqlDataSource()
{
SQLDataSource.UpdateCommand = "Update";
SQLDataSource.UpdateParameters.Clear();

List<string> paramList = GetColumnNameList();

foreach (string col in paramList)
{
if (!IsFieldNull(col))
{
if (col != "EncryptedData")
{
SQLDataSource.UpdateParameters.Add(col, "@" + col);
}
else
{
SQLDataSource.UpdateParameters.Add("EncrptedData",
EncryptionHeper.Encrypt(GetDecryptedValue()));
}
}
}
}

The method checks to make sure the data being sent isn’t blank or null by executing the ‘IsFieldNull’ method. This method uses the ‘ExtractValuesFromCell’ method to grab values from the DetailsView control. This method isn’t exactly well documented on MSDN, so it’s not obvious at first that the passed parameter, IOrderedDictionary dictionary, is being used as a referenced parameter, not as a copy. [another weird quirk about the .NET framework] The method, GetValues, used is identical to what is covered at the article written at David Fowler’s blog.

private bool IsFieldNull(string fieldName)
{
OrderedDictionary vals = GetValues(DetailsView) as OrderedDictionary;

if (vals[fieldName] == null)
{
return true;
}

return string.IsNullOrEmpty(vals[fieldName].ToString());
}

public IDictionary GetValues(DetailsView detailsView)
{
IOrderedDictionary values = new OrderedDictionary();

foreach (DetailsViewRow row in detailsView.Rows)
{
if (row.RowType != DataControlRowType.DataRow)
{
continue;
}

DataControlFieldCell dataCell = (DataControlFieldCell)row.Cells[0];

if (dataCell.ContainingField.ShowHeader)
{
dataCell = (DataControlFieldCell)row.Cells[1];
}

dataCell.ContainingField.ExtractValuesFromCell(values, dataCell, row.RowState, true);
}

return values;
}

Angelo’s BBQ in Ft. Worth

It’s hard to find decent BBQ, especially in larges cities. Dallas, TX has its share of franchise BBQ restaurants. If you’ve lived in any of the surrounding DFW areas for a few months, places like ‘Sonny Bryans’, ‘Spring Creek BBQ’, and ‘Dickey’s BBQ Pit’ can be found easily. Each place has its own specials and caters towards a specific taste, however, I think they all fail to capture BBQ that’s comparable to what you find at cook offs at rodeos or competitions in the locale. I thought this would be a problem that would never be solved, until recently.

A few weeks ago, I was invited to dine in at Angelo’s BBQ in Ft. Worth. Don’t let the location or the decor of the restaurant fool you, the food in this place is extremely good. I got to try the beef brisket and the chopped beef, both were tasty and extremely tender. A quality that seems to have all but disappeared in all the franchise restaurants I’ve stepped into so far.

No place is perfect though. The place could use some fresh decor and maybe upgrade the serving ware from styrofoam to dishes.  I also advise to get there before nightfall. The location isn’t exactly in the most tourist friendly area. The restaurant location is near an industrial area, so I’m not sure how safe it is in the evenings.

Pricing isn’t bad for what you get, expect to pay around $10 for a dinner platter or $5 for a sandwich. Alcohol is also served there.

All in all, if you’re looking for BBQ that doesn’t taste dry or like it was reheated after being stored in the freezer, I recommend trying Angelo’s BBQ.

Back Online

After a long hiatus, I finally got my blog back online. I’ve been looking into quite a few new technologies the past few months and will certainly start writing about them this week.

Ubuntu Server 7.10: Installing Proftpd and ‘warning: unable to determine IP address of’

I had some trouble installing proftpd today on a Linode.com VPS server. For anyone that hasn’t used linode, it’s a colo service that lets people lease different grades of Linux VPS servers at reasonable prices. I’ve been using them for 4 months with few problems.

Installing proftpd in Ubuntu is easy. All you have to do is open a Terminal window or SSH into your server and type:

sudo apt-get install proftpd

However for some reason, the Ubuntu distros mounted on Linode’s servers have a quirk or two that require a few tweaks to get the server working. The first thing is whether to run the server in standalone or xinetd mode. The server won’t be able access port 21 unless it’s operating in standalone mode. So, set it to standalone mode.

The second quirk, which is not unique to Linode VPS servers, regards resolving the host’s address. You must add the server’s static address to the hosts file. For example, if your machine’s name is ‘johnny’ and the static IP address is 79.221.23.12.

To edit your hosts file type

sudo vi /etc/hosts

Your hosts file  should look like this:

127.0.0.1       localhost
79.221.23.12 johnny

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

You can determine your machine’s host name by opening the file,  ‘/etc/hostname’.

Aftewards, check your proftpd configuration. To do this, type:

sudo proftpd -td5

If you would like more information on setting up Proftpd or Ubuntu’s network configuration. Please consult the following:

Baking Ribs

For those of you enjoying your Super Bowl Sunday, I would like to extend to you a quick tip about preparing pork ribs. Baking ribs isn’t that hard. Just do the following:

  1. Preheat the oven to 200 degrees.
  2. Defrost ribs. If they’re still in those water / air tight plastic packages, you can defrost them quickly by submerging the plastic package in warm water.
  3. Season ribs and wrap the slabs in foil.
  4. After oven is preheated, put them on a cookie sheet and let them bake in the oven for 3 to 3.5 hours.

After baking them, just put BBQ sauce on it and sear the ribs either on a grill or hot skillet to get the BBQ sauce nice and toasted. For sauce, I usually use Original K.C. Masterpeice, and for seasoning, one of those dry rub seasonings made by Weber (yes, the same company that makes BBQ Grills).

PostgreSQL 8.2: Recovering from a Corrupted Database

Today I encountered my second database corruption or crash on my development machine. The problem is that my IDE drive sucks, an old IBM 75GXP. For those of you who don’t remember, these drives were notorious for having all sorts of problems when they were released. Most of the problems revolved around dying after six months. Regardless, I’m still using this thing for another month.

You can tell if your database became corrupted by restarting the postmaster or typing:

sudo /etc/init.d/posrgresql-8.2 restart

The result of the command will look similar to the following:

2007-11-26 13:14:48 CST LOG:  server process (PID 16312) was terminated by signal 11
2007-11-26 13:14:48 CST LOG: terminating any other active server processes
2007-11-26 13:14:48 CST LOG: all server processes terminated; reinitializing
2007-11-26 13:14:48 CST LOG: database system was interrupted at 2007-11-26 13:14:47 CST
2007-11-26 13:14:48 CST LOG: checkpoint record is at 0/965E4130
2007-11-26 13:14:48 CST LOG: redo record is at 0/965E4130; undo record is at 0/0; shutdown TRUE
2007-11-26 13:14:48 CST LOG: next transaction ID: 0/73629; next OID: 10952256
2007-11-26 13:14:48 CST LOG: next MultiXactId: 1; next MultiXactOffset: 0
2007-11-26 13:14:48 CST LOG: database system was not properly shut down; automatic recovery in progress
2007-11-26 13:14:48 CST LOG: record with zero length at 0/965E4178

Before initiating any recovery options, I recommend backing up your $PGDATA directory. The following directions will attempt to reset the transaction log, and you may loose the latest changes to your databases. The first option for recovering from a crash is to open your ‘/etc/postgresql/8.2/main/postgresql.conf’ file and turn on the following option:

...
zero_damaged_pages = true

After saving the changes, type the following to restart postgresql:

sudo /etc/init.d/postgresql-8.2 restart

If you still see error messages, you may need to restart the entire computer in order to get postgresql to start.

After restarting postgresql, reopen the ‘/etc/postgresql/8.2/main/postgresql.conf’ file and comment out the zero_damaged_pages option. The line should look similar to the following:

#zero_damaged_pages = true

After reading a few posts, it seems that IDE drives in general should not be used when working with databases because of corruption issues. If you are using an IDE drive, I recommend adding the following options to the ‘/etc/postgresql/8.2/main/postgresql.conf’ file.

fsync = on				# turns forced synchronization on or off
wal_sync_method = fsync

The problem with turning on fsync is that it slows down the performance of Postgresql, however, it should prevent further corruption issues with your database.

You can find more information about these configuration options in the Postgresql 8.2 documentation.

Reference:

When Not to Run on an Open Source Platform

The other day I had a nice chat with a colleague about the benefits and pitfalls of using an open source platform. I know this subject has been discussed deeply on various forums articles and blogs. I’ve personally done research for Alcatel-Lucent and the University of Texas at Dallas on the subject. For businesses, the bottom line is cost. Their are certain scenarios in a business where switching platforms, whether its open source or not, does not make sense. The scenario I discussed with my colleague revolved around the cost of switching from a Microsoft platform to a Linux platform for its custom server applications.

My colleague’s business faces two main hurdles before considering a migration: the time it will take to migrate custom applications and the expenses incurred during the migration. If the current system is meeting customer needs and does not need further changes, then migrating over to an open source platform may not be the best choice, especially if the lifetime of the system is not close to expiring. Open source software is best used when continuous scaling is demanded. Because of the licensing fees incurred when scaling can become astronomical, development of the system needs to consider the benefits given to a proprietary solution over a open source solution. For example, Microsoft’s Visual Studio has excellent tools for building .NET applications. If the time saved from developing on this platform is justifiable and the scaling needs do not supercede the cost  saved using an open source platform, then using a Microsoft solution may be warranted.

Despite my overall excellent experience developing on the Microsoft platform, I still have a hard time recommending it as the platform of choice, mainly because of cost and time savings. These savings are not obvious to people who have not used an open platform before. One way to explain the cost and time savings incurred is the notion of barriers or hurdles towards completing a goal. When developing or deploying a system, one of the hurdles towards completion is paying for licenses, which incurs some amount of expense in time and money. If the entire application stack is free, then you eliminate that expense altogether for the rest of the system’s lifetime. This allows the complete application stack to be cloned or deployed multitudes of times, whether in a test system, developer system, or production system, without incurring the expense of time and money you would encounter in a purchased product. Removing this hurdle has changed the way entire systems are deployed and general deployment of systems in both open source and commercial projects. One simple example is Debian’s software distribution, ‘apt’. A user can script out the default software configuration for a server with one command line. For example, installing a web server, office suite, browser, a couple of games, a couple of compilers, and a IDE with one shell command. Removing the purchasing barrier also paves the way towards completely automating the scaling of a system’s infrastructure. For the small software vendor, you just can’t do that on a Windows platform.