The intelligence of machines and the branch of computer science which aims to create it

Artificial Intelligence Journal

Subscribe to Artificial Intelligence Journal: eMailAlertsEmail Alerts newslettersWeekly Newsletters
Get Artificial Intelligence Journal: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn

Artificial Intelligence Authors: Corey Roth, Pat Romanski, Liz McMillan, Yeshim Deniz, Kevin Benedict

Related Topics: Artificial Intelligence Journal

Artificial Intelligence: Article

Timing the Market with Distributed Genetics - Part 2

Picking the best stocks for investing a fixed amount of money

This event handler is passed as TaskStatusEventArgs. On this TaskStatusEventArgs is a parameter called TaskResult that has a property called TaskData. This TaskData property can be set by the code running on the remote agent and whatever value is set will be automatically returned to the client in this property.

There are two key catches to this property. The first is possibly the most obvious, which is that whatever class type you try to return must be marked as serializable. This stands to reason, because whatever you are returning is going to have to be serialized across the network. Fine!

My initial stab at this tried returning the full StockPicker instances, however, and this gave me a bizarre "TypeLoadExceptionHolder" error, with no further explanation (and, coincidentally, this exception type - although a part of the standard .NET Framework, is not documented by Microsoft!). After a good bit of digging and support from Digipede, what I realized was that .NET's serialization won't permit any type to be retrieved unless you have the exact type's assembly available on the retrieving computer. In my case, since I was randomly generating these types on the Grid, of course I didn't have these assemblies available on the central machine - nor did I want to return them.

Fortunately, as I said, we really don't need the implementation returned to the client - only the metadata (for breeding the next generation) and the success ranking - since this will form the basis for choosing which algorithms get to breed and which do not. Both of these things are encapsulated by the TypeExecutionResult class, so this is what we return.

Our final code listing - Listing 5 - shows the Executive Executable that is automatically distributed to Agent machines in the Digipede framework, and which remains in memory for the duration of a given Job (generation) in our solution. A key thing to understand about Executives and Jobs in the Digipede framework, however, is that an Executive can be temporarily assigned another Job to execute, if it is given higher priority. This is the reason why we key all of our collections at least partially by the Job ID.

In the constructor for our Executive, we connect to two sources of market data - MSN Money's MoneyCentralRemote Web Service and WebServiceX's INET market data publisher. A list of stocks is retrieved from WebServiceX and then this list is iterated through and detailed market data is retrieved from MoneyCentralRemote. This data will not change between Jobs, so it can be kept at the class level, unattached to a specific job instance.

When a given Job is added (see the JobAdded method), we find out if there are two or three parameters passed to it. If there are two, we have not been passed "seeds" for a next generation, so this must be the first generation, and we create it completely randomly. On the other hand, if we have a third parameter, we perform a clever bit of manual decoding.

This bit of encoding (see MasterApp) and decoding (shown here) was another key hurdle in implementing this solution. I knew that I needed to pass a list of TypeExecutionResults to serve as the basis for breeding another generation, but the Parameters in Digipede want to take strings. To make this work then I serialize it to a byte stream, Base64 encode it, and pass the resulting string.

It should be pointed out that you could add a stream-based FileDef to the job and pass the seeds down this way, rather than as a parameter. This would have the benefit of not having to perform the Base64 encoding and de-coding. In general, FileDefs represent a more flexible method for passing all different kinds of data using Digipede - but I felt that the Base64 parameter was sufficient for my needs.

The DoWork portion of the class is where the randomly generated (or bred) algorithms are actually loaded and run on the grid. The results of execution are loaded into a TypeExecutionResult and returned to the MasterApp, as explained in the previous section.

Final Thoughts
You may notice as you read through the code in Listing 4 that we pass a total of four arguments to our MasterApp application. The first two are used by Digipede and have already been explained. The second two are the number of algorithms to randomly generate or breed in each generation and the maximum depth of logic (how many if conditions to nest within each other) to allow in any given algorithm. It's important that you bump both of these numbers to a fairly high number to see the true benefits of grid-enabling such an application.

For example, if each algorithm takes a few seconds to run, simply having 1,000 algorithms in a generation isn't going to take long enough to really warrant putting such a solution on a grid. On the other hand, if you expand the maximum depth of logic enough, it could start to take a minute or two to generate and run each algorithm. If you increased the size of a generation to, say, 100,000 algorithms, you are now looking at a few days to complete a single generation, in the absence of a grid.

The approach I have outlined in this article will give you near-linear scalability as you increase the number of machines dedicated to processing for this solution. In this case, deploying to 1,000 computers could bring the runtime back down to a few minutes for a single generation.

Future Enhancements
The basic pattern for producing the solution outlined in this article was to adapt Brian Connolly's initial ant-finding-food genetic solution to the problem of picking stocks. In the opposite direction, I was adapting Digipede's out-of-the-box grid samples to encapsulate a genetic algorithm solution.

Along the way, I was able to introduce Generics and change most of the collections used in the initial genetic solution (created under .NET 1.1) to be strongly typed. An enterprising developer could almost certainly take the Gene portion of the solution and use genetics to create a master class that would work equally well for stock picking and guiding an ant.

Another area that seems ripe for further enhancement is the kind of properties supported by this genetic engine. Currently only bool properties can be used, allowing only "yes/no" decisions. It isn't difficult to envision an engine that could accept numeric properties and automatically compare them against each other and/or against randomly generated constants.

Finally, as I was writing this article, Digipede released version 2.1 of their software, letting developers easily leverage multiple cores by allowing more than one job to run concurrently on a single machine. Even in the version that I was using, there is a Concurrency setting on JobTemplates that allows you to state whether the tasks in a single job are safe to run concurrently. Given that increased parallelism is an absolute necessity for computing progress going forward (now that we will no longer be getting processors that are twice as fast every X years), this is an area of this product that I would like to have examined much more thoroughly.

I now pass the torch onto the rest of you for further exploration. Have fun - and feel free to contact me at [email protected] with any comments, questions, or concerns as always.

More Stories By Derek Ferguson

Derek Ferguson, founding editor and editor-in-chief of .Net Developer's Journal, is a noted technology expert and former Microsoft MVP.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.