I recently bought a small cloud server from Rackspace with the goal of amalgamating various services I had hosted previously at Webfaction.

The SVN –> GIT migration was simple and painless (thanks to Jon Maddox for the reference instructions).  Importantly, all the commit history from Subversion was maintained.

Create a new git user:

adduser git

As the git user

mkdir repos

cd repos

mkdir trunk_tmp

cd trunk_tmp

git-svn init http://x.y.z/svn/trunk –no-metadata

git config svn.authorsfile ~/repos/users.txt

git-svn fetch

 

The users.txt should contain a mapping of svn to git users in the following format:

svnusername = My Full Name <my email address>

After the git-svn fetch operation has completed (it will download all revisions from subversion and may take a while), do the following:

git clone trunk_tmp trunk

rm –rf trunk_tmp

~git/repos/trunk is the new home of your GIT repository.

To access it remotely (over ssh):

git clone ssh://git@<host>/~git/repos/trunk

Simple as that.  I’m the only one using the repository so tunnelling it over a single user account is satisfactory.  I’d recommend setting up ssh private/public keys to make authentication more seamless.

Resource

• http://git.or.cz/course/svn.html – Provides a reasonable mapping of SVN –> GIT commands.
• http://www.jonmaddox.com/2008/03/05/cleanly-migrate-your-subversion-repository-to-a-git-repository/ – More detailed instructions on the repository migration.

The boys at JBoss had recently released Hibernate Search 3.2.0 and it looked pretty sweet. The possibility of performance improvements around indexing was enough to make me upgrade.

Unfortunately, like most software frameworks, Hibernate Search introduced/forced some new dependencies on us. We had previously been using Seam 2.2.0 and Hibernate 3.3.0 (both JPA1), neither of which were compatible with Hibernate Search 3.2.0.

We’re using Maven, so I’ll quickly outline the dependency changes I had to make (all dependences are available via the JBoss Nexus Repository):

<dependency>
    <groupId>javax.persistence</groupId>
    <artifactId>persistence-api</artifactId>
    <!– bit of a hack to make sure we aren’t transitively pulling in the persistence api (should be referencing hibernate-jpa-2.0-api now) –->
    <version>explicitly-fail</version>
</dependency>

javax.persistence:persistence-api is the JPA1 API and is no longer relevant and should never be included as a dependency. The explicitly-fail version will trigger a build failure if it ever manages to sneak in.

<dependency>
<groupId>org.hibernate.javax.persistence</groupId>
<artifactId>hibernate-jpa-2.0-api</artifactId>
<version>1.0.0.Final</version>
</dependency>

org.hibernate.javax.persistence:hibernate-jpa-2.0-api is the new JPA2 API published by Hibernate. 

All previous hibernate dependencies had to be upgraded to 3.5.2.Final, and Hibernate Search bumped up to 3.2.0.Final.

If you’re using Seam, you’ll also need to upgrade it to 2.2.1.CR1.  Note that this is not yet a final release but is necessary in order to support JPA2.

Unfortunately, Seam 2.2.1.CR1 lacks a complete Hibernate Search integration.  I could no longer able properly inject a FullTextEntityManager (via @In).

Instead I was forced to obtain a FullTextSession programmatically:

private FullTextSession getFullTextSession()
{
    Session session = (Session) entityManager.getDelegate();
    return Search.getFullTextSession  (session.getSessionFactory().getCurrentSession());
}

Seam also enhanced it’s EntityManager so that it now returns a JDK proxy when getDelegate() is called. 

That would be fine and dandy, but unfortunately the proxy does not implement org.hibernate.classic.Session, as required by the FullTextSession in Hibernate Search. 

Session.getSessionFactory().getCurrentSession() will return an org.hibernate.classic.Session and does allow you to move forward. 

A bit annoying and hopefully something that will get corrected in the near future.  I’m hoping that either Hibernate Search will be upgraded to not require a classic session, or Seam will provide an EntityManager delegate implementing the classic session interface. Better yet, Seam should just provide a fully supported integration with Hibernate Search 3.2.0.

Honestly, that was about it as far as the upgrade went.  We were able to index using either the new MassIndexer API or the old suggested best practice from Hibernate Search 3.1. 

It’s interesting to note that on my test system, the MassIndexer API was slower than the previous indexing code. The documentation does cover many different tuning possibilities so it’s entirely possible the situation could be improved.

Also, the MassIndexer API runs in a series of transactions and unless your timeout is set sufficiently high (ie. higher than the default JBoss 5 minutes), you will see transaction timeouts.

I didn’t attempt to work around this and opt’d to stick with existing indexing code that already runs in a separate Seam @Asynchronous method (thus avoiding timeouts).

Somewhat less than ideal, but if you’re keen on running Hibernate Search 3.2.0 w/ Seam in JBoss 4.2.3, it is possible and not too much work.

The objective is to learn enough about the platform during these initial largely custom implementations, to be in a position to hopefully create and deliver a largely standalone solutions faster than we would using the typical Java technology stack.

The jury is still out, but what follows are some initial thoughts after working with the platform.  The team in question are a group of 3 senior developers working collectively on an application deployed in a single Salesforce Organization. 

Forgive my use or miss-use of Salesforce/DeveloperForce terminology.  It can get confusing at times

Version Control

Like any good development shop, we store all code and build artifacts in source control.  Subversion is our poison of choice.

Salesforce development is a bit different in that a lot of the configuration and development activities can occur in either the browser or their Eclipse-based IDE. 

Our strategy for getting code and configurations under source control was to use Eclipse and regularly sync with our development organizations (Salesforce jargon for a development space or area). 

Unbeknownst to us at the onset, there are certain configurations you can make using the browser that are not possible to make with the IDE.  It’s important that these configurations (user/queue creation, etc.) are documented externally (ie. in a spreadsheet).

Code and configurations are moved between developers using Subversion and the Eclipse synchronization capabilities (or the Salesforce migration tool).  By storing these artifacts in Subversion, we’re also able to do code reviews on them via Crucible.

Your mileage may vary with this approach, as we found the current Salesforce tooling to be quite poor at conflict resolution when it came to structural changes.  Moreover, we frequently had to fall back to manually making changes via the web browser. 

Very annoying. 

I think it’s safe to assume that despite all the redeeming qualities of the DeveloperForce platform, some significant gaps remain when it comes to the expectations of traditional software developers.  By traditional, I mean those of us that are used to using source control, doing code reviews, etc.  I guess Salesforce considers us a dying breed

Would I choose the platform to build a product I expected to maintain in the long-term and derive significant value from? No. 

Would I use it in the short-term as a RAD environment to build a prototype? Possibly, but not without first considering some of the more traditional alternatives (there are a number of schema-less data storage products available these days, combine that with a Grails, Seam, Rails or Django and you may very well be off to the races).

That’s enough for one night.  I plan on doing a follow-up once the team has more experience maintaining and migrating code after it’s been deployed to DeveloperForce.

Particularly interesting for me was his overview of the different mechanisms by which you can support multiple customer schemas securely and with decent performance.  The product I’m actively working on is likely headed in this direction, looks like Oracle VPD is Emmanuel’s preferred solution.  A quick search has turned up an add-on for Postgres called Veil which aims to provide row/column-level security similar to Oracle VPD.  Good to at least have a choice, but I imagine that when push comes to shove, Oracle will win out.

Shards has always looked interesting, will be nice when it finally hits GA and supports the JPA API.  Would make it slightly easier to incorporate and play around with.

Interesting thoughts on clustering Lucene/Hibernate Search.  We’re currently running it asynchronously (ie. @Asynchronous in Seam) on a single node but will likely need to look at pushing it to a second box and trying to get indexes in near real time without noticeable degradation to the front-end.

That was on top of the better part of a day that was spent by another developer digging into the code.

Development ain’t easy, and frameworks for all their glory strive to make the easy stuff easier and the difficult stuff… well it’s still difficult.

Take Seam for example, a couple simple @In annotations here and there, and all of a sudden you have an application up and running. 

But throw transactions and exceptions into the mix and the expected behaviour is up in the air.

Imagine the following scenario:

Component A calls Component B.update() which in turn calls EntityManager.persist(someEntity).

someEntity fails a database constraint and an EntityAlreadyExists exception is propagated from Component B.update(). 

Can Component A turn around and update someEntity and call Component B.update() again?

It depends.  In Seam there is a RollbackInterceptor behind the scenes that will rollback any transaction crossing an injection boundary (it’s slightly more complicated than that but we’ll leave that for another day). 

If Component A was a POJO Transactional Seam component, and Component B was a Seam EntityHome component, this wouldn’t work.  In this sitation, Component B throwing an exception would actually rollback the transaction before control was returned to Component A.  Component A could very well handle the exception but the underlying transaction would still be rolled back.

From the looks of the implementation, the EntityHome (Component B) is relying on the entity existing in its PersistenceContext when doing an update.  You can update() multiple times as long as the transaction remains open and long-running.

Throw an exception into the mix and that transaction is likely to get rolled back by the RollbackInterceptor and the persistence context reset.

From that point on, calling Component B.update() (equivalent to EntityHome.update()) is going to report a success but do nothing but flush an empty persistence context. 

The short term fix was to have Component B.update() re-merge in someEntity.  However, rather than perverting the framework, it likely makes sense to dig deeper into the implementations and merge Component A and Component B to prevent the RollbackInterceptor from firing and rolling back the transaction on an exception that is recoverable.

The things you learn!

The Eclipse Memory Analyzer is a fast and feature-rich Java heap analyzer that helps you find memory leaks and reduce memory consumption.

The Memory Analyzer was developed to analyze productive heap dumps with hundreds of millions of objects. Once the heap dump is parsed, you can re-open it instantly, immediately get the retained size of single objects and quickly approximate the retained size of a set of objects. The reference chain to the Garbage Collection Roots then details why the object is not garbage collected.

Using these features, a report automatically extracts leak suspects. It includes details about the objects accumulated, the path to the GC Roots, plus general information like system properties.

The tool was actually contributed by SAP and is currently an incubation project over at Eclipse. It’s available as both an eclipse feature and a standalone eclipse RCP application.

Kudos to the team for taking the time to provide a standalone package!

About caBIG

caBIG™ stands for the cancer Biomedical Informatics Grid™. caBIG™ is an information network enabling all constituencies in the cancer community – researchers, physicians, and patients – to share data and knowledge. The components of caBIG™ are widely applicable beyond cancer as well.

The mission of caBIG™ is to develop a truly collaborative information network that accelerates the discovery of new approaches for the detection, diagnosis, treatment, and prevention of cancer, ultimately improving patient outcomes.

In a nutshell, caBIG is an initiative of the National Cancer Institute built heavily upon open-source components that aims to motivate and facilitate the sharing of data. It strongly suggests a front-loaded UML workflow (using MDA) and incorporates certain aspects of ontologies and common data definitions to help guarantee consistent semantics (and syntax).

From a purely technical perspective, I’ve never been sold on the idea of MDA. I’ve had experience with both open-source and commercial modeling tools that have never fulfilled on the promise of true round-tripping (and if you don’t have round-tripping… well, you’re in for a world of hurt). Now in the caBIG case, there is a pipeline of transformations that you’re more or less required to run through…

1. Create a UML representation of your object and domain models
2. Annotate the model with caBIG specific annotations and stereotypes
3. Run the annotated model through the Semantic Integration Workbench (another caBIG tool)
4. Submit the final model (XMI) to caBIG for approval and insertion into the caDSR (cancer data standards repository)

Make a change in the future and you’re more or less required to run through steps #1-4 again.

Once you have a validated UML model, you can then run through the caCORE SDK and generate skeleton code for a 3-tiered application consisting of (at a high level) a Hibernate data model, an external API and some middleware code to glue the API and data model together. Round-tripping is essentially non-existent from what I’ve heard and seen.

You’re done! Congratulations on achieving Silver-level compliance.

…

wait a second.

It’s a little bit too process heavy for my liking. I would have liked to see the NIH/caBIG be first and foremost focused on data interoperability and less on tools, particularly those that dictate particular workflows (like UML -> annotation -> MDA -> Code Generation).

I’d much rather see an extensible API with pluggable end-points, a meta data registration service and a suite of validation test cases. Define suitable goals and keep it simple. Provide suitable incentives and vendors will support it.

There’s more than one way to provide interoperability.

As a developer working on existing products that are considering support for caBIG, the requirement to fundamentally change my development process is a bit unnerving. Speaking generally, there’s no guarantee that everyone has UML models for their systems and even if they did, attempting to do full MDA transformations on them would be fairly ambitious.

That’s it for now. It’s been an interesting conference and I’ve learned a lot about the various initiatives and their progress. Off to visit customers in Cincinnati tomorrow!

Unfortunately for me, today was more or less a day of false starts. Like always, I had a number of ideas but quickly realized most of them were too much work or simply not interesting enough to devote time to. I was contemplating playing around with GWT or Android but couldn’t think of an interesting enough problem to warrant the effort setting up the development environment. I just couldn’t bring myself to install Eclipse.

On the Eclipse front, it’s a constant struggle. I think the community support it gets is great but wish other IDEs would get the same love (obviously it’s a matter of user demand). I’m an IntelliJ fan and find its lack of serious plugins for many of the popular open-source frameworks a bit disappointing. It’s Groovy support is top notch, things are definitely lacking when it comes to other popular frameworks (like Seam which has JBoss Tools for Eclipse but not much for IntelliJ).

I’ll leave the topic of Maven artifacts for another day. Needless to say that some of the more popular frameworks and libraries are starting to come around to it.

The day was not a complete loss and I did manage to make progress on a couple of things. The first involved some UI work that could eventually replace our current excel-based way of doing expenses. I’ve been seeing more and more references to Ext JS and wanted to check it out. I mocked up a bit of a GUI using some of their components but didn’t get much farther than that. I was quite impressed with their examples (Feed Viewer and Web Desktop). If you’re a web developer, I’d be interesting in hearing your thoughts on the library. I don’t do a lot of web-centric development but I have played a bit with Prototype, JQuery and Mochikit previously in conjunction with Rails and Django.

The more interesting thing I played around with today was OpenGrok.

OpenGrok is a fast and usable source code search and cross reference engine. It helps you search, cross-reference and navigate your source tree. It can understand various program file formats and version control histories like Mercurial, SCCS, RCS, CVS, Subversion, Teamware and Bazaar. In other words it lets you grok (profoundly understand) the open source, hence the name OpenGrok. It is written in Java.

OpenGrok is the tool used for the OpenSolaris source browser and search.

It’s actually pretty cool. I installed on to a Ubuntu 7.10 virtual machine without too many problems. There are a few gotchas (documented and undocumented) when it came to indexing our source code (from subversion) but in the end it worked pretty well. OpenGrok uses Lucene behind the scenes and provides a nice simplistic web application for searching and navigating the source code (there are also command-line and Swing interfaces).

Performance was very good. Indexing a checkout consisting of ~1.1G (lots of libraries and test data) didn’t take much longer than a couple minutes. Indexing the 11,000 change sets from subversion only added about a minute. This is on a VM w/ 512M on a MacBook Pro. Searches are instantaneous.

I haven’t played around with too many other source code x-referencers so I’m not sure I fully understand it’s potential or how it compares to it’s commercial and OSS competitors. That being said, I love the speed and if the OpenSolaris guys are using it, it can’t be that bad right?

It’s far from my day job and really just serves as a bit of an outlet. You’re free to argue about whether it’s a creative outlet or not The project stemmed from code written during one of hack days and has grown organically from there.

Essentially the goals of the project are as follows:

• Provide an interface to easily gather statistics from a JDBC connection. Currently this is accomplishable in two ways.
  • The delegating JDBC driver will write all executing queries to a log file alongside some basic statistical information (ie. how long it spent executing).
  • More interestingly, the delegating JDBC driver also registers an MBean that exposes a slightly more comprehensive set of statistics (ie. how much time has been spent executing sql, # of queries executed, etc.)
  • There’s also an example implementation of a TestNG listener that will track JDBC statistics on a per-test method basis.
• Provide a means of visualizing the gathered statistics.
  • There is a JDBCSpy Viewer under development. It has the ability to connect real-time to the JDBCSpy MBean and visualize activity. It can also reply a previously logged session by loading the JDBCSpy log file. It makes use of a bunch of the Orson and JFreeChart work so things don’t look half bad.

I’ve been using Maven for all build activities which satisfies one of the non-functional objectives.

If pressed, I’d say things are available in a beta-quality form. The code isn’t perfect nor particularly tuned. Most of the (recent) time I’ve spent on the project has been focused towards GUI refactoring. There’s plenty of more work to do on the JDBC driver side but it’s more or less taken a back seat right now. Part of the reason for this lies in the fact that there’s a few other projects that do something similar for JDBC logging.

Click here for a screenshot of the current viewer.

I’ll probably be adding some basic Groovy support (to enable user-defined filtering) to the viewer in the next commit or two. More or less just for kicks but it would also provide an opportunity to play a bit more with the language.

Cheers.

A Google of One

It’s just another phase in the continued commoditization of infrastructure. It started a decade or two ago with the OS and has been followed in quick succession by the web and J2EE stacks.

While not free, cloud computing offerings by the likes of Amazon.com have definitely opened the realm of possibility to the average developer.

Hadoop is just the next logical procession in all of this, an implementation of something Google pioneered and has been running for 4 or 5 years now. Continued contributions to the project from Yahoo! (assuming they continue to open-source the majority of their improvements) and others can only strengthen it’s appeal.

In the case of Google, they’ve been very successful because of what they’ve been able to build on top of their infrastructure. They have long realized the importance of well architected distributed systems and their developers see those benefits today. As developers, it’s amazing how simple things are when you’re able to think about them serially. Google and Yahoo! understand that, so should we.

The resources that used to be precious are now freely available. So then what becomes the bottleneck? What becomes the skill that is valued?

Two things:

* Implementation. The ability to leverage all the pieces together effectively.

* Business Acumen. By this, I mean having a smart business plan. If you are going to pull this off, you are going to need to cover the cost of using the cloud computing and the initial servers. Yet if you can come up with a way to make enough money off of each individual using the service, it could potentially be self sustaining. Google and Yahoo could even help you compete against them by supplying you with revenue from ads.

I’ll add Availability of useful and interesting data to this list. You can have all the processing power in the world but if it’s not directed towards solving interesting problems, you’ll have a very small audience.