How To Approach Manage Package Release Testing

While there are multiple places where testing occurs in the life of releasing a managed package to the Salesforce AppExchange, let’s take a look at the transition from deployed code in a packaging org to a released version of a managed packaged.

 

Now, you may ask, is there a need to do more testing in , phase?  Aren’t the unit tests, functional testing, regression testing, etc. already performed enough?  I believe it is imperative to test the packaging of your app before releasing, as there is still room for defects to find their way into the final product.  Testing on the final product is always a good idea.

 

For example, when packaging up your app, did you include every single new component or remember to delete that obsolete field?  Perhaps you hit a ‘working as designed feature’ where custom labels used in an apex composition component are not included in package causing the install to fail.  When including permission sets in the package, do they include permissions for a custom object that is the child of a master/detail relationship with a standard object (hint – the package install will fail)?  What about the upgrade experience for existing customers, will everything work correctly and do your upgrade notes cover all the necessary steps to updated layouts, views and other non-upgradable components?

 

As many possible issues can arise, you owe it to your customers (not to mention the team’s effort to reach this point) to ensure a quality release.  I will cover the testing from when release ready code is available in a packaging org through the managed release version.  Before uploading a release version of the managed package to the app exchange, here is what I do.

Beta Packages

A great option in the packaging org is the ability to create beta packages.  A beta package lets you create a package, install it into another org and test the package without having to commit to the components in a package.  Conversely, with a managed package release version, the components are permanent.  With each Salesforce release, there are additional capabilities for ISVs to remove components from released packages.  However, certain components like global interfaces cannot be deleted or changed.  In many cases, going right to a released version without final testing can leave you stuck with components you don’t want.

Although beta packages have a lot of benefits, there are a few limitations to outline.   First, beta packages cannot be upgraded.  The practical impact of this limitation is the following.  When testing a beta package, if you find an issue and plan to release a second beta, it cannot be installed into the same org on top of the first beta package.  You have to uninstall the beta package first.  While this seems like a minor annoyance first, keep in mind that any related components must be removed or updated first.  For example, if your installation process calls for button overrides, page layout assignments, data creation, etc., it must be undone first.  This can greatly increase the effort involved.  Next, beta packages can only be installed in developer orgs or sandboxes.  So if you wanted to spin up a trial force org as a starting point and then apply the beta you are out of luck. 

 

Creating a beta package is easy.  From the packaging org, select a Release Type of Managed – Beta.  Then you can install in a Developer, Sandbox or test organization for registered partners.

 

Beta INSTALL org

My first step is a simple new installation test of the beta package.  I have a dedicated beta test org that is dedicated for this purpose.  I installed the package to verify that there are no unexpected installation problems (like the master/detail permission set issue) and to test out post install scripts.
While many managed packaged do not include permission sets, I encourage you to consider adding them.  A simple search of Ideas on the Success Community shows that setup hurdles with managed packages is a source of common complaint.  Permission sets assist with the setup processing, making it easy to assign functionality to groups of users.  While permission sets are great, it is not possible to include standard objects in permission sets for a managed package.  In addition, a permission set that includes permissions on a custom object that is child in a master/detail relationship with a standard object will cause the install to fail. 
My approach has been to include everything possible in a permission set that is installed with the app.  While details are included in the installation guide to create the additional permission sets manually, it is not the best admin experiences.  I had tried to create the permission sets as part of a post-install script, but the context that this script runs does not have permission to do this.  Feel free to vote for my idea to allow this.  Instead, I offer an apex class that can be installed in the org, executed to create the additional permission sets and then removed.
While performing a beta install test, I do not go through extensive post-install setup or create sample data.  Any additional setup or data, must be undone to uninstall.  Remember, the point is to test the install, not the entire app.  You will be installing and uninstalling in this org, a lot.  It is possible to install and uninstall the managed package through the API, which can be a time saver.

Beta Upgrade org

The second step in the process is to test the upgrade from the last released version of the package (betas cannot be upgraded).  Here I have a second, dedicated beta upgrade org.  This is installed with the latest, managed release version of the package prior to the beta.  I would rather find out early in the process that there is an upgrade issue.  Similar to the Beta Install Org, there is only minimal setup on the prior version.  After upgrading to the latest beta, the package will need to be uninstalled to test again.
To further facilitate the installation and setup testing, I am working on setup and teardown processes that will use the meta-data API to further automate the testing process.  More on that in the future.

Beta Testing org

As I now know the beta package will install and upgrade, it is time for more thorough testing.  It is best to start with new install testing first.  I want to preserve my previously setup orgs for testing on the final beta version, where only potential upgrade issues exist.  This helps to preserve more realistic data from the prior version.  While I could use the Beta Install or Beta Upgrade org for this testing, it would defeat the purpose of quick and easy installs.  In this org, it’s time to run the app through its paces, including testing of permission sets or profile settings with different users.  Ideally, no issues are found and testing can move forward to the last step.  If issues are found, its time to build a new beta version, uninstall and start testing again.

Upgrade Testing

At this point, its time to test the upgrade from a prior version with full setup and data.  As many package components are not upgradeable (such as page layouts, list views and reports), it is good to check out the upgrade experience as well.  You can verify that upgrade notes accurately capture all of the needed changes.  Saving this for last minimizes the amount of testing setup that is needed. In my case, I have a several different testing configuration to try out.  Although, you may only need one depending on your app. It is worth mentioning that I have setup automated data loading to get an environment up and running quickly.  All of the data needed in an org is loaded with a master script using data loader from the command line.

Managed Released

One of the best benefits using a beta is moving to a released version.  Uploading a new package is simply changing a radio button on the package.  No package content has to be changed, which means no room for error.  This ensures what was tested against is what you release.  Remember, to go through and deprecate any beta packages created during testing.  This will not effect any installs, but prevent any new installations from occurring with this package version.  Finally, I go through one final smoke test by installing the managed package.  I will do this into the Beta Upgrade org, after uninstalling the beta version.  I will need the new version installed here next time.  While this is probably not necessary, I don’t want the first customer experience to be a failed install.

Wrap Up

With the above steps, I am confident in the app that I am releasing to customers.  Let me know your thoughts and if you have a different approach to release testing.  For example, if you are using a throw-away test namespace and going right to managed released version, I would enjoy hearing your results.  I am always looking to make the process better and faster.