While there are multiple places where testing occurs in the life of releasing a managed package to the Salesforce AppExchange, let’s take a look at the transition from deployed code in a packaging org to a released version of a managed packaged.
Now, you may ask, is there a need to do more testing in , phase? Aren’t the unit tests, functional testing, regression testing, etc. already performed enough? I believe it is imperative to test the packaging of your app before releasing, as there is still room for defects to find their way into the final product. Testing on the final product is always a good idea.
For example, when packaging up your app, did you include every single new component or remember to delete that obsolete field? Perhaps you hit a ‘working as designed feature’ where custom labels used in an apex composition component are not included in package causing the install to fail. When including permission sets in the package, do they include permissions for a custom object that is the child of a master/detail relationship with a standard object (hint – the package install will fail)? What about the upgrade experience for existing customers, will everything work correctly and do your upgrade notes cover all the necessary steps to updated layouts, views and other non-upgradable components?
As many possible issues can arise, you owe it to your customers (not to mention the team’s effort to reach this point) to ensure a quality release. I will cover the testing from when release ready code is available in a packaging org through the managed release version. Before uploading a release version of the managed package to the app exchange, here is what I do.
Beta PackagesA great option in the packaging org is the ability to create beta packages. A beta package lets you create a package, install it into another org and test the package without having to commit to the components in a package. Conversely, with a managed package release version, the components are permanent. With each Salesforce release, there are additional capabilities for ISVs to remove components from released packages. However, certain components like global interfaces cannot be deleted or changed. In many cases, going right to a released version without final testing can leave you stuck with components you don’t want.
Although beta packages have a lot of benefits, there are a few limitations to outline. First, beta packages cannot be upgraded. The practical impact of this limitation is the following. When testing a beta package, if you find an issue and plan to release a second beta, it cannot be installed into the same org on top of the first beta package. You have to uninstall the beta package first. While this seems like a minor annoyance first, keep in mind that any related components must be removed or updated first. For example, if your installation process calls for button overrides, page layout assignments, data creation, etc., it must be undone first. This can greatly increase the effort involved. Next, beta packages can only be installed in developer orgs or sandboxes. So if you wanted to spin up a trial force org as a starting point and then apply the beta you are out of luck.
Creating a beta package is easy. From the packaging org, select a Release Type of Managed – Beta. Then you can install in a Developer, Sandbox or test organization for registered partners.
Beta INSTALL orgMy first step is a simple new installation test of the beta package. I have a dedicated beta test org that is dedicated for this purpose. I installed the package to verify that there are no unexpected installation problems (like the master/detail permission set issue) and to test out post install scripts.
While many managed packaged do not include permission sets, I encourage you to consider adding them. A simple search of Ideas on the Success Community shows that setup hurdles with managed packages is a source of common complaint. Permission sets assist with the setup processing, making it easy to assign functionality to groups of users. While permission sets are great, it is not possible to include standard objects in permission sets for a managed package. In addition, a permission set that includes permissions on a custom object that is child in a master/detail relationship with a standard object will cause the install to fail.
My approach has been to include everything possible in a permission set that is installed with the app. While details are included in the installation guide to create the additional permission sets manually, it is not the best admin experiences. I had tried to create the permission sets as part of a post-install script, but the context that this script runs does not have permission to do this. Feel free to vote for my idea to allow this. Instead, I offer an apex class that can be installed in the org, executed to create the additional permission sets and then removed.
While performing a beta install test, I do not go through extensive post-install setup or create sample data. Any additional setup or data, must be undone to uninstall. Remember, the point is to test the install, not the entire app. You will be installing and uninstalling in this org, a lot. It is possible to install and uninstall the managed package through the API, which can be a time saver.
Beta Upgrade orgThe second step in the process is to test the upgrade from the last released version of the package (betas cannot be upgraded). Here I have a second, dedicated beta upgrade org. This is installed with the latest, managed release version of the package prior to the beta. I would rather find out early in the process that there is an upgrade issue. Similar to the Beta Install Org, there is only minimal setup on the prior version. After upgrading to the latest beta, the package will need to be uninstalled to test again.
To further facilitate the installation and setup testing, I am working on setup and teardown processes that will use the meta-data API to further automate the testing process. More on that in the future.