Bundling SAC custom widgets

Please note: Information in this post will be extremely simplified so it is suitable for getting an understanding of the problem, it’s context, and a possible solution. It might not be fully accurate in some places, and might contain some bias based on personal preferences. The post is about a highly complicated topic which requires experience in various technologies related to JavaScript development, and is only meant as a very rough guidance – structured in a way to showcase the process how we get from the problem to the solution. Sorry: to one of the possible solutions. It is definitely not a step by step guide.

Custom widgets for SAC can be developed in JavaScript by implementing an HTML custom element. In a very simple scenario, this can be done as a single class definition in a single JavaScript file. However as requirements grow and features are added, you will want to structure your code into multiple classes, separate files, and maybe use some third party libraries – possibly even UI5 controls. But how can the browser load all these separate files and libraries, if you can only put a single URL in the manifest of your SAC widget? And how can you actually get all your bits and pieces together into a single file, so you can host your widget directly in SAC – which does not support any additional files except a single one for main view, designer panel and styling panel?

While we will first investigate how a multi-file application can be loaded into the browser in order to understand the loading process, remember, we have one ultimate goal: Package our widget – including all its JavaScript files, XML files (e.g. for UI5 XML views) and any other resources – into one single file. SAC will load that one single file, from there on no further request are allowed to fetch additional pieces or resources of our application.

An introduction to JavaScript module loading

While real programming languages and runtimes typically offer a native way of loading further software modules from day one (e.g. assemblies in .NET), JavaScript started off as a dumbass scripting language without any standardized modularization concept. You would put any number of <script> tags in your HTML, and it would load the source from the referenced locations and parse it top to bottom. Obviously, this will not work, if you have no control over the HTML DOM, and is quite inflexible anyway. For this reason, different groups of developers invented different mechanisms, formats and libraries for dealing with a modularized way of loading code, the most common ones being:

CommonJS: This way of defining and loading modules was primarily intended for non-browser scenarios, like for backends running on NodeJS. You would define your modules with module.exports = … syntax, and use the require(‘path/to/module’) to load the in the code where you need it. Require will make sure your module is only loaded a single time only. And – as it is meant for backend enviroments – it will load synchronously, it will only continue with the next statement once the requested source code has finished loading. While require() is a built-in function in NodeJS, a browser will not recognize this syntax. While there might exist some libraries trying to support require in a browser evironment, it is in general not a good idea, as the concept is missing asynchronous loading.

AMD: Asynchronous module definition format typically requires that your source files outermost statement is a call to a specific function – let’s call it ‘define’ for our example – where you pass the list of dependencies (the list of other modules the current module depends on), and a callback function which contains the ‘content’ of the module. The implementation of the define function will make sure all dependencies are loaded first – this happens conditionally (only load one module once), recursively (dependencies of dependencies) and, as the name suggests, asynchronously, potentially with parallelism – before the callback function is executed to evaluate the actual content of the current module. Typically a class (or function) definition wrapped within the callback function. The dependencies’ exported objects (typically classes or functions themselves) are passed as parameters to the callback function. This is how you can access the dependencies.
While the define() function is not known by browsers either, there is a bunch of libraries implementing the AMD concept (e.g. require.js) and also UI5 will use a very similar module loading concept – an AMD spin-off actually.

UMD: A kind of mixture of the above two, trying to make a module reusable in both environments.

ES6 Modules: With time, the concept of modules made it into the ECMAScript standard. Modern browsers support a native way of loading javascript modules. For this to work, the ‘root’ module has to be included with ‘type=”module”‘ in the script tag. From here on the source can use the ‘import’ keyword to load other modules, and has to specify the (typically relative) URL of the module to be loaded. (Or some custom name, along with an import map pointing to the URL). The other modules being loaded will need to use the export keyword in the source code to specify what is the content they are “sharing” with the outside world.
Loading is done in a similar way like for AMD. The biggest difference is the syntax (import/export) and that there is direct browser support without the need for any library.

When setting up your development environment and your build pipeline you can use transpilers (like babel) and bundlers (like webpack or Rollup) which can translate your source from one module loader syntax to the other and/or compile your code from multiple source files into a single file to loaded into the browser.

This means that you can actually code with the modern ES6 module syntax using the import / export keywords – which is especially handy when programing in TypeScript – and then have your application ‘translated’ e.g. into AMD style so it is loadable in older browsers. More on that later…

Module loading possibilities in SAC

In case of an SAC custom widget, it is the SAC runtime which is in control, and decides how it is going to load our root module. Let’s explore what is happening:

1. The widget manifest contains the path to the source as a single URL in the “url” property of the “webcomponents” section.
2. SAC will add a <script> tag with the ‘src’ attribute set to the above URL, and without ‘type=”module”‘. That means, our entry point javascript file is not loaded as an ES6 module, and thus it can not use the import keyword to load further modules with browser support.
3. Our code should register a custom HTML element – with the same name es defined in the widget manifest in the “tag” property.
4. SAC will add an element with this name to the DOM.

As you can see, our ‘top level’ widget code is not added as an ES6 module, so we need to use one of the other loading mechanisms.

Piggybacking the UI5 loader

If you have seen the user interface of SAC you have probably noticed it is quite ‘fiori like’, in other words it is very likely to be built with UI5. Indeed, you will find a script tag loading UI5 in the DOM.

In turn, this also means that UI5’s module loader (which comes very close to AMD) is active, and we could trick the UI5 loader into loading our modules as well.

Is that technically possible? For sure! We have already implemented SAC widgets which make of use of the fact that SAC is built with UI5. A ‘top level bootstrap code’ is what we load first, which will reconfigure the path mappings of the UI5 loader (using sap.ui.loader.config()) and call require() to load our second piece of code, which is already in AMD-like module format. From here on all further modules are automatically loaded using UI5’s module loader.

Is it a good idea? Depends… First of all, SAC has never officially stated that widgets can rely on a UI5 runtime being loaded when the widget is being instantiated. They could change the environment from one version to the next – use a different version of UI5, use different configuration settings, or not use UI5 at all. Second, setting up your code to be compatible with the UI5 loader also requires some effort and thus does not come for free. And third, in a basic setup (without Component-preload) it will load your modules one by one, which will not yield the best possible performance.

Using the UI5 loader for a widget which does not use any UI5 functionality (i.e. it does not use UI5 for the user interface within the widget) is probably not a good idea. It introduced an unnecessary dependency on UI5.

If you plan to use UI5 controls for the user interface anyway, using UI5’s module loader throughout the whole code might be a good idea. Without any additional tricks however, such widgets still consist of multiple files, and can not be hosted on SAC, but require a separate web server.

Bundling tools

As usual, there is no free lunch. You need to set up some kind of ‘build pipeline’ to process the source files into a format that is optimized for loading. Either the UI5 way, or by using one of the bundling tools. Still, lunch can be cheap – and yummy – as long as you know how to cook it.

In our example we decide to use Rollup for bundling. (There are also other open source tools, e.g. webpack.) What exactly will it do? It will be given a list of “entry point” source files, parse the dependency tree from there to determine which pieces of the source codes are relevant, and merge them into a single output file (or into multiple chunks). The format of this output file can be controlled from configuration, and could be AMD (for use with e.g. RequireJS), CommonJS or ES6 (mostly relevant if our output would be a library to be re-used in other projects), or IIFE. Latter stands for Immediately Invoked Function Expression, and is a way of encapsulating and separating a piece of JavaScript code from the rest of the codebase (e.g. to avoid global variable name conflicts.)

This IIFE format is the one we need to go for – at least at first glance, see later… – as our widget code will not be loaded by SAC via module loader, but included into the DOM with a script tag directly. Using IIFE means that “code-splitting builds” are not possible, that is, everything will be merged into a single output file. Exactly what we need! No module loading will be necessary / happening within our code. Everything is included right from the start, when that single file is loaded by SAC.

 {
        input: [
            'path/to/main.js'
        ],
        output: {
            name: 'MyWidget',
            dir: 'build/',
            format: 'iife'
        },
        plugins: [
            nodeResolve(), // so Rollup can find node modules using the node resolution algorithm
            commonjs() // so Rollup can convert CommonJS modules to ES6 and include them in a bundle
        ]
    }

Could we also opt for AMD? Yes. But that would require us to use a module loader library, and would only make sense if we want to split our code into separate files, some of which are loaded only on demand. This typically does not pay off for a simple project such as a simple widget. Or does it? Let’s first try without…

So how does the big picture look like?

1. We write the code using the keyword ‘import’ to reference dependencies. These imports can point to another file in our own source (a relative path), a node module we want to use (e.g. “dayjs” for formatting dates), and, if using UI5, something like “sap/ui/core/mvc/XMLView”.
2. These JavaScripts – including the source code of any referenced node modules – are bundled by Rollup into a single file.
3. This single output file is the one registered into SAC via the widget manifest.
4. SAC will include this single file into the page with a <script> tag.

What’s in the bundle?

If you look at the output bundle, you will see that depencies that are part of our project (a relative path in the import statement) are included in the bundle 1:1. If you import class X, you will literally find the code of class X in the bundle.

The second type of dependency often used are libraries imported from npm packages. If configured correctly – that is when using @rollup/plugin-node-resolve – these are also part of the bundle. For example when using the dayjs library, you will see the (minified) source code of dayjs within your bundle. The exports of dayjs are assigned to a variable ‘dayjs’ which is how our custom code will reference it.

As far as the widget code contains only custom code and references to libraries that can be combined into a self-contained bundle, everything will work as expected. (Not.)

Finally, there is a third group of imports – the ones which are not resolvable by the Rollup configuration, and will not be bundled. Rollup will report a warning during bundling – listing all these dependencies, and calls them ‘external dependencies‘. These dependencies will need to be supplied by the runtime environment. That is: they must already be loaded before our bundle file is loaded, and must be accessible to our code with the right global names (which can be specified in the config file of Rollup). This is where it might get tricky – especially in the SAC context wich is out of our control, and especially when using UI5 classes as dependencies.

Bundling for UI5 widgets: accessing the UI5 library classes

You will immediately stumble into this issue around the external dependencies when using Rollup to bundle a widget that uses UI5 for presenting its UI: In the JavaScript code, you would be using an import like “sap/ui/core/mvc/XMLView”, which will be treated as an external dependency by Rollup. Why? Because there is no node package for UI5 in your project that would export these classes. And anyway, you would not want to include the whole UI5 library into you single file package. It is huuuge. (Pun intended.)

So what can you do?

If we assume that SAC has already initialized the UI5 core components (because it relies on UI5 itself), a very central class like XMLView is already loaded, because it was required already to display parts of SAC itself, and is accessible under the global name sap.ui.core.mvc.XMLView. You can check that by entering sap.ui.core.mvc.XMLView into the browser console. If we rely on this ‘fact’, we could just map the dependencies with the following Rollup configuration:

output: {
            name: 'MyWidget',
            dir: 'build/rollup/',
            format: 'iife',
            globals: {
                "sap/ui/core/mvc/XMLView": "sap.ui.core.mvc.XMLView",
                "sap/ui/core/mvc/Controller": "sap.ui.core.mvc.Controller",

                [... all the other dependencies ...]
            }
        },

Rollup will still give a warning about the dependencies being external, but produce the following output: (simplified sample)

var MyWidget = (function (Controller, XMLView) {
        ... your source code with classes etc. is copied here ...
})(sap.ui.core.mvc.Controller, sap.ui.core.mvc.XMLView);

And this solution actually works quite well for common classes – even though it is not a fully correct architecture – but it might fail for any class which happens not to be loaded already, at the time your widget fires up. For example: any UI5 controls which are not used in SAC, but used in your widget. Such classes should ideally be loaded on the fly (asynchronously) by the default loader mechanism of UI5. But how do we tell UI5 that a certain dependency is now required?

Programming classic UI5 from scratch you would tell by using the sap.ui.define(…) method as the topmost line of your source file. Whether you know it or not, you are using the AMD like loader of UI5. Does that ring a bell? Yes, Rollup does support AMD as one of its output formats. So let’s change that to:

 output: {
            ...
            format: 'amd',
            amd: {
                define: "sap.ui.define"
            }            
        },

We need to specify that the loader function for UI5 is called sap.ui.define – the default used by Rollup would be just define(), without the sap.ui namespace.

So what the hell, didn’t we say we should use IIFE as output format? A few lines above I said:

Could we also opt for AMD? Yes. But that would require us to use a module loader library, and would only make sense if we want to split our code into separate files, some of which are loaded only on demand. This typically does not pay off for a simple project such as a simple widget. Or does it? Let’s first try without…

Well, if we use UI5 widgets, we do need to load (UI5 library) modules on demand, that is more or less by nature of UI5. Plus, in this case we anyway rely on SAC to be built with UI5 itself, and thus the module loader we need for AMD format is already included in UI5 and comes for free. So: For a simple widget which does not use UI5 controls, IIFE might be the better approach. It is lean and clean and makes no assumptions on the environment. For a widget based on UI5, AMD format goes hand in hand with the UI5 module loading mechanism.

Bundling for UI5 widgets: our own controller classes

Now we will ‘find’ all UI5 dependencies (UI5 controls etc.). But how will UI5 find us? If we reference a controller class by name in a UI5 XML view, how will UI5 know where to load that controller file from? Well, it won’t, unless we tell him.

One way to go would be changing the configuration of the UI5 loader (see the section ‘Piggybacking the UI5 loader’ above). We anyway have a ‘non-UI5’ part of the code which will instantiate the first UI5 view. There is 3 difficulties with that:

• If working with URLs, we would need to either hardcode the base URL of the widget hosting server, or figure it out dynamically.
• UI5 loader would in general load these files individually – not from our bundle. We would need to find a way to inject the modules straight from our bundle rather than having the loader load them on its own.
• It would be best to avoid tampering with the loader configuration at all, and make bundle just “fit in” to the original concept.

To understand what is happening under the hood, you can try to set up a test case where an XML view is instantiated (based on XML hardcoded as a string, as we cant yet load XML files neither 🙂 ) and references the controller class from the XML:

<mvc:View xmlns:mvc="sap.ui.core.mvc"
 controllerName="com.mywidget.MyMainView">

You will see that this view will lead to a load request to module ‘com/mywidget/MyMainView.controller.js’ which will fail. It will try to load this file from the same server as from where the UI5 library sources are originating. Our fix for that is to define the controller class as a named module while our single bundle file is loaded. This way the module is already known to the loader, implicitly stored at the point in time when our single bundle is parsed, and it does not need to be loaded again when it is required for the XML file. Our bundled source has to go something like this:

sap.ui.define('com/mywidget/MyMainView.controller', ['sap/ui/core/mvc/Controller', 'sap/ui/model/json/JSONModel'], (function (Controller, JSONModel) {
   class MyMainView extends ... {
   ... real code ...
   }
   return MyMainView;
}));

If you want some more insights into this, check out sap.ui.loader._.getAllModules() in the console. This will list all loaded modules, and can be used to check if your module was loaded correctly, and under what name. You will also see that .js is added automatically to the module name – it is not part of the define in the bundle, but internally part of the module name as tracked by the loader, and that there is a loader name for the module with the slashes, and an internal ui5 one with dots (and without .js in the end.) Finally, the fact that .controller is added automatically to the controller reference in the XML comes from the mandatory naming conventions for controller classes. This is how you can figure out with trial, error and pain, that the module name should be exactly com/mywidget/MyMainView.controller – with controller, without .js, and with slashes, not dots.

Now we know how define statements in our bundle should look like for controller classes – and that also means, we can not have all our code within one define statement, but need separate ones (with the correct module name!) for each resource that UI5 will be looking for. How do we trick Rollup into generating such a bundle?

• Configure Rollup to generate multiple chunks.
• One individual chunk for each controller class. Make sure that they get an AMD module id – use the right output bundle name, and use the autoId setting.
• One chunk for the rest. That is the main widget class and any helper classes not needed (=loaded) by UI5 directly. The module name does not matter here.
• Concatenate the individual chunk files one after the other, e.g. with the npm package ‘concat’.
• The order in which you concatenate might matter due to the dependencies between your classes:
  • Note this: If you import the classes for type checking purposes only – TypeScript to check the types during compile time – but are not using any class level artifacts, the class itself is not required at runtime, and thus it is not a real dependency. You will use the runtime instance of the class by reference, call its instance methods, but this works by the fact that those methods are available on the object – no need to ‘load’ any external code. When would the class itself be required? For example, if you create a new instance, or if you call static methods on the class.
  • In the main widget class you might typically manually create the instance of controller class yourself, which leads to a dependency. In such a case, when the main class is loaded, the controller class already has to be available, so that its constructor or static methods can be used. Otherwise the loader would try to load it via URL and fail.
  • Thus the right order (for most cases) is: Controller chunks first, then the main chunk.
  • If you import your main class into the controller class for typescript typing purposes only – that is no new instance creation, no static method calls etc. – you will not have any (real) dependency to the main class.
    If you use static methods you will get a circular dependency, which is a no-go, or you do this in one direction only (controller to main), in which case concatenation order needs to be swapped.
    If you do create a new instance of the main widget class within your controller class… Well… Consider a another job – something without programming…
• When importing a class, use the fully qualified module name for the import (instead of a relative path). This makes sure the dependencies in the bundle are named consistently. Note that Rollup will report imported classes as an external dependency because of this – this can be ignored.

Rollup configuration:
{
        input: {
            'com/mywidget/_MainChunk': 'source/WidgetClass.js',
            'com/mywidget/MyMainView.controller': 'source/MyMainView.controller.js',

        },
        output: {
            name: 'MyWidget',
            dir: 'build/',
            format: 'amd',
            amd: {
                define: "sap.ui.define",
                autoId: true
            }
        },
        plugins: [
            nodeResolve(), 
            commonjs() 
        ]
}

Build commands:
rollup -c
concat build/com/mywidget/MyMainView.controller.js build/com/mywidget/_MainChunk.js -o build/MyWidgetBundle.js

Excellent! Now we managed to have all our code in one file, but still registered piece by piece as own modules, so that the UI5 loader keeps track of them in its internal ‘database’.

By the way: if you were reading carefully, you might notice a bit of redundancy here. We have been working on a solution so that UI5 can load the controller class if it is referenced in the view XML. At the same time we mentioned you might want to create the controller instance yourself in the main class (and thus have a module dependency from main to controller.) Of course with manual instantiation of the controller, it would never be loaded based on the XML. But in a more complex solution you might mix these approaches – manual instantiation for main view controller, with implicitly loaded ones for subviews.

Bundling for UI5 widgets: and the really special ones…

Finally, there is one special topic: XML views (and other non-code resources). How can a sap.ui.core.mvc.XMLView load the XML source code automatically? Are XML files of XML views also loaded via the module loader? And if yes, how can they be bundled?

To answer the first question, take a look at sap.ui.loader._.getAllModules() for one of the UI5 demo apps, e.g. the shopping cart app. You will find a module ‘sap/ui/demo/cart/view/Product.view.xml’. If you have a peek into how the XMLView will load the XML, it will call jQuery.sap.loadResource which in turn will call sap.ui.loader._.getModuleContent(sResourceName, mOptions.url). So, luckily, there we are again, at the same module loader.

Unfortunately, were are only half way there yet…

In the demo apps you will find that the XML content is already loaded at the point when it is first requested. If you check the same for your widget: the XML will not be there, getModuleContent() will return undefined, and the method one stack level further up (jQuery.sap.loadResource = LoaderExcetions.loadResource) will execute an AJAX fetch to get the file from the server. This will obviously fail.

In the demo apps the XML is already available for a reason: It has been loaded with a Component-preload in advance. We do not have such a thing in our solution (yet). The component preload’s data is ‘injected’ into the loader class. Can we achieve the same from our bundle? Let’s try the same approach as for the controller class:

sap.ui.define('com/mywidget/MyMainView.view', [], (function () {
   let xml = `<?xml ... [xml content goes here] ... >`;
   return xml;
}));

“Computer says: no!”. The module loader will load this resource, but remember it as com/mywidget/MyMainView.view.js while the resource is being searched for as com/mywidget/MyMainView.view.xml. Can we fix that? Unfortunately not, here is why:

function ui5Define(sModuleName, aDependencies, vFactory, bExport) {
   ...
   sResourceName = sModuleName + '.js';

The extension .js is hardcoded in the UI5 source. How about simulating what a Component-preload is doing? Let’s look at a Component-prload.js file an another project:

sap.ui.require.preload({
	"com/whatever/SomeView.view.xml":'<?xml version="1.0" encoding="UTF-8"?><mvc:View>...</mvc:View>',
	"com/whatever/SomeView.controller.js":function(){sap.ui.define([...],function(...) { ... } },
        "com/whatever/manifest.json":'{...whatever...}'
});

That is, instead of just concatenating our chunks, we could instead ‘merge’ them into a preload file like above – at the same time we can also merge our XML view content. We don’t even need to put define() around the XML, we can simply include it as a string. Mind, that in the preload file you need to specify the file names including the extension.

If you try to do this for both your XML and JavaScript resources you will face issues though. Your main class will preload, but not execute – thus the implementation of the custom element tag will not be registered. You might try to ‘require()’ the main module at end of the bundles file to make it load and execute… Yes, you might, if you want to spend two sleepless nights trying to figure out why that does not work. Or, you use a less complicated approach: Only put the XML / JSON resources into preload(), and leave the JavaScript modules with a plain define() afterwards.

That way XML and other non-code resources will be registered as preloads, code artifacts will be registered as modules immediately, and finally your widget will load with all its bits and pieces.

(One watchout point: the trick with component preload will not work if preloads are disabled in the UI5 configuration, e.g. if you are in UI5 debug mode.)

I’m confused, what’s the conclusion?

Why did we go in circles, from UI5 loader to IIFE, then back to loading with UI5 loader?

First of all, using UI5 loading mechanisms without bundling load resources individually in it’s basic setup. But we wanted a single file.

Second, if your widgets don’t require UI5 components, you can use plain IIFE bundling method.

And third, investigating all these options is an excellent opportunity to learn about how on-demand JavaScript module loading, bundling, and the UI5 loader works.

Art the end of the day, everything described above are just options and strategies – you need to make your choice depending on the exact requirements and the size of the development. Frameworks come and frameworks go, but the knowledge you acquire during the problem-solving process will be yours forever.

PS: If you understood everything above at first read, don’t forget to send us your job application 😉