Claims about scale are cheap, so we measured. On our MAS test system, Fiedler Data Import imported 10,000 job plans with their tasks and labor and deleted them again. It planned a workbook of a million rows, and it wrote that workbook’s result, inside the 2 GiB of memory its container may use.
10,000 job plans, with tasks and labor
The workbook had two sheets. The first held 10,000 job plans, each with two tasks. The second named the same job plans again, each with two labor lines. That makes 60,000 records to plan. Mike Wilson, a user of the test system, ran the job through the business object path.
| Stage | Time |
|---|---|
| Plan | 38.1 s |
| Run | 25 min 2 s |
| Result workbook | 7.0 s |
| Whole job | 25 min 47 s |
The job ended without an error. It inserted 50,000 records: the 10,000 job plans, their 20,000 tasks and 20,000 labor lines. The 10,000 job plans on the second sheet read unchanged, because the first sheet had created them by then, as the plan article explains. The delete below found all 10,000 job plans in Manage.
That is about 33 new records a second, each saved through Manage’s business objects with its rules. While the job ran, the app’s container used at most 89 MiB of its 2 GiB. On average it used 0.05 of a CPU core, and Manage’s container used 0.53, against 0.21 before the job. The time goes into Manage’s saves, and the app itself stays light.
Then a second workbook with $DELETE removed the 10,000 job plans with their tasks and labor. It
planned in 49.6 seconds and ran in 17 minutes 54 seconds. Afterwards Manage counted none of them.
A workbook of a million rows
The second test used a workbook as large as Excel allows: 1,048,576 rows of assets with 15 columns, 94 MB. Its data starts in row 4, so the plan has 1,048,573 rows, each a new asset.
| Stage | Time |
|---|---|
| Reading the workbook | 2.0 s |
| Plan, reading included | 53 min 49 s |
The plan ran inside the app’s container on the test system, without a restart. It used 1.1 to 1.2 GiB of memory for most of the time and up to 1.7 GiB in its last 15 minutes. The first attempt ran out of memory at its very end: the plan’s CSV export held the whole sheet at once, about 3 GB. We changed it to write as it reads, which needs 214 MB, and planned again.
We stopped after the plan, because a run would have written a million assets into our test system.
A result workbook of a million rows
After a run, the app writes the result workbook with a text and colours for every row, as the results article shows. For a million rows, the first version of that step held everything at once and needed 6.8 GB.
The result writer now streams: it reads the sheet, asks for each row’s result, and writes the row. On a copy of the million-row plan, in a container with the app’s 2 GiB limit and the same runtime, it wrote the result in 65 seconds with at most 1,070 MB. The first version died in the same container. Both versions wrote byte-identical files, for the million rows and for our 14 example workbooks.
What the numbers say
- The app stays light. At 89 MiB during the 10,000 job plans, it leaves its 2 GiB mostly unused, and a million rows fit as well.
- Manage sets the pace. A row group is saved through Manage’s business objects, with every rule and every automation script of yours, and that takes Manage’s time.
- The plan grows with the rows. 60,000 records planned in 38.1 seconds, and a million rows in under an hour.
What comes next
The next article is about running Fiedler Data Import: the installation, the metrics, the backup, the command line and the test suites. The Fiedler Data Import page lists the whole series.
Want to see how your largest workbook behaves? Get in touch.
IBM and Maximo are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide.