Batch Processing with pro Fit
Table of contents
Introduction
This TechNote explains how to do batch processing with pro Fit, i.e. how to process several files automatically.
The techniques described here can be used for various applications where a large number of data files must be processed in some way. You may want to combine the data of all files into a single file. Or you may want to open each file, calculate the average of each column, and create a new data file that contains the average values of all data files.
In this TechNote we discuss the following example:
You have a large number of files, each file containing the data of an exponential decay. You want to fit the data from each file to pro Fit’s built-in exponential function. The results of all these fits are to be collected in a new data file.
The techniques we present for solving this problem can easily be applied to any other type of batch processing.
This TechNote and all example files can also be downloaded as a zipped file (16 kB).
Methods for batch processing
There are three basic methods that can be used for batch fitting our data files in pro Fit:
- You open all the files by hand (select them all in the Finder and double-click). Then run a pro Fit script that goes through all open data windows and runs a fit for each of them.
- You write an Python script in pro Fit that runs through all the files. For each file, the script tells pro Fit to open it, then calls a pro Fit script for fitting and collecting the fit results, and then closes the file again.
- You write an Apple Script in Apple’s Script Editor that runs through all the files. For each file, the script tells pro Fit to open it, then calls a pro Fit script for fitting and collecting the fit results, and then closes the file again.
- You write an external module for cycling through all the files. The module opens each file in pro Fit, runs a fit, stores the fit results, and closes the file again.
In the following, we will discuss methods 1, 2 and 3. Method 4 requires an in-depth knowledge of MacOS programming, but is otherwise similar to methods 2 and 3 – we will therefore not discuss it in this TechNote.
In all the examples presented here, we will fit the data in each file to pro Fit’s built-in function “Exp” by varying the parameters “A”, “t0” and “const”. The parameter “x0” is to be set to 0 and held constant (because “A” and “x0” are mathematically redundant and they cannot be fitted simultaneously).
Method 1: Batch processing from a Pascal script within pro Fit
This method is easiest to script because the script does not have to find the files within the file system, but it rather assumes that you have already opened them within pro Fit. Its drawback lies in the fact that before running the batch process, you must manually open all files to be processed. You can easily open a large batch of files from the Finder: Select all the files to be opened, then hit command-O or drag the files onto the pro Fit icon.
The short script that follows does then all the work for us.
program MultipleFit; var wind, myDataWindow; windowCount; fResults: Object; begin NewWindow(type dataType); {open a new window for storing results} myDataWindow := FrontWindow; {save a reference to it} windowCount := 0; wind := NextWindow(myDataWindow); {get next window behind it} SelectFunction('Exp'); {the function we will fit} SetParameterProperties(parameter 2, mode inactive); {we don't want to fit x0} while (wind <> 0) do {cycle through all windows} begin if GetWindowProperty(wind, type) = dataType then {if data window} begin windowCount := windowCount+1; Writeln('Fitting window ', windowCount); SetColumnName(windowCount, GetWindowProperty(wind, name)); fResults := CurveFit(window wind, xColumn 1, yColumn 2); {run the fit} if FitResult(fitResultObject fResults, result nrFittedParameters) <> 0 then begin {if the fit was successful, get the results} data[1, windowCount] := FitResult(fitResultObject fResults, result fittedParameter, index1 1); data[2, windowCount] := FitResult(fitResultObject fResults, result fittedParameter, index1 2); data[3, windowCount] := FitResult(fitResultObject fResults, result fittedParameter, index1 3); data[4, windowCount] := FitResult(fitResultObject fResults, result fittedParameter, index1 4); end; end;{if} wind := NextWindow(wind); {go to next window} end; end;
The script first opens a data window to store the fit results. This window becomes the current data window. Then it cycles through all other presently open data windows, fits each one, and stores the results into the first data window.
To cycle through the data windows, the program uses the commands NextWindow and GetWindowType. NextWindow returns a reference ID for a window that lies behind another window. GetWindowProperty(wind, type) returns the window type, i.e. tells if a window is a data window, drawing window or text window. When the script finds a data window, it runs the fit (by calling CurveFit), and it retrieves the results calling FitResult.
Obviously, the same script could also be written as a Python script. For details, see the next section, where a Python script is used to retrieve the data files from the file system.
Method 2: Batch processing from a Python script
Method 1 has the disadvantage that you have to open all the files manually before running the program.
It might be more convenient to have a method that runs a fit over all files in a given folder. There is no way to do that from a Pascal script, but it can be implemented by means of a Python script. This is described in the following.
Open a new Function window, and paste the following script into it:
import os # the directory to iterate through, adapt this to your needs: dataDir = pf.ChooseDirectory() # select the function to fit and set parameter x0 to inactive pf.SelectObject(pf.GetFunctionObject(function = "Exp")) pf.SetParameterProperties(parameter = 2, mode = pf.paramInactive) # open a new data window for storing the results pf.NewWindow(type = pf.dataType) myDataWindow = pf.FrontWindow() windowCount = 0 # recusrively iterate through the directory for root, dirs, files in os.walk(dataDir): for name in files: # open the file p = os.path.join(root, name) print ('processing', p) pf.OpenFile(file = p) wind = pf.GetWindowObject(window = pf.FrontWindow()) #if the opened window is a data window if wind.type == pf.dataType: windowCount = windowCount+1 windowName = wind.name # fit the function, and close the opened window fResults = pf.CurveFit(window = wind, xColumn = 1, yColumn = 2) pf.CloseWindow(window = wind, saveOption = pf.dontSave) # get the results pf.SetColumnName(windowCount, windowName) if pf.FitResult(fitResultObject = fResults, result = pf.nrFittedParameters) != 0: pf.SetData(1, windowCount, pf.FitResult(fitResultObject = fResults, \ result = pf.fittedParameter, index1 = 1)) pf.SetData(2, windowCount, pf.FitResult(fitResultObject = fResults, \ result = pf.fittedParameter, index1 = 2)) pf.SetData(3, windowCount, pf.FitResult(fitResultObject = fResults, \ result = pf.fittedParameter, index1 = 3)) pf.SetData(4, windowCount, pf.FitResult(fitResultObject = fResults, \ result = pf.fittedParameter, index1 = 4)) else: pf.CloseWindow(window = wind, saveOption = pf.dontSave) print ('done')
This script looks similar to the Pascal script above, but instead of getting the data from the presently opened data windows, it brings up a dialog that allows choosing a directory, whereupon it iterates through all files in that directory (assuming that pro Fit can open them). For each data file, it runs the fit, just as in the first example.
Method 3: Batch processing from an Apple Script
Alternatively to using a Python script, you can also use an Apple Script for the same purpose.
To define the AppleScript, start Apple’s Script Editor, open a new AppleScript window, and enter the following:
-- bring up a dialog for selecting the folder set myFolder to choose folder with prompt "Choose a folder with data files:" -- create a list with all files in the folder set myFiles to list folder myFolder -- a list of files in myFolder set myFileCount to count myFiles -- the number of files in myFolder -- now start fitting with pro Fit tell application "pro Fit NAS" activate -- bring pro Fit to front set error alerts to false -- disable error reports within pro Fit make new table -- open a data window for storing our results set name of front window to "Result data" -- and set its name set globalData 0 to 0 -- this is our window counter repeat with i from 1 to myFileCount set theFile to item i of myFiles -- get the i-th file try open file ((myFolder as string) & theFile) as table -- open file write line "processing: " & theFile run program "SingleFit" -- run the program in pro Fit close window theFile saving no -- close without saving on error errText write line "cannot process: " & theFile & " (" & errText & ")" end try end repeat set error alerts to true -- enable error reports within pro Fit end tell
The above script first brings up a dialog where you can choose the folder that contains your files. Then it opens a new data window in pro Fit and sets its name to “Result data”. This data window will contain our results. Then it sets the variable globalData[0] to 0. This variable can be accessed by the program described below and is used as a file counter. Then the script tries opening each file of the folder as a data file. If this is successful, it runs a program called “SingleFit” that must be defined in pro Fit. This program is described below. Then the script closes the data window and goes to the next one.
Note that you can save your script as a “compiled script” from your script editor. The script can then be opened as any other pro Fit module using the “Load Module” command in the Customize menu. Or you can put it in the “pro Fit modules” folder. In this way the script is always accessible inside pro Fit and you don’t need anything else to do your batch processing.
Before running the script, you must define the program “SingleFit”. To do this, switch to pro Fit and define the following program:
program SingleFit; var windowCount; fResults : Object; begin windowCount := GetGlobalData(0) + 1; {increase window counter} SetGlobalData(0, windowCount); SetCurrentWindow(FrontWindow); {use data of front window for fit} SetFunctionParam('Exp', 1, 1); {set starting parameters} SetFunctionParam('Exp', 2, 0); {these values depend on} SetFunctionParam('Exp', 3, 1); {your model and data} SetFunctionParam('Exp', 4, 0); fResults := CurveFit(function 'Exp', xColumn 1, yColumn 2); {run the fit} SetCurrentWindow(GetWindowID('Result data')); {window for results} if FitResult(fitResultObject fResults, result nrFittedParameters) <> 0 then begin {if the fit was successful} data[1, windowCount] := FitResult(fitResultObject fResults, result fittedParameter, index1 1); data[2, windowCount] := FitResult(fitResultObject fResults, result fittedParameter, index1 2); data[3, windowCount] := FitResult(fitResultObject fResults, result fittedParameter, index1 3); data[4, windowCount] := FitResult(fitResultObject fResults, result fittedParameter, index1 4); end; end;
This program simply fits the data in the front window (which was opened by our AppleScript) and writes the fitting results to the window titled “Result data”.
It uses SetGlobalData/GetGlobalDate for storing the window counter so that the value can also be accessed from the AppleScript
Notes
- Both above examples set the starting parameters of the function Exp before fitting. The values used here may have to be varied depending on the data sets to be fitted.
- The pro Fit package comes with an Apple Script called “batch processing”. This script can be found in the folder “Examples:AppleScript” – it shows some other tricks that can be done with Apple Scripts. The use of an Apple Script might appear a bit clumsy at first sight because it requires writing the script in a separate application. But Apple Scripts are a very powerful tool, and once you get used to their weird syntax, there’s nearly nothing you cannot do with them.
- If you use an Apple Script often, you may want to add it to pro Fit’s Prog menu. To do this, save the Apple Script as a “compiled script” from your script editor, then switch to pro Fit and choose “Load Module” from the Customize menu to load the script into pro Fit. To add the script permanently to pro Fit, save it as a “compiled script” and place it into the folder “pro Fit Modules” in the pro Fit folder.
Further reading
pro Fit User Manual, Chapter 11 “Apple Script”