0 Votes

Batch processings

Last modified by Omid Abouali on 2026/09/14 15:10

Previous (Customized reports) | Back to Start | Next (None)

Introduction to Batch Processings

Batch processing is a method of processing and managing data or tasks in large quantities, or "batches," rather than individually. It is commonly used in various fields such as data processing, manufacturing, finance, and more. Batch processing offers several advantages, including efficiency, resource optimization, and the ability to process large volumes of data or tasks without constant manual intervention.

Key Concepts:

Batch: A batch refers to a collection of data or tasks that are processed together as a group. These batches can be composed of various types of data, transactions, or tasks.
Batch Jobs: A batch job is a specific set of instructions or a program that is designed to process a batch of data or tasks. It defines the operations to be performed on the data and the sequence in which these operations should be executed - For cluster usage mainly, not included in the present example.
Batch Processing System: This is a software framework or environment that facilitates the execution of batch jobs. It manages the scheduling, execution, and monitoring of batch jobs. Examples include Apache Hadoop, Apache Spark, and IBM InfoSphere DataStage - For cluster usage mainly, not included in the present example.

Parallel and Serial Tasks in Batch:

In the context of batch processing, tasks can be organized and executed in two main ways: parallel processing and serial processing (also known as sequential processing). Each approach has its own advantages and considerations based on the specific requirements of the batch job. Let's explore both parallel and serial tasks in batch processing:

Multiple tasks are executed simultaneously in parallel processing, taking advantage of available system resources to speed up overall processing time. Each task is independent of the others and can be processed concurrently. Parallel processing is particularly useful when the tasks are not dependent on each other and can be divided into smaller, parallelizable units.
Serial Tasks (Sequential Tasks): In serial processing, tasks are executed one after the other in a sequential manner. Each task depends on the successful completion of the previous task. This approach is suitable when tasks are interdependent and the output of one task serves as input to the next.

Relevance to Present Example:

In serial runs, each task refers to an opening of one IDA Tunnel.exe and performs saving/calculations. Each single task uses the full power of your computer, whilst other tasks wait for its completion.
In parallel runs, multiple tasks will run simultaneously with each on a single core/processor of your computer.

Steps:

Step 1: Setup the API:

First, you need to set up the API. For this, you need to find the file util.py and set the variable path_to_tunnel to point to the bin folder in your IDA Tunnel installation (or another folder if you have IDA Tunnel installed elsewhere):

path_to_tunnel = "C:\\Program Files (x86)\\IDA Tunnel <YOUR IDA VERSION>\\bin\\"

Note 1: Replace <YOUR IDA VERSION> with your IDA Tunnel version, e.g., 2.0 or 2.0 beta1.

Note 2: If you use a Windows system, always use double backslashes \\ for specifying paths.

Save the util.py after changing it.
Second, you also need to go to Services in Windows and make sure that the IDAMessageBrokerService is running. You can find Services by simply searching it in the Windows search bar.

Step 2: Looking at the example Python script

Before running the script, you can choose to either use our provided tunnel model named 1.IDM (in folder "example\\tunnels") or prepare an example system for the report yourself. For the latter, open IDA Tunnel and create a new system. Save the system in our provided folder, and make sure the system name is 1.IDM.
Copy and paste the 1.IDM several times, rename them as 2.IDM, 3.IDM, etc. The number of copies depends on the number of tasks (Ntasks) you want to execute in parallel or in serial.
Now you can open the script Batch_processing.py. Find and set the variable file_path to point to the folder as,

file_path = "C:\\<USER DIR>\\example\\Batch processings\\tunnels\\"

Note 3: Replace <USER DIR> with your user directory where the downloaded examples are saved to.

Define file_paths which points to all the systems, i.e., 1.IDM, 2.IDM, 3.IDM, etc. The number of readings will depend on the number of tasks (Ntasks) you specify.

file_paths = [file_path + str(i ) + ".IDM" for i in range(1, Ntasks + 1)]

Define result_paths that point to the systems you will be saving your results to. With the following line, the same file_path will be used for saving, but the names of the saved systems will be result1.IDM, result2.IDM, etc.

result_paths = [file_path + "result" + str(i ) + ".IDM" for i in range(1, Ntasks + 1)]

Note 4: The defined file_paths and result_paths are for the convenience of running our script in parallel and serial manners. You can explicitly define your own paths as, e.g., file_paths = [".\\example\\Batch processings\\tunnels\\a.IDM", ".\\example\\Batch processings\\tunnels\\b.IDM" ]

To run the script, open the command prompt terminal by searching cmd in the Windows search bar.

Then go to IDA Tunnel distributed python directory by typing:

>> cd "C:\\Program Files (x86)\\IDA Tunnel <YOUR IDA VERSION>\\bin\\python"

Step 3: Parallel and Serial Tasks

Now you can run the Batch_processing.py script by typing:

>> python.exe "C:\\<USER DIR>\\example\\Batch processings\\Batch_processing.py"

The given mode = "serial", Ntasks = 2 will start IDA Tunnel tasks in a consecutive (serial) manner. The task to perform in this example is (a) reading in systems one after another until all systems are opened; (b) starting from the first window: showing zones containing windows facing north, then saving the system.
No optimization will be performed in this tutorial because starting multiple IDA Tunnel instances for simulation with just a single license is not permitted.
Let the script finish, then change mode = "parallel" and rerun the Batch_processing.py
This time, 2 IDA Tunnel tasks will execute simultaneously (parallel) to read the systems, find windows facing north, and then save the systems. (Note: If there is no pop-out IDA Tuunel window, the only workaround is to use The Conda Python Distribution, see Introduction to Python)

Important: Make sure you have sufficient number of .IDM files for trying serial/parallel use of IDA Tunnel API. For instance, if your Ntasks = 10, you should have 10 IDM files available for reading (1.IDM, 2.IDM to 9.IDM). However, you can avoid creating multiple .IDM files by explicitly claiming your own path of reading.

Previous (Customized reports) | Back to Start | Next (None)