2 The automatic processing flowchart for UAV image

Figure 2. The automatic processing flowchart for UAV image

In the flowchart, each step corresponds to a function in UAV modules as follows:

(1) Importation of orientation parameters from third party software. Currently in MASI Version 6.0 there is not a function of aero-triangulation for UAV images, but the results of aero-triangulation from mainstream software can be used in MASI software. Clicking the icon showing “Orientate” in the main interface,

The following menu will pop up.

Please use the program, i.e., importFromPix4d.exe, if users need to directly import the results of aero-triangulation from Pix4D software; please use the program, i.e., importFromPhotoScan.exe, if users need to directly import the results of aero-triangulation from PhotoScan software; please use the program, i.e., importFromCC.exe, if users need to directly import the results of aero-triangulation from ContextCapture software. Others’ results, e.g., from Inpho software, can be used in MASI software directly.

(2) Undistortion (optional for some cases). UAV image is usually captured by the calibrated civil camera. Thus, the undistortion operation is demanded to remove the distortion caused by lens, which is convenient for the further processing. If the image is undistorted, the step is discarded. For instance, the output undistorted images from Pix4D software (stored in the path: ..\project name\1_initial\images\undistorted_images) is images after undistortion. If they are used in MASI, the step does not need. If in the first step the orientation results from Pix4D software are imported into MASI, the undistorted images from Pix4D software must be used. Or the programs, undistortForPix4D.exe and undistort4Pix4DForm.exe, are used to undistort UAV images according to calibrated parameters exported from Pix4D software, the calibrated model adopted in the programs is the same as that of Pix4D software. The undistorted results for UAV images are also the same as those undistorted results exported from Pix4D software. But, the module in MASI software supports more data types and more bands, and supports hyper-spectral image very well. For more details, please refer to user manual. The case is similar for usage of the results of aero-triangulation from PhotoScan software. The undistorted UAV images from PhotoScan software should be used, or UAV images are undistorted by the lensCorrection.exe provided by MASI software.

Please ensure that the distortion model used in the course of calibrating the camera is the same as the model used in undistortion modules of MASI software. The sameness is a prerequisite. Currently in MASI software two calibrated models are supported for undistortion. One is called General model, which corresponds to the programs, lensCorrection.exe and lensUndistortForm.exe. The coordinate definition and calibrated model adopted in the program, lensCorrection.exe, is the same as that of Australis software (i.e., 10-parameter model employed in digital close-range photogrammetry). Nowadays in mainland China, most cameras mounted on UAV and middle format cameras are calibrated by using the model. Thus, images captured by these pre-calibrated cameras can be undistorted by the module. The model is also suitable for the case where the self-calibrated camera parameters are exported from PhotoScan software (the results in the form of Australis software are saved). The other model supported in MASI software (via the program, undistortForPix4D.exe) is the same as that of Pix4D software. Thus, the calibrated parameters exported from Pix4D software can be used in the model and the undistortion function is also the same as that of Pix4D software.

There are two ways to incorporate third party software with MASI software. One is that the undistorted images and orientation results both from the same third party software are imported into MASI software. The other is for the third party software without undistortion function. In this way, undistortion is carried out firstly in MASI software (ensure the sameness of the distortion model in calibration as the model used in MASI), the undistorted images are then used in the third party software for aero-triangulation. After aero-triangulation, the orientation results from the third party and the distorted images are used in the following steps, such as multi-view stereo and DSM generation, and ortho-rectification.

Clicking the icon showing “Undistort” in the main interface,

The following menu will pop up.

(3) Automatic multi-view stereo and DSM generation. For each image in a UAV block, the program selects, in these images which are overlapped with the image, some suitable images to fulfill multiple view stereo and intersection of 3D points together with the image. And then, the program tries to generate dense point clouds for each image (users can decide whether the text-formatted point clouds files with extension .xyz are saved by setting the configuration option). These point clouds are transformed to DSMs and these DSMs are mosaicked. At last, a mosaicked DSM covering the whole UAV block is generated (the DSM before interpolation). Clicking the icon showing “DSM” in the main interface,

The following menu will pop up.

The command item, Easily used Model, in the above menu is to call the function of multi-view stereo and DSM generation. After processing, a mosaicked DSM (the DSM before interpolation) is generated.

Please note that the input images must be undistorted image and the program can only deal with the image whose pixels is less than 60 million. If images’ pixels are more than 60 million, aerial modules should be used or new modules should be customized by contacting us.

(4) Post processing for DSM: The above mosaicked DSM has some areas which have not height value (before interpolation these areas are filled by using no-data values). In this step, these areas with no-data value are interpolated, and outliers (or small spots of errors) in the resultant DSM after interpolation are removed. The output of this step is the product of dense DSM, which is generated by this software. The DSM product can be used in different applications followed (e.g., volume calculation, finding change of surface height, extraction of the height of building / tree). The command item, Post Processing, in the above menu is to call the function of post processing for DSM.

(5) Automatic true ortho-rectification: The program uses the DSM source generated in Step 4 to ortho-rectify all images in a UAV block. For each image, an ortho-rectified image is generated. These ortho-rectified images are mosaicked and an ortho-mosaic image covering the whole UAV block is generated. An alternative way is that DTM as a height source, which can be from third party or generated by MASI software, is used to ortho-rectify images. This way is more common. Because UAV images have high overlapping ratios, DSM generated by MASI software is highly dense and has high quality. We highly recommend that the highly dense DSM is used to fulfill true ortho-rectification. Clicking the icon showing “True Ortho” in the main interface,

The following menu will pop up.

The command item, UAV Ortho, in the above menu is to call the function of ortho-rectification. After processing, an ortho-mosaicked image is generated.

(6) Comparing surface height and change finding. By comparing DSMs of different phases (generated by MASI software), the program can automatically find change of the surface height and calculate the value of changed height. It can be used in the automatic finding of new buildings, the unplanned buildings and the removed buildings, and estimating their corresponding accurate height. The map of height difference produced in this step can be used in the following extraction of building attributes (the center position of building, height of building, number of layers, area of ground, and construction area of building). Clicking the icon showing “Surface Change” in the main interface,

The operating interface of the function will be launched.

(7) Automatic DSM2DTM: The program deletes the non-grounded objects like buildings, trees in DSM and transforms DSM to DTM where the pure ground height is saved. Clicking the icon showing “DSM to DTM” in the main interface,

The operating interface of the function will be launched.

(8) Height estimation of building / tree. The height difference between DSM and DTM (nDSM) is the height of building / tree. The height difference (raster form) produced in this step can be used in the following extraction of building attributes (the center position of building, height of building, number of layers, area of ground, and construction area of building). DSM is generated by the functions in MASI software (including post processing for DSM), and DTM can be produced in the step 7 in this flowchart. Clicking the icon showing “Building height” in the main interface,

The operating interface of the function will be launched.

(9) Applications. The generated dense DSM, map of height difference, nDSM (i.e., height of building / tree) in above steps can be passed to the application functions which fulfill volume calculation and extraction of building attributes (the center position of building, height of building, number of layers, area of ground, and construction area of building). Clicking the icon showing “App” in the main interface,

The following menu will pop up.

Currently two applications modules are included, i.e., volume calculation and automatic extraction of building attributes in the light of buildings footprints. The command item, Calculate Volume, in the above menu is to call the function of volume calculation. The GUI program called from the command item is calVolumeForm.exe. Also, the GUI program can be launched by double-clicking the program directly. For more details, please refer to “Usage of the calVolume.exe and calVolumeForm.exe programs”. The command item, Extract Buildings’ Attributes, in the above menu is to call the function of extraction of building attributes. The GUI program called from the command item is extBldgAttributesForm.exe. Also, the GUI program can be launched by double-clicking the program directly. For more details, please refer to “Usage of the extBldgAttributes.exe and extBldgAttributesForm.exe programs”. In the course of volume calculation and automatic extraction of building attributes, the bounds in the form of polygon are needed. Users can click the command item, Polygons Drawing Tool, in above menu to launch the interactive tool, collectPolygons.exe, to collect polygons and save them.

(10) 3D modeling with true color textures. Images and highly dense DSM generated in previous steps are used to generate 3D TIN models with true color textures. The 3D models are in a form of tiled mesh. The data type of the images which are used for texture mapping is unsigned 8 bit integer. If it is not, user should stretch the images to unsigned 8 bit using the autoStretch.exe program provided by MASI software. Clicking the icon showing “Textured 3D Model” in the main interface,

The following menu will pop up.

The command item, textured 3D modeling, in the above menu is to call the function of 3D modeling. The GUI program called from the command item is aerialScene3DForm.exe. Also, the GUI program can be launched by double-clicking the program directly. For more details, please refer to “Usage of the aerialScene3D.exe and aerialScene3DForm.exe programs”.

In addition, other functions are included: image displaying, interactively editing of DSM/DEM, collecting polygons, automatic mosaic (mosaic of DSMs, mosaic of ortho-rectified images, mosaic of histogram matched images), transforming RGB image to grey image, rotation of image, reflection of image, transforming point clouds to surface in the form of raster, transforming DSM with raster form to point clouds, image cropping, creating overviews for image.

The method of format transform for point clouds generated by MASI software: the file of point clouds generated by the software is text format, which can be transformed to LAS format through the following steps. In the course of transform, a third party software package OSGeo4W (including the open source library, i.e., libLAS) is exploited. The package can be downloaded from the official website of OSGeo4W. The steps are as follows:

(1) Translation to LAS format: the command, namely txt2las.exe (OSGeo4W package includes the program), in the open source library, libLAS, can be used to translate the point clouds with text format to the LAS file. The corresponding command is as follows:

>txt2las.exe -parse xyz -i filename.xyz -o filename.las

(2) Assigning RGB values: the command, namely las2las.exe (OSGeo4W package includes the program), in the open source library, libLAS, can be used to assign the RGB values from truly ortho-rectified image to the points in LAS point clouds file. The colored point clouds are obtained. One example of the command is as follows:

>las2las.exe -i points.las --color-source ortho.tif -o points_rgb.las --file-format 1.2 --point-format 3 --color-source-scale 256 --color-source-bands 1 2 3

3 The flowchart sample of UAV images

In the following, the flowchart will be illustrated through a sample. First, the main interface of the UAV modules is launched via the Windows Start menu, i.e., calling the program, uavMain.exe. The launched main interface is as follows:

Clicking the icons in the main interface is to call the corresponding functions. The main steps of the flowchart sample are as follows:

Step 1: Importation of orientation parameters from third party software

Clicking the icon showing “Orientate” in the main interface,

The following menu will pop up.

(1) Using the results of aero-triangulation from Pix4D software

It is assumed that the results of aero-triangulation from Pix4D software are imported. If the results of aero-triangulation are from other software, please refer to the coming description. Clicking the button of “import from Pix4D” in the above menu is to trigger the action of importation. The operating interface is as follows:

In the section of input setting, the files outputted from Pix4D software are selected, including the camera file and the file of orientation parameters defined in Pix4D software. The two files are located in the direcotry, ..\project name\1_initial\params\. In the section of output setting, the names of transformed files, i.e., the file of interior parameters of camera and the list file of exterior elements defined and used in MASI software, are set.

After files are transformed correctly, the dialog box will pop up.

Defining of each line in the file (the definition of the underlying coordinate system is outlined in Appendix III of user manual):

Line 1 is the X coordinate of principal point in image coordinate system;

Line 2 is the Y coordinate of principal point in image coordinate system;

Line 3 is the X coordinate (Unit: mm) of principal point in camera coordinate system (the center of the camera as the original point in the coordinate system);

Line 4 is the Y coordinate (Unit: mm) of principal point in camera coordinate system (the center of the camera as the original point in the coordinate system);

Line 5 is the focal length of the camera (Unit: mm);

Line 6 is the pixel size of the camera (Unit: mm).

The content of the list file of exterior elements is as follows:

Each line corresponds to exterior elements of an image, and these lines are arranged in sequence. First element is the image ID which is followed by six exterior elements (Attention: the first element, image ID, do not include extension of image file). The six elements are three linear elements and three angle elements. The order of these exterior elements is first linear elements then angle elements, and rotation system is omega-phi-kappa. The unit of rotational angle is degree.

Because the results of aero-triangulation from Pix4D software will be used in MASI software, the undistorted images exported from Pix4D software should be adopted. Or the programs, undistortForPix4D.exe and undistort4Pix4DForm.exe, provided by MASI software, are used to undistort the images according to calibrated parameters exported from Pix4D software. The undistortion function is the same as that of Pix4D software. For more details, please refer to user manual. The undistorted images, which are saved in the direcotry, ..\project name\1_initial\images\undistorted_images, will be directly used in multi-view stereo and DSM generation, and ortho-rectification. If the directory does not exist, the following operation can be used to generate the undistorted images:

Clicking the menu, Process -> Save Undistorted Images,

$说明: D:\dev\manuals_program\MASI_v3_manual\pictures\undistorted.PNG$

(The figure is from the official website of Pix4D software. For more details, please refer to the technical support articles on the website)

(2) Using the results of aero-triangulation from PhotoScan software

If the results of aero-triangulation from PhotoScan software need to be used in MASI software, firstly the camera parameters and the camera positions (i.e., orientation) should be exported from PhotoScan software as well as the undistorted images via the following operations:

Exporting camera parameters:

Clicking the menu, Tools -> Camera Calibration,

The following interface will pop up, and then the tab “Adjusted” is selected.

Clicking the saving button in the right side:

The “Save As” dialog box will pop up as shown as follows:

$说明: C:\Users\yang\Pictures\photoscan\saving_camera_adjusted.PNG$

The type “Agisoft Camera Calibration (*.xml)” is selected in Save as type. After input the file name, click the button of Save. A XML file with .xml extension is generated. The content of the file is as follows:

$说明: C:\Users\yang\Pictures\photoscan\agisoft_photoscan_camera_xml.PNG$

Exporting camera positions (orientation):

Clicking the menu, File -> Export -> Export Cameras,

First, the dialog box to select coordinate system will pop up.

$说明: C:\Users\yang\Pictures\photoscan\export_ext_setting.PNG$

The coordinate system is the one which is used to depict camera positions for each photo (i.e., the three linear elements in the exterior elements). Since MASI software only supports the cases where the exterior elements can be depicted by local rectangular coordinate system or projected coordinate system, here the projected or local coordinate system can only be selected as coordinate system, e.g., UTM, TM or the defined local rectangular coordinate system. After the OK button is clicked, the “Save As” dialog box will pop up as shown as follows:

The type “Omega Phi Kappa (*.txt)” is selected in Save as type. After input the file name, click the button of Save. A text file with .txt extension is generated. The file includes the exterior elements of photos and will be used in the coming operation of importing.

After camera parameters and camera positions (orientation) are exported from PhotoScan software, the coming step is to export undistorted images. Clicking the menu, File -> Export -> Undistort Photos,

The setting dialog box will pop up:

The setting options are configured as the above figure. After the OK button is clicked, the undistorted images are generated. The undistorted images generated in this step will be used in the following multi-view stereo and DSM generation and ortho-rectification in MASI software. Or UAV images are undistorted by the lensCorrection.exe provided by MASI software. But the calibrated camera parameters to be used in MASI software must be the form used in Australis software. The camera parameters can be exported from PhotoScan software via the menu, Tools -> Camera Calibration, as described previously. But in the “Save As” dialog box, the type “Australis Camera Parameters (*.txt)” is selected in Save as type. Please also see the following section. For more details, please refer to user manual of PhotoScan software.

After finishing the above exporting operations from PhotoScan software, users then call the corresponding importing operations provided in MASI software. Clicking the button of “import from PhotoScan” in the above menu is to trigger the action of importation. The operating interface is as follows:

In the section of input setting, the files exported from PhotoScan software are selected, including the camera file and the file of camera positions defined in PhotoScan software. In the section of output setting, the names of transformed files, i.e., the file of interior parameters of camera and the list file of exterior elements defined and used in MASI software, are set. The setting is shown as the above figure. After the execute button is clicked, the transformation will carry out.

After files are transformed correctly, the dialog box will pop up.

The output files, the file of interior parameters of camera (interiorfile.dat) and the list file of exterior elements (oriList.dat) can be directly used in the coming steps, such as multi-view stereo and DSM generation, and true ortho-rectification. The formats of the resultant files are described as the previous sections. The format of exterior file: the order of the six exterior elements after image ID is first linear elements then angle elements, and rotation system is omega-phi-kappa. The unit of rotational angle is degree.

(3) Using the results of aero-triangulation from ContextCapture software

If the results of aero-triangulation from ContextCapture software need to be used in MASI software, firstly a file with BlocksExchange XML format should be exported from ContextCapture software as well as the undistorted images via the following operations:

Clicking the menu, Block -> Export -> Export block, users should select the output format, output file and options; and click on Export block to create the XML file.

说明: https://docs.bentley.com/LiveContent/web/ContextCapture%20Help-v17/en/GUID-CD27AAEA-E9E4-4B58-844F-F45E82B50A83-low.png

(This interface is the possible pop-up dialog, the setting on the dialog is not suitable for MASI software, please see the following description)

Output format: BlocksExchange XML format should be selected.

Output file: the filename set for the exported XML file

The setting of options is as follows:

Spatial reference system: select the coordinate system used to write 3D positions and rotations. Only local rectangular coordinate system or projected coordinate system should be selected in order to be used in MASI software.

Rotation format: select how rotations are written, Rotation matrix or Omega, Phi, Kappa angles. Only Omega, Phi, Kappa angles should be selected in order to be used in MASI software.

Camera orientation: select the axis convention for rotations. X right, Y up should be selected in order to be used in MASI software.

Include automatic tie points: include automatic tie points in the export. Users can select No for this option (or uncheck this option).

Export photos without lens distortion: photos are undistorted according to the block's distortion data and exported to JPG files in a sub-directory. Users should select Yes for this option (or check this option) in order to be used in MASI software.

After finishing the above exporting operations from ContextCapture software, users then call the corresponding importing operations provided in MASI software. Clicking the button of “import from ContextCapture” in the above menu is to trigger the action of importation. The operating interface is as follows:

In the section of input setting, the file with BlocksExchange XML format, exported from ContextCapture software, is selected. In the section of output setting, the names of outcome files, i.e., the file of interior parameters of camera and the list file of exterior elements defined and used in MASI software, are set. The setting is shown as the above figure. After the execute button is clicked, the transformation will carry out.

After files are generated correctly, the dialog box will pop up.

Because in ContextCapture software, photos grouping is supported. All photos taken using the same physical camera, with identical focal length and dimensions must be gathered in a photo group. The file with BlocksExchange XML format may include multiple photo groups. The importing function provided in MASI software can generate a pair of files (i.e., the file of interior parameters of camera and the list file of exterior elements) for each photo group in the file with BlocksExchange XML format. The generated files for each group are named in the following rules: the filename set by users ends with _X, where X is the serial number of group (1 based). For instance, in the above importing setting, the outcome files, interiorfile_1.dat (the file of interior parameters of camera) and oriList_1.dat (the list file of exterior elements) are generated for the first photo group.

The output files for each photo group, i.e., the file of interior parameters of camera and the list file of exterior elements, can be directly used in the coming steps, such as multi-view stereo and DSM generation, and true ortho-rectification. The formats of the resultant files are described as the previous sections. The format of exterior file: the order of the six exterior elements after image ID is first angle elements then linear elements. The order is different from the cases of Pix4D and PhotoScan. Please be careful. For more details, please refer to user manual of MASI software. Rotation system is omega-phi-kappa. The unit of rotational angle is degree.

(4) Using the results of aero-triangulation from other softwares

The results of aero-triangulation from other software, such as, Inpho, PhotoMOD can be used in MASI software directly. The exterior elements from different software may have different formats in terms of order between linear and angle elements, rotation system (or the rotation system is indicated by rotation matrix directly), and the unit of angle, comparing with the above cases. Users only need set the provided three options, in the light of the real case of the exterior elements, in the coming steps, such as multi-view stereo and DSM generation, and ortho-rectification (please refer to the coming steps), then the results of aero-triangulation can be used directory.

Step 2: Undistortion (in this sample, this step does not need to be carried out in MASI software)

This step does not need to be carried out in MASI software. The undistorted images from Pix4D software (stored in the path: ..\project name\1_initial\images\undistorted_images) are exploited. Because in the first step the orientation results from Pix4D software are imported into MASI and will be used in the coming steps, the undistorted images from Pix4D software must be used. Or the programs, undistortForPix4D.exe and undistort4Pix4DForm.exe (called under the menu, Undistort -> Undistort for Pix4D), are used to undistort UAV images according to calibrated parameters exported from Pix4D software, the calibrated model adopted in the programs is the same as that of Pix4D software. The undistorted results for UAV images are also the same as those undistorted results exported from Pix4D software. But, the module in MASI software supports more data types and more bands, and supports hyper-spectral image very well. For more details, please refer to user manual.

If the orientation results from PhotoScan software are used in MASI software, the undistorted images from PhotoScan software must be adopted. Please refer to Step 1 to learn how to export undistorted images from PhotoScan software. Or UAV images are undistorted by the programs, lensCorrection.exe and lensUndistortForm.exe (called under the menu, Undistort -> General Undistort), provided by MASI software. But the calibrated camera parameters to be used in MASI software must be the form used in Australis software, and they are the self-calibrated camera parameters exported from PhotoScan software. The camera parameters can be exported from PhotoScan software via the menu, Tools -> Camera Calibration, as described previously. But in the “Save As” dialog box, the type “Australis Camera Parameters (*.txt)” is selected in Save as type. The values of self-calibrated camera parameters in the saved file can be used for the parameters setting in the GUI program, lensUndistortForm.exe. For more details, please refer to user manual of PhotoScan software.

If the orientation results from ContextCapture software are used in MASI software, the undistorted images from ContextCapture software must be adopted. Please refer to Step 1 to learn how to export undistorted images from ContextCapture software. Or the programs, undistortForPix4D.exe and undistort4Pix4DForm.exe (called under the menu, Undistort -> Undistort for Pix4D), are used to undistort images according to calibrated parameters which are extracted from the BlocksExchange XML format file exported from ContextCapture software. The extraction is fulfilled by the program, fetchDistortFromCC.exe (called under the menu, Undistort -> Extract distortion parameters from ContextCapture), provided by MASI software. The operating interface is as follows:

The program, fetchDistortFromCC.exe, can generate a file of distortion parameters for each photo group in the file with BlocksExchange XML format. The generated file for each group is named in the following rules: the filename set by users ends with _X, where X is the serial number of group (1 based). For instance, in the above extraction setting, the outcome file, camMASI_1.cam, is generated for the first photo group.

The outcome files (i.e., files of distortion parameters) can be directly used in the programs, undistortForPix4D.exe and undistort4Pix4DForm.exe (called under the menu, Undistort -> Undistort for Pix4D). Images are undistorted according to calibrated parameters indirectly exported from ContextCapture software, the calibrated model adopted in the programs is the same as that of ContextCapture software. The undistorted results for images are also the same as those undistorted results exported from ContextCapture software. But, the module in MASI software supports more data types and more bands, and supports hyper-spectral image very well. For more details, please refer to user manual.

If a third party software has not the function of undistortion, the following operations can be adopted. Firstly, undistortion is carried out in MASI software (ensure the sameness of the distortion model in calibration as the model used in MASI), the undistorted images are then used in the third party software for aero-triangulation. After aero-triangulation, the orientation results from the third party and the distorted images are used in the coming steps, such as multi-view stereo and DSM generation, and ortho-rectification.

Step 3: Automatic multi-view stereo and DSM generation

Clicking the icon showing “DSM” in the main interface,

The following menu will pop up.

By clicking the command item, Easily used Model, in the above menu, the GUI program is launched. After processing, a mosaicked DSM (the DSM before interpolation) is generated. The graphical interface of the program is as follows:

First, all files required in the course of multi-view stereo and generation of DSM, such as images, the file of interior parameters of camera and the list file of exterior elements (images are the undistorted images from Pix4D software, the file of interior parameters of camera and the list file of exterior elements are obtained in Step 1) are copied to a directory, e.g., H:\uav_tests\MASI, and set it as work directory. When MASI graphical interface is used, all required files should be put in the same directory and the directory is set as work directory.

Then, the file of interior parameters of camera and the list file of exterior elements are selected, respectively, in the section of input files in the interface. In the section of configuration options, Extension of image means the adopted format of images. It can be TIFF or JPEG, which are commonly used format for UAV image. The following three options should be set according to the real case of the exterior elements. For the orientation parameters from Pix4D software (or PhotoScan software), the order of these exterior elements is first linear elements then angle elements, and rotation system is omega-phi-kappa. The unit of rotational angle is degree. Thus, the selected values of the three options are shown in the above figure of the operating interface.

If the results of aero-triangulation from other software, such as, Inpho, PhotoMOD, PixelGrid, are used, users should set the provided three options in the interface (Order of exterior elements, Rotation system, and Unit of angle), according to the real case of the exterior elements. The different combination of the values of three options provided in this software can support almost all formats of exterior elements from the available aero-triangulation software. For more details, please refer to the sections of multi-view stereo and DSM generation for UAV images in user manual.

Height limit has two choices: setting maximum and minimum heights from right side, and selecting third part elevation file (The coordinate system of elevation file must be the same as that of exterior elements), respectively. If users choose to set height limit from right side, the maximum and minimum heights are values of surface height (including height of the man-made objects) of the highest and lowest points above sea level in the UAV block. The height value above sea level is based on the adopted coordinates system. The maximum and minimum height value of elevation can be extended, to some extent, i.e., increasing the maximum value a little bit and decreasing the minimum value a little bit.

Rotation or not first: whether image should be rotated in advance. For some cases, a specific rotation is required: the layout of image that is used to determine orientation of image is different from the layout of image which is saved in computer and used as input of the program. There are four cases: none rotation, rotation of clockwise 90 degree; rotation of anti-clockwise 90 degree, and rotation of 180 degree. In this sample, no rotation is needed in advance. If the exterior elements with PATB format exported by Inpho software are adopted, the images before multi-view stereo may need to be rotated 180 degree because the directions of coordinates defined in Inpho software are different. Users can set to rotate images 180 degree via the option.

Ratio of image size: the ratio of the size of original image to the size of matched image. The size of matched image can be the same as the original image, or can be reduced in the light of some specific scale. Two cases are supported in the software: 1 means the size of matched image in each dimension is the same as the original image, while 2 means the size of matched image is the 1/2 size of the original image in each dimension. Reduction of size in the course of multi-view stereo can lead to reduction of computational requirement and of the amount of data, and also accelerate the process. But it also leads to the loss of precision. Because multi-view stereo is adopted, the loss of precision is relatively small. Users can select the value in the light of specific requirements. In most cases, the default values can be used.

Delete intermediate files or not: users can determine whether the intermediate files (epipolar and disparity images) are deleted by setting the option. The epipolar images can be used in the stereo displaying and further in the line-drawing in the stereo displaying.

Method to select height value: the method how to select the height value when there are multiple 3D points in a grid. The software supports three methods: max, min, average, which select maximum, minimum, average height value, respectively. The recommended method is max.

Overlapping ratio: the program can flexibly deal with different cases where different users may acquire UAV images with different overlapping ratios. Two choices are provided: high or low. If the flight / side overlapping ratio is above 70/60, high can be selected; while the flight / side overlapping ratio is about 60/30, low should be selected.

Save point clouds or not: users can determine whether point clouds generated from each image are saved as a text-formatted file with extension .xyz. The default setting is no-saving.

Partitions and Part ID: The two parameters are used for distributed computing on multiple machines. The program, mvsNew.exe, can equally partition all images of the whole block into different parts, in the light of the number of used computers. Each part corresponds to a computer and is identified by a serial number. The serial number is 0-based and integer type, its range is 0 ~ (Partitions - 1). For the default setting in this form, partitions are 1, which means only one computer is used for computation. All images are treated as one part and are processed on this machine whose part ID is 0. Attention: MASI software need to be installed on all computers used for computation and it should be ensured that all used files including images, the list file exterior elements of the whole block, the file of interior parameters of the camera and the configuration file are accessed by each machine.

Number of used threads per processor: setting the number of used threads (CPU cores or the virtual CPU cores by using hyper-threading technology) per processor. For the cases where CPU cores are enough, we suggest that number of 4, 6 or 8 is set here.

Bounding polygon: users can determine whether a bounding polygon is used to limit the region of multi-view stereo. After checking the checkbox, a shape format file which includes a polygon for bounding should be selected. Compared to the region bounded by the used polygon, the resultant region after the utilization of bounding polygon in multi-view stereo may be extended a little bit. The bounding polygon file can include only one polygon and its coordinate system must be the same as that of exterior elements. For default setting, bounding polygon is not used. The button of “Selecting File” in the right side is invalid.

After the above options in the interface are set, users can click “Save configuration” to save these options. These configuration options will be saved in a configuration file, uavDSM.conf, and next time the options can be loaded from the configuration file by clicking “Load configuration”. The configuration file, uavDSM.conf, is also stored in the work directory. If “Execute” is clicked, the configuration file will also be generated in the same way.

In the output setting, users should set the filename of the mosaicked DSM (the DSM before interpolation), the grid spacing (cell size) of the raster file used to store height value (the value of grid spacing can not be better than the actual ground sample distance, GSD, of image), the no-data value used to fill in the area without height value in the raster DSM file (no-data value).

Because the computational requirements for multi-view stereo matching are large, the software supports parallel computing to leverage advances of the multi-core computer. Serial or parallel processing can be selected. If parallel processing is selected, number of processors should be inputted as well as the current user name using the Windows OS and password of the user. Next time users can load the information by clicking the load button, and do not need input the information again. Users should be very careful when selecting the number of processors, which should be determined in the light of the number of available CPU cores in the adopted machine, the size of physical memory and the size of image frame. There are three points for selecting the number of processors. First, the product of the value set in the option, Number of used threads per processor, and the number of used processors here can not exceed the total number of CPU cores (or the virtual CPU cores by using hyper-threading technology). Second, all consumed memory can not exceed the physical memory on the machine. In general, 3 G - 8 G memory is consumed in one processor. The total number, i.e., multiplication of the number of processors by the memory consumed in a processor, can not exceed the physical memory on the machine. Third, requirements of disk reading and writing brought by multiprocessor processing can not exceed the disk capability provided by the machine. User can get the information of consumed memory from Task Manager of Windows OS, and the information of busy intensity of disk reading and writing and its speed from Resource Monitor of Windows OS. Users who want to use parallel computing in the software should be trained for a period of time, and know the software and the principle behind the software more. At the beginning, ordinary users are not expected to use parallel computing. In the course of using parallel computing, if there are any problems encountered, please contact us as soon as possible.

After finishing the setting of configuration options, input and output setting, users can click the button of “Execute”. The program starts to run and the required time to finish the whole procedure is different according to the size of the UAV block. In the course of computation, the buttons of “Execute” and “Cancel” on the bottom of the form is invalid. The two buttons can not be clicked in case they are clicked many times in the course of running. Users can learn the progress and the running status via the re-directed output printing (sometime printing may delay) on the right side. User can also check the information of resources (CPU, memory and Disks) possessed by the program from Task Manager and Resource Monitor of Windows OS. Once the execution of mvsNew.exe is finished, a message box will pop up showing that the processing is closed successfully or terminated (aborted). After clicking OK button on the message box, the GUI program returns to the available status. Users can either click the button of “Cancel” on the bottom or click the closing icon on the up-right corner of the interface to close this GUI program. Or users can select and set new input and output to fulfill a new task.

User can click the button of “Terminate” on the bottom if they need to abort the processing (not the closing of GUI form). The GUI program, uavDSMForm.exe, will kill the process of mvsNew.exe which is being called. Killing the process of mvsNew.exe may delay (less than one second or several seconds). Once the process is killed, a message box will pop up showing that the processing is terminated (aborted) or closed successfully. After clicking OK button on the message box, the GUI program returns to the available status. Users can either click the button of “Cancel” on the bottom or click the closing icon on the up-right corner of the interface to close this GUI program. Or users can select and set new input and output to fulfill a new task.

Attention: the button of “Cancel” on the bottom is only for closing the GUI program. If the called logical program (i.e., mvsNew.exe) is running, users should first click the button of “Terminate” to abort the processing task or wait until the processing task is finished, and then close the GUI program. Users can not click the button of “Cancel” or click the closing icon on the up-right corner of the interface to close this GUI program when the called logical program is running.

The blank area of the right side will display the information printed by the command program, and the printed information will also be saved in a log file named as uavDSM_ddMMMyyyyHHmmss_log.txt. In the filename, dd are two digital numbers indicating a date, MMM are three English characters indicating an abbreviation of a month, yyyy are four digital numbers indicating a year, and HHmmss are digital numbers indicating hour, minute and second in the form of 24 hours.

All configuration options are saved in the configuration file, uavDSM.conf, through a certain type of form, in the course of running of program. The user name and password used in the parallel processing are saved in the file named as pwd.txt (If parallel processing is not used, there is no such file). The configuration file, uavDSM.conf, can be used in the function, Load Configuration. The file, pwd.txt, can be used to load user name and the password which are used in the parallel processing. The printed information in the course of running is saved in a log file named as uavDSM_ddMMMyyyyHHmmss_log.txt.

In the course of running of the program, the intermediate files, i.e., epipolar and disparity images are generated (these intermediate files can remain or be deleted according to the corresponding configuration option). A text-formatted file of point clouds with extension .xyz is generated for each image which has a suitable overlapping with other images, if point clouds are set to be saved. In this file, each line corresponds to one point. A mosaicked DSM covering the whole UAV block (if the default case is adopted, partitions are 1, which means only one computer is used for computation) is generated (the filename is set in output DSM file in output setting in the interface, and DSM is the one before interpolation). If point clouds are set to be saved, a list file named “points_part%d_of_%dparts_List.txt” is generated at the same time. The first %d from left to right in the filename is replaced by the serial number of the partition (partID) and the second %d is replaced by the number of partitions (Partitions). The list file includes all filenames of point clouds extracted from multi-view stereo matching which are distributed to the machine. The format of the list is as follows:

DSC00305_points.xyz

DSC00306_points.xyz

DSC00307_points.xyz

DSC00308_points.xyz

DSC00309_points.xyz

….

All these files are stored in the work directory. Except the above mentioned files, other files generated will be deleted in the end of the program.

A useful method to decide if arguments, parameters setting and command operations in this step are correct is to check if the objects in the generated epipolar pair are aligned strictly in horizontal direction. The vertical swipe function in the module of image viewer, imgViewer.exe, provided by MASI software can be used to view horizontal alignment. For more details, please refer to the program usage of imgViewer.exe.

The DSM result obtained in this step is the DSM before interpolation, which needs to pass to the next step (i.e., post processing for DSM) to yield the final highly dense DSM. Before post processing for DSM, if the un-interpolated DSM has big areas of mismatching (e.g., water body, cloud and snow covered area, area with repeated textures) which can be found by the manner of visual checking, the interactive tool for editing DSM/DEM (i.e., collectPolygons.exe) can be used to edit the DSM with big areas of mismatching. For more details, please refer to “Usage of the collectPolygons.exe program”.

Step 4 Post processing for DSM

In this step, these areas with no-data value in the above mosaicked DSM are interpolated, and outliers (or small spots of errors) in the resultant DSM after interpolation are removed. The output of this step is the product of dense DSM, which is generated by this software. The DSM product can be used in the different applications followed.

By clicking the command item, Post Processing, in the above menu, the GUI program is launched. The graphical interface of the program is as follows:

The concept of work directory is the same as that in automatic multi-view stereo and DSM generation. All files required in the course of ortho-rectification should be put in the work directory. In the section of input files in this interface, the file of interior parameters of camera and the list file of exterior elements are the same as those in the step of multi-view stereo and DSM generation. Thus, the directory, H:\uav_tests\MASI, is still set as the work directory in this step.

All options before the option, File of elevation source, are the same as those in the step of multi-view stereo and DSM generation. File of elevation source: the full file path of the external elevation source used in the ortho-rectification. The result in Step 4, post processing for DSM, is selected as the elevation source. The coordinate system of elevation file must be the same as that of exterior elements and the elevation file covers all the areas of these images. If third party elevation source is used, please make sure the sameness of coordinate system. Resolution: the image resolution (Unit: m) of the ortho-rectified image. The resolution of ortho-rectified image can not be better than the actual GSD (ground spacing distance). Ratios deleted in the X end and the Y end: the ratios need to be deleted in the two ends for X, Y directions, respectively. The remained center part of ortho-rectified image after deleted in ends is a portion with high quality.

Number of used threads: setting the number of used threads (CPU cores or the virtual CPU cores by using hyper-threading technology). For the cases where CPU cores are enough, we suggest that number of 4, 6 or 8 is set here.

After the above options in the interface are set, users can click “Save configuration” to save these options. These configuration options will be saved in a configuration file, uavOrtho.conf, and next time the options can be loaded from the configuration file by clicking “Load configuration”. The configuration file, uavOrtho.conf, is also stored in the work directory. If the button of “Execute” is clicked, the configuration file will also be generated in the same way.

In the output setting, users should set the output directory and the filename of the ortho-mosaicked image. In the output directory, the ortho-rectified image for each image will be stored. The resultant images after ortho-rectification are named in a specific rule: image ID_ortho.tif. The data type of ortho-rectified images, as well as the number of bands, is the same as that of original images, e.g., unsigned 8 for original image, still unsigned 8 for ortho-rectified image. The file format of resultant images is TIFF. The output ortho-mosaicked image should also be stored in the output directory.

No-data value in the output setting is the value used to fill in the area without intensity value (area with no-data value) in the resultant ortho-image. The value is also called as invalid value or background value.

After finishing the setting of configuration options, input and output setting, users can click the button of “Execute”. The called program, uavOrtho.exe, starts to run and the required time to finish the whole procedure is different according to the size of the UAV block. In the course of computation, the buttons of “Execute” and “Cancel” on the bottom of the form is invalid. The two buttons can not be clicked in case they are clicked many times in the course of running. Users can learn the progress and the running status via the re-directed output printing (sometime printing may delay) on the right side. User can also check the information of resources (CPU, memory and Disks) possessed by the program from Task Manager and Resource Monitor of Windows OS. Once the execution of uavOrtho.exe is finished, a message box will pop up showing that the processing is closed successfully or terminated (aborted). After clicking OK button on the message box, the GUI program returns to the available status. Users can either click the button of “Cancel” on the bottom or click the closing icon on the up-right corner of the interface to close this GUI program. Or users can select and set new input and output to fulfill a new task.

User can click the button of “Terminate” on the bottom if they need to abort the processing (not the closing of GUI form). The GUI program, uavOrthoForm.exe, will kill the process of uavOrtho.exe which is being called. Killing the process of uavOrtho.exe may delay (less than one second or several seconds). Once the process is killed, a message box will pop up showing that the processing is terminated (aborted) or closed successfully. After clicking OK button on the message box, the GUI program returns to the available status. Users can either click the button of “Cancel” on the bottom or click the closing icon on the up-right corner of the interface to close this GUI program. Or users can select and set new input and output to fulfill a new task.

Attention: the button of “Cancel” on the bottom is only for closing the GUI program. If the called logical program (i.e., uavOrtho.exe) is running, users should first click the button of “Terminate” to abort the processing task or wait until the processing task is finished, and then close the GUI program. Users can not click the button of “Cancel” or click the closing icon on the up-right corner of the interface to close this GUI program when the called logical program is running.

The blank area of the right side will display the information printed by the command program, and the printed information will also be saved in a log file named as uavOrtho_ddMMMyyyyHHmmss_log.txt. In the filename, dd are two digital numbers indicating a date, MMM are three English characters indicating an abbreviation of a month, yyyy are four digital numbers indicating a year, and HHmmss are digital numbers indicating hour, minute and second in the form of 24 hours. The log file is also stored in the work directory.

All configuration options are saved in the configuration file, uavOrtho.conf, through a certain type of form in the course of running of program. The configuration file can be used in the action of loading configuration.

At the same time, a list file named ortho_List.txt is generated, which is saved in the output directory. The list file includes, in sequence, all filenames of ortho-rectified images whose order is the same as that in the list of exterior elements of images. The format of the list is as follows:

H:\uav_tests\MASI\orthoDir\DSC00305_ortho.tif

H:\uav_tests\MASI\orthoDir\DSC00306_ortho.tif

H:\uav_tests\MASI\orthoDir\DSC00307_ortho.tif

H:\uav_tests\MASI\orthoDir\DSC00308_ortho.tif

H:\uav_tests\MASI\orthoDir\DSC00309_ortho.tif

….

Step 6: Comparing surface height and change finding

By comparing DSMs of two different phases, the program can automatically find change of the surface height and calculate the value of changed height. Clicking the icon showing “Surface Change” in the main interface,

The operating interface of change of surface height is launched.

Attention: the datasets in the interface may probably be not the results from UAV images. The purpose of this step is to demonstrate how to operate.

The corresponding DSM files are selected in the sections of the previous DSM and the next DSM, respectively. After processing, the file of height difference is obtained. Users can set two parameters, the height threshold and no-data value.

The height threshold: If the height difference is less than the threshold, the difference is not interesting and it will be assigned as no-data value in the resultant file. For the case where the height difference is more than the threshold, the real difference will saved in the resultant file.

No-data: no-data is the value used to fill in the area without change of surface height (or invalid area) in the resultant file. No-data can be determined either from input DSM or from the right side list box.

Step 7: Automatic DSM2DTM

Clicking the icon showing “DSM to DTM” in the main interface,

The operating interface of automatic DSM2DTM is launched.

Attention: the datasets in the interface may probably be not the results from UAV images. The purpose of this step is to demonstrate how to operate.

The program adopts an algorithm with multilevel triangle-based ground filtering. It deletes the non-grounded objects like buildings, trees in DSM and transforms the DSM to DTM where the pure ground height is saved. The input file is DSM source and the output file is resultant DTM.

The meaning of parameters is as follows:

The predefined size: the defined size in which the lowest point is selected as a ground point; unit: the same as the grid spacing (cell size), e.g., 40.0. It means that the lowest point in a grid, whose sizes of width and height are both 40 meters (assuming the unit of grid spacing is meter), is taken as a ground point. The higher is the predefined size, the more levels in the pyramids are generated. Level number of pyramids decrease, then the built triangles is to be dense. This parameter is usually decided by the size of the biggest non-ground object in the terrain area. If the predefined size is less than the size of non-ground object, these objects will not be filtered out. On the contrary, if the predefined size is too large to maintain some terrain details. Thus, users need find a balance between filtering out big non-ground objects and maintaining the terrain details in the light of the requirements of applications.

Level, to which level the DSM is filtered. The higher is the level set, the less ground points are generated, the less terrain details remain, and the more efficient is the processing. The lower is the level set, the more ground points are generated, the more terrain details remain, and the less efficient is the processing. Usually the triangulation of mass points and searching a specific triangle in a huge triangulation is inefficient. The level set can not be higher than (the highest level - 1), level is 0-based, e.g., 3.

The threshold of distance: the threshold of distance from a point to the corresponding triangle. In the course of filtering, a point whose distance to its corresponding triangle is more than the threshold is deemed as a non-ground point, vice versa, e.g., 1.0 meter. If the value of threshold is low, there are less ground points which will survive in the filtering processing and there are also some risks that the true ground point may be filtered out by taking them as non-ground points.

The size of block: for processing large raster, block division is needed. Filtering is fulfilled, block by block, and its size must bigger than two times of overlapping size, e.g., 6144. Usually the default value is used. The reason that block division is used is that a large DSM will result in huge ground points which require huge computation for the filtering processing. The block division is a way which promotes computational efficiency and saves memory consumption.

The size of overlap between blocks: the size of overlap between the adjacent blocks, e.g. 128. The overlay ensures the rightness of border of each block. Usually the default value is used.

No-data: no-data is the value used to fill in the area without height value. Because each grid in raster file must be assigned a value, for some cases where there are no real height values in some areas, the concept of no-data is introduced. No-data can be determined either from input DSM or from the right side list box in this interface.

If there are some non-ground objects which are not filtered out, such as buildings and vegetation areas, or small areas of errors in the resultant DTM in this step, the interactive tool for editing DSM/DEM (i.e., collectPolygons.exe) can be used for editing. For more details, please refer to “Usage of the collectPolygons.exe program”.

Step 8: Height estimation of building / tree

Clicking the icon showing “Building height” in the main interface,

The operating interface of height estimation of building / tree is launched.

Attention: the datasets in the interface may probably be not the results from UAV images. The purpose of this step is to demonstrate how to operate.

The corresponding files are selected in the sections of the ground height (DTM) and the surface height (DSM), respectively. After processing, the file of building / tree height is obtained. Users can set two parameters, the height threshold and no-data value.

The height threshold: the height being less than the threshold is the value which is not interesting or the possible existing errors. The value will be replaced by no-data in the file of building / tree height. The height being more than the threshold is an interesting and reliable value, which as a true value will be saved in the file of building / tree height.

No-data: no-data is the value used to fill in the area without building / tree (or invalid area) in the resultant file. No-data can be determined either from input DSM or from the right side list box.

Step 9: Applications

Clicking the icon showing “App” in the main interface,

The following menu will pop up.

Currently two applications modules are included, i.e., volume calculation and automatic extraction of building attributes in the light of buildings footprints.

Volume calculation: a polygon in the vector form in which volume is calculated and highly dense DSM are given, the module estimates the volume of objects (e.g., sands, small stones, coal, mineral particles and garbage) occupying the area. For more details, please refer to “Usage of the calVolume.exe and calVolumeForm.exe programs”. Users can click the command item, Polygons Drawing Tool, in above menu to launch the interactive tool, collectPolygons.exe, to collect the polygon and save it.

Automatic extraction of building attributes: a vector file with ERSI shape format, which including multiple polygons of building footprint, and the corresponding nDSM dataset are given, the module automatically extracts the center position (x, y coordinates) of building, area of ground, height of building, number of layers, and construction area of building for each building. Moreover, these extracted values are set as new attributes for these polygons. nDSM can either be obtained by subtracting DTM from DSM in the module of buildingHeightForm.exe, or be obtained by calculating difference between two DSMs in the module of surfaceChangeForm.exe. The former can be used to extract building height from the bottom to top, number of layers, etc., while the latter can be used to automatically find new, illegal, added, demolished, or height-exceeded buildings and extract their attributes, such as position, height, layers, ground area, and construction area of these found buildings. For more details, please refer to “Usage of the extBldgAttributes.exe and extBldgAttributesForm.exe programs”. Users can click the command item, Polygons Drawing Tool, in above menu to launch the interactive tool, collectPolygons.exe, to collect polygons on nDSM and save them to a vector file. Polygons of building footprint in vector file can also be from third party source. For both cases, it is required that building footprints are accurately aligned with the actual boundaries of buildings. And, the vector file including polygons and the nDSM file should be geo-coded and the underlying coordinate system is the same.

Step 10: 3D modeling with true color textures

Clicking the icon showing “Textured 3D Model” in the main interface,

The following menu will pop up.

By clicking the command item, textured 3D modeling, the GUI program is launched as follows:

$说明: D:\VisionOnSky\LOGO\空视\logo.png$
	VisionOnSky Co., Ltd.
	www.visiononsky.com